Dynamic Skyline Computation with LSD Trees ^†

Abstract

Given a set of high-dimensional feature vectors $S\subset {\mathbb{R}}^n$, the skyline or Pareto problem is to report the subset of vectors in S that are not dominated by any vector of S. Vectors closer to the origin are preferred: we say a vector x is dominated by another distinct vector y if x is equally or further away from the origin than y with respect to all its dimensions. The dynamic skyline problem allows us to shift the origin, which changes the answer set. This problem is crucial for dynamic recommender systems where users can shift the parameters and thus shift the origin. For each origin shift, a recomputation of the answer set from scratch is time intensive. To tackle this problem, we propose a parallel algorithm for dynamic skyline computation that uses multiple local split decision (LSD) trees concurrently. The geometric nature of the LSD trees allows us to reuse previous results. Experiments show that our proposed algorithm works well if the dimension is small in relation to the number of tuples to process.

1. Introduction

Dynamic skyline queries appear as common query types of recommender systems: walking tourists querying for restaurants with respect to price and distance, route planning systems on dynamic routing networks involving temporal congestions or road closures (e.g., [1]), online shopping portals with dynamically responsive input masks such as sliders for prices, sizes, and so on. All these use cases have in common that often a user does not issue a single query, but a sequence of queries, where successive queries only have slightly different parameters (movement of the tourists, the time passed in temporal routing networks, and the sliders of a webpage input mask). A naïve approach is to start the computation of the skyline set each time from scratch. Dynamic skyline algorithms provide fast results for recurring queries of the same kind by caching [2]. The key to dynamic skyline is a generalized mapping of the input data to the feature vector space. Instead of reconstructing this mapping for each query, it stays the same during recurring queries. Effectively, the vector space is parameterized by a query vector, i.e., it is shifted by an offset.

Let us examine a concrete example with the feature vectors $p_1=(55,90),p_2=(10,30),p_3=(70,70),p_4=(40,70),p_5=(5,80),p_6=(55,45),p_7=(90,50),p_8=(52,10),p_9=(25,20)$. The answer to the classic skyline problem (the query vector is $q=(0,0)$) is the skyline set $\{p_5,p_2,p_9,p_8\}$, because all points in this set are incomparable (i.e., none of them dominates each other), while the remaining elements such as $p_1$ are dominated by one of the elements of the skyline set (e.g., $p_2$ dominates $p_1$). In the classic skyline problem, we regard vectors closer to the origin as better than others that are farther away in all dimensions. A geometric interpretation is to connect the skyline set with line segments such that all points to the upper right of these line segments are dominated. See Figure 1 for a geometric interpretation, which we keep as a running example in this article.

The skyline set changes if we shift the origin as we are allowed to do in the dynamic skyline problem. Let us select $q=(45,55)$ as an example. Then the skyline set is $\{p_4,p_6,p_7\}$, and the answer remains unchanged when shifting q to $(40,55)$ or $(45,50)$. For instance, $p_1$ is dominated by $p_4$ because $p_4-q=(-5,10)$ and $p_1-q=(10,30)$, and the absolute value of the difference in both dimensions is smaller for $p_4$ than for $p_1$. With

Table 1. Feature vectors of our running example with skyline set (with respect to $q_0$) and dynamic skyline set for three different query points $q_1,q_2$, and $q_3$. The columns $q_i-p_j$ show the absolute difference between a query vector $q_i$ and a feature vector $p_j$ in each dimension. We show in the column dom by which vector the vector in the respective row got dominated; here a dash (−) says that this vector is not dominated by any vectors of the input set. Dominated vectors are not part of the skyline set.

Vector ${\mathit{p}}_{\mathit{j}}$	${\mathit{q}}_0=(0,0)$	${\mathit{q}}_1=(45,55)$		${\mathit{q}}_2=(40,55)$		${\mathit{q}}_3=(45,50)$
	dom.	${\mathit{q}}_1-{\mathit{p}}_{\mathit{j}}$	dom.	${\mathit{q}}_2-{\mathit{p}}_{\mathit{j}}$	dom.	${\mathit{q}}_3-{\mathit{p}}_{\mathit{j}}$	dom.
$p_1$ = $(55,90)$	$p_5$	(10, 35)	$p_6$	(15, 35)	$p_4$	(10, 40)	$p_4$
$p_2$ = $(10,30)$	−	(35, 25)	$p_6$	(30, 25)	$p_4$	(35, 20)	$p_4$
$p_3$ = $(70,70)$	$p_4$	(25, 15)	$p_6$	(30, 15)	$p_4$	(25, 20)	$p_4$
$p_4$ = $(40,70)$	$p_2$	(5, 15)	−	(0, 15)	−	(5, 20)	−
$p_5$ = $(5,80)$	−	(40, 25)	$p_6$	(35, 25)	$p_4$	(40, 30)	$p_4$
$p_6$ = $(55,45)$	$p_2$	(10, 10)	−	(15, 10)	−	(10, 5)	−
$p_7$ = $(90,50)$	$p_2$	(45, 5)	−	(50, 5)	−	(45, 0)	−
$p_8$ = $(52,10)$	−	(7, 45)	$p_4$	(12, 45)	$p_4$	(7, 40)	$p_4$
$p_9$ = $(25,20)$	−	(20, 35)	$p_4$	(15, 35)	$p_4$	(20, 30)	$p_4$

, it is easy to verify the calculation.

It is easy to see that a naïve computation of the skyline set involves comparing each vector with all other vectors. Assuming that we can compare two vectors in ${\mathbb{R}}^n$ in $\mathcal{O}\left(n\right)$ time, we can compute the skyline set in $\mathcal{O}\left(n{m}^2\right)$ time, where m is the input set size. The question we pose in this article is whether we need to compare all elements with each other, and whether a recomputation for a different query point is needed. To this end, we review an algorithm using a geometrical data structure, called the LSD tree, which leverages geometric distributions and is in favor of pruning subsets of the input in certain cases. This tree data structure also gives us means for reasoning about when a recomputation of the dynamic skyline set is necessary.

1.1. Dynamic Skylines

Consider a data set $\mathcal{D}$ and a scoring function $f:\mathcal{D}\to {\mathbb{R}}^n$. We call an elements $x\in \mathcal{D}$ a tuple, and $f\left(x\right)$ its respective (feature) vector. (Instead of directly working on feature vectors, by mapping the input element to feature vectors, we can support the case that multiple elements can have the same feature vector, i.e., we allow the geometrical representation of tuples by not necessarily distinct feature vectors.) Let us say that we have a query vector $q\in {\mathbb{R}}^n$ and want to compute the dynamic skyline of q [3], i.e., we want to find all vectors of $f\left(\mathcal{D}\right)$ that are not dominated by any other vectors with respect to q. We say a vector dominates another vector if its distance to q has an equal or lower value than the other vector’s distance to q in all respective coordinates, and a strictly lower value in at least one coordinate. According to this definition, the query vector $q\in {\mathbb{R}}^n$ always dominates all other vectors, but q is not part of $f\left(\mathcal{D}\right)$ in general. We call q an offset vector because it moves the best location $0\in {\mathbb{R}}^n$ of the classical skyline problem to its own location (e.g., we translate the vector space by ${\mathbb{R}}^n\to {\mathbb{R}}^n,v\mapsto v+q$). Vectors are often illustrated as points of the ${\mathbb{R}}^n$ vector space equipped with the $l_1$ norm, given by ${\left|x\right|}_{l_1}={\sum }_{i=1}^n\left|x_i\right|$ for $x=(x_1,\dots ,x_n)\in {\mathbb{R}}^n$. The latter is useful, as two distinct vectors are incomparable with respect to domination when their norm is equal. For instance, $(1,4)$ and $(4,1)$ are feature vectors in ${\mathbb{R}}^2$ with ${\left|(1,4)\right|}_{l_1}=5={\left|(4,1)\right|}_{l_1}$ and therefore both are incomparable.

1.2. Structure of the Paper

In what follows, we tackle the problem of answering dynamic skyline queries. Given a data set $\mathcal{D}$, we precompute a data structure based on LSD trees, such that for a sequence of query points $q_1,q_2,\dots$, we can use this data structure to quickly answer ${\mathcal{S}}^{q_1}\left(\mathcal{D}\right)$, and reuse this answer for computing ${\mathcal{S}}^{q_2}\left(\mathcal{D}\right)$ if possible.

To this end, we study a modification of the Skyline Breaker algorithm [4] using LSD trees. While the original paper suggests an algorithm for classic skyline computation, we extend its usage to dynamic skyline queries. Therefore, we review the main ideas before addressing our new enhancements. The rest of our paper is organized as follows: Section 2 contains an overview of related work with regards to skyline computation on tree data structures. We introduce basic definitions and helpful lemmata in Section 3 with the sLSD tree structure in Section 4 before focusing on the algorithm itself in Section 5. There, we present the problem and highlight different techniques for optimizations. Finally, Section 6 reflects possible performance speedups and slowdowns.

2. Related Work

Computing the skyline by means of geometric representation of data already produces many results. Kossmann et al. [5] proposed nearest neighbor (NN) search operating on an R*-tree [6] structure. This idea was extended by Papadias et al. [7], featuring the branch-and-bound skyline (BBS) algorithm. They were also the first to propose the dynamic skyline problem [3]. Using the M-tree [8] as a geometrical representation for the data, Chen and Lian [9] reasoned about pruning techniques for fast dynamic skyline computation. In addition to providing a dynamic skyline algorithm, Sacharidis et al. [2] proposed caching methods to speed up computation in subsequent queries. Recent research approaches can compute the dynamic skyline for massive datasets [10], work in parallel [11], or work on distributed systems [12]. Along with new proposals for skyline computation, the focus of recent research projects tends to parallelization of classic skyline algorithms. Selke et al. [13] discussed a variant of block-nested loop (BNL) using the Lazy List [14] data structure. Techniques like optimistic locking are shown as a good trade-off between redundant synchronization steps and race conditions. By doing so, they manage to successfully shrink the sequential fraction of BNL. A parallelized variant of BBS has been proposed by Im et al. [15]. Obviously, dynamic skylines and parallel skyline algorithms are part of an active research area. The combination of both fields is the topic of this paper.

A related problem, but not studied in this paper, is to allow the underlying dataset to be dynamic. Considering dynamic datasets, Essiet et al. [16] studied multi-objective optimization; Hu et al. [17] proposed routing on dynamical environments; and Gulzar et al. [18] gave an algorithm for computing the skyline by pruning and selecting superior local skylines (see also references within the PhD thesis of Alami [19]).

3. Preliminaries

As in common calculus, ${\pi }_i:{\mathbb{R}}^n\to \mathbb{R}$ denotes the projection to the i- th coordinate, i.e.,  $\mathrm{for}\mathrm{all}(v_1,\dots ,v_n)\in {\mathbb{R}}^n$

$${\pi }_i(v_1,\dots ,v_n)=v_i\mathrm{with}i\in \left\{1,\dots ,n\right\}.$$

We call a function $f:\mathcal{D}\to {\mathbb{R}}^n$ a scoring function. Here $\mathcal{D}$ is a subset of an arbitrary universe. Further, for a scoring function $f:\mathcal{D}\to {\mathbb{R}}^n$, we call $v:=f\left(x\right)\in {\mathbb{R}}^n$ the feature vector of $x\in \mathcal{D}$. So that there is no possibility for confusion, we write $\left|S\right|$ to express the cardinality of a set S, i.e., the number of elements of S.

Scoring and Dominance. In what follows, we first consider a fixed offset $q\in {\mathbb{R}}^n$, which represents our query point. The motivation for the definitions is that f and q induce a strict weak ordering ${\prec }_q$ on $\mathcal{D}$, called a dynamic scoring of $\mathcal{D}$. For two vectors $v,w\in {\mathbb{R}}^n$, we call v better than w with respect to q if there exists a $j\in \left\{1,\dots ,n\right\}$ such that $|{\pi }_j\left(v-q\right)|<|{\pi }_j\left(w-q\right)|$ and $|{\pi }_i\left(v-q\right)|\le |{\pi }_i\left(w-q\right)|$ for every $i\in \left\{1,\dots ,n\right\}$. We write $v{\prec }_{q}w$ for short. Analogously, we write $x{\prec }_{q}y$ for two tuples $x,y\in \mathcal{D}$ if and only if $f\left(x\right){\prec }_{q}f\left(y\right)$ holds. We can put this statement the other way around: if there is no $j\in \{1,\dots ,n\}$ such that $|{\pi }_j\left(f\left(x\right)-q\right)|<|{\pi }_j\left(f\left(y\right)-q\right)|$, then either $f\left(x\right)=f\left(y\right)$ or y dominates x.

For instance, given two data points $d_1$ and $d_2$ with $f\left(d_1\right)=p_1=(55,90)$ and $f\left(d_2\right)=p_2=(10,30)$, then ${\pi }_1\left(p_1\right)=55$ and ${\pi }_2\left(p_1\right)=90$. For $q_0=(0,0)$, we have that $p_2{\prec }_{q_0}{p}_1$, whereas for $q^{\prime }=(100,100)$, $p_1{\prec }_{q^{\prime }}{p}_2$ because $|{\pi }_1(p_1-q^{\prime })|=45<90=|{\pi }_1(p_2-q^{\prime })|$ and $|{\pi }_2(p_1-q^{\prime })|=10<70=|{\pi }_2(p_2-q^{\prime })|$.

Formal Problem Definition. The dynamic skyline parameterizes the classic skyline by feature vector q as an offset [3]. More concretely, the dynamic skyline set ${\mathcal{S}}^q\left(\mathcal{D}\right)\subset \mathcal{D}$ holds the condition

$$\begin{array}{cc}\hfill x\in {\mathcal{S}}^q\left(\mathcal{D}\right):\iff & \forall o\in \mathcal{D},\mathrm{there}\mathrm{either}\mathrm{exists}\mathrm{a}j\in \left\{1,\dots ,n\right\}\hfill \\ \hfill & \mathrm{such}\mathrm{that}|{\pi }_j\left(f\left(x\right)-q\right)|<|{\pi }_j\left(f\left(o\right)-q\right)|\hfill \\ \hfill & \mathrm{or}f\left(x\right)=f\left(o\right)\hfill \\ \hfill \iff & \mathrm{there}\mathrm{exists}\mathrm{no}o\in \mathcal{D}\mathrm{with}o{\prec }_{q}x.\hfill \end{array}$$

We also call $x\in {\mathcal{S}}^q\left(\mathcal{D}\right)$ a tuple that is not dominated by any other tuple of $\mathcal{D}$, or more briefly, we call x a skyline point (if the context with $\mathcal{D}$ and q is clear). In the following, when the sense is obvious, we identify a tuple $x\in \mathcal{D}$ with its value $f\left(x\right)$, e.g., we do not explicitly mention any notion of f. So when speaking about coordinates of x, we actually mean the coordinates of $f\left(x\right)$.

Lemma 1.

Let $U_1,\dots ,U_k\subset \mathcal{D}$ be a cover of $\mathcal{D}$, i.e., ${\bigcup }_{i=1}^k{U}_i=\mathcal{D}$. Then ${\mathcal{S}}^q\left(\mathcal{D}\right)\subset {\bigcup }_{i=1}^k{\mathcal{S}}^q\left(U_i\right).$

Proof. 

Let $x\in {\mathcal{S}}^q\left(\mathcal{D}\right)\cap U_i$, then for all $o\in U_i\subset \mathcal{D}$, there either exists a $j\in \left\{1,\dots ,n\right\}$ such that $|{\pi }_j\left(f\left(x\right)-q\right)|<|{\pi }_j\left(f\left(o\right)-q\right)|\mathrm{or}f\left(x\right)=f\left(o\right)$. Hence $x\in {\mathcal{S}}^q\left(U_i\right)$.    □

4. The sLSD Tree

The LSD tree is a hybrid tree that has bucket nodes as leaves and directory nodes as internal nodes. Dictionary nodes have always two children, and therefore an LSD tree is a full binary tree. Bucket nodes are the actual nodes that store the data.

In the beginning, the tree consists of a single bucket node. Whenever a bucket node contains more than M elements, this bucket node will be split. In our case, the split point is determined by the median of all elements according to one dimension. After the split, we have a left and a right bucket node: the left bucket contains those elements that have smaller values than the split position in the split dimension. The information about the split is saved in a new directory node that replaces the previous bucket node and adopts both new bucket nodes as children. We can consider a dictionary node as a separation of its two bucket children with a hyperplane. While bucket nodes store the actual data, a directory node only saves the split position and the two bucket nodes split by this hyperplane. Figure 2 gives an example.

Exactly as the kd tree [20], the split dimension can be determined by the tree-depth of the bucket node that was split. In another perspective, the LSD tree partitions data of ${\mathbb{R}}^n$ into disjoint cuboids where each cuboid contains all elements of exactly one bucket. Each bucket can therefore be represented as a cuboid whose coordinates are saved in the ancestor nodes’ split positions. We call this particular cuboid the bounding cuboid of the bucket. Computing the nearest neighbor (NN) of an arbitrary point $q\in {\mathbb{R}}^n$ is done by the following steps: First, traverse the tree in a top-down manner in order to locate the bucket in which q would belong. Next, recursively climb the tree upwards while scanning the sibling node at each depth for near points. If this sibling is an internal node, we additionally scan its subtree recursively. Henrich [21] proposes an algorithm that takes the distances to already found points for a bound to stop the naïve search prematurely. The difference between the LSD tree and our sLSD tree is the lack of external directory nodes as we restrict the problem for in-memory use cases.

4.1. Dealing with Skewed Distributions

The sLSD tree splits an overfull bucket by sorting its elements in a certain dimension and taking the median. For a kd tree, the split dimension of a bucket is usually the split dimension of its parent node, shifted by one, i.e., if the parent node has a split dimension d, then the bucket gets split at dimension $(d+1)modn$. In the worst-case scenario, all elements share the same value in this dimension. A split would therefore create two new buckets: one assigned with the entire content and the other left empty. Henrich [22] avoids this bad behavior by simply iterating over the split dimension until he finds different values of the elements in this dimension. For termination, we just have to check if at least two elements have different feature vectors. Another method would take those dimensions into account as a split dimension, for which only a few objects’ values collide with the median value in this dimension. If there are multiple dimensions with the same number of minimal collisions, then we take the dimension with the highest variance of feature vector values out of this remaining set. This additional filter tries to hinder any bucket from becoming too elongated in specific dimensions.

4.2. Complexity Analysis

As a member of the kd tree family, the sLSD tree shares complexity traits with its ancestor. In fact, the kd tree is a specialization of the sLSD tree with a maximum bucket size of $M=1$. On the one hand, for negligibly small M, the sLSD tree provides operations like inserting or locating an element in $\mathcal{O}\left({log}_2\left|\mathcal{D}\right|\right)$ average and $\mathcal{O}\left(\right|\mathcal{D}\left|\right)$ worst time. On the other hand, an $M\ge \left|\mathcal{D}\right|$ lets the sLSD tree store the entire data in a single bucket. Hence, the time complexity is based on the data structure used for the buckets. For the other cases with $1\ll M<\left|\mathcal{D}\right|$, we have to take care with the more complex insertion step. If we assume contrarily to Section 4.1 that the median value is always unique under all considered objects during a split, then any split of an overfull bucket at the median creates two buckets, each containing at least $⌊M/2⌋$ elements. Moreover, our algorithm will not remove any element of the sLSD tree. After inserting $M+1$ elements, there is at no time any bucket with less than $⌊M/2⌋$ elements. Let us use the random variables $b_i\in [⌊M/2⌋,M)$ with $i=1,\dots ,k$ for counting the elements of each bucket of an sLSD tree with k leaves. These variables satisfy the constraint that ${\sum }_{i=1}^k{b}_i=\left|\mathcal{D}\right|$. In the average case, the occupancy rate is distributed uniformly over all intervals, hence we get $\frac{3}{4}kM=\left|\mathcal{D}\right|$, i.e., $k=\frac{4\left|\mathcal{D}\right|}{3M}=\mathcal{O}\left(\frac{\left|\mathcal{D}\right|}{M}\right)$. Therefore, the sLSD tree has an average depth of $\mathcal{O}({log}_2\frac{\left|\mathcal{D}\right|}{M})$. The costs for inserting an element are a combination of

• finding the bucket in which the element shall be inserted; the time is logarithmic in the depth of the tree; and 
• splitting this bucket if it is overfull. The split is done by

  (a)

  sorting the elements in one dimension that takes $\mathcal{O}\left(M{log}_{2}M\right)$ time, and

  (b)

  creating a new bucket that takes $\mathcal{O}\left(M\right)$ time for moving half of the elements to the new bucket.

  Overall, a newly created bucket can take $M/2$ new elements on average before it has to be split. Hence, this step takes $\mathcal{O}\left({log}_{2}M\right)$ amortized time.

To sum up, insertion takes $\mathcal{O}\left({log}_2\left|\mathcal{D}\right|\right)$ amortized time on average. In the worst case, all buckets have minimal occupancy, i.e., $b_i=⌊M/2⌋$. Thus, we have $k=\frac{2\left|\mathcal{D}\right|}{M}$. Let us take an empty sLSD tree. If we insert a sequence of elements $\mathcal{D}$ that are strictly ordered, we obtain, as for kd trees, a caterpillar tree with the worst-case depth of $\mathcal{O}\left(\mathcal{D}\right)$ if we drop the constraint of unique values for determining the split dimension (i.e., all feature vectors must not have the same value in the split dimension). It is easy to see that the distribution of elements will affect not only the number of buckets, but also the time complexity. Fortunately, Henrich [23] provides a heuristic online strategy that tries to redistribute the occupancy. His strategy tries to shift the split position of the parent node of an overflowing bucket in order to prevent the split of the bucket. However, we have to take into account what kind of node the sibling of the brimful bucket is:

• If the sibling is a bucket that has space left, we just shift the parent node’s split position such that the number of elements gets rebalanced.
• If the sibling is a subtree that can adopt a new element without splitting one of its children, we shift again the split position of the bucket’s parent node. This time, however, we also have to recompute the split information of the internal nodes of this subtree.

If none of these conditions can be applied, we declare the bucket’s parent node as overfull and try to shift the split position of its respective parent. Hence, if a local redistribution is not possible, we recursively go up the tree and reconstruct the split information. Regarding aggressive shifting strategies, there is a certain trade-off between overall bucket utilization and the number of directory nodes [23].

4.3. Geometrical Characteristics

In the context of the Skyline Breaker algorithm, we are particularly interested in catching skyline points out of the sLSD tree quickly. Therefore, we need to compute the bounding cuboids for nodes with arbitrary depth. For that, we denote with $\mathrm{depth}\left(e\right)$ the depth of any node e of an sLSD tree. The following lemmas will help us to solve this task.

Lemma 2

([4]). Let b be an arbitrary leaf node. To determine the bounding cuboid of b, we need to examine n ancestors of b.

Let us recall that $q=(q_1,\dots ,q_n)$ is our query vector.

Lemma 3.

Let $o=(o_1,\dots ,o_n)\in f\left(\mathcal{D}\right)$ be a feature vector. The pairwise disjoint, non-bounded cuboids $D_1:=(q_1+d_1,\infty )\times (q_2+d_2,\infty )\cdots \times (q_n+d_n,\infty ),D_2:=(-\infty ,q_1-d_1)\times (q_2+d_2,\infty )\cdots \times (q_n+d_n,\infty ),\cdots ,D_{2^n}:=(-\infty ,q_1-d_1)\times \cdots \times \left(-\infty ,q_n-d_n)\right)$ do not contain any skyline point, where $d:=\left(\right|q_1-o_1|,\dots ,|q_n-o_n\left|\right)$. Furthermore, ${\mathbb{R}}^n\setminus {\bigcup }_{j=1}^{2^n}{D}_j$ is connected.

Proof. 

For every feature vector $u\in {\bigcup }_{j=1}^{2^n}{D}_j$, we have $|{\pi }_j\left(u-q\right)|>|{\pi }_j\left(o-q\right)|$ for every $j=1,\dots ,n$, and thus u is dominated by o. See Figure 3 for an illustration. Since all cuboids $D_j$ are pairwise disjoint, and each point v that is not dominated by o is not in ${\bigcup }_{j=1}^{2^n}{D}_j$, we conclude that ${\mathbb{R}}^n\setminus {\bigcup }_{j=1}^{2^n}{D}_j$ is connected.    □

Lemma 4.

Both the siblings of $B_q$ and the descendants of $B_q$’s ancestors with depth at least $\mathrm{depth}\left(B_q\right)-n+1$ can contain skyline points.

Proof. 

Without loss of generality, we assume $\mathrm{depth}\left(B_q\right)>n-1$. Let p be the ancestor of $B_q$ with a depth of $\mathrm{depth}\left(B_q\right)-n$, and let c be the child that is the ancestor of $B_q$. The node c has a depth of $\mathrm{depth}\left(B_q\right)-n+1$. The descendant directory nodes of c and c itself cannot have any split position farther away from q than p. Otherwise, the bucket $B_q$ would be at a different location than q. Because of the definition of $B_q$, no directory node that is both ancestor of $B_q$ and descendant of p has the split-dimension of p. Hence, there is no leaf node of c whose bounding cuboid is contained (entirely) in D (due to Lemma 3 with $B_o\leftarrow B_q$).    □

5. The Algorithm

While the original algorithm restricts itself to non-negative feature vectors, we had to relinquish this constraint in order to parameterize the skyline computation with an arbitrary offset q. Therefore, we reintroduce basic original concepts while providing our alterations.

5.1. The Dynamic Skyline Breaker Algorithm

We generate an sLSD tree for each thread. Each thread starts with the SBDyn (Algorithm 1) immediately after its tree is filled with the input data. Firstly, we search the nearest neighbor of q with respect to the $l_1$ norm in terms of bucket nodes by Henrich ’s distance-scan algorithm [22]. While locating this node, at most $2^n-1$ buckets have to be examined. Because we have not yet found any dominator, these buckets might contain skyline points in addition to a. Hence, we collect these buckets for local skyline computation and call them ${\left\{B_i\right\}}_{i\in I}$ for some I. Let $a\in f\left(\mathcal{D}\right)\subset {\mathbb{R}}^n$ be a NN of q, and let $B_q$ denote the bucket in which q would be added. Now, having Lemma 2 in mind, we start to traverse the tree reversely, counting our steps upwards. We initialize a vector $v\leftarrow q$ that keeps track of the geometric position. While climbing the tree upwards, we examine the sibling c of the node we came from. If we have not counted upwards to the dimension n, at least one of v’s coordinates is zero (see Lemma 4). Therefore, we cannot discard c. In other words, we need to traverse c and collect all of its buckets. When we have reached the number n with our counting, one of the descendants of c could be discarded. That is because some descendants’ buckets might be contained in the region without skyline points considered in Lemma 3. Therefore, we start a new task that traverses c, done by the tree clipping described in Section 5.3 and Algorithm 2. We have traversed the tree when we have reached the root node while climbing up the tree from $B_q$. The last step is the allocation of the global skyline. For overview, we present a flow chart of the algorithm in Figure 4.

Algorithm 1 Dynamic Skyline Breaker: The ternary ? operator has the same semantic as in C/C++, i.e., $a?b:c$ means if a then b else c.

1: function sbDyn($q\in {\mathbb{R}}^n$)

2:    $Q\leftarrow \mathrm{Set}(\varnothing )\subset \mathcal{D}$

3:    $(a,B_q,{\left\{B_i\right\}}_i)\leftarrow NearestNeighbor\left(q\right)$

4:    $Assert(a={\mathrm{minarg}}_{a\in f\left(\mathcal{D}\right)}{∥a-q∥}_{l_1})$

5:    $Q.\mathtt{insert}\left({\mathcal{S}}^q\left(B_q\cup {\left\{B_i\right\}}_i\right)\right)$

6:    $N\leftarrow B_q$

7:    for $i\leftarrow 0\mathrm{to}n$ do

8:        $c\leftarrow N,N\leftarrow N.\mathtt{parent}$

9:        $c\leftarrow N.\mathtt{left}=c?N.\mathtt{right}:N.\mathtt{left}$

10:        $Q.\mathtt{insert}\left({\mathcal{S}}^q\left(c\right)\right)$

11:    while $N\ne root$ do

12:        $c\leftarrow N,N\leftarrow N.\mathtt{parent}$

13:        $c\leftarrow N.\mathtt{left}=c?N.\mathtt{right}:N.\mathtt{left}$

14:        if $c\mathrm{is}\mathrm{a}\mathtt{BucketNode}$ then

15:           $Q.\mathtt{insert}\left({\mathcal{S}}^q\left(c\right)\right)$

16:        else

17:           $v\leftarrow q\in {\mathbb{R}}^n$

18:           $w\leftarrow \mathrm{split}\mathrm{position}\mathrm{of}N$

19:           $v.\left(\mathrm{split}\mathrm{dimension}\mathrm{of}N\right)\leftarrow w$

20:           clip($q,c,v,Q$)

21:    return ${\mathcal{S}}^q\left(Q\right)$

Algorithm 2 Clipping the Tree

1: function clip($q\in {\mathbb{R}}^n,N:\mathrm{node},v\in {\mathbb{R}}^n,Q\subset \mathcal{D}$)

2:    $w\leftarrow \mathrm{split}\mathrm{position}\mathrm{of}N$

3:    $d\leftarrow \mathrm{split}\mathrm{dimension}\mathrm{of}N$

4:    $c\leftarrow q.d<w?N.\mathtt{left}:N.\mathtt{right}$

5:    if $c\mathrm{is}\mathrm{a}\mathtt{BucketNode}$ then

6:        $Q.\mathtt{insert}\left({\mathcal{S}}^q\left(c\right)\right)$

7:    else

8:        $\mathrm{clip}(q,c,v,Q)$

9:    $v^{\prime }\leftarrow v$

10:    $v^{\prime }.d\leftarrow w$

11:    if $a\neg {\prec }_q{v}^{\prime }$ then

12:        $c\leftarrow c=N.\mathtt{left}?N.\mathtt{right}:N.\mathtt{left}$

13:        if $c\mathrm{is}\mathrm{a}\mathtt{BucketNode}$ then

14:           $Q.\mathtt{insert}\left({\mathcal{S}}^q\left(c\right)\right)$

15:        else

16:           $\mathrm{clip}(q,c,v^{\prime },Q)$

5.2. Skyline in a Bucket

For local skyline calculations, we can modify any skyline algorithm to work with an offset of q. We did this with a simple BNL-based algorithm that works directly on the bucket, while subtracting the offset ahead of comparison.

5.3. The Tree Clip Algorithm

Algorithm 2 works with a divide-and-conquer (DC) strategy. We start with a directory node and the split information of its parent node. It is advisable to hold these data in a variable $v\in {\mathbb{R}}^n$ by setting the coordinate entry of p at the split dimension to the split position, while all other coordinates are kept identical to q’s position for a start. The other child c of our current directory node can be illustrated as a cuboid with the corner point $e_c=v$ that is closest to q. Therefore, while we traverse the children of our current directory node, we note down the split information in v until we have gained coordinates for all dimensions. With this information represented by the split point $v\in {\mathbb{R}}^n$, using the condition $a{\prec }_{q}v$, we now check whether discarding the child node c is feasible. If the condition holds, we know that all elements contained in the cuboid of c are dominated by a. Thus, we discard the node c, even if it is a directory node. Otherwise, we have to traverse the right node recursively, again denoting its split data. Figure 5 gives a geometric interpretation of an LSD tree, where we have located the bucket of q and a bucket containing a.

Remark 1.

According to Lemma 4, if the depth of the sLSD tree is less than $n-1$, we cannot make use of the clipping technique.

6. Analysis and Optimization

Let us retrace the versatile steps involved in the algorithm while analyzing the time complexity. At the start, locating $B_q$ takes $\mathcal{O}({log}_2\frac{\left|\mathcal{D}\right|}{M})$ time on average and $\mathcal{O}\left(\right|\mathcal{D}\left|\right)$ time in the worst case, according to Section 4.2. Subsequently, a is found while examining at most $2^n-1$ that takes $\mathcal{O}\left((2^n-1)M\right)$ time in the worst case. If we take a balanced sLSD tree for granted, the preparation step of Section 5.3, which moves upwards until reaching at least n different dimensions, will also take $\mathcal{O}\left(2^n\right)$ buckets into consideration. For local skyline computation of any bucket of size $\Theta \left(M\right)$, an algorithm like BNL [24] applied in Section 5.2 needs at least $\Omega \left(M\right)$ time (assuming that it is left to compare all feature vectors by a single remaining dimension at which they differ) and has $\mathcal{O}\left(n{M}^2\right)$ worst-case time complexity. The time taken by the actual tree clipping algorithm is heavily dependent on the distribution of split information and takes at least the time proportional to the remaining depth of the current node in the sLSD tree after the preparation step. By building our algorithm on the sLSD tree, we obtain a hybrid approach that employs both the DC strategy of Section 5.3 and some nested-loop algorithms of Section 5.2. Both strategies are weighted by the maximum bucket size M. On the one extreme, for $M=1$, we get a kd tree-like structure with singletons as buckets. According to the complexities of Section 4.2, setting $M=1$ creates a much larger tree structure and exploits entirely the DC strategy. On the other hand, taking any $M>\left|\mathcal{D}\right|$ puts the entire content in a single bucket and just evaluates the nested-loop algorithm. For taking advantage of both algorithms, a $M\in (1,\frac{\left|\mathcal{D}\right|}{2^{n+1}}]$ has to be chosen, dependent of the input data’s distribution. Obviously, there are data sets for which choosing a small or large M results in a speed-up. A best-case scenario for large M is a correlated data set with $\mathcal{O}\left(\right|U\left|\right)$ time complexity for the BNL of any subset $U\subset \mathcal{D}$. If the sLSD tree is balanced, we have a depth of $\mathcal{O}({log}_2\frac{\left|\mathcal{D}\right|}{M})$ and thus $\frac{\left|\mathcal{D}\right|}{M}$ leaves. Hence, the algorithm will take at most $\mathcal{O}\left(\right|\mathcal{D}|{log}_2\left|\mathcal{D}\right|)$ time. On the other hand, extremely anti-correlated data can pose the worst-case running time for the BNL. However, if the clipping condition holds for most of the time, we have for $\frac{\left|\mathcal{D}\right|}{M}\ge 2^n$ the best case running time $\mathcal{O}({log}_2\frac{\left|\mathcal{D}\right|}{M}-n)$ for the tree climbing algorithm that collects $\mathcal{O}\left(2^n\right)$ buckets of a balanced LSD tree in addition. For merging these buckets, we have $2^n\mathcal{O}\left(M\right)$ for a best-case and $2^{2^n}\mathcal{O}\left(M^2\right)$ for a worst-case time. Note that both times are independent of $\left|\mathcal{D}\right|$. Hence, we argue that the dimensional factor plays only a minor role for a reasonably large $\left|\mathcal{D}\right|>2^{2n}$.

Proposition 1.

The Skyline Breaker algorithm accesses the optimal number of nodes for computing the dynamic skyline set of q, i.e., no algorithm will solve the same problem while visiting fewer nodes on the same data structure without knowledge of the contents of any bucket.

Proof. 

Let us assume that our algorithm unnecessarily inspects a bucket u that can be safely pruned. The bucket u has a bounding rectangle with corner points $(u_0,\dots ,u_{2^n})$. Let $u_L$ be a best corner, i.e.,  $u_L{\prec }_q{u}_i$ for every $i=1,\dots ,2^n$. In the contrary, there has to be an element $v\in \mathcal{D}$ with $f\left(v\right)\prec u_L$. Otherwise, without further knowledge, we have to inspect the bucket u. We also know that the NN of q is always part of the skyline set. Hence, every algorithm that computes the dynamic skyline set has to access the bucket $B_a$ that would take a as a new element of the global skyline. Due to the fact that Algorithm 2 traverses the tree from $B_q$ upwards, we will encounter nodes in an ascending, weakly-sorted order with respect to its best corner. In other words, the algorithm will either find a bucket containing v before accessing u or find a bucket that dominates v. In both cases, we will not inspect the bucket u; we clip either a subtree containing u or u itself.    □

7. Comparison to Other Geometrical Data Structures

Our usage of the LSD tree is motivated by modern computer hardware architectures featuring large caches. While the well-known kd tree can be seen as a special case of an LSD tree with a bucket of size 1, large bucket sizes leverage cache effects: for the skyline computation within a bucket, we stick to a basic block-nested loop algorithm with a naïve time complexity of $\mathcal{O}\left(n{M}^2\right)$. However, this algorithm is practically fast if it operates on data within the cache size of the CPU. Additionally, parallel evaluation is easy since the dominance check for each element can be performed independently. In that sense, large bucket sizes shrink the height of the LSD tree compared to the kd tree while spending only a negligibly additional cost for the computation within the relatively large buckets. Another well-known geometric representation is an r tree [25]. Since the internal node of an r tree stores the complete geometric information about its bounding cuboid, computing this information is cheap compared to our approach with the need to have a node-to-ancestor path of length n to have the split information for all n dimensions. Nevertheless, skewed distributions of the input can cause inefficiencies like empty or mostly empty cuboids. It can also often happen that cuboids intersect such that it is harder to detect whether pruning of nodes can be done.

7.1. Improving the Clipping Condition

The tree clipping (Algorithm 2) could be extended to use a list of known skyline points instead of solely a (a as defined in Section 5.1). A large list ${\mathcal{S}}^{\prime }\subset \mathcal{S}$ of global skyline points reinforces the clipping in Section 5.3 by testing $o{\prec }_{q}p$ for any $o\in {\mathcal{S}}^{\prime }$ instead of only a. Thus, a skyline subset will provide a higher chance of clipping. Actually, we can already enlist some points while finding a; let us recall that the search for the NN of q can take at most $2^n-1$ buckets ${\left\{B_i\right\}}_i$ into account. Due to the distance-scan algorithm [22], each of these buckets is a neighbor of $B_q$. A potentially huge number arises from the number of orthants that are spawned by q (see Lemma 3). Because the bounding cuboids form a connected region that contains q, there exists a certain area in which we can be sure that local skyline points belong to the global dynamic skyline ${\mathcal{S}}^q\left(\mathcal{D}\right)$. More precisely, we compute the maximal cube (i.e., a cuboid with equal sides) that is contained in the union of all buckets. We denote its side length with s. Then any $o\in {\mathcal{S}}^q\left({\bigcup }_i{B}_i\right)$ with ${∥o-q∥}_{l_1}\le s$ is in the global skyline ${\mathcal{S}}^q\left(\mathcal{D}\right)$. If there is a $p{\prec }_{q}o$, then ${∥p-q∥}_{l_1}\le s$ and hence $p\in {\mathcal{S}}^q\left({\bigcup }_i{B}_i\right)$, a contradiction for o being a local skyline point. By taking any as-yet encountered local skyline point for filtering during the clipping, Proposition 1 is tightened to the statement that the algorithm is optimal with respect to the class of algorithms that may also use knowledge of the buckets’ contents.

7.2. Analysis of Parallelism

At first glance, it seems an easy task to divide the algorithm into independent subtasks: firstly, the input data are partitioned and each part gets processed by a different thread. Particularly for a concurrent input source like data stored in the RAM of a CRCW (Concurrent Read Concurrent Write) machine, there is no sequential bottleneck. In the main step, each thread computes independently the skyline of its fetched input data. Finally, we just merge the local skylines until a single (i.e., the global) skyline remains. Unfortunately, the described algorithm is not embarrassingly parallel; it has several delicate parts that we have to take care of when adding concurrent structures to our code. In particular, this involves the queue that holds the computed, local skylines. For this job, Java 7’s ConcurrentLinkedQueue with wait-free access support [26] seemed fitting for us. Moreover, before climbing up its LSD tree, every thread must have at least one close neighbor of q in order to make use of the clipping technique. This may be either a point of the thread’s own input data set, a global NN of q, or even a list comprising every already found skyline point. Hence, we either use a global synchronization barrier (meaning all threads wait for a global communication process to start) or a concurrent list that is filled with newly found local skyline points and trimmed when an already stored point gets dominated by a recently collected point; this point cannot be dominated by another point from the list, otherwise the trimmed point would have been trimmed in advance. The other parts of the implementation can be written in the MapReduce and Fork/Join model. Both are two different models for parallel execution. They share the common goal of easing parallelization of a given sequential task. Nevertheless, their field of application is orthogonal. While MapReduce targets distributed computing, Fork/Join works merely on a single node. We describe our algorithm as a hybrid approach that employs both models—MapReduce for distributing the work-load to different nodes and Fork/Join for local multi-threaded computation. For the latter, we just translate the ideas of [4] to dynamic skyline computation. In order to describe our idea in the MapReduce framework, let us take a partition ${\left\{U_j\right\}}_{j\in J}$ of $\mathcal{D}$, i.e., ${\bigcup }_{j\in J}{U}_j=\mathcal{D}$ with $U_i\cap U_j=\varnothing$ for $i,j\in J$ pairwise different. The three functions $\mathtt{map}:(\left\{j\right\},U_j)\mapsto (\left\{j\right\},{\mathcal{S}}^q\left(U_j\right)),$ $\mathtt{combine}:\left((a,{\mathcal{S}}^q\left(A\right)),(b,{\mathcal{S}}^q\left(B\right))\right)\mapsto (a\cup b,{\mathcal{S}}^q\left(A\cup B\right))$ for each pairwise different subsets $a,b\subset J$ with any $A,B\subset \mathcal{D}$, and $\mathtt{reduce}:(J,{\mathcal{S}}^q\left(\mathcal{D}\right))\mapsto {\mathcal{S}}^q\left(\mathcal{D}\right)$ represent the common work flow of our MapReduce application. In particular, our implementation is a deterministic, multi-threaded program [27] because its schedule is predetermined by a fixed number of tasks that consists of the starting threads and the threads that combine the local skylines. Nevertheless, the code of SubSection 5.3 is non-deterministic with respect to thread spawning. That is caused by the fact that the number of tasks is dependent on the effectiveness of the clipping and the recursive divide-and-conquer technique. The latter tells us that the tree clipping is a fully-strict computation [28]. Because common frameworks like Hadoop do not allow dynamic task creation [29], the clipping cannot be described in terms of the MapReduce model. Fortunately, translating this part of the algorithm to the Fork/Join model seemed the right choice; Blumofe et al. [30] showed that the work-stealing technique is optimal for scheduling fully strict computations.

7.3. Caching Dynamic Skylines

The construction of the sLSD trees in Section 5.1 is done on the fly, i.e., the tree is not getting rebalanced during the initial bulk insertion of the data tuples $\mathcal{D}$. For reuse, we can restructure the sLSD trees to prevent worst-case scenarios for the subsequent queries. Therefore, we exchange our median-based split strategy with a distribution-dependent one [31] in order to cope, for instance, with skew-distributed data. An optimal refactoring of the tree results in a balanced tree with $\mathcal{O}(\frac{\left|\mathcal{D}\right|}{M})$ depth. Additionally, we save after each query the offset vector q along with its dynamic skyline set ${\mathcal{S}}^q\left(\mathcal{D}\right)$ for reuse [2]. For effective caching, we have to introduce the notion of orthants of q:

Definition 1.

The offset q splits the space into $2^n$ orthants. If we give each orthant a number, we can write the j-th orthant of q as $O_j^q$ for every $j=1,\dots ,2^n$. By Lemma 1, we have ${\mathcal{S}}^q\left(\mathcal{D}\right)\subset {\bigcup }_{j=1}^{2^n}{\mathcal{S}}^q\left(O_j^q\right)$.

Now, Proposition 1 and a variation of ([2], Lemma 2) comes in handy:

Lemma 5.

Let $q^{\prime }$ be an old offset and b a node of the sLSD tree whose bounding cuboid $C_b\subset O_j^{q^{\prime }}$ belongs to the j-th orthant of $q^{\prime }$, but does not contain any points of the j-th orthant skyline set ${\mathcal{S}}^{q^{\prime }}(O_j^{q^{\prime }})$. If q is an offset and $q^{\prime }$ belongs to the j-th orthant of q, then b does not intersect with ${\mathcal{S}}^q\left(\mathcal{D}\right)$ and thus can be clipped.

The cache is used to provide additional conditions for the clipping algorithm. Before employing the cache, we take only these elements into consideration that effectively help clipping:

Corollary 1

([2]). Let $q^{\prime }$ and $q^{\prime \prime }$ be two old queries for which we have cached their results. If $q^{\prime }{\prec }_q{q}^{″}$, then elements of the dynamic skyline set of $q^{″}$ are either part or dominated by elements of ${\mathcal{S}}^{q^{\prime }}\left(\mathcal{D}\right)$. Hence, $q^{\prime \prime }$ will not clip additional nodes of the sLSD tree.

7.4. Evaluation

For the following evaluation, we used the implementation of the Skyline Breaker [4] algorithm, called SB in the following, which is freely accessible from https://github.com/koeppl/skylinebreaker (accessed on 30 December 2022). This implementation is written in the Java language. In this evaluation, we used the SBQuick class applying the pruning technique on the LSD trees with parallel threading. To compare our solution with other skyline algorithms, we used the skyline benchmark software of Wiemann [32] (https://github.com/sven-wi/SkylineCompare) (accessed on 30 December 2022). Upon request, we received Java implementations for pskyline and parallelBBS of Im et al. [15] written in this framework. While pskyline applies MapReduce on lists of tuples, parallelBBS is based on the algorithm of Papadias et al. [7] using r trees as the geometric representation of the data (cf. Section 7).

We ran our experiments on a Debian 11 machine with 128 GB of RAM and an Intel Core i3–9100 CPU with four cores. The experiments are split into two parts. In the first part (Figure 6), we scale the number of input tuples while evaluating the parallel algorithms with four threads. In the second part (Figure 7), we scale the number of threads from one up to four. In both parts, we select different correlations of the input data, which are randomly generated by the framework based on the dataset generation of ([24], Figures 10–12).

Overall Evaluation. We observe that SB excels at small dimensions for which we can quickly detect the boundaries of a bucket. For larger dimensions, the times for these checks become longer, and thus the overall performance deteriorates with n more than the performance of the other solutions. Following the curves while scaling the input, we observe that SB is still in the competitive range with the other solutions, yet not the best choice if we want to evaluate pure skyline queries. In all different distributions, we observe a similar progression of SB’s time curve, which supports the claim that the usage of the LSD trees as geometric data structures makes the algorithm robust against different data distributions.

Parallel Evaluation. When scaling the number of threads, we observe that SB becomes faster most of the time or stays roughly at a constant speed level. Other implementations deteriorate at several instances, becoming slower with a higher number of threads. We can conclude that SB is suitable for the computation of the skyline set in parallel.

8. Conclusions

We studied a geometrical interpretation of the dynamic skyline problem by using LSD trees. LSD trees have shown in practice that they can handle high-dimensional feature vectors with low impact on the input data’s distribution. For the skyline computation, we made use of an already existing algorithm, the Skyline Breaker algorithm. Some parts of the parallel algorithm were changed in order to cope with the dynamic version of skyline computation. The theoretical evaluation addresses the combinatorial nature of the algorithm and treats both best- and worst-case scenarios with respect to different maximum bucket sizes. Computing the boundaries of a selected bucket is easy in small dimensions since it involves visiting only some closest ancestor nodes. Unfortunately, the boundary computation has an exponential dependency on the dimension such that the approach can quickly become less appealing if the dimension is relatively high compared to the input size. In that case, we have seen in the experiments that other approaches are more suitable.