Skip to Content
You are currently on the new version of our website. Access the old version .
Applied SciencesApplied Sciences
  • Article
  • Open Access

22 July 2022

24 Pages

Generalized Sketches for Streaming Sets

,
,
,
and
1
State Key Laboratory for Manufacturing Systems Engineering, Xi’an Jiaotong University, Xi’an 710049, China
2
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
3
MOE Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an 710049, China
*
Authors to whom correspondence should be addressed.

Abstract

Many real-world datasets are given as a stream of user–interest pairs, where a user–interest pair represents a link from a user (e.g., a network host) to an interest (e.g., a website), and may appear more than once in the stream. Monitoring and mining statistics, including cardinality, intersection cardinality, and Jaccard similarity of users’ interest sets on high-speed streams, are widely employed by applications such as network anomaly detection. Although estimating set cardinality, set intersection cardinality, and set Jaccard similarity, respectively, is well studied, there is no effective method that provides a one-shot solution for estimating all these three statistics. To solve the above challenge, we develop a novel framework, SimCar. SimCar online builds an order-hashing (OH) sketch for each user occurring in the data stream of interest. At any time of interest, one can query the cardinalities, intersection cardinalities, and Jaccard similarities of users’ interest sets. Specially, using OH sketches, we develop maximum likelihood estimation (MLE) methods to estimate cardinalities and intersection cardinalities of users’ interest sets. In addition, we use OH sketches to estimate Jaccard similarities of users’ interest sets and build locality-sensitive hashing tables to search for users with similar interests with sub-linear time. We evaluate the performance of our methods on real-world datasets. The experimental results demonstrate the superiority of our methods.

1. Introduction

Many real-world networks (e.g., computer networks, phone networks, and financial networks) generate data (e.g., packets, calling records, and transactions) in a stream fashion, where an element (e.g., a packet) records a link from a user (e.g., a source IP address) to an interest (e.g., a destination IP address). The data can be modeled as a data stream of user–interest pairs, including duplicates, because multiple data records (e.g., packets from a source IP address to a destination IP address) may relate to the same user–interest pair. Due to the large size and high-speed nature of these data streams, it is prohibitive to collect all the data, especially when the computation and memory resources of data-collection devices (e.g., network routers) are limited. To solve this challenge, considerable attention has been paid to designing fast and memory-efficient data streaming algorithms for monitoring and mining user behaviors. For example, Count-Min sketch [1] and its variants have been successfully used to detect heavy hitters (e.g., user–interest pairs frequently occurred).
In addition to statistics using the frequency of duplicates, significant effort has also been made to mine users’ interest sets, such as:
  • User cardinality. A user’s cardinality is defined to be the number of distinct interests that the user links to, i.e., the cardinality of the user’s interest set. Monitoring user cardinalities in computer networks is fundamental for applications such as network anomaly detection [2]. One can use a variety of sketch methods, such as LPC [3], LogLog [4], HyperLogLog [5], MinCount [6], RoughEstimator [7], and their variants [2,8,9,10,11,12,13] to generate a compact data summary (or sketch) for each user’s interest set, and then infer user cardinalities from the generated sketches.
  • Common-interest count. Two users’ common-interest count is defined as the number of interests that they both link to, i.e., the cardinality of the intersection of two users’ interest sets. It is popularly used for applications such as friend recommendation [14] and network anomaly detection [15]. One can use sketch and sampling methods in [13,14,15] to estimate users’ common-interest counts.
  • User Jaccard similarity. The Jaccard similarity is a popular similarity measure. Two users’ similarity can be evaluated as the Jaccard similarity of their interest sets, which is defined as their common-interest count divided by the number of distinct interests at least one of them links to. One can use sketch methods such as MinHash [16], OPH [17], and their variants [18,19,20,21,22] to estimate users’ Jaccard similarities. Some of these sketch methods, such as Minhash, can be used for locality-sensitive hashing (LSH) indexing [16,23,24,25,26], which is capable of searching a query user’s similar users with sub-linear time.
The above three duplicate-irrelevant statistics may all be desired for applications such as network anomaly detection. For example, given a detected abnormal user (e.g., the IP address of an attacker captured by Honeynet systems), network administers may want to quickly search for similar users among all network users, which may also be abnormal users, and then estimate their cardinalities, Jaccard similarities, and common-interest counts for further inspection. Although many cardinality estimation methods can be easily extended to estimate the above three statistics because they generate mergeable sketches (mergeable cardinality estimation sketches of two sets S 1 and S 2 can be used to generate a sketch of S 1 ∪ S 2 , which is used to estimate the cardinality of S 1 ∪ S 2 .), Cohen et al. [15] observed that their extensions exhibit large errors for estimating common-interest counts and Jaccard similarities. Even worse, the sketches generated by these cardinality methods cannot be used for LSH indexing, which results in a high computational cost for searching users with similar interests. One can combine multiple existing methods to generate different sketches for estimating user cardinalities, common-interest counts, and Jaccard similarities, respectively. This straightforward method increases the required memory and computational resources, which limits its usage for high-speed data streams.
To address the above challenges, we develop a novel framework, SimCar, which is shown in Figure 1. The framework consists of two modules: (1) the online processing module, which uses Giroire’s algorithm [6] to build an order-hashing (OH) sketch of the interest set of each user occurring in the stream; (2) the online query module, which provides functions for estimating the cardinalities, intersection cardinalities, and Jaccard similarities of users’ interest sets at any time of interest. Specially, using OH sketches, we develop maximum likelihood estimation (MLE) methods for estimating cardinalities and intersection cardinalities of users’ interest sets. In addition, we use the technique optimal densification [22] to generate densification OH (DOH) sketches and then use DOH sketches to estimate Jaccard similarities of users’ interest sets and build locality-sensitive hashing tables to search for users with similar interests with sub-linear time. Our main contributions are summarized as follows:
Figure 1. Framework of our framework SimCar.
  • For sets given in a stream fashion, our method, SimCar, builds only one type of sketches (i.e., OH sketch) to effectively mine a variety of their statistics, including cardinality, intersection cardinality, and Jaccard similarity, as well as fast search similar sets. It outperforms the straightforward method of combining existing sketch-based cardinality estimation methods and Jaccard similarity estimation methods.
  • We develop maximum likelihood estimation (MLE) methods to estimate set cardinalities and intersection cardinalities, which are more accurate than the original OH-sketch-based cardinality estimation method [6].
  • Compared with state-of-the-art methods, experimental results on real-world datasets demonstrate that our method, SimCar, is up to 1000 times faster for online data streams processing, and reduces the memory usage by 13.5–23.8% to achieve the same performance accuracy.
The rest of this paper is organized as follows. The problem formulation is presented in Section 3. Section 4 introduces the preliminaries. Section 5 presents our framework, SimCar. The performance evaluation is presented in Section 6. Section 2 summarizes related work. Concluding remarks then follow.

3. Problem Formulation

Let Γ = e ( 1 ) , … , e ( t ) , … denote the stream of interest, where e ( t ) = ( u ( t ) , w ( t ) ) is the t th element of stream Γ , which represents a link from a user u ( t ) to an interest w ( t ) . Note that a pair ( u , w ) may appear more than once in stream Γ . Denote by U the set of occurring users, and W the set of occurring interests. Let N u be the interest set of user u, i.e., the set of interests that user u links to. User u’s cardinality is defined as the number of its distinct interests, i.e., d u = | N u | . For two users u and v, we define their common-interest count as d u ∩ v = | N u ∩ N v | , and their Jaccard similarity as J u , v = | N u ∩ N v | | N u ∪ N v | . In this paper, we aim to design a fast method to build a compact data summary (or, sketch) for each user in a single-pass stream Γ . For any specific query user u, generated sketches are effective for rapidly searching for nearest neighbors of user u (i.e., users v ∈ U with large Jaccard similarity J u , v ) without enumerating each user in U. More importantly, they are also accurate for estimating the cardinality d v of any user v, Jaccard similarity J u , v , as well as the common-interest count d u ∩ v of any two users u and v. For ease of reference, we list the notation used throughout the paper in Table 1.
Table 1. Table of notation.

4. Preliminaries

In this section, we present Giroire’s algorithm [6] for cardinality estimation, and the OPH method [17] as well as the optimal densification [22] for Jaccard similarity estimation, which are closely related with our framework, SimCar.

4.1. Giroire’s Algorithm

Giroire [6] developed a fast method, CORE, for estimating a stream’s cardinality, i.e., the number of distinct elements occurring in the stream. The basic idea behind CORE is that one can assign each element in the stream a rank that is randomly and uniformly selected from the range (0,1), and to some extent, the minimum of occurring elements’ ranks reflects the number of distinct occurring elements (i.e., the stream’s cardinality). To accelerate the performance, CORE splits elements in the stream into k subsets, and keeps track of the minimum rank for each subset, which is finally used to infer the stream’s cardinality. In detail, CORE uses k registers x 1 , …, x k to generate a sketch of the stream, where x 1 , …, x k are initialized to 1. Let r e be the rank of an element e that is uniformly selected from range (0, 1) at random; that is, r e ∼ Uniform ( 0 , 1 ) . For an element e arriving on the stream, CORE computes i e = ⌊ r e k ⌋ + 1 and then updates the i e th register x i e as
x i e ← min ( x i e , r e k − i e − 1 ) .
At the end of the stream, the stream’s cardinality is estimated as d * = k ( k − 1 ) ∑ i = 1 k x i . Since d * is severely biased for small cardinalities, CORE treats the list of registers x 1 , … , x k as an LPC sketch (i.e., a bitmap of k bits) [3]. Let n 1 be the number of registers among x 1 , … , x k that equal 1 at the end of the stream. When n 1 k ≤ 86 % , CORE estimates the stream’s cardinality as − k log n 1 k .

4.2. OPH

OPH [17] can be viewed as a sampling method, which samples (at most) k distinct elements from each set of interest without replacement and then estimates the Jaccard similarity of two sets based on their sampled elements. Compared with the well-known sketch method MinHash [16], which is used to estimate set Jaccard similarity, OPH reduces the time complexity of processing each element in set N u from O ( k ) to O ( 1 ) . In detail, to build a sketch of set N u ⊆ W , OPH [17] uses a single permutation function π ( · ) : W → W to process each element in a set N u . Specifically, it evenly divides W = { 1 , … , p } into k bins: W j = p ( j − 1 ) k + 1 , … , p j k , 1 ≤ j ≤ k . The OPH sketch of set N u is a vector x → u = ( x u , 1 , … , x u , k ) T , where each element x u , j , 1 ≤ j ≤ k , is computed as
x u , j = min w ∈ N u ∧ π ( w ) ∈ W j π ( w ) , ∃ w ∈ N u , π ( w ) ∈ W j ∅ , otherwise .
Given that at least one of x u , j and x v , j does not equal ∅ Li et al. [17] observed that x u , j = x v , j occurs with probability J u , v , and proposed estimating J u , v based on this observation. Let 1 ( P ) be the indicator function that equals one when the predicate P is true, and zero otherwise. Formally, OPH estimates the Jaccard similarity J u , v of two sets N u and N v based on their OPH sketches x → u and x → v as
J ^ u , v = ∑ i = 1 k 1 ( x u , i = x v , i ∧ x u , i ≠ ∅ ∧ x v , i ≠ ∅ ) ∑ i = 1 k 1 ( x u , i ≠ ∅ ∨ x v , i ≠ ∅ ) .
As shown in Algorithm 1, one can apply the above OPH algorithm online to stream Γ and compute the OPH sketch of each user’s interest set N u , u ∈ U .
Algorithm 1: Online update procedure of OPH [17].
Applsci 12 07362 i001

4.3. Optimal Densification

Clearly, the OPH method generates sketches (i.e., vectors) that may have empty elements (i.e., elements equal ∅), which hinders its application for LSH-based similarity search with sub-linear time, and also results in a large error in the Jaccard similarity estimation. To solve this problem, refs. [20,21,22] proposed several densification techniques, which reassign empty elements using the values of non-empty elements. The state-of-the-art densification technique, optimal densification [22], is shown in Algorithm 2. Specifically, when the i th element of OPH sketch x → u is empty (i.e., x u , i = ∅ ), optimal densification iteratively uses a hash function j = h ( i , l ) : { 1 , … , k } × { 1 , 2 , … , } → { 1 , … , k } , to find a non-empty element x u , j (i.e., x u , j ≠ ∅ ). Function h ( i , l ) takes two arguments: (1) i, the current empty element that needs to be reassigned; (2) l, the number of failed attempts made so far to reach a non-empty element, which is initialized to 1. Let x → u * denote the optimal densification of OPH sketch x → u . For any sets N u and N v , their Jaccard similarity J u , v is estimated as
J ^ u , v * = ∑ i = 1 k 1 ( x u , i * = x v , i * ) k .
Shrivastava [22] proves that J ^ u , v * is an unbiased estimator of J u , v and is more accurate than the original OPH method, as well as other densificiation techniques in [20,21].
Algorithm 2: Optimal densification [22].
Applsci 12 07362 i002

5. Our Framework SimCar

In this section, we use Giroire’s algorithm [6] (Algorithm 3) to build an OH sketch for each occurring user. Based on generated OH sketches, we first present our methods for estimating the Jaccard similarity of users’ interest sets and quickly searching users with similar interests. Then, we propose methods for estimating the cardinalities and common-interest counts of users’ interest sets.
Algorithm 3: The CORE algorithm [59].
Applsci 12 07362 i003

5.1. Jaccard Similarity Estimation and Nearest Neighbor Search

Jaccard Similarity Estimation. Let x → u ( * ) be the DOH sketch of OH sketch x → u , u ∈ U , which is computed using the optimal densification technique introduced in Section 4. Similar to optimal densification [22], we find that P ( x u , i * = x v , i * ) = J u , v for any 1 ≤ i ≤ k . Then, we estimate the Jaccard similarity of J u , v as
J ^ u , v * = ∑ i = 1 k 1 ( x u , i * = x v , i * ) k .
Following the error analysis in [22], we find that J ^ u , v * is an unbiased estimate of J u , v and has the same accurate performance as the optimal densification of OPH.
Nearest Neighbor Search. A variety of LSH methods [16,23,24,25,26] have been developed to solve the task of nearest neighbor search with sub-linear time. The basic idea behind these LSH methods is to place similar users into the same bucket of a hash table with high probability. Following the principle of LSH, our nearest neighbor search method consists of two basic phases:
  • Pre-processing phase. We first construct b hash tables H 1 , … , H b for all users in U. Let k = b c , where c is an integer parameter used to determine the performance of LSH. Let x → u ( * , i , c ) , 1 ≤ i ≤ b , denote the vector consisting of the ( ( i − 1 ) c + 1 ) th to ( i c ) th elements of DOH sketch x → u * , i.e., x → u ( * , i , c ) = ( x u , ( i − 1 ) c + 1 * , … , x u , i c * ) T . Then, we insert each user u and its densification sketch x → u * into hash tables H 1 , … , H b as
    H i [ x → u ( * , i , c ) ] ← H i [ x → u ( * , i , c ) ] ∪ { u } , 1 ≤ i ≤ b .
  • Retrieving phase. Given a specific user u, we search for its similar users. Instead of scanning all users in set U, we only probe b different buckets H 1 [ x → u ( * , i , c ) ] , … , H b [ x → u ( * , i , c ) ] from b hash tables, respectively, and report the users in any of these buckets as the potential candidates. Last, nearest neighbors of user u are detected by enumerating all users in set H 1 [ x → u ( * , i , c ) ] ∪ … ∪ H b [ x → u ( * , i , c ) ] , and computing and sorting their Jaccard similarity estimations.

5.2. User Cardinality Estimation

In this section, we estimate the cardinality of any queried user. Specially, we first build a probabilistic model according to the state of the OH sketch, and then solve it using MLE methods.
Exact probabilistic model. For any user u ∈ U , let N u , i denote the set of its interests whose hash values equal i with respect to hash function f; that is,
N u , i = { w : i w = i , w ∈ N u } , 1 ≤ i ≤ k .
At any time of stream Γ , from Equation (1), we have
x u , i = min w ∈ N u , i r w k − ⌊ r w k ⌋ .
Define d u , i = | N u , i | . Let f ( x u , i = x | d u , i = d ) denote the probability density function (PDF) of random variable x u , i at x given d u , i = d . Then, we have
f ( x u , i = x | d u , i = d ) = d ( 1 − x ) d − 1 , d > 0 ∧ 0 < x < 1 1 , d = 0 ∧ x = 1 0 , otherwise .
Our algorithm randomly splits users in set N u into k subsets N u , 1 , …, N u , k . Using the classical balls-in-urns model, we derive the probability distribution of vector ( d u , 1 , … , d u , k ) as
P ( d u , 1 , … , d u , k | d u ) = d u d u , 1 , … , d u , k k d u ,
where d u d u , 1 , … , d u , k = d u ! d u , 1 ! ⋯ d u , k ! . Therefore, the PDF of x → u given d u is computed as
f ( x → u | d u ) = ∑ d u , 1 + … + d u , k = d u d u d u , 1 , … , d u , k k d u ∏ i = 1 k f ( x u , i | d u , i ) .
Poisson approximation model. Clearly, d u , 1 , … , d u , k are not independent, which hinders us when it comes to obtaining the MLE of d u . Similar to [5], we use the Poisson approximation technique to remove the dependence of d u , 1 , … , d u , k . Specifically, we assume that the value of d u is distributed according to a Poisson distribution with parameter λ u , i.e., d u ∼ Poisson ( λ u ) . Then, the PDF of x → u given λ u is
f ( x → u | λ u ) = ∑ d = 0 + ∞ ℒ ( x → u | d u ) e − λ u λ u d u d u ! = ∑ d u = 0 + ∞ ∑ d u , 1 + … + d u , k = d u d u d u , 1 , … , d u , k k d u ∏ i = 1 k f ( x u , i | d u , i ) e − λ u λ u d u d u ! = ∑ d u , 1 = 0 + ∞ … ∑ d u , k = 0 + ∞ ∏ i = 1 k f ( x u , i | d u , i ) e − λ u k λ u d u , i d u , i ! k d u , i = ∏ i = 1 k ∑ d u , i = 0 + ∞ f ( x u , i | d u , i ) e − λ u k λ u d u , i d u , i ! k d u , i = ∏ i = 1 k f ( x u , i | λ u ) .
Given d u ∼ Poisson ( λ u ) , the above equation indicates that the values of x u , 1 , … , x u , k are independent and identically distributed. In addition, the values of d u , 1 , … , d u , k are independent and identically distributed according to a Poisson distribution Poisson ( λ u k ) . Formally, we have
d u , i ∼ Poisson λ u k , i = 1 , … , k .
For an estimator ϕ ( x → u ) of d u (e.g., Equation (5), derived later), we let E d u ( ϕ ( x → u ) ) and Var d u ( ϕ ( x → u ) ) denote the expectation and the variance of ϕ ( x → u ) under the exact probabilistic model (i.e., f ( x → u | d u ) ), and let E Poisson ( λ u ) ( ϕ ( x → u ) ) and Var Poisson ( λ u ) ( ϕ ( x → u ) ) denote the expectation and the variance of ϕ ( x → u ) under the Poisson approximation model (i.e., f ( x → u | d u ) ). The authors of [5,68,69] reveal that statistical properties of E d u ( ϕ ( x → u ) ) and Var d u ( ϕ ( x → u ) ) are well approximated by E Poisson ( λ u ) ( ϕ ( x → u ) ) and Var Poisson ( λ u ) ( ϕ ( x → u ) ) when setting λ u = d u , which is the depoissonization step.
Estimator of λ u . Next, we elaborate our method to derive the MLE of λ u under the Poisson approximation model. For any 0 < x < 1 , the following equation holds
f ( x u , i = x | λ u ) = ∑ d u , i = 0 + ∞ f ( x u , i = x | d u , i ) e − λ u k λ u d u , i d u , i ! k d u , i = e − λ u k + ∑ d u , i = 1 ∞ d u , i ( 1 − x ) d u , i − 1 e − λ u k λ u d u , i d u , i ! k d u , i = λ u k e − λ u x k ,
where the last equation holds because of the Taylor series of λ u k e − λ u x k at x 0 = 1 . Then, we compute the PDF of x u , i at x given λ u as
f ( x u , i = x | λ u ) = λ u k e − λ u x k , 0 < x < 1 e − λ u k , x = 1 .
The last equation holds because we have d u , i = 0 when x = 1 , which occurs with probability P ( d u , i = 0 | λ u ) = e − λ u k . Let Φ u denote the set of elements in vector x → u that are smaller than 1, i.e.,
Φ u = { i : x u , i < 1 , 1 ≤ i ≤ k } .
Denoted by k u is the number of elements in vector x → u that equal 1. Then, the likelihood function of λ u given by k independent observed variables x u , 1 , … , x u , k is computed as
ℒ ( x → u | λ u ) = ∏ i = 1 k f ( x u , i | λ u ) = e − λ u k u k ∏ i ∈ Φ u λ u k e − λ u x u , i k = λ u k k − k u e − λ u ( k u + ∑ i ∈ Φ u x u , i ) k = λ u k k − k u e − λ u ∑ i = 1 k x u , i k .
Let λ ^ u = arg max λ u log L ( x → u | λ u ) denote the MLE of λ u . We compute the derivative of log L ( x → u | λ u ) as
∂ log L ( x → u | λ u ) ∂ λ u = k − k u λ u − ∑ i = 1 k x u , i k .
Let ∂ log L ( x → u | λ u ) ∂ λ u = 0 . Then, we obtain the MLE of λ u as
λ ^ u = k ( k − k u ) ∑ i = 1 k x u , i .
To analyze the estimation error of λ ^ u , we first compute the Fisher information of λ u as
I ( λ u ) = − E ∂ 2 log L ( x → u | λ u ) ∂ λ u 2 = E k − k u λ u 2 ∣ λ u = k ( 1 − e − λ u k ) λ u 2 .
The last equation is obtained because E ( k u | λ u ) = k e − λ u k , which is easily derived from Equation (4). From [70], we find that the MLE λ ^ u is an asymptotically efficient unbiased estimator of λ u , and its mean square error (MSE) asymptotically approaches the Cramér–Rao lower bound (CRLB) of λ u , which is defined as 1 I ( λ u ) . Formally, we have lim k → + ∞ λ ^ u → λ u and lim k → + ∞ MSE ( λ ^ u ) → 1 I ( λ u ) . Lastly, at the depoissonization step, we use λ ^ u to approximate d u . In our experiments, we will show that the above MLE estimator is more accurate than the Giroire’s algorithm (Section 4.1), especially for small cardinalities.

5.3. Common-Interest Count Estimation

Similarly, we use the MLE and Poisson approximation techniques to estimate the common-interest count d u ∩ v of any two users u , v ∈ U . Specifically, we assume
d u ∩ v ∼ Poisson λ u , v .
Define N u ∖ v = N u ∖ N v and N v ∖ u = N v ∖ N u . Let d u ∖ v = | N u ∖ v | and d v ∖ u = | N v ∖ u | . We also assume
d u ∖ v ∼ Poisson ( λ u − λ u , v ) ,
d v ∖ u ∼ Poisson ( λ v − λ u , v ) ,
where λ u ≥ λ u , v and λ v ≥ λ u , v . Similar to N u , i , 1 ≤ i ≤ k , we define
N u ∩ v , i = { w : i w = i , w ∈ N u ∩ v } ,
N u ∖ v , i = { w : i w = i , w ∈ N u ∖ v } ,
N v ∖ u , i = { w : i w = i , w ∈ N v ∖ u } .
Let d u ∩ v , i = | N u ∩ v , i | , d u ∖ v , i = | N u ∖ v , i | , and d v ∖ u , i = | N v ∖ u , i | . Similar to Equation (2), we find that d u ∩ v , 1 , …, d u ∩ v , k , d u ∖ v , 1 , …, d u ∖ v , k , d v ∖ u , 1 , …, d v ∖ u , k are independent, and they are distributed according to the following Poisson distributions
d u ∩ v , i ∼ Poisson λ u , v k ,
d u ∖ v , i ∼ Poisson λ u − λ u , v k ,
d v ∖ v , i ∼ Poisson λ v − λ u , v k .
Define x u ∩ v , i = min w ∈ N u ∩ v , i r w , x u ∖ v , i = min w ∈ N u ∖ v , i r w , and x v ∖ u , i = min w ∈ N v ∖ u , i r w . Then, we find that
x u , i = min ( x u ∩ v , i , x u ∖ v , i ) ,
x v , i = min ( x u ∩ v , i , x v ∖ u , i ) .
Let f ( x u , i = x , x v , i = x ′ | λ u , λ v , λ u , v ) denote the PDF of the random variable pair ( x u , i , x v , i ) at ( x , x ′ ) , given λ u , λ v , and λ u , v . In what follows, we omit the conditions λ u , λ v , and λ u , v for simplicity when no confusion is raised. We derive f ( x u , i = x , x v , i = x ′ ) for all possible relations between x u , i and x v , i as follows:
Case 1: x u , i = x v , i = 1 . This case indicates d u ∩ v , i = d u ∖ v , i = d v ∖ u , i = 0 . Therefore, we have
f ( x u , i = 1 , x v , i = 1 ) = P ( d u ∩ v , i = 0 ) P ( d u ∖ v , i = 0 ) P ( d v ∖ u , i = 0 ) = e − λ u , v k e − λ u − λ u , v k e − λ v − λ u , v k = e − λ u + λ v − λ u , v k .
Case 2: x u , i = 1 ∧ 0 < x v , i < 1 .This case indicates d u ∩ v , i = d u ∖ v , i = 0 and 0 < x v ∖ u , i = x v , i < 1 . Similar to Equation (4), we have f ( x v ∖ u , i = x ) = λ v − λ u , v k e − ( λ v − λ u , v ) x k . Then, we have
f ( x u , i = 1 , x v , i = x ) = f ( x v ∖ u , i = x ) P ( d u ∩ v , i = 0 ) P ( d u ∖ v , i = 0 ) = λ v − λ u , v k e − λ u + ( λ v − λ u , v ) x k , 0 < x < 1 .
Case 3: 0 < x u , i < 1 ∧ x v , i = 1 . Similar to Case 2, we have
f ( x u , i = x , x v , i = 1 ) = f ( x u ∖ v , i = x ) P ( d u ∩ v , i = 0 ) P ( d v ∖ u , i = 0 ) = λ u − λ u , v k e − λ v + ( λ u − λ u , v ) x k , 0 < x < 1 .
Case 4: 0 < x u , i = x v , i < 1 . This case indicates that x u ∩ v , i < x u ∖ v , i and x u ∩ v , i < x v ∖ u , i . Similar to Equation (4), we obtain f ( x u ∩ v , i = x ) = λ u , v k e − λ u , v x k , f ( x u ∖ v , i > x ) = e − ( λ u − λ u , v ) x k , and f ( x v ∖ u , i > x ) = e − ( λ v − λ u , v ) x k . Therefore, we have
f ( x u , i = x , x v , i = x ) = f ( x u ∩ v , i = x ) f ( x u ∖ v , i > x ) f ( x v ∖ u , i > x ) = λ u , v k e − ( λ u + λ v − λ u , v ) x k , 0 < x < 1 .
Case 5: 0 < x u , i < x v , i < 1 . This case indicates that x u ∖ v , i = x u , i and min ( x u ∩ v , i , x v ∖ u , i ) = x v , i . We find that d v , i = d u ∩ v , i + d v ∖ u , i ∼ Poisson λ v k , and f ( min ( x u ∩ v , i , x v ∖ u , i ) = x ′ ) = λ v k e − λ v x ′ k . Therefore, we have
f ( x u , i = x , x v , i = x ′ ) = f ( x u ∖ v , i = x ) f ( min ( x u ∩ v , i , x v ∖ u , i ) = x ′ ) = ( λ u − λ u , v ) λ v k 2 e − ( λ u − λ u , v ) x + λ v x ′ k , 0 < x < x ′ < 1 .
Case 6: 0 < x v , i < x u , i < 1 . Similar to Case 5, we have
f ( x u , i = x , x v , i = x ′ ) = f ( min ( x u ∩ v , i , x u ∖ v , i ) = x ) f ( x v ∖ u , i = x ′ ) = ( λ v − λ u , v ) λ u k 2 e − ( λ v − λ u , v ) x ′ + λ u x k , 0 < x ′ < x < 1 .
Let Φ u , v ( j ) denote the set of integers i ∈ { 1 , … , k } , such that the relation between x u , i and x v , i is Case j, 1 ≤ j ≤ 6 . Define k u , v ( j ) = | Φ u , v ( j ) | . Then, we have k u = k u , v ( 1 ) + k u , v ( 2 ) , k v = k u , v ( 1 ) + k u , v ( 3 ) , Φ u = Φ u , v ( 3 ) ∪ Φ u , v ( 4 ) ∪ Φ u , v ( 5 ) ∪ Φ u , v ( 6 ) , and Φ v = Φ u , v ( 2 ) ∪ Φ u , v ( 4 ) ∪ Φ u , v ( 5 ) ∪ Φ u , v ( 6 ) . Now, we obtain the likelihood function of λ → = ( λ u , λ v , λ u , v ) given by k independent observed variable pairs ( x u , 1 , x v , 1 ) , … , ( x u , k , x v , k ) as
L ( x → u , x → v ) = Π i = 1 k f ( x u , i , x v , i ) = e − ( λ u + λ v − λ u , v ) k u , v ( 1 ) k × λ v − λ u , v k k u , v ( 2 ) e − λ u k u , v ( 2 ) + ( λ v − λ u , v ) ∑ i ∈ Φ u , v ( 2 ) x v , i k × λ u − λ u , v k k u , v ( 3 ) e − λ v k u , v ( 3 ) + ( λ u − λ u , v ) ∑ i ∈ Φ u , v ( 3 ) x u , i k × λ u , v k k u , v ( 4 ) e − ( λ u + λ v − λ u , v ) ∑ i ∈ Φ u , v ( 4 ) x u , i k × ( λ u − λ u , v ) λ v k 2 k u , v ( 5 ) e − ∑ i ∈ Φ u , v ( 5 ) ( λ u − λ u , v ) x u , i + λ v x v , i k
× ( λ v − λ u , v ) λ u k 2 k u , v ( 6 ) e − ∑ i ∈ Φ u , v ( 6 ) ( λ v − λ u , v ) x v , i + λ u x u , i k = e − λ u ∑ i = 1 k x u , i k e − λ v ∑ i = 1 k x v , i k e λ u , v ∑ i = 1 k min ( x u , i , x v , i ) k × λ u − λ u , v k k u , v ( 3 ) + k u , v ( 5 ) λ v − λ u , v k k u , v ( 2 ) + k u , v ( 6 ) × λ u k k u , v ( 6 ) λ v k k u , v ( 5 ) λ u , v k k u , v ( 4 ) .
We use the Newton–Raphson method [71] to compute the MLE of λ → = ( λ u , λ v , λ u , v ) , i.e., λ → ^ = arg max λ → L ( x → u , x → v ) . The Newton–Raphson method starts from an initial estimation, λ → ( 0 ) , and then repeats the following procedure
λ → ( l + 1 ) ← λ → ( l ) − H ( λ → ( l ) ) − 1 g ( λ → ( l ) ) ,
until a sufficiently accurate root of function g ( λ → ) = ( 0 , 0 , 0 ) T is reached, where g ( λ → ) and H ( λ → ) are the gradient and the Hessian matrix of log L ( x → u , x → v ) at λ → is defined as
g ( λ → ) = ∂ log L ( x → u , x → v ) ∂ λ u , ∂ log L ( x → u , x → v ) ∂ λ v , ∂ log L ( x → u , x → v ) ∂ λ u , v T ,
H ( λ → ) = ∂ 2 log L ( x → u , x → v ) ∂ λ u 2 , ∂ 2 log L ( x → u , x → v ) ∂ λ u ∂ λ v , ∂ 2 log L ( x → u , x → v ) ∂ λ u ∂ λ u , v ∂ 2 log L ( x → u , x → v ) ∂ λ v ∂ λ u , ∂ 2 log L ( x → u , x → v ) ∂ λ v 2 , ∂ 2 log L ( x → u , x → v ) ∂ λ v ∂ λ u , v ∂ 2 log L ( x → u , x → v ) ∂ λ u , v ∂ λ u , ∂ 2 log L ( x → u , x → v ) ∂ λ u , v ∂ λ v , ∂ 2 l o g L ( x → u , x → v ) ∂ λ u , v 2 .
The closed formulas of g ( λ → ) and H ( λ → ) are given in Appendix A. To set λ → ( 0 ) = ( λ u ( 0 ) , λ v ( 0 ) , λ u , v ( 0 ) ) T , we initialize λ u ( 0 ) and λ v ( 0 ) using the estimates of λ u and λ v given in Section 5.2 (i.e., λ u ( 0 ) = k ( k − k u ) ∑ i = 1 k x u , i and λ v ( 0 ) = k ( k − k v ) ∑ i = 1 k x v , i ) and initialize λ u , v ( 0 ) = J ^ u , v ( λ ^ u ( 0 ) + λ v ( 0 ) ) J ^ u , v + 1 , where J ^ u , v = ∑ i = 1 k 1 ( x u , i * = x v , i * ) k is an estimate of Jaccard similarity J u , v , which is introduced in Section 5.1.
Next, we analyze the error of obtained MLE λ ^ u , v * . The Fisher information matrix of λ → is defined as
I ( λ → ) = − E ( H ( λ → ) ) .
The closed formula of E ( H ( λ → ) ) is given in Appendix A. Let ( I ( λ → ) ) 3 , 3 − 1 be the ( 3 , 3 ) element of the inverse of matrix I ( λ → ) . From [70], we find that the MLE λ ^ u , v * is an asymptotically efficient unbiased estimator of λ ^ u , v , and its MSE asymptotically approaches the CRLB of λ u , v , i.e., ( I ( λ → ) ) 3 , 3 − 1 . Lastly, we use λ ^ u , v * to approximate d u ∩ v .

5.4. Space and Time Complexities

Complexities of online generating OH sketches. We use k registers for each occurring user in stream Γ , and the time complexity of processing each element of stream Γ is O ( 1 ) .
Complexities of densification. No extra memory space is required for densification. From [22], we find that computing a user u’s DOH sketch from its OH sketch x → u requires time complexity O ( k ( k u k − k u + 2 ) ) .
Complexities of user cardinality estimation. No extra memory space is required for user cardinality estimation. The time complexity of estimating a user’s cardinality is O ( k ) .
Complexities of common-interest count estimation. No extra memory space is required for common-interest count estimation. We find that our method only requires several Newton–Raphson iterations to converge, where each iteration has a negligible computational cost. Therefore, the time complexity of estimating two users’ common-interest counts is O ( k ) .
Complexities of Jaccard similarity estimation. No extra memory space is required for common-interest count estimation. Based on two users’ DOH sketches, time O ( k ) is required when estimating two users’ Jaccard similarity.
Complexities of nearest neighbor search. Given DOH sketches of users in U, space O ( | U | b ) and time O ( | U | b ) are required for building LSH tables, and searching a query user’s nearest neighbors requires time O ( k log | U | ) .

6. Evaluation

In this section, we evaluate the runtime and accuracy of our framework, SimCar, compared to the state of the art. All experiments were run on the same machine with an Intel Xeon CPU E5-2620 v3 with 2.4 GHz, and all algorithms were implemented in Python.

6.1. Datasets

We performed our experiments on six real-world datasets from different areas. The dataset Flickr [72] is a bipartite network of Flickr users and their group memberships. The dataset Movie [73] is a bipartite network consisting of one million movie ratings from MovieLens (http://movielens.umn.edu/, accessed on 10 March 2022).
The dataset Wikipedia [74] is a bipartite network of excellent articles in the English Wikipedia (http://en.wikipedia.org/wiki/Wikipedia, accessed on 10 March 2022) which meet a core set of editorial standards, and the words they contain. The dataset Reuters [75] is a bipartite network of article-word inclusions in documents that appeared on Reuters newswire in 1987. The dataset Tropes [76] is a bipartite network of TV tropes (http://tvtropes.org, accessed on 10 March 2022), characterizing artistic works by their tropes. The dataset TREC [77] is a bipartite network of 1.7 million text documents from the Text Retrieval Conference’s (TREC) (http://trec.nist.gov/data/docs_eng.html, accessed on 10 March 2022), containing 556,000 words of different languages. The detailed statistics of these real-world datasets are summarized in Table 2.
Table 2. Real-world datasets used in our experiments, where “size” refers to the number of elements in the stream.

6.2. Baselines

We compared our framework with the following state-of-the-art methods on all four tasks: user cardinality estimation, common-interest count estimation, Jaccard similarity estimation, and nearest neighbor search.
  • MinHash and HLL. For common-interest count estimation, the baseline is the method [15], which estimates common-interest counts by generating both MinHash [16] and HyperLogLog [5] (short for HLL in the following paper) sketches for each user’s interest set. Clearly, one can use generated HLL sketches to estimate user cardinalities, and use generated Minhash sketches to estimate Jaccard similarities and build LSH tables for a nearest neighbor search.
  • OPH+HLL. One can also solve all four tasks by combining OPH [17] and HLL, where OPH [17] is much faster than MinHash and exhibits comparable accuracy for applications such as Jaccard similarity estimation and nearest neighbor search [20,21,22]. In detail, HLL sketches were used to estimate user cardinalities, and optimal densified OPH sketches were used to estimate Jaccard similarity and build LSH tables for a rapid nearest neighbor search. To estimate the common-interest count of any two users u and v, we first easily obtained (1) estimations d ^ u , d ^ v , and d ^ u ∪ v of d u , d v , and d u ∪ v = | N u ∪ N v | from HLL sketches and our OH sketches; (2) estimation J ^ u , v of Jaccard similarity J u , v from optimal densified OPH sketches and DOH sketches. As shown in [15], the common-interest count d u ∩ v can be estimated by each of the following schemes: (scheme 1) d ^ u ∩ v = d ^ u + d ^ v − d ^ u ∪ v ; (scheme 2) d ^ u ∩ v = J ^ u , v d ^ u ∪ v ; (scheme 3) d ^ u ∩ v = J ^ u , v J ^ u , v + 1 ( d ^ u + d ^ v ) . Similar to MinHash and HLL method [15], we initialize the parameter λ → ( 0 ) = ( λ u ( 0 ) , λ v ( 0 ) , λ u , v ( 0 ) ) T = ( d ^ u , d ^ v , d ^ u ∩ v ) T for our common-interest count estimation method in Section 5.3 and then run only a single Newton–Raphson iteration to obtain a more accurate estimation of d u ∩ v . In our experiments, common-interest count estimations given by OPH and HLL are computed by the above schemes 1, 2, and 3 in a direct manner.
In our experiments, by default, we let MinHash, OPH, and our sketch method OH use the same k, i.e., the number of registers used to generate one MinHash/OPH/OH sketch. We let k 1 denote the number of registers used for an HLL sketch. Therefore, our method, SimCar, uses k 1 fewer registers than baselines MinHash and HLL and OPH and HLL. Compared with MinHash, OPH, and OH using 32-bit registers, HLL uses 5-bit registers [5]; thus, our method, SimCar, reduces the memory usage by 13.5% and 23.8% when k 1 = k and k 1 = 2 k , respectively.

6.3. Metric

We evaluated both the efficiency and effectiveness of our method, SimCar, in comparison with the above two baseline methods. For efficiency, we evaluated the running time of all methods. Specially, we used the online sketching time to measure the time of online processing stream Γ . For user cardinality estimation, common-interest count estimation, and Jaccard similarity estimation, we evaluated the error of estimation μ ^ with respect to its true value μ using the normalized root mean square error (NRMSE), which is defined as NRMSE ( μ ^ ) = MSE ( μ ^ ) / μ , where MSE ( μ ^ ) = E ( ( μ ^ − μ ) 2 ) = Var ( μ ^ ) + E ( μ ^ ) − μ 2 . For the nearest neighbor search, given a query user, we report the average size of the retrieved user set, as well as the recall of the top 10 most similar users among that set. In our experiments, we computed the above metrics over 100 independent runs.

6.4. Runtime

As shown in Figure 2a, where we set k 1 = k = 2 9 , our framework, SimCar, significantly outperforms MinHash and HLL and OPH and HLL on all datasets in Table 2. For example, for Reuters, the online sketching time of SimCar is about 2 seconds, while MinHash and HLL and OPH and HLL need about 764 and 5.3 seconds, respectively. On average, SimCar is about 400 and 2.5 times faster than MinHash and HLL and OPH and HLL, respectively. Figure 2b shows the online sketching time for different k. We can see that the online sketching time of SimCar and OPH+HLL is almost constant, while the online sketching time of MinHash and HLL increases linearly to k, because Minhash needs to update each of its k registers when updating each element that occurs in stream Γ .
Figure 2. Online sketching time of our framework, SimCar, in comparison with MinHash and HLL and OPH and HLL. (a) Sketching time on different datasets with k 1 = k = 2 9 . (b) Sketching time on dataset Reuters with different k.

6.5. Accuracy

Results of cardinality estimation. We compared our method, SimCar, with HyperLogLog used by MinHash and HLL and OPH and HLL, as well as the CORE algorithm [59], which is introduced in Algorithm 3 in Section 5. In this experiment, all three sketch methods—HLL, CORE, and SimCar—use the same number of registers, i.e., k 1 = k . As shown in Figure 3, our method, SimCar, is significantly more accurate than HLL and CORE on these real-world datasets. To be more specific, SimCar gives a gain of 18 % ∼ 40 % (resp. 10 % ∼ 23 % ) improvement to CORE (resp. HLL). For example, on Wikipedia, the average NRMSE of SimCar is 0.0285 , while those of HLL and CORE are 0.0337 and 0.0475 , respectively. Figure 4 further shows the average NRMSEs for different k. We can see that the average NRMSEs of all methods decrease as k increases, and our method is consistently more accurate than HLL and CORE. In addition, we compute the fine-grained NRMSEs with respect to the user cardinality ranging from 1 to 10 , 000 with k = 2 8 and k = 2 9 . As shown in Figure 5, both HLL and CORE exhibit large estimation errors when user cardinality is small because they both use two different estimators for cardinalities within two different ranges, respectively. However, our method, SimCar, just uses one estimator, and we find that SimCar decreases the NRMSEs of HLL and CORE by 10 % − 42 % for different user cardinalities.
Figure 3. User cardinality estimation errors of our framework, SimCar, compared with HLL and CORE, k 1 = k = 2 9 by default.
Figure 4. User cardinality estimation errors of our framework, SimCar, compared with HLL and CORE, k 1 = k = 2 9 by default.
Figure 5. User cardinality estimation errors of our framework, SimCar, compared with HLL and CORE.
Results of common-interest count estimation. As shown in Figure 6a–c, where we set k 1 = k , we can see that our method, SimCar, is more accurate than the other two methods for all three schemes introduced in Section 6.2, and scheme 1 performs the worst among the three schemes, whose average NRMSEs are up to 8.7 × those of the other two schemes. We raised k 1 to 2 k , and the results are shown in Figure 6d–f. We can see that SimCar still outperforms MinHash and HLL and OPH and HLL for most of the datasets. Figure 7 show the average NRMSEs for different k on datasets Reuters, Tropes, and TREC, where we set k 1 = k . We can see that the average NRMSEs of all methods decrease as k increases. For scheme 1, our method, SimCar, is up to four and three times more accurate than MinHash and HLL and OPH and HLL, respectively. For schemes 2 and 3, we find that OPH and HLL gives results comparable to SimCar when k is medium. However, our method, SimCar, still outperforms OPH and HLL when k ≥ 2 10 .
Figure 6. Average NRMSEs of SimCar compared with MinHash and HLL and OPH and HLL for estimating common-interest counts.
Figure 7. Average NRMSEs of SimCar compared with MinHash and HLL and OPH and HLL for estimating common-interest counts.
Results of Jaccard similarity estimation and nearest neighbor search. We compared our method to the MinHash sketch and densified OPH sketch for Jaccard similarity estimation and nearest neighbor search. We used the technique discussed in Section 5.1 to build LSH for all three methods. The experimental results in Table 3 demonstrate that our method, SimCar, is comparable to MinHash and densified OPH on all the datasets.
Table 3. Performance of our framework, SimCar, compared with MinHash and OPH for Jaccard similarity estimation and nearest neighbor search with k = 2 9 , c = 2 .

7. Conclusions and Future Work

In this paper, a framework—SimCar—was developed for mining users’ interest sets. We built an OH sketch for each user that occurred in the data stream of interest, and one can query several mining results of users’ interest sets at any time of the data stream. Specially, we developed accurate methods for estimating cardinalities, common-interest counts, and Jaccard similarities of users’ interest sets. In addition, we used OH sketches to build LSH tables to quickly search for users with similar interests with sub-linear time. We evaluated the performance of our methods on real-world datasets. The experimental results demonstrated the effectiveness and efficacy of our methods. In future, we plan to use techniques such as register sharing (i.e., compress the OH sketches of all users into a large register array) to further reduce the memory usage.

Author Contributions

Conceptualization, K.Y. and Y.Q.; methodology, K.Y. and Y.Q.; software, K.Y. and Y.Q.; validation, K.Y.; formal analysis, K.Y. and Y.Q.; investigation, K.Y. and Y.Q.; resources, K.Y. and P.J.; data curation, Y.Q.; writing—original draft preparation, K.Y.; writing—review and editing, W.G., Y.Q., P.W. and P.J.; visualization, K.Y.; supervision, W.G., P.W. and P.J.; project administration, W.G.; funding acquisition, W.G. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by National Key R&D Program of China (2021YFB1715600).

Data Availability Statement

Publicly available datasets were analyzed in this study. This data can be found here: [http://movielens.umn.edu/; http://en.wikipedia.org/wiki/Wikipedia; http://tvtropes.org; http://trec.nist.gov/data/docseng.html].

Conflicts of Interest

The authors declare that they have no conflict of interest.

Appendix A

(1) Formula of the gradient g ( λ → ) . Each element of g ( λ → ) is computed as
∂ log L ( x → u , x → v ) ∂ λ u = k u , v ( 6 ) λ u + k u , v ( 3 ) + k u , v ( 5 ) λ u − λ u , v − ∑ i = 1 k x u , i k ,
∂ log L ( x → u , x → v ) ∂ λ v = k u , v ( 5 ) λ v + k u , v ( 2 ) + k u , v ( 6 ) λ v − λ u , v − ∑ i = 1 k x v , i k ,
∂ log L ( x → u , x → v ) ∂ λ u , v = k u , v ( 4 ) λ u , v − k u , v ( 3 ) + k u , v ( 5 ) λ u − λ u , v − k u , v ( 2 ) + k u , v ( 6 ) λ v − λ u , v + ∑ i = 1 k min ( x u , i , x v , i ) k .
(2) Formula of Hessian matrix H ( λ → ) . Each element of H ( λ → ) is computed as
∂ 2 log L ( x → u , x → v ) ∂ λ u 2 = − k u , v ( 6 ) λ u 2 − k u , v ( 3 ) + k u , v ( 5 ) ( λ u − λ u , v ) 2 ,
∂ 2 log L ( x → u , x → v ) ∂ λ v 2 = − k u , v ( 5 ) λ v 2 − k u , v ( 2 ) + k u , v ( 6 ) ( λ v − λ u , v ) 2 ,
∂ 2 log L ( x → u , x → v ) ∂ λ u , v 2 = − k u , v ( 4 ) λ u , v 2 − k u , v ( 3 ) + k u , v ( 5 ) ( λ u − λ u , v ) 2 − k u , v ( 2 ) + k u , v ( 6 ) ( λ v − λ u , v ) 2 ,
∂ 2 log L ( x → u , x → v ) ∂ λ u ∂ λ v = ∂ 2 log L ( x → u , x → v ) ∂ λ v ∂ λ u = 0 ,
∂ 2 log L ( x → u , x → v ) ∂ λ u ∂ λ u , v = ∂ 2 log L ( x → u , x → v ) ∂ λ u , v ∂ λ u = k u , v ( 3 ) + k u , v ( 5 ) ( λ u − λ u , v ) 2 ,
∂ 2 log L ( x → u , x → v ) ∂ λ v ∂ λ u , v = ∂ 2 log L ( x → u , x → v ) ∂ λ u , v ∂ λ v = k u , v ( 2 ) + k u , v ( 6 ) ( λ v − λ u , v ) 2 .
(3) Formula of E ( H ( λ → ) ) . From Equations (4)–(11), we derive expectations of variables k u , k v , E ( k u , v ( 1 ) ) , … , E ( k u , v ( 6 ) ) as
E ( k u ) = k e − λ u k , E ( k v ) = k e − λ v k ,
E ( k u , v ( 1 ) ) = k e − λ u + λ v − λ u , v k ,
E ( k u , v ( 2 ) ) = k ( e − λ u k − e − λ u + λ v − λ u , v k ) ,
E ( k u , v ( 3 ) ) = k ( e − λ v k − e − λ u + λ v − λ u , v k ) ,
E ( k u , v ( 4 ) ) = k λ u , v λ u + λ v − λ u , v ( 1 − e − λ u + λ v − λ u , v k ) ,
E ( k u , v ( 5 ) ) = k ( λ u − λ u , v ) λ u + λ v − λ u , v ( 1 − e − λ u + λ v − λ u , v k ) − E ( k u , v ( 3 ) ) ,
E ( k u , v ( 6 ) ) = k ( λ v − λ u , v ) λ u + λ v − λ u , v ( 1 − e − λ u + λ v − λ u , v k ) − E ( k u , v ( 2 ) ) .
Based the above equations and the formula of H ( λ → ) , we easily obtain the formula of each entry of matrix E ( H ( λ → ) ) .

References

  1. Cormode, G.; Muthukrishnan, S. An Improved Data Stream Summary: The Count-min Sketch and Its Applications. J. Algorithms 2005, 55, 58–75. [Google Scholar] [CrossRef] [Scilit]
  2. Estan, C.; Varghese, G.; Fisk, M. Bitmap algorithms for counting active flows on high speed links. In Proceedings of the SIGCOMM, Karlsruhe, Germany, 25–29 August 2003; pp. 182–209. [Google Scholar]
  3. Whang, K.; Vander-zanden, B.T.; Taylor, H.M. A linear-time probabilistic counting algorithm for database applications. IEEE Trans. Database Syst. 1990, 15, 208–229. [Google Scholar] [CrossRef] [Scilit]
  4. Durand, M.; Flajolet, P. Loglog Counting of Large Cardinalities; Springer: Berlin/Heidelberg, Germany, 2003; pp. 605–617. [Google Scholar]
  5. Flajolet, P.; Fusy, E.; Gandouet, O.; Meunier, F. Hyperloglog: The analysis of a near-optimal cardinality estimation algorithm. In Proceedings of the AOFA, Nice, France, 17–22 June 2007. [Google Scholar]
  6. Giroire, F. Order statistics and estimating cardinalities of massive data sets. Discret. Appl. Math. 2009, 157, 406–427. [Google Scholar] [CrossRef] [Scilit]
  7. Kane, D.M.; Nelson, J.; Woodruff, D.P. An Optimal Algorithm for the Distinct Elements Problem. In Proceedings of the PODS, Indianapolis, IN, USA, 6–11 June 2010; pp. 41–52. [Google Scholar]
  8. Zhao, Q.; Kumar, A.; Xu, J. Joint data streaming and sampling techniques for detection of super sources and destinations. In Proceedings of the ACM SIGCOMM IMC 2005, Berkeley, CA, USA, 19–21 October 2005; pp. 77–90. [Google Scholar]
  9. Yoon, M.; Li, T.; Chen, S.; Peir, J.K. Fit a spread estimator in small memory. In Proceedings of the IEEE INFOCOM 2009, Rio de Janeiro, Brazil, 19–25 April 2009; pp. 504–512. [Google Scholar]
  10. Wang, P.; Guan, X.; Qin, T.; Huang, Q. A Data Streaming Method for Monitoring Host Connection Degrees of High-Speed Links. IEEE Trans. Inf. Forensics Secur. 2011, 6, 1086–1098. [Google Scholar] [CrossRef] [Scilit]
  11. Xiao, Q.; Chen, S.; Chen, M.; Ling, Y. Hyper-Compact Virtual Estimators for Big Network Data Based on Register Sharing. In Proceedings of the SIGMETRICS, Portland, OR, USA, 15–19 June 2015; pp. 417–428. [Google Scholar]
  12. Chen, A.; Cao, J.; Shepp, L.; Nguyen, T. Distinct counting with a self-learning bitmap. J. Am. Stat. Assoc. 2011, 106, 879–890. [Google Scholar] [CrossRef] [Scilit]
  13. Ting, D. Towards Optimal Cardinality Estimation of Unions and Intersections with Sketches. In Proceedings of the SIGKDD, San Francisco, CA, USA, 13–17 August 2016. [Google Scholar]
  14. Zhao, P.; Aggarwal, C.C.; He, G. Link prediction in graph streams. In Proceedings of the 32nd IEEE International Conference on Data Engineering, (ICDE 2016), Helsinki, Finland, 16–20 May 2016; pp. 553–564. [Google Scholar]
  15. Cohen, R.; Katzir, L.; Yehezkel, A. A Minimal Variance Estimator for the Cardinality of Big Data Set Intersection. In Proceedings of the SIGKDD, Halifax, NS, Canada, 13–17 August 2017. [Google Scholar]
  16. Broder, A.Z.; Charikar, M.; Frieze, A.M.; Mitzenmacher, M. Min-Wise Independent Permutations. J. Comput. Syst. Sci. 2000, 60, 630–659. [Google Scholar] [CrossRef] [Scilit]
  17. Li, P.; Owen, A.B.; Zhang, C. One Permutation Hashing. In Proceedings of the NIPS, Lake Tahoe, NV, USA, 3–6 December 2012; pp. 3122–3130. [Google Scholar]
  18. Li, P.; König, A.C. b-Bit minwise hashing. In Proceedings of the WWW, Raleigh, NC, USA, 26–30 April 2010; pp. 671–680. [Google Scholar]
  19. Mitzenmacher, M.; Pagh, R.; Pham, N. Efficient estimation for high similarities using odd sketches. In Proceedings of the WWW, Doha, Qatar, 7–11 April 2014; pp. 109–118. [Google Scholar]
  20. Shrivastava, A.; Li, P. Improved Densification of One Permutation Hashing. In Proceedings of the UAI, Quebec City, QC, Canada, 23–27 July 2014; pp. 732–741. [Google Scholar]
  21. Shrivastava, A.; Li, P. Densifying One Permutation Hashing via Rotation for Fast Near Neighbor Search. In Proceedings of the ICML, Beijing, China, 21–26 June 2014; pp. 557–565. [Google Scholar]
  22. Shrivastava, A. Optimal Densification for Fast and Accurate Minwise Hashing. In Proceedings of the ICML, Sydney, Australia, 6–11 August 2017; pp. 3154–3163. [Google Scholar]
  23. Indyk, P.; Motwani, R. Approximate Nearest Neighbors: Towards Removing the Curse of Dimensionality. In Proceedings of the STOC, Dallas, TX, USA, 23–26 May 1998; pp. 604–613. [Google Scholar]
  24. Gionis, A.; Indyk, P.; Motwani, R. Similarity Search in High Dimensions via Hashing. In Proceedings of the PVLDB, Edinburgh, UK, 7–10 September 1999; pp. 518–529. [Google Scholar]
  25. Charikar, M. Similarity estimation techniques from rounding algorithms. In Proceedings of the STOC, Montreal, QC, Canada, 19–21 May 2002; pp. 380–388. [Google Scholar]
  26. Datar, M.; Immorlica, N.; Indyk, P.; Mirrokni, V.S. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the SOCG, Brooklyn, NY, USA, 8–11 June 2004; pp. 253–262. [Google Scholar]
  27. Wang, P.; Qi, Y.; Zhang, Y.; Zhai, Q.; Wang, C.; Lui, J.C.S.; Guan, X. A Memory-Efficient Sketch Method for Estimating High Similarities in Streaming Sets. In Proceedings of the KDD, Anchorage, AK, USA, 4–8 August 2019; pp. 25–33. [Google Scholar]
  28. Li, X.; Li, P. C-OPH: Improving the Accuracy of One Permutation Hashing (OPH) with Circulant Permutations. arXiv 2021, arXiv:2111.09544. [Google Scholar]
  29. Fernandez, R.C.; Min, J.; Nava, D.; Madden, S. Lazo: A Cardinality-Based Method for Coupled Estimation of Jaccard Similarity and Containment. In Proceedings of the 2019 IEEE 35th International Conference on Data Engineering (ICDE), Macao, China, 8–11 April 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1190–1201. [Google Scholar]
  30. Manasse, M.; McSherry, F.; Talwar, K. Consistent Weighted Sampling; Technical Report; John Cappelen: Copenhagen, Denmark, 2010. [Google Scholar]
  31. Haeupler, B.; Manasse, M.S.; Talwar, K. ling Made Fast, Small, and Easy. CoRR 2014. abs/1410.4266. [Google Scholar]
  32. Ioffe, S. Improved Consistent Sampling, Weighted Minhash and L1 Sketching. In Proceedings of the ICDM, Sydney, Australia, 13–17 December 2010; pp. 246–255. [Google Scholar]
  33. Li, P. 0-Bit Consistent Weighted Sampling. In Proceedings of the KDD, Sydney, Australia, 10–13 August 2015; pp. 665–674. [Google Scholar]
  34. Wu, W.; Li, B.; Chen, L.; Zhang, C. Canonical Consistent Weighted Sampling for Real-Value Weighted Min-Hash. In Proceedings of the ICDM, Barcelona, Spain, 12–15 December 2016; pp. 1287–1292. [Google Scholar]
  35. Shrivastava, A. Simple and Efficient Weighted Minwise Hashing. In Proceedings of the NIPS, Barcelona, Spain, 5–10 December 2016; pp. 1498–1506. [Google Scholar]
  36. Wu, W.; Li, B.; Chen, L.; Zhang, C. Consistent Weighted Sampling Made More Practical. In Proceedings of the WWW, Seville, Spain, 16–18 November 2017; pp. 1035–1043. [Google Scholar]
  37. Ertl, O. BagMinHash-Minwise Hashing Algorithm for Weighted Sets. CoRR 2018. abs/1802.03914. [Google Scholar]
  38. Li, P.; Li, X.; Samorodnitsky, G.; Zhao, W. Consistent Sampling Through Extremal Process. In Proceedings of the Web Conference 2021, Ljubljana, Slovenia, 19–23 April 2021; pp. 1317–1327. [Google Scholar]
  39. Moulton, R.; Jiang, Y. Maximally Consistent Sampling and the Jaccard Index of Probability Distributions. arXiv 2018, arXiv:1809.04052. [Google Scholar]
  40. Qi, Y.; Wang, P.; Zhang, Y.; Zhao, J.; Tian, G.; Guan, X. Fast Generating A Large Number of Gumbel-Max Variables. In Proceedings of the WWW, Taipei, Taiwan, 20–24 April 2020; pp. 796–807. [Google Scholar]
  41. Panigrahy, R. Entropy based nearest neighbor search in high dimensions. In Proceedings of the SODA, Miami, FL, USA, 22–26 January 2006; pp. 1186–1195. [Google Scholar]
  42. Lv, Q.; Josephson, W.; Wang, Z.; Charikar, M.; Li, K. Multi-probe LSH: Efficient indexing for high-dimensional similarity search. In Proceedings of the VLDB, Vienna, Austria, 23–27 September 2007; pp. 950–961. [Google Scholar]
  43. Huang, Q.; Feng, J.; Zhang, Y.; Fang, Q.; Ng, W. Query-aware locality-sensitive hashing for approximate nearest neighbor search. PVLDB 2015, 9, 1–12. [Google Scholar] [CrossRef] [Scilit]
  44. Gan, J.; Feng, J.; Fang, Q.; Ng, W. Locality-sensitive hashing scheme based on dynamic collision counting. In Proceedings of the SIGMOD, Scottsdale, AZ, USA, 20 May 2012; pp. 541–552. [Google Scholar]
  45. Liu, Y.; Cui, J.; Huang, Z.; Li, H.; Shen, H.T. SK-LSH: An efficient index structure for approximate nearest neighbor search. PVLDB 2014, 7, 745–756. [Google Scholar] [CrossRef] [Scilit]
  46. Tao, Y.; Yi, K.; Sheng, C.; Kalnis, P. Quality and efficiency in high dimensional nearest neighbor search. In Proceedings of the SIGMOD, Providence, RI, USA, 29 June–2 July 2009; pp. 563–576. [Google Scholar]
  47. Satuluri, V.; Parthasarathy, S. Bayesian locality sensitive hashing for fast similarity search. PVLDB 2012, 5, 430–441. [Google Scholar] [CrossRef] [Scilit]
  48. Gao, J.; Visvesvaraya Jagadish, H.; Lu, W.; Chin Ooi, B. DSH: Data Sensitive Hashing for high-dimensional k-NN search. In Proceedings of the SIGMOD, Snowbird, UT, USA, 22–27 June 2014. [Google Scholar]
  49. Wang, Y.; Shrivastava, A.; Ryu, J. Randomized Algorithms Accelerated over CPU-GPU for Ultra-High Dimensional Similarity Search. In Proceedings of the SIGMOD, Houston, TX, USA, 10–15 June 2018. [Google Scholar]
  50. Ahle, T.D.; Pagh, R.; Razenshteyn, I.; Silvestri, F. On the complexity of inner product similarity join. In Proceedings of the 35th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems, San Francisco, CA, USA, 26 June–1 July 2016; pp. 151–164. [Google Scholar]
  51. Neyshabur, B.; Srebro, N. On Symmetric and Asymmetric LSHs for Inner Product Search. In Proceedings of the International Conference on Machine Learning, Lille, France, 6 July–11 July 2015; pp. 1926–1934. [Google Scholar]
  52. Shrivastava, A.; Li, P. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canafa, 8–13 December 2014; pp. 2321–2329. [Google Scholar]
  53. Ting, D. Streamed Approximate Counting of Distinct Elements: Beating Optimal Batch Methods. In Proceedings of the SIGKDD, New York, NY, USA, 24–27 August 2014; pp. 442–451. [Google Scholar]
  54. Bachrach, Y.; Finkelstein, Y.; Gilad-Bachrach, R.; Katzir, L.; Koenigstein, N.; Nice, N.; Paquet, U. Speeding up the xbox recommender system using a euclidean transformation for inner-product spaces. In Proceedings of the 8th ACM Conference on Recommender Systems, Silicon Valley, CA, USA, 6 October 2014; pp. 257–264. [Google Scholar]
  55. Ballard, G.; Kolda, T.G.; Pinar, A.; Seshadhri, C. Diamond sampling for approximate maximum all-pairs dot-product (MAD) search. In Proceedings of the ICDM, Atlantic City, NJ, USA, 14–17 November 2015; pp. 11–20. [Google Scholar]
  56. Flajolet, P.; Martin, G.N. Probabilistic counting algorithms for data base applications. J. Comput. Syst. Sci. 1985, 31, 182–209. [Google Scholar] [CrossRef] [Scilit]
  57. Xiao, Q.; Zhou, Y.; Chen, S. Better with fewer bits: Improving the performance of cardinality estimation of large data streams. In Proceedings of the INFOCOM, Atlanta, GA, USA, 1–4 May 2017; pp. 1–9. [Google Scholar]
  58. Cohen, E.; Kaplan, H. Summarizing Data Using Bottom-k Sketches. In Proceedings of the PODC, Portland, OR, USA, 12–15 August 2007; pp. 225–234. [Google Scholar]
  59. Lumbroso, J. An optimal cardinality estimation algorithm based on order statistics and its full analysis. In Proceedings of the AofA, Vienna, Austria, 28 June–2 July 2010; pp. 489–504. [Google Scholar]
  60. Chen, W.; Liu, Y.; Guan, Y. Cardinality change-based early detection of large-scale cyber-attacks. In Proceedings of the INFOCOM, Turin, Italy, 14–19 April 2013; IEEE: Piscataway, NJ, USA, 2013; pp. 1788–1796. [Google Scholar]
  61. Flajolet, P. On Adaptive Sampling. Computing 1990, 43, 391–400. [Google Scholar] [CrossRef] [Scilit]
  62. Gibbons, P.B. Distinct Sampling for Highly-Accurate Answers to Distinct Values Queries and Event Reports. In Proceedings of the PVLDB, Roma, Italy, 11–14 September 2001; pp. 541–550. [Google Scholar]
  63. Mao, Y.; Gan, D.; Mwakapesa, D.S.; Nanehkaran, Y.A.; Tao, T.; Huang, X. A MapReduce-based K-means clustering algorithm. J. Supercomput. 2022, 78, 5181–5202. [Google Scholar] [CrossRef] [Scilit]
  64. Corizzo, R.; Pio, G.; Ceci, M.; Malerba, D. DENCAST: Distributed density-based clustering for multi-target regression. J. Big Data 2019, 6, 1–27. [Google Scholar] [CrossRef] [Scilit]
  65. Corizzo, R.; Dauphin, Y.; Bellinger, C.; Zdravevski, E.; Japkowicz, N. Explainable image analysis for decision support in medical healthcare. In Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA, 15–18 December 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 4667–4674. [Google Scholar]
  66. Cao, M.; Jia, W.; Lv, Z.; Zheng, L.; Liu, X. Superpixel-Based Feature Tracking for Structure from Motion. Appl. Sci. 2019, 9, 2961. [Google Scholar] [CrossRef] [Scilit]
  67. Ding, K.; Yang, Z.; Wang, Y.; Liu, Y. An improved perceptual hash algorithm based on u-net for the authentication of high-resolution remote sensing image. Appl. Sci. 2019, 9, 2972. [Google Scholar] [CrossRef] [Scilit]
  68. Jacquet, P.; Szpankowski, W. Analytical Depoissonization and its Applications. Theor. Comput. Sci. 1998, 201, 1–62. [Google Scholar] [CrossRef] [Scilit]
  69. Mitzenmacher, M.; Upfal, E. Probability and Computing—Randomized Algorithms and Probabilistic Analysis; Cambridge University Press: Cambridge, UK, 2005. [Google Scholar]
  70. Bickel, P.J.; Doksum, K.A. Mathematical Statistics: Basic Ideas and Selected Topics, 2nd ed.; Prentice-Hall: Hoboken, NJ, USA, 2001; Volume 1. [Google Scholar]
  71. Ypma, T.J. Historical Development of the Newton-Raphson Method. SIAM Rev. 1995, 37, 531–551. [Google Scholar] [CrossRef] [Scilit]
  72. Mislove, A.; Marcon, M.; Gummadi, K.P.; Druschel, P.; Bhattacharjee, B. Measurement and analysis of online social networks. In Proceedings of the SIGCOMM, Kyoto, Japan, 27–31 August 2007; pp. 29–42. [Google Scholar]
  73. GroupLens Research. MovieLens Data Sets. 2006. Available online: http://www.grouplens.org/node/73 (accessed on 1 March 2022).
  74. Wikimedia Foundation. Wikimedia Downloads. 2010. Available online: http://dumps.wikimedia.org/ (accessed on 13 June 2021).
  75. Lewis, D.D.; Yang, Y.; Rose, T.G.; Li, F. RCV1: A New Benchmark Collection for Text Categorization Research. J. Mach. Learn. Res. 2004, 5, 361–397. [Google Scholar]
  76. Kunegis, J. KONECT: The Koblenz Network Collection. In Proceedings of the 22nd International Conference on World Wide Web, Rio de Janeiro, Brazil, 13–17 May 2013; pp. 1343–1350. [Google Scholar]
  77. National Institute of Standards and Technology. Text REtrieval Conference (TREC) English Documents. 2010; Volume 4–5. Available online: http://trec.nist.gov/data/docs_eng.html (accessed on 11 June 2021).
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.