Text Indexing for Faster Gapped Pattern Matching

Abstract

We revisit the following version of the Gapped String Indexing problem, where the goal is to preprocess a text $T\left[1..n\right]$ to enable efficient reporting of all $\mathrm{occ}$ occurrences of a gapped pattern $P=P_1\left[\alpha ..\beta \right]P_2$ in T. An occurrence of P in T is defined as a pair $(i,j)$ where substrings $T[i..i+|P_1\left|\right)$ and $T[j..j+|P_2\left|\right)$ match $P_1$ and $P_2$, respectively, with a gap $j-(i+|P_1\left|\right)$ lying within the interval $\left[\alpha ..\beta \right]$. This problem has significant applications in computational biology and text mining. A hardness result on this problem suggests that any index with polylogarithmic query time must occupy near quadratic space. In a recent study [STACS 2024], Bille et al. presented a sub-quadratic space index using space $\tilde{\mathcal{O}}\left(n^{2-\delta /3}\right)$, where $0\le \delta \le 1$ is a parameter fixed at the time of index construction. Its query time is $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }·(1+\mathrm{occ}))$, which is sub-linear per occurrence when $\delta <1$. We show how to achieve a gap-sensitive query time of $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }·(1+{\mathrm{occ}}^{1-\delta })+{\sum }_{g\in \left[\alpha ..\beta \right]}{\mathrm{occ}}_g·g^{\delta })$ using the same space, where ${\mathrm{occ}}_g$ denotes the number of occurrences with gap g. This is faster when there are many occurrences with small gaps.

1. Introduction

Let $T\left[1..n\right]$ be a string (called the text) over a polynomially sized alphabet and $P=P_1\left[\alpha ..\beta \right]P_2$ be a gapped pattern, where $P_1$ and $P_2$ are strings and $\left[\alpha ..\beta \right]$ is an integer interval called the gap range. An occurrence of P in T is represented as a pair $(i,j)$ such that $T[i..i+|P_1\left|\right)=P_1$, $T[j..j+|P_2\left|\right)=P_2$ with gap $j-(i+|P_1\left|\right)\in \left[\alpha ..\beta \right]$. Locating occurrences of gapped patterns has numerous applications in computational biology [1,2,3,4,5,6,7] and text mining [8,9,10,11]. The algorithmic variant of the gapped pattern matching problem is well studied  [12,13] and can be solved in $\tilde{\mathcal{O}}(n+m+\mathrm{occ})$ time ($\tilde{\mathcal{O}}(·)$ suppresses polylogarithmic factors, in particular, ${(logn)}^k=\tilde{\mathcal{O}}\left(1\right)$ for any constant k.) [3,6,14,15,16].

This work focuses on an indexing version of the problem where the text T is known during preprocessing. The gapped pattern $P=P_1\left[\alpha ..\beta \right]P_2$ is provided as a query. Formally, we consider Problem 1.

Problem 1 

(Gapped String Indexing). 

• Preprocess: A text $T\left[1..n\right]$.
• Query: Given a gapped pattern $P=P_1\left[\alpha ..\beta \right]P_2$, report all occurrences of P in T.

In recent work, Bille et al. [17] showed that for all $0\le \delta \le 1$, an index for Problem 1 can be constructed occupying $\tilde{\mathcal{O}}\left(n^{2-\delta /3}\right)$ or $\tilde{\mathcal{O}}\left(n^{3-2\delta }\right)$ space, and answering queries $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }·(\mathrm{occ}+1))$ time. Our main result is an index that requires similar space but achieves query times parameterized by the gaps present in the occurrences. This is stated formally in Theorem 1 below. In particular, our result improves the case where gaps for most occurrences are small.

Theorem 1. 

For all $0\le \delta \le 1$, there exists an index for Gapped String Indexing that occupies $\tilde{\mathcal{O}}\left(n^{2-\delta /3}\right)$ space and answers queries in time

$$\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }·(1+{\mathrm{occ}}^{1-\delta })+\sum _{g\in \left[\alpha ..\beta \right]}{\mathrm{occ}}_g·g^{\delta })$$

where ${\mathrm{occ}}_g$ is the number of occurrences with gap g.

Our main technique revolves around solving the following bounded variant of Gapped String Indexing, which may be of independent interest.

Problem 2 

(Bounded Gapped String Indexing).  

• Preprocess: A text $T\left[1..n\right]$ and integer G.
• Query: Given a gapped pattern $P=P_1\left[\alpha ..\beta \right]P_2$ where $\beta <G$, report all occurrences of P in T.

For Problem 2, we provide a solution with space and query complexity stated in Theorem 2.

Theorem 2. 

For every $0\le \delta \le 1$, there exists an index for Bounded Gapped String Indexing that occupies $\tilde{\mathcal{O}}\left(n^{2-\delta /3}\right)$ space and answers queries in time

$$\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }·(1+{\mathrm{occ}}^{1-\delta })+G^{\delta }·\mathrm{occ})$$

where $\mathrm{occ}$ is the size of the output.

To prove Theorem 2, we build on previous results for the Gapped Set Intersection Problem (defined formally in Section 2), developing techniques for the bounded-gap case. We utilize a blocking technique based on binary trees, which when combined with a generalized form of the Kraft–McMillan inequality, allows us to achieve an improved query time (Lemma 9). Our preprocessing techniques differ from those of Bille et al. [17] in that we rely on the bounded nature of the gaps to perform the proposed blocking technique. Indeed, Theorem 2 is accomplished through extra preprocessing (blocking and building data structures for blocks) that is possible only when an upper bound $G>\beta$ is known in advance. As described in Section 3.6, Theorem 1 is then achieved by applying Theorem 2 for different ranges of G.

1.1. Previous Work

A long line of research has contributed to the current results on Gapped String Indexing. Many of these place some restrictions on the problem. The earliest results are by Peterlongo et al. [18], and are for the heavily restricted version where lengths $|P_1|$, $|P_2|$, and gap $g=\alpha =\beta$ are known for preprocessing. Given these restrictions, their solution uses $\mathcal{O}\left(n\right)$ space and achieves an optimal query time of $\mathcal{O}(m+\mathrm{occ})$.

In a slightly generalized variant where only the gap length $g=\alpha =\beta$ is given at preprocessing, Iliopoulos and Rahman [19] present an index using space $\mathcal{O}\left(n{log}^{1+ϵ}n\right)$, where $ϵ>0$ is an arbitrarily small constant, with query time $\mathcal{O}(m+loglogn+\mathrm{occ})$. For the same case where g is known in advance, Bille and Gørtz [17] introduced an improved approach, achieving optimal query time with $\mathcal{O}\left(n{log}^{ϵ}n\right)$ space. In the case where only an upper bound G is given on $\beta$, the problem can be reduced to 3D range searching with an index using $\tilde{\mathcal{O}}\left(Gn\right)$ space and $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+\mathrm{occ})$ query time [20]. Conditioned on the Strong Set-Disjointness Conjecture [21], Bille et al. [22] also demonstrated that any solution with $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+\mathrm{occ})$ query time must use $\tilde{\mathcal{O}}\left(Gn\right)$ space. Recently, Ganguly et al. [23] proposed a variant (called Bounded Ratio Gapped String Indexing) where the gap $\beta$ satisfies $\beta \le \gamma ·\left(\right|P_1|+|P_2\left|\right)$. Here, $\gamma$ represents a gap ratio fixed at index construction time. Under this relaxed constraint, an index can be constructed occupying $\tilde{\mathcal{O}}(\gamma ·n)$ space and having $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+\mathrm{occ})$ query time.

The framework employed by Bille et al. [17] utilizes the results for 3-SUM Indexing by Golovnev [24] (see also [25]). In 3-SUM indexing, one needs to preprocess two sets of integers, $S_1$ and $S_2$, so that given a query integer c, one can efficiently determine if there exists $a\in S_1$ and $b\in S_2$ such that $a+b=c$. The reduction from Gapped String Indexing to 3-SUM Indexing, through a series of intermediate problems, forms the basis of both [17] and this work. Leveraging one of these intermediate problems (between Gapped Indexing and 3-SUM Indexing) alleviates some steps for this work relative to Bille et al.’s results [17].

1.2. Notation and Technical Preliminaries

We use $\left[i..j\right]$ to refer to the set $\{i,i+1,\cdots ,j-1,j\}$ and $\left[i..j\right)$ to the set $\{i,i+1,\cdots ,j-1\}$. For an array of integers A, we use $A\left[i..j\right]$ to denote the subarray $A\left[i\right]\cdots A\left[j\right]$ and $A\left[i..j\right)$ to denote $A\left[i\right]\cdots A[j-1]$.

For a string T, we use $T\left[i\right]$ to refer to the $i^{th}$ symbol in T, $T\left[i..j\right]$ to denote the substring $T\left[i\right]\cdots T\left[j\right]$, and $T\left[i..j\right)$ the substring $T\left[i\right]\cdots T[j-1]$. We call a substring of the form $T\left[i..n\right]$ a suffix of T and a substring of the form $T\left[1..i\right]$ a prefix of T. We use $T^R$ to denote the reverse of the string T. The suffix tree [26] of a string $T\left[1..n\right]$ is a compact tree of all suffixes with leaves in corresponding lexicographic order. The suffix array, denoted $SA\left[1..n\right]$, is defined such that $T[SA[i\left]..n\right]$ is the $i^{th}$ suffix when all suffixes are sorted in lexicographic order. For a pattern P, its suffix range $\left[a..b\right]$ is the maximal range such that for all $h\in \left[a..b\right]$, $T[SA[h\left]..n\right]$ has prefix P. The suffix range exists if P occurs in T; otherwise, the suffix range is empty.

For convenience, we assume that all strings are over a polynomially-sized integer alphabet so that the suffix tree and suffix array can be constructed in linear time [27]. Given the suffix tree and suffix array, the suffix range of a string $P\left[1..m\right]$ can be found in $\mathcal{O}\left(m\right)$ time.

2. A Preliminary Solution

Before introducing our main solution, we first present a preliminary approach. While this solution does not achieve the query efficiency required to prove Theorem 1, it serves as a valuable foundation for our solution. As an initial step, we introduce the following two problem formulations from [17].

Problem 3 

(Gapped Set Intersection (with Reporting)).

• Preprocess: A collection of subsets $S_1$, ⋯, $S_k$ of total size $N={\sum }_{i=1}^k\left|S_i\right|$ over integer universe $\left[0..U\right]$.
• Query: Given $(i,j,\alpha ,\beta )$, report if there exists (report all, resp.) $(a,b)\in S_i\times S_j$ where there exists $s\in \left[\alpha ..\beta \right]$ such that $a+s=b$.

We also define the bounded version of Problem 3, analogous to Problem 2. Specifically, the bounded problem formulation provides a collection of subsets $S_1$, ⋯, $S_k$, and an integer G for preprocessing. A query consists of the tuple $(i,j,\alpha ,\beta )$ with the additional guarantee that $\beta <G$. We will utilize the following results from Bille et al. [17].

Lemma 1 

(Index for Gapped Set Intersection (Theorems 5 and 6 from [17])). For every $0\le \delta \le 1$, there is a data structure for the Gapped Set Intersection that occupies $\tilde{\mathcal{O}}\left(N^{2-\delta /3}\right)$ space and answers existential queries in time $\tilde{\mathcal{O}}\left(N^{\delta }\right)$ and reporting queries in time $\tilde{\mathcal{O}}(N^{\delta }·(\mathrm{occ}+1))$, where $\mathrm{occ}$ is the size of the output.

Lemma 2 allows us to focus on Problem 3 for the majority of the remaining work.

Lemma 2 

(Reduction to Bounded Gapped Set Intersection (Adapted from [17]). Assume there is a data structure for the Bounded Gapped Set Intersection (with Reporting) using $s\left(N\right)$ space that answers existential queries in time $t\left(N\right)$ and reporting queries in time $t\left(N\right)·(1+\mathrm{occ})$. Then, there exists a data structure for Bounded Gapped String Indexing using $\tilde{\mathcal{O}}(n+s\left(n\right))$ space. It can answer existential queries in time $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+t\left(n\right))$ and reporting queries in time $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+t\left(n\right)·(1+\mathrm{occ}))$, where n is the length of the input text and $\mathrm{occ}$ is the size of the output.

For completeness, we sketch an adaptation of the proof from [17].

Proof. 

We begin with the array $S_2\left[1..n\right]$ such that $S_2\left[i\right]=SA\left[i\right]$, where $SA[·]$ is the suffix array of T. We also define the array $S_1\left[1..n\right]$, where $S_1\left[i\right]=n-SA{\left[i\right]}^R+1$, and $SA{[·]}^R$ is the suffix array of the reverse string $T^R$. Both arrays are decomposed into subsets corresponding to dyadic intervals of the form $[1+\kappa ·2^j..(\kappa +1)·2^j]$, where $0\le \kappa \le \lfloor n/2^j\rfloor -1$ and $0\le j\le \lfloor logn\rfloor$. These subsets serve as the input to the Gapped Set Intersection instance. Notably, the sum of the subset cardinalities is $\mathcal{O}(nlogn)$.

Given a query $P_1\left[\alpha ..\beta \right]P_2$, we decompose the suffix range for $P_1^R$ into a collection of $\mathcal{O}(logn)$ dyadic-sized subarrays of $S_1$, corresponding to precomputed subsets. We denote this collection by $\mathcal{A}$. Similarly, we decompose the suffix range of $P_2$ into a collection of $\mathcal{O}(logn)$ dyadic intervals corresponding to precomputed subsets of $S_2$ denoted by $\mathcal{B}$. We then perform $\mathcal{O}\left({log}^{2}n\right)$ queries (For notational brevity here, we abuse notation slightly. The actual query would be on the indices corresponding to subsets A and B.) $(A,B,\alpha ,\beta )$ for all $A\in \mathcal{A}$ and $B\in \mathcal{B}$. It is important to note that the bounds $\alpha$ and $\beta$ remain unchanged throughout the reduction.    □

We next present a preliminary solution for the Bounded Gapped Set Intersection with space $\tilde{\mathcal{O}}\left(n{G}^{1-\delta /3}\right)$ and reporting time $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n/G^{1-\delta }+G^{\delta }\mathrm{occ})$.

2.1. The Data Structure

Let $A\left[1..N\right]$ be an array containing the elements of ${\cup }_{i=1}^k{S}_i$ in sorted order where we now define $N=\left|{\cup }_{i=1}^k{S}_i\right|$. We subdivide A into overlapping blocks of size $2G$, with each consecutive block overlapping by G elements. Formally, for $i\in [1..\lfloor N/G\rfloor )$, we define block $B_i=A[1+(i-1)G..1+(i+1)G)$. See Figure 1.

Figure 1. A preliminary blocking scheme for $N=32$ and $G=4$.

Given a query $(i,j,\alpha ,\beta )$ where $\beta <G$, consider the following: if $(a,b)\in S_i\times S_j$ and there exists an $s\in \left[\alpha ..\beta \right]$ such that $a+s=b$, then, since $s\le \beta <G$, a and b must lie in the same block. Consequently, we construct the data structure from Lemma 1 for each block, $B_h$, using the subsets $S_1^{\prime }=S_1\cap B_h$, ⋯, $S_k^{\prime }=S_k\cap B_h$, excluding empty subsets. Thus, the number of subsets in a given block may be fewer than k. We store in sorted order, for each block $B_h$, the original subset indices i for each non-empty subset $S_t^{\prime }=S_i\cap B_h$. We also store the associated t value in the same sorted order.

2.2. Querying

Given query $(i,j,\alpha ,\beta )$, we iterate through $h\in [1..\lfloor N/G\rfloor ]$. For block $B_h$, we first determine if $S_i\cap B_h$ is empty. This can be accomplished using binary search over the stored indices for $B_h$, which were described above. If $S_i\cap B_h=\varnothing$, we are finished for $B_h$ as there are no solutions in this block. Otherwise, we obtain $t_i$ such that $S_{t_i}^{\prime }=B_h\cap S_i$. Similarly, we perform a binary search for j over the indices for $B_h$. If $S_j\cap B_h=\varnothing$, we are finished for $B_h$. Otherwise, we obtain $t_j$ such that $S_{t_j}^{\prime }=B_h\cap S_j$. We then make the query $(t_i,t_j,\alpha ,\beta )$ to the data structure for block $B_h$ and store the reported solutions.

After all blocks are processed, a final sort of the stored solutions is performed and the duplicates are removed. We then output the resulting list of occurrences.

2.3. Analysis

The above approach requires $\tilde{\mathcal{O}}\left(G^{2-\delta /3}\right)$ space per block, resulting in an overall space requirement of $\tilde{\mathcal{O}}((N/G)·G^{2-\delta /3})=\tilde{\mathcal{O}}\left(N{G}^{1-\delta /3}\right)$. Reporting all occurrences within a single block $B_h$ takes $\tilde{\mathcal{O}}\left(G^{\delta }({\mathrm{occ}}_h+1)\right)$, where ${\mathrm{occ}}_h$ represents the number of occurrences in $B_h$. The total time across all blocks is $\tilde{\mathcal{O}}(G^{\delta }·N/G+G^{\delta }\mathrm{occ})=\tilde{\mathcal{O}}(N/G^{1-\delta }+G^{\delta }\mathrm{occ})$. Note that each solution occurs in at most two blocks, so the final sorting and duplicate removal step does not change the asymptotic query time.

Applying the reduction in Lemma 2, this yields a solution for Bounded Gapped String Indexing with Reporting that requires $\tilde{\mathcal{O}}\left(n{G}^{1-\delta /3}\right)$ space and achieves a query time of $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n/G^{1-\delta }+G^{\delta }\mathrm{occ})$. However, this does not provide the desired query time complexity, particularly for large values of G. We now present an improved solution.

3. An Improved Solution

The basis of the improved solution is to carefully decompose the array A (defined in Section 2) to avoid having to check every block for occurrences as was done in the preliminary solution. We rely heavily on techniques from [28] and its extensions in [22,29].

3.1. The Data Structure

We will construct a tree data structure over the array A. Each node in the tree will have an associated subarray of A. We construct the tree structure over the array A recursively as follows: The tree’s root is associated with the entire array $A\left[1..N\right]$. We designate the root’s midpoint as $m=\lfloor (1+N)/2\rfloor$. The root node is given a middle child, that is a leaf representing the subarray $A[m-G+1..m+G]$. We then recursively create two child subtrees: the left child subtree corresponds to $A\left[1..m\right]$, and the right child subtree corresponds to $A[m+1..N]$. If at any point the size of a subarray is at most $2G$, we treat the node as a leaf node. See Figure 2. For each node in the tree, we create the Gapped Set Intersection Data Structure from Lemma 1. For each leaf node, we create the Gapped Set Intersection Data Structure outlined in Lemma 1. See Algorithm 1 for pseudocode. Like in Section 2, these data structures are constructed over the non-empty subsets of each block, and we maintain the mapping from the query i and j to the corresponding non-empty subset if it exists. These details are omitted from the pseudocode.

Algorithm 1 Construction Algorithm

1: procedure Construct($A,G,l,r$)

2:     if $r-l+1>2G$ then

3:      $v\leftarrow$create_internal_node($A,l,r$)

4:      $m\leftarrow \lfloor (l+r)/2\rfloor$

5:      $v.\mathrm{middle}\_\mathrm{child}\leftarrow$ create_leaf_node($A,m-G+1,m+G$)

6:      $v.\mathrm{left}\_\mathrm{child}\leftarrow$ Construct($A,G,l,m$)

7:      $v.\mathrm{right}\_\mathrm{child}\leftarrow$ Construct($A,G,m+1,r$)

8:     else

9:      $v\leftarrow$create_leaf_node($A,l,r$)

10:     end if

11:     return v

12: end procedure

13: procedure Construct(A, G)

14:      $root\leftarrow$Construct($A,G,1,N$)

15: end procedure

Figure 2. Tree structure constructed by Algorithm 1 with $N=32$ and $G=4$.

In terms of notation, we call the leaves created in Line 5 of Algorithm 1 middle children leaves. For a node v with associated subarray $A\left[l..r\right]$, we call $m=\lfloor (l+r)/2\rfloor$ the midpoint of v.

3.2. Querying

To query the tree structure, we begin at the root. We first use the data structure from Lemma 1 to check whether any occurrence is contained in the current nodes’ associated subarray. If the current node is a leaf and it contains an occurrence, we report all occurrences using the data structure from Lemma 1. If the current node is not a leaf and contains an occurrence, we recursively search all of its children. This is shown in pseudocode in Algorithm 2.

Algorithm 2 Query Algorithm

1: procedure Search($v,i,j,\alpha ,\beta$)

2:       if not v.contains_occurrence($i,j,\alpha ,\beta$) then

3:             return

4:       else if v.contains_occurrence($i,j,\alpha ,\beta$) and v is leaf then

5:             v.report_all_occurrences($i,j,\alpha ,\beta$)

6:       else if v.contains_occurrence($i,j,\alpha ,\beta$) and v is internal node then

7:             Search($v.\mathrm{middle}\_\mathrm{child},i,j,\alpha ,\beta$)

8:             Search($v.\mathrm{left}\_\mathrm{child},i,j,\alpha ,\beta$)

9:             Search($v.\mathrm{right}\_\mathrm{child},i,j,\alpha ,\beta$)

10:       end if

11: end procedure

12: procedure Query($i,j,\alpha ,\beta$)

13:       Search($root,i,j,\alpha ,\beta$)

14: end procedure

3.3. Correctness

The key observation from Figure 2 is that the union of the leaf nodes in the tree constructed by Algorithm 1 resembles the blocking scheme described in Section 2 (see Figure 1 for comparison). Based on this observation, we now formalize the following key lemmas.

Lemma 3. 

Every subarray $A^{\prime }$ of size at most G is contained in the subarray of some leaf.

Proof. 

For the sake of contradiction, assume that subarray $A^{\prime }$ is not contained in the subarray of any leaf. Let v be the node of maximum height that contains $A^{\prime }$. Let l and r denote the bounds for the subarray for node v, and let $m=\lfloor (l+r)/2\rfloor$.

Since $A^{\prime }$ is not contained in any leaf, it must be that $A^{\prime }$ is not fully contained within the range $[m-G+1..m+G]$. Therefore, $A^{\prime }$ either starts in the range $[l..m-G]$ or ends in the range $[m+G+1..r]$. In the former case, since $m-G+|A^{\prime }|-1\le m-G+G=m$, we have that $A^{\prime }$ must be contained within the subarray of $v.$left_child, which contradicts the assumption that v is the highest node containing $A^{\prime }$. In the latter case, since $m+G+1-|A^{\prime }|+1\ge m+G+1-G+1>m+1$, we have that $A^{\prime }$ must be contained within the subarray of v.right_child, which again contradicts the assumption that v is the highest node containing $A^{\prime }$. □

The correctness of the above query procedure then follows from the fact that every block of size at most G is contained within the subarray of some leaf node v. All ancestors u of leaf v will report that they contain an occurrence, allowing the DFS traversal to continue until leaf v is reached and its occurrences are reported.

3.4. Space Analysis

First, considering only the Gapped Set Intersection Data Structure from Lemma 1 on non-leaf nodes, this requires space logarithmic factors from

$$\begin{array}{c}\hfill \sum _{i=0}^{logN}{2}^i{\left({\displaystyle \frac{N}{2^i}}\right)}^{2-\delta /3}=N^{2-\delta /3}\sum _{i=0}^{logN}{\left({\displaystyle \frac{1}{2^{1-\delta /3}}}\right)}^i.\end{array}$$

Since $0\le \delta \le 1$, we have $1/2^{1-\delta /3}<1$, and the geometric series converges to a constant. Hence, ignoring leaf nodes, the space is $\tilde{\mathcal{O}}\left(N^{2-\delta /3}\right)$.

Next, we include the leaves. We first show that the number of leaves is $\mathcal{O}(N/G)$.

Lemma 4. 

Every non-middle child leaf’s associated subarray has size at least G.

Proof. 

Suppose, for the sake of contradiction, there exists a leaf u that has a subarray size less than G. Let v be the parent of u with range l to r and midpoint m. If u is a left child, it has a subarray size

$$m-l+1=⌊{\displaystyle \frac{l+r}{2}}⌋-l+1<G.$$

The above implies

$${\displaystyle \frac{l+r}{2}}-1-l+1<G,$$

which leads to $r-l<2G$. However, this implies $r-l+1\le 2G$, so v had a subarray size of at most $2G$. In such a case, our algorithm would not recursively create a left child for v, a contradiction.

Similarly, if u is a right child with subarray size less than G, it has size

$$r-(m+1)+1=r-\left(⌊{\displaystyle \frac{l+r}{2}}⌋+1\right)+1<G,$$

meaning $r-\lfloor (l+r)/2\rfloor <G$. This implies $r-(l+r)/2<G$. Hence, $r-l<2G$. Again, we conclude v has a subarray size small enough that our algorithm would not recursively create a right child for v, a contradiction. □

Lemma 5. 

The number of leaves in the tree structure created by Algorithm 1 is $\mathcal{O}(N/G)$.

Proof. 

Since all leaves created as non-middle children have disjoint associated subarrays, their union represents a set of cardinality N, and (by Lemma 4) each represents a disjoint subset of size at least G. Therefore, there are at most $\mathcal{O}(N/G)$ non-middle child leaves. Next, because the tree (still excluding middle children) is a binary tree, the total number of internal nodes is also $\mathcal{O}(N/G)$. Furthermore, including the middle child leaves at most doubles the total number of nodes in the tree. □

Because each leaf contains a subarray of size $\mathcal{O}\left(G\right)$, the space for the data structures for each leaf is $\tilde{\mathcal{O}}\left(G^{2-\delta /3}\right)$. Combined with Lemma 5, the total space for leaves is $\tilde{\mathcal{O}}(G^{2-\delta /3}·N/G)=\tilde{\mathcal{O}}(G^{1-\delta /3}·N)$, which, since $G\le N$, is also $\tilde{\mathcal{O}}\left(N^{2-\delta /3}\right)$.

3.5. Query Time Analysis

We now analyze the run time of Algorithm 2. Our first step can be seen as a modification of the Kraft–McMillan inequality.

Lemma 6. 

For a rooted binary tree $\mathcal{T}=(V,E)$, let $h\left(v\right)$ denote the height of node v in $\mathcal{T}$. Then

$$\sum _{v\in V}{2}^{-h\left(v\right)}\le 1+log\left|V\right|.$$

Proof. 

We use induction on the tree height. The base case holds with a single node having height 0 since $2^0=1\le 1+log1=1$. For an arbitrary tree $\mathcal{T}=(V,E)$ with $\left|V\right|>1$ nodes, let the left subtree of the root be ${\mathcal{T}}_{\mathcal{L}}=(V_L,E_L)$ with relative height function $h_L$, and the right child of the root, ${\mathcal{T}}_R=(V_R,E_R)$, with relative height function $h_R$. Then,

$$\begin{array}{cc}\hfill \sum _{v\in V}{2}^{-h\left(v\right)}& =2^0+\sum _{v\in V_L}{2}^{-(h_L\left(v\right)+1)}+\sum _{v\in V_R}{2}^{-(h_R\left(v\right)+1)}\hfill \\ \hfill & =1+{\displaystyle \frac{1}{2}}\sum _{v\in V_L}{2}^{-h_L\left(v\right)}+{\displaystyle \frac{1}{2}}\sum _{v\in V_R}{2}^{-h_R\left(v\right)}\hfill \\ \hfill & \le 1+1+{\displaystyle \frac{1}{2}}log|V_L|+{\displaystyle \frac{1}{2}}log\left|V_R\right|(\mathrm{By}\mathrm{Inductive}\mathrm{Hypothesis})\hfill \\ \hfill & =1+1+log\sqrt{|V_L\left|\right|V_R|}\hfill \\ \hfill & \le 1+1+log{\displaystyle \frac{|V_L|+|V_R|}{2}}(\mathrm{Inequality}\mathrm{of}\mathrm{Arithmetic}\mathrm{and}\mathrm{Geometric}\mathrm{Means})\hfill \\ \hfill & =1+log\left(\right|V_L|+|V_R\left|\right)\hfill \\ \hfill & \le 1+log(1+|V_L|+|V_R\left|\right)=1+log\left|V\right|.\hfill \end{array}$$

□

Lemma 7. 

Let s be the number of leaves for which we have to run the “report_all_occurrences” subroutine in Algorithm 2. Let $\mathcal{V}$ be the set of nodes on whichSearchis executed in Algorithm 2. Then, $\left|\mathcal{V}\right|=\mathcal{O}(1+slogN)$.

Proof. 

The height of the tree constructed by Algorithm 1 is $\mathcal{O}(logN)$. Each root-to-v path for all v where “report_all_occurrences” is called contributes at most $\mathcal{O}(logN)$ calls of Search. Thus, the contribution overall of these paths is $\mathcal{O}(1+slogN)$. What remains to be counted are nodes where Search is called and “contains_occurrence” reports there are no occurrences. For these, observe that each node on the root-to-v path for all v where “report_all_occurrences” is called has at most two children where this can be the case. Thus, including these nodes at most triples the total number of nodes on which Search is called. □

We take s and $\mathcal{V}$ as defined in Lemma 7. First, we consider the time used by calls to “contains_occurrence”. Because the size of a subarray for a node v at height $h\left(v\right)$ is $\mathcal{O}(N/2^{h\left(v\right)})$, by Lemma 1, a call to “contains_occurrence” for a node v at height $h\left(v\right)$ requires time $\tilde{\mathcal{O}}\left({(N/2^{h\left(u\right)})}^{\delta }\right)$. The combined time used for “contains_occurrence” calls is polylogarithmic factors from

$$\sum _{v\in \mathcal{V}}{\left({\displaystyle \frac{N}{2^{h\left(u\right)}}}\right)}^{\delta }=N^{\delta }\sum _{v\in \mathcal{V}}{\left({\displaystyle \frac{1}{2^{h\left(v\right)}}}\right)}^{\delta }.$$

(1)

We next apply Hölder’s inequality to obtain the bound

$$\sum _{v\in \mathcal{V}}{\left({\displaystyle \frac{1}{2^{h\left(v\right)}}}\right)}^{\delta }=\sum _{v\in \mathcal{V}}{\left({\displaystyle \frac{1}{2^{h\left(v\right)}}}\right)}^{\delta }{1}^{1-\delta }\le {\left(\sum _{v\in \mathcal{V}}{\displaystyle \frac{1}{2^{h\left(v\right)}}}\right)}^{\delta }{\left(\sum _{v\in \mathcal{V}}1\right)}^{1-\delta }.$$

Applying Lemma 6 and 7, we can further bound this as

$${\left(\sum _{v\in \mathcal{V}}{\displaystyle \frac{1}{2^{h\left(v\right)}}}\right)}^{\delta }{\left(\sum _{v\in \mathcal{V}}1\right)}^{1-\delta }\le {(1+log|\mathcal{V}\left|\right)}^{\delta }{(1+slogN)}^{1-\delta }.$$

Substituting into Equation (1), we obtain that the time used for “contains_occurrence” calls is $\tilde{\mathcal{O}}\left(N^{\delta }{(1+s)}^{1-\delta }\right)$.

Next, we consider the time used by calls to “report_all_occurrences”. By Lemma 1, each leaf v on which “report_all_occurrences” is called takes time $\tilde{\mathcal{O}}\left(G^{\delta }(1+{\mathrm{occ}}_v)\right)$, where ${\mathrm{occ}}_v$ denotes the number of occurrences contained in the subarray for node v. To bound the time complexity, we should bound the number of times an occurrence can be reported over all blocks. To this end, we prove Lemma 8.

Lemma 8. 

Every subarray $A^{\prime }=A\left[a..b\right]$ of size $b-a+1\le G$ has a non-empty intersection with $\mathcal{O}\left(1\right)$ leaf’s subarrays.

Proof. 

Consider first the tree structure without any middle child leaves. In this case, the leaves are all disjoint and, by Lemma 4, have subarray size at least G. Hence, at most two non-middle children leaves have non-empty intersections with $A^{\prime }$.

Now, we incorporate the middle children. We consider the middle children leaves as being ordered according to their midpoint.

• Claim: The difference between consecutive midpoints of middle children leaves is at least G. To see this, consider a middle child of u with midpoint $m_u=\lfloor (l_u+r_u)/2\rfloor$ and a middle child of v with midpoint $m_v=\lfloor (l_v+r_v)/2\rfloor$ immediately preceding $m_u$ in the order. If v is in the left subtree of u, since

$$m_v+G<{\displaystyle \frac{l_v+r_v}{2}}+{\displaystyle \frac{r_v-l_v+1}{2}}=r_v+{\displaystyle \frac{1}{2}},$$

we have $m_v+G\le r_v\le m_u$, where the last inequality follows from Line 6 in Algorithm 1. Hence, $m_u-m_v\ge G$.

If, on the other hand, v is not in the left subtree of u, then since the middle child of v is ordered before the middle child of u, v cannot be in the right subtree of u. If v is the parent of u, then by a similar argument,

$$m_u-G+1>{\displaystyle \frac{l_u+r_u}{2}}-1-{\displaystyle \frac{r_u-l_u+1}{2}}+1=l_u-{\displaystyle \frac{1}{2}}$$

and we have $m_u-G+1\ge l_u\ge m_v+1$, where the last inequality follows from Line 7 in Algorithm 1. Hence, $m_u-m_v\ge G$. In any other remaining cases, u and v must share either some lowest common ancestor or an intermediate vertex on the path from v to u. Call this vertex w. Observe that the middle child of w sits in the ordering between the middle children of u and v, a contradiction that makes the last remaining cases impossible.

Applying the above claim, we first observe that the range of possible midpoints of middle children leaves that can have a non-empty intersection with $A^{\prime }$ is $[a-G..b+G-1]$. Therefore, an upper bound on the number of middle children leaves that can have a non-empty intersection with $A^{\prime }$ is given by

$$⌈{\displaystyle \frac{b+G-1-(a-G)+1}{G}}⌉=⌈{\displaystyle \frac{b-a+2G}{G}}⌉=⌈{\displaystyle \frac{b-a}{G}}⌉+2=3.$$

Hence, at most three middle children leaves have intersections with $A^{\prime }$. Combined with at most two non-middle children leaves intersecting $A^{\prime }$, we arrive at the desired result. □

As a result of Lemma 8, each occurrence is reported at most $\mathcal{O}\left(1\right)$ times, and the removal of potential duplicates does not affect the total asymptotic time complexity. Over all “report_all_occurrences” calls, the time is $\tilde{\mathcal{O}}\left(G^{\delta }(s+\mathrm{occ})\right)$.

Summing the time for calls to “contains_occurrence” and “report_all_occurrences” we obtain a total time of $\tilde{\mathcal{O}}(N^{\delta }{(1+s)}^{1-\delta }+G^{\delta }(s+\mathrm{occ}))$. Because each leaf on which “report_all_occurrences” is called contains at least one occurrence, and by Lemma 8, each occurrence is contained in $\mathcal{O}\left(1\right)$ leaves, we have $s=\mathcal{O}\left(\mathrm{occ}\right)$. Furthermore, we have ${(1+\mathrm{occ})}^{1-\delta }=\mathcal{O}(1+{\mathrm{occ}}^{1-\delta })$. This gives us the following lemma.

Lemma 9. 

For every $0\le \delta \le 1$, there is a data structure for the Bounded Gapped Set Intersection with Reporting that occupies $\tilde{\mathcal{O}}\left(N^{2-\delta /3}\right))$ space and answers queries in time

$$\tilde{\mathcal{O}}(N^{\delta }(1+{\mathrm{occ}}^{1-\delta })+G^{\delta }·\mathrm{occ})$$

where $\mathrm{occ}$ is the size of the output.

Combining Lemma 9 with the reduction used in Lemma 2, we obtain the result in Theorem 2, which is an $\tilde{\mathcal{O}}\left(n^{2-\delta /3}\right)$ space index for Bounded Gapped String Indexing with $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }(1+{\mathrm{occ}}^{1-\delta })+G^{\delta }·\mathrm{occ})$ query time.

3.6. Obtaining Theorem 1

We now apply Theorem 2 to obtain Theorem 1. For convenience, we assume that n is a power of two (if not, we can pad T with extra symbols # not in T’s alphabet until its length is a power of two. We can accomplish this while at most doubling its length). We construct the data structure from Lemma 9 for all G in 1, 2, 4, 8, $...$, n. The space required across all data structures is polylogarithmic factors from

$$\sum _{i=0}^{logn}{n}^{2-\delta /3}=n^{2-\delta /3}\sum _{i=0}^{logn}1=\tilde{\mathcal{O}}\left(n^{2-\delta /3}\right).$$

To answer a query $P_1\left[\alpha ..\beta \right]P_2$, we split $\left[\alpha ..\beta \right]$ into logarithmically many ranges $R_1=\left[a_1..b_1\right]=\left[\alpha ..2^{\lceil log\alpha \rceil }\right]$, $R_2=\left[a_2..b_2\right]=[2^{\lceil log\alpha \rceil }+1..2^{\lceil log\alpha \rceil +1}]$, $...$, $R_k=\left[a_k..b_k\right]=\left[2^{\lfloor log\beta \rfloor }..\beta \right]$, where in the case $\beta \le 2^{\lceil log\alpha \rceil }$, no split is performed. The query $P_1\left[\alpha ..2^{\lceil log\alpha \rceil }\right]P_2$ is given to the data structure for $G=2^{\lceil log\alpha \rceil }$. By Theorem 2, it reports occurrences in time $\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }{(1+{\mathrm{occ}}_{R_1})}^{1-\delta }+2^{\lceil log\alpha \rceil }{\mathrm{occ}}_{R_1})$, where ${\mathrm{occ}}_{R_1}$ is the number of occurrences with a gap in the range $R_1=\left[\alpha ..2^{\lceil log\alpha \rceil }\right]$. Continuing in this fashion for each split, the overall complexity is polylogarithmic factors from

$$\begin{array}{cc}\hfill \sum _{i=1}^k\left(\right|P_1|& +|P_2|+n^{\delta }{(1+{\mathrm{occ}}_{R_i})}^{1-\delta }+b_i^{\delta }{\mathrm{occ}}_{R_i})\hfill \\ \hfill & \le \tilde{\mathcal{O}}\left(\right|P_1|+|P_2\left|\right)+n^{\delta }\sum _{i=1}^k(1+{\mathrm{occ}}_{R_i}^{1-\delta })+\sum _{i=1}^k{b}_i^{\delta }{\mathrm{occ}}_{R_i}\hfill \\ \hfill & =\tilde{\mathcal{O}}\left(\right|P_1|+|P_2|+n^{\delta }(1+{\mathrm{occ}}^{1-\delta }))+\sum _{i=1}^k{b}_i^{\delta }{\mathrm{occ}}_{R_i}\hfill \end{array}$$

where ${\mathrm{occ}}_{R_i}$ is the number of occurrences with a gap in range $R_i$. The last equality holds since ${\sum }_{i=1}^k{\mathrm{occ}}_{R_i}^{1-\delta }=\tilde{\mathcal{O}}\left({\mathrm{occ}}^{1-\delta }\right)$.

We observe that for a given range $R_i$, ${\mathrm{occ}}_{R_i}={\sum }_{g\in R_i}{\mathrm{occ}}_g$, where ${\mathrm{occ}}_g$ is the number of occurrences with gap exactly g. Furthermore, $b_i\le 2g$ for all $g\in R_i$. Hence,

$${\left({\displaystyle \frac{b_i}{2}}\right)}^{\delta }{\mathrm{occ}}_{R_i}={\left({\displaystyle \frac{b_i}{2}}\right)}^{\delta }\sum _{g\in R_i}{\mathrm{occ}}_g\le \sum _{g\in R_i}{g}^{\delta }{\mathrm{occ}}_g.$$

We conclude that

$$\sum _{i=1}^k{\left({\displaystyle \frac{b_i}{2}}\right)}^{\delta }{\mathrm{occ}}_{R_i}\le \sum _{i=1}^k\sum _{g\in R_i}{g}^{\delta }{\mathrm{occ}}_g=\sum _{g\in \left[\alpha ..\beta \right]}{g}^{\delta }{\mathrm{occ}}_g,$$

giving us an overall query time complexity of

$$\tilde{\mathcal{O}}\left(|P_1|+|P_2|+n^{\delta }(1+{\mathrm{occ}}^{1-\delta })+\sum _{g\in \left[\alpha ..\beta \right]}{g}^{\delta }{\mathrm{occ}}_g\right)$$

as desired. This completes the proof of Theorem 1.

4. Conclusions

We have presented an index for Gapped String Indexing with a reporting time parameterized by the gap lengths of the occurrences. Potential directions for further development include the following:

• Establishing matching conditional lower bounds based on the Strong Set-Disjointness Conjecture or other conjectures used in fine-grained complexity.
• Extensions to the multi-gap case: That is, preprocess a text to answer queries of the form $P_1\left[{\alpha }_1..{\beta }_1\right]\cdots \left[{\alpha }_{k-1}..{\beta }_{k-1}\right]P_k$. It is not immediate how to adapt Problem 3-based techniques to this setting.
• Extensions to the bounded ratio gapped setting of Ganguly et al. [23].

We acknowledge the likelihood that the gap-sensitive approach proposed here may have a worse query time compared to the prior approach by Bille et al. [17] for large gap values due to polylogarithmic factors. In such instances, it may be advantageous to consider some form of a meta-algorithm that employs our gap-sensitive approach for small gaps and the algorithm of Bille et al. [17] for larger gaps. We leave this as a possible direction for future research.