Linear-Time Text Compression by Longest-First Substitution

Abstract

We consider grammar-based text compression with longest first substitution (LFS), where non-overlapping occurrences of a longest repeating factor of the input text are replaced by a new non-terminal symbol. We present the first linear-time algorithm for LFS. Our algorithm employs a new data structure called sparse lazy suffix trees. We also deal with a more sophisticated version of LFS, called LFS2, that allows better compression. The first linear-time algorithm for LFS2 is also presented.

1. Introduction

Data compression is a task of reducing data description length. Not only does it enable us to save space for data storage, but also it reduces time for data communication. This paper focuses on text compression where the data to be compressed are texts (strings). Recent research developments show that text compression has a wide range of applications, e.g., pattern matching [1, 2, 3], string similarity computation [4, 5], detecting palindromic/repetitive structures [4, 6], inferring hierarchal structure of natural language texts [7, 8], and analyses of biological sequences [9].

Grammar-based compression [10] is a kind of text compression scheme in which a context-free grammar (CFG) that generates only an input text w is output as a compressed form of w. Since the problem of computing the smallest CFG which generates w is NP-hard [11], many attempts have been made to develop practical algorithms that compute a small CFG which generates w. Examples of grammar-based compression algorithms are LZ78 [12], LZW [13], Sequitur [7], and Bisection [14]. Approximation algorithms for optimal grammar-based compression have also been proposed [15, 16, 17]. The first compression algorithm based on a subclass of context-sensitive grammars was introduced in [18].

Grammar-based compression based on greedy substitutions has been extensively studied. Wolff [19] introduced a concept of most-frequent-first substitution (MFFS) such that a digram (a factor of length 2) which occurs most frequently in the text is recursively replaced by a new non-terminal symbol. He also presented an $O\left(n^2\right)$-time algorithm for it, where n is the input text length. A linear-time algorithm for most-frequent-first substitution, called Re-pair, was later proposed by Larsson and Moffat [20]. Apostolico and Lonardi [21] proposed a concept of largest-area-first substitution such that a factor of the largest “area” is recursively replaced by a new non-terminal symbol. Here the area of a factor refers to the product of the length of the factor by the number of its non-overlapping occurrences in the input text. It was reported in [22] that compression by largest-area-first substitution outperforms gzip (based on LZ77 [23]) and bzip2 (based on the Burrows-Wheeler Transform [24]) on DNA sequences. However, to the best of our knowledge, no linear-time algorithm for this compression scheme is known.

This paper focuses on another greedy text compression scheme called longest-first substitution (LFS), in which a longest repeating factor of an input text is recursively replaced by a new non-terminal symbol. For example, for input text $w=\mathtt{abaaabbababb}\$$, the following grammar

$$\begin{array}{ccc}\hfill S& \to & B\mathtt{aa}ABA\$;\hfill \\ \hfill A& \to & \mathtt{abb};\hfill \\ \hfill B& \to & \mathtt{ab},\hfill \end{array}$$

which generates only w is the output of LFS.

In this paper, we propose the first linear-time algorithm for text compression by LFS substitution. A key idea is the use of a new data structure called sparse lazy suffix trees. Moreover, this paper deals with a more sophisticated version of longest-first text compression (named LFS2), where we also consider repeating factors of the right-hand of the existing production rules. For the same input text $w=\mathtt{abaaabbababb}\$$ as above, we obtain the following grammar:

$$\begin{array}{ccc}\hfill S& \to & B\mathtt{aa}ABA\$;\hfill \\ \hfill A& \to & B\mathtt{b};\hfill \\ \hfill B& \to & \mathtt{ab}.\hfill \end{array}$$

This method allows better compression since the total grammar size becomes smaller. In this paper, we present the first linear-time algorithm for text compression based on LFS2. Preliminary versions of our paper appeared in [25] and [26].

Related Work

It is true that several algorithms for LFS or LFS2 were already proposed, however, in fact none of them runs in linear time in the worst case. Bentley and McIlroy [27] proposed an algorithm for LFS, but Nevill-Manning and Witten [8] pointed out that the algorithm does not run in linear time. Nevill-Manning and Witten also claimed that the algorithm can be improved so as to run in linear time, but they only noted a too short sketch for how, which is unlikely to give a shape to the idea of the whole algorithm. Lanctot et al. [28] proposed an algorithm for LFS2 and stated that it runs in linear time, but a careful analysis reveals that it actually takes $O\left(n^2\right)$ time in the worst case for some input string of length n. See Appendix for our detailed analysis.

2. Preliminaries

2.1. Notations

Let Σ be a finite alphabet of symbols. We assume that Σ is fixed and $|\Sigma |$ is constant. An element of ${\Sigma }^{*}$ is called a string. Strings x, y, and z are said to be a prefix, factor, and suffix of string $w=xyz$, respectively.

The length of a string w is denoted by $\left|w\right|$. The empty string is denoted by $\epsilon$, that is, $\left|\epsilon \right|=0$. Also, we assume that all strings end with a unique symbol $\$\in \Sigma$ that does not occur anywhere else in the strings. Let ${\Sigma }^{+}={\Sigma }^{*}\backslash \left\{\epsilon \right\}$. The i-th symbol of a string w is denoted by $w\left[i\right]$ for $1\le i\le \left|w\right|$, and the factor of a string w that begins at position i and ends at position j is denoted by $w[i:j]$ for $1\le i\le j\le \left|w\right|$. For convenience, let $w[i:j]=\epsilon$ for $j<i$, and $w[i:]=w[i:|w\left|\right]$ for $1\le i\le \left|w\right|$. For any strings $x,w$, let ${\mathit{BP}}_w\left(x\right)$ denote the set of the beginning positions of all the occurrences of x in w. That is, ${\mathit{BP}}_w\left(x\right)=\{i\mid x=w[i:i+\left|x\right|-1\left]\right\}$.

We say that strings $x,y$ overlap in w if there exist integers $i,j$ such that $x=w[i:i+|x|-1]$, $y=w[j:j+|y|-1]$, and $i\le j\le i+\left|x\right|-1$ or $j\le i\le j+\left|y\right|-1$.

Let $\#{\mathit{occ}}_w\left(x\right)$ denote the possible maximum number of non-overlapping occurrences of x in w. If $\#{\mathit{occ}}_w\left(x\right)\ge 2$, then x is said to be repeating in w. We abbreviate a longest repeating factor of w to an LRF of w. Remark that there can exist more than one LRF for w.

Let Σ and Π be the set of terminal and non-terminal symbols, respectively, such that $\Sigma \cap \Pi =\varnothing$. A context-free grammar $\mathcal{G}$ is a formal grammar in which every production rule is of the form $A\to u$, where $A\in \Pi$ and $u\in {(\Sigma \cup \Pi )}^{*}$. Let $u=xBy$ and $v=x\beta y$ with $x,y,\beta \in {(\Sigma \cup \Pi )}^{*}$ and $B\in \Pi$. If there exists a production rule $B\to \beta$ in $\mathcal{G}$, then $v=x\beta y$ is said to be directly derived from $u=xBy$ by $\mathcal{G}$, and it is denoted by $u{\Rightarrow }_{\mathcal{G}}v$. If there exists a sequence $w_0,w_1,\dots ,w_n$ such that $w_i\in {(\Sigma \cup \Pi )}^{*}$ and

$$u=w_0{\Rightarrow }_{\mathcal{G}}{w}_1{\Rightarrow }_{\mathcal{G}}\cdots {\Rightarrow }_{\mathcal{G}}{w}_n=v,$$

then we say that v is derived from u. The length of a non-terminal symbol A, denoted $\left|A\right|$, is the length of the string $z\in {\Sigma }^{*}$ that is derived from the production rule $A\to v$. For convenience, we assume that any non-terminal symbol A in $\mathcal{G}$ has $\left|A\right|$ positions. The size of the production rule is the number of terminal and non-terminal symbols v contains.

Figure 1. $\mathit{STree}\left(w\right)$ with $w=\mathtt{ababa}\$$. Solid arrows represent edges, and dotted arrows are suffix links.

2.2. Data Structures

Our text compression algorithm uses a data structure based on suffix trees [29]. The suffix tree of string w, denoted by $\mathit{STree}\left(w\right)$, is defined as follows:

Definition 1 (Suffix Trees) 

$\mathit{STree}\left(w\right)$ is a tree structure such that: (1) every edge is labeled by a non-empty factor of w, (2) every internal node has at least two child nodes, (3) all out-going edge labels of every node begin with mutually distinct symbols, and (4) every suffix of w is spelled out in a path starting from the root node.

Assuming any string w terminates with the unique symbol $ not appearing elsewhere in w, there is a one-to-one correspondence between a suffix of w and a leaf node of $\mathit{STree}\left(w\right)$. It is easy to see that the numbers of the nodes and edges of $\mathit{STree}\left(w\right)$ are linear in $\left|w\right|$. Moreover, by encoding every edge label x of $\mathit{STree}\left(w\right)$ with an ordered pair $(i,j)$ of integers such that $x=w[i:j]$, each edge only needs constant space. Therefore, $\mathit{STree}\left(w\right)$ can be implemented with total of $O\left(\right|w\left|\right)$ space. Also, it is well known that $\mathit{STree}\left(w\right)$ can be constructed in $O\left(\right|w\left|\right)$ time (e.g. see [29]).

$\mathit{STree}\left(w\right)$ for string $w=\mathtt{ababa}\$$ is shown in Figure 1. For any node v of $\mathit{STree}\left(w\right)$, $\mathit{str}\left(v\right)$ denotes the string obtained by concatenating the labels of the edges in the path from the root node to node v. The length of node v, denoted $\mathit{len}\left(v\right)$, is defined to be $\left|\mathit{str}\right(v\left)\right|$. It is an easy application of the Ukkonen algorithm [29] to compute the lengths of all nodes while constructing $\mathit{STree}\left(w\right)$. The leaf node $\ell$ such that $\mathit{str}\left(\ell \right)=w[i:]$ is denoted by ${\mathit{leaf}}_i$, and i is said to be the id of the leaf. Every node v of $\mathit{STree}\left(w\right)$ except for the root node has a suffix link, denoted by $\mathit{suf}\left(v\right)$, such that $\mathit{suf}\left(v\right)=v^{\prime }$ where $\mathit{str}\left(v^{\prime }\right)$ is a suffix of $\mathit{str}\left(v\right)$ and $\mathit{len}\left(v^{\prime }\right)+1=\mathit{len}\left(v\right)$. Linear-time suffix tree construction algorithms (e.g., [29]) make extensive use of the suffix links.

A sparse suffix tree [30] of $w\in {\Sigma }^{*}$ is a kind of suffix tree which represents only a subset of the suffixes of w. The sparse suffix tree of $w\in {(\Sigma \cup \Pi )}^{*}$ represents the subset $\left\{w\right[i:]\mid w[i]\in \Sigma \}$ of suffixes of w which begin with a terminal symbol. Let $\ell$ be the length of the LRFs of w. A reference node of the sparse suffix tree of $w\in {(\Sigma \cup \Pi )}^{*}$ is any node v such that $\mathit{len}\left(v\right)\ge \ell +1$, and there is no node u such that $\mathit{str}\left(u\right)$ is a proper prefix of $\mathit{str}\left(v\right)$ and $\mathit{len}\left(u\right)\ge \ell +1$.

Our algorithm uses the following data structure.

Definition 2 (Sparse Lazy Suffix Trees) 

A sparse lazy suffix tree (SLSTree) of string $w\in {(\Sigma \cup \Pi )}^{*}$, denoted by $\mathit{SLSTree}\left(w\right)$, is a kind of sparse suffix tree such that: (1) All paths from the root node to the reference nodes coincide with those of the sparse suffix tree of w, and (2) Every reference node v stores an ordered triple $⟨min\left(v\right),max\left(v\right),\mathrm{card}\left(v\right)⟩$ such that $min\left(v\right)=min{\mathit{BP}}_w\left(\mathit{str}\left(v\right)\right)$, $max\left(v\right)=max{\mathit{BP}}_w\left(\mathit{str}\left(v\right)\right)$, and $\mathrm{card}\left(v\right)=\left|{\mathit{BP}}_w\left(\mathit{str}\left(v\right)\right)\right|$.

$\mathit{SLSTree}\left(w\right)$ is called “lazy” since its subtrees that are located below the reference nodes may not coincide with those of the corresponding sparse suffix tree of w. Our algorithms of Section 3. run in linear time by “neglecting” updating these subtrees below the reference nodes.

Proposition 1 

For any string $w\in {\Sigma }^{*}$, $\mathit{SLSTree}\left(w\right)$ can be obtained from $\mathit{STree}\left(w\right)$ in $O\left(\right|w\left|\right)$ time.

Proof. 

By a standard postorder traversal on $\mathit{STree}\left(w\right)$, propagating the id of each leaf node. □

Since $\mathit{STree}\left(w\right)$ can be constructed in $O\left(\right|w\left|\right)$ time [29], we can build $\mathit{SLSTree}\left(w\right)$ in total of $O\left(\right|w\left|\right)$ time.

3. Off-Line Compression by Longest-First Substitution

Given a text string $w\in {\Sigma }^{*}$, we here consider a greedy approach to construct a context-free grammar which generates only w. The key is how to select a factor of w to be replaced by a non-terminal symbol from Π. Here, we consider the longest-first-substitution approach where we recursively replace as many LRFs as possible with non-terminal symbols.

Example. 

Let $w=\mathtt{abaaabbababb}\$$. At the beginning, the grammar is of the following simple form $S\to \mathtt{abaaabbababb}\$$, where the right-hand of the production rule consists only of terminal symbols from Σ. Now we focus on the right-hand of S which has two LRFs $\mathtt{aba}$ and $\mathtt{abb}$. Let us here choose $\mathtt{abb}$ to be replaced by non-terminal $A\in \Pi$. We obtain the following grammar: $S\to \mathtt{abaa}A\mathtt{ab}A\$$; $A\to \mathtt{abb}$. The other LRF $\mathtt{aba}$ of length 3 is no longer present in the right-hand of S. Thus we focus on an LRF $\mathtt{ab}$ of length 2. Replacing $\mathtt{ab}$ by non-terminal $B\in \Pi$ results in the following grammar: $S\to B\mathtt{aa}ABA\$$; $A\to \mathtt{abb}$; $B\to \mathtt{ab}$. Since the right-hand of S has no repeating factor longer than 1, we are done.

Let $w_0=w$, and let $w_k$ denote the string obtained by replacing an LRF of $w_{k-1}$ with a non-terminal symbol $A_k$. $\mathit{LRF}\left(w_{k-1}\right)$ denotes the LRF of $w_{k-1}$ that is replaced by $A_k$, namely, we create a new production rule $A_k\to \mathit{LRF}\left(w_{k-1}\right)$. In the above example, $w_0=w=\mathtt{abaaabbababb}\$$, $\mathit{LRF}\left(w_0\right)=\mathtt{abb}$, $A_1=A$, $w_1=\mathtt{abaa}A\mathtt{ab}A\$$, $\mathit{LRF}\left(w_1\right)=\mathtt{ab}$, $A_2=B$, and $w_2=B\mathtt{aa}ABA\$$.

Due to the property of the longest first approach, we have the following observation.

Observation 1 

Let $A_1,\dots ,A_k\in \Pi$ be the non-terminal symbols which replace $\mathit{LRF}\left(w_0\right),$$\dots ,$$\mathit{LRF}\left(w_{k-1}\right)$, respectively. For any $1\le i\le k$, the right-hand of the production rule of $A_i$ contains none of $A_1,\dots ,A_{i-1}$.

In what follows, we will show our algorithm which outputs a context-free grammar which generates a given string. Our algorithm heavily uses the SLSTree structure.

3.1. How to Find $\mathit{LRF}\left(w_k\right)$ Using $\mathit{SLSTree}\left(w_k\right)$

In this section, we show how to find an LRF of $w_k$ from $\mathit{SLSTree}\left(w_k\right)$.

The next lemmas characterize an LRF of $w_k$ that is not represented by a node of $\mathit{SLSTree}\left(w_k\right)$.

Lemma 1 

If an LRF x of $w_k$ is not represented by a node of $\mathit{SLSTree}\left(w_k\right)$, then $max{\mathit{BP}}_{w_k}\left(x\right)=min{\mathit{BP}}_{w_k}\left(x\right)+\left|x\right|$.

Proof. 

Let $i=min{\mathit{BP}}_{w_k}\left(x\right)$ and $j=max{\mathit{BP}}_{w_k}\left(x\right)$. Since x is a repeating factor of $w_k$, $|{\mathit{BP}}_{w_k}\left(x\right)|\ge 2$, which means that $i\ne j$. If $w_k[i+|x\left|\right]\ne w_k[j+|x\left|\right]$, then it contradicts the precondition that x is not represented by a node of $\mathit{SLSTree}\left(w_k\right)$. Hence we have $w_k[i+|x\left|\right]=w_k[j+|x\left|\right]$. Moreover, since x is an LRF of $w_k$, we have $j\ge i+\left|x\right|$. However, if we assume $j>i+\left|x\right|$, this contradicts the precondition that x is an LRF of $w_k$, since $w_k[i+|x\left|\right]=w_k[j+|x\left|\right]$ and we obtain a longer LRF $w_k[i:i+|x\left|\right]=w_k[j:j+|x\left|\right]$. Hence we have $j=i+\left|x\right|$. □

The above lemma implies that an LRF x is not represented by a node of $\mathit{SLSTree}\left(w_k\right)$ only if the first and the last occurrences of x form a square $xx$ in $w_k$. For example, see Figure 1 that illustrates $\mathit{SLSTree}\left(w_0\right)$ for $w=\mathtt{ababa}\$$. One can see that $\mathtt{ab}$ is an LRF of $w_0$ but it is not represented by a node of $\mathit{SLSTree}\left(w_0\right)$.

However, the following lemma guarantees that it is indeed sufficient to consider the strings represented by nodes of $\mathit{SLSTree}\left(w_k\right)$ as candidates for $\mathit{LRF}\left(w_k\right)$.

Lemma 2 

Let x be an LRF of $w_k$ that is not represented by a node of $\mathit{SLSTree}\left(w_k\right)$. Then, there exists another LRF y of $w_k$ that is represented by a node of $\mathit{SLSTree}\left(w_k\right)$ such that $\left|x\right|=\left|y\right|$. Moreover, x is no longer present in $w_{k+1}$ after a substitution for y (see also Figure 2).

Proof. 

Let $i=min{\mathit{BP}}_{w_k}\left(x\right)$ and $j=max{\mathit{BP}}_{w_k}\left(x\right)$. It follows from Lemma 1 that $j=i+\left|x\right|$. Suppose that x is represented on an edge from some node s to some node t of $\mathit{STree}\left(w\right)$. Let $u=\mathit{str}\left(t\right)$. Then we have ${\mathit{BP}}_{w_k}\left(x\right)={\mathit{BP}}_{w_k}\left(u\right)$. Let y be the suffix of u of length $\left|x\right|$. It is clear that $i+\left|u\right|-\left|y\right|,j+\left|u\right|-\left|y\right|\in {\mathit{BP}}_{w_k}\left(y\right)$. Since $j=i+\left|x\right|=i+\left|y\right|$, $\#{\mathit{occ}}_{w_k}\left(y\right)\ge 2$. Thus y is an LRF of $w_k$. Since u is represented by node t and $i=min{\mathit{BP}}_{w_k}\left(u\right)$ and $j=max{\mathit{BP}}_{w_k}\left(u\right)$, we know that $w_k[i+|u\left|\right]\ne w_k[j+|u\left|\right]$. Hence y is represented by a node of $\mathit{SLSTree}\left(w_k\right)$. Since x occurs only within the region $w_k[i:j+|u|-1]$, x does not occur in $w_{k+1}$ after a substitution for y. □

In the running example of Figure 1, $\mathtt{ba}$ is an LRF of $w_0$ that is represented by a node of $\mathit{SLSTree}\left(w_0\right)$. After its two occurrences are replaced by a non-terminal symbol $A_1$, then $\mathtt{ab}$, which is an LRF of $w_0$ not represented by a node of $\mathit{SLSTree}\left(w_0\right)$, is no more present in $w_1=\mathtt{a}{A}_1{A}_1\$$.

After constructing $\mathit{SLSTree}\left(w_0\right)=\mathit{SLSTree}\left(w\right)$, we create a bin-sorted list of the internal nodes of $\mathit{SLSTree}\left(w\right)$ in the decreasing order of their lengths. This can be done in linear time by a standard

Figure 2. Illustration for proof of Lemma 2. Since u is represented by a node of $\mathit{SLSTree}\left(w_k\right)$, we know that $w_k[i+|u\left|\right]\ne w_k[j+|u\left|\right]$.

traversal on $\mathit{SLSTree}\left(w\right)$. We remark that a new internal node v may appear in $\mathit{SLSTree}\left(w_k\right)$ for some $k\ge 1$, which did not exist in $\mathit{SLSTree}\left(w_{k-1}\right)$. However, we have that $\mathit{len}\left(v\right)\le |\mathit{LRF}\left(w_{k-1}\right)|$. Thus, we can maintain the bin-sorted list by inserting node v in constant time.

Given a node s in the bin-sorted list, we can determine whether $\mathit{str}\left(s\right)$ is repeating or not by using $\mathit{SLSTree}\left(w_k\right)$, as follows.

Lemma 3 

Let s be any node of $\mathit{SLSTree}\left(w_k\right)$ with $\mathit{len}\left(s\right)\le |\mathit{LRF}\left(w_k\right)|$ and let $s_1,\dots ,s_{\ell }$ be the children of s. Then ${\mathit{BP}}_{w_k}\left(\mathit{str}\left(s\right)\right)$ is a disjoint union of ${\mathit{BP}}_{w_k}\left(\mathit{str}\left(s_1\right)\right),$$\dots ,$${\mathit{BP}}_{w_k}\left(\mathit{str}\left(s_{\ell }\right)\right)$.

Proof. 

Clear from the definition of $\mathit{SLSTree}\left(w_k\right)$. □

Lemma 4 

For any node s of $\mathit{SLSTree}\left(w_{k-1}\right)$ such that $|\mathit{LRF}\left(w_k\right)|\le \mathit{len}\left(s\right)\le \left|\mathit{LRF}\left(w_{k-1}\right)\right|$, it takes amortized constant time to check whether or not $\mathit{str}\left(s\right)$ is an LRF of $w_k$.

Proof. 

Let $s_1,\dots ,s_{\ell }$ be the children of s. Then, $\mathit{str}\left(s\right)$ is repeating if and only if

$$max\{max{\mathit{BP}}_{w_{k-1}}\left(s_i\right)\mid 1\le i\le \ell \}-min\{min{\mathit{BP}}_{w_{k-1}}\left(s_j\right)\mid 1\le j\le \ell \}\ge \mathit{len}\left(s\right).$$

Remark that the values of $min{\mathit{BP}}_{w_{k-1}}\left(s_i\right)$ and $max{\mathit{BP}}_{w_{k-1}}\left(s_i\right)$ are stored in node $s_i$ and can be referred to in constant time. Since the above inequality is checked at most once for each node s, it takes amortized constant time. □

Suppose we have found an LRF of $w_k$ as mentioned above. In the sequel, we show our greedy strategy to select occurrences of the LRF in $w_k$ to be replaced with a new non-terminal symbol.

The next lemma is essentially the same as Lemma 2 of Kida et al. [1].

Lemma 5 

For any non-repeating factor x of $w_k$, ${\mathit{BP}}_{w_k}\left(x\right)$ forms a single arithmetic progression.

Therefore, for any non-repeating factor x of $w_k$, ${\mathit{BP}}_{w_k}\left(x\right)$ can be expressed by an ordered triple consisting of minimum element $min{\mathit{BP}}_{w_k}\left(x\right)$, maximum element $max{\mathit{BP}}_{w_k}\left(x\right)$, and cardinality $\left|{\mathit{BP}}_{w_k}\left(x\right)\right|$, which takes constant space.

Lemma 6 

Let s be any node of $\mathit{SLSTree}\left(w_k\right)$ such that $\mathit{str}\left(s\right)$ is an LRF of $w_k$, and $s^{\prime }$ be any child of s. Then, ${\mathit{BP}}_{w_k}\left(\mathit{str}\left(s^{\prime }\right)\right)$ contains at most two positions corresponding to non-overlapping occurrences of $\mathit{str}\left(s\right)$ in $w_k$.

Proof. 

Assume for contrary that ${\mathit{BP}}_{w_k}\left(\mathit{str}\left(s^{\prime }\right)\right)$ contains three non-overlapping occurrences of $\mathit{str}\left(s\right)$, and let them be $i_1,i_2,i_3$ in the increasing order. Then we have

$$i_3-(i_1+\mathit{len}\left(s\right)-1)\ge i_3-i_2\ge \mathit{len}\left(s\right)\ge 1,$$

which implies that $w_k[i_1:i_1+\mathit{len}\left(s\right)]$ and $w_k[i_3:i_3+\mathit{len}\left(s\right)]$ are non-overlapping. Moreover, since $\mathit{len}\left(s^{\prime }\right)>\mathit{len}\left(s\right)$, we have $w_k[i_1:i_1+\mathit{len}\left(s\right)]=w_k[i_3:i_3+\mathit{len}\left(s\right)]$. However, this contradicts the precondition that $\mathit{str}\left(s\right)$ is an LRF of $w_k$. □

From Lemma 6, each child $s^{\prime }$ of node s such that $\mathit{str}\left(s\right)$ is an LRF, corresponds to at most two non-overlapping occurrences of $\mathit{str}\left(s\right)$. Due to Lemma 3, we can greedily select occurrences of $\mathit{str}\left(s\right)$ to be replaced by a new non-terminal symbol, by checking all children $s_1,\dots ,s_{\ell }$ of node s. According to Lemma 5, it takes amortized constant time to select such occurrences for each node s.

Note that we have to select occurrences of $\mathit{str}\left(s\right)$ so that no occurrences of $\mathit{str}\left(s\right)$ remain in the text string, and at least two occurrences of $\mathit{str}\left(s\right)$ are selected. We remark that we can greedily choose at least $max\{2,\#\mathit{occ}(\mathit{str}\left(s\right))/2\}$ occurrences.

3.2. How to Update $\mathit{SLSTree}\left(w_k^{i-1}\right)$ to $\mathit{SLSTree}\left(w_k^i\right)$

Let L be the set of the greedily selected occurrences of $\mathit{LRF}\left(w_k\right)$ in $w_k$. For any $0\le i\le \left|L\right|$, let $w_k^i$ denote the string obtained after replacing the first i occurrences of $\mathit{LRF}\left(w_k\right)$ with non-terminal symbol $A_{k+1}$. Namely, $w_k^0=w_k$ and $w_k^{\left|L\right|}=w_{k+1}$.

In this section we show how to update $\mathit{SLSTree}\left(w_k^{i-1}\right)$ to $\mathit{SLSTree}\left(w_k^i\right)$. Let p be the beginning position of the i-th occurrence in L. Assume that we have $\mathit{SLSTree}\left(w_k^{i-1}\right)$, and that we have replaced $w_k^{i-1}[p:p+|\mathit{LRF}\left(w_k\right)|-1]$ with non-terminal symbol $A_{k+1}$ such that $|A_{k+1}|=|\mathit{LRF}\left(w_k\right)|$. We now have $w_k^i$, and we have to update $\mathit{SLSTree}\left(w_k^{i-1}\right)$ to $\mathit{SLSTree}\left(w_k^i\right)$.

A naive way to obtain $\mathit{SLSTree}\left(w_k^i\right)$ is to remove all the suffixes of $w_k^{i-1}$ from $\mathit{SLSTree}\left(w_k^{i-1}\right)$ and insert all the suffixes of $w_k^i$ into it. However, since only the nodes not longer than $\mathit{LRF}\left(w_k\right)$ are important for our longest-first strategy, only the suffixes $w_k^{i-1}[p-t:]$ such that $1\le t\le \left|\mathit{LRF}\right(w_k\left)\right|$ and $w_k^{i-1}\left[r\right]\in \Sigma$ for any $p-t\le r<p$ have to be removed from $\mathit{SLSTree}\left(w_k^{i-1}\right)$, and only the suffixes $w_k^i[p-t:]$ have to be inserted into the tree (see the light-shaded suffixes of Figure 3).

Lemma 7 

For any t, let r be the shortest node of $\mathit{SLSTree}\left(w_k^{i-1}\right)$ such that $w_k^i[p-t:p-1]$ is a prefix of $\mathit{str}\left(r\right)$. Assume $p-t=min{\mathit{BP}}_{w_k^{i-1}}\left(\mathit{str}\left(r\right)\right)$.

• If $\mathit{len}\left(r\right)>|\mathit{LRF}\left(w_k\right)|+t-1$, then there exists an edge in $\mathit{SLSTree}\left(w_k^i\right)$ from the root node to ${\mathit{leaf}}_{p-t}$ labeled with $w_k^i[p-t:]$.
• If $\mathit{len}\left(r\right)\le |\mathit{LRF}\left(w_k\right)|+t-1$, then there exists a node s in $\mathit{SLSTree}\left(w_k^i\right)$ such that $\mathit{str}\left(s\right)=w_k^i[p-t:p-1]$ and s has an edge labeled with $w_k^i[p:]=A_k{w}_k^i[p+|A_k|:]$ and leading to ${\mathit{leaf}}_{p-t}$.

Proof. 

Consider Case 1 (see also Figure 4). Since $t\ge 1$, $\mathit{len}\left(r\right)>|\mathit{LRF}\left(w_k\right)|$. Hence $\mathit{str}\left(r\right)$ is a non-repeating factor of $w_k^i$. By Lemma 5, ${\mathit{BP}}_{w_k^{i-1}}\left(\mathit{str}\left(r\right)\right)$ forms a single arithmetic progression. Also, since $\mathit{len}\left(r\right)>|\mathit{LRF}\left(w_k\right)|$, $max{\mathit{BP}}_{w_k^{i-1}}\left(\mathit{str}\left(r\right)\right)-min{\mathit{BP}}_{w_k^{i-1}}\left(\mathit{str}\left(r\right)\right)\le \left|\mathit{LRF}\left(w_k\right)\right|$. Therefore, if

Figure 3. $\mathit{LRF}\left(w_k\right)$ at position p of $w_k^{i-1}$ is replaced by non-terminal symbol $A_k$ in $w_k^i$. Every $w_k^{i-1}[p-t:]$ is removed from the tree and every $w_k^i[p-t:]$ is inserted into the tree (the light-shaded suffixes in the right figure). In addition, every $w_k^{i-1}[p+h:]$ for $1\le h\le \left|\mathit{LRF}\right(w_k\left)\right|-1$ is removed from the tree (the dark-shaded suffixes in the right figure).

Figure 4. Illustration of Case 1 of Lemma 7.

$p-t=min{\mathit{BP}}_{w_k^{i-1}}\left(\mathit{str}\left(r\right)\right)$, then ${\mathit{BP}}_{w_k^i}\left(w_k^i[p-t:]\right)=\{p-t\}$. Hence there exists an edge from the root node to ${\mathit{leaf}}_{p-t}$ labeled with $w_k^i[p-t:]$ in $\mathit{SLSTree}\left(w_k^i\right)$.

Consider Case 2 (see also Figure 5). Let $u=w_k^{i-1}[p-t:p-1]=w_k^i[p-t:p-1]$. Then $\left|u\right|=t-1$. Since $\mathit{len}\left(r\right)\le |\mathit{LRF}\left(w_k\right)|+t-1$, and since r is not longer than the reference node in the path spelling out $u\mathit{LRF}\left(w_k\right)$ from the root node of $\mathit{SLSTree}\left(w_k^i\right)$, there exists at least one integer m such that $m\in {\mathit{BP}}_{w_k^i}\left(\mathit{str}\left(r\right)\right)$ and $m\notin {\mathit{BP}}_{w_k^i}\left(u{A}_k\right)$. Hence there exists a node s in $\mathit{SLSTree}\left(w_k^i\right)$ such that $\mathit{str}\left(s\right)=u$ and has an out-going edge labeled with $w_k^i[p:]=A_k{w}_k^i[p+|A_k|:]$ and leading to ${\mathit{leaf}}_{p-t}$.□

It is not difficult to see that the edge in each case of Lemma 7 does not exist in $\mathit{SLSTree}\left(w_k^{i-1}\right)$. Hence we create the edge when we update $\mathit{SLSTree}\left(w_k^{i-1}\right)$ to $\mathit{SLSTree}\left(w_k^i\right)$.

The next lemma states how to locate node s of Case 2 of Lemma 7.

Lemma 8 

For each t, we can locate node s such that $\mathit{str}\left(s\right)=w_k^i[p-t:p-1]$ in amortized constant time.

Proof. 

Let $x_{p-t}$ be the longest node in the tree such that $\mathit{str}\left(x_{p-t}\right)$ is a prefix of $w_k^i[p-t:p-1]$.

Figure 5. Illustration of Case 2 of Lemma 7.

Consider the largest possible t and denote it by $t_{max}$. Since $t_{max}\le \left|\mathit{LRF}\left(w_k\right)\right|$, the node $x_{p-t_{max}}$ can be found in $O\left(\right|\mathit{LRF}\left(w_k\right)\left|\right)$ time by going down the path that spells out $w_k^i[p-t_{max}:p-1]$ from the root node (recall that Σ is fixed). Let $z\in {\Sigma }^{*}$ be the string such that $\mathit{str}\left(x_{p-t_{max}}\right)z=w_k^i[p-t_{max}:p-1]$. If $z\ne \epsilon$, then we create a new child node $s_{p-t_{max}}$ of $x_{p-t_{max}}$ such that $\mathit{str}\left(s_{p-t_{max}}\right)=w_k^i[p-t_{max}:p-1]$. Otherwise, we set $s_{p-t_{max}}=x_{p-t_{max}}$.

Now assume that we have located nodes $x_{p-t}$ and $s_{p-t}$. We can then locate $s_{p-t+1}$ as follows. Consider node $x_{p-t+1}$. Remark that $\mathit{str}\left(\mathit{suf}\left(x_{p-t}\right)\right)$ is a prefix of $\mathit{str}\left(x_{p-t+1}\right)$, and thus we can detect $x_{p-t+1}$ in $O\left(\right|\mathit{str}\left(x_{p-t+1}\right)|-|\mathit{str}\left(\mathit{suf}\left(x_{p-t}\right)\right)\left|\right)$ time by using the suffix link. After finding $x_{p-t+1}$, we can locate or create $s_{p-t+1}$ in constant time.

The total time cost for detecting $x_{p-t}$ for all $1\le t\le t_{max}$ is linear in

$$\begin{array}{cc}\hfill {\displaystyle \sum _{t=2}^{t_{max}}}& \left(\right|\mathit{str}\left(x_{p-t+1}\right)|-|\mathit{str}\left(\mathit{suf}\left(x_{p-t}\right)\right)\left|\right)\hfill \\ \hfill =& |\mathit{str}\left(x_{p-1}\right)|-|\mathit{str}\left(\mathit{suf}\left(x_{p-2}\right)\right)|\hfill \\ \hfill +& |\mathit{str}\left(x_{p-2}\right)|-|\mathit{str}\left(\mathit{suf}\left(x_{p-3}\right)\right)|\hfill \\ & \cdots \cdots \hfill \\ \hfill +& |\mathit{str}\left(x_{p-t_{max}+1}\right)|-|\mathit{str}\left(\mathit{suf}\left(x_{p-t_{max}}\right)\right)|\hfill \\ \hfill =& |\mathit{str}\left(x_{p-1}\right)|-|\mathit{str}\left(\mathit{suf}\left(x_{p-t_{max}}\right)\right)|+t_{max}-2\hfill \\ \hfill =& |\mathit{str}\left(x_{p-1}\right)|-|\mathit{str}\left(x_{p-t_{max}}\right)|+t_{max}-1\hfill \\ \le & t_{max}\le \left|\mathit{LRF}\left(w_k\right)\right|.\hfill \end{array}$$

Hence we can locate each $s_{p-t}$ in amortized constant time. □

Let v be the reference node in the path from the root to some ${\mathit{leaf}}_{p-t}$. Assume that ${\mathit{leaf}}_{p-t}$ is removed from the subtree of v, and redirected to node s in the same path, such that $\mathit{str}\left(s\right)=w_k^i[p-t:p-1]$. In order to update $\mathit{SLSTree}\left(w_k^{i-1}\right)$ to $\mathit{SLSTree}\left(w_k^i\right)$, we have to maintain triple $⟨min\left(v\right),max\left(v\right),\mathrm{card}\left(v\right)⟩$ for node v. One may be concerned that if $p-t$ is neither $min\left(v\right)$ or $max\left(v\right)$ and $\mathrm{card}\left(v\right)\ge 4$ in

Figure 6. Illustration of proof for Lemma 9.

$\mathit{SLSTree}\left(w_k^{i-1}\right)$, the occurrences of $\mathit{str}\left(v\right)$ in $\mathit{SLSTree}\left(w_k^i\right)$ do not form a single arithmetic progression any more. However, we have the following lemma. For any factor y of $w_k^i$, let ${\mathit{Dead}}_{w_k^i}\left(y\right)={\mathit{BP}}_{w_k^{i-1}}\left(y\right)\backslash {\mathit{BP}}_{w_k^i}\left(y\right)$, namely, ${\mathit{Dead}}_{w_k^i}\left(y\right)$ denotes the occurrences of y in $w_k^{i-1}$ that overlap with the i-th greedily selected occurrence of $\mathit{LRF}\left(w_k\right)$ in $w_k$.

Lemma 9 

Let v be any reference node of $\mathit{SLSTree}\left(w_k^i\right)$ such that $\#{\mathit{occ}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)=1$. For any integer $m,n$, if $m,n\in {\mathit{BP}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)$, then there is no integer r such that $m<r<n$ and $r\in {\mathit{Dead}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)$. (See Figure 6).

Proof. 

Assume for contrary that there exists integer r such that $r\in {\mathit{Dead}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)$ and $m<r<n$. Since $r\in {\mathit{Dead}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)$, there exist integers $a,b$ such that $a\le r\le b$, and $b-a+1=2\left|\mathit{LRF}\right(w_k\left)\right|$. For any integer j such that $a\le j\le b$ and $j\in {\mathit{BP}}_{w_k^{i-1}}\left(\mathit{str}\left(v\right)\right)$, we have $j\in {\mathit{Dead}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)$. Since $m,n\notin {\mathit{Dead}}_{w_k^i}\left(\mathit{str}\left(v\right)\right)$, $m<a<b<n$. As $\mathit{str}\left(v\right)$ is non-repeating, $n<m+\mathit{len}\left(v\right)-1$. Since $m<a<b<m+\mathit{len}\left(v\right)-1$, $w[a:b]$ is a factor of $\mathit{str}\left(v\right)$. Therefore, there exist two integers $a^{\prime },b^{\prime }$ such that $w[a^{\prime }:b^{\prime }]=w[a:b]$. Since $m<a<b<n<a^{\prime }<b^{\prime }<n+\mathit{len}\left(v\right)-1$, $w[a:b]$ is repeating and $\left|w[a:b]\right|=b-a+1=2|\mathit{LRF}\left(w_k\right)|>|\mathit{LRF}\left(w_k\right)|$. It contradicts that $\mathit{LRF}\left(w_k\right)$ is an LRF of $w_k$. □

Recall that p is the beginning position of the i-th largest greedily selected occurrence of $\mathit{LRF}\left(w_k\right)$ in $w_k$. Also, for any $1\le t\le \left|\mathit{LRF}\right(w_k\left)\right|$ such that $w_k^{i-1}\left[r\right]\in \Sigma$ for every $p-t\le r<p$, we have removed ${\mathit{leaf}}_{p-t}$ from the subtree rooted at the reference node v and have reconnected it to node s such that $\mathit{str}\left(s\right)=w_k^i[p-t:p-1]$. According to the above lemma, if $min\left(v\right)<p-t<max\left(v\right)$, ${\mathit{leaf}}_j$ for every $p-t\le j\le max\left(v\right)$ is removed from the subtree of v. After processing ${\mathit{leaf}}_{p-t}$, then $max\left(v\right)$ is updated to $p-t-d$ where $d=(min(v)+max(v\left)\right)/\mathrm{card}\left(v\right)$ is the step of the progression, and $\mathrm{card}\left(v\right)$ is updated to $(max(v)-(p-t\left)\right)/d+1$.

Notice that ${\mathit{leaf}}_{p+h}$ for every $0\le h\le \left|\mathit{LRF}\right(w_k\left)\right|-1$ has to be removed from the tree, since $w_k^i[p+h]\notin \Sigma$ and therefore this leaf node should not exist in $\mathit{SLSTree}\left(w_k^i\right)$ (see the dark-shaded suffixes of Figure 3). Removing each leaf can be done in constant time. Maintaining the information about the triple for the arithmetic progression of the reference nodes can be done in the same way as mentioned above.

The following lemma states how to locate each reference node.

Lemma 10 

Let p be the i-th greedily selected occurrence of $\mathit{LRF}\left(w_k\right)$ in $w_k$. For any integer $\ell$ such that $w_k^{i-1}\left[\ell \right]\in \Sigma$, let $v\left(\ell \right)$ denote the reference node of $\mathit{SLSTree}\left(w_k^{i-1}\right)$ in the path from the root spelling out suffix $w_k^{i-1}[\ell :]$. For each j such that $p-|\mathit{LRF}\left(w_k\right)|\le j\le p+|\mathit{LRF}\left(w_k\right)|-1$, we can locate the reference node $v\left(j\right)$ in amortized constant time.

Figure 7. The left figure illustrates how to find $v\left(j\right)$ from $v(j-1)$. The right one illustrates a special case where $v\left(j\right)={\mathit{leaf}}_j$. Once $v\left(j\right)={\mathit{leaf}}_j$, it stands that $v\left(k\right)={\mathit{leaf}}_k$ for any $j\le k\le p-1$.

Proof. 

Let $\ell =\left|\mathit{LRF}\right(w_k\left)\right|$. We find $v(p-\ell )$ by spelling out $w_k^{i-1}[p-\ell :]$ from the root in $O\left(\ell \right)$ time, since there can be at most $\ell +1$ nodes in the path from the root to $v(p-\ell )$.

Suppose we have found $v(j-1)$. We find $v\left(j\right)$ as follows. Let $u(j-1)$ be the parent node of $v(j-1)$. We have $\mathit{len}\left(u\right(j-1\left)\right)\le \ell$ and $\mathit{len}\left(v\right(j-1\left)\right)\le \ell +1$. We go to $\mathit{suf}\left(u\right(j-1\left)\right)$. Since $\mathit{len}\left(\mathit{suf}\right(u(j-1)\left)\right)+1=\mathit{len}\left(u\right(j-1\left)\right)$, we have $\mathit{len}\left(\mathit{suf}\right(u(j-1)\left)\right)\le \ell +1$. Thus, we can find $v\left(j\right)$ by going down the path starting from $\mathit{suf}\left(u\right(j-1\left)\right)$ and spelling out $w_k^{i-1}[j-1+\mathit{len}\left(u(j-1)\right):j-1+\mathit{len}\left(v(j-1)\right)]=w_k^{i-1}[j+\mathit{len}\left(\mathit{suf}\left(u(j-1)\right)\right):j-1+\mathit{len}\left(v(j-1)\right)]$. (See also the left illustration of Figure 7).

A special case happens when there exists a node s in the path from the root to ${\mathit{leaf}}_j$, such that $\mathit{len}\left(s\right)=\ell$ and the edge from s in the path starts with some non-terminal symbol $A_h$ with $h<k$. Namely, $w_k^i[j+\ell ]=A_h$. Due to the property of the longest first approach, we have $|A_h|\ge \ell$. Thus $v_j={\mathit{leaf}}_j$. Moreover, for any $j\le k\le p-1$, $v\left(k\right)={\mathit{leaf}}_k$. (See also the right illustration of Figure 7). It is thus clear that each $v\left(k\right)$ can be found in constant time. Since $|A_h|\ge \ell =\mathit{LRF}\left(w_k\right)$, the leaves corresponding to $w_k^{i-1}[p+x-1:]$ with $1\le x\le \ell$ do not exist in $\mathit{SLSTree}\left(w_k^{i-1}\right)$. □

From the above discussions, we conclude that:

Theorem 1 

For any string $w\in {\Sigma }^{*}$, the proposed algorithm for text compression by longest first substitution runs in $O\left(\right|w\left|\right)$ time using $O\left(\right|w\left|\right)$ space.

Pseudo-codes of our algorithms are shown in Algorithms 1, 2, and 3.

3.3. Reducing Grammar Size

In the above sections we considered text compression by longest first substitution, where we construct a context free grammar $\mathcal{G}$ that generates only a given string w. By Observation 1, for any production rule $A_k\to x_k$ of $\mathcal{G}$, $x_k$ contains only terminal symbols from Σ. In this section, we take the factors of $x_k$ into consideration for candidates of LRFs, and also replace LRFs appearing in $x_k$. This way we can reduce

Algorithms 1: Recursively find longest repeating factors.

Algorithm 2: updateSLSTree

Algorithm 3: getGreedilySelectedOccurrences

the total size of the grammar. In so doing, we consider an LRF of string $z_k=w_k{\$}_0{x}_1{\$}_1\cdots x_k{\$}_k$, where $z_0=w_0=w$ and each ${\$}_i$ appears nowhere else in $z_k$.

Example. 

Let $w=w_0=z_0=\mathtt{abaaabbababb}{\$}_{\mathtt{0}}$. We replace an LRF $\mathtt{abb}$ with A, and obtain the following grammar: $S\to \mathtt{abaa}A\mathtt{ab}A{\$}_{\mathtt{0}}$; $A\to \mathtt{abb}$. Then, $w_1=\mathtt{abaa}A\mathtt{ab}A{\$}_{\mathtt{0}}$ and $\mathit{LRF}\left(z_0\right)=\mathtt{abb}$. Now, $z_1=\mathtt{abaa}A\mathtt{ab}A{\$}_{\mathtt{0}}\mathtt{abb}{\$}_{\mathtt{1}}$. We replace an LRF $\mathtt{ab}$ of $z_1$ with a non-terminal B, getting $S\to B\mathtt{aa}ABA{\$}_{\mathtt{0}}$; $A\to B\mathtt{b}$; $B\to \mathtt{ab}$. Then, $w_2=B\mathtt{aa}ABA{\$}_{\mathtt{0}}$ and $\mathit{LRF}\left(z_1\right)=\mathtt{ab}$. Now, $z_2=B\mathtt{aa}ABA{\$}_{\mathtt{0}}B\mathtt{b}{\$}_{\mathtt{1}}\mathtt{ab}{\$}_{\mathtt{2}}$. Since there is no LRF of length more than 1 in $z_2$, we are done.

We call this method of text compression LFS2.

Theorem 2 

Given a string w, the LFS2 strategy compresses w in linear time and space.

Proof. 

We modify the algorithm proposed in the previous sections. If we have a generalized SLSTree for set $\{w_k$, $x_1{\$}_1,\dots ,x_k{\$}_k\}$ of strings, we can find an LRF of $z_k=w_k{x}_1{\$}_1\cdots x_k{\$}_k$. It follows from the property of the longest first substitution strategy that $|x_i|\ge |x_j|$ for any $i<j$. Therefore, any new node inserted into the generalized SLSTree for $\{w_k$, $x_1{\$}_1,\dots ,x_{k-1}{\$}_{k-1}\}$ is shorter than the reference nodes of the tree. Thus, using the Ukkonen on-line algorithm [29], we can obtain the generalized SLSTree of $\{w_k$, $x_1{\$}_1,\dots ,x_k{\$}_k\}$, by inserting the suffixes of each $x_k{\$}_k$ into the generalized SLSTree of $\{w_k$, $x_1{\$}_1,\dots ,x_{k-1}{\$}_{k-1}\}$ in $O\left(\right|x_k{\$}_k\left|\right)$ time. It is easy to see that the total length of $x_1{\$}_1,\dots ,x_k{\$}_k,\dots$ is $O\left(\right|w\left|\right)$. □

4. Conclusions and Future Work

This paper introduced a linear-time algorithm to compress a given text by longest-first substitution (LFS). We employed a new data structure called sparse lazy suffix trees in the core of the algorithm.

We also gave a linear-time algorithm for LFS2 that achieves better compression than LFS.

A related open problem is the following: Does there exist a linear time algorithm for text compression by largest-area-first substitution (LAFS)? The algorithm presented in [21] uses minimal augmented suffix trees (MASTrees) [31] which enable us to efficiently find a factor of the largest area. The size of MASTrees is known to be linear in the input size [32], but the state-of-the-art algorithm of [32] to construct MASTrees takes $O(nlogn)$ time, where n is the input text length. Also, the algorithm of [21] for LAFS reconstructs the MASTree from scratch, every time a factor of the largest area is replaced by a new non-terminal symbol. Would it be possible to update a MASTree or its relaxed version for following substitutions?