Practical Grammar Compression Based on Maximal Repeats ^†

Abstract

This study presents an analysis of RePair, which is a grammar compression algorithm known for its simple scheme, while also being practically effective. First, we show that the main process of RePair, that is, the step by step substitution of the most frequent symbol pairs, works within the corresponding most frequent maximal repeats. Then, we reveal the relation between maximal repeats and grammars constructed by RePair. On the basis of this analysis, we further propose a novel variant of RePair, called MR-RePair, which considers the one-time substitution of the most frequent maximal repeats instead of the consecutive substitution of the most frequent pairs. The results of the experiments comparing the size of constructed grammars and execution time of RePair and MR-RePair on several text corpora demonstrate that MR-RePair constructs more compact grammars than RePair does, especially for highly repetitive texts.

1. Introduction

Grammar compression is one of the lossless data compression methods. For a given text, grammar compression constructs a small size context free grammar that derives only the given text. While the problem of constructing the smallest such grammar for a given text is known to be NP-hard [1], several approximation algorithms have been proposed. One of them is RePair [2], which is an off-line grammar compression algorithm. Despite its simple scheme, RePair is known for its high compression in practice [3,4,5], and hence, it has been comprehensively studied. Some examples of studies on the RePair algorithm include its extension to an online algorithm [6], practical working time/space improvements [7,8], applications to various fields [3,9,10], and theoretical analysis of generated grammar sizes [1,11,12].

In the field of text processing, the repetitiveness of a text is considered an important property. Furthermore, it has been suggested that the extent of the repetitiveness of a given text directly relates to the number of maximal repeats in the text. Belazzougui et al. [13] demonstrated theoretical relations between the number of extensions of maximal repeats and famous other properties of text such as the number of factors in the Lempel–Ziv parsing and the number of runs in the Burrows–Wheeler transform. Also, several text indexing data structures, whose sizes are bounded by the number of extensions of maximal repeats, have been proposed in the literature [14,15,16].

In this study, we analyzed the properties of RePair, focusing on their relationship to maximal repeats. Although RePair has been extensively studied, to the best of our knowledge, no previous study has associated RePair with maximal repeats. Furthermore, we propose MR-RePair, which is a novel grammar compression algorithm based on the property of maximal repeats. Several off-line grammar compression techniques based on the properties of (non-maximal) repeats have been proposed previously [17,18,19]. Recently, Gańczorz and Jeż introduced a heuristic method that improves the practical compression ratio of RePair in terms of the grammar size [20]. However, none of the previously proposed methods use the properties of maximal repeats. In this study, we aim to demonstrate that there is a theoretical guarantee for the size of constructed grammars; under a specific condition, the size of the grammar constructed by MR-RePair is smaller than or equal to that constructed by RePair. Our experiments show that MR-RePair constructs smaller grammars compared to RePair. We emphasize that generating a grammar of small size is of great importance since most, if not all, existing algorithms/data structures that work on grammar-compressed texts have running time dependent on the grammar sizes (see e.g., [21,22,23,24,25,26,27,28] and the references therein) and not directly on the encoded sizes.   

Contributions: The primary contributions of this study are as follows.

• We show interesting relations between maximal repeats and grammars constructed by RePair.
• We propose MR-RePair, which is a novel variant of RePair based on replacing the most frequent maximal repeats.
• We implement MR-RePair and experimentally demonstrate that MR-RePair produces smaller grammars than all tested implementations of RePair. For a highly repetitive text used in the experiments, MR-RePair decreased the size of the constructed grammar to about 55% of that of RePair.

The rest of this paper is organized as follows. In Section 2, we introduce basic notations and review maximal repeats, grammar compression, and RePair. In Section 3, we present an analysis of the properties of RePair and demonstrate its relationship to maximal repeats. The definition and implementation of MR-RePair and its comparison with RePair are provided in Section 4. In Section 5, we report the experimental results of comparing RePair and MR-RePair. Finally, in Section 6, we conclude the paper.

A preliminary version of this work was previously and partially presented at the Data Compression Conference (DCC 2019) [29]. Compared to the preliminary version, the additional and extended parts of this paper are as follows.

• We discuss how the sizes of generated grammars differ depending on the implementation of RePair, and prove that a lower bound of the maximum difference of the sizes is $\frac{1}{6}(\sqrt{6n+1}+13)$ (Definition 1 and Theorem 3).
• We describe Naïve-MR-RePair, which is a naïve version of our MR-RePair. Furthermore, we prove that there is the case where the grammar size of Naïve-MR-RePair becomes larger than that of RePair in logarithmic order of the length of input string (Theorem 4).
• We performed our experiment again following the advice of a DCC reviewer.
• For some of the lemmas we omitted the proofs in the previous version, in this version, we show them all.

2. Preliminaries

In this section, we provide some notations and definitions to be used in the following sections. In addition, we recall grammar compression and review the RePair.

2.1. Basic Notations and Terms

Let $\mathsf{\Sigma }$ be an alphabet, that is, an ordered finite set of symbols. An element $T=t_1\cdots t_n$ of ${\mathsf{\Sigma }}^{*}$ is called a string or a text, where $\left|T\right|=n$ denotes its length. Let $ϵ$ be an empty string of length 0, that is, $\left|ϵ\right|=0$. Let ${\mathsf{\Sigma }}^{+}={\mathsf{\Sigma }}^{*}\backslash \left\{ϵ\right\}$ and $T=t_1\cdots t_n\in {\mathsf{\Sigma }}^n$ be any text of length n. If $T=usw$ with $u,s,w\in {\mathsf{\Sigma }}^{*}$, then s is called a substring of T. Let $T[i..j]=t_i\cdots t_j$ for any $1\le i\le j\le n$ denote a substring of T beginning at i and ending at j in T, and let $T\left[i\right]=t_i$ denote the ith symbol of T. For a finite set S of texts, text T is said to be a superstring of S if T contains all texts of S as substrings. Let $\#\mathrm{occ}\left(s\right)$ denote the frequency of s, i.e., the number of occurrences of s in a text as a substring. If there exists an isomorphism from an alphabet $\mathsf{\Sigma }$ to another alphabet $\widehat{\mathsf{\Sigma }}$, texts ${\mathsf{\Sigma }}^{*}$ and ${\widehat{\mathsf{\Sigma }}}^{*}$ are said to be isomorphic for $\mathsf{\Sigma }$ and $\widehat{\mathsf{\Sigma }}$.

2.2. Maximal Repeats

Let s be a substring of text T. If the frequency of s is greater than 1, s is called a repeat. A left (or right) extension of s is any substring of T in a form of $ws$ (or $sw$), where $w\in {\mathsf{\Sigma }}^{*}$. We define s as a left (or right) maximal if left (or right) extensions of s occur a strictly less number of times in T than s. Accordingly, s is a maximal repeat of T if s is both left and right maximal. In this paper, we only consider strings with a length of more than 1 as maximal repeats. For example, substring abra of $T=$ abracadabra is a maximal repeat, whereas br is not.

2.3. Grammar Compression

A context free grammar (CFG or simply grammar) G is defined as a four-tuple $G=\{V,\mathsf{\Sigma },S,R\}$, where V denotes an ordered finite set of variables, $\mathsf{\Sigma }$ denotes an ordered finite alphabet, R denotes a finite set of binary relations called production rules (or rules) between V and ${(V\cup \mathsf{\Sigma })}^{*}$, and $S\in V$ denotes a special variable called start variable. A production rule refers to the situation, where a variable is substituted and written in a form of $v\to w$, with $v\in V$ and $w\in {(V\cup \mathsf{\Sigma })}^{*}$. Let $X,Y\in {(V\cup \mathsf{\Sigma })}^{*}$. If there are $x_l,x,x_r,y\in {(V\cup \mathsf{\Sigma })}^{*}$ such that $X=x_{l}x{x}_r$, $Y=x_{l}y{x}_r$, and $x\to y\in R$, we write $X\Rightarrow Y$, and denote the reflexive transitive closure of ⇒ as $\stackrel{*}{\Rightarrow }$. Let $\mathit{val}\left(v\right)$ be a string derived from v, i.e., $v\stackrel{*}{\Rightarrow }\mathit{val}\left(v\right)$. We define grammar $\widehat{G}=\{\widehat{V},\widehat{\mathsf{\Sigma }},\widehat{S},\widehat{R}\}$ as a subgrammar of G if $\widehat{V}\subseteq V$, $\widehat{\mathsf{\Sigma }}\subseteq (V\cup \mathsf{\Sigma })$, and $\widehat{R}\subseteq R$.

Given a text T, grammar compression is a method for lossless text data compression that constructs a restricted CFG uniquely deriving the text T. For G to be deterministic, a production rule for each variable $v\in V$ must be unique. In what follows, we assume that every grammar is deterministic and each production rule is $v_i\to {\mathit{expr}}_i$, where ${\mathit{expr}}_{\mathit{i}}$ is an expression either ${\mathit{expr}}_i=a$ ($a\in \mathsf{\Sigma }$) or ${\mathit{expr}}_i=v_{j_1}{v}_{j_2}\cdots v_{j_n}$ ($i>j_k$ for all $1\le k\le j_n$).

For estimating the effectiveness of compression, we use the size of the constructed grammar, which is defined as the total length of the right-hand side of all production rules of the grammar.

2.4. RePair

RePair is a grammar compression algorithm proposed by Larsson and Moffat [2]. For input text T, let $G=\{V,\mathsf{\Sigma },S,R\}$ be the grammar constructed by RePair. Then, the RePair procedure can be described with the following steps:

• Step 1. Replace each symbol $a\in \mathsf{\Sigma }$ with a new variable $v_a$ and add $v_a\to a$ to R.
• Step 2. Find the most frequent pair p in T.
• Step 3. Replace every occurrence (or, as many occurrences as possible, when p is a pair consisting of the same symbol) of p with a new variable v, and then, add $v\to p$ to R.
• Step 4. Re-evaluate the frequencies of pairs for the updated text generated in Step 3. If the maximum frequency is 1, add $S\to \left(\mathrm{current}\mathrm{text}\right)$ to R, and terminate. Otherwise, return to Step 2.

Figure 1 illustrates an example of the grammar generation process of RePair.

Figure 1. An example of the grammar generation process of RePair for the text abracadabra. The generated grammar is $\{\{v_{\mathtt{a}},v_{\mathtt{b}},v_{\mathtt{r}},v_{\mathtt{c}},v_{\mathtt{d}},v_1,v_2,v_3,S\},\left\{\mathtt{a},\mathtt{b},\mathtt{r},\mathtt{c},\mathtt{d}\right\},S,\{v_{\mathtt{a}}\to \mathtt{a},v_{\mathtt{b}}\to \mathtt{b},v_{\mathtt{r}}\to \mathtt{r},v_{\mathtt{c}}\to \mathtt{c},v_{\mathtt{d}}\to \mathtt{d},v_1\to v_{\mathtt{a}}{v}_{\mathtt{b}},v_2\to v_1{v}_{\mathtt{c}},v_3\to v_2{v}_{\mathtt{d}},S\to v_3{v}_{\mathtt{c}}{v}_{\mathtt{a}}{v}_{\mathtt{d}}{v}_3\}\}$ with a size of 16.

Lemma 1

([2]). RePair works in $\mathcal{O}\left(n\right)$ expected time and $5n+4k^2+4k^{\prime }+\lceil \sqrt{n+1}\rceil -1$ words of space, where n is the length of the source text, k denotes the cardinality of the source alphabet, and $k^{\prime }$ denotes the cardinality of the final dictionary.

3. Analysis of RePair

This section presents an analysis of RePair with respect to its relationship to maximal repeats and introduces an important concept, called MR-order.

3.1. RePair and Maximal Repeats

The following theorem describes an essential property of RePair, that is, RePair recursively replaces the most frequent maximal repeats.

Theorem 1.

Let T be a given text, assuming that every most frequent maximal repeat of T does not appear with overlaps with itself. Let f be the frequency of the most frequent pairs of T, and t be a text obtained after all pairs with frequency f in T are replaced by variables. Then, there is a text s such that s is obtained after all maximal repeats with frequency f in T are replaced by variables, and s and t are isomorphic to each other.

We need two lemmas and a corollary to prove Theorem 1. The following lemma shows a fundamental relation between the most frequent maximal repeats and the most frequent pairs in a text.

Lemma 2.

A pair p of variables is most frequent in a text T if and only if p occurs once in exactly one of the most frequent maximal repeats of T.

Proof. 

(⇒) Let r be a most frequent maximal repeat containing p as a substring. It is clear that p can only occur once in r, since otherwise, $\#\mathrm{occ}\left(p\right)>\#\mathrm{occ}\left(r\right)$ would hold, implying the existence of a frequent maximal repeat that is more frequent than r, contradicting the assumption that r is most frequent. Suppose that there exists a different most frequent maximal repeat $r^{\prime }$ containing p as a substring. Similarly, p occurs only once in $r^{\prime }$. Furthermore, since r and $r^{\prime }$ can be obtained by left and right extensions to p, $\#\mathrm{occ}\left(r\right)=\#\mathrm{occ}\left(r^{\prime }\right)=\#\mathrm{occ}\left(p\right)$, and any occurrence of p is contained in an occurrence of both r and $r^{\prime }$. Since $r^{\prime }$ cannot be a substring of r, there exists a string w that is a superstring of r and $r^{\prime }$, such that $\#\mathrm{occ}\left(w\right)=\#\mathrm{occ}\left(r\right)=\#\mathrm{occ}\left(r^{\prime }\right)=\#\mathrm{occ}\left(p\right)$. However, this contradicts that r and $r^{\prime }$ are maximal repeats.

(⇐) Let r be the most frequent maximal repeat such that p occurs once in it. By definition, $\#\mathrm{occ}\left(r\right)=\#\mathrm{occ}\left(p\right)$. If p is not the most frequent symbol pair in T, there exists a pair $p^{\prime }$ in T such that $\#\mathrm{occ}\left(p^{\prime }\right)>\#\mathrm{occ}\left(p\right)=\#\mathrm{occ}\left(r\right)$. However, this implies that there is a maximal repeat $r^{\prime }$ with $\#\mathrm{occ}\left(r^{\prime }\right)=\#\mathrm{occ}\left(p^{\prime }\right)>\#\mathrm{occ}\left(r\right)$, contradicting that r is most frequent. □

The following corollary is derived directly from Lemma 2.

Corollary 1.

For a given text, the frequency of the most frequent pairs and that of the most frequent maximal repeats are the same.

The following lemma shows an important property of the most frequent maximal repeats.

Lemma 3.

The length of the overlap between any two occurrences of most frequent maximal repeats is at most 1.

Proof. 

Let $xw$ and $wy$ be the most frequent maximal repeats that have an overlapping occurrence $xwy$, where $x,y,w\in {\mathsf{\Sigma }}^{+}$. If we assume that $\left|w\right|\ge 2$, since $xw$ and $wy$ are most frequent maximal repeats, it holds that $\#\mathrm{occ}\left(w\right)=\#\mathrm{occ}\left(xw\right)=\#\mathrm{occ}\left(wy\right)$, i.e., every occurrence of w is preceded by x and followed by y. This implies that $\#\mathrm{occ}\left(xwy\right)=\#\mathrm{occ}\left(xw\right)=\#\mathrm{occ}\left(wy\right)$ as well, but contradicts that $xw$ and $wy$ are maximal repeats. □

Theorem 1 can now be proved based on the above lemmas and corollary.

Proof of Theorem 1.

According to Corollary 1, the frequency of the most frequent maximal repeats in T is f. Let p be one of the most frequent pairs in T. According to Lemma 2, there is a unique maximal repeat that is most frequent and contains p once. We denote such maximal repeat as r. Let us assume that there is a substring $zxpyw$ in T, where $z,w\in \mathsf{\Sigma }$, $x,y\in {\mathsf{\Sigma }}^{*}$, and $xpy=r$. We denote $r\left[1\right]$ and $r\left[\right|r\left|\right]$ as $\dot{x}$ and $\dot{y}$, respectively. There are the following two cases to consider:

(i)$\#\mathrm{occ}\left(z\dot{x}\right)<f$and$\#\mathrm{occ}\left(\dot{y}w\right)<f$. If $\left|r\right|=2$, the replacement of p directly corresponds to the replacement of the most frequent maximal repeat, since $p=r$. If $\left|r\right|>2$, after p is replaced with a variable v, r is changed to $xvy$. This occurs f times in the updated text, and according to Lemma 2, the frequency of every pair occurring in $xvy$ is still f. Because the maximum frequency of pairs does not increase, f is still the maximum frequency. Therefore, we replace all pairs contained in $xvy$ in the following steps, whereas $z\dot{x}$ and $\dot{y}w$ are not replaced. This holds for every occurrence of p, implying that replacing the most frequent pairs while the maximum frequency does not change, corresponds to replacing all pairs (old and new) contained in the most frequent maximal repeats of the same frequency until they are replaced by a single variable. Then, s can be generated by replacing r.

(ii)$\#\mathrm{occ}\left(z\dot{x}\right)=f$or$\#\mathrm{occ}\left(\dot{y}w\right)=f$. We consider the case where $\#\mathrm{occ}\left(z\dot{x}\right)=f$. Note that $\#\mathrm{occ}\left(zxpy\right)<f$ according to the assumption that $xpy$ is a maximal repeat. Suppose RePair replaces $z\dot{x}$ by a variable v before p is replaced. Note that according to Lemma 2, there is a maximal repeat occurring f times and including $z\dot{x}$ once (we denote the maximal repeat as $r^{\prime }$), and $r^{\prime }\ne r$ by assumption. According to Lemma 3, the length of the overlap of r and $r^{\prime }$ is at most 1, and then, only $\dot{x}$ is a symbol present in both r and $r^{\prime }$. After that, $xpy=r$ is no longer the most frequent maximal repeat because some of its occurrences are changed to $vr[2..|r\left|\right]$. However, $r[2..|r\left|\right]$ still occurs f times in the updated text. Since $\#\mathrm{occ}\left(zxpy\right)<f$ and $\#\mathrm{occ}\left(xpy\right)=f$, $\#\mathrm{occ}\left(vr\left[2\right]\right)<f$ and $r[2..|r\left|\right]$ is a maximal repeat. Then, $r[2..|r\left|\right]$ will become a variable in subsequent steps, similarly to (i). Here, $r^{\prime }$ would also become a variable. Thus, we can generate s by first replacing $r^{\prime }$ and then replacing $r[2..|r\left|\right]$. Similarly, this holds for $\dot{y}w$ when $\#\mathrm{occ}\left(\dot{y}w\right)=f$ and $\#\mathrm{occ}\left(z\dot{x}\right)=\#\mathrm{occ}\left(\dot{y}w\right)=f$. □

3.2. MR-Order

According to Theorem 1, if there is just one most frequent maximal repeat in the current text, then RePair replaces its all occurrences step by step. However, a problem arises if there are two or more most frequent maximal repeats, with some of them overlapping. In this case, the selection order of pairs (of course, they are most frequent) affects the priority of maximal repeats. We call this order of selecting (summarizing) maximal repeats as the maximal repeat selection order (or simply MR-order). Note that, the selection order of pairs actually depends on the implementation of RePair.

For instance, consider the text abcdeabccde, where abc and cde are the most frequent maximal repeats occurring twice. There are two MR-orders, depending on which of the two maximal repeats abc or cde is given priority. The results of the replacement using RePair with the MR-order are (i) $xyx\mathtt{c}x$ with variables x and y such that $x\stackrel{*}{\Rightarrow }\mathtt{abc}$ and $y\stackrel{*}{\Rightarrow }\mathtt{de}$, and (ii) $zwz\mathtt{c}w$ with variables z and w such that $z\stackrel{*}{\Rightarrow }\mathtt{ab}$ and $w\stackrel{*}{\Rightarrow }\mathtt{cde}$. More precisely, there are 12 possible ways in which RePair can compress the text, with the following generated rule sets:

• $\{v_1\to \mathtt{ab},v_2\to v_1\mathtt{c},v_3\to \mathtt{de},S\to v_2{v}_3{v}_2\mathtt{c}{v}_3\}$,
• $\{v_1\to \mathtt{ab},v_2\to \mathtt{de},v_3\to v_1\mathtt{c},S\to v_3{v}_2{v}_3\mathtt{c}{v}_2\}$,
• $\{v_1\to \mathtt{bc},v_2\to \mathtt{a}{v}_1,v_3\to \mathtt{de},S\to v_2{v}_3{v}_2\mathtt{c}{v}_3\}$,
• $\{v_1\to \mathtt{bc},v_2\to \mathtt{de},v_3\to \mathtt{a}{v}_1,S\to v_3{v}_2{v}_3\mathtt{c}{v}_2\}$,
• $\{v_1\to \mathtt{ed},v_2\to \mathtt{ab},v_3\to v_2\mathtt{c},S\to v_3{v}_1{v}_3\mathtt{c}{v}_1\}$,
• $\{v_1\to \mathtt{ed},v_2\to \mathtt{bc},v_3\to \mathtt{a}{v}_2,S\to v_3{v}_1{v}_3\mathtt{c}{v}_1\}$,
• $\{v_1\to \mathtt{ab},v_2\to \mathtt{cd},v_3\to v_2\mathtt{e},S\to v_1{v}_3{v}_1\mathtt{c}{v}_3\}$,
• $\{v_1\to \mathtt{ab},v_2\to \mathtt{de},v_3\to \mathtt{c}{v}_2,S\to v_1{v}_3{v}_1\mathtt{c}{v}_3\}$,
• $\{v_1\to \mathtt{cd},v_2\to \mathtt{ab},v_3\to v_1\mathtt{e},S\to v_2{v}_3{v}_2\mathtt{c}{v}_3\}$,
• $\{v_1\to \mathtt{cd},v_2\to v_1\mathtt{e},v_3\to \mathtt{ab},S\to v_3{v}_2{v}_3\mathtt{c}{v}_2\}$,
• $\{v_1\to \mathtt{ed},v_2\to \mathtt{ab},v_3\to \mathtt{c}{v}_1,S\to v_2{v}_3{v}_2\mathtt{c}{v}_3\}$,
• $\{v_1\to \mathtt{ed},v_2\to \mathtt{c}{v}_1,v_3\to \mathtt{ab},S\to v_3{v}_2{v}_3\mathtt{c}{v}_2\}$.

Here, 1–6 have the same MR-order because $\mathtt{abc}$ precedes $\mathtt{cde}$ in all of them. At the same time, 7–12 have the same MR-order for the same reason: $\mathtt{cde}$ precedes $\mathtt{abc}$.

If there are several distinct most frequent pairs with overlaps, RePair constructs grammars with different sizes according to the selection order of the pairs. For example, consider the text bcxdabcyabzdabvbcuda. There are three most frequent pairs, namely, $\mathtt{ab}$, $\mathtt{bc}$, and $\mathtt{da}$, occurring three times each. If RePair takes $\mathtt{ab}$ first, the rule set of the generated grammar may become $\{v_1\to \mathtt{ab},v_2\to \mathtt{bc},v_3\to \mathtt{d}{v}_1,S\to v_2\mathtt{x}{v}_3\mathtt{c}\mathtt{y}{v}_1\mathtt{z}{v}_3\mathtt{v}{v}_2\mathtt{u}\mathtt{d}\mathtt{a}\}$ and its size is 19. If RePair takes $\mathtt{da}$ first, the rule set of the generated grammar may become $\{v_1\to \mathtt{da},v_2\to \mathtt{bc},S\to v_2\mathtt{x}{v}_1{v}_2\mathtt{y}\mathtt{a}\mathtt{b}\mathtt{z}{v}_1\mathtt{b}\mathtt{v}{v}_2\mathtt{u}{v}_1\}$ and its size is 18.

Remark 1.

If there are several distinct pairs with the same maximum frequency, the size of the grammar generated by RePair depends on their replacement order.

However, the following theorem states that the MR-order rather than the replacement order of pairs determines the size of the grammar generated by RePair.

Theorem 2.

The sizes of grammars generated by RePair are the same if they are generated in the same MR-order.

Proof. 

Let T be a variable sequence appearing in the grammar generation process of RePair and f be the maximum frequency of pairs in T. Suppose that $T^{\prime }$ is a variable sequence generated after RePair replaces every pair occurring f times. According to Theorem 1, all generated $T^{\prime }$ are isomorphic to one another, then the length of all of them is the same, regardless of the replacement order of pairs. Let $r_1$ be the most frequent maximal repeats of T with $r_1$ preceding all other maximal repeats in this MR-order. As a result, $r_1$ is converted into a variable, and according to Lemma 2, all pairs included in $r_1$ are distinct. Then, the size of the subgrammar which exactly derives $r_1$ is $2\left(\right|r_1|-1)+1=2|r_1|-1$. This holds for the next prioritized maximal repeat (we denote it as $r_2$) with the following slight difference: the pattern actually replaced would be a substring of $r_2$ excluding its beginning or end if there are occurrences of overlap with $r_1$. However, these strings are common in the same MR-order. Then, the sizes of generated subgrammars are the same, regardless of the order of selecting pairs. Similarly, this holds for all most frequent maximal repeats and every maximum frequency of pairs through the entire process of RePair. □

3.3. Greatest Size Difference of RePair

We consider the problem of determining the greatest size difference between possible outcomes of RePair.

Definition 1

(Greatest size difference). Let g and $g^{\prime }$ be the sizes of any two possible grammars that can be generated by RePair for a given text. Then, the greatest size difference of RePair (GSDRP) is $\max \left(\right|g-g^{\prime }\left|\right)$.

A lower bound of the GSDRP can be established according to the following theorem.

Theorem 3.

Given a text with a length of n, a lower bound of GSDRP is $\frac{1}{6}(\sqrt{6n+1}+13)$.

Proof. 

Let B, L, and R be strings such that

$$\begin{array}{cc}\hfill B& =l_{1}xy{r}_1{l}_{2}xy{r}_2\cdots {\mathit{l}}_{\mathit{f}-1}xy{\mathit{r}}_{\mathit{f}-1}{\mathit{l}}_{\mathit{f}}xy{\mathit{r}}_{\mathit{f}},\hfill \\ \hfill L& =♢l_{1}x♢l_{2}x\cdots ♢{\mathit{l}}_{\mathit{f}}x,\hfill \\ \hfill R& =♢y{r}_1♢y{r}_2\cdots ♢y{\mathit{r}}_{\mathit{f}},\hfill \end{array}$$

where $x,y,l_1,\dots ,l_f,r_1,\dots ,r_f$ denote distinct symbols, and each occurrence of ♢ denotes a distinct symbol. Consider text $T=B{L}^{f-1}{R}^{f-1}$. Here, $xy$, $l_{1}x$, ⋯, $l_{f}x$, ${\mathit{yr}}_{\mathit{1}}$, ⋯, ${\mathit{yr}}_{\mathit{f}}$ are the most frequent maximal repeats with a frequency f in T. Let G and $G^{\prime }$ be grammars generated by RePair for T in different MR-order, such that (i) $xy$ precedes all other maximal repeats and (ii) $xy$ follows all other maximal repeats, respectively. We denote the sizes of G and $G^{\prime }$ as g and $g^{\prime }$, respectively.

First, we consider G and how RePair generates it. The first rule generated by the replacement is $v_1\to xy$ considering the MR-order. After the replacement, L and R remain unchanged, whereas B becomes the following text:

$$B_1=l_1{v}_1{r}_1{l}_2{v}_1{r}_2\cdots {\mathit{l}}_{\mathit{f}-1}{v}_1{\mathit{r}}_{\mathit{f}-1}{\mathit{l}}_{\mathit{f}}{v}_1{\mathit{r}}_{\mathit{f}}.$$

Each pair in $B_1$ occurs only once in the entire text $B_1{L}^{f-1}{R}^{f-1}$. This means that $B_1$ can never be shortened from the current length of $3f$. In the remaining steps, ${\mathit{l}}_{\mathit{i}}x$ and $y{\mathit{r}}_{\mathit{i}}$ (for $i=1,\cdots ,f$) are replaced. L and R are changed to texts with a length of $2f$ each. Hence, the following holds:

$$g=3f+2·2f+2(1+2f)=11f+2.$$

(1)

Next, we consider $G^{\prime }$ and how RePair generates it. According to their MR-order, $l_{1}x$, ⋯, $l_{f}x$, ${\mathit{yr}}_{\mathit{1}}$, ⋯, ${\mathit{yr}}_{\mathit{f}}$ are replaced before $xy$ is selected. They do not overlap with each other, and after they are replaced, $xy$ does not occur in the generated text. Therefore, there are $2f$ rules in $G^{\prime }$ deriving ${\mathit{l}}_{\mathit{i}}x$ and $y{\mathit{r}}_{\mathit{i}}$ (for $i=1,\cdots ,f$), whereas the rule deriving $xy$ is absent. L and R are changed to texts with a length of $2f$ each, and B is changed to a text with a length $2f$. Hence, the following holds:

$$g^{\prime }=2f+2·2f+2·2f=10f.$$

(2)

Let us denote the length of the original text $T=B{L}^{f-1}{R}^{f-1}$ by n. Then, the following holds:

$$\begin{array}{c}\hfill n=4f+2\left(3f\right)(f-1)=6f^2-2f.\end{array}$$

Therefore,

$$\begin{array}{c}\hfill f=\frac{1}{6}(\sqrt{6n+1}+1)\end{array}$$

(3)

holds. According to Equations (1)–(3),

$$\begin{array}{cc}\hfill g-g^{\prime }& =11f+2-10f=f+2\hfill \\ \hfill & =\frac{1}{6}(\sqrt{6n+1}+13)\hfill \end{array}$$

holds and the theorem follows. □

4. MR-RePair

The main strategy of the proposed method is to recursively replace the most frequent maximal repeats instead of the most frequent pairs.

In this section, we first explain the naïve version of our method called Naïve-MR-RePair. Although it can have a bad performance in certain cases, it is simple and helpful in understanding our main result. Then, we describe the proposed MR-RePair.

4.1. Naïve-MR-RePair

Definition 2

(Naïve-MR-RePair). For an input text T, let $G=\{V,\mathsf{\Sigma },S,R\}$ be the grammar generated by Naïve-MR-RePair. Naïve-MR-RePair constructs G through the following steps:

• Step 1.Replace each symbol $a\in \mathsf{\Sigma }$ with a new variable $v_a$ and add $v_a\to a$ to R.
• Step 2.Find the most frequent maximal repeat r in T.
• Step 3.Replace every occurrence (or as many occurrences as possible, when there are overlaps) of r in T with a new variable v and then add $v\to r$ to R.
• Step 4.Re-evaluate the frequencies of maximal repeats for the updated text generated inStep 3. If the maximum frequency is 1, add $S\to \left(\mathrm{current}\mathrm{text}\right)$ to R and terminate. Otherwise, return toStep 2.

We can easily extend the concept of the MR-order to Naïve-MR-RePair.

Figure 2 illustrates an example of the grammar generation process of Naïve-MR-RePair. Figure 1 and Figure 2 explain why the strategy of using maximal repeats is more effective compared to that using pairs. When compressing the text $v_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{r}}{v}_{\mathtt{a}}{v}_{\mathtt{c}}{v}_{\mathtt{a}}{v}_{\mathtt{d}}{v}_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{r}}{v}_{\mathtt{a}}$, both RePair and Naïve-MR-RePair generate subgrammars deriving the most frequent maximal repeat $v_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{r}}{v}_{\mathtt{a}}$. The rule set of the subgrammar generated by RePair is $\{v_1\to v_{\mathtt{a}}{v}_{\mathtt{b}},v_2\to v_1{v}_{\mathtt{r}},v_3\to v_2{v}_{\mathtt{a}}\}$ with a size of 6. At the same time, the rule set of the subgrammar generated by Naïve-MR-RePair is $\{v_1\to v_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{r}}{v}_{\mathtt{a}}\}$ with a size of 4.

Figure 2. An example of the grammar generation process of Naïve-MR-RePair for the text abracadabra. The generated grammar is $\{\{v_{\mathtt{a}},v_{\mathtt{b}},v_{\mathtt{r}},v_{\mathtt{c}},v_{\mathtt{d}},v_1,S\},\left\{\mathtt{a},\mathtt{b},\mathtt{r},\mathtt{c},\mathtt{d}\right\},S,\{v_{\mathtt{a}}\to \mathtt{a},v_{\mathtt{b}}\to \mathtt{b},v_{\mathtt{r}}\to \mathtt{r},v_{\mathtt{c}}\to \mathtt{c},v_{\mathtt{d}}\to \mathtt{d},v_1\to v_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{r}}{v}_{\mathtt{a}},S\to v_1{v}_{\mathtt{c}}{v}_{\mathtt{a}}{v}_{\mathtt{d}}{v}_1\}\}$ with a size of 14.

However, the following theorem indicates that the size of the grammar generated by Naïve-MR-RePair is larger than that generated by RePair in certain cases, even when they work in the same MR-order. Roughly speaking, this is caused by the overlaps of maximal repeats. When there is an occurrence of the most frequent maximal repeat that overlaps with its occurrence, little difference would arise in grammar constructing processes of RePair and Naïve-MR-RePair from the view point of maximal repeats, that is, the targeted maximal repeats would vary in RePair and in Naïve-MR-RePair (RePair replaces the targeted maximal repeat step by step and Naïve-MR-RePair replaces it at once). Indeed, if maximal repeats are carefully embedded in a text for increasing the difference, the case presented in the following theorem occurs.

Theorem 4.

Given a text T with a length of n, let $g_{\mathit{rp}}$ and $g_{\mathit{nmr}}$ be the sizes of the grammars generated by RePair and Naïve-MR-RePair for T, respectively, assuming that RePair and Naïve-MR-RePair work in the same MR-order. Then, there is a case when $g_{\mathit{nmr}}=g_{\mathit{rp}}+\Omega (logn)$ holds. (We show a concrete example of this theorem in Appendix A.)

Proof. 

Let $G_{\mathit{rp}}=\{V_{\mathit{rp}},{\mathsf{\Sigma }}_{\mathit{rp}},S_{\mathit{rp}},R_{\mathit{rp}}\}$ and $G_{\mathit{nmr}}=\{V_{\mathit{nmr}},{\mathsf{\Sigma }}_{\mathit{nmr}},S_{\mathit{nmr}},$ $R_{\mathit{nmr}}\}$ be the grammars generated by RePair and Naïve-MR-RePair, respectively. Let $T^{\prime }$ be the text generated just after Step 1 of RePair or Naïve-MR-RePair (the Step 1 is common in both algorithms), that is, $T^{\prime }=v_1\cdots v_n$ such that $v_i\in V_{\mathit{rp}}\cap V_{\mathit{nmr}}$ and $v_i\to T\left[i\right]\in R_{\mathit{rp}}\cap R_{\mathit{nmr}}$ (for $i=1,\cdots ,n$), and ${\widehat{G}}_{\mathit{rp}}=\{{\widehat{V}}_{\mathit{rp}},{\widehat{\mathsf{\Sigma }}}_{\mathit{rp}},{\widehat{S}}_{\mathit{rp}},{\widehat{R}}_{\mathit{rp}}\}$ (or ${\widehat{G}}_{\mathit{nmr}}=\{{\widehat{V}}_{\mathit{nmr}},{\widehat{\mathsf{\Sigma }}}_{\mathit{nmr}},{\widehat{S}}_{\mathit{nmr}},{\widehat{R}}_{\mathit{nmr}}\}$) be a subgrammar of $G_{\mathit{rp}}$ (or $G_{\mathit{nmr}}$) deriving $T^{\prime }$. Let $T^{\prime }={\left(uw\right)}^{2^{m+1}-1}u$, where $u\in V_{\mathit{rp}}\cap V_{\mathit{nmr}}$, $w\in {(V_{\mathit{rp}}\cap V_{\mathit{nmr}})}^{+}$ such that $uwu$ is the most frequent maximal repeat of $T^{\prime }$, and $m\in {\mathbb{N}}^{+}$. Note that $2^{m+1}-1={\sum }_{i=0}^m{2}^i$. Here ${\widehat{R}}_{\mathit{rp}}$ and ${\widehat{R}}_{\mathit{nmr}}$ are defined as follows:

${\widehat{R}}_{\mathit{rp}}$:

Assume that $x_i\in {\widehat{V}}_{\mathit{rp}}$ for $1\le i\le m$ and $y_j\in {\widehat{V}}_{\mathit{rp}}\cup {\widehat{\mathsf{\Sigma }}}_{\mathit{rp}}$ for $1\le j\le \left|w\right|$, then ${\widehat{R}}_{\mathit{rp}}$ consists of

• $\left|w\right|$ rules $y_j\to y_l{y}_r$ with $\mathit{val}\left(y_{\left|w\right|}\right)=uw$,
• one rule $x_1\to y_{\left|w\right|}{y}_{\left|w\right|}$ and ${log}_2\lfloor 2^{m+1}-1\rfloor -1=m-1$ rules $x_i\to x_{i-1}{x}_{i-1}$ for $2\le i\le m$, and
• one rule ${\widehat{S}}_{\mathit{rp}}\to x_m{x}_{m-1}\cdots x_1{y}_{\left|w\right|}$.

${\widehat{R}}_{\mathit{nmr}}$:

Assume that $d=|{\widehat{V}}_{\mathit{nmr}}|=|{\widehat{R}}_{\mathit{nmr}}|$ and $z_i\in {\widehat{V}}_{\mathit{nmr}}$ for $1\le i\le d$, then ${\widehat{R}}_{\mathit{nmr}}$ consists of

• one rule $z_1\to uwu$, and
• $d-1$ rules $z_i\to z_{i-1}w{z}_{i-1}$ for $2\le i\le d$ and $z_d={\widehat{S}}_{\mathit{nmr}}$.

Let ${\widehat{g}}_{\mathit{rp}}$ and ${\widehat{g}}_{\mathit{nmr}}$ be the sizes of ${\widehat{G}}_{\mathit{rp}}$ and ${\widehat{G}}_{\mathit{nmr}}$, respectively. Then, the following holds:

$$\begin{array}{cc}\hfill {\widehat{g}}_{\mathit{rp}}& =2\left|w\right|+2m+(m+2)=3m+2\left|w\right|+2,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\widehat{g}}_{\mathit{nmr}}& =\left|w\right|+2+\left(\right|w|+2)(d-1)=\left(\right|w|+2)d.\hfill \end{array}$$

(5)

Here, with regard to the length of $T^{\prime }$, we have

$$\begin{array}{cc}\hfill n& =(2(2^m-1)+1)\left(\right|w|+1)+1,\mathrm{and}\hfill \\ \hfill n& =(2^d-1)\left|w\right|+2^d.\hfill \end{array}$$

From these, $d=m+1$ holds. Hence, according to Equations (4) and (5), the following holds:

$${\widehat{g}}_{\mathit{nmr}}-{\widehat{g}}_{\mathit{rp}}=(m-1)\left(\right|w|-1)-1.$$

Therefore, ${\widehat{g}}_{\mathit{nmr}}>{\widehat{g}}_{\mathit{rp}}$ holds for some $(m,|w\left|\right)$, and the proposition holds. □

4.2. MR-RePair

The grammar size of Naïve-MR-RePair becomes larger than that of RePair as shown in Theorem 4 because Naïve-MR-RePair cannot replace all occurrences of most frequent maximal repeats if it overlaps with another occurrence of itself. In the remainder of this section, we describe MR-RePair, which is an improved version of Naïve-MR-RePair.

Definition 3

(MR-RePair). For an input text T, let $G=\{V,\mathsf{\Sigma },S,R\}$ be the grammar generated by MR-RePair. MR-RePair constructs T through the following steps:

• Step 1.Replace each symbol $a\in \mathsf{\Sigma }$ with a new variable $v_a$ and add $v_a\to a$ to R.
• Step 2.Find the most frequent maximal repeat r in T.
• Step 3.Check if $\left|r\right|>2$ and $r\left[1\right]=r\left[\right|r\left|\right]$, and if so, use $r[1..|r|-1]$ instead of r inStep 4.
• Step 4.Replace every occurrence of r with a new variable v and then add $v\to r$ to R.
• Step 5.Re-evaluate the frequencies of maximal repeats for the updated text generated inStep 4. If the maximum frequency is 1, add $S\to \left(\mathrm{current}\mathrm{text}\right)$ to R and terminate. Otherwise, return toStep 2.

We can easily extend the concept of the MR-order to MR-RePair. We do not care if it uses $r[2..|r\left|\right]$ in Step 3, instead of $r[1..|r|-1]$. MR-RePair can replace all occurrences of r even if it overlaps with itself in some occurrences since, according to Lemma 3, the length of the overlaps of the most frequent maximal repeats is at most 1. If $r\left[1\right]=r\left[\right|r\left|\right]$ but r does not overlap with itself, then $vr\left[\right|r\left|\right]$ becomes the most frequent maximal repeat after $r[1..|r|-1]$ is replaced by v and $vr\left[\right|r\left|\right]$ is replaced immediately. Similar to RePair, MR-RePair still cannot replace all of them if $\left|r\right|=2$.

Figure 3 illustrates an example of the grammar generation process of MR-RePair. Although the size of the grammar generated by MR-RePair as shown in Figure 3 is larger than that generated by Naïve-MR-RePair as shown in Figure 2, it is still smaller than that generated by RePair as shown in Figure 1.

Figure 3. An example of the grammar generation process of MR-RePair for the text abracadabra. The generated grammar is $\{\{v_{\mathtt{a}},v_{\mathtt{b}},v_{\mathtt{r}},v_{\mathtt{c}},v_{\mathtt{d}},v_1,S\},\left\{\mathtt{a},\mathtt{b},\mathtt{r},\mathtt{c},\mathtt{d}\right\},S,\{v_{\mathtt{a}}\to \mathtt{a},v_{\mathtt{b}}\to \mathtt{b},v_{\mathtt{r}}\to \mathtt{r},v_{\mathtt{c}}\to \mathtt{c},v_{\mathtt{d}}\to \mathtt{d},v_1\to v_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{r}},v_2\to v_1{v}_{\mathtt{a}},S\to v_2{v}_{\mathtt{c}}{v}_{\mathtt{a}}{v}_{\mathtt{d}}{v}_2\}\}$ with a size of 15.

Theorem 5.

Assume that RePair and MR-RePair work based on the same MR-order for a given text. Let $g_{\mathit{rp}}$ and $g_{\mathit{mr}}$ be the sizes of the grammars generated by RePair and MR-RePair, respectively. Then, $\frac{1}{2}{g}_{\mathit{rp}}<g_{\mathit{mr}}\le g_{\mathit{rp}}$ holds.

Proof. 

Assume that $G_{\mathit{rp}}=\{V_{\mathit{rp}},{\mathsf{\Sigma }}_{\mathit{rp}},S_{\mathit{rp}},R_{\mathit{rp}}\}$ and $G_{\mathit{mr}}=\{V_{\mathit{mr}},{\mathsf{\Sigma }}_{\mathit{mr}},S_{\mathit{mr}},R_{\mathit{mr}}\}$ are grammars generated by RePair and MR-RePair, respectively, for a given text T with a length of n. Let $T^{\prime }$ be the text generated just after Step 1 of RePair or Naïve-MR-RePair (the Step 1 is common in both algorithms), that is, $T^{\prime }=v_1\cdots v_n$ such that $v_i\in V_{\mathit{rp}}\cap V_{\mathit{mr}}$ and $v_i\to T\left[i\right]\in R_{\mathit{rp}}\cap R_{\mathit{mr}}$ (for $i=1,\cdots ,n$).

Let $f_1$ be the maximum frequency of the maximal repeats in $T^{\prime }$. According to Corollary 1, the maximum frequency of the pairs in $T^{\prime }$ is also $f_1$. Let ${\widehat{G}}_{\mathit{rp}}^{\left(f_1\right)}$ (or ${\widehat{G}}_{\mathit{mr}}^{\left(f_1\right)}$) be a subgrammar of $G_{\mathit{rp}}$ (or $G_{\mathit{mr}}$) generated while RePair (or MR-RePair) replaces pairs (or maximal repeats) with the frequency $f_1$, ${\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}$ (or ${\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}$) be the size of this subgrammar, and $T_{\mathit{rp}}^{\left(f_1\right)}$ (or $T_{\mathit{mr}}^{\left(f_1\right)}$) be the updated text after all pairs (or maximal repeats) with the frequency $f_1$ are replaced. Let $r_1^{\left(f_1\right)},\cdots ,r_{m_1}^{\left(f_1\right)}$ be maximal repeats with frequency $f_1$ in $T^{\prime }$ assuming that they are prioritized in this order by the MR-order. Let $l_i^{\left(f_1\right)}$ (for $i=1,\cdots ,m_1$) be the length of the longest substring of $r_i^{\left(f_1\right)}$ such that there are variables that derive the substring in both ${\widehat{G}}_{\mathit{rp}}^{\left(f_1\right)}$ and ${\widehat{G}}_{\mathit{mr}}^{\left(f_1\right)}$. Note that this substring is common to RePair and MR-RePair, and each $l_i^{\left(f_1\right)}$ is at least 2. Since RePair replaces such substring step by step and MR-RePair replaces it at once, the following holds:

$$\begin{array}{cc}\hfill {\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}& =\sum _{i=1}^{m_1}2(l_i^{\left(f_1\right)}-1),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}& =\sum _{i=1}^{m_1}{l}_i^{\left(f_1\right)}.\hfill \end{array}$$

(7)

From these,

$$\begin{array}{cc}\hfill {\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}-{\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}& =2\sum _{i=1}^{m_1}{l}_i^{\left(f_1\right)}-2m_1-\sum _{i=1}^{m_1}{l}_i^{\left(f_1\right)}\hfill \\ \hfill & =\sum _{i=1}^{m_1}{l}_i^{\left(f_1\right)}-2m_1\hfill \\ \hfill & \ge 2m_1-2m_1=0(\because l_i\ge 2\mathrm{holds}\mathrm{for}\mathrm{each}i).\hfill \end{array}$$

Hence,

$$\begin{array}{c}\hfill {\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}\le {\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}\end{array}$$

(8)

holds. According to Equation (6),

$$\begin{array}{cc}\hfill {\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}& =2\sum _{i=1}^{m_1}{l}_i^{\left(f_1\right)}-2m_1\hfill \\ \hfill & =2{\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}-2m_1\left(\mathrm{by}\mathrm{Equation}\left(7\right)\right).\hfill \end{array}$$

Hence,

$$\begin{array}{c}\hfill \frac{1}{2}{\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}<{\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}\end{array}$$

(9)

holds. Therefore, according to Equations (8) and (9),

$$\begin{array}{c}\hfill \frac{1}{2}{\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}<{\widehat{g}}_{\mathit{mr}}^{\left(f_1\right)}\le {\widehat{g}}_{\mathit{rp}}^{\left(f_1\right)}\end{array}$$

(10)

holds. The updated texts $T_{\mathit{rp}}^{\left(f_1\right)}$ and $T_{\mathit{mr}}^{\left(f_1\right)}$ are isomorphic for $V_{\mathit{rp}}$ and $V_{\mathit{mr}}$. Let $f_2$ be the maximum frequency of the maximal repeats in $T_{\mathit{rp}}^{\left(f_1\right)}$ (and $T_{\mathit{mr}}^{\left(f_1\right)}$). Then, a similar analysis holds for ${\widehat{G}}_{\mathit{rp}}^{\left(f_2\right)}$ and ${\widehat{G}}_{\mathit{mr}}^{\left(f_2\right)}$. Hence, $\frac{1}{2}{\widehat{g}}_{\mathit{rp}}^{\left(f_2\right)}<{\widehat{g}}_{\mathit{mr}}^{\left(f_2\right)}\le {\widehat{g}}_{\mathit{rp}}^{\left(f_2\right)}$ holds similarly to Equation (10), and the updated texts $T_{\mathit{rp}}^{\left(f_2\right)}$ and $T_{\mathit{mr}}^{\left(f_2\right)}$ are isomorphic. Inductively, for every maximum frequency of maximal repeats $f_i$, $\frac{1}{2}{\widehat{g}}_{\mathit{rp}}^{\left(f_i\right)}<{\widehat{g}}_{\mathit{mr}}^{\left(f_i\right)}\le {\widehat{g}}_{\mathit{rp}}^{\left(f_i\right)}$ holds and the updated texts $T_{\mathit{rp}}^{\left(f_i\right)}$ and $T_{\mathit{mr}}^{\left(f_i\right)}$ are isomorphic. Let k be a natural number such that $f_k>1$ and $f_{k+1}=1$, that is, k is the number of times that the maximum frequency decreases through the entire process of RePair and MR-RePair. Then,

$$\begin{array}{cc}\hfill g_{\mathit{rp}}& =\sum _{j=1}^k{\widehat{g}}_{\mathit{rp}}^{\left(f_j\right)}+\left|\mathsf{\Sigma }\right|+|T_{\mathit{rp}}^{\left(f_k\right)}|\hfill \\ \hfill & =\sum _{j=1}^k\sum _{i=1}^{m_j}2(l_i^{\left(f_j\right)}-1)+\left|\mathsf{\Sigma }\right|+|T_{\mathit{rp}}^{\left(f_k\right)}|,\mathrm{and}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill g_{\mathit{mr}}& =\sum _{j=1}^k{\widehat{g}}_{\mathit{mr}}^{\left(f_j\right)}+\left|\mathsf{\Sigma }\right|+|T_{\mathit{mr}}^{\left(f_k\right)}|\hfill \\ \hfill & =\sum _{j=1}^k\sum _{i=1}^{m_j}{l}_i^{\left(f_j\right)}+\left|\mathsf{\Sigma }\right|+|T_{\mathit{mr}}^{\left(f_k\right)}|\hfill \end{array}$$

(12)

hold. Recall that each symbol $a\in \mathsf{\Sigma }$ is replaced with a new variable in the first step both in RePair and in MR-RePair. $\left|\mathsf{\Sigma }\right|$ is the size of the subgrammar consisting of the rules generated in the first step. Since every $l_i^{\left(f_j\right)}\ge 2$ and $|T_{\mathit{rp}}^{\left(f_k\right)}|=|T_{\mathit{mr}}^{\left(f_k\right)}|$, $\frac{1}{2}{g}_{\mathit{rp}}<g_{\mathit{mr}}\le g_{\mathit{rp}}$ follows Equations (11) and (12), and thus, the proposition holds. $g_{\mathit{mr}}=g_{\mathit{rp}}$ holds when every length $l_i^{\left(f_j\right)}$ is 2. □

However, when the MR-orders of RePair and MR-RePair are different, then the grammar generated by MR-RePair can be larger than that generated by RePair, as the following theorem indicates:

Theorem 6.

Unless the MR-order of RePair and MR-RePair are the same, there is a case where the size of the grammar generated by MR-RePair becomes larger than that generated by RePair.

Proof. 

We show a concrete example of the case stated in the proposition. Consider text $\mathtt{abcxabcyabczcxcxcycyczcz}$. There are four most-frequent maximal repeats, $\mathtt{abc}$, $\mathtt{cx}$, $\mathtt{cy}$, and $\mathtt{cz}$. Let A and B be two different MR-orders such that A prioritizes the maximal repeats in order of $\mathtt{abc}$, $\mathtt{cx}$, $\mathtt{cy}$, $\mathtt{cz}$ and B prioritizes the maximal repeats in order of $\mathtt{cx}$, $\mathtt{cy}$, $\mathtt{cz}$, $\mathtt{abc}$, respectively. MR-RePair working in A generates a grammar whose rules are $\{v_{\mathtt{a}}\to \mathtt{a},$ $v_{\mathtt{b}}\to \mathtt{b},$ $v_{\mathtt{c}}\to \mathtt{c},$ $v_{\mathtt{x}}\to \mathtt{x},$ $v_{\mathtt{y}}\to \mathtt{y},$ $v_{\mathtt{z}}\to \mathtt{z},$ $v_1\to v_{\mathtt{a}}{v}_{\mathtt{b}}{v}_{\mathtt{c}},$ $v_2\to v_{\mathtt{c}}{v}_{\mathtt{x}},$ $v_3\to v_{\mathtt{c}}{v}_{\mathtt{y}},$ $v_4\to v_{\mathtt{c}}{v}_{\mathtt{z}},$ $S\to v_{1}x{v}_{1}y{v}_{1}z{v}_2{v}_2{v}_3{v}_3{v}_4{v}_4\}$, where S is the start variable. Meanwhile, RePair working in B generates a grammar whose rules are $\{v_{\mathtt{a}}\to \mathtt{a},$ $v_{\mathtt{b}}\to \mathtt{b},$ $v_{\mathtt{c}}\to \mathtt{c},$ $v_{\mathtt{x}}\to \mathtt{x},$ $v_{\mathtt{y}}\to \mathtt{y},$ $v_{\mathtt{z}}\to \mathtt{z},$ $v_1\to v_{\mathtt{c}}{v}_{\mathtt{x}},$ $v_2\to v_{\mathtt{c}}{v}_{\mathtt{y}},$ $v_3\to v_{\mathtt{c}}{v}_{\mathtt{z}},$ $v_4\to v_{\mathtt{a}}{v}_{\mathtt{b}},$ $S\to v_4{v}_1{v}_4{v}_2{v}_4{v}_3{v}_1{v}_1{v}_2{v}_2{v}_3{v}_3\}$, where S is the start variable. The size of the grammar generated by MR-RePair working in A is 27, whereas the size of that generated by RePair working in B is 26. □

While Theorem 6 indicates that the grammar can be larger in MR-RePair than in RePair, in Section 5 we demonstrate that MR-RePair outperforms RePair in practice.

We can implement MR-RePair by extending the original implementation of RePair stated in [2] and holding the same complexity.

Theorem 7.

Let $G=\{V,\mathsf{\Sigma },S,R\}$ be the grammar generated by MR-RePair for a given text with a length of n. Then, MR-RePair works in $\mathcal{O}\left(n\right)$ expected time and $5n+4k^2+4k^{\prime }+\lceil \sqrt{n+1}\rceil -1$ word space, where k and $k^{\prime }$ denote the cardinalities of Σ and V, respectively.

Proof. 

Compared to RePair, the additional operations performed by MR-RePair are (i) extending the selected pair to left and right until it becomes a maximal repeat and (ii) checking and excluding either the beginning or the end of the obtained maximal repeat if they are the same. These additional operations can be realized using the same data structures as those employed in RePair. Then, the space complexity of MR-RePair follows Lemma 1.

We can clearly execute operation (ii) in a constant time. Hence, we consider how the time complexity is affected by operation (i). Let l be the length of the maximal repeat containing the focused pair, as well as f be the frequency of the pair. Then, $\mathcal{O}\left(fl\right)$ more time is required for MR-RePair to check the left- and right-extensions for all occurrences of the focused pair compared to RePair. However, the length of the entire text is shortened by at least $f(l-1)$ by the replacement. Therefore, MR-RePair works in $\mathcal{O}\left(n\right)$ expected time according to possible counts of the replacement through all of the steps of the algorithm. □

Remark 2.

We can convert a grammar of RePair to that of MR-RePair by repeating the following transform: If a variable v appears only once on the right-hand side of other rules, the rule can be removed for v, and the one occurrence of v can be replaced with the right-hand side of the removed rule. However, the time and space complexity stated in Theorem 7 cannot be achieved in this manner, since additional operations and memory for searching and storing such variables are required.

5. Experiments

We implemented and conducted some comparative experiments. In particular, we compared the sizes of constructed grammars and execution times of the proposed MR-RePair, several existing RePair implementations, and Re-PairImp (https://bitbucket.org/IguanaBen/repairimproved), which was recently proposed in [20] as an improvement of RePair.

As stated in Remark 1, the MR-order affects the size of a constructed grammar. In practice, the MR-order varies depending on the implementation of the priority queue that manages pairs. For this reason, we used four different implementations of RePair in the comparative analysis, and they were implemented by Maruyama (https://code.google.com/archive/p/re-pair/), Navarro (https://www.dcc.uchile.cl/~gnavarro/software/index.html), Prezza (https://github.com/nicolaprezza/Re-Pair) [7], and Wan (https://github.com/rwanwork/Re-Pair); we ran it with level 0 (no heuristic option), respectively.

Table 1 lists the details of the texts that we used in the experiments. In particular, we employed three texts as highly repetitive texts: one is a randomly generated text (rand77.txt), and the other two are a Fibonacci string (fib41) and a German text (einstein.de.txt) selected from the Repetitive Corpus of the Pizza&Chili Corpus (http://pizzachili.dcc.uchile.cl/repcorpus.html). The randomly generated text, rand77.txt, consists of alphanumeric symbols and some special symbols. It was generated by concatenating 32 copies of a block that includes 1024 random strings of length 64 each, i.e., the size of the randomly generated text is $64\times 1024\times 32=\mathrm{2,097,152}$ byte. In addition, we used three texts (E.coli, bible.txt, and world192.txt) selected from the Large Corpus (http://corpus.canterbury.ac.nz/descriptions/#large) to consider a real-data case. We executed each program seven times for each text and measured the elapsed CPU time only for the grammar generation process. We calculated the average time across five results, excluding the minimum and maximum values among the seven runs. The experiments were run on a computer equipped with an Intel(R) Core i7-8700 3.2–4.6 GHz 6core, 32GB RAM, and using Ubuntu 16.04. All of the programs were compiled using gcc version 7.4 with the “-O3” option.

Table 1. Text files used in our experiments.

Text	Size (bytes)	$\left|\mathbf{\Sigma }\right|$	Content
rand77.txt	2,097,152	77	32 copies of 1024 random patterns with a length of 64
fib41	267,914,296	2	Fibonacci string from the Pizza&Chili Corpus
einstein.de.txt	92,758,441	117	Edit history of the Wikipedia for Albert Einstein
E.coli	4,638,690	4	Complete genome of the E. Coli bacterium
bible.txt	4,047,392	63	The King James version of the bible
world192.txt	2,473,400	94	The CIA world fact book

Table 2 summarizes the experimental results. Unfortunately, Re-PairImp was unable to process fib41 in our experimental environment because of a lack of memory. Here, we excluded the number of rules generating a single terminal symbol from the number of rules since they are the same between RePair and MR-RePair. As shown in the table, the sizes of grammars constructed by each RePair implementation differ from each other for all texts except fib41. In any case, MR-RePair is not inferior to RePair in terms of the size of grammars while in Theorem 6 we show that the grammar can be larger in MR-RePair than in RePair if their MR-orders are different. For rand77.txt, the number of rules and size of the grammars for MR-RePair decreased to about 11% and 55% of those for RePair, respectively. Long maximal repeats occur more frequently in rand77.txt than in other texts and we consider this is a main reason of the remarkable effectiveness of MR-RePair for the text.

Table 2. Sizes of generated grammars and execution times of the considered algorithms. Each cell in the table represents the number of generated rules, total lengths of the right side of all of the rules except for the start variable, length of the right side of the start variable, and the total grammar size in the order from the top row. The total grammar size presented in the fourth row is the total of the values presented in the second row and the third row. The fifth row separated by a line represents the execution time for compression in seconds. The best results are highlighted in bold.

Text File		RePair				Re-Pair	MR-
		Maruyama	Navarro	Prezza	Wan	Imp	RePair
rand77.txt	Rules	41,651	41,642	41,632	41,675	41,661	4,492
	Total length	83,302	83,284	83,264	83,350	83,322	46,143
	Start variable	9	2	7	2	2	9
	Grammar size	83,311	83,286	83,271	83,352	83,324	46,152
	Execution time	0.22	0.34	2.94	0.94	2.48	0.20
fib41	Rules	38	38	38	38	-	38
	Total length	76	76	76	76	-	76
	Start variable	3	3	3	3	-	3
	Grammar size	79	79	79	79	-	79
	Execution time	9.99	14.38	48.85	85.39	-	14.88
einstein.de.txt	Rules	49,968	49,949	50,218	50,057	49,933	21,787
	Total length	99,936	99,898	100,436	100,114	99,866	71,709
	Start variable	12,734	12,665	13,419	12,610	12,672	12,683
	Grammar size	112,670	112,563	113,855	112,724	112,538	84,392
	Execution time	9.04	13.74	136.49	40.24	213.73	9.73
E.coli	Rules	66,664	66,757	66,660	67,368	66,739	62,363
	Total length	133,328	133,514	133,320	134,736	133,478	129,138
	Start variable	651,875	649,660	650,538	652,664	650,209	650,174
	Grammar size	785,203	783,174	783,858	787,400	783,687	779,312
	Execution time	0.52	0.65	9.82	2.00	11.29	0.58
bible.txt	Rules	81,193	81,169	80,999	81,229	81,282	72,082
	Total length	162,386	162,338	161,998	162,458	162,564	153,266
	Start variable	386,514	386,381	386,992	386,094	385,989	386,516
	Grammar size	548,900	548,719	548,990	548,552	548,553	539,782
	Execution time	0.51	0.65	8.41	1.85	11.32	0.57
world192.txt	Rules	55,552	55,798	55,409	55,473	55,437	48,601
	Total length	111,104	111,596	110,812	110,946	110,874	104,060
	Start variable	213,131	213,962	213,245	212,647	212,857	212,940
	Grammar size	324,235	325,558	324,057	323,593	323,731	317,000
	Execution time	0.32	0.55	4.92	1.09	6.81	0.36

For einstein.de.txt, the number of rules and size of the grammar decreased to about 44% and 72% of those for RePair, respectively. By contrast, it turned out that the effect of the improvement was limited for the texts from the Large Corpus, which are not highly repetitive. Note that fib41 does not contain any maximal repeats longer than 2 without overlaps. Therefore, MR-RePair generated the same rules as RePair in this case. It should be also be noted that MR-RePair runs at a speed comparable to the fastest implementation of RePair.

6. Conclusions

In this study, we analyzed the process of RePair and revealed that the RePair algorithm replaces the most frequent pairs step by step within their corresponding most frequent maximal repeats. On the basis of this analysis, we designed MR-RePair, a novel variant of RePair. Instead of consecutively substituting the most frequent pairs, MR-RePair performs one-time substitution of the most frequent maximal repeats. Furthermore, we implemented MR-RePair and compared the sizes of its constructed grammars to those of the grammars constructed by several implementations of RePair. Through the experiments, we confirmed the effectiveness of MR-RePair especially for highly repetitive texts.

We defined the greatest size difference of any two possible grammars that can be generated by RePair for a given text, naming it GSDRP. We demonstrated that a lower bound of GSDRP is $\frac{1}{6}(\sqrt{6n+1}+13)$ for a given text of length n. We left improving the lower bound and showing an upper bound of GSDRP as our future work.

We estimated the effectiveness of the compression using the size of the generated grammars instead of the length of the output bits. Reducing the grammar size has important implications since the majority of the existing text algorithms applied to grammar-compressed texts, including grammar-based self indexes [21,22], edit distance computation [23], q-gram mining [24,25], and pattern matching [26,27,28], have time/space complexities that are dependent on the input grammar size. For instance, the compressed indexes proposed by Claude and Navarro [21,22] can be directly built on MR-RePair grammar-compressed texts. Algorithms specifically designed for straight-line programs (SLPs), which are text compressions with grammars in Chomsky normal form, can also be easily modified to work on grammars that are not in Chomsky normal form similar to MR-RePair grammars. Hence, MR-RePair serves as a base for practical improvements of these algorithms.

From the viewpoint of storing data more compactly, developing a method for encoding constructed grammars is another important issue. Implementing an efficient encoding method for MR-RePair is one of our future tasks.