Suffix-Sorting via Shannon-Fano-Elias Codes

Abstract

Given a sequence $T=t_0{t}_1\dots t_{n-1}$ of size $n=\left|T\right|$, with symbols from a fixed alphabet Σ, $\left(\right|\Sigma |\le n)$, the suffix array provides a listing of all the suffixes of T in a lexicographic order. Given T, the suffix sorting problem is to construct its suffix array. The direct suffix sorting problem is to construct the suffix array of T directly without using the suffix tree data structure. While algorithims for linear time, linear space direct suffix sorting have been proposed, the actual constant in the linear space is still a major concern, given that the applications of suffix trees and suffix arrays (such as in whole-genome analysis) often involve huge data sets. In this work, we reduce the gap between current results and the minimal space requirement. We introduce an algorithm for the direct suffix sorting problem with worst case time complexity in $O\left(n\right)$, requiring only $(1\frac{2}{3}nlogn-nlog|\Sigma |+O\left(1\right))$ bits in memory space. This implies $5\frac{2}{3}n+O\left(1\right)$ bytes for total space requirment, (including space for both the output suffix array and the input sequence T) assuming $n\le 2^{32},|\Sigma |\le 256$, and 4 bytes per integer. The basis of our algorithm is an extension of Shannon-Fano-Elias codes used in source coding and information theory. This is the first time information-theoretic methods have been used as the basis for solving the suffix sorting problem.

1. Introduction

The suffix array provides a compact representation of all the n suffixes of T in a lexicographic order. The sequence of positions in T of the first symbols from the sorted suffixes in this lexicographic order is the suffix array for the sequence. The suffix sorting problem is to construct the suffix array of T. Mamber and Myers [1] were the first to propose an $O(nlogn)$ algorithm to construct the suffix array with three to five times less space than the traditional suffix tree. Other methods for fast suffix sorting in $O(nlogn)$ time have been reported in [2], while memory efficient constructions were considered in [3]. Puglisi et al. [4] provide a detailed comparison of different recently proposed linear time algorithms for suffix sorting.

Suffix trees and suffix arrays have drawn a significant attention in recent years due to their theoretical linear time and linear space construction, and their logarithmic search performance. Gusfield [5] provides a detailed study on the use of suffix trees in the analysis of biological sequences. Suffix sorting is also an important problem in data compression, especially for compression schemes that are based on the Burrows-Wheeler Transform [6,7]. In fact, it is known that the suffix sorting stage is a major bottleneck in BWT-based compression schemes[6,7,8]. The suffix array is usually favored over suffix trees due to its smaller memory footprint. This is important given that the applications of suffix trees and suffix arrays (such as in whole-genome sequence analysis) often involve huge data sets. An important problem therefore is to construct suffix arrays in linear time, while requiring space for the input sequence, in addition to $nlogn$ bits, the minimal space required to hold the suffix array. This translates to $5n$ bytes of total space requirment, assuming $n\le 2^{32},|\Sigma |$ = 256, and 4 bytes per integer. Existing algorithms, such as the KS Algorithm [9] require $13n$ bytes, while the KA Algorithm [10] requires $10n$ bytes.

This paper presents a space-efficient linear time algorithm for solving the direct suffix sorting problem, using only $5\frac{2}{3}n$ bytes. The basis of the algorithm is an extension of the Shannon-Fano-Elias codes pupularly used in arithmetic coding. In the next section, we provide a background to the problem, and briefly describe related work. Section 3 presents our basic algorithm for suffix sorting. Section 4 improves the complexity of the basic algorithm using methods from source coding and information theory.

2. Background

The BWT [6,7] performs a permutation of the characters in the input sequence, such that characters in lexically similar contexts will be near to each other. Thus, the BWT is very closely related to the suffix tree and suffix array - two important data structures used in pattern matching, analysis of sequence redundancy, and in data compression. The major link is the fact the BWT provides a lexicographic sorting of the contexts as part of the permutation of the input sequence. The recent book [7] provides a detailed treatment of the link between the BWT and suffix arrays and suffix trees. Apart from the BWT, other popular compression schemes have also been linked with the suffix tree data structure. For instance, the PPM* compression was implemented using suffix trees and suffix tries in [11]. The sequence decomposition step required for constructing the dictionary in LZ-based schemes can be performed by the use of tree structures, such as binary search trees, or suffix tries [12]. A detailed analysis on the use of suffix trees in data compression is provided in Szpankowski [13]. When decorated with the array of longest common prefixes, the suffix array can be used in place of the suffix tree in almost any situation where a suffix tree is used [14]. This capability is very important for applications that require huge storage, given the smaller memory requirement of the suffix array. An important issue is how to devise efficient methods for constructing the suffix array, without the suffix tree.

Some methods for constructing suffix arrays first build the suffix tree, and then construct the suffix array by performing an inorder traversal of the suffix tree. Farach al.[15] proposed a divide and conquer method to construct the suffix tree for a given sequence in linear time. The basic idea is to divide the sequence into odd and even sequences, based on the position of the symbols. Then, the suffix tree is constructed recursively for the odd sequence. Using the suffix tree for the odd sequence, they construct the suffix tree for the even sequence. The final step merges the suffix tree from the odd and even sequences into one suffix tree using a coupled depth-first search. The result is a linear time algorithm for sufix tree construction. This divide and conquer approach is the precursor to many recent algorithms for direct suffix sorting.

Given the memory requirement and implementation difficulty of the suffix tree, it is desirable to construct the suffix array directly, without using the suffix tree. Also, for certain applications such as in data compression where only the suffix array is needed, avoiding the construction of the suffix tree will have some advantages, especially with respect to memory requirements. More importantly, direct suffix sorting without the suffix tree raises some interesting algorithmic challenges. Thus, more recently, various methods have been proposed to construct the suffix array directly from the sequence [9,10,16,17,18], without the need for a suffix tree.

Kim et al. [16] followed an approach similar to Farach’s above, but for the purpose of constructing the suffix array directly. They introduce the notion of equivalent classes between strings, which they use to perform coupled depth-first searches at the merging stage. In [9] a divide and conquer approach similar to Farach’s method was used, but for direct construction of the suffix array. Here, rather than dividing the sequence into two symmetric parts, the sequence was divided into two unequal parts, by considering suffixes that begin at positions $(imod3\ne 0)$ in the sequence. These suffixes are recursively sorted, and then the remaining suffixes are sorted based on information in the first part. The two sorted suffixes are then combined using a merging step to produce the final suffix array. Thus, a major difference is in the way they divided the sequences into two parts, and in the merging step.

Ko and Aluru [10] also used recursive partitioning, but following a fundamentally different approach to construct the suffix array in linear space and linear time. They use a binary marking strategy whereby each suffix in T is classified as either an S-suffix or an L-suffix, depending on its relative order with its next neighbor. Let $T_i^{\prime }=t_i{t}_{i+1}{t}_{i+2}\dots t_{n-1}$ denote the suffix $T_i^{\prime }$ of sequence T starting at position i. An S-suffix is a suffix that is lexicographically smaller than its right neighbor in T, while an L-suffix is one that is lexicographically larger than its right neighbor. That is, $T_i^{\prime }$ is an S-suffix if $T_i^{\prime }\prec T_{i+1}^{\prime }$, otherwise $T_i^{\prime }$ is an L-suffix. This classification is motivated by the observation that an S-suffix is always lexicographically greater than any L-suffix that starts with the same first character. The two types of suffixes are then treated differently, whereby the S-suffixes are sorted recursively by performing some special distance computations. The L-suffixes are then sorted using the sorted order of the S-suffixes. The classification scheme is very similar to the approach earlier used by Itoh and Tanaka [19]. But the algorithm in [19] runs in $O(nlogn)$ time on average.

Lightweight suffix sorting algorithms [3,17,20,21] have also been proposed which pay more attention on the extra working space required. Working space excludes the space for the output suffix array. In general, the suffix array requires $n\lceil logn\rceil$ bit space, while the input text requires another $n\lceil log|\Sigma |\rceil$ bits, or in the worst case $n\lceil logn\rceil$ bits. In [22], an algorithm that runs in $O(nlogn)$ worst case time, requiring $O(nlogn+(n/\sqrt{logn}))$ working space was proposed, while Hon et al. [20] constructed suffix arrays for integer alphabets in $O(nlogn)$ time, using $O(nlogn)$-bit space. Nong and Zhang [17] combined ideas from the KS algorithm [9] and the KA algorithm [10] to develop a method that works in $O(nlog|\Sigma \left|\right)$ time, using $(nlog|\Sigma |+|\Sigma |logn)$-bit working space (without the output suffix array). In [23,24] they extended the method to use $2n+O\left(1\right)$ bytes of working space (or a total space of $7n+O\left(1\right)$ bytes, including space for the suffix array, and the original sequence), by exploiting special properties of the L and S suffixes used in the KA algorithm, and of the leftmost S-type substrings. In-place suffix sorting was considered in [25], where suffix sorting was performed for strings with symbols from a general alphabet using $O\left(1\right)$ working space in $O(nlogn)$ time. In some other related work [2,26], computing the BWT was viewed as a suffix sorting problem. Okanohara and Sadakane [26] modified a suffix sorting algorithm to compute the BWT using a working space of $O(nlog|\Sigma |log{log}_{|\Sigma |}n)$ bits. There has also been various efforts on compressed suffix arrays and compressed suffix trees as a means to tackle the space problem with suffix trees and suffix arrays. (See [27,28,29] for examples). We do not consider compressed data structures in this work. A detailed survey on suffix array construction algorithms is provided in [4].

2.1. Main Contribution

We propose a divide-and-conquer sort-and-merge algorithm for direct suffix sorting on a given input string. Given a string of length n, our algorithm runs in $O\left(n\right)$ worst case time and space. The algorithm recursively divides an input sequence into two parts, performs suffix sorting on the first part, then sorts the second part based on the sorted suffix from the first. It then merges the two smaller sorted suffixes to provide the final sorted array. The method is unique in its use of Shannon-Fano-Elias codes in efficient construction of a global partial order for the suffixes. To our knowledge, this is the first time information-theoretic methods have been used as the basis for solving the suffix sorting problem.

Our algorithm also differs from previous approaches in the use of a simple partitioning step, and how it exploits this simple partitioning scheme for conflict resolution. The total space requirement for the proposed algorithm is $(\frac{5}{3}n\lceil logn\rceil -n\lceil log|\Sigma |\rceil )+O\left(1\right)$ bits, including the space for the output suffix array, and for the original sequence. Using the standard assumption of 4 bytes per integer, with $|\Sigma |\le 256,n\le 2^{32}$, we get a total space requirement of $5\frac{2}{3}n$ bytes, including the n bytes for the original string and the $4n$ bytes for the output suffix array. This is a significant improvement when compared with other algorithms, such as the $10n$ bytes required by the KA algorithm [10], or the $13n$ bytes required by the KS algorithm [9].

3. Algorithm I: Basic Algorithm

3.1. Notation

Let $T=t_0{t}_1{t}_2\dots t_{n-1}$ be the input sequence of length n, with symbol alphabet $\Sigma =\{{\sigma }_0,{\sigma }_1,\dots ,{\sigma }_{|\Sigma |-1}\}$. Let $T_i^{\prime }=t_i{t}_{i+1}{t}_{i+2}\dots t_{n-1}$ denote the suffix of T starting at position $i(i=0,1,2,\dots n-1)$. Let $T\left[i\right]=t_i$ denote the i-th symbol in T. For any two strings, say α and β, we use $\alpha \prec \beta$ to denote that the α preceeds β in lexicographic order. (Clearly, α and β could be individual symbols, from the same alphabet, i.e. $\left|\alpha \right|=\left|\beta \right|=1$.) We use $ as the end of sequence symbol, where $\$\notin \Sigma$ and $\$<\sigma ,\forall \sigma \in \Sigma$. Further, we use $SA$ to denote the suffix array of T, and S to denote the sorted list of first characters in the suffixes. Essentially, $S\left[i\right]=T\left[SA\right[i\left]\right]$. Given $SA$, we use $S{A}^{\prime }$ to denote its inverse. We define $S{A}^{\prime }$ as follows: $S{A}^{\prime }\left[i\right]=k\text{if}SA\left[k\right]=i;i,k=0,1,\dots ,n-1$. That is, $S\left[k\right]=S\left[S{A}^{\prime }\left[i\right]\right]=T\left[i\right]$. We use $p_i$ to denote the probability of symbol $T\left[i\right]$, and $P_i$ the probability of the substring $T[i\dots i+m-1]$, the m-length substring starting at position i.

3.2. Overview

We take the general divide and conquer approach:

• Divide the sequence into two groups;
• Construct the suffix array for the first group;
• Construct the suffix array for the second group;
• Merge the suffix arrays from the two groups to form the suffix array for the parent sequence;
• Perform the above steps recursively to construct the complete suffix array for the entire sequence.

Figure 1 shows a schematic diagram for the working of the basic algorithm using an example sequence, $\mathtt{T}=\mathtt{aaaabaaaabxaaaab}$. The basic idea is to recursively partition the input sequence in a top-down manner into two equal-length subsequences according the odd and even positions in the sequence. After reaching the subsequences with length $\le 2$, the algorithm then recursively merges and sorts the subsequences using a bottom-up approach, based on the partial suffix arrays from the lower levels of the recursion. Thus, the algorithm does not start the merging procedure until it reaches the last partition on a given branch.

Each block in the figure contains two rows. The first row indicates the position of the current block of symbols in the original sequence T. The second row indicates the current symbols. The current symbols are unsorted in the downstream dividing block and sorted in the upstream merging block. To see how the algorithm works, starting from the top left block follow the solid division arrow, the horizontal trivial sort arrow (dotted arrow), and then the dashed merge arrows. The procedure ends at the top right block. We briefly describe each algorithmic step in the following. Later, we modify this basic algorithm for improved complexity results.

Figure 1. Basic working of the proposed algorithm using an example. The original sequence is indicated at the top left. The sorted sequence and the suffix array are indicated in the top right box.

Figure 1. Basic working of the proposed algorithm using an example. The original sequence is indicated at the top left. The sorted sequence and the suffix array are indicated in the top right box.

3.3. Algorithm I

3.3.1. Divide T

If $\left|T\right|\ge 2$, divide T into 2 subsequences, $T_1$ and $T_2$. $T_1$ contains all the symbols at the even positions of T. $T_2$ contains all the symbols at the odd positions of T. That is, $T_1=\{T\left[j\right],j\in [0,\left|T\right|\left)\right|jmod2=0\}$, $T_2=\{T\left[j\right],j\in [0,\left|T\right|\left)\right|jmod2=1\}$

3.3.2. Merge SA of $T_1$ and SA of $T_2$

Let $S{A}_1$ and $S{A}_2$ be the suffix array of $T_1$ and $T_2$ respectively. Let $SA$ be the suffix array of T. If $T_1$ and $T_2$ have been sorted to obtain their respective suffix arrays $S{A}_1$ and $S{A}_2$, then we can merge $S{A}_1$ and $S{A}_2$ in linear time on average, to form the suffix array $SA$. Without loss in generality, we assume $S{A}_1=a_0{a}_1{a}_2\dots a_u$, $S{A}_2=b_0{b}_1{b}_2\dots b_v$ and $SA=c_0{c}_1{c}_2\dots c_{u+v}$ are the sorted indices of $T_1$, $T_2$ and T respectively. Given $a_k$, we use ${\widehat{a}}_k$ to denote its corresponding position in T. That is, $T_1\left[S{A}_1\left[k\right]\right]=T_1\left[a_k\right]=T\left[{\widehat{a}}_k\right]$. Similarly, for ${\widehat{b}}_k$. ${\widehat{a}}_k$ and ${\widehat{b}}_k$ are easily obtained from $a_k$, based on the level of recursion, and whether we are on a left branch or a right branch. For $k=0(0\le k\le u),l=0(0\le l\le v)\text{and}g=0(0\le g\le u+v)$, we compare the partially ordered subsequences using $a_k$ and $b_l$, viz.

$$\begin{array}{cc}\text{If }T\left[{\widehat{a}}_k\right]\prec T[{\widehat{b}}_l],& \{SA\left[C_g\right]\leftarrow {\widehat{a}}_k,S[C_g]\leftarrow T\left[{\widehat{a}}_k\right];k++,g++\\ \text{If }T[{\widehat{b}}_l]\prec T\left[{\widehat{a}}_k\right],& \{SA[C_g]\leftarrow {\widehat{b}}_l,S[C_g]\leftarrow T[{\widehat{b}}_l];l++,g++\\ T\left[{\widehat{a}}_k\right]=T[{\widehat{b}}_l],& \{\begin{array}{c}{c}_x\leftarrow \text{ResolveConflict}({\widehat{a}}_k,{\widehat{b}}_l)\hfill \\ SA\left[C_g\right]\leftarrow {\widehat{c}}_x,S\left[C_g\right]\leftarrow T\left[{\widehat{c}}_x\right];g++\hfill \end{array}\end{array}$$

Whenever we compare two symbols, $T\left[{\widehat{a}}_k\right]$ from $T_1$ and $T[{\widehat{b}}_l]$ from $T_2$, we might get into the ambiguous situation whereby the two symbols are the same (i.e., $T[{\widehat{a}}_k]=T[{\widehat{b}}_l]$). Thus, we cannot easily decide which suffix precedes the other, (i.e., whether $T_{{\widehat{a}}_k}^{\prime }\prec T_{{\widehat{b}}_l}^{\prime }$, or $T_{{\widehat{b}}_l}^{\prime }\prec T_{{\widehat{a}}_k}^{\prime }$), based on the individual symbols. We call this situation a conflict. The key to the approach is how efficiently we can resolve potential conflicts as the algorithm progresses. We exploit the nature of the division procedure, and the fact that we are dealing with substrings (suffixes) of the same string, for efficient conflict resolution. Thus, the result of the merging step will be a partially sorted subsequence, based on the sorted order of the smaller child subsequences. An important difference here is that unlike in other related approaches [9,10,16], our sort-order at each recursion level is global with respect to T, rather than being local to the subsequence at the current recursion step. This is important, as it significantly simplifies the subsequent merging step.

3.3.3. Recursive Call

Using the above procedure, we recursively construct the suffix array of $T_1$ from its two children $T_{11}$ and $T_{12}$. Similarly, we obtain the suffix array for $T_2$ from its children $T_{21}$ and $T_{22}$. We follow this recursive procedure until the base case is reached (when the length of the subsequence is $\le 2$).

3.4. Conflict Resolution

We use the notions of conflict sequence or conflict pairs. Two suffixes $T_i^{\prime }$ and $T_j^{\prime }$ form a conflict sequence in T if $T_i^{\prime }\left[0\dots k\right]=T_j^{\prime }\left[0\dots k\right]$, (that is, $T[i\dots i+k]=T[j\dots j+k]$ ) for some $k\ge 1$. Thus, the two suffixes can not be assigned a total order after considering their first k symbols. We say that the conflict between $T_i^{\prime }$ and $T_j^{\prime }$ is resolved whenever $T[i\dots i+l_k-1]=T[j\dots j+l_k-1]$, and $T[i+l_k]\ne T[j+l_k]$. Here, $l_k$ is called the conflict length. We call the triple $(T_i^{\prime },T_j^{\prime },l_k)$ a conflict pair, or conflict sequence. We use the notation $CP(i,j,l_k)$ to denote a conflict pair $T_i^{\prime }$ and $T_j^{\prime }$ with a conflict length of $l_k$. We also use $CP(i,j)$ to denote a conflict pair where the conflict length is yet to be determined, or is not important given the context of the discussion.

The challenge, therefore, is how to resolve conflicts efficiently given the recursive partitioning framework proposed. Obviously, $l_k\le n$, the sequence length. Thus, the minimum distance from the start of each conflicting suffix to the end of the sequence ($min\{n-i,n-j\}$), or the distance from an already sorted symbol can determine how long it will take to resolve a conflict. Here, we consider how the previously resolved conflicts can be exploited for a quick resolution of other conflicts. We maintain a traceback record on such previously resolved conflicts in a conflict tree, or a conflict table.

Figure 2 shows the conflict tree for the example sequence, $\mathtt{T}=\mathtt{aaaabaaabxaaaab}$. Without loss in generality and for easier exposition of the basic methodology, we assume that $n=2^x$, for some positive integer x. Our conflict resolution strategy is based on the following properties of conflict sequences and conflict trees.

• Given the even-odd recursive partitioning scheme, at any given recursion level, conflicts can only occur between specific sets of positions in the original sequence T. For instance, at the lowest level, conflicts can only occur between $T_0$, and $T_{n/2}$, or more generally, between $T_i$, and $T_{i+n/2}$. See Figure 2.
• Given the conflict sequence, $T_i^{\prime }$ and $T_j^{\prime }$ the corresponding conflict pair $CP(i,j,l_k)$ is unique in T. That is, only one conflict pair can have the pair of start positions $(i,j)$. Thus the conflict pairs $CP(i,j,l_k)$ and $CP(j,i,l_k)$ are equivalent. Hence, we represent both $CP(i,j,l_k)$ and $CP(j,i,l_k)$ as $CP(i,j,l_k)$, where, $i<j$.
• Consider $CP(i,j)$ at level $h,0\le h\le \lceil logn\rceil$ in the tree. We define its neighbors as the conflict pairs: $CP(i^{\prime },j^{\prime })$, with $(i^{\prime }=i+2^q,j^{\prime }=j+2^q)$, or $(i^{\prime }=i-2^q,j^{\prime }=j-2^q)$, where $q>0$, with $q=h,h-1,h-2,\dots ,1$, and $i^{\prime },j^{\prime }\le n$. Essentially, neighboring conflicts are found on the same level in the conflict tree, from the leftmost node to the current node. For example, for $R_6=CP(2,14)$ at $h=2$, the neighbor will be: $\{R_2=CP(0,12)\}$. We notice that, by our definition, not all conflicts in the same node are neighbors.

Figure 2. Conflict tree for the example sequence $T=aaaabaaaabxaaaab$. The original sequence is indicated at the root node. The notation $R_i\mapsto R_j$ indicates that conflict pair $R_i$ is resolved by $R_j$ after a certain number of steps, given that $R_j$ has been resolved. Conflict pairs are labeled following a depth-first traversal on the tree.

Figure 2. Conflict tree for the example sequence $T=aaaabaaaabxaaaab$. The original sequence is indicated at the root node. The notation $R_i\mapsto R_j$ indicates that conflict pair $R_i$ is resolved by $R_j$ after a certain number of steps, given that $R_j$ has been resolved. Conflict pairs are labeled following a depth-first traversal on the tree.

Given the conflict tree, we can determine whether a given conflict pair $CP(i,j)$ can be resolved based on previous trace back information (i.e., previously resolved conflicts). From the conflict tree, we see that this determination can be made by checking only a fixed number of neighboring conflict pairs. In fact, from the definition above, we see that the size of the neighborhood is at most 6. For $CP(i,j)$, the neighbors will be the conflict pairs in the set $CP(i^{\prime },j^{\prime })$, with $(i^{\prime }=i-2^h,j^{\prime }=j-2^h)$; $(i^{\prime }=i+2^h,j^{\prime }=j+2^h)$; $(i^{\prime }=i-2^{h-1},j^{\prime }=j-2^{h-1})$; $(i^{\prime }=i+2^{h-1},j^{\prime }=j+2^{h-1})$; $(i^{\prime }=i-2^{h-2},j^{\prime }=j-2^{h-2})$; and $(i^{\prime }=i+2^{h-2},j^{\prime }=j+2^{h-2})$. Some of these pair of indices may not be valid indices for a conflict pair. For example, using the previous example, $R_6=CP(2,14)$ has only one valid neighbor, $\{R_2=CP(0,12)\}$. Also, given that we resolve the conflicts based on the depth first traversal order, neighbors of $CP(i,j)$ that are located on a node to the right of $CP(i,j)$ in the conflict tree cannot be used to resolve conflict $CP(i,j)$. Therefore, in some cases, less than six probes may be needed.

Let $CP(i,j)$ be the current conflict. Let $CP(i^{\prime },j^{\prime },l_k^{\prime })$ be a previously resolved neighboring conflict. We determine whether $CP(i,j)$ can be resolved based on $CP(i^{\prime },j^{\prime },l_k^{\prime })$ using a simple check: Let $\Delta =(i^{\prime }-i)=(j^{\prime }-j)$. If Δ is negative, then $CP(i,j)$ can be resolved with $CP(i^{\prime },j^{\prime },l_k^{\prime })$ iff: $|\Delta |\le l_k^{\prime }-1$ Conversely, if Δ is positive, then $CP(i,j)$ can be resolved with $CP(i^{\prime },j^{\prime },l_k^{\prime })$ iff: $l_k^{\prime }\ge 0$ If any of the two conditions holds, we say that $CP(i,j)$ is resolved by $CP(i^{\prime },j^{\prime },l_k^{\prime })$, after Δ steps. Essentially, this means that after Δ comparison steps, $CP(i,j)$ becomes equivalent to $CP(i^{\prime },j^{\prime },l_k^{\prime })$. We denote this equivalence using the notation: $CP(i,j)\mapsto CP(i^{\prime },j^{\prime },l_k^{\prime })$. The following algorithm shows how we can resolve a given conflict pair, $CP(i,j)$.

ResolveConflict$(i,j)$

Generate neighbors for $CP(i,j)$

Remove invalid neighbors

Probe the valid neighbors for previously resolved conflict pairs

for each valid neighbor, compute the quantity $d=(l_k^{\prime }-|\Delta \left|\right)$; end for

Select the neighbor $CP(i^{\prime },j^{\prime },l_k^{\prime })$ with maximum value for d

if ($\Delta <0$ and $|\Delta |\le l_k^{\prime }$), then /* no extra work is needed */

Compute $l_k=l_k^{\prime }+\Delta -1$.

else if ($\Delta >0$ and $l_k^{\prime }\ge 0$), then

Perform at most ($\Delta -1$) extra comparison steps

if (conflict is not resolved after the $\Delta -1$ comparisons), then

/*Resolve conflict using $CP(i^{\prime },j^{\prime },l_k^{\prime })$ */

Compute $l_k=l_k^{\prime }+\Delta$

end if

end if

else /* conflict cannot be resolved using earlier conflicts */

Resolve conflict using direct symbol-wise comparisons

end if

The complexity of conflict resolution (when it can be resolved using previous conflicts) thus depends on the parameter Δ. Table 1 shows the conflict pairs for the example sequence used to generate the conflict tree of Figure 2. That table includes the corresponding Δ value where a conflict can be resolved based on a previous conflict.

The final consideration is how to store the conflict tree to ensure constant-time access to the resolved conflict pairs. Since for a given $CP(i,j)$, the $(i,j)$ pair is unique, we can use these for direct access to the conflict tree. We can store the conflict tree as a simple linear array, where the positions in the array are determined based on the $(i,j)$ values, and hence the height in the tree. To avoid the sorting that may be needed for fast access using the $(i,j)$ indices, we use a simple hash function, where each hash value can be computed in constant time. The size of this array will be $O(nlogn)$ in the worst case, since there are at most $nlogn$ conflicts (see analysis below). The result will be an $O\left(1\right)$ time access to any given conflict pair, given its $(i,j)$ index in T.

Table 1. Conflict pairs for $\mathtt{T}=\mathtt{aaaabaaaabxaaaab}$. $R_q\mapsto R_p$ indicates that conflict pair $R_q$ is resolved by a previously resolved conflict pair $R_p$, after a fixed number of steps. That is, after the fixed number of steps Δ, conflict pair $R_q$ becomes equivalent to conflict pair $R_p$.

R

$CP(i,j,l_k)$

↦

Δ

R

$CP(i,j,l_k)$

↦

Δ

R

$CP(i,j,l_k)$

↦

Δ

R

$CP(i,j,l_k)$

↦

Δ

$R_1$

$(0,8,0)$

−

−

$R_9$

$(6,8,1)$

−

−

$R_{17}$

$(3,7,1)$

$R_5$

$-1$

$R_{25}$

$(1,12,4)$

$R_{22}$

$-1$

$R_2$

$(0,12,3)$

−

−

$R_{10}$

$(2,8,1)$

$R_7$

$-2$

$R_{18}$

$(5,11,5)$

$R_8$

$+1$

$R_{26}$

$(1,6,4)$

$R_{23}$

$-1$

$R_3$

$(8,12,1)$

−

−

$R_{11}$

$(8,14,2)$

$R_8$

$-2$

$R_{19}$

$(5,7,2)$

$R_9$

$+1$

$R_{27}$

$(6,13,2)$

$R_{24}$

$-1$

$R_4$

$(6,14,1)$

−

−

$R_{12}$

$(5,13,2)$

$R_4$

$+1$

$R_{20}$

$(1,7,2)$

$R_{10}$

$+1$

$R_{28}$

$(2,13,3)$

$R_{25}$

$-1$

$R_5$

$(2,6,2)$

$R_3$

$+6$

$R_{13}$

$(1,5,3)$

−

−

$R_{21}$

$(7,13,3)$

$R_{11}$

$+1$

$R_{29}$

$(2,7,3)$

$R_{26}$

$-1$

$R_6$

$(2,14,1)$

$R_2$

$-2$

$R_{14}$

$(1,13,2)$

$R_5$

$+1$

$R_{22}$

$(0,11,5)$

−

−

$R_{30}$

$(7,14,1)$

$R_{27}$

$-1$

$R_7$

$(0,6,3)$

−

−

$R_{15}$

$(3,11,1)$

$R_6$

$+1$

$R_{23}$

$(0,5,5)$

−

−

$R_{31}$

$(3,14,2)$

$R_{28}$

$-1$

$R_8$

$(6,12,4)$

−

−

$R_{16}$

$(7,11,2)$

$R_3$

$+1$

$R_{24}$

$(5,12,3)$

−

−

$R_{32}$

$(3,8,2)$

$R_{29}$

$-1$

Table 1. Conflict pairs for $\mathtt{T}=\mathtt{aaaabaaaabxaaaab}$. $R_q\mapsto R_p$ indicates that conflict pair $R_q$ is resolved by a previously resolved conflict pair $R_p$, after a fixed number of steps. That is, after the fixed number of steps Δ, conflict pair $R_q$ becomes equivalent to conflict pair $R_p$.

R

$CP(i,j,l_k)$

↦

Δ

R

$CP(i,j,l_k)$

↦

Δ

R

$CP(i,j,l_k)$

↦

Δ

R

$CP(i,j,l_k)$

↦

Δ

$R_1$

$(0,8,0)$

−

−

$R_9$

$(6,8,1)$

−

−

$R_{17}$

$(3,7,1)$

$R_5$

$-1$

$R_{25}$

$(1,12,4)$

$R_{22}$

$-1$

$R_2$

$(0,12,3)$

−

−

$R_{10}$

$(2,8,1)$

$R_7$

$-2$

$R_{18}$

$(5,11,5)$

$R_8$

$+1$

$R_{26}$

$(1,6,4)$

$R_{23}$

$-1$

$R_3$

$(8,12,1)$

−

−

$R_{11}$

$(8,14,2)$

$R_8$

$-2$

$R_{19}$

$(5,7,2)$

$R_9$

$+1$

$R_{27}$

$(6,13,2)$

$R_{24}$

$-1$

$R_4$

$(6,14,1)$

−

−

$R_{12}$

$(5,13,2)$

$R_4$

$+1$

$R_{20}$

$(1,7,2)$

$R_{10}$

$+1$

$R_{28}$

$(2,13,3)$

$R_{25}$

$-1$

$R_5$

$(2,6,2)$

$R_3$

$+6$

$R_{13}$

$(1,5,3)$

−

−

$R_{21}$

$(7,13,3)$

$R_{11}$

$+1$

$R_{29}$

$(2,7,3)$

$R_{26}$

$-1$

$R_6$

$(2,14,1)$

$R_2$

$-2$

$R_{14}$

$(1,13,2)$

$R_5$

$+1$

$R_{22}$

$(0,11,5)$

−

−

$R_{30}$

$(7,14,1)$

$R_{27}$

$-1$

$R_7$

$(0,6,3)$

−

−

$R_{15}$

$(3,11,1)$

$R_6$

$+1$

$R_{23}$

$(0,5,5)$

−

−

$R_{31}$

$(3,14,2)$

$R_{28}$

$-1$

$R_8$

$(6,12,4)$

−

−

$R_{16}$

$(7,11,2)$

$R_3$

$+1$

$R_{24}$

$(5,12,3)$

−

−

$R_{32}$

$(3,8,2)$

$R_{29}$

$-1$

3.5. Complexity Analysis

Clearly, the complexity of the algorithm depends on the overall number of conflicts, the number of conflicts that require explicit symbol-wise comparisons, the number that can be resolved using earlier conflicts, and how much extra comparsions are required for those. We can notice that we require symbol-wise comparison mainly for the conflicts at the leftmost nodes on the conflict tree, and the first few conflicts in each node. For the sequence used in Figure 2 and Table 1, with $n=16$, we have the following: 32 total number of conflict pairs; 20 can be resolved with only look-ups based on the neighbors (with no extra comparisons); 1 requires extra comparisons after lookup (a total of 6 extra comparisons); 11 cannot be resolved using lookups, and thus requires direct symbol-wise; comparisons (a total of 20 such comparisons in all). This gives 44 overall total comparisons, and 21 total lookups.

This can be compared with the results for the worst case sequence with the same length, $T={\mathtt{a}}^{16}=\mathtt{aaaaaaaaaaaaaaaa}$. Here we obtain: 49 total number of conflict pairs; 37 can be resolved with only look-ups based on the neighbors (with no extra comparisons); 8 requires extra comparisons after lookup (a total of 12 extra comparisons); 4 cannot be resolved using lookups, and thus require direct symbol-wise; comparisons (a total of 19 such comparisons in all). This results in 41 overall total comparisons and 45 total lookups. While the total number of comparisons is similar for both cases, the number of lookups is significantly higher for $T=a^n$, the worst case sequence.

The following lemma establishes the total number of symbol comparisons and the number of conflicts that can be encountered using the algorithm.

Lemma 1: Given an input sequence $T=t_0,t_2\dots ,t_{n-1}$, with length n, the maximum number of symbol comparisons and the maximum number of conflict pairs are each in $O(nlogn)$.

Proof. Deciding on whether we can resolve a given conflict based on its neigbors requires only constant time, using Algorithm ResolveConflict(). Assume we are sorting the worst case sequence, $T=a^n$. Consider level h in the conflict tree. First, we resolve the left-most conflict at this level, say conflict $R_L$. This will require at most $\frac{n}{2^{h_{max}-h}}=\frac{n}{2^h}$ symbol comparisons. For example, at $h=h_{max}-1$, the level just before the lowest level, the leftmost conflict will be $R_1=CP(0,\frac{n}{2})$, requiring $\frac{n}{2}$ comparsions. However, each conflict on the same level with $R_L$ can be resolved with at most $\frac{n}{2^{h_{max}-h+1}}=2^{h-1}$ comparisons, plus constant time lookup using the earlier resolved neighbors. A similar analysis can be made for conflict pairs $CP(i,j)$, where both i and j are odd. There are are at most $(\frac{n}{2^h}-1)$ conflicts for each node at level h of the tree. Thus, the total number of comparisons required to resolve all conflicts will be ${\sum }_{i=0}^{\lceil logn\rceil }{2}^i(\frac{n}{2^i}-1)\le nlogn$. In the worst case, each node in the tree will have the maximum $(\frac{n}{2^h}-1)$ number of conflicts. Thus, similar to the overall number of comparisons, the worst case total number of conflicts will be in $O(nlogn)$. □

We state our complexity results in the following theorem:

Theorem 1: Given an input sequence $T=t_0,t_2\dots ,t_{n-1}$, with symbols from an alphabet Σ, Algorithm I solves the suffix sorting problem in $O(nlogn)$ time and $O\left(n\right)$ space, for both the average case and the worst case.

Proof. The worst case result on required time follows from Lemma 1 above. Now consider the random string with symbols in Σ unifomly distributed. Here, the probablity of encountering one symbol is $\frac{1}{|\Sigma |}$. Thus, the probability of matching two substrings from T will decrease rapidly as the length of the substrings increase. In fact, the probability of a conflict of length $l_k$ will be $Pr\left\{CP(i,j,l_k)\right\}=\frac{1}{{|\Sigma |}^{2l_k}}$. Following [30], the maximum value of $l_k$, the length of the longest common prefix for a random string is given by $l_{max}=O\left({log}_{|\Sigma |}n\right)$. Thus, the average number of matching symbol comparisons for a given conflict will be:

$${\eta }_{compare}={\sum }_{l_k=1}^{l_{max}}\left(l_k\right)Pr\left\{CP(i,j,l_k)\right\}=\frac{1}{{|\Sigma |}^{2.1}}+\frac{2}{{|\Sigma |}^{2.2}}+\frac{3}{{|\Sigma |}^{2.3}}+\dots +\frac{l_{max}}{{|\Sigma |}^{2.l_{max}}}$$

This gives:

$${\eta }_{compare}=\frac{1-{|\Sigma |}^{2.l_{max}}}{1-|\Sigma |}=\frac{1-\frac{1}{n^2}}{1-\frac{1}{{|\Sigma |}^2}}=\frac{n^2-1}{n^2}.\frac{{|\Sigma |}^2}{{|\Sigma |}^2-1}\le 2$$

For each conflict pair, there will be exactly one mismatch, requiring one comparison. Thus, on average, we require at most 3 comparisons to resolve each conflict pair.

The probability of a conflict is just the probablity that any two randomly chosen symbols will match. This is simply $\frac{1}{{|\Sigma |}^2}$. However, we still need to make at least one comparison for each potential conflict pair. The number of this potential conflict pairs is exacty the same as the worst case number of conflicts. Thus, although the average number of comparisons per conflict is constant, the total time required to resolve all conflicts is still in $O(nlogn)$.

We can reduce the space required for conflict resolution as follows. A level-h conflict pair can be resolved based on only previous conflicts (its neighbors), all at the same level, h. Thus, rather than the current depth-first traversal, we change the traversal order on the tree, and use a bottom-up breadth-first traversal. Then, starting with the lowest level, $h=\lceil logn\rceil$, we resolve all conflicts at a given level (starting from the leftmost), before moving up to the next level. Then, we re-use the space used for the lower levels in the tree. This implies a maximum of $n-1$ entries in the hash table at any given time, or $O\left(n\right)$ space.

We make a final observation about the nature of the algorithm above. It may be seen that, in deed, we do not need to touch $T_2$, the odd tree at all, until the very last stage of final merging. Since we can sort $T_1$ to obtain $S{A}_1$ without reference to $T_2$, we can therefore use $S{A}_1$ to sort $T_2$ using radix sort, since positions in $T_1$ and $T_2$ are adjacent in T. This will eliminate consideration of the $(n-1)$ worst case conflicts at level $h=0$, and all the conflicts in $T_2$. This will however not change the complexity results, since the main culprit is the $O(nlogn)$ total number of conflicts in the worst case. We make use of this observation in the next section, to develop an $O\left(n\right)$ time and space algorithm for suffix sorting.

4. Algorithm II: Improved Algorithm

The major problem with Algorithm I is the time taken for conflict resolution. Since the worst case number of conflicts is in $O(nlogn)$, an algorithm that performs a sequential resolution of each conflict can do no better than $O(nlogn)$ time in the worst case. We improve the algorithm by modifying the recursion step, and the conflict resolution strategy. Specifically, we still use binary partitioning, but we use a non-symmetric treatment of the two branches at each recursion step. That is, only one branch will be sorted, and the second branch will be sorted based on the sorted results from the first branch. This is motivated by the observation at the end of the last section. We also use a preprocessing stage inspired by methods in information theory to facilitate fast conflict resolution.

4.1. Overview of Algorithm II

In Algorithm I, we perform symmetric division of T into two subsequences $T_1$ and $T_2$, and then merge their respective suffix arrays $S{A}_1$ and $S{A}_2$ to form $SA$, the suffix array of T. $S{A}_1$ and $S{A}_2$ in turn are obtained by recursive division of $T_1$ and $T_2$ and subsequent merging of the suffix arrays of their respective children. The improvement in Algorithm II is that when we divide T into $T_1$ and $T_2$, the division is no longer symetric. Similar to the KS Algorithm [9], here we form $T_1$ and $T_2$ as follows: $T_1=\{T\left[j\right],j\in [0,\left|T\right|\left)\right|jmod3=0\}$, $T_2=\{T\left[j\right],j\in [0,\left|T\right|\left)\right|jmod3\ne 0\}$. This is a 1:2 asymetric partitioning. The division schemes are special cases of the general $1:\eta$ partitioning, where $\eta =2$ in the above, while $\eta =1$ in the symetric partitioning scheme of Algorithm I. An important parameter is α, defined as: $\alpha =\frac{1+\eta }{\eta }$. In the current case, $\alpha =1.5$.

Further, the recursive call is now made on only one branch of T, namely $T_2$. After we obtain the suffix array $S{A}_2$ for $T_2$, we radix sort $T_1$ based on the values in $S{A}_2$ to construct $S{A}_1$. This radix sorting step only takes linear time. The merging step remains similar to Algorithm I, though with some important changes to accomodate the above non-symetric partitioning of T. We also now use an initial ordering stage for faster conflict resolution.

4.2. Sort $T_2$ to Form $S{A}_2$

After dividing T into $T_1$ and $T_2$, we need to sort each child sequence to form its own smaller suffix array. Consider $T_2$. We form $S{A}_2$ by performing the required sorting recursively, using a non-symmetric treatment of each branch. $T_2$ is thus sorted by a recursive subdivision, and local sorting at each step. The suffix arrays of the two children are then merged to form the suffix array of their parent. Figure 3 shows this procedure for the example sequence $T=\mathtt{aaaabaaaabxaaaab}$. Merging can be performed as before (with changes given the non-symetric nature of $T_1$ and $T_2$). The key to the algorithm is how to obtain the sorted array of the left child from that of the right child at each recursion step.

Figure 3. Asymmetric recursive partitioning for improved algorithm, using $T=\mathtt{aaaabaaaabxaaaaab}$. Recursive partitioning is performed on the right branch.

Figure 3. Asymmetric recursive partitioning for improved algorithm, using $T=\mathtt{aaaabaaaabxaaaaab}$. Recursive partitioning is performed on the right branch.

4.3. Sort $T_1$ by Inducing $S{A}_1$ from $S{A}_2$

Without loss in generality, we assume $T_1$ is unsorted and $T_2$ has been sorted to form $S{A}_2$. Given the partitioning scheme, we have that for any $t_k\in T_1$, $t_{k+1}\in T_2$. There must exist an ordering of $t_{k+1}\in S{A}_2$, since the indices in $S{A}_2$ are unique. For each k, we construct the set of pairs P, given by $P=\left\{\langle t_k,S{A}_2^{\prime }[k+1])\right|t_k\in T_1\rangle \}$. Each pair in P is unique. Then, we radix sort P to generate the suffix array $S{A}_1$ of $T_1$. This step can thus be accomplished in linear time. This only works for the highest level (i.e., obtaining $S{A}_1$ from $S{A}_2$). However, we can use a similar procedure, but with some added work, at the other levels of recursion.

Consider a lower level of recursion, say at level h. See Figure 3. We can append the values in the $SA$ from the right tree so that we can perform successive bucket sorts. Thus, we use the $SA$ from the right tree as the tie breaker, after a certain number of steps. In general, this will not change the number of comparisons at the lowest level of the tree. However, for the general case, this will ensure that at most ${\alpha }^{h-1}$ bucket sorts are performed, involving ${\alpha }^{h_{max}-h}$ symbols in each bucket sort, where $h_{max}=\lceil {log}_{\alpha }n\rceil$ is the lowest level of recursion. For instance, using the example in Figure 3, at $h=3$, we will have $T_1=a^2{a}^8$, and $T_2=b^4{a}^7{a}^{11}{a}^{13}$ (for convenience, we have used superscripts to denote the respective positions in the original sequence T). Assume $T_2$ has been sorted to obtain $S{A}_2=\left[3210\right]$. Using $S{A}_2$, we now form the sequences:

$T\left[2\right]T\left[4\right]\circ 3$

$T\left[8\right]T\left[11\right]\circ 1$

where symbol ’∘’ denotes concatenation. The last symbol in each sequence is obtained from $S{A}_2$, the suffix array of the right tree. These sequences are then bucket-sorted to obtain $S{A}_1$. Since bucket sorting is linear in the number of input symbols, in the worst case, this will result in a total of $O\left(n\right)$ time at each level. Thus, the overall time for this procedure will still be in $O(nlogn)$ worst case.

We can also observe that with the asymetric partitioning, and by making the recursive call on only one partition at each level, we reduce both the overall number of conflicts and number of comparisons to $O\left(n\right)$. But the number of lookups could still be in $O(nlogn)$ worst case.

4.4. Improved Sorting - Using Shannon-Fano-Elias Codes

The key to improved sorting is to reduce the number of bucket sorts in the above procedure. We do this by pre-computing some information before hand, so that the sorting can be performed based on a small block of symbols, rather than one symbol at a time. Let m be the block size. With the precomputed information, we can perform a comparison involving an m-block symbol in $O\left(1\right)$ time. This will reduce the number of bucket sorts required at each level h from ${\alpha }^{h-1}$ to $\frac{{\alpha }^{h-1}}{m}$, each involving ${\alpha }^{h_{max}-h}$ symbols. By an appropriate choice of m, we can reduce the complexity of the overall sorting procedure. For instance, with $m=loglogn$, this will lead to an overall worst case complexity in $O\left(\frac{nlogn}{loglogn}\right)$ time for determining the suffix array of $T_1$ from that of $T_2$. With $m=logn$, this gives $O\left(n\right)$ time. We use $m=logn$ in subsequent discussions.

The question that remains then is how to perform the required computations, such that all the needed block values can be obtained in linear time. Essentially, we need a pair-wise global partial ordering of the suffixes involved in each recursive step. First, we observe that we only need to consider the ordering between pairs of suffixes at the same level of recursion. The relative order between suffixes at different levels is not needed. For instance, using the example sequence of Figure 3, the sets of suffixes for which we need the pair-wise orderings will be those at positions: $\left\{\right\{1,5,10,14\};\{2,8\};\{4,13\left\}\right\}$. Each subset corresponds to a level of recursion, from $h=2$ to $h=h_{max}-1$. Notice that we don’t necessarily need those at level $h=1$, as we can directly induce the sorted order for these suffixes from $S{A}_2$, after sorting $T_2$.

We need a procedure to return the relative order between each pair of suffix positions in constant time. Given that we already have an ordering from the right tree in $S{A}_2$, we only need to consider the prefixes of the suffixes in the left tree up to the corresponding positions in $T_2$, such that we can use entries in $S{A}_2$ to break the tie, after a possible $\frac{{\alpha }^{h-1}}{m}$ bucket sorts. Let $Q_i$ be the m-length prefix of the suffix $T_i^{\prime }$: $Q_i=T[i\dots i+m-1]$. We can use a simple hash function to compute a representative of $Q_i$, for instance using the polynomial hash function:

$$h\left(Q_i\right)={\sum }_{j=0}^{m-1}{|\Sigma |}^{m-1}f\left(Q_i\left[j\right]\right)mod{n}^{\prime }$$

where $f\left(x\right)=k$, if x is the k-th symbol in $\Sigma ,k=0,1,\dots ,|\Sigma |-1$, and $n^{\prime }$ is the nearest prime number $\ge n$. The problem is that the ordering information is lost in the modulus operation. Although order-preserving hash functions exist (see [31]), these run in $O\left(n\right)$ time on average, without much guarantees on their worst case. Also, with the m-length blocks, this may require $O\left(mn\right)=O(nlogn)$ time on average.

We use an information theoretic approach to determine the ordering for the pairs. We consider the required representation for each m-length block as a codeword that can be used to represent the block. The codewords are constrained to be order preserving: That is, $\mathcal{C}\left(Q_i\right)<\mathcal{C}\left(Q_j\right)$ iff $Q_i\prec Q_j$ and $\mathcal{C}\left(Q_i\right)=\mathcal{C}\left(Q_j\right)$ iff $Q_i=Q_j$, where $\mathcal{C}\left(x\right)$ is the codeword for sequence x. Unlike in traditional source coding where we are given one long sequence to produce its compact representation, here, we have a set of short sequences, and we need to produce their respective compact representations, and these representations must be order preserving.

Let $P_i$ be the probability of $Q_i$, the m-length block starting at position i in T. Let $p_i$ be the probability of symbol $t_i=T\left[i\right]$. If necessary, we can pad T with a maximum of $(m-1)$ ’$’ symbols, to form a valid m-block at the end of the sequence. We compute the quantity: $P_i^{\prime }={\prod }_{k=i}^{i+m-1}{p}_k$. Recall that $t_i=T\left[i\right]\in \Sigma$, and ${\sum }_{j=0}^{|\Sigma |-1}Pr\left\{{\sigma }_j\right\}=1$. For a given sequence T, we should have: ${\sum }_{i=0}^{n-1}{P}_i^{\prime }=1$. However, since T may not contain all the possible $m-$length blocks in ${\Sigma }^m$, we need to normalize the product of probabilities to form a probability space:

$$P_i=\frac{P_i^{\prime }}{{\sum }_{i=0}^{n-1}{P}_i^{\prime }}$$

(1)

To determine the code for $Q_i$, we then use the cumulative distribution function (cdf) for the $P_i$’s, and determine the corresponding position for each $P_i$ in this cdf. Essentially, this is equivalent to dividing a number line in the range [0 1], such that each $Q_i$ is assigned a range proportional to its probability, $P_i$. See Figure 4. The total number of divisions will be equal to the number of unique m-length blocks in T. The problem then is to determine the specific interval on this number line that corresponds to $Q_i$, and to choose a tag $q_i$ to represent $Q_i$.

Figure 4. Code assignment by successive partitioning of a number line.

Figure 4. Code assignment by successive partitioning of a number line.

We use the following assignment procedure to compute the tag, $q_i$. First we determine the interval for the tag, based on which we compute the tag. Define the cumulative distribution function for the symbols in $\Sigma =\{{\sigma }_0,{\sigma }_1,\dots ,{\sigma }_{|\Sigma |-1}\}$: $F_x\left({\sigma }_j\right)={\sum }_{v=0}^{j}Pr\left\{{\sigma }_v\right\}$. The symbol probabilities, $Pr\left\{{\sigma }_v\right\}$’s are simply obtained based on the $p_i$’s. For each symbol ${\sigma }_k$ in Σ, we have an open interval in the cdf: $\left[F_x\left({\sigma }_{k-1}\right)F_x\left({\sigma }_k\right)\right)$. Now, given the sequence $Q_i=s_1{s}_2\dots s_k\dots s_m$, $s_i\in \Sigma$, the procedure steps through the sequence. At each step $k,k=1,2,\dots ,m$ along the sequence, we can compute $U\left(k\right)$ and $L\left(k\right)$, the respective current upper and lower ranges for the tag using the following relations:

$$\begin{array}{c}L\left(0\right)=0;U\left(0\right)=1\hfill \\ \mathbf{for}k=1\text{ to }m:\hfill \\ U\left(k\right)=L(k-1)+[U(k-1)-L(k-1)]F_x\left(s_k\right)\hfill \\ L\left(k\right)=L(k-1)+[U(k-1)-L(k-1)]F_x(s_k-1)\hfill \end{array}$$

The procedure stops at $k=m$, and the values of $U\left(k\right)$ and $L\left(k\right)$ at this final step will be the range of the tag, $q_i$. We can choose the tag $q_i$ as any number in the range: $L\left(m\right)\le q_i<U\left(m\right)$. Thus we chose $q_i$ as the mid point of the range at the final step: $q_i=\frac{U\left(m\right)+L\left(m\right)}{2}$. Figure 5(a) shows an example run of this procedure for a short sequence: $Q_i=acabd$ with a simple alphabet, $\Sigma =\{a,b,c,d\}$, and where each symbol has an equal probability $p_i=\frac{1}{4}$. This gives $q_i=\frac{271}{2048}$.

Figure 5. Code assignment procedure, using an example sequence. The vertical line represents the current state of the number line. The current interval at each step in the procedure is shown with a darker shade. The symbol considered at each step is listed under their respective number lines (a) Using the sequence $Q_i=acabd$. (b) Evolution of code assignment procedure, after removing the first symbol (in previous sequence acabd), and bringing in a new symbol a to form a new sequence: $Q_{i+1}=cabda$.

Figure 5. Code assignment procedure, using an example sequence. The vertical line represents the current state of the number line. The current interval at each step in the procedure is shown with a darker shade. The symbol considered at each step is listed under their respective number lines (a) Using the sequence $Q_i=acabd$. (b) Evolution of code assignment procedure, after removing the first symbol (in previous sequence acabd), and bringing in a new symbol a to form a new sequence: $Q_{i+1}=cabda$.

Lemma 2: The tag assignment procedure results in a tag $q_i$ that is unique to $Q_i$.

Proof. The procedure described can be seen as an extension of the Shanon-Fano-Elias coding procedure used in source coding and information theory [32]. Each tag $q_i$ is analogous to an arithmetic coding sequence of the given m-length block, $Q_i$. The open interval defined by $\left[\right(L\left(m\right)U\left(m\right))$ for each m-length sequence is unique to the sequence.

To see this uniqueness, we notice that the final number line at step m represents the cdf for all m-length blocks that appeared in the original sequence T. Denote this cdf for the m-blocks as $F_x^m$. Given $Q_i$, the i-th m-block in T, the size of its interval is given by $(U\left(m\right)-L\left(m\right))=P_i$. Since all probabilities are positive, we see that $F_x^m\left(Q_i\right)\ne F_x^m\left(Q_j\right)$ whenever $Q_i\ne Q_j$. Therefore, $F_x^m\left(Q_i\right)$ determines $Q_i$ uniquely. Thus, $F_x^m\left(Q_i\right)$ serves as a unique code for $Q_i$. Choosing any number $q_i$ within the upper and lower bounds for each $Q_i$ define a unique tag for $Q_i$. Thus the chosen tag defined by the midpoint of this interval is unique to $Q_i$. □

Lemma 3: The tags generated by the assignment procedure are order preserving.

Proof. Consider the ordering of the tags for different m-length blocks. Each step in the assignment procedure uses the same fixed order of the symbols on a number line, based on their order in Σ. Thus, the position of the upper and lower bounds at each step depends on the previous symbols considered, and the position of the current symbol in the ordered list of symbols in Σ. Therefore the $q_i$’s are ordered with respect to the lexicographic ordering of the $Q_i$’s: $q_i<q_j\iff Q_i\prec Q_j$, and $q_i=q_j\iff Q_i=Q_j$. □

Lemma 4: Given $T=t_0{t}_1\dots t_{n-1}$, all the required tags $q_i,\forall i$, can be computed in $O\left(n\right)$ time.

Proof. Suppose we have already determined $P_i$ and $q_i$ for the m-block $Q_i$ as described above. For efficient processing, we can compute $P_{i+1}$ and the tag $q_{i+1}$ in the $\left[01\right]$ number line, using the previous values for $P_i$ and $q_i$. This is based on the fact that $Q_i$ and $Q_{i+1}$ are consecutive positions in T (In practice, we need only a fraction of the positions in T, which will mean less time and space are required. But here we describe the procedure for the entire T since the complexity remains the same.). In particular, given $Q_i=T[i\dots i+m-1]$, and $Q_{i+1}=T[i+1\dots i+m]$. We compute $P_{i+1}^{\prime }$ as:

$$P_{i+1}^{\prime }=\frac{P_i^{\prime }.p_{i+m}}{p_i}$$

Then, we compute $P_{i+1}$ using Equation (1). Thus, all the required $P_i$’s can be computed in $O(n+m)$ = $O(n+logn)=O\left(n\right)$ time.

Similarly, given the tag $q_i$ for $Q_i$, and its upper and lower bounds $U\left(m\right)$ and $L\left(m\right)$, we can compute the new tag $q_{i+1}$ for the incoming m-block, $Q_{i+1}$ based on the structure of the assignment procedure used to compute $q_i$, (see Figure 5(b)). We compute the new tag $q_{i+1}$ by first computing it’s upper and lower bounds. Denote the respective upper and lower bounds for $q_i$ as: $U^i\left(m\right),L^i\left(m\right)$. Similarly, we use $U^{i+1}\left(m\right),L^{i+1}\left(m\right)$ for the respective bounds for $q_{i+1}$. Let $s=T\left[i\right]=Q_i\left[0\right]$ be the first symbol in $Q_i$. Its probability is given by $p_i$. Also, let $s_{new}=T[i+m]$ be the new symbol that is shifted in. Its probability is given by $p_{i+m}$, and we also know it’s position in the cdf. We first compute the intermediate bounds at step $k=m-1$ when using $Q_{i+1}$, namely:

$U^{i+1}(m-1)=[U\left(m\right)-F_x\left(s\right)]\left(\frac{1}{p_i}\right)$

$L^{i+1}(m-1)=[L\left(m\right)-F_x(s-1)]\left(\frac{1}{p_i}\right)$

Multiplying by $\left(\frac{1}{p_i}\right)$ changes the probability space from the previous range of $\left[F_x(s-1)F_x\left(s\right)\right)$ to $\left[01\right]$. After the computations, we can then perform the last step in the assignment procedure to determine the final range for the new tag:

$U^{i+1}\left(m\right)=L^{i+1}(m-1)+[U^{i+1}(m-1)-L^{i+1}(m-1)]F_x\left(s_{new}\right)$

$L^{i+1}\left(m\right)=L^{i+1}(m-1)+[U^{i+1}(m-1)-L^{i+1}(m-1)]F_x(s_{new}-1)$

The tag $q_{i+1}$ is then computed as the average of the two bounds as before. The worst case time complexity of this procedure is in $O(n+m+|\Sigma \left|\right)=O(n+logn+|\Sigma \left|\right)$. The $|\Sigma |$ component comes from the time needed to sort the unique symbols in T before computing the cdf. This can be performed in linear time using counting sort. Since $|\Sigma |\le n$, this gives a worst case time bound of $O\left(n\right)$ to compute the required codes for all the $O\left(n\right)$ m-length blocks. □

Figure 5(b) shows a continuation of the previous example, with the old m-block: $Q_i=acabd$, and a new m-block $Q_{i+1}=cabda$. That is, the new symbol a has been shifted in, while the first symbol in the old block has been shifted out. We observe that the general structure in Figure 5(a) is not changed by the incoming symbol, except only at the first step and last step. For the running example, the new value will be $q_{i+1}=\frac{1051}{2048}$. Table 2 shows the evolution of the upper and lower bounds for the two adjacent m-blocks. The bounds are obtained from the figures.

Table 2. Upper and lower bounds on the current interval on the number line.

	a	c	a	b	d	c	a	b	d	a
$U_k$	1	$\frac{1}{4}$	$\frac{3}{16}$	$\frac{9}{64}$	$\frac{17}{128}$	1	$\frac{3}{4}$	$\frac{9}{16}$	$\frac{17}{32}$	$\frac{17}{32}$
$L_k$	0	0	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{33}{256}$	0	$\frac{1}{2}$	$\frac{1}{2}$	$\frac{33}{64}$	$\frac{135}{256}$
$U_k-L_k$	1	$\frac{1}{4}$	$\frac{1}{16}$	$\frac{1}{64}$	$\frac{1}{256}$	1	$\frac{1}{4}$	$\frac{1}{16}$	$\frac{1}{64}$	$\frac{1}{256}$

Table 2. Upper and lower bounds on the current interval on the number line.

	a	c	a	b	d	c	a	b	d	a
$U_k$	1	$\frac{1}{4}$	$\frac{3}{16}$	$\frac{9}{64}$	$\frac{17}{128}$	1	$\frac{3}{4}$	$\frac{9}{16}$	$\frac{17}{32}$	$\frac{17}{32}$
$L_k$	0	0	$\frac{1}{8}$	$\frac{1}{8}$	$\frac{33}{256}$	0	$\frac{1}{2}$	$\frac{1}{2}$	$\frac{33}{64}$	$\frac{135}{256}$
$U_k-L_k$	1	$\frac{1}{4}$	$\frac{1}{16}$	$\frac{1}{64}$	$\frac{1}{256}$	1	$\frac{1}{4}$	$\frac{1}{16}$	$\frac{1}{64}$	$\frac{1}{256}$

Having determined $q_i$, which is fractional, we can then assign the final code for $Q_i$ by mapping the tags to an integer in the range $[0n-1]$. This can be done using a simple formula:

$$c_i=\mathcal{C}\left(Q_i\right)=\lfloor (n-1)\frac{q_i-q_{min}}{q_{max}-q_{min}}\rfloor$$

where $q_{min}={min}_i\left\{q_i\right\}$, and $q_{max}={max}_i\left\{q_i\right\}$. Notice that here, the $c_i$’s computed will not necessarily be consecutive. But they will be ordered. Also, the number of distinct $q_i$’s is at most n. The difference between $c_i$ and $c_{i+1}$ will depend on $P_i$ and $P_{i+1}$. The floor function, however, could break down the lexicographic ordering. A better approach is to simply record the position where each $Q_i$ fell on the number line. We then read off these positions from 0 to 1, and use the count at which each $Q_i$ is encountered as its code. This is easily done using the cummulative count of occurence of each distinct $Q_i$. Since the $q_i$’s are implicitly sorted, so are the $c_i$’s. We have thus obtained an ordering of all the m-length substrings in T. This is still essentially a partial ordering of all the suffixes based on their first m symbols, but a total order on the distinct m-length prefixes of the suffixes.

We now present our main results in the following theorems:

Theorem 2: Given the sequence $T=t_0{t}_1\dots t_{n-1}$, from a fixed alphabet Σ, $|\Sigma |\le n$, all the m-length prefixes of the suffixes of T can be ordered in linear time and linear space in the worst case.

Proof. The theorem follows from Lemmas 2 to 4. The correctness of the ordering follows from Lemma 2 and Lemma 3. The time complexity follows from Lemma 3. What remains is to prove is the space complexity. We only need to maintain two extra $O\left(\right|\Sigma \left|\right)$ arrays, one for the number line at each step, and the other to keep the cumulative distribution function. Thus, the space needed is also linear in $O(n+m+|\Sigma \left|\right)=O\left(n\right)$. □

Theorem 3: Given the sequence $T=t_0{t}_1\dots t_{n-1}$, with symbols from a fixed alphabet Σ, $|\Sigma |\le n$, Algorithm II computes the suffix array of T in $O\left(n\right)$ worst case time and space.

Proof. At each iteration, the recursive call applies only to the $\frac{2}{3}n$ suffixes in $T_2$. Thus, the running time for the algorithm is given by the solution to the recurrence : $\phi \left(n\right)=\phi \left(\lceil \frac{2}{3}n\rceil \right)+O\left(n\right)$. This gives $\phi \left(n\right)=O\left(n\right)$. Combined with Theorem 2, this establishes the linear time bound for the overall algorithm.

We improve the space requirement using an alternative merging procedure. Since we now have $S{A}_1$ and $S{A}_2$, we can modify the merging step by exploiting the fact that any conflict that can arise during the merging can be resolved by using only $S{A}_2$. To resolve a conflict between suffix $T_{1_i}$ in $T_1$ and suffix $T_{2_j}$ in $T_2$, we need to consider two cases:

• Case 1: If $j\text{ mod }3=1$, we compare $\langle T_1\left[i\right],S{A}_2\left[S{A}_2^{\prime }[i+1]\right]\rangle$ versus $\langle T_2\left[j\right],S{A}_2\left[S{A}_2^{\prime }[j+1]\right]\rangle$, since the relative order of both $T_{1_{i+1}}$ and $T_{2_{j+1}}$ are available from $S{A}_2$.
• Case 2: If $j\text{ mod }3=2$, we compare $\langle T_1\left[i\right],T_1[i+1],S{A}_2\left[S{A}_2^{\prime }[i+2]\right]\rangle$ versus $\langle T_2\left[j\right],T_2[j+1],S{A}_2\left[S{A}_2^{\prime }[j+1]\right]\rangle$. Again, for this case, the tie is broken using the triplet, since the relative order of both $T_{1_{i+2}}$ and $T_{2_{j+2}}$ are also available from $S{A}_2$.

Consider the step just before we obtain $S{A}_1$ from $S{A}_2$ as needed to obtain the final $SA$. We needed the codes for the m-blocks in sorting to obtain $S{A}_2$. Given the 1:2 non-symetric partitioning used, at this point, the number of such m-blocks needed for the algorithm will be $\frac{2}{3}.\frac{2}{3}n$. These require $\frac{4}{9}n$ integers to store. We need $\frac{2}{3}n$ integers to store $S{A}_2$. At this point, we also still have the inverse SA used to merge the left and right suffix arrays to form $S{A}_2$. This requires $\frac{4}{9}n$ integers for storage. Thus, the overall space needed at this point will be $1\frac{5}{9}n$ integers, in addition to the space for T. However, after getting $S{A}_2$, we no longer need the integer codes for the m-length blocks. Also, the merging does not involve $S{A}_1^{{}^{\prime }}$, so this need not be computed. Thus, we compute $S{A}_2^{\prime }$, and re-use the space for the m-block codes. $S{A}_2^{\prime }$ requires $\frac{2}{3}n$ integers. Further, since we are merging $S{A}_1$ and $S{A}_2$ from the same direction, we can construct the final SA in-place, by re-using part of the space used for the already merged sections of $S{A}_1$ and $S{A}_2$. (See for example [33,34]). Thus, the overall space requirement in bits will be $(nlog|\Sigma |+nlogn+\frac{2}{3}nlogn)=(nlog|\Sigma |+1\frac{2}{3}nlogn)$, where we need $nlog|\Sigma |$ bits to store T, $nlogn$ bits for the output suffix array, and $\frac{2}{3}nlogn$ bits for $S{A}_2^{{}^{\prime }}$. □

The above translates to a total space requirement of $7\frac{2}{3}n$ bytes, using standard assumptions of 4 bytes per integer, and 1 byte per symbol.

Though the space above is $O\left(n\right)$, the $\frac{2}{3}nlogn$ bits used to store $S{A}_2^{{}^{\prime }}$ could further be reduced. We do this by by making some observations in the above two cases encountered during merging. We can notice that after obtaining $S{A}_2$ and $S{A}_1$, we do not really need to store the text T anymore. The key observation is that, merging of $S{A}_2$ and $S{A}_1$ proceeds in the same direction for each array, for instance, from the least to the largest suffix. Thus, at the k-th step, the symbol at position $i=S{A}_2^{{}^{\prime }}\left[k\right]$ (that is, $T_2\left[i\right]$) can easily be obtained using $S{A}_2$, and two $O\left(\right|\Sigma \left|\right)$ arrays, namely, $B_1$: which stores the symbols in Σ in lexicographic order, and $B_2$ that stores the cummulative count for each symbol in $T_2$.

For $T_2[i+1]$, we compute $S{A}_2[S{A}_2^{{}^{\prime }}\left[i\right]+1])$ and use the value as index into $B_2$. We then use the position in the $B_2$ arrray to determine the symbol value from $B_1$. Similarly we obtain the symbol $T_1\left[i\right]=T_1\left[S{A}_1^{{}^{\prime }}\left[k\right]\right]$, using a second set of $O\left(\right|\Sigma \left|\right)$ arrays. For symbol $T_1[i+1]$ we do not have $S{A}_1^{{}^{\prime }}$. However, we can observe that symbol $T_1[i+1]$ will be some symbol $T_2\left[j\right]$ in $T_2$. Hence, we can use $S{A}_2$ and $S{A}_2^{{}^{\prime }}$ to determine the symbol, as described above.

Thus, we can now release the space currently used to store T and use this in part to store $S{A}_2^{{}^{\prime }}$, and then merge $S{A}_1$ and $S{A}_2$ using $S{A}_2^{{}^{\prime }}$ and the two sets of $O\left(\right|\Sigma \left|\right)$$B_1,B_2$ arrays. The space saving gained in doing this will be: ($nlog|\Sigma |-2\left(\right|\Sigma |log|\Sigma |+|\Sigma |logn))\approx nlog|\Sigma |$ bits. Using this in the previous space analysis leads to a final space requirement of $(nlogn+\frac{2}{3}nlogn+2|\Sigma |(log|\Sigma |+logn)-nlog|\Sigma |)\approx (1\frac{2}{3}nlogn-nlog|\Sigma \left|\right)$ bits. This gives $5\frac{2}{3}n$ bytes, assuming $n\le 2^{32},|\Sigma |\le 256$ at 4 bytes per integer.

Finally, since we do not need $S{A}_2^{{}^{\prime }}$ anymore, we can release the space it is occupying. Compute a new set of $B_1$ and $B_2$ arrays (in place) for the newly computed SA. The second set of $O\left(\right|\Sigma \left|\right)$ arrays are no longer needed. Using SA and the new $B_1$ and $B_2$ arrays, we now recover the original sequence T, at no extra space cost.

5. Conclusion and Discussion

We have proposed two algorithms for solving the suffix sorting problem. The first algorithm runs in $O(nlogn)$ time and $O\left(n\right)$ space, for both the averge case and worst case. Using ideas from Shannon-Fano-Elias codes used in infomation theory, the second algorithm improved the first to an $O\left(n\right)$ worst case time and space complexity. The algorithms proposed perform direct suffix sorting on the input sequence, circumventing the need to first construct the suffix tree.

We mention that the proposed algorithms are generally independent of the type of alphabet, Σ. The only requirement is that Σ be fixed during the run of the algorithm. Any given fixed alphabet can be mapped to a corresponding integer alphabet. Also, since practically the number of unique symbols in $\left|T\right|$ cannot be more than n, the size of T, it is safe to say that $n\ge |\Sigma |$.

For practical implementation, one will need to consider the problem of practical code assignment, since the procedure described may involve dealing with very small fractions, depending on m, the block length, and $|\Sigma |$. This is a standard problem in practical data compression. With $n\le 2^{32}$, $|\Sigma |=256$, we have $m=32$,and thus we may need to store values as small as $\frac{1}{{|\Sigma |}^{32}}=\frac{1}{{256}^{32}}$ while computing the tags. This translates to $\frac{1}{2^{256}}$, or about $1.158\times {10}^{-77}$. In most implementations, a variable of type $\mathtt{double}$ can store values as small as $1.7\times {10}^{-308}$. For the case of 1:2 asymetric partitioning used above, we need only only $\frac{4n}{9}$ of the m-blocks, and hence, the overall space needed to store them will still be n integers, since the size of double is typically twice that of integers. To use the approach for much larger sequences, periodic re-scaling and re-normalization schemes can be used to address the problem. Another approach will be to consider occurrence counts, rather than occurrence probabilities for the m-blocks. Here, the final number line will be a cummulative count, rather than a cdf, with the total counts being n. Then, the integer code for each m-block can easily be read off this number line, based on its overall frequency of occurrence. Therefore, we will need space for at most $\frac{4n}{9}+|\Sigma |$ integers in computing the integer codes for the positions needed. Moffat et al. [35] provide some ideas on how to address practical problems in arithmetic coding.

Overall, by a careful re-use of previously allocated space, the algorithm requires $(1\frac{2}{3}nlogn-nlog|\Sigma \left|\right)$ bits, including the n bytes needed to store the original string. This translates to $5\frac{2}{3}n$ bytes, using standard assumptions of 4 byte per integer, and 1 byte for each symbol. This is a significant improvement over the $10n$ bytes required by the KA algorithm [10], or the $13n$ bytes required by the KS algorithm [9]. Our algorithm is also unique in its use of Shannon-Fano-Elias codes, traditionally used in source coding, for efficient suffix sorting. This is the first time information-theoretic methods have been used as the basis for solving the suffix sorting problem. We believe that this new idea of using an information theoretic approach to suffix sorting could shed a new light on the problems of suffix array construction, their analysis, and applications.