Burrows Wheeler Transform on a Large Scale: Algorithms Implemented in Apache Spark ^†

Abstract

With the rapid growth of Next Generation Sequencing (NGS) technologies, large amounts of “omics” data are daily collected and need to be processed. Indexing and compressing large sequences datasets are some of the most important tasks in this context. Here, we propose a novel approach for the computation of Burrows Wheeler transform relying on Big Data technologies, i.e., Apache Spark and Hadoop. We implement three algorithms based on the MapReduce framework, distributing the index computation and not only the input dataset, differently than previous approaches from the literature. Experimental results performed on real datasets show that the proposed approach is promising.

1. Background

The enormous amount of data produced by the Next-Generation Sequencing (NGS) technologies opens the way for a more comprehensive characterization of mechanisms at the molecular level, at the basis of the cellular life and may have a role in the occurrence and progress of disorders and diseases. This may help to approach fundamental questions in biological and clinical research, such as how the interactions between cellular components and chromatin structure may affect gene activity, or to what extent complex diseases such as diabetes or cancer may involve specific (epi)genomic traits. Indexing NGS data is an important problem in this context [1].

In particular, an index is a data structure that enables efficient retrieval of stored objects. Indexing strategies used in NGS allow space-efficient storage of biological sequences in a full-text index that facilitates fast querying, in order to return exact or approximate string matches. Popular full-text index data structures include variants of suffix arrays [2], FM-index based on the Burrows–Wheeler transform (BWT) and some auxiliary tables [3], and hash tables [4]. The choice of a specific index structure is often a trade-off between query speed and memory consumption. For example, hash tables can be very fast but their memory footprint is sometimes prohibitive for large string collections [5].

Here, we address the problem of computing BWT in a distributed environment, relying on Big Data technologies such as Apache Spark [6] and Hadoop [7]. To the best of our knowledge, this is the first attempt to combine Spark and Hadoop together, in order to improve the performance due to both memory and cloud optimization.

Previous research has been proposed in the literature on the BWT computation based on Apache Hadoop, such as the approaches presented in [8,9] (BigBWA). While one of the algorithms proposed in [8] computes the BWT in a MapReduce [10] fashion, in [9] the parallelism is intended only to split the input sequences and then apply another existing framework, i.e., BWA [11], in order to align them via BWT. Another related approach from the literature is RopeBWT2 [12]. However, the latter is a single-node FM-index constructor, optimized for high-end shared-memory systems, whereas our purpose is scaling the BWT computation on commodity clusters, using distributed-memory paradigms. Different from the approach proposed here, the performance of most previous methods is closely linked to hardware assumptions, rather than to tasks beyond the BWT construction itself, therefore a direct comparison with them is not suitable. The only algorithm comparable with our approach is the MapReduce one proposed in [8].

We propose an approach for the BWT computation which combines the advantages of the MapReduce paradigm [10] and the Spark Resilient Distributed Datasets (RDD) [6] (a preliminary version of it appeared in [13]). In particular, our strategy is based on the computation of Suffix Arrays in a distributed environment, by revisiting the idea of prefix doubling presented in [14]. We have also implemented in Apache Spark the MapReduce naive algorithm presented in [8], according to two different variants, and compared them with the novel one proposed here.

Validation results obtained on real biological datasets, including genomic and proteomic data, show that our approach improves the performance for BWT computation as compared to its competitors.

1.1. Preliminaries

Let S be a string of n characters defined on the alphabet $\mathrm{\Sigma }$. We denote by $S\left(i\right)$ the i-th character in S and by $S_i$ its i-th suffix. We recall the following basic notions.

1.1.1. BWT

The Burrows-Wheeler transform of S is useful in order to rearrange it into runs of similar characters. This may have advantages both for indexing and for compressing more efficiently S. The BWT applied to S returns:

• a permutation $bwt\left(S\right)$ of S, obtained by sorting all its circular shifts in lexicographic order, and then extracting the last column;
• the index (0-based) I of the row containing the original string S.

Among the most important properties of BWT, it is reversible. Figure 1 shows an example of BWT for the string S = $BANANA\$$. In particular, $bwt\left(S\right)=BNN\$AAA$, and $I=3$.

1.1.2. Suffix Array

The suffix array $SA$ of S is defined as an array of integers providing the starting positions of suffixes of S in lexicographical order. Therefore, an entry $SA\left[i\right]$ contains the starting position of the i-th suffix in S among those in lexicographic order. Figure 2 shows the Suffix Array for the same example of BWT.

1.1.3. Inverse Suffix Array

Inverse Suffix Array of S, $ISA\left[i\right]=j$ means that the rank of the suffix i is j, i.e., $SA\left[j\right]=i$.

2. Implementation

We use the notation $[i,j]$ for denoting the set $\{i, \dots , j\}$. Let $S\in \mathrm{\Sigma }*$ be a string of length n. For $i\in [0,n-1]$, let $S_i$ denote the suffix of S starting in position i and let $S_{i,j}$ denote the sub-string $S\left[i\right]S[i+1]\dots S\left[j\right]$. In order for the bwt calculation to be easily reversed, we assume that the string S always ends with a $ sentinel character, i.e., the smallest character in $\mathrm{\Sigma }$. In addition, we assume that $S\left[i\right]=\$$ for $i>n$.

In the following, we first introduce the Sorting based MapReduce algorithm proposed in [8], in two variants; then we describe in detail the Prefix Doubling algorithm proposed here.

2.1. Sorting Based MapReduce (SMR)

The first step of the algorithm SMR is suffix partitioning. The goal is to partition the set of possible suffixes into sub-sets $K_1, \dots , K_r,$ where r is a positive integer representing the desired number of partitions (i.e., the number of nodes within the cluster). The partitioning has to comply with the following property.

Property 1.

For each pair $K_i,K_j$, all suffixes in $K_i$ are lower or greater in lexicographical order than all those ones in $K_j$.

To this aim, suffixes are discriminated by their first k characters (i.e., k-mers in the following). Here, k is a positive integer chosen in advance. The algorithm SMR maps each suffix $S_i$ of S in a key-value tuple (k-mer, i) (Algorithm 1). Therefore, suffixes are partitioned according to the key, and then the tuples are sorted with respect to this key to maintain the order of the partitions.

Algorithm 1 Preparation to Partitioning
1:	procedure MAP($S_i$)
2:	return (k-mer of $S_i$, i)

Partitioning

A fairly important issue is how to implement partitioning in practice. An efficient technique is to sample the set of keys and then determine ranges based on the desired number of partitions. Then, the set of keys is partitioned according to the determined ranges, thus the partitions are balanced. Partitioning and sorting at the same time is possible this way. In [8] such a technique is used and optimized for the case of genomics sequences. It is also provided by the Spark framework [6] through the RangePartitioner functionality https://archive.apache.org/dist/spark/docs/2.3.0/api/java/index.html (accessed on 13 February 2026). The following example clarifies this aspect.

Example:

         $S =$ C A T T A T T A G G A

                    0 1 2 3 4 5 6 7 8 9 10

                        * *   * *

For $k=3$

Prefix	Sorted Partitions
(CAT, 0)	{($$$, 11), (A$$, 10)}
(ATT, 1)	{(ATT, 1), (ATT, 4), (AGG, 7)}
(TTA, 2)	{(CAT, 0), (GA$, 9), (GGA, 8)}
(TAT, 3)	{(TTA, 2), (TAT, 3), (TTA, 5)}
(ATT, 4)
(TTA, 5)
(TAG, 6)
(AGG, 7)
(GGA, 8)
(GA$, 9)
(A$$, 10)
($$$, 11)

The second step consists of completing the work by ordering suffixes partition by partition (see Algorithm 2). Suffix indexes are collected in the partition p by the procedure Calculate Partial SA. The idea is to reduce the problem of calculating the partial SA $S{A}_p$ to the calculation of another SA, $S{A}_t$, that refers to a new string T built from p. The order of suffixes in T implicitly defines the order of suffixes indexed from p in S. This approach is inspired by the recursive step in the DC3 algorithm [15].

Algorithm 2 Partial Suffix Arrays Computation
1:	procedure CALCULATE PARTIAL SA(p)
2:	Calculate $l_{max}$ the maximum distance between two elements in p
3:	Generate $\mathcal{L}$ the list of $S[p\left[i\right],l_{max}]$ for each $i\in p$
4:	Sort $\mathcal{L}$ using AlgorithmX
5:	return$S{A}_p$

Two different variants of SMR are considered: ${\mathrm{SMR}}_r$, such that AlgorithmX is the Radix Sort, and ${\mathrm{SMR}}_m$, if it is the Merge sort.

2.2. Prefix Doubling Algorithm (PDA)

The more crucial aspect for the BWT computation considered here is the calculation of the Suffix Array of the input string. Indeed, BWT can be obtained from the Suffix Array in a MapReduce fashion via join operation. Therefore, the algorithm proposed for the computation of the Suffix Array, based on the idea of prefix doubling inspired by [14], is described below (see Algorithms 3 and 4).

• Input:

Let S be a string of length n, the input tuples set is:

$$\mathrm{Input}=\left\{\right(\mathtt{null},S\left(i\right)):i=1, \dots , n)\}$$

• Output:

A set of tuples of the form $(i,r)$, where i is the index of a suffix in S and r is its rank (i.e., its position in the list of the sorted suffixes). In the literature this is referred to as the ISA. For our purpose, the resulting output is inverted in order to obtain the Suffix Array of S and, then, its BWT.

Algorithm 3 Sketch Algorithm Iterative with Prefix Doubling
1:	procedure CALCULATEISA(S)
2:	$\mathit{Input}=\left\{\right(\mathit{null},S\left(i\right)):i=1,...,n\}$
3:	Initialize set ISA with Input
4:	for ($k\leftarrow 0$ to $\lceil lo{g}_{2}n\rceil$) do
5:	Apply the operation of Shifting to ISA obtaining two sets
6:	Join the two sets obtained with the operation of Pairing
7:	Update ISA by calling RE-RANKING
8:	end for
9:	return ISA

Algorithm 4 Re-ranking
1:	procedure RE-RANKING(Pairs)
2:	Sort tuple of pairs in Pairs by value
3:	for all ($(i,(r_{a1},r_{a2}))\in Pairs$) do
4:	$\mathit{Let} (j,(r_{b1},r_{b2}))$ the previous pair to $(i,(r_{a1},r_{a2}))$
5:	if$(r_{a1},r_{a2})=(r_{b1},r_{b2})$then
6:	$r_{new}=j$
7:	else
8:	Assign to $r_{new}$ the position in the sorted set Pairs of $(i,(r_{a1},r_{a2}))$
9:	end if
10:	Update tuple $(i,r)$ in ISA with tuple $(i,r_{new})$
11:	end for

2.2.1. Initialization

The first step is starting from the Input set and initialize the set of tuples $(i,r)$, as described in the previous paragraph. In this step, the rank is based on the first character of the suffix. In particular, let $Occ\left(c\right)$ be the number of occurrences of the character lexicographically smaller of c in the string S, then the rank of the suffix i can be determined as $Occ\left(S\right(i\left)\right)$.

In a MapReduce fashion, this can be accomplished by first counting the occurrences of each character in S, and then computing the cumulative sum $Occ$ on the sorted counts. The map and reduce steps are:

$$\mathtt{map}:(\mathtt{null},S(i\left)\right)\to \left(S\right(i),1)$$

$$\mathtt{reduce}:(c,\mathtt{list}[1,1, \dots , 1\left]\right)\to (c,\mathrm{sum}\mathrm{of}\mathrm{ones})$$

From this, $Occ$ is calculated locally by collecting the result.

The $ISA$ set can be then initialized with the following map step:

$$\mathtt{map}:(\mathtt{null},S(i\left)\right)\to (i,occ(S\left(i\right)\left)\right)$$

2.2.2. ISA Extending

The next step is to extend each rank contained in the initialized $ISA$ by the whole suffix. Here, we use a technique called Prefix Doubling which is based on the following statement:

Given that the suffixes of a string are already sorted by their prefix of length h, we can deduce their ordering by their prefix of length 2 h.

Given two suffixes $S_i$ and $S_j$ with an identical prefix of length h, we can deduce their sorting by comparing the order of the suffixes $S_{i+h}$ and $S_{j+h}$. Thus, the idea is to pair, for each suffix $S_i$, its rank with the rank of the suffix $S_{i+h}$ (i.e., $\left(ISA\right[i\left]\right),ISA[i+h])$) and sort all these pairs in order to obtain the sorting by the prefix of length $2h$. Indeed, an iteration doubles the prefix, since the longest suffix has size n, all suffixes will be sorted after at most ${log}_2\left(n\right)$ iterations.

2.2.3. Shifting and Pairing

To implement the above idea in a MapReduce fashion, we apply the two considered map steps to the latest $ISA$ calculated to obtain two different sets:

$$\mathtt{map}:(i,r)\to (i,(r,0\left)\right)$$

$$\mathtt{map}:(i,r)\to (i-2^k,(-r,0))$$

where k is the number of the iterations minus one. The indices of rank are shifted this way, then the rank is paired by a reduce step. It is worth noticing that a negative number is used to denote a shifted rank, and the value is mapped as a tuple with a zero term, in order to consider the ranks shifted that overflow the string length.

The union of the two obtained sets is considered and all tuples with a negative key are discarded (the corresponding ranks do not pair with any other rank in the set). The following reduce step is applied to the union:

$$\mathtt{reduce}:(i,\mathtt{list}[(r1,0),(r2,0)\left]\right)\to (i,(r1,-r2\left)\right)$$

where $r2$ is the rank shifted. Some ranks may occur that are not reduced due to the unique key. These ranks overflow the length of S and remain paired with zero. We denote the final set derived from this phase by $Pairs$.

2.2.4. Re-Ranking

Our purpose is to extend the previous rank with a new rank, obtained by considering the prefix doubled. Therefore, we compute the new rank according to the tuple in $Pairs$ as follows: first we sort all tuples by value, then we compare each tuple at position i (after sorting) with the one in position $i-1$. If they are equal, the new rank is equal to the rank of the previous tuple, otherwise the new rank is i. Finally, a new ISA set with rank extended is obtained, and the procedure is iterated on it again. All operations described above can be achieved also in a distributed manner:

• For the sorting operation, a certain number of partitions can be identified by range into roughly equal ranges of the elements in the set (the ranges can be determined by sampling the data). Then, for each partition, a sorting algorithm is applied that sorts each partition locally. This is easily provided by the framework Apache Spark.
• In order to compute the new rank, the partition identified previously is considered and the procedure above is applied locally, as described before, using the length of the partition and the offset (i.e., the number of elements in the previous partition) for computing the position of the tuples.

2.2.5. Example

Let S = BANANA$ be the input string of length $n=7$. The input pairs are:

$$\begin{array}{c}\mathrm{Input}=\left\{\right(\mathtt{null},B),(\mathtt{null},A),(\mathtt{null},N),\\ (\mathtt{null},A),(\mathtt{null},N),(\mathtt{null},A),(\mathtt{null},\$)\}\end{array}$$

As for $Occ\left(c\right)$, it is shown in

Table 1. Computation of $Occ\left(c\right)$.

c	A	B	N	$
$Occ\left(c\right)$	0	3	4	6

.

After the initialization, the initial ISA set is:

$$\begin{array}{c}\mathrm{ISA}=\left\{\right(0,3),(1,0),(2,4),(3,0),\\ (4,4),(5,0),(6,6)\}\end{array}$$

After the first iteration, the shifted tuples are:

$$\begin{array}{c}\mathrm{Shifted}=\left\{\right(-1,(-3,0)),(0,(0,0)),(1,(-4,0)),\\ (2,(0,0\left)\right),(3,(-4,0\left)\right),(4,(0,0\left)\right),(5,(-6,0\left)\right)\}\end{array}$$

After the pairing, we obtain the set:

$$\begin{array}{c}\mathrm{Pairs}=\left\{\right(0,(3,0)),(1,(0,4)),(2,(4,0),(3,(0,4),\\ (4,(4,0),(5,(0,6),(6,(6,0\left)\right\}\end{array}$$

Finally, we sort by value and we re-rank the indices. Then, the new ISA is:

$$\begin{array}{c}\mathrm{ISA}=\left\{\right(0,3),(1,1),(2,4),(3,1),\\ (4,4),(5,0),(6,6)\}\end{array}$$

We observe that the only rank updated in this iteration is the one with index 5, indeed, shifting by 1, it is possible to distinguish among the prefixes $AN$, $AN$ and $A\$$ corresponding to the suffixes $S_1$, $S_3$ and $S_5$.

3. Results and Discussion

The presented algorithms have been evaluated on real datasets taken from the Pizza&Chili website [16], where a set of text collections of various types and sizes are available to test experimentally compressed indexes. In particular, the text collections stored on this website have been selected to form a representative sample of different applications where indexed text searching might be useful. From this collection, we have chosen the following three datasets:

• PROTEINS, containing a sequence of newline-separated protein sequences obtained from the Swissprot database.
• DNA, a sequence of newline-separated gene DNA sequences obtained from files of the Gutenberg Project.
• ENGLISH, the concatenation of English text files selected from collections of the Gutenberg Project.

We have implemented in Apache Spark all algorithms considered here. The experimental evaluations have been performed on a cluster from the GARR Cloud Platform, configured with 1 master and 48 slave nodes, each node with 6 VCore, 24 GB of RAM and 1 TB for disk. Apache Hadoop $3.1.3$ and Spark $2.3.4$ have been used.

For the PROTEINS and DNA datasets, we considered the first 50 MB, the first 100 MB, and the entire 200 MB dataset. Figure 3 focuses on the performance of the compared algorithms, showing running times for all successful executions (executions that failed or required more than 10 h to complete are omitted).

Figure 3 reports the end-to-end runtime of the three BWT implementations ${\mathrm{SMR}}_m$, ${\mathrm{SMR}}_r$ and PDA, respectively, across datasets ranging from 50 MB to 1 GB. The results are shown on a logarithmic scale to capture runtime differences exceeding two orders of magnitude. The observed trends reflect both the algorithmic structure of each variant and the way they interact with Spark’s distributed execution model.

${\mathrm{SMR}}_m$ achieves the lowest runtime on all datasets up to 200 MB, completing in approximately 0.4–2 min regardless of data type. Although algorithmically simple, ${\mathrm{SMR}}_m$ maps extremely well to Spark: it performs a single large distributed sort on the full set of suffixes/rotations and avoids multiple rounds of shuffle, intermediate data materialization, and JNI transitions. The entire computation remains within the JVM, minimizing serialization overhead and garbage-collection pressure. The high memory footprint of ${\mathrm{SMR}}_m$ (up to 1 GB for 200 MB inputs) does not significantly impact the performance at this scale, due to the fact that executors can retain the entire working set in memory. However, this prevents ${\mathrm{SMR}}_m$ from scaling to large inputs, due to the quadratic space requirements of full suffix representation, therefore it is not able to process the ENGLISH dataset.

${\mathrm{SMR}}_r$ is consistently the slowest implementation and exhibits poor scalability. While competitive on 50 MB datasets, its runtime increases linearly with input size, from about 12 min on 100 MB DNA up to more than $3.4$ h on the 200 MB PROTEINS dataset. This behavior stems from its hybrid design. Indeed, radix sorting is delegated to C++ via JNI calls. This requires the executor to repeatedly marshal data into a native format, invoke the native routine and then reconstruct the JVM objects afterwards. The associated JNI overhead grows with the number of partitions and becomes dominant for large datasets. Moreover, the interaction between radix partitioning and the highly repetitive structure of biological sequences leads to partition skew, excessive shuffle volume, and frequent spill-to-disk events, all of which significantly degrade performance. Therefore, the theoretically linear complexity of radix sorting does not materialize in practice within Spark, resulting in runtimes far worse than both ${\mathrm{SMR}}_m$ and PDA.

In contrast, PDA exhibits the most stable and predictable scaling behavior. Runtime increases gradually from 4–6 min for the 50 MB datasets to about 1.06 h for the ENGLISH dataset. This algorithm follows a prefix-doubling strategy and refines suffix ranks over multiple iterations. Each iteration triggers a distributed sort on compact integer tuples, rather than on full suffix strings, significantly reducing memory traffic. Although PDA performs several shuffle stages, the amount of data exchanged in each step is small and regular, generating well-balanced partitions and keeping executor memory pressure low (typically 150–400 MB). As a result, PDA is the only variant able to handle the 1 GB dataset reliably, confirming that a multi-pass approach with lightweight intermediate structures is more compatible with Spark’s execution model than a single-pass strategy involving heavy per-record state. Overall, the runtime results highlight that achieving optimal performance for an algorithm on a single machine does not necessarily translate into efficient execution in a distributed environment. ${\mathrm{SMR}}_m$ benefits from a single well-optimized global sort, PDA benefits from predictable multi-stage refinement with compact state, and ${\mathrm{SMR}}_r$ suffers from JNI overhead, skew-induced imbalance, and increased shuffle costs. These systems-level effects ultimately shape the performance hierarchy observed in Figure 3.

Figure 4 shows the throughput achieved by the three BWT implementations across all considered datasets. Throughput is defined as the amount of input processed per second (MB/s), and therefore higher values correspond to better performance. The results highlight three distinct behavioral patterns.

First, ${\mathrm{SMR}}_m$ achieves the highest throughput across all datasets, with values ranging from approximately $1.7$ to $2.3$ MB/s. This behavior is largely attributable to its low computational overhead. ${\mathrm{SMR}}_m$ performs significantly fewer intermediate operations than the other two implementations, indeed it benefits from reduced communication and shuffle overhead in Spark. However, this performance advantage is deceptive, as ${\mathrm{SMR}}_m$ does not scale to larger datasets and lacks robustness when the amount of data increases or distribution becomes more complex. Its superior throughput should, therefore, be interpreted as a side-effect of algorithmic simplicity, rather than genuine efficiency or scalability.

In contrast, PDA exhibits stable and predictable throughput, consistently ranging between $0.20$ and $0.32$ MB/s across all datasets and data types, including the largest ENGLISH text corpus. This stability indicates that the prefix-doubling approach maintains a balanced computation-communication profile, with resource consumption that grows proportionally to the input size. As a result, PDA is the most scalable and reliable implementation among the three. PDA consistently delivers similar throughput independent of the dataset family (DNA, PROTEINS, ENGLISH) and maintains strong performance even on large-scale inputs.

Finally, ${\mathrm{SMR}}_r$ shows the lowest throughput, with values dropping as low as $0.02$–$0.10$ MB/s on the larger datasets. This degradation is expected, due to the fact that radix-based sorting techniques require substantial initialization and partitioning overhead in distributed environments, which become progressively more expensive as the input grows. The heavy use of bucket-sorting structures and the associated data-shuffling operations significantly penalize performance, particularly on datasets above 100 MB. Notably, ${\mathrm{SMR}}_r$ could not be executed (did not complete successfully) on the 1 GB ENGLISH corpus, which further reinforces its limited suitability for large-scale distributed settings.

Overall, the throughput analysis reveals that, while ${\mathrm{SMR}}_m$ appears faster on small datasets, PDA is the only variant that maintains consistent and scalable performance across all dataset families and sizes, making it the most appropriate choice for large and heterogeneous workloads. ${\mathrm{SMR}}_r$, although theoretically efficient on uniform data distributions, proves unsuitable for distributed BWT construction at scale due to its substantial communication and initialization costs.

Figure 5 shows the peak executor-side memory usage observed during the execution of the three BWT construction strategies. The results reveal key architectural differences in how each method interacts with Spark’s execution model. ${\mathrm{SMR}}_m$ incurs the highest memory pressure, exceeding 1 GB for medium-sized inputs (200 MB). This behavior is consistent with its algorithmic structure. ${\mathrm{SMR}}_m$ explicitly materializes the full set of string rotations and sorts them lexicographically. In Spark, this results in large wide dependencies and multi-stage shuffles where each executor must temporarily hold sizable segments of the rotation matrix. Because each rotation has the same length as the input and rotations are represented as full strings or large byte arrays, the RDDs expand quadratically and stress both JVM heap and off-heap memory. The massive shuffle map output also triggers spill events and memory amplification typical of sort-based shuffle operations.

${\mathrm{SMR}}_r$ reduces the cost of comparing large rotations by performing digit-wise comparisons on fixed-size segments, but it still materializes multiple rounds of key-value buffers during each radix iteration. Since these buffers must be fully materialized before each partition-level sort, executor memory remains high and relatively invariant to input size (around 700 MB). This plateau is a consequence of radix grouping since the memory footprint scales primarily with the number of radix passes and alphabet characteristics, not with input length. Additionally, the use of wide transformations in each digit pass forces Spark to retain intermediate broadcasted ranks and local buckets in memory, further increasing peak usage.

In contrast, PDA exhibits the most favorable memory profile, maintaining a stable footprint between 150 and 400 MB across all datasets. This is directly attributable to the prefix-doubling approach. Rather than generating the full rotation matrix, the algorithm stores compact rank pairs and refines them iteratively. At each iteration, only small fixed-size records are exchanged across the cluster, leading to lightweight shuffles and minimal executor memory residency. Spark is able to pipeline transformations efficiently because intermediate state is small enough to avoid both spilling and repeated materialization. Crucially, this compact representation enables PDA to handle the ENGLISH dataset, where both ${\mathrm{SMR}}_m$ and ${\mathrm{SMR}}_r$ would exceed executor memory, while PDA requires only 157 MB at peak.

The obtained results demonstrate that the scalability of BWT construction on distributed systems is strongly influenced by data representation and shuffle patterns, rather than by input size alone. Methods that minimize wide dependencies and materialized intermediate state, such as the prefix-doubling strategy implemented in PDA, achieve significantly better memory locality, predictable peak usage, and overall robustness at scale.

To assess the scalability of the proposed distributed BWT, we measured the runtime of PDA on the ENGLISH dataset while increasing the number of worker nodes. As shown in Figure 6, execution time decreases substantially from 7400 s (24 nodes) to 5050 s (36 nodes) and 4000 s (48 nodes), confirming that the algorithm benefits from additional parallel resources.

The observed speedup, however, is sublinear, reaching around 92% parallel efficiency. The parallel efficiency ($\eta$) when scaling from 24 to 48 worker nodes is computed as

$$\eta ={\displaystyle \frac{\mathrm{Speedup}}{\mathrm{Number}\mathrm{of}\mathrm{nodes}/\mathrm{Baseline}\mathrm{nodes}}}.$$

$$\eta ={\displaystyle \frac{{\mathrm{Speedup}}_{48}}{48/24}}={\displaystyle \frac{T_{24}/T_{48}}{2}}={\displaystyle \frac{7400/4000}{2}}=0.925\approx 92\%.$$

This behavior is expected for distributed sorting–based workloads and it is mainly due to Spark-specific overheads, including shuffle traffic, synchronization barriers between iterations, partition skew, and spill-related disk I/O. Despite these bottlenecks, the iterative BWT maintains good large-scale scalability, successfully processing a ENGLISH dataset while continuing to exhibit runtime reductions as resources increase. The results indicate that the iterative variant is the most robust and scalable implementation among those evaluated.

While the empirical evaluation provides meaningful insights into the behavior of the three BWT construction strategies in Spark, several limitations must be acknowledged. First, ${\mathrm{SMR}}_m$ exhibits the highest throughput and lowest runtime on small datasets. However, this advantage is largely superficial as its algorithmic structure is inherently non-scalable. Because ${\mathrm{SMR}}_m$ materializes the full rotation matrix, its memory footprint grows quadratically with input size, preventing execution on large datasets such as the 1 GB English corpus. Consequently, large-scale measurements for ${\mathrm{SMR}}_m$ are not available, limiting the ability to perform uniform comparisons across the full dataset.

${\mathrm{SMR}}_r$ faces even more severe limitations, tied to its algorithmic structure and its interaction with Spark’s distributed execution model. Experimentally, ${\mathrm{SMR}}_r$ presents very long and often impractical processing times, even on moderately sized inputs. Although the implementation is written in C++ to maximize performance, the cost remains prohibitive. From a theoretical perspective, this behavior is expected: the computational complexity of ${\mathrm{SMR}}_r$ is $O\left(\right|p|-l_{max})$, where p is the partition under consideration and $l_{max}$ denotes the maximum distance between any two indices in p, including the final index. When partition indices are not uniformly distributed, $l_{max}$ can become very large, leading to substantial slowdowns. In distributed settings, these theoretical issues are compounded by the practical overheads of JNI transitions, repeated bucket initialization, and large-volume shuffle operations. As a consequence, ${\mathrm{SMR}}_r$ exhibits non-linear performance degradation with increasing input size and could not be executed on the largest dataset (1 GB), preventing a complete three-way comparison at scale.

${\mathrm{SMR}}_m$, although generally faster than ${\mathrm{SMR}}_r$, exhibits analogous scalability limitations. The version implemented with Merge sort benefits from a more favorable comparison cost but still cannot process very large inputs because the underlying representation of full rotations imposes excessive memory demands on executors. Thus, both SMR variants encounter fundamental limitations that hinder their applicability to large-scale distributed BWT construction.

In contrast, PDA is the only method capable of handling all dataset sizes. Despite performing multiple prefix-doubling iterations, each involving a global distributed sort, PDA maintains predictable memory usage and stable runtime behavior. Its compact representation of intermediate state minimizes shuffle volume and avoids the quadratic blow-up observed in ${\mathrm{SMR}}_m$, while also preventing the partition-skew amplification that degrades ${\mathrm{SMR}}_r$. As a result, PDA fully exploits Spark’s distributed parallelism and remains robust even under varying cluster conditions. This aligns with theoretical expectations that distributing the refinement of suffix ranks across lightweight iterations, PDA achieves consistent scalability and emerges as the only practical choice for large and heterogeneous workloads.

4. Conclusions

A novel approach for the implementation of a full-text index, that is, the Burrows Wheeler transform, has been proposed here. Three algorithms based on the MapReduce paradigm have been implemented in Apache Spark and they have been validated on real datasets.

Among the various applications where an efficient and distributed implementation of BWT may be useful (e.g., data compression, pattern matching, etc.), we mention that searching for a suitable combination of Indexing and Machine Learning techniques has recently proved to be a promising issue [17,18,19]. Therefore, the construction of an FM-index begins with text transformation using the Burrows-Wheeler Transform (BWT), which is computationally intensive for large texts such as those found in bioinformatics or search engine contexts. We plan to focus our future studies on the application of the approach proposed here for building a large FM-index more efficiently. Indeed, the BWT phase is often the bottleneck in the construction of an FM-index, so any optimization of the BWT would improve the overall performance of the index.

Further research will include the exploration of BWT construction in distributed environments for video and image compression, which are yet challenging tasks [20,21,22].