Time-Universal Data Compression ^†

Abstract

Nowadays, a variety of data-compressors (or archivers) is available, each of which has its merits, and it is impossible to single out the best ones. Thus, one faces the problem of choosing the best method to compress a given file, and this problem is more important the larger is the file. It seems natural to try all the compressors and then choose the one that gives the shortest compressed file, then transfer (or store) the index number of the best compressor (it requires $logm$ bits, if m is the number of compressors available) and the compressed file. The only problem is the time, which essentially increases due to the need to compress the file m times (in order to find the best compressor). We suggest a method of data compression whose performance is close to optimal, but for which the extra time needed is relatively small: the ratio of this extra time and the total time of calculation can be limited, in an asymptotic manner, by an arbitrary positive constant. In short, the main idea of the suggested approach is as follows: in order to find the best, try all the data compressors, but, when doing so, use for compression only a small part of the file. Then apply the best data compressors to the whole file. Note that there are many situations where it may be necessary to find the best data compressor out of a given set. In such a case, it is often done by comparing compressors empirically. One of the goals of this work is to turn such a selection process into a part of the data compression method, automating and optimizing it.

1. Introduction

Nowadays lossless data compressors, or archivers, are widely used in systems of information transmission and storage. Modern data compressors are based on the results of the theory of source coding, as well as on the experience and intuition of their developers. Among the theoretical results, we note, first of all, such deep concepts as entropy, information, and methods of source coding discovered by Shannon [1]. The next important step was done by Fitingoff [2] and Kolmogorov [3], who described the first universal code, as well as Krichevsky who described the first such a code with minimal redundancy [4].

Now practically used data compressors are based on the PPM universal code [5] (which is used along with the arithmetic code [6]), the Lempel–Ziv (LZ) compression methods [7], the Burrows–Wheeler transform [8] (which is used along with the book-stack (or MTF) code [9,10,11]), the class of grammar-based codes [12,13] and some others [14,15,16]. All these codes are universal. This means that, asymptotically, the length of the compressed file goes to the smallest possible value (i.e., the Shannon entropy per letter), if the compressed sequence is generated by a stationary source.

In particular, the universality of practically used codes means that we cannot compare their performance theoretically, because all of them have the same limit ratio of compression. On the other hand, the experiments show that the performance of different data compressors depends on a compressed file and it is impossible to single out one of the best or even remove the worst ones. Thus, there is no theoretical or experimental way to select the best data compressors for practical use. Hence, if someone is going to compress a file, he should first select the appropriate data compressor, preferably giving the best compression. The following obvious two-step method can be applied: first, try all available compressors and choose the one that gives the shortest compressed file. Then place a byte representation of its number and the compressed file. When decoding, the decoder first reads the number of the selected data compressor, and then decodes the rest of the file with the selected data compressor. An obvious drawback of this approach is the need to spend a lot of time in order to first compress the file by all the compressors.

In this paper we show that there exists a method that encodes the file with the (close to) optimal compressor, but uses a relatively small extra time. In short, the main idea of the suggested approach is as follows: in order to find the best, try all the compressors, but, when doing it, use for compression only a small part of the file. Then apply the best data compressor for the compression of the whole file. Based on experiments and some theoretical considerations, we can say that under certain conditions this procedure is quite effective. That is why we call such methods “time-universal.”

It is important to note that the problems of data compression and time series prediction are very close mathematically (see, for example, [17]). That is why the proposed approach can be directly applied to time series forecasting.

To the best of our knowledge, the suggested approach to data compression is new, but the idea to organize the computation of several algorithms in such a way that any of them worked at certain intervals of time, and their course depends on intermediate results, is widely used in the theory of algorithms, randomness testing and artificial intelligence; see [18,19,20,21].

2. The Statement of the Problem and Preliminary Example

Let there be a set of data compressors $F=\{{\phi }_1,{\phi }_2,\dots \}$ and $x_1{x}_2\dots$ be a sequence of letters from a finite alphabet A, whose initial part $x_1\dots x_n$ should be compressed by some $\phi \in F$. Let $v_i$ be the time spent on encoding one letter by the data compressor ${\phi }_i$ and suppose that all $v_i$ are upper-bounded by a certain constant $v_{max}$, i.e., ${sup}_{i=1,2,\dots ,}{v}_i\le v_{max}.$ (It is possible that $v_i$ is unknown beforehand.)

The considered task is to find a data compressor from F which compresses $x_1\dots x_n$ in such a way that the total time spent for all calculations and compressions does not exceed $T(1+\delta )$ for some $\delta >0$. Note that $T=v_{max}n$ is the minimum time that must be reserved for compression and $\delta T$ is the additional time that can be used to find the good compressor (among ${\phi }_1,{\phi }_2,\dots$). It is important to note that we can estimate $\delta$ without knowing the speeds $v_1,v_2,\dots$.

If the number of data compressors F is finite, say, $\{{\phi }_1,{\phi }_2,\dots ,{\phi }_m\}$, $m\ge 2$, and one chooses ${\phi }_k$ to compress the file $x_1{x}_2\dots x_n$, he can use the following two step procedure: encode the file as $<k>{\phi }_k(x_1{x}_2\dots x_n)$, where $<k>$ is $\lceil logm\rceil$-bit binary presentation of k. (The decoder first reads $\lceil logm\rceil$ bits and finds k, then it finds $x_1{x}_2\dots x_n$ decoding ${\phi }_k(x_1{x}_2\dots x_n)$.) Now our goal is to generalize this approach for the case of infinite F = $\{{\phi }_1,{\phi }_2,\dots \}.$ For this purpose we take a probability distribution $\omega$ = ${\omega }_1,{\omega }_2,\dots$ such that all ${\omega }_i>0$. The following is an example of such a distribution:

$${\omega }_k=\frac{1}{k(k+1)},k=1,2,3,\dots .$$

(1)

Clearly, it is a probability distribution, because ${\omega }_k=1/k-1/(k+1)$.

Now we should take into account the length of a codeword which presents the number k, because those lengths must be different for different k. So, we should find such ${\phi }_k$ that the value

$$\lceil -log{\omega }_k\rceil +|{\phi }_k(x_1{x}_2\dots x_n)|$$

is close to minimal. As earlier, the first part $\lceil -log{\omega }_k\rceil$ is used for encoding number k (codes achieving this are well-known, e.g., [22].) The decoder first finds k and then $x_1{x}_2\dots x_n$ using the decoder corresponding to ${\phi }_k$. Based on this consideration, we give the following

Definition 1.

We call any method that encodes a sequence $x_1{x}_2\dots x_n$, $n\ge 1$, $x_i\in A$, by the binary word of the length $\lceil -log{\omega }_j\rceil +|{\phi }_j(x_1{x}_2\dots x_n)|$ for some ${\phi }_j\in F$, a time-adaptive code and denote it by ${\widehat{\mathrm{\Phi }}}_{compr}^{\delta }$. The output of ${\widehat{\mathrm{\Phi }}}_{compr}^{\delta }$ is the following word:

$${\widehat{\mathrm{\Phi }}}_{compr}^{\delta }(x_1{x}_2\dots x_n)=<{\omega }_i>{\phi }_i(x_1{x}_2\dots x_n),$$

(2)

where $<{\omega }_i>$ is $\lceil -log{\omega }_i\rceil$-bit word that encodes i, whereas the time of encoding is not grater than $T(1+\delta )$ (here $T=v_{max}n$).

If for a time-adaptive code ${\widehat{\mathrm{\Phi }}}_{compr}^{\delta }$ the following equation is valid

$$\underset{n\to \infty }{lim}{\widehat{\mathrm{\Phi }}}_{compr}^{\delta }(x_1\dots x_n)/n=\underset{1=1,2,\dots }{inf}\underset{n\to \infty }{lim}{\phi }_i(x_1\dots x_n)/n,$$

this code is called time-universal.

Comment 1

It will be convenient to reckon that the whole sequence is compressed not letter-by-letter, but by sub-words, each of which, say, a few kilobytes in length. More formally, let, as before, there be a sequence $x_1{x}_2\dots$, where $x_i$, $i=1,2,\dots$ are sub-words whose length (say, L) can be a few kilobytes. In this case $x_i\in {\{0,1\}}^{8L}$.

Comment 2

Here and below we did not take into account the time required for the calculation of $log{\omega }_i$ and some other auxiliary calculations. If in a certain situation this time is not negligible, it is possible to reduce $\widehat{T}$ in advance by the required value.

This description and the following discussion are fairly formal, so we give a brief preliminary example of a time-adaptive code. To do this, we took 22 data compressors from [23] and 14 files of different lengths. For each file we applied the following three-step scheme: first we took 1% of the file and sequentially compressed it with all the data compressors. Then we selected the three best compressors, took 5% of the file, and sequentially compressed it with the three compressors selected. Finally, we selected the best of these compressors and compressed the file with this compressor. Thus, the total extra time is limited to 22 × 0.01 + 3 × 0.05 = 0.37, i.e., $\delta \le 0.37$.

Table 1. Three-step compression. Extra-time $\delta$ = 0.37.

$\mathbf{File}$	Length (bites)	Best Compressor	Chosen Compressor	Chosen/Best (Ratio of Length)
BIB	111,261	nanozip	lpaq8	1.06
BOOK1	768,771	nanozip	nanozip	1
BOOK 2	610,856	nanozip	nanozip	1
GEO	102,400	nanozip	ccm	1.07
NEWS	377,109	nanozip	nanozip	1
OBJ1	21,504	nanozip	tornado	1.23
OBJ2	246,814	nanozip	lpaq8	1.08
PAPER1	53,161	nanozip	tornado	1.52
PAPER2	82,199	nanozip	tornado	1.54
PIC	513,216	zpaq	bbb	1.25
PROGC	39,611	nanozip	tornado	1.42
PROGL	71,646	nanozip	tornado	1.44
PROGP	49,379	lpaq8	tornado	1.4
TRANS	93,695	lpaq8	lpaq8	1

contains the obtained data.

Table 2. Three-step compression. Extra-time $\delta$ = 0.74.

$\mathbf{File}$	Legth	Best Compressor	Chosen Compressor	Chosen/Best (Ratio of Length)
BIB	111,261	nanozip	nanozip	1
BOOK1	768,771	nanozip	nanozip	1
BOOK 2	610,856	nanozip	nanozip	1
GEO	102,400	nanozip	nanozip	1
NEWS	377,109	nanozip	lpq1v2	1.14
OBJ1	21,504	nanozip	ccm	1.17
OBJ2	246,814	nanozip	nanozip	1
PAPER1	53,161	nanozip	lpaq8	1.19
PAPER2	82,199	nanozip	nanozip	1
PIC	513,216	zpaq	bbb	1.25
PROGC	39,611	nanozip	lpaq8	1.04
PROGL	71,646	nanozip	lpaq8	1.03
PROGP	49,379	lpaq8	lpaq8	1
TRANS	93,695	lpaq8	lpaq8	1

shows that the larger the file, the better the compression. The following table gives some insight into the effect of the extra time. Here we used the same three-step scheme, but the size of the parts was $2\%$ and $10\%$ for the first step and the second, respectively, while the extra time was 0.74.

From the tables it can be seen that the performance of the considered scheme increases significantly when the additional time increases. It worth noting, that if one applied all 22 data compressors to the whole file, the extra time would be 21 instead of 0.74.

3. The Time-Universal Code for the Finite Set of Data Compressors

3.1. Theoretical Consideration

Suppose that there is a file $x_1{x}_2\dots x_n$ and data compressors ${\phi }_1,\dots ,{\phi }_m$, $n\ge 1,m\ge 1$. Let, as before, $v_i$ be the time spent on encoding one letter by the data compressor ${\phi }_i$,

$$v_{max}=\underset{i=1,\dots ,n}{max}{v}_i,T=nv,$$

(3)

and let

$$\widehat{T}=T(1+\delta ),\delta >0.$$

(4)

The goal is to find the data compressor ${\phi }_j$, $j=1,\dots ,m$, that compresses the file $x_1{x}_2\dots x_n$ in the best way in time $\widehat{T}$.

Apparently, the following two-step method is the simplest.

Step 1. Calculate $r=\lfloor \delta T/\left(m{v}_{max}\right)\rfloor$.

Step 2. Compress the file $x_1{x}_2\dots x_r$ by ${\phi }_1$ and find the length of compressed file $|{\phi }_1(x_1\dots x_r)|$, then, likewise, find $|{\phi }_2(x_1\dots x_r)|$, etc.

Step 3. Calculate $s=arg{min}_{i=1,\dots ,m}$$|{\phi }_i(x_1\dots x_r)|$

Step 4. Compress the whole file $x_1{x}_2\dots x_n$ by ${\phi }_s$ and compose the codeword $\langle s\rangle$ ${\phi }_s(x_1\dots x_n)$, where $\langle s\rangle$ is $\lceil logm\rceil$-bit word with the presentation of s.

It will be shown that even this simple method is time universal. On the other hand, there are a lot of quite reasonable approaches to build the time-adaptive codes. For example, it could be natural to try a three step procedure, which was considered in the previous part (see Table 1 and Table 2), as well as many other versions. Probably, it could be useful to use multidimensional optimization approaches, such as machine learning, so-called deep learning, etc. That is why, we consider only some general conditions needed for time-universality.

Let us give some needed definitions. Suppose, a time-adaptive data-compressor $\widehat{\mathrm{\Phi }}$ is applied to $x=x_1\dots x_t$. For any ${\phi }_i$ we define

$${\tau }_i\left(t\right)=max\{r:{\phi }_i(x_1\dots x_r)wascalculated,whenextratime\delta Twasexhausted\}.$$

Theorem 1.

Let there be an infinite word $x_1{x}_2\dots$ and time-adaptive method $\widehat{\mathrm{\Phi }}$ which is based on the finite set of data compressors ${\varphi }_1,\dots ,{\varphi }_m$. If its additional time of calculation is not grater than $\delta T$ and the following properties are valid:

(i) the limits ${lim}_{t\to \infty }{\phi }_i(x_1\dots x_t)/t$ exist for $i=1,2,\dots ,m$,

(ii) for $i=1,2,\dots ,m$

$$\underset{t\to \infty }{lim}{\tau }_i\left(t\right)=\infty ,$$

(5)

(iii) for any t the method $\widehat{\mathrm{\Phi }}(x_1\dots x_t)$ uses such a compressor ${\phi }_s$ for which, for any i

$$(-log{\omega }_s+|{\phi }_s(x_1\dots x_{{\tau }_s\left(t\right)})|)/{\tau }_s\left(t\right)\le (-log{\omega }_i+|{\phi }_i(x_1\dots x_{{\tau }_i\left(t\right)})|)/{\tau }_i\left(t\right),$$

(6)

Then $\widehat{\mathrm{\Phi }}(x_1\dots x_n)$ is time universal, that is

$$\underset{t\to \infty }{lim}\widehat{\mathrm{\Phi }}(x_1\dots x_t)/t=\underset{i=1,2,\dots }{inf}\underset{t\to \infty }{lim}|{\phi }_i(x_1\dots x_t)|/t$$

(7)

A proof is given in the Appendix A, but here we give some informal comments. First, note that property (i) means that any data compressor will participate in the competition to find the best one. Second, if the sequence $x_1{x}_2\dots$ is generated by a stationary source and all ${\phi }_i$ are universal codes, then the property (iii) is valid with probability 1 (See, for example, [22]). Hence, this theorem is valid for this case. Besides, note that this this theorem is valid for methods described earlier.

3.2. Experiments

We conducted several experiments to evaluate the effectiveness of the proposed approach in practice. For this purpose we took 20 data compressor from the “squeeze chart (lossless data compression benchmarks)”, http://www.squeezechart.com/index.html and files from this site http://corpus.canterbury.ac.nz/descriptions/, and http://tolstoy.ru/creativity/90-volume-collection-of-the-works/ (Information about their size is given in the tables below). It is worth noting, that we do not change the collection of the data compressors and the files during experiments. The results are presented in the following tables, where the expression “worst/best” means the ratio of the longest length of the compressed file and the shortest one (for different data compressors). More formally, $worst/best={max}_{i,j=1,\dots ,20}(|{\phi }_i|/|{\phi }_j|)$. The expression “chosen/best” is a similar value for a chosen data compressor and the best one. The value “chosen/best” is the frequency of occurrence of the event “the best compressor was selected”.

Table 3. Two-step compression. Extra-time $\delta$ = 20 × 0.03 = 0.6.

Length of File (byte)	Number of Files	Ratio “Chosen Best”	Average “Worst/best”	Average “Chosen/Best”
≤${10}^5$	1496	8%	112.87%	103.57%
${10}^5–{10}^6$	1122	45.72%	131.22%	102.04%
${10}^6–{10}^8$	384	71%	147.95%	100.99%

shows the results of the two-step method, where we took 3% in the first step. Thus, the total extra time is limited to 20 × 0.03 = 0.6, i.e., $\delta \le 0.6$.

Here ratio “chosen best” means a proportion of cases in which the best method was chosen.

Table 4. Two-step compression. Extra-time $\delta$ = 20 × 0.05 = 1.

Length of File (byte)	Number of Files	Ratio “Chosen Best”	Average “Worst/Best”	Average “Chosen/Best”
≤${10}^5$	1496	16%	112.87%	102.14%
${10}^5–{10}^6$	1122	53.63%	131.22%	101.33%
${10}^6–{10}^8$	384	73%	147.95%	100.84%

shows the effect of the extra time $\delta$ on the efficiency of the method (In this case we took 5% in the first step).

Table 5. Three-step compression. Extra-time $\delta$ = $20\times 0.03+5\times 0.05$ = $0.85$.

Length of File (byte)	Number of Files	Ratio “Chosen Best”	Average “Worst/Best”	Average “Chosen/Best”
≤${10}^5$	1496	14%	112.87%	102.48%
${10}^5–{10}^6$	1122	54.9%	131.22%	101.92%
${10}^6–{10}^8$	384	73%	147.95%	100.86%

contains information about the three step method. Here we took 3% in the first step and then took five data compressors with the best performance. Then, in the second step, we tested those five data compressors taking 5% from the file. Hence, the extra time equals $20\times 0.03+5\times 0.05$ = $0.85$.

Table 6. Four-step compression. Extra-time $20\times 0.01+5\times 0.02$ + $3\times 0.05=0.45$.

Length of File (byte)	Number of Files	Ratio “Chosen Best”	Average “Worst/Best”	Average “Chosen/Best”
≤${10}^5$	1496	10%	112.87%	103.12%
${10}^5–{10}^6$	1122	44.69%	131.22%	102.54%
${10}^6–{10}^8$	384	72%	147.95%	100.88%

gives an example of four step method. Here we took 1% in the first step and then took five data compressors with the best performance. Then, in the second step, we tested those five data compressors taking 2% from each file. Basing on the obtained data, we chose three best and tested them on 5% parts. At last, the best of them was used for compression of the whole file. Hence, the extra time equals $20\times 0.01+5\times 0.02$ + $3\times 0.05=0.45$.

If we compare Table 6 and Table 3, we can see that the performance of the four step method is better than two step method, where the extra time is significantly less for the four step method. The same is valid for the considered example of the three step method.

We can see that the three- and four-step methods make sense because they make it possible to reduce the additional time while maintaining the better quality of the method. Also, we can make another important conclusion. All tables show that the method is more efficient for large files. Indeed, the ratio “chosen/best” and the average value “chosen/best” decreases where the file lengths increases. Moreover, the average value “worst/best” increases where the file lengths increases.

4. The Time-Universal Code for Stationary Ergodic Sources

In this section we describe a time-universal code for stationary sources. It is based on optimal universal codes for Markov chains, developed by Krichevsky [4,24] and the twice-universal code [25]. Denote by $M_i$, $i=1,2,\dots$ the set of Markov chains with memory (connectivity) i, and let $M_0$ be the set of Bernoulli sources. For stationary ergodic $\mu$ and an integer r we denote by $h_r\left(\mu \right)$ the r-order entropy (per letter) and let $h_{\infty }\left(\mu \right)$ be the limit entropy; see for definitions [22].

Krichevsky [4,24] described the codes ${\psi }_0,{\psi }_1,\dots$ which are asymptotically optimal for $M_0,M_1,\dots$, correspondingly. If the sequence $x_1{x}_2\dots x_n$, $x_i\in A$, is generated by a source $\mu$ $\in M_i$, the following inequalities are valid almost surely (a.s.):

$$h_i\left(\mu \right)\le |{\psi }_i(x_1\dots x_t)|/t\le h_i\left(\mu \right)+\left(\right(|A|-{1)|A|}^i+C)/t,$$

(8)

where t grows. (Here C is a constant.) The length of a codeword of the twice-universal code $\rho$ is defined as the following “mixture”:

$$|\rho (x_1\dots x_t)|=-log\sum _{i=0}^{\infty }{\omega }_{i+1}{2}^{-|{\psi }_i(x_1\dots x_t)|}.$$

(9)

(It is well-known in information theory [22] that there exists a code with such codeword lengths, because ${\sum }_{x_1\dots x_t\in A^t}$$2^{-|\rho (x_1\dots x_t)|}$ = $1$.) This code is called twice-universal because for any $M_i$, $i=0,1,\dots$, and $\mu \in M_i$ the equality (8) is valid (with different C). Besides, for any stationary ergodic source $\mu$ a.s.

$$\underset{t\to \infty }{lim}|{\rho }_i(x_1\dots x_t)|/t=h_{\infty }\left(\mu \right).$$

(10)

Let us estimate the time of calculations necessary when using $\rho$. First, note that it suffices to sum a finite number of terms in (9), because all the terms $2^{-|{\psi }_i(x_1\dots x_t)|}$ are equal for $i\ge t$. On the other hand, the number of different terms grows, where $t\to \infty$ and, hence, the encoder should calculate $2^{-|{\psi }_i(x_1\dots x_t)|}$ for growing number i’s. It is known [24] that the time spent on coding one letter is close for different codes ${\psi }_i,i=0,1,2,\dots$.

Hence, the time spent for encoding one letter by the code $\rho$ grows to infinity, when t grows. The described below time-universal code ${\mathrm{\Psi }}^{\delta }$ has the same asymptotic performance, but the time spent for encoding one letter is a constant.

In order to describe the time-universal code ${\mathrm{\Psi }}^{\delta }$ we give some definitions. Let, as before, v be an upper-bound of the time spent for encoding one letter by any ${\psi }_i$, $x_1\dots x_t$ be the generated word,

$$T=tv,N\left(t\right)=\delta T/v=\delta t,$$

$$m\left(t\right)=\lfloor loglogN\left(t\right)\rfloor ,s\left(t\right)=\lfloor N\left(t\right)/\left(m\right(t)+1)\rfloor .$$

(11)

Denote by ${\mathrm{\Psi }}^{\delta }$ the following method:

Step 1. Calculate $m\left(t\right),s\left(t\right)$ and

$$|{\psi }_0(x_1\dots x_{s\left(t\right)})|,|{\psi }_1(x_1\dots x_{s\left(t\right)})|,\dots ,|{\psi }_{m\left(t\right)}(x_1\dots x_{s\left(t\right)})|.$$

Step 2. Find such a j that

$$|{\psi }_j(x_1\dots x_{s\left(t\right)})|=\underset{i=0,\dots ,m\left(t\right)}{min}|{\psi }_i(x_1\dots x_{s\left(t\right)})|.$$

Step 3. Calculate the codeword ${\psi }_j(x_1\dots x_t)$ and output

$${\mathrm{\Psi }}^{\delta }(x_1\dots x_t)=<j>{\psi }_j(x_1\dots x_t),$$

where $<j>$ is the $\lceil -log{\omega }_{j+1}\rceil$-bit codeword of j. The decoding is obvious.

Theorem 2.

Let $x_1{x}_2\dots$ be a sequence generated by a stationary source and the code ${\mathrm{\Psi }}^{\delta }$ be applied. Then this code is time-universal, i.e., a.s.

$$\underset{t\to \infty }{lim}|{\mathrm{\Psi }}^{\delta }(x_1\dots x_t)|/t=\underset{i=0,1,\dots }{inf}\underset{t\to \infty }{lim}|{\psi }_i(x_1\dots x_t)|/t.$$

(12)