A Note on Sequence Prediction over Large Alphabets

Abstract

Building on results from data compression, we prove nearly tight bounds on how well sequences of length n can be predicted in terms of the size $\sigma$ of the alphabet and the length k of the context considered when making predictions. We compare the performance achievable by an adaptive predictor with no advance knowledge of the sequence, to the performance achievable by the optimal static predictor using a table listing the frequency of each $(k+1)$-tuple in the sequence. We show that, if the elements of the sequence are chosen uniformly at random, then an adaptive predictor can compete in the expected case if $k\le {log}_{\sigma }n-3-ϵ$, for a constant $ϵ>0$, but not if $k\ge {log}_{\sigma }n$.

1. Introduction

The relation between compression and prediction dates back at least as far as William of Ockham in the fourteenth century. This relation was not properly formalized, however, until the notion of Kolmogorov complexity was developed in the twentieth century [1,2,3]. Since then, there have been many efforts to harness compression algorithms for prediction, with a number of researchers focusing particularly on prediction for prefetching either disk pages or web pages. We refer the reader to the text by Cesa–Bianchi and Lugosi [4] and references therein for a thorough discussion. For example, Krishnan and Vitter [5] showed that a prefetcher based on LZ78 [6] is asymptotically competitive with the best finite-state prefetcher. For prefetching, however, the alphabet of possible elements—i.e., all pages on the disk or web—is huge. In this paper we investigate, therefore, what effect the size of the alphabet has on predictability.

Krishnan and Vitter considered the problem of pure prefetching, in which the prefetcher can replace all the contents of the cache between each page request. They combined a predictor by Hannan [7], which is asymptotically competitive against any memoryless predictor, with an instantaneous version of LZ78, thus obtaining a predictor such that, for any finite-state predictor and any sequence, their predictor’s success rate converges to or exceeds the finite-state predictor’s success rate with probability approaching 1 as the sequence length increases. (As Krishnan and Vitter noted, a similar construction for predicting binary sequences was given by Feder, Merhav and Gutman [8].) Notice that this bound cannot be improved to hold with certainty instead of with high probability: the predictor must be randomized because, for any deterministic predictor, an adversary can choose each bit of the sequence to be the opposite of what the predictor guesses. Krishnan and Vitter’s proof is based on the fact that, for any finite-state compressor and any sequence, LZ78’s asymptotic compression ratio is at most that of the finite-state compressor; this necessarily involves the assumption that both the alphabet and the context length used in prediction are fixed. It is known what effect the size of the alphabet has on compressibility, both when we make certain assumptions about the source of the sequence [9,10] and when we want to bound the size of the encoding in terms of the kth-order empirical entropy of the sequence [11,12]. We will define a notion of predictability that is analogous to empirical entropy and use similar arguments to give nearly tight bounds on how large the alphabet can be before we cannot guarantee good prediction.

The rest of this paper is laid out as follows: in Section 2 we briefly review several notions of entropy in computer science—Shannon entropy, empirical entropy, Rényi entropy and min-entropy—before defining empirical predictability; in Section 3 we show that we can achieve good expected prediction in terms of a sequence’s kth-order empirical predictability when $k\le {log}_{\sigma }n-3-ϵ$, where k is the length of the contexts considered when making predictions, $\sigma$ is the size of the alphabet, n is the length of the sequence and $ϵ>0$ is a constant; in Section 4 we show we cannot achieve the same bound when $k\ge {log}_{\sigma }n$. A preliminary version of these results [13] was reported at the 9th Canadian Workshop on Information Theory (CWIT ’05) while the author was at the University of Toronto.

2. Empirical Predictability

Shannon [14] defined the entropy of a random variable to be our uncertainty about its value. Specifically, if a random variable $\mathcal{X}$ takes on one of $\sigma$ values according to a probability distribution $P=p_1,\dots ,p_{\sigma }$, then

$$H\left(\mathcal{X}\right)=\sum _{j=1}^{\sigma }{p}_{j}log\frac{1}{p_j}$$

The base of the logarithm determines the unit of uncertainty; in computer science, the base is usually assumed to be 2 with the result that the unit is the bit (i.e., our uncertainty about the outcome of flipping a fair coin). Throughout the rest of this paper we write log to mean ${log}_2$. Given P, the expected number of bits needed to encode the value of $\mathcal{X}$ is at least $H\left(\mathcal{X}\right)$ and less than $H\left(\mathcal{X}\right)+1$.

The 0th-order empirical entropy $H_0\left(S\right)$ of a sequence $S[1\dots n]$ is simply our uncertainty about an element chosen uniformly at random from S, i.e.,

$$H_0\left(S\right)=\frac{1}{n}\sum _{a\in S}\mathsf{occ}(a,S)log\frac{n}{\mathsf{occ}(a,S)}$$

where $a\in S$ means that element a occurs in S and $\mathsf{occ}(a,S)$ is its frequency. For $k\ge 1$, the kth-order empirical entropy $H_k\left(S\right)$ of S is our expected uncertainty about the random variable $s\left[i\right]$ in the following experiment: i is chosen uniformly at random between 1 and n; if $i\le k$, then we are told $s\left[i\right]$; if $i>k$, then we are told $s[i-k\dots i-1]$ and asked to guess $s\left[i\right]$. Specifically,

$$H_k\left(S\right)=\frac{1}{n}\sum _{\left|\alpha \right|=k}\left|S_{\alpha }\right|H_0\left(S_{\alpha }\right)$$

where $S_{\alpha }$ is the concatenation of the elements in S immediately following occurrences of the k-tuple $\alpha$. Notice that $|S_{\alpha }|=\mathsf{occ}(\alpha ,S)$ unless $\alpha$ is a suffix of S, in which case it is 1 less. Given a table listing the frequency of each $(k+1)$-tuple in S, it takes about $n{H}_k\left(S\right)$ bits to encode S. For further discussion of empirical entropy, we refer readers to Manzini’s analysis [15] of the Burrows–Wheeler Transform.

The Rényi entropy of order t of $\mathcal{X}$ is defined as

$$\frac{1}{1-t}log\left(\sum _{j=1}^{\sigma }{p}_j^t\right)$$

for $0\le t\ne 1$, where the random variable $\mathcal{X}$ again takes on values according to the probability distribution $P=p_1,\dots ,p_{\sigma }$. The Rényi entropy of order 0 of $\mathcal{X}$ is the logarithm of the size of the support of P. The limit of the Rényi entropy of order t of $\mathcal{X}$ as t approaches 1 is the Shannon entropy $H\left(\mathcal{X}\right)$ of $\mathcal{X}$; as t approaches infinity, the limit is $-log{sup}_{1\le j\le \sigma }{p}_j$, which is often called the min-entropy of X. Min-entropy is related to predictability because, given P and asked to guess the value of X, our best strategy is to choose the most probable value and, thus, guess correctly with probability ${max}_{1\le i\le \sigma }\left\{p_i\right\}$.

If we are asked to guess the value of an element $s\left[i\right]$ chosen uniformly at random from the sequence S with no context given, then our best strategy is to choose the most frequent element and, thus, guess correctly with probability ${max}_{a\in S}\frac{\mathsf{occ}(a,S)}{n}$. Following the example of empirical entropy, we call this probability the 0th-order empirical predictability $P_0\left(S\right)$ of S. We define the kth-order empirical predictability $P_k\left(S\right)$ of S to be the expected predictability of the random variable $s\left[i\right]$ in the following experiment: i is chosen uniformly at random between 1 and n; if $i\le k$, then we are told $s\left[i\right]$; if $i>k$, then we are told $s[i-k\dots i-1]$ and asked to guess $s\left[i\right]$. Specifically,

$$P_k\left(S\right)=\frac{1}{n}\left(k+\sum _{\left|\alpha \right|=k}\left|S_{\alpha }\right|P_0\left(S_{\alpha }\right)\right)$$

for $k\ge 1$, where $S_{\alpha }$ is again the concatenation of the elements in S immediately following occurrences of the k-tuple $\alpha$. For example, if $S=\mathsf{TORONTO}$ then $P_0\left(S\right)=3/7\approx 0.429$,

$$\begin{array}{ccc}\hfill P_1\left(S\right)& =& \frac{1}{7}\left(1+P_0\left(S_{\mathsf{N}}\right)+2P_0\left(S_{\mathsf{O}}\right)+P_0\left(S_{\mathsf{R}}\right)+2P_0\left(S_{\mathsf{T}}\right)\right)\hfill \\ & =& \frac{1}{7}\left(1+P_0\left(\mathsf{T}\right)+2P_0\left(\mathsf{RN}\right)+P_0\left(\mathsf{O}\right)+2P_0\left(\mathsf{OO}\right)\right)\hfill \\ & =& \frac{6}{7}\approx 0.857\hfill \end{array}$$

and all higher-order empirical entropies are 1. In other words, if someone asks us to guess an element chosen uniformly at random from $\mathsf{TORONTO}$ then, given no context, we should choose $\mathsf{O}$, in which case the probability of our prediction being correct is $3/7$. If we are given the preceding element (or told there is no preceding element) and it is not an $\mathsf{O}$, then we can answer with certainty; if it is an $\mathsf{O}$, which has probability $2/7$, then we should guess either $\mathsf{R}$ or $\mathsf{N}$ and be right with probability $1/2$; overall, the probability of our prediction being correct is $6/7$. If we are given the two preceding elements (or told how many preceding elements there are), then we can always answer with certainty.

Given a table listing the frequency of each $(k+1)$-tuple in S, we can build a static predictor that, after seeing k elements, always predicts the element that follows that k-tuple most often in S; this predictor guesses correctly $n{P}_k\left(S\right)$ times when predicting all the elements in S, which is optimal for a static predictor that uses contexts of length at most k.

3. Upper Bound

Having defined the kth-order predictability $P_k\left(S\right)$ of S, it is natural to ask when an adaptive predictor with no advance knowledge of S can achieve success rate $P_k\left(S\right)$. Whenever both k and the size $\sigma$ of the alphabet are fixed, Krishnan and Vitter’s predictor [5] almost certainly achieves a success rate asymptotically approaching $P_k\left(S\right)$ as n goes to infinity. If S is a randomly-chosen permutation, however, then $P_1\left(S\right)=1$ but the expected success rate of any predictor without advance knowledge of S, approaches 0 as n increases. In this section we show that if $k\le {log}_{\sigma }n-3-ϵ$, for a constant $ϵ>0$, then an adaptive predictor can achieve expected success rate $P_k\left(S\right)$ on any sufficiently long sequence. For simplicity we assume that k is given although, in practice, a predictor should find an optimal or nearly optimal context length by itself.

The most obvious 0th-order predictor is the one that always guess that the next element will be the most frequent element seen so far. Hannan [7] randomized this predictor and obtained a predictor A whose expected success rate converges to $P_0\left(S\right)$ when $n=\omega \left({\sigma }^3\right)$. We now consider the most obvious generalization $A_k$ of Hannan’s predictor to use contexts of a given length k: after seeing a k-tuple $\alpha$, we apply Hannan’s predictor to the subsequence of elements consisting of the concatenation of elements so far that immediately followed occurrences of $\alpha$; i.e.,

$$A_k\left(S[1\dots i]\right)=A\left({\left(S[1\dots i]\right)}_{\alpha }\right)$$

where $\alpha =S[i-k\dots i-1]$ and ${\left(S[1\dots i]\right)}_{\alpha }$ is as defined in Section 2.

Fix $ϵ>0$ and assume $k\le {log}_{\sigma }n-3-ϵ$. Consider the subsequences into which $A_k$ partitions S before applying A, and let $\mathcal{L}$ be the subset of them that are each of length at least ${\sigma }^{3+ϵ/2}$. Notice that A achieves expected success rate $P_0\left(S^{\prime }\right)$ on any subsequence $S^{\prime }\in \mathcal{L}$ so, by linearity of expectation, $A_k$ achieves expected success rate at least

$$\frac{1}{n}\sum _{\genfrac{}{}{0pt}{}{\left|\alpha \right|=k,}{S_{\alpha }\in \mathcal{L}}}\left|S_{\alpha }\right|P_0\left(S_{\alpha }\right)$$

On the other hand, the total length of the subsequences not in $\mathcal{L}$ is less than ${\sigma }^k·{\sigma }^{3+ϵ/2}<\frac{n}{{\sigma }^{ϵ/2}}=o\left(n\right)$, so

$$\begin{array}{ccc}\hfill P_k\left(S\right)& =& \frac{1}{n}\left(k+\sum _{\genfrac{}{}{0pt}{}{\left|\alpha \right|=k,}{S_{\alpha }\in \mathcal{L}}}|S_{\alpha }|P_0\left(S_{\alpha }\right)+\sum _{\genfrac{}{}{0pt}{}{\left|\alpha \right|=k,}{S_{\alpha }\notin \mathcal{L}}}\left|S_{\alpha }\right|P_0\left(S_{\alpha }\right)\right)\hfill \\ & =& \frac{1}{n}\sum _{\genfrac{}{}{0pt}{}{\left|\alpha \right|=k,}{S_{\alpha }\in \mathcal{L}}}\left|S_{\alpha }\right|P_0\left(S_{\alpha }\right)+o\left(1\right)\hfill \end{array}$$

Therefore, when S is sufficiently long, $A_k$ achieves expected success rate $P_k\left(S\right)$.

Theorem 1

If the n elements of a sufficiently long sequence S are chosen arbitrarily from an alphabet of size σ and $k<{log}_{\sigma }n-3-ϵ$, for a constant $ϵ>0$, then $A_k$ achieves expected success rate $P_k\left(S\right)$.

4. Lower Bound

Compression researchers (see, e.g., [16] for a survey) have shown how to store S in $n{H}_k\left(S\right)+o(nlog\sigma )$ for all $k\le (1-ϵ){log}_{\sigma }n$ simultaneously, where $ϵ$ is a positive constant. In a previous paper [11] we showed that it is impossible to prove a worst-case bound of this form when $k\ge {log}_{\sigma }n$:

• in $\sigma$-ary De Bruijn cycles [17] of order k, each k-tuple appears exactly once, so such cycles have length ${\sigma }^k$ and kth-order empirical entropy 0;
• there are ${(\sigma !)}^{{\sigma }^{k-1}}/{\sigma }^k$ such cycles [18] and ${log}_2\left({(\sigma !)}^{{\sigma }^{k-1}}/{\sigma }^k\right)=\mathsf{\Theta }({\sigma }^{k}log\sigma )$;
• by the pigeonhole principle, there is no injective mapping from $\sigma$-ary strings of length n with kth-order empirical entropy 0, to binary strings of length $o(nlog\sigma )$;
• therefore, if $k\ge {log}_{\sigma }n$, then in the worst case we cannot store S in $\lambda n{H}_k\left(S\right)+o(nlog\sigma )$ bits for any coefficient $\lambda$.

In a recent paper [12] we used similar but more sophisticated arguments to show that, if $k\ge (1+ϵ){log}_{\sigma }n$ for some positive constant $ϵ$, then in the expected case we cannot store S in $\lambda n{H}_k\left(S\right)+o(nlog\sigma )$ bits for any coefficient $\lambda =o\left(n^{ϵ}\right)$; if $k\ge (2+ϵ){log}_{\sigma }n$, then with high probability we cannot store S in that many bits for any coefficient $\lambda$.

We now turn our attention to proving lower bounds for prediction and show that, if $k\ge {log}_{\sigma }n$ and the elements of S are chosen uniformly at random, then no predictor without advance knowledge of S can achieve an expected success rate close to $P_k\left(S\right)$. Notice that $n{P}_k\left(S\right)$ is, by definition, at least the number of distinct k-tuples in S minus 1: for any distinct k-tuple $\alpha$ that occurs in S and is not a suffix of S, the optimal static predictor described in Section 2 correctly guesses the element after at least one occurrence of $\alpha$ in S. Suppose $k\ge {log}_{\sigma }n$—implying that $\sigma \ge 2$—and let $c={\sigma }^k/n\ge 1$. Janson, Lonardi and Szpankowski [19] showed that the expected number of distinct k-tuples in S is

$${\sigma }^k\left(1-\frac{1}{e^{n/{\sigma }^k}}\right)+\mathcal{O}\left(k\right)+\mathcal{O}\left(\frac{nk}{{\sigma }^k}\right)=cn\left(1-\frac{1}{e^{1/c}}\right)+\mathcal{O}\left(k\right)+\mathcal{O}\left(k/c\right)$$

so

$$\mathrm{E}\left[P_k\left(S\right)\right]\ge c\left(1-\frac{1}{e^{1/c}}\right)\ge 1-\frac{1}{e}>0.632$$

On the other hand, no predictor without advance knowledge of S can achieve an expected success rate greater than $1/\sigma \le 1/2$.

Theorem 2

If the n elements of a sequence S are chosen uniformly at random from an alphabet of size σ and $k\ge {log}_{\sigma }n$, then S’s expected kth-order empirical predictability $\mathrm{E}\left[P_k\left(S\right)\right]\ge 1-1/e>0.632$ but no predictor without advance knowledge of S can achieve an expected success rate greater than $1/\sigma \le 1/2$.