Generalized Forms of the Kraft Inequality for Finite-State Encoders

Abstract

We derive a few extended versions of the Kraft inequality for information lossless finite-state encoders. The main basic contribution is in defining a notion of a Kraft matrix and in establishing the fact that a necessary condition for information losslessness of a finite-state encoder is that none of the eigenvalues of this matrix have modulus larger than unity, or equivalently, the spectral radius of the Kraft matrix cannot exceed one. We then derive several equivalent forms of this condition, which are based on well-known formulas for spectral radius. Even stronger results are presented for the important special case where the finite-state encoder is assumed irreducible. Finally, two extensions are outlined—one concerns the case of side information available to both encoder and decoder, and the other is for lossy compression.

1. Introduction

Kraft’s inequality plays a pivotal role in information theory. It provides a complete and elegant characterization of the feasibility of variable-length uniquely decodable (UD) codes by imposing a simple constraint on codeword lengths. In 1949, Kraft [1] introduced this inequality for prefix codes, establishing a condition on codeword lengths necessary for prefix decodability. Seven years later, McMillan [2] generalized this to UD codes, leading to the Kraft–McMillan inequality, which is widely used in information theory, first and foremost, to furnish a necessary and sufficient condition for the existence of a UD code with a given code-length function, and thereby also to prove the converse to the lossless source coding theorem, asserting that no UD source code can yield a coding rate below the entropy rate of the source. Once this necessary and sufficient condition is satisfied, there exists not only a general UD code, but also more specifically, a prefix code with that length function. Beyond its immediate operational meaning, Kraft’s inequality underlies many fundamental principles in lossless compression, such as the equivalence between lossless source coding and probability assignment. In general, its importance stems from the fact that it connects combinatorial properties of codes with analytical bounds in a precise and tractable manner. Classical treatments can be found in standard texts such as [3,4].

When memory is introduced into the encoder, however, the classical Kraft inequality (CKI) no longer applies directly. Finite-state (FS) encoders constitute a natural and widely studied model for compression with memory, arising in universal source coding, individual-sequence coding, and FS prediction. In this setting, the encoder’s output depends not only on the current source symbol, but also on an internal state that evolves over time in a manner that depends on past inputs. As a result, the set of admissible codeword length assignments is no longer characterized by a single scalar inequality, and the extension of Kraft’s condition becomes substantially more subtle.

Significant progress in this direction was made by Ziv and Lempel [5], who derived a generalized Kraft inequality (GKI) for information-lossless (IL) FS encoders by considering blocks over large super-alphabets, see Lemma 2 in [5]. When reading Ziv and Lempel’s article, the reader might get the impression that their GKI was established merely as an auxiliary result needed in their way of proving that the FS compressibility of a sequence is lower bounded by its asymptotic empirical entropy. Their focus was not on the Kraft inequality in its own right. Consequently, their formulation of Kraft’s inequality suffers from two main limitations: (i) it does not reduce exactly to the CKI when the encoder has merely one state, and (ii) it is based on super-alphabet extensions to long blocks rather than being formulated in a single-letter manner, or at the level in which the encoder is defined in the first place. More precisely, while the inequality remains valid even for short block lengths, it yields tight results only asymptotically for long blocks. But nevertheless, a direct, state-level generalization of Kraft’s inequality that mirrors the simplicity and sharpness of the classical result has remained elusive.

In this paper, we present several new forms of GKIs for IL FS encoders. Our approach associates with every given IL FS encoder a nonnegative matrix, termed the Kraft matrix, whose entries are determined by the encoder’s single-symbol output lengths and state transitions. We show that information losslessness imposes a spectral-radius constraint on this matrix, which serves as a natural analog of Kraft’s inequality. Unlike Ziv and Lempel’s GKI mentioned above, this inequality, as well as its several equivalent forms presented herein, reduces exactly to the CKI in the single-state case and avoids the use of super-alphabet extensions.

We then further refine the analysis for irreducible FS encoders, where the Perron–Frobenius theory yields stronger, uniform bounds on matrix powers. These results lead to transparent lower bounds on achievable compression rates for both stochastic sources and individual sequences. In addition, we extend the framework to settings with side information (SI) available at both the encoder and decoder, where the relevant constraint is expressed in terms of the joint spectral radius (JSR) of a finite set of Kraft matrices [6]. This extension clarifies the structural limitations imposed by SI and highlights the role of common sub-invariant vectors. Finally, another extension is associated with lossy source coding in the spirit of those of [7,8,9].

Overall, the proposed framework provides a unified and exact characterization of feasibility conditions for FS encoders, sharpening existing results and offering new tools for the analysis of compression and prediction under finite-memory constraints.

The outline of the remaining part of this article is as follows. In Section 2, we establish notation conventions, define the setting, and provide some background on the GKI of Ziv and Lempel. In Section 3, we present our basic GKI, asserting that the spectral radius of the Kraft matrix must not exceed unity for an IL FS encoder. Stronger and more explicit statements are then provided for irreducible encoders in Section 4. In Section 5, we apply the GKI of Section 4 to obtain converse bounds on compression and prediction of irreducible machines, both in the probabilistic setting and for individual sequences. Finally, in Section 6, we extend the GKI to the case of availability of SI, and in Section 7, we extend it to the lossy case.

2. Notation, Setting and Background

Throughout this paper, scalar random variables (RV’s) will be denoted by capital letters, their sample values will be denoted by the respective lower case letters, and their alphabets will be denoted by the respective calligraphic letters. A similar convention will apply to random vectors and their sample values, which will be denoted with the same symbols superscripted by the dimension. Thus, for example, $X^n$ (n – positive integer) will denote a random n-vector $(X_1,\dots ,X_n)$, and $x^n=(x_1,\dots ,x_n)$ is a specific vector value in ${\mathcal{X}}^n$, the n–th Cartesian power of $\mathcal{X}$, which is the alphabet of each component of $x^n$. For two positive integers, i and j, where $i\le j$, $x_i^j$ and $X_i^j$ will designate segments $(x_i,\dots ,x_j)$ and $(X_i,\dots ,X_j)$, respectively, where for $i=1$, the subscript will be omitted (as above). For $i>j$, $x_i^j$ (or $X_i^j$) will be understood as the null string. An infinite sequence $(x_1,x_2,\dots )$ will be denoted by x. Logarithms and exponents, throughout this paper, will be understood to be taken to the base 2 unless specified otherwise. The indicator function of an event $\mathcal{A}$ will be denoted by $\mathcal{I}\left\{\mathcal{A}\right\}$, i.e., $\mathcal{I}\left\{\mathcal{A}\right\}=1$ if $\mathcal{A}$ occurs and $\mathcal{I}\left\{\mathcal{A}\right\}=0$ if not.

Following the FS encoding model of [5], an FS encoder is defined by the quintuple, $E=(\mathcal{X},\mathcal{Y},\mathcal{Z},f,g)$, whose five ingredients are defined as follows:

• $\mathcal{X}$ is the finite alphabet of each symbol of the source sequence to be compressed. The cardinality of $\mathcal{X}$ will be denoted by $\alpha$.
• $\mathcal{Y}$ is a finite collection of binary variable-length strings, which is allowed to consist of empty string, denoted ‘null’ (whose length is zero);
• $\mathcal{Z}$ is a finite set of s states of the encoder;
• $f:\mathcal{Z}\times \mathcal{X}\to \mathcal{Y}$ is the output function, and
• $g:\mathcal{Z}\times \mathcal{X}\to \mathcal{Z}$ is the next-state function.

Given an infinite source sequence to be compressed, $\mathit{x}=(x_1,x_2,\dots )$, with $x_i\in \mathcal{X}$, the FS encoder E produces an infinite output sequence, $\mathit{y}=(y_1,y_2,\dots )$ with $y_i\in \mathcal{Y}$, forming the compressed bit-stream, while passing through a sequence of states $\mathit{z}=(z_1,z_2,\dots )$ with $z_i\in \mathcal{Z}$, $i=1,2,\dots$. The encoder is governed by the recursive equations:

$$\begin{array}{cccc}\hfill & \hfill y_i& =& f(z_i,x_i),\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill z_{i+1}& =& g(z_i,x_i),\hfill \end{array}$$

(2)

for $i=1,2,\dots$, with a fixed initial state $z_1\in \mathcal{Z}$. If at any step $y_i=null$, this is referred to as idling as no output is generated, but only the state evolves in response to the input. At each time instant i, the encoder emits $L\left(y_i\right)=L\left[f(z_i,x_i)\right]$ bits, and it is understood that $L\left(\mathrm{null}\right)=0$.

Remark 1.

The null string option (which also appears in [5]) is motivated by the wish to allow the encoder to “idle” for certain combinations of inputs and states rather than “enforcing” it to output compressed bits at each and every time instant. This idling option opens the door to having a lot more flexibility and sometimes it is even necessary. For example, even when considering a simple block code as an example of an FS encoder (as will be done in the sequel in Example 1), then, in general, the encoder can output nothing before having read the entire input block. So formally, if the block length is k, then the encoder idles for $k-1$ time instants, and only upon reading the last input symbol, it produces the compressed codeword for that block.

An encoder with s states, henceforth called an s-state encoder, is one for which $\left|\mathcal{Z}\right|=s$. For the sake of simplicity, we adopt a few notation conventions from [5]: given a segment of input symbols $x_i^j$, where i and j are positive integers with $i\le j$, and an initial state $z_i$, we use $f(z_i,x_i^j)$ to denote the corresponding output segment $y_i^j$ produced by E. Similarly, $g(z_i,x_i^j)$ will denote the final state $z_{j+1}$ after processing the inputs $x_i^j$, beginning from state $z_i$. Thus, in response to an input $x^n$, the encoder produces a compressed bit string of length $L\left(y^n\right)=L\left[f(z_1,x^n)\right]={\sum }_{i=1}^{n}L\left[f(z_i,x_i)\right]={\sum }_{i=1}^{n}L\left(y_i\right)$ bits.

Definition 1.

An FS encoder E is called IL if, given any initial state $z_i\in \mathcal{Z}$, any positive integer n, and any input string, $x_i^{i+n}$, the triplet $(z_i,f(z_i,x_i^{i+n}),g(z_i,x_i^{i+n}))$ uniquely determines the corresponding input string $x_i^{i+n}$.

Remark 2.

The IL property can be considered as the FS counterpart of the notion of unique decodability for ordinary single-state codes (that is, codes with no memory). Indeed, every UD code can be viewed as a single-state IL encoder. But in general, an FS code is not necessarily prefix-free, because the codewords emitted at each time instant may depend on the internal state, which carries over additional information. It should be stressed that the IL property (required for each and every i and n) is attributed merely to the encoder, and it has nothing necessary to do with the mode of operation of the decoder. In particular, one may wonder why the final state, $g(z_i,x_i^{i+n})$, plays a role in the ‘reconstruction’ of $x_i^{i+n}$ as defined in Definition 1. The final state is needed because an FS encoder can “carry information forward” in its state instead of emitting it out immediately. Consequently, without knowing the final state, some of the input information may still be stored in the encoder’s memory rather than in the emitted bits. It is easy to see this even in the above-mentioned simple example of a UD block code when viewed as an instance of an FS encoder: one can verify that the IL property holds in this case, and the reconstruction of the input according to Definition 1 indeed requires the final state in general (for details, the reader is referred to the discussion between Equations (15) and (16) in [10]).

In Lemma 2 of [5], Ziv and Lempel presented a GKI for IL FS encoders. It asserts that for every IL encoder with s states and every positive integer ℓ,

$$\sum _{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}{2}^{-{\min }_{z\in \mathcal{Z}}L\left[f(z,x^{\mathsf{\ell }})\right]}\le s^2\left[1+\log \left(1+{\displaystyle \frac{{\alpha }^{\mathsf{\ell }}}{s^2}}\right)\right],$$

(3)

where we remind the reader that $\alpha$ is the alphabet size of the input sequence to be compressed. Ziv and Lempel’s GKI was a perfect tool for their purpose of proving that the compression ratio achieved by an IL FS encoder cannot be smaller than the asymptotic empirical entropy rate (defined in [5]) for any infinite source sequence x. However, when examined for finitely long sequences, and from the perspective of serving as a necessary condition for information losslessness, this inequality suffers from two main weaknesses.

• It does not exactly recover the CKI for the special case, $s=1$, as in that case, the right-hand side (r.h.s.) becomes $1+\log (1+{\alpha }^{\mathsf{\ell }})>1$. Moreover, even if $\mathsf{\ell }=1$, the right-hand side (r.h.s.), which is $1+{\log }_2(1+\alpha )$, is even larger than 2 for every $\alpha \ge 2$. On a related note, a close inspection of the proof of Lemma 2 in [5] reveals that the inequality in Equation (3) is actually a strong inequality (<), in other words, this inequality is always loose.
• It is significant only upon an extension from single symbols into the super-alphabet of ℓ-strings for large ℓ, unlike the ordinary Kraft inequality, which is asserted in the same level that the code is defined. For example, the CKI for a code that is defined in the level of single symbols of $\mathcal{X}$ is asserted in that level, i.e., ${\sum }_{x\in \mathcal{X}}{2}^{-L\left[f\right(x\left)\right]}\le 1$.

Our objective in this work is first and foremost to establish another GKI for IL FS encoders that is free of the above-mentioned drawbacks. In other words, for the case $s=1$, it would recover the traditional Kraft inequality exactly, and it will be posed in the single-letter level without recourse to alphabet extensions. The latter property will enable one to verify relatively easily that this inequality holds in a given situation.

Our first proposed GKI serves as the basis for our subsequent derivations. Having derived it, we then confine attention to the subclass of irreducible IL FS encoders, namely, FS encoders for which every state can be reached from every state in a finite number of steps. For this important subclass of encoders, we provide several alternative formulations of the GKI and provide a stronger upper bound to the growth rate of the Kraft sum as function of the block length. Again, all these forms are smooth extensions of the CKI in the sense that in the special case $s=1$, they degenerate to the CKI. Finally, we consider extensions in two directions (one at a time): the first is the case where SI is available to both encoder and decoder, and the second is the case of lossy compression.

3. The Basic Generalized Kraft Inequality

For a given IL FS encoder E with s states, let us define an $s\times s$ Kraft matrix K, whose $(z,z^{\prime })$ entry is given by

$$K_{z{z}^{\prime }}=\sum _{\{x:g(z,x)=z^{\prime }\}}{2}^{-L\left[f\right(z,x\left)\right]},(z,z^{\prime })\in {\mathcal{Z}}^2,$$

(4)

where the summation over an empty set is understood as zero. Since K is a non-negative matrix, then according to Theorem 8.3.1 in [11], the spectral radius of K, $\rho \left(K\right)$, is an eigenvalue of K. (We remind the reader that the spectral radius is the maximum absolute value (magnitude) of the eigenvalues of a square matrix.)

Our first form of a GKI is the following.

Theorem 1.

For every IL FS encoder,

$$\rho \left(K\right)\le 1.$$

(5)

As can be seen, this GKI has the two desired properties we mentioned above:

• The case $s=1$ obviously recovers the CKI, since in this case, K degenerates to a scalar, which is nothing but the Kraft sum, ${\sum }_{x\in \mathcal{X}}{2}^{-L\left[f\right(x\left)\right]}$, and then Equation (5) asserts that ${\sum }_{x\in \mathcal{X}}{2}^{-L\left[f\right(x\left)\right]}\le 1$, as desired.
• The matrix K is defined in terms of the functions f and g only. These functions are defined in the level of the single symbols and states.

The first property sets the stage of establishing the condition $\rho \left(K\right)\le 1$ as a necessary condition for information losslessness of a given FS encoder, in analogy to the fact that ordinary Kraft inequality is a necessary (and sufficient) condition for the existence of unique decodability in the case $s=1$. Since there is no involvement of summations over super-alphabets of long vectors, this condition is relatively easy to check, similarly as the CKI, which is a necessary condition for the unique decodability property of ordinary lossless source codes.

Proof. 

The proof is in the footsteps of Karush [12]. Let $L_{\max }\stackrel{▵}{=}{\max }_{z,x}L\left[f(z,x)\right]$. For every positive integer ℓ, the $(z,z^{\prime })$ entry of the ℓ-th order power, $K^{\mathsf{\ell }}$, is given by

$$\begin{array}{ccc}\hfill {\left[K^{\mathsf{\ell }}\right]}_{z{z}^{\prime }}& =& \sum _{z_2\in \mathcal{Z}}\sum _{z_3\in \mathcal{Z}}\cdots \sum _{z_{\mathsf{\ell }}\in \mathcal{Z}}\prod _{i=1}^{\mathsf{\ell }}\left(\sum _{\{x_i:g(z_i,x_i)=z_{i+1}\}}{2}^{-L\left[f(z_i,x_i)\right]}\right)\hfill \\ & =& \sum _{\{x^{\mathsf{\ell }}:g(z,x^{\mathsf{\ell }})=z^{\prime }\}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\hfill \\ & =& \sum _{l=0}^{\mathsf{\ell }·L_{\max }}{2}^{-l}·\left|\{x^{\mathsf{\ell }}:L\left[f(z,x^{\mathsf{\ell }})\right]=l,g(z,x^{\mathsf{\ell }})=z^{\prime }\}\right|\hfill \\ & \le & \sum _{l=0}^{\mathsf{\ell }·L_{\max }}{2}^{-l}·2^l\hfill \\ & =& 1+\mathsf{\ell }·L_{\max },\hfill \end{array}$$

(6)

where in the first line, $z_1=z$ and $z_{\mathsf{\ell }+1}=z^{\prime }$, and the inequality is due to the postulated IL property (as z and $z^{\prime }$ are fixed). Alternatively, we can also bound ${\left[K^{\mathsf{\ell }}\right]}_{z{z}^{\prime }}$ by $1+\log (1+{\alpha }^{\mathsf{\ell }})$ using the same considerations as in the proof of Lemma 2 in [5], except that the factor $s^2$ is missing since z and $z^{\prime }$ are fixed. The choice of which is better between these two bounds depends, of course, on $L_{\max }$. In any case, both expressions are essentially linear in ℓ. Continuing with the first bound, it follows that

$$\sum _{z^{\prime }\in \mathcal{S}}{\left[K^{\mathsf{\ell }}\right]}_{z{z}^{\prime }}=\sum _{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\le s(1+\mathsf{\ell }·L_{\max }).$$

(7)

Let ${\mathit{e}}_z$ be a column vector of dimension s whose entries are all zero except the entry corresponding to state z, which is 1, and let 1 denote the all-one column vector of dimension s. Then, Equation (7) can be rewritten as

$${\mathit{e}}_z^{\top }{K}^{\mathsf{\ell }}\mathbf{1}\le s(1+\mathsf{\ell }·L_{\max }).$$

(8)

To prove that $\rho \left(K\right)\le 1$, we proceed by contradiction. Assume conversely, that $\lambda \stackrel{▵}{=}\rho \left(K\right)>1$. Since K has non-negative entries, the Perron–Frobenius theorem (see again Theorem 8.3.1 in [11]) guarantees that the right eigenvector v corresponding to $\lambda$ has non-negative components and at least one strictly positive component. Since $\mathbf{1}={(1,\dots ,1)}^T$ has strictly positive components, there exists a constant $\delta >0$ such that $\mathbf{1}\ge \delta v$ component-wise. Multiplying by $K^{\mathsf{\ell }}$ from the left and using the non-negativity of K, we obtain

$$K^{\mathsf{\ell }}\mathbf{1}\ge \delta K^{\mathsf{\ell }}v=\delta {\lambda }^{\mathsf{\ell }}v.$$

(9)

Taking the z-th component yields

$${\mathit{e}}_z^{\top }{K}^{\mathsf{\ell }}\mathbf{1}\ge \delta {\lambda }^{\mathsf{\ell }}{v}_z.$$

(10)

For any index z with $v_z>0$, the r.h.s. grows exponentially in ℓ since $\lambda >1$, but this contradicts Equation (8) which establishes an upper bound that grows only linearly in ℓ. Therefore the postulate $\rho \left(K\right)>1$ cannot hold true, and we conclude that $\rho \left(K\right)\le 1$, which completes the proof. □

Since $\rho \left(K\right)\le 1$, it is clear that for every natural ℓ, $\rho \left(K^{\mathsf{\ell }}\right)={\left[\rho \left(K\right)\right]}^{\mathsf{\ell }}\le \rho \left(K\right)\le 1$. In other words, the spectral radius of

$$K^{\mathsf{\ell }}={\left\{\sum _{\{x^{\mathsf{\ell }}:g(z,x^{\mathsf{\ell }})=z^{\prime }\}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\right\}}_{z,z^{\prime }\in \mathcal{Z}}$$

(11)

is also never larger than unity, which is an extension of our GKI to super-alphabets, which is again, a smooth extension that degenerates to the CKI for $s=1$.

Example 1.

Consider a binary source sequence and a block code of length 2, which maps the source strings 00, 01, 10, and 11, into 0, 10, 110, and 111, respectively. This code can be implemented by an FS encoder with $s=3$ states, labeled ‘S’, ‘O’, and ‘I’, using the following functions, f and g (see also Figure 1):

$$\begin{array}{ccc}\hfill g(S,0)& =& O,\hfill \\ \hfill g(S,1)& =& I,\hfill \\ \hfill g(O,0)& =& g(O,1)=g(I,0)=g(I,1)=S,\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill f(S,0)& =& null,\hfill \\ \hfill f(S,1)& =& 11,\hfill \\ \hfill f(O,0)& =& 0,\hfill \\ \hfill f(O,1)& =& 10,\hfill \\ \hfill f(I,0)& =& 0,\hfill \\ \hfill f(I,1)& =& 1.\hfill \end{array}$$

State ‘S’ designates the start of a block. State ‘O’ remembers that the first input of the block was ‘0’ and state ‘I’ remembers that the first input was ‘1’. Upon moving to state ‘I’, the encoder can already output ‘11’, because the entire codeword will be either ‘110’ or ‘111’ if the first source symbol is ‘1’, so the first two coded bits are ‘11’ in either case. After state ‘I’, the encoder can complete the codeword according to the second input in the block. After state ‘O’, outputs are generated only upon receiving the second symbol. After both states ‘O’ and ‘I’, the encoder must return to state ‘S’ in order to start the next block. The corresponding Kraft matrix (with row and column indexing in the order of (S,O,I)) is given by:

$$K=\left(\begin{array}{ccc}0& 2^{-0}& 2^{-2}\\ 2^{-1}+2^{-2}& 0& 0\\ 2^{-1}+2^{-1}& 0& 0\end{array}\right)=\left(\begin{array}{ccc}0& 1& 0.25\\ 0.75& 0& 0\\ 1& 0& 0\end{array}\right)$$

(12)

whose eigenvalues are 1, 0, and $-1$, and so the spectral radius is $\rho \left(K\right)=1$. As can be seen, the sums of the second and third rows do not exceed unity, so when the initial state is either ‘O’ or ‘I’, the Kraft sum does not exceed 1. On the other hand, the Kraft sum corresponding to the first row (pertaining to ‘S’) exceeds unity. This demonstrates an important observation: the model of a general IL FS encoder is broader and more general than a model of an FS encoder for which given every state, the encoder implements a certain prefix (or UD) code for the variety of incoming symbols. For $\mathsf{\ell }=100$, we find that

$$K^{100}=\left(\begin{array}{ccc}1& 0& 0\\ 0& 0.75& 0.1875\\ 0& 1& 0.25\end{array}\right)$$

where eigenvalues are 0, 1, and 1. Here, the sums of the first and the second rows do not exceed unity, so when the initial state is either ‘S’ or ‘O’, the Kraft sum does not exceed 1. On the other hand, the Kraft sum corresponding to the third row exceeds unity, and so, the above comment with regard to K applies here too. This concludes Example 1.

Figure 1. State transition diagram of the encoder in Example 1. The various state transitions are labeled in a form $x/y$, where x denotes the input and $y=f(z,x)$ denotes the output.

Figure 1. State transition diagram of the encoder in Example 1. The various state transitions are labeled in a form $x/y$, where x denotes the input and $y=f(z,x)$ denotes the output.

Earlier, we said that $\rho \left(K\right)\le 1$ is a necessary condition for a given code with next-state function g and code-lengths $\left\{L\right[f(z,x)\left]\right\}$ to be IL. One might naturally wonder whether it is also a sufficient condition. This question is open in general, but we have two comments related to this issue.

The first is that the answer is obviously affirmative for the subclass of IL encoders, which satisfies the CKI for each and every state, i.e., ${\sum }_{z^{\prime }\in \mathcal{Z}}{K}_{z{z}^{\prime }}={\sum }_{x\in \mathcal{X}}{2}^{-L\left[f\right(z,x\left)\right]}\le 1$: Simply construct a separate prefix code with length function $\left\{L\right[f(z,x)],x\in \mathcal{X}\}$ for each $z\in \mathcal{Z}$. However, in general, an IL code does not necessarily satisfy the ordinary Kraft inequality for each z. Indeed, in Example 1, the sum of the first row of K is larger than 1.

The second comment is that we can give an affirmative answer in the level of longer blocks. Let $z_1\in \mathcal{Z}$ be an arbitrary initial state and consider the lengths, $l(z_1,x^n)={\sum }_{i=1}^{n}L\left[f(z_i,x_i)\right]$. Then, as we have seen in (6):

$$\sum _{x^n}{2}^{-L\left[f(z_1,x^n)\right]}\le s(1+n{L}_{\max }),$$

(13)

where the factor of s stems from taking the sum of $K_{z{z}^{\prime }}$ over $z^{\prime }\in \mathcal{Z}$. Equivalently,

$$\sum _{x^n}{2}^{-[l\left[f(z_1,x^n)\right]+\log \left[s(1+n{L}_{\max })\right]}\le 1,$$

(14)

and so, there exists a prefix code with lengths $l^{\prime }\left(x^n\right)=l(z_1,x^n)+\log \left[s(1+n{L}_{\max })\right]+\log s$, which are relatively only slightly longer than those of the original code. Here, the additional $\log s$ term is a header that notifies $z_1$.

4. Irreducible FS Encoders

IL FS encoders for which the next-state function g allows transition from every state to every state within a finite number of steps, are henceforth referred to as irreducible FS encoders. Equivalently, defining the $s\times s$ adjacency matrix A such that $A_{z{z}^{\prime }}=1$ whenever $\exists x\in \mathcal{X}$ such that $g(z,x)=z^{\prime }$ and $A_{z{z}^{\prime }}=0$ otherwise, then an IL FS encoder is irreducible if the matrix A is irreducible. Likewise, an IL FS encoder is irreducible if the matrix K is irreducible. For an irreducible FS encoder, the shortest path from every state z to every other state $z^{\prime }$ lasts no longer than $s-1$ steps, because any longer path must visit a certain state $z^{″}$ at least twice, meaning that this path contains a loop starting and ending at $z^{″}$, which can be eliminated. Clearly, the encoder of Example 1 is irreducible.

Intuitively, it makes sense to use irreducible encoders, because for reducible ones, once the machine leaves a certain subset of transient states, it can never return, and so, effectively, reducible encoders use eventually a smaller number of states after finite time. Specifically, given a reducible machine and an infinite individual sequence $x_1,x_2,\dots$, suppose the machine starts at a transient state. Then, there are two possibilities: either the machine quits the subset of transient states after finite time, or it stays in that subset forever. In the former case, the transient states are in use for finite time only and then never used again. In the latter case, the recurrent states are never used. In either case, asymptotically, only a subset of the available states are used, and so, effectively, the number of states actually used is smaller than s. Let ${\mathcal{Z}}_{\infty }$ denote the set of states visited infinitely many times along the sequence. This set is necessarily closed and induces a strongly connected subgraph. Consequently, the asymptotic behavior of the encoder along the given sequence is governed entirely by its restriction to ${\mathcal{Z}}_{\infty }$, which constitutes an irreducible FS encoder with strictly fewer than s states. Therefore, reducible encoders cannot offer asymptotic advantages over irreducible ones, even for individual sequences.

Assume next that the next-state function g induces an irreducible matrix $K^{\mathsf{\ell }}$, where ℓ be an arbitrary positive integer. Since $K^{\mathsf{\ell }}$ is non-negative and irreducible, the Collatz–Wielandt formulas [13,14] for the spectral radius of $K^{\mathsf{\ell }}$ hold true. These are given by

$$\rho \left(K^{\mathsf{\ell }}\right)=\underset{w\in {\mathcal{W}}_{+}}{\max }\underset{\{z:w_z>0\}}{\min }{\displaystyle \frac{{\left[K^{\mathsf{\ell }}w\right]}_z}{w_z}}=\underset{w\in {\mathcal{W}}_{+}}{\min }\underset{\{z:w_z>0\}}{\max }{\displaystyle \frac{{\left[K^{\mathsf{\ell }}w\right]}_z}{w_z}}.$$

(15)

where w is an s-dimensional column vector and ${\mathcal{W}}_{+}$ is the set of all such vectors with non-negative components not all of which are zero. These lead to the two following GKIs:

$$\forall w\in {\mathcal{W}}_{+}\exists z\mathrm{such} \mathrm{that}{w}_z>0\mathrm{and}\sum _{z^{\prime }\in \mathcal{S}}{w}_{z^{\prime }}·\sum _{\{x^{\mathsf{\ell }}:g(z,x^{\mathsf{\ell }})=z^{\prime }\}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\le w_z,$$

(16)

and

$$\exists w\in {\mathcal{W}}_{+}\forall z\mathrm{such} \mathrm{that}{w}_z>0:\sum _{z^{\prime }\in \mathcal{S}}{w}_{z^{\prime }}·\sum _{\{x^{\mathsf{\ell }}:g(z,x^{\mathsf{\ell }})=z^{\prime }\}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\le w_z,$$

(17)

The first formulation can be simplified at the price of a possible loss of tightness, by selecting w to be the all-one vector and thereby bounding $\rho \left(K^{\mathsf{\ell }}\right)$ from below. This results in the conclusion that an IL FS encoder always satisfies yet another GKI:

$$\exists z\in \mathcal{S}\sum _{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\le 1.$$

(18)

In words, for every given irreducible FS encoder, $(f,g)$, and for every natural ℓ, there is at least one initial state, $z\in \mathcal{Z}$, for which the Kraft sum is less than unity, but again, not all states must satisfy this condition (as we saw in Example 1, the Kraft sum exceeds unity when the initial state is ‘S’). All these are also smooth extensions of the CKI in the sense that for $s=1$ we are back to the CKI.

But there is an even stronger GKI that applies to irreducible encoders. It asserts that in the irreducible case, $K^n$ does not even grow linearly as in (6), but is rather bounded by a constant, independent of n. For $s=1$, this constant is 1, again in agreement with the CKI.

Theorem 2.

Let K be an irreducible Kraft matrix. Then, for all $z,z^{\prime }\in \mathcal{Z}$ and for every natural n,

$${\left(K^n\right)}_{z{z}^{\prime }}=\sum _{\{x^n:g(z,x^n)=z^{\prime }\}}{2}^{-L\left[f(z,x^n)\right]}\le 2^{(s-1)L_{\max }}.$$

(19)

Consequently, for every $z\in \mathcal{S}$,

$$\sum _{x^n\in {\mathcal{X}}^n}{2}^{-L\left[f(z,x^n)\right]}\le s·2^{(s-1)L_{\max }},$$

(20)

and

$$\sum _{z\in \mathcal{S}}\sum _{x^n\in {\mathcal{X}}^n}{2}^{-L\left[f(z,x^n)\right]}\le s^2·2^{(s-1)L_{\max }}.$$

(21)

Proof. 

It is sufficient to prove the first inequality, as the two other ones will follow trivially by a summation over $z^{\prime }\in \mathcal{Z}$ and then also over $z\in \mathcal{Z}$, respectively. Since K is non-negative and irreducible, the Perron–Frobenius theorem applies. This theorem asserts that the spectral radius, $\rho \left(K\right)$, is positive and simple, with left and right eigenvectors, u and v, respectively, that have only strictly positive components. In Theorem 1 we have already proved that $\rho \left(K\right)\le 1$. Assume first that $\rho \left(K\right)=1$. Then, $u^T{K}^n=u^T$, or, equivalently,

$$\sum _{z\in \mathcal{S}}{u}_z{\left(K^n\right)}_{z{z}^{\prime }}=u_{z^{\prime }}\forall z^{\prime }\in \mathcal{Z}.$$

(22)

Since all terms are non-negative, the left-hand side is lower bound by $u_z{\left(K^n\right)}_{z{z}^{\prime }}$ for any $z\in \mathcal{Z}$. This implies for every $z,z^{\prime }\in \mathcal{Z}$

$${\left(K^n\right)}_{z{z}^{\prime }}\le {\displaystyle \frac{u_{z^{\prime }}}{u_z}}\le {\displaystyle \frac{{\max }_{z\in \mathcal{Z}}{u}_z}{{\min }_{z\in \mathcal{Z}}{u}_z}}.$$

(23)

Let $z_{\star }\in \mathcal{Z}$ and $z^{\star }\in \mathcal{Z}$ be achievers of ${\min }_{z\in \mathcal{Z}}{u}_z$ and ${\max }_{z\in \mathcal{Z}}{u}_z$, respectively. Then, for every $z,z^{\prime }\in \mathcal{Z}$,

$${\left(K^n\right)}_{z{z}^{\prime }}\le {\displaystyle \frac{u_{z^{\star }}}{u_{z_{\star }}}}.$$

(24)

Since K is irreducible and since $z^{\star }$ and $z_{\star }$ are distinct, there exists a path of length $\mathsf{\ell }\le s-1$ from $z^{\star }$ to $z_{\star }$, say, $z^{\star }\to z_1\to \cdots \to z_{\mathsf{\ell }-1}\to z_{\star }$ such that

$${\left(K^{\mathsf{\ell }}\right)}_{z^{\star }{z}_{\star }}\ge K_{z^{\star }{z}_1}·K_{z_1{z}_2}\cdots K_{z_{\mathsf{\ell }-1}{z}_{\star }}>0.$$

(25)

Since all positive entries of K are at least as large as $2^{-L_{\max }}$, this product is at least as large as $2^{-\mathsf{\ell }{L}_{\max }}\ge 2^{-(s-1)L_{\max }}$. It follows then that

$${\left(K^{\mathsf{\ell }}\right)}_{z^{\star }{z}_{\star }}\ge 2^{-(s-1)L_{\max }}.$$

(26)

Now,

$$u_{z_{\star }}=\sum _{z\in \mathcal{S}}{u}_z{\left(K^{\mathsf{\ell }}\right)}_{z{z}_{\star }}\ge u_{z^{\star }}{\left(K^{\mathsf{\ell }}\right)}_{z^{\star }{z}_{\star }}\ge u_{z^{\star }}{2}^{-(s-1)L_{\max }},$$

(27)

which implies that

$$2^{(s-1)L_{\max }}\ge {\displaystyle \frac{u_{z^{\star }}}{u_{z_{\star }}}}\ge {\left(K^n\right)}_{z{z}^{\prime }},$$

(28)

for every $z,z^{\prime }\in \mathcal{S}$. This completes the proof for the case $\rho \left(K\right)=1$. The case $\rho \left(K\right)<1$ is obtained from the case $\rho \left(K\right)=1$ by simply defining $\widehat{K}=K/\rho \left(K\right)$ and using the fact that all non-negative entries of $\widehat{K}$ are lower bounded by $2^{-L_{\max }}/\rho \left(K\right)$. Since $\widehat{K}$ is also irreducible and since $\rho \left(\widehat{K}\right)=1$, we now have

$${\left({\widehat{K}}^n\right)}_{z{z}^{\prime }}\le {\left[\rho \left(K\right)2^{L_{\max }}\right]}^{s-1}.$$

(29)

But ${\widehat{K}}^n=K^n/{\left[\rho \left(K\right)\right]}^n$, and so,

$${\left(K^n\right)}_{z{z}^{\prime }}\le {\left[\rho \left(K\right)\right]}^{n+s-1}·2^{(s-1)L_{\max }}<2^{(s-1)L_{\max }}.$$

(30)

This completes the proof of Theorem 2. □

5. Converse Bounds Derived from the GKI

In this section, we demonstrate how the GKI of Section 4 can be used to obtain lower bounds on the performance of irreducible machines in compression and in prediction problems. For compression, both probabilistic sources and individual sequences are considered. For prediction, only the individual sequence version is presented, but the probabilistic counterpart can also be derived straightforwardly using the same ideas.

5.1. Compression of Probabilistic Sources

Let $\{P(z,x^{\mathsf{\ell }}),z\in \mathcal{Z},x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}\}$ be a joint probability distribution of random variables Z and $X^{\mathsf{\ell }}$. Then,

$$\begin{array}{ccc}\hfill s^2·2^{(s-1)L_{\max }}& \ge & \sum _{z\in \mathcal{Z}}\sum _{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}{2}^{-L\left[f(z,x^{\mathsf{\ell }})\right]}\hfill \\ & =& \sum _{z\in \mathcal{Z}}\sum _{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}P(z,x^{\mathsf{\ell }})·2^{-L\left[f(z,x^{\mathsf{\ell }})\right]-\log P(z,x^{\mathsf{\ell }})}\hfill \\ & \ge & {\exp }_2\left\{-\sum _{z\in \mathcal{Z}}\sum _{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}P(z,x^{\mathsf{\ell }})L\left[f(z,x^{\mathsf{\ell }})\right]+H(Z,X^{\mathsf{\ell }})\right\}\hfill \\ & =& {\exp }_2\left[-\mathbb{E}\left\{L\left[f(Z,X^{\mathsf{\ell }})\right]\right\}+H(Z,X^{\mathsf{\ell }})\right],\hfill \end{array}$$

(31)

where the inequality follows from Jensen’s inequality and the convexity of the exponential function. By taking logarithms of both sides, rearranging terms, and normalizing by ℓ, we get

$$\begin{array}{ccc}\hfill R& =& {\displaystyle \frac{\mathbb{E}\left\{L\left[f(Z,X^{\mathsf{\ell }})\right]\right\}}{\mathsf{\ell }}}\hfill \\ & \ge & {\displaystyle \frac{H(Z,X^{\mathsf{\ell }})}{\mathsf{\ell }}}-{\displaystyle \frac{{\log }_2\left(s^2·2^{(s-1)L_{\max }}\right)}{\mathsf{\ell }}}\hfill \\ & \ge & {\displaystyle \frac{H\left(X^{\mathsf{\ell }}\right)}{\mathsf{\ell }}}-{\displaystyle \frac{2{\log }_{2}s+(s-1)L_{\max }}{\mathsf{\ell }}},\hfill \end{array}$$

(32)

and if the source P is stationary, $H\left(X^{\mathsf{\ell }}\right)/\mathsf{\ell }$ can be further lower bounded by $H\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})$, to obtain

$$R\ge H\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2{\log }_{2}s+(s-1)L_{\max }}{\mathsf{\ell }}}.$$

(33)

Since this bound applies to every positive integer ℓ, we may maximize the lower bound over ℓ, and obtain

$$R\ge \underset{\mathsf{\ell }\ge 1}{\sup }\left\{H\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2{\log }_{2}s+(s-1)L_{\max }}{\mathsf{\ell }}}\right\}.$$

(34)

We see that thanks to Theorem 2, the vanishing term subtracted from the entropy decays at the rate of $1/\mathsf{\ell }$ as opposed to the $(\log \mathsf{\ell })/\mathsf{\ell }$ rate that stems from Lemma 2 of [5] as well as from the more general inequality of $1+\mathsf{\ell }·L_{\max }$, that is obtained when reducible machines are allowed.

5.2. Compression of Individual Sequences

In the context of individual sequences, we can arrive at an analogous lower bound, provided that we define a shift-invariant empirical distribution. Specifically, let $x^n$ be a given individual sequence, let ℓ be a positive integer smaller than n, and let $z_1$ be a given initial state of the encoder. We assume that $x^n$ cyclic with respect to (w.r.t.) g in the sense that $g(z_n,x_n)=z_1$. If this is not the case, consider an extension of $x^n$ by concatenating a suffix $x_{n+1}^{n+m}$ such that the extended sequence would be cyclic w.r.t. g. Since g is assumed irreducible, this is always possible and the length m of the extension need not be larger than $s-1$. To avoid cumbersome notation, we redefine $x^n$ to be the sequence after the cyclic extension (if needed), and we shall keep in mind that this cyclic extension adds no more than $m·L_{\max }\le (s-1)L_{\max }$ bits to the compressed description, or equivalently, $(s-1)L_{\max }/n$ to the compression ratio, and so, this extra rate should be subtracted back upon returning to the original sequence before the cyclic extension. For every $w^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}$ and $z\in \mathcal{S}$, let

$$\delta (z_i,x_i^{\left(\right(i-1)\oplus (\mathsf{\ell }-1\left)\right)+1};z,w^{\mathsf{\ell }})=\left\{\begin{array}{cc}1\hfill & z_i=z\mathrm{and}{x}_i^{\left(\right(i-1)\oplus (\mathsf{\ell }-1\left)\right)+1}=w^{\mathsf{\ell }}\hfill \\ 0\hfill & \mathrm{elsewhere}\hfill \end{array}\right.$$

(35)

where ⊕ denotes modulo-n addition. Next, define the empirical distribution

$$\widehat{P}(z,w^{\mathsf{\ell }})={\displaystyle \frac{1}{n}}\sum _{i=1}^n\delta (z_i,x_i^{\left(\right(i-1)\oplus (\mathsf{\ell }-1\left)\right)+1};z,w^{\mathsf{\ell }}),$$

(36)

Now,

$$\begin{array}{ccc}\hfill {\displaystyle \frac{1}{n}}\sum _{i=1}^{n}L\left[f(z_i,x_i)\right]& =& {\displaystyle \frac{1}{n\mathsf{\ell }}}\sum _{i=1}^n\mathsf{\ell }·L\left[f(z_i,x_i)\right]\hfill \\ & =& {\displaystyle \frac{1}{n\mathsf{\ell }}}\sum _{i=1}^n\sum _{j=0}^{\mathsf{\ell }-1}L\left[f(z_i,x_{\left(\right(i-1)\oplus j)+1})\right]\hfill \\ & =& {\displaystyle \frac{1}{n\mathsf{\ell }}}\sum _{i=1}^{n}L\left[f(z_i,x_i^{\left(\right(i-1)\oplus (\mathsf{\ell }-1\left)\right)+1})\right]\hfill \\ & =& {\displaystyle \frac{1}{n\mathsf{\ell }}}\sum _{i=1}^n\sum _{z\in \mathcal{S}}\sum _{w^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}\delta (z_i,x_i^{\left(\right(i-1)\oplus (\mathsf{\ell }-1\left)\right)+1};z,w^{\mathsf{\ell }})L\left[f(z,w^{\mathsf{\ell }})\right]\hfill \\ & =& {\displaystyle \frac{1}{n\mathsf{\ell }}}\sum _{z\in \mathcal{S}}\sum _{w^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}\sum _{i=1}^n\delta (z_i,x_i^{\left(\right(i-1)\oplus (\mathsf{\ell }-1\left)\right)+1};z,w^{\mathsf{\ell }})L\left[f(z,w^{\mathsf{\ell }})\right]\hfill \\ & =& {\displaystyle \frac{1}{\mathsf{\ell }}}\sum _{z\in \mathcal{S}}\sum _{w^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}}\widehat{P}(z,w^{\mathsf{\ell }})L\left[f(z,w^{\mathsf{\ell }})\right]\hfill \\ & \ge & \widehat{H}\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2{\log }_{2}s+(s-1)L_{\max }}{\mathsf{\ell }}},\hfill \end{array}$$

(37)

where $\widehat{H}\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})$ is the empirical conditional entropy derived from the shift-invariant distribution $\widehat{P}$. The last inequality follows from a similar derivation as in the probabilistic case considered above, except that the earlier distribution P is now replaced by the empirical one, $\widehat{P}$, and therefore the corresponding entropies are replaced by their empirical counterparts. Using the fact that this is true for every natural $\mathsf{\ell }<n$ and returning to the original sequence before the cyclic extension, we find that

$${\displaystyle \frac{1}{n}}\sum _{i=1}^{n}L\left[f(z_i,x_i)\right]\ge \underset{1\le \mathsf{\ell }<n}{\max }\left\{\widehat{H}\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2\log s+(s-1)L_{\max }}{\mathsf{\ell }}}\right\}-{\displaystyle \frac{(s-1)L_{\max }}{n}}.$$

(38)

Furthermore, invoking Ziv’s inequality (see Equation (13.125) in [4]), this can be further lower bounded in terms of the LZ complexity. Specifically, according to Equation (13.125) in [4], for every Markov source, $Q_{\mathsf{\ell }-1}$, of order $\mathsf{\ell }-1$ and every $x^n\in {\mathcal{X}}^n$,

$$c\left(x^n\right)\log c\left(x^n\right)\le -\log Q_{\mathsf{\ell }-1}\left(x^n\right|x_{-(\mathsf{\ell }-2)}^0)+{ϵ}_{\mathsf{\ell }}\left(n\right),$$

(39)

where $c\left(x^n\right)$ is the maximum number of distinct phrases whose concatenation forms $x^n$, and where ${ϵ}_{\mathsf{\ell }}\left(n\right)$ tends to zero at the rate of $O\left(\log \right(\log n)/\log n)$ for every fixed ℓ. By minimizing the r.h.s. w.r.t. $Q_{\mathsf{\ell }-1}$, we get

$$c\left(x^n\right)\log c\left(x^n\right)\le n\widehat{H}\left(X_{\mathsf{\ell }}\right|X_0^{\mathsf{\ell }-1})+n·{ϵ}_{\mathsf{\ell }}\left(n\right),$$

(40)

and so,

$${\displaystyle \frac{1}{n}}\sum _{i=1}^{n}L\left[f(z_i,x_i)\right]\ge {\displaystyle \frac{c\left(x^n\right)\log c\left(x^n\right)}{n}}-\underset{\mathsf{\ell }}{\min }\left[{ϵ}_{\mathsf{\ell }}\left(n\right)+{\displaystyle \frac{2\log s+(s-1)L_{\max }}{\mathsf{\ell }}}\right]-{\displaystyle \frac{(s-1)L_{\max }}{n}}.$$

(41)

The minimizing ℓ can be found to be proportional to $\sqrt{n}$, but the dominant term of ${ϵ}_{\mathsf{\ell }}\left(n\right)$ remains of the order of ${\displaystyle \frac{\log \left(\log n\right)}{\log n}}$.

5.3. Prediction of Individual Sequences

We next derive a lower bound to the prediction error of any FS predictor that is based on an irreducible FS machine. The idea is to harvest the compression lower bound to induce a lower bound on prediction by considering an FS encoder that is based on FS prediction and encoding the prediction error (predictive coding)—see Figure 2.

Consider an FS predictor with q states, defined by the following recursion, for $i=1,2,\dots$

$$\begin{array}{ccc}\hfill {\widehat{x}}_{i+1}& =& u(x_i,{\sigma }_i),\hfill \\ \hfill {\sigma }_{i+1}& =& v(x_i,{\sigma }_i),\hfill \end{array}$$

(42)

where $\mathit{\sigma }=({\sigma }_1,{\sigma }_2,\dots )$, ${\sigma }_i\in \Sigma$, $i=1,2,\dots$, is a corresponding infinite state sequence, whose alphabet, $\Sigma$, is a finite set of states of cardinality q, and $\widehat{\mathit{x}}=({\widehat{x}}_1,{\widehat{x}}_2,\dots )$, ${\widehat{x}}_i\in \mathcal{X}$, $i=1,2,\dots$, is the resulting predictor output sequence. Without loss of generality, the initial state, ${\sigma }_1$, and the initial prediction, ${\widehat{x}}_1$, are assumed fixed members, ${\sigma }_{\star }\in \Sigma$ and ${\widehat{x}}_{\star }\in \mathcal{X}$, respectively. Here, $u:\mathcal{X}\times \Sigma \to \mathcal{X}$ is the predictor output function and $v:\mathcal{X}\times \Sigma \to \Sigma$ is the next-state function.

It is assumed that $\mathcal{X}$ is a group with well-defined addition and subtraction operations. For example, if $\mathcal{X}=\{0,1,\dots ,\alpha -1\}$ then it is natural to equip $\mathcal{X}$ with addition and subtraction modulo $\alpha$. Let $\rho :\mathcal{X}\to {\mathbb{R}}^{+}$ denote a given loss function. Then, the performance of a predictor across the time range, $1\le t\le n$ is measured in terms of the time-average,

$${\displaystyle \frac{1}{n}}\sum _{i=1}^n\rho (x_i-{\widehat{x}}_i).$$

(43)

Given an arbitrary irreducible FS predictor $(u,v)$ as defined above, consider the auxiliary conditional probability distribution,

$$Q_{\theta }\left(x_{i+1}\right|x_i,{\sigma }_i)={\displaystyle \frac{e^{-\rho (x_{i+1}-u(x_i,{\sigma }_i))/\theta }}{Z\left(\theta \right)}},\theta \ge 0,$$

(44)

where

$$Z\left(\theta \right)=\sum _{x\in \mathcal{X}}{e}^{-\rho \left(x\right)/\theta }.$$

(45)

Define also the function

$$\Delta \left(R\right)=\underset{\theta \ge 0}{\sup }\theta ·[R-\log Z\left(\theta \right)],R\ge 0.$$

(46)

Now, define

$$Q_{\theta }\left(x^n\right)=\prod _{i=1}^n{Q}_{\theta }\left(x_i\right|x_{i-1},{\sigma }_{i-1})$$

(47)

where ${\sigma }_0$ and $x_0$ are arbitrary members of $\Sigma$ and $\mathcal{X}$, respectively, such that ${\sigma }_1=v(x_0,{\sigma }_0)=s_{\star }$, and ${\sigma }_2,{\sigma }_3,\dots ,{\sigma }_{n-1}$ are generated from $x_1,x_2,\dots ,x_{n-1}$ as in (42).

Let k divide n and consider the lossless compression of $x_0^{n-1}$ in blocks of length k, ${\mathit{x}}_j=x_{jk+1}^{jk+k}$, $j=0,1,\dots ,n/k-1$, by using the Shannon code, whose length function for a vector $x^k$ is $\lceil -\log Q_{\theta }\left(x^k\right)\rceil$. This is equivalent to predictive coding, where the prediction error signal, $z_n=x_n-f(x_{n-1},s_{n-1})$ is compressed losslessly under a model of a memoryless source with a marginal $Q_{\theta }\left(z\right)$ (see Figure 2 for illustration). In this case, since the ceiling operation is carried over k-blocks, and there are $n/k$ such k-blocks, the upper bound to $L\left(x^n\right)$ becomes

$$L\left(x^n\right)=\sum _{i=0}^{n/k-1}\lceil -\log Q_{\theta }\left(x_{ik+1}^{ik+k}\right)\rceil \le {\displaystyle \frac{1}{\theta }}·\sum _{i=1}^n\rho (x_i-u(x_{i-1},{\sigma }_{i-1}))+n\log Z\left(\theta \right)+{\displaystyle \frac{n}{k}}.$$

(48)

On the other hand, the corresponding encoder of Figure 2 can be viewed as an encoder with $q·M_k$ states, where $M_k=({\alpha }^k-1)/(\alpha -1)$, since this is the number of combinations of a state of the q-state predictor and a state of the lossless block encoder, whose number of states is ${\sum }_{j=0}^{k-1}{\alpha }^j=M_k$. Thus,

$${\displaystyle \frac{L\left(x^n\right)}{n}}\ge \widehat{H}\left(X^{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2\log \left(q{M}_k\right)+(q{M}_k-1)L_{\max }}{\mathsf{\ell }}}-{\displaystyle \frac{L_{\max }}{n}}$$

(49)

where it should be kept in mind that $L_{\max }$ is expected to grow linearly with k. Thus, by comparing the upper bound and the lower bound to $L\left(x^n\right)$, we have

$$\begin{array}{ccc}& & {\displaystyle \frac{1}{n\theta }}·\sum _{i=1}^n\rho (x_i-u(x_{i-1},{\sigma }_{i-1}))+\log Z\left(\theta \right)+{\displaystyle \frac{1}{k}}\hfill \\ & \ge & \widehat{H}\left(X^{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2\log \left(q{M}_k\right)+(q{M}_k-1)L_{\max }}{\mathsf{\ell }}}-{\displaystyle \frac{L_{\max }}{n}}.\hfill \end{array}$$

(50)

or, equivalently,

$$\begin{array}{ccc}& & {\displaystyle \frac{1}{n}}\sum _{i=1}^n\rho (x_i-u(x_{i-1},{\sigma }_{i-1}))\hfill \\ & \ge & \theta \left[\widehat{H}\left(X^{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2\log \left(q{M}_k\right)+(q{M}_k-1)L_{\max }}{\mathsf{\ell }}}-{\displaystyle \frac{L_{\max }}{n}}-{\displaystyle \frac{1}{k}}-\log Z\left(\theta \right)\right].\hfill \end{array}$$

(51)

Maximizing the r.h.s. over $\theta \ge 0$, we get

$${\displaystyle \frac{1}{n}}\sum _{i=1}^n\rho (x_{i+1}-u(x_i,{\sigma }_i))\ge \Delta \left(\widehat{H}\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})-{\displaystyle \frac{2\log \left(q{M}_k\right)+(q{M}_k-1)L_{\max }}{\mathsf{\ell }}}-{\displaystyle \frac{L_{\max }}{n}}-{\displaystyle \frac{1}{k}}\right).$$

(52)

The bound is meaningful if $k\gg 1$ and $\mathsf{\ell }\gg q{M}_k$, so that the two subtracted terms in the argument of the function $\Delta (·)$ are small compared to the main term, $\widehat{H}\left(X_{\mathsf{\ell }}\right|X^{\mathsf{\ell }-1})$. It is tight essentially for sequences of the form $x_i=u(x_{i-1},{\sigma }_{i-1})+z_i$, $i=1,2,\dots$, where $z^n=(z_1,\dots ,z_n)$ is typical to an i.i.d. source and where the marginal empirical distribution of each $z_i$ is close to $e^{-\rho \left(z\right)/\theta }/Z\left(\theta \right)$ for some $\theta \ge 0$.

6. GKI in the Presence of Side Information

We will now discuss briefly an extension of the GKI for IL FS encoders in the case where SI is available at both the encoder and the decoder. The resulting condition is expressed in terms of the joint spectral radius (JSR) of a finite set of nonnegative matrices indexed by the various side-information symbols. We identify verifiable sufficient conditions for subexponential growth of Kraft sums and discuss the limitations inherent in the presence of SI.

Let $\mathcal{X}$ be the source alphabet as before and let $\mathcal{W}$ denote the finite alphabet of the SI sequence, $w_1,w_2,\dots$, whose symbols are synchronized with the corresponding source symbols. As before, let $\mathcal{Z}$ be the finite set of states with $\left|\mathcal{Z}\right|=s$. An FS encoder with SI is specified by an output function $f:\mathcal{Z}\times \mathcal{X}\times \mathcal{W}\to \mathcal{Y}$, ($\mathcal{Y}$ being defined as a subset of ${\{0,1\}}^{*}$, similarly as before) and a next-state function $g:\mathcal{Z}\times \mathcal{X}\times \mathcal{W}\to \mathcal{Z}$. Given an initial state, $z_1=z$, a source sequence, $\mathit{x}=(x_1,x_2,\dots )$, and a SI sequence, $\mathit{w}=(w_1,w_2,\dots )$, the encoder implements the equations:

$$\begin{array}{ccc}\hfill y_i& =& f(z_i,x_i,w_i),\hfill \\ \hfill z_{i+1}& =& g(z_i,x_i,w_i),\hfill \end{array}$$

(53)

for $i=1,2,\dots$, and the total code-length produced by the encoder after n steps is

$$L\left[f(z,x^n,w^n)\right]=\sum _{i=1}^{n}L\left(f(z_i,x_i,w_i)\right).$$

(54)

Definition 2.

An FS encoder is said to be information-lossless with side information if for every n, the quadruple $(z_1,y^n,w^n,z_{n+1})\in \mathcal{Z}\times {\mathcal{Y}}^n\times {\mathcal{W}}^n\times \mathcal{Z}$ dictates $x^n\in {\mathcal{X}}^n$.

For each SI symbol, $w\in \mathcal{W}$, define the corresponding Kraft matrix

$${\left[K\left(w\right)\right]}_{z{z}^{\prime }}=\sum _{\{x\in \mathcal{X}:g(z,x,w)=z^{\prime }\}}{2}^{-L\left[f\right(z,x,w\left)\right]},z,z^{\prime }\in \mathcal{Z}.$$

(55)

Each $K\left(w\right)$ is a nonnegative $s\times s$ matrix. For a given SI sequence, $w^n$, define the product matrix

$$K\left(w^n\right)=K\left(w_1\right)·K\left(w_2\right)\cdots K\left(w_n\right).$$

(56)

Now, let $\mathcal{K}=\left\{K\right(w),w\in \mathcal{W}\}$. The growth rate of the Kraft products, $K\left(w^n\right)$, over arbitrary SI sequences, $\left\{w^n\right\}$, is governed by the JSR of $\mathcal{K}$, which is defined as follows.

Definition 3.

The JSR of $\mathcal{K}$ is defined as

$${\rho }_{\mathrm{JSR}}\left(\mathcal{K}\right)=\underset{n\to \infty }{\lim }\underset{w^n\in {\mathcal{W}}^n}{\max }{\parallel K\left(w^n\right)\parallel }^{1/n},$$

(57)

where $\parallel ·\parallel$ is any matrix norm.

It is a classical result that this limit exists and is independent of the chosen norm. The GKI in the presence of SI can be formulated as follows.

Theorem 3.

For an IL FS encoder with SI,

$${\rho }_{\mathrm{JSR}}\left(\mathcal{K}\right)\le 1.$$

(58)

Proof. 

Fix an arbitrary SI sequence, $w^n\in {\mathcal{W}}^n$ and states $z,z^{\prime }\in \mathcal{Z}$. The $(z,z^{\prime })$ entry of $K\left(w^n\right)$ is given by

$$\sum _{\{x^n:g(z,x^n,w^n)=z^{\prime }\}}{2}^{-L\left[f(z,x^n,w^n)\right]}.$$

(59)

Since the encoder is IL for the fixed sequence $w^n$, the mapping between $x^n$ and $(z,x^n,w^n,z^{\prime })$ is injective over all paths from z to $z^{\prime }$. Grouping sequences according to their total code-length (similarly as before) and using a standard counting argument yield a linear upper bound (in n) on each matrix entry of $K\left(w^n\right)$, uniformly over $w^n$. Exponential growth of $\parallel K\left(w^n\right)\parallel$ is therefore impossible, and the JSR must satisfy ${\rho }_{\mathrm{JSR}}\left(\mathcal{K}\right)\le 1$. □

The following proposition can sometimes help. For example, if $v=\mathbf{1}$ satisfies Proposition 1, this means that the Kraft sum is less than or equal to unity for every initial state and every SI sequence. In such a case, one can simply design a separate prefix code for every combination of initial state and SI sequence.

Proposition 1.

If there exists a vector $v\in {\mathbb{R}}^s$ with strictly positive components such that $K\left(w\right)v\le v$ for every $w\in \mathcal{W}$, then for every SI sequence $w^n$, $K\left(w^n\right)v\le v$, and hence the family $\left\{K\left(w^n\right)\right\}$ is uniformly bounded.

Proof. 

The claim follows by induction on n. Since $v>0$, uniform boundedness of all products implies ${\rho }_{\mathrm{JSR}}\left(\mathcal{K}\right)\le 1$. □

In contrast to the case without SI, bounding the spectral radius of each individual Kraft matrix $K\left(w\right)$ is necessary but insufficient to control the growth rate of arbitrary products. In other words, even if $\rho \left[K\right(w\left)\right]\le 1$ for every $w\in \mathcal{W}$ individually, the JSR may exceed unity, and in fact, may be arbitrarily large. As an example, let $ϵ$ be an arbitrarily small positive real and consider the matrices

$$A=\left(\begin{array}{cc}ϵ& {\displaystyle \frac{1}{ϵ}}\\ 0& ϵ\end{array}\right)$$

and $B=A^T$. While $\rho \left(A\right)=\rho \left(B\right)=ϵ$, which is arbitrarily small, it turns out that

$$\rho (A·B)={ϵ}^2+{\displaystyle \frac{1}{2{ϵ}^2}}+\sqrt{1+{\displaystyle \frac{1}{4{ϵ}^4}}}\approx {\displaystyle \frac{1}{{ϵ}^2}},$$

(60)

which is accordingly, arbitrarily large. The JSR is therefore the correct quantity governing feasibility.

Exact computation of the JSR is undecidable in general, even for nonnegative rational matrices. Consequently, the above result should be interpreted as a structural constraint rather than a computational criterion. Nonetheless, there is a plethora of upper and lower bounds to the JSR. Also, as mentioned earlier, the existence of a common positive sub-invariant vector provides a meaningful and verifiable sufficient condition for subexponential growth.

7. GKI for Lossy Compression

For lossy compression, we adopt a simple encoder model, where each source vector $x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}$ is first mapped into a reproduction vector ${\widehat{x}}^{\mathsf{\ell }}=Q\left(x^{\mathsf{\ell }}\right)\in {\widehat{\mathcal{X}}}^{\mathsf{\ell }}$ within distortion $\mathsf{\ell }D$ and then ${\widehat{x}}^{\mathsf{\ell }}$ is losslessly compressed by an IL FS encoder with s states exactly as before. The latter may work in the level of single letters or in the level of ℓ-blocks. Let us define $\mathcal{B}\left({\widehat{x}}^{\mathsf{\ell }}\right)=\{x^{\mathsf{\ell }}\in {\mathcal{X}}^{\mathsf{\ell }}:d(x^{\mathsf{\ell }},{\widehat{x}}^{\mathsf{\ell }})\le \mathsf{\ell }D\}$ and let $B_{\mathsf{\ell }}={\max }_{{\widehat{x}}^{\mathsf{\ell }}\in {\widehat{\mathcal{X}}}^{\mathsf{\ell }}}\left|\mathcal{B}\left({\widehat{x}}^{\mathsf{\ell }}\right)\right|$. Now,

$$\begin{array}{ccc}\hfill K_{z{z}^{\prime }}& \stackrel{▵}{=}& \sum _{\{x^{\mathsf{\ell }}:g(z,Q\left(x^{\mathsf{\ell }}\right))=z^{\prime }\}}{2}^{-L\left[f(z,Q\left(x^{\mathsf{\ell }}\right))\right]}\hfill \\ & =& \sum _{\{{\widehat{x}}^{\mathsf{\ell }}:g(z,{\widehat{x}}^{\mathsf{\ell }}))=z^{\prime }\}}\sum _{\{x^{\mathsf{\ell }}:Q\left(x^{\mathsf{\ell }}\right)={\widehat{x}}^{\mathsf{\ell }}\}}{2}^{-L\left[f(z,{\widehat{x}}^{\mathsf{\ell }})\right]}\hfill \\ & \le & \sum _{\{{\widehat{x}}^{\mathsf{\ell }}:g(z,{\widehat{x}}^{\mathsf{\ell }}))=z^{\prime }\}}\left|\mathcal{B}\left({\widehat{x}}^{\mathsf{\ell }}\right)\right|·2^{-L\left[f(z,{\widehat{x}}^{\mathsf{\ell }})\right]}\hfill \\ & \le & B_{\mathsf{\ell }}·\sum _{\{{\widehat{x}}^{\mathsf{\ell }}:g(z,{\widehat{x}}^{\mathsf{\ell }}))=z^{\prime }\}}{2}^{-L\left[f(z,{\widehat{x}}^{\mathsf{\ell }})\right]}\hfill \\ & \stackrel{▵}{=}& B_{\mathsf{\ell }}·{\widehat{K}}_{z{z}^{\prime }},\hfill \end{array}$$

(61)

and so, $K\le B_{\mathsf{\ell }}·\widehat{K}$ entry-wise. Now, $\widehat{K}$ has all the properties that we have proved for the lossless case, it is just defined in the super-alphabet of ℓ-blocks. Since $\rho \left(\widehat{K}\right)\le 1$, we readily have:

$$\rho \left(K\right)=\rho (B_{\mathsf{\ell }}·\widehat{K})=B_{\mathsf{\ell }}·\rho \left(\widehat{K}\right)\le B_{\mathsf{\ell }}.$$

(62)

Inequality (62) can be viewed as the FS analog of very similar earlier results derived in [7,8,9], for lossy D-semifaithful codes combined with UD codes, i.e., codes that consist of a cascade of a reproduction encoder (within distortion D as above) followed by UD lossless compression of the resulting reproduction vector. In those earlier articles, the main result was a generalized Kraft inequality, where the Kraft sum (or integral, in the continuous case) is upper bounded by the volume of a ball of normalized radius D in terms of the distortion measure, which, in essence, is exactly $B_{\mathsf{\ell }}$.

For additive distortion measures, the quantity $B_{\mathsf{\ell }}$ can be estimated using the method of types [15], or the Chernoff bound, or saddle-point integration [16,17]. If the method of types is used, then $B_{\mathsf{\ell }}$ is upper bounded by $2^{\mathsf{\ell }\Phi \left(D\right)}$, where

$$\Phi \left(D\right)=\underset{\{P_{X\widehat{X}}:d(X,\widehat{X})\le D\}}{\max }H\left(X\right|\widehat{X}).$$

(63)

Thus, the corresponding GKI reads

$$\rho \left(K\right)\le 2^{\mathsf{\ell }\Phi \left(D\right)}.$$

(64)

But this bound is tight only in terms of the exponential order as a function of ℓ and hence is meaningful mainly for very large ℓ. For example, if the source and the reproduction vectors are binary, and the Hamming distortion measure is adopted, then it turns out that

$$\Phi \left(D\right)=h_2\left(D\right)=-D{\log }_{2}D-(1-D){\log }_2(1-D).$$

(65)

But we can say somewhat more in this case: here, $B_{\mathsf{\ell }}$ is simply the cardinality of a Hamming sphere of radius $\mathsf{\ell }D$, which upon careful analysis (see for example [17]), can be shown to be

$$B_{\mathsf{\ell }}={\displaystyle \frac{2^{\mathsf{\ell }{h}_2\left(D\right)}}{\sqrt{2\pi \mathsf{\ell }D(1-D)}}}·(1+o\left(1\right)).$$

(66)

In [9], more general results are available, including the case of multiple simultaneous distortion constraints.