Are Guessing, Source Coding and Tasks Partitioning Birds of A Feather? ^†

Abstract

This paper establishes a close relationship among the four information theoretic problems, namely Campbell source coding, Arikan guessing, Huleihel et al. memoryless guessing and Bunte and Lapidoth tasks’ partitioning problems in the IID-lossless case. We first show that the aforementioned problems are mathematically related via a general moment minimization problem whose optimum solution is given in terms of Renyi entropy. We then propose a general framework for the mismatched version of these problems and establish all the asymptotic results using this framework. The unified framework further enables us to study a variant of Bunte–Lapidoth’s tasks partitioning problem which is practically more appealing. In addition, this variant turns out to be a generalization of Arıkan’s guessing problem. Finally, with the help of this general framework, we establish an equivalence among all these problems, in the sense that, knowing an asymptotically optimal solution in one problem helps us find the same in all other problems.

1. Introduction

The concept of entropy is very central to information theory. In source coding, the expected number of bits required (per letter) to encode a source with (finite) alphabet set $\mathcal{X}$ and probability distribution P is the Shannon entropy $H\left(P\right):=-{\sum }_{x\in \mathcal{X}}P\left(x\right)logP\left(x\right)$. If the compressor does not know the true distribution P, but assumes a distribution Q (mismatch), then the number of bits required for compression is $H\left(P\right)+I(P,Q)$, where

$$I(P,Q):=-{\sum }_{x\in \mathcal{X}}P\left(x\right)logQ\left(x\right)+{\sum }_{x\in \mathcal{X}}P\left(x\right)logP\left(x\right),$$

(1)

is the entropy of P relative to Q (or the Kullback–Leibler divergence). In his seminal paper, Shannon [1] argued that $H\left(P\right)$ can also be regarded as a measure of uncertainty. Subsequently, Rényi [2] introduced an alternate measure of uncertainty, now known as Rényi entropy of order $\alpha$, as

$$\begin{array}{c}\hfill H_{\alpha }\left(P\right):=\frac{1}{1-\alpha }log{\sum }_{x\in \mathcal{X}}P{\left(x\right)}^{\alpha },\end{array}$$

(2)

where $\alpha >0$ and $\alpha \ne 1$. Rényi entropy can also be regarded as a generalization of the Shannon entropy as ${lim}_{\alpha \to 1}{H}_{\alpha }\left(P\right)=H\left(P\right)$. Refer Aczel and Daroczy [3] and the references therein for an extensive study of various measures of uncertainty and their characterizations.

In 1965, Campbell [4] gave an operational meaning to Rényi entropy. He showed that, instead of expected code lengths, if one minimizes the cumulants of code lengths, then the optimal cumulant is Rényi entropy $H_{\alpha }\left(P\right)$, where $\alpha =1/(1+\rho )$ with $\rho$ being the order of the cumulant. He also showed that the optimal cumulant can be achieved by encoding sufficiently long sequences of symbols. Sundaresan (Theorem 8 of [5]) (c.f. Blumer and McElice [6]) showed that, in the mismatched case, the optimal cumulant is $H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)$, where

$$I_{\alpha }(P,Q):=\frac{\alpha }{1-\alpha }log{\sum }_{x}P\left(x\right){\left(\frac{Q{\left(x\right)}^{\alpha }}{{\sum }_{y}Q{\left(y\right)}^{\alpha }}\right)}^{\frac{\alpha -1}{\alpha }}-H_{\alpha }\left(P\right)$$

(3)

is called α-entropy of P relative to Q or Sundaresan’s divergence [7]. Hence, $I_{\alpha }(P,Q)$ can be interpreted as the penalty for not knowing the true distribution. The first term in (3) is sometimes called the Renyi cross-entropy and is analogous to the first term of (1). $I_{\alpha }(P,Q)\ge 0$ with equality if and only if $P=Q$. $I_{\alpha }$-divergence can also be regarded as a generalization of the Kullback–Leibler divergence as ${lim}_{\alpha \to 1}{I}_{\alpha }(P,Q)=I(P,Q)$. Refer to [5,8,9] for detailed discussions on the properties of $I_{\alpha }$. Lutwak et al. also independently identified $I_{\alpha }$ in the context of maximum Rényi entropy and called it an α-Rényi relative entropy (Equation (4) of [10]). $I_{\alpha }$, for $\alpha >1$, also arises in robust inference problems (see [11] and the references therein).

In [12], Massey studied a guessing problem where one is interested in the expected number of guesses required to guess a random variable X that assumes values from an infinite set, and found a lower bound in terms of Shannon entropy. Arıkan [13] studied it for a finite alphabet set and showed that Rényi entropy arises as the optimal solution in minimizing moments of the number of guesses. Subsequently, Sundaresan [5] showed that the penalty in guessing according to a distribution Q when the true distribution is P, is given by $I_{\alpha }(P,Q)$. It is interesting to note that guesswork has also been studied from a large deviations point of view [14,15,16,17,18]. Bunte and Lapidoth [8] studied a problem on partitioning of tasks and showed that Rényi entropy and Sundaresan’s divergence play a similar role in the optimal number of tasks performed. We propose, in this paper, a variant of this problem where the tasks in each subset of the partition are performed according to the decreasing order of probabilities. We show that Rényi antropy and Sundaresan’s divergences arise as optimal solutions in this problem too. Huleihel et al. [19,20] studied the memoryless guessing problem, a variant of Arıkan’s guessing problem, with i.i.d. (independent and identically distributed) guesses and showed that the minimum attainable factorial moments of number of guesses is the Rényi entropy. We show, in this paper, that the minimum factorial moment in the mismatched case is measured by the Sundaresan’s divergence.

We observe that, in all these problems, the objective is to minimize usual moments or factorial moments of random variables, and Rényi entropy and Sundaresan’s divergence arise in optimal solutions. The relationship between source coding and guessing is well-known in the literature. Arıkan and Merhav established a close relationship between lossy source coding and guessing with distortion using large deviation techniques [14,21]. The same for the lossless case was done by Hanawal and Sundaresan [17]. In this paper, we establish a general framework for all the five problems in the IID-lossless case. We then use this to establish upper and lower bounds for the mismatched version of these problems. This helps us find an equivalence among all these problems, in the sense that knowing an asymptotically optimal solution in one problem helps us find the same in all other problems.

Our Contributions in the Paper:

(a)

a general framework for the problems on source coding, guessing and tasks partitioning;

(b)

lower and upper bounds for the general framework of these problems both in matched and mis-matched cases;

(c)

a unified approach to derive bounds for the mismatched version of these problems;

(d)

a generalized tasks partitioning problem; and

(e)

establishing operational commonality among the problems.

Organisation of the Paper:

In Section 2, we present our unified framework, and find conditions under which lower and upper bounds are attained. In Section 3, we present four well-known information-theoretic problems, namely, Campbell’s source coding, Arıkan’s guessing, Huleihel et al.’s memoryless guessing, and Bunte–Lapidoth’s tasks partitioning, and re-establish and refine major results pertaining to these problems. In Section 4, we propose and solve a generalized tasks partitioning problem. In Section 5, we establish a connection among the aforementioned problems. Finally, we summarize and conclude the paper in Section 6.

2. A General Minimization Problem

In this section, we present a general minimization problem whose optimum solution evaluates to Rényi entropy. We will later show that all problems stated in Section 3 are particular instances of this general problem.

 Proposition 1. 

Let $\psi :\mathcal{X}\to (0,\infty )$ be such that ${\sum }_{x\in \mathcal{X}}\psi {\left(x\right)}^{-1}\le k$ for some $k>0$. Then, for $\rho \in (-1,0)\cup (0,\infty )$,

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[\psi {\left(X\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-logk,$$

(4)

where ${\mathbb{E}}_P[·]$ denotes the expectation with respect to probability distribution P on $\mathcal{X}$, $H_{\alpha }\left(P\right)$ is the Rényi entropy of order α, and $\alpha :=\alpha \left(\rho \right)=1/(1+\rho )$. The lower bound is achieved if and only if

$$\psi {\left(x\right)}^{-1}=k·P{\left(x\right)}^{\alpha }/Z_{P,\alpha }\mathrm{for}x\in \mathcal{X},$$

(5)

where $Z_{P,\alpha }:={\sum }_{x\in \mathcal{X}}P{\left(x\right)}^{\alpha }$.

 Proof. 

Observe that

$$\begin{array}{ccc}\hfill \mathrm{sgn}\left(\rho \right){\sum }_{x\in \mathcal{X}}P\left(x\right)\psi {\left(x\right)}^{\rho }& =& \mathrm{sgn}\left(\rho \right){\sum }_{x\in \mathcal{X}}P{\left(x\right)}^{\alpha }{\left(\frac{\psi {\left(x\right)}^{-1}}{P{\left(x\right)}^{\alpha }}\right)}^{-\rho }\hfill \\ & \stackrel{\left(a\right)}{\ge }& \mathrm{sgn}\left(\rho \right)\left(\sum _{x}P{\left(x\right)}^{\alpha }\right)·{\left(\frac{{\sum }_x\psi {\left(x\right)}^{-1}}{{\sum }_{x}P{\left(x\right)}^{\alpha }}\right)}^{-\rho }\hfill \\ & =& \mathrm{sgn}\left(\rho \right){\left(\sum _{x}P{\left(x\right)}^{\alpha }\right)}^{1+\rho }·{\left(\sum _x\psi {\left(x\right)}^{-1}\right)}^{-\rho }\hfill \\ & \stackrel{\left(b\right)}{\ge }& \mathrm{sgn}\left(\rho \right){\left(\sum _{x}P{\left(x\right)}^{\alpha }\right)}^{1+\rho }·k^{-\rho },\hfill \end{array}$$

where $\left(a\right)$ is due to the generalised log-sum inequality (Equation (4.1) of [22]) applied to the function $f\left(x\right)=\mathrm{sgn}\left(\rho \right)·x^{-\rho }$; and $\left(b\right)$ follows from the hypothesis that ${\sum }_x\psi {\left(x\right)}^{-1}$$\le k$. By taking log and then dividing by $\rho$, we obtain (4). Equality holds in $\left(a\right)$ if and only if $\psi {\left(x\right)}^{-1}=\nu P{\left(x\right)}^{\alpha }$ for some constant $\nu$ and in (b) if and only if ${\sum }_x\psi {\left(x\right)}^{-1}=k$. This completes the proof. □

The left-side of (4) is called normalised cumulant of $\psi \left(X\right)$ of order ρ. The measure $P^{\left(\alpha \right)}\left(x\right):=P{\left(x\right)}^{\alpha }/Z_{P,\alpha }$ in (5) that attains the lower bound in (4) is called an α-scaled measure or escort measure of P. This measure also arises in robust inference (Equation (7) of [11]) and statistical physics [23]. The above proposition can also be proved using a variational formula as follows. By a version of Donsker–Varadhan variational formula (Propostion 4.5.1 of [24]), for any real-valued f on $\mathcal{X}$, we have

$$log{\mathbb{E}}_P[2^{f\left(X\right)}]=\underset{Q}{max}\{{\mathbb{E}}_Q\left[f\left(X\right)\right]-D(Q\parallel P)\},$$

(6)

where the max is over all probability distributions Q on $\mathcal{X}$. Taking $\rho >0$ and $f\left(x\right)=\rho log\psi \left(x\right)$ in (6), we have

$$\begin{array}{ccc}\hfill log{\mathbb{E}}_P\left[\psi {\left(X\right)}^{\rho }\right]& =& \underset{Q}{max}\{\rho {\mathbb{E}}_Q[log\psi \left(X\right)]-D(Q\parallel P)\}\hfill \\ & =& \underset{Q}{max}\left\{-\rho \sum _{x}Q\left(x\right)log\frac{\psi {\left(x\right)}^{-1}Q\left(x\right)}{Q\left(x\right)}-D(Q\parallel P)\right\}\hfill \\ & =& \underset{Q}{max}\left\{\rho H\left(Q\right)-\rho \sum _{x}Q\left(x\right)log\frac{\psi {\left(x\right)}^{-1}}{Q\left(x\right)}-D(Q\parallel P)\right\}\hfill \\ & \stackrel{\left(a\right)}{\ge }& \underset{Q}{max}\left\{\rho H\left(Q\right)-\rho log\sum _x\psi {\left(x\right)}^{-1}-D(Q\parallel P)\right\}\hfill \\ & \stackrel{\left(b\right)}{\ge }& \rho \underset{Q}{max}\left\{H\left(Q\right)-\frac{1}{\rho }D(Q\parallel P)\right\}-\rho logk,\hfill \end{array}$$

where (a) is by the log-sum inequality (Equation (4.1) of [22]) and (b) is by applying the constraint ${\sum }_x\psi {\left(x\right)}^{-1}\le k$. For $\rho \in (-1,0)$, the inequalities in (a) and (b) are reversed, and the last max is replaced by min. Hence, (4) follows as the last max is equal to $H_{\alpha }\left(P\right)$ by (Theorem 1 of [25]). Equality in $\left(a\right)$ and $\left(b\right)$ holds if and only if $\psi {\left(x\right)}^{-1}=k·Q\left(x\right)$. In addition, the last max is attained when $Q\left(x\right)=P{\left(x\right)}^{\alpha }/Z_{P,\alpha }\mathrm{for}x\in \mathcal{X}$. This completes the proof. The following is the analogous one for Shannon entropy.

 Proposition 2. 

Let $\psi :\mathcal{X}\to (0,\infty )$ be such that ${\sum }_{x\in \mathcal{X}}\psi {\left(x\right)}^{-1}\le k$. Then,

$$\begin{array}{c}\hfill {\mathbb{E}}_P\left[log\psi \left(X\right)\right]\ge H\left(P\right)-logk.\end{array}$$

(7)

Equality in (7) is achieved if and only if $\psi {\left(x\right)}^{-1}=k·P\left(x\right)\forall x\in \mathcal{X}$.

Proof. 

$$\begin{array}{ccc}\hfill {\mathbb{E}}_P\left[log\psi \left(X\right)\right]& =& -\sum _{x}P\left(x\right)log\frac{P\left(x\right)·\psi {\left(x\right)}^{-1}}{P\left(x\right)}=H\left(P\right)-\sum _{x}P\left(x\right)log\frac{\psi {\left(x\right)}^{-1}}{P\left(x\right)}\hfill \\ & \ge & H\left(P\right)-log\sum _x\psi {\left(x\right)}^{-1}\ge H\left(P\right)-logk,\hfill \end{array}$$

where the penultimate inequality is due to the log-sum inequality. Equality holds in both inequalities if and only if $\psi {\left(x\right)}^{-1}=k·P\left(x\right)\forall x\in \mathcal{X}$. □

It is interesting to note that $log{\mathbb{E}}_P\left[\psi {\left(X\right)}^{\rho }\right]/\rho \to {\mathbb{E}}_P[log\left(\psi \left(X\right)\right)]$ and $H_{\alpha }\left(P\right)\to H\left(P\right)$ as $\rho \to 0$ in (4). We now extend Propositions 1 and 2 to sequences of random variables. Let ${\mathcal{X}}^n$ be the set of all n-length sequences of elements of $\mathcal{X}$, and $P_n$ be the n-fold product distribution of P on ${\mathcal{X}}^n$, that is, for $x^n:=x_1,\cdots ,x_n\in {\mathcal{X}}^n$, $P_n\left(x^n\right)={\prod }_{i=1}^{n}P\left(x_i\right)$.

 Corollary 1. 

Given any $n\ge 1$, if ${\psi }_n:{\mathcal{X}}^n\to [0,\infty )$ is such that ${\sum }_{x^n\in {\mathcal{X}}^n}{\psi }_n{\left(x^n\right)}^{-1}$ $\le k_n$ for some $k_n>0$, then

(a)

For $\rho \in (-1,0)\cup (0,\infty )$,

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[{\psi }_n{\left(X^n\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-\underset{n\to \infty }{lim\; sup}\frac{log{k}_n}{n}.$$

(b)

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n}{\mathbb{E}}_{P_n}[log{\psi }_n\left(X^n\right)]\ge H\left(P\right)-\underset{n\to \infty }{lim\; sup}\frac{log{k}_n}{n},$$

where ${\mathbb{E}}_{P_n}[·]$ denotes the expectation with respect to probability distribution $P_n$ on ${\mathcal{X}}^n$.

 Proof. 

It is easy to see that $H_{\alpha }\left(P_n\right)=n{H}_{\alpha }\left(P\right)$ and $H\left(P_n\right)=nH\left(P\right)$. Applying Propositions 1 and 2, dividing throughout by n and taking $lim\; infn\to \infty$, the results follow. □

A General Framework for Mismatched Cases

In this sub-section, we establish a unified approach for cases when there is mismatch between assumed and true distributions.

 Proposition 3. 

Let $\rho >-1$, $\alpha =1/(1+\rho )$, and Q be a probability distribution on $\mathcal{X}$. For $n\ge 1$, let $Q_n$ be the n-fold product distribution of Q on ${\mathcal{X}}^n$. If ${\psi }_n:{\mathcal{X}}^n\to (0,\infty )$ is such that

$${\psi }_n\left(x^n\right)\le \frac{c_n·Z_{Q_n,\alpha }}{Q_n{\left(x^n\right)}^{\alpha }}$$

(8)

for some $c_n>0$, then

(a)

for $\rho \ne 0$, we have

$$\mathit{sgn}\left(\rho \right)·{\mathbb{E}}_{P_n}\left[{\psi }_n{\left(X^n\right)}^{\rho }\right]\le \mathit{sgn}\left(\rho \right)·2^{n\rho [H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)+n^{-1}log{c}_n]},$$

(b)

for $\rho \ne 0$, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[{\psi }_n{\left(X^n\right)}^{\rho }\right]\le H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)+\underset{n\to \infty }{lim\; sup}\frac{1}{n}log{c}_n,$$

(c)

for $\rho =0$, we have

$${\mathbb{E}}_{P_n}[log{\psi }_n\left(X^n\right)]\le n[H\left(P\right)+I(P,Q)+n^{-1}log{c}_n],$$

(d)

for $\rho =0$, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n}{\mathbb{E}}_{P_n}[log{\psi }_n\left(X^n\right)]\le H\left(P\right)+I(P,Q)+\underset{n\to \infty }{lim\; sup}\frac{1}{n}log{c}_n.$$

 Proof. 

Part (a): From (8), we have

$$\begin{array}{cc}\hfill \mathrm{sgn}\left(\rho \right)·{\mathbb{E}}_{P_n}\left[{\psi }_n{\left(X^n\right)}^{\rho }\right]& =\mathrm{sgn}\left(\rho \right)·{\sum }_{x^n\in {\mathcal{X}}^n}{P}_n\left(x^n\right){\psi }_n{\left(x^n\right)}^{\rho }\hfill \\ \hfill & \le \mathrm{sgn}\left(\rho \right)·c_n^{\rho }{Z}_{Q_n,\alpha }^{\rho }{\sum }_{x^n\in {\mathcal{X}}^n}{P}_n\left(x^n\right)Q_n{\left(x^n\right)}^{-\alpha \rho }\hfill \\ \hfill & =\mathrm{sgn}\left(\rho \right)·2^{\rho [H_{\alpha }\left(P_n\right)+I_{\alpha }(P_n,Q_n)+log{c}_n]}\hfill \\ \hfill & =\mathrm{sgn}\left(\rho \right)·2^{n\rho [H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)+n^{-1}log{c}_n]},\hfill \end{array}$$

(9)

where the penultimate equality holds from the definition of $I_{\alpha }$, and the last one holds because $H_{\alpha }\left(P_n\right)=n{H}_{\alpha }\left(P\right)$, $I_{\alpha }(P_n,Q_n)=n{I}_{\alpha }(P,Q)$.

Part (b): Taking log, dividing throughout by $n\rho$, and then applying lim sup successively on both sides of (9), the result follows.

Part (c): When $\rho =0$, we have $\alpha =1$ and (8) becomes ${\psi }_n\left(x^n\right)\le \frac{c_n}{Q_n\left(x^n\right)}$. Hence,

$$\begin{array}{cc}\hfill {\mathbb{E}}_{P_n}[log{\psi }_n\left(X^n\right)]& ={\sum }_{x^n\in {\mathcal{X}}^n}{P}_n\left(x^n\right)log{\psi }_n\left(x^n\right)\hfill \\ \hfill & \le log{c}_n+{\sum }_{x^n\in {\mathcal{X}}^n}{P}_n\left(x^n\right)log(1/Q_n\left(x^n\right))\hfill \\ \hfill & =log{c}_n+H\left(P_n\right)+I(P_n,Q_n)\hfill \\ \hfill & =n(H\left(P\right)+I(P,Q)+n^{-1}log{c}_n),\hfill \end{array}$$

(10)

where the last equality holds because $H\left(P_n\right)=nH\left(P\right)$, and $I(P_n,Q_n)=nI(P,Q)$.

Part (d): Dividing (10) throughout by n, and taking limsup on both sides, the result follows. □

 Proposition 4. 

Let $\rho >-1$, $\alpha =1/(1+\rho )$, and Q be a probability distribution on $\mathcal{X}$. For $n\ge 1$, let $Q_n$ be the n-fold product distribution of Q on ${\mathcal{X}}^n$. Suppose ${\psi }_n:{\mathcal{X}}^n\to (0,\infty )$ is such that

$${\psi }_n\left(x^n\right)\ge \frac{a_n{Z}_{Q_n,\alpha }}{Q_n{\left(x^n\right)}^{\alpha }}$$

for some $a_n>0$, then

(a)

for $\rho \ne 0$, we have

$$\mathit{sgn}\left(\rho \right)·{\mathbb{E}}_{P_n}\left[{\psi }_n{\left(X^n\right)}^{\rho }\right]\ge \mathit{sgn}\left(\rho \right)·2^{n\rho (H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)+n^{-1}log a_n)},$$

(b)

for $\rho \ne 0$, we have

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[{\psi }_n{\left(X^n\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)+\underset{n\to \infty }{lim\; inf}\frac{1}{n}log{a}_n,$$

(c)

for $\rho =0$, we have

$${\mathbb{E}}_{P_n}[log{\psi }_n\left(X^n\right)]\ge n(H\left(P\right)+I(P,Q)+n^{-1}log{a}_n),$$

(d)

for $\rho =0$, we have

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n}{\mathbb{E}}_{P_n}[log{\psi }_n\left(X^n\right)]\ge H\left(P\right)+I(P,Q)+\underset{n\to \infty }{lim\; inf}\frac{1}{n}log{a}_n.$$

 Proof. 

Similar to proof of Proposition 3. □

3. Problem Statements and Known Results

In this section, we discuss Campbell’s source coding problem, Arıkan’s guessing problem, Huleihel et al.’s memoryless guessing problem, and Bunte–Lapidoth’s tasks partitioning problem. Using the general framework presented in the previous section, we re-establish known results, and present a few new results relating to these problems.

3.1. Source Coding Problem

Let X be a random variable that assumes values from a finite alphabet set $\mathcal{X}=\{a_1,\dots ,a_m\}$ according to a probability distribution P. The tuple $(\mathcal{X},P)$ is usually referred to as a source. A binary codeC is a mapping from $\mathcal{X}$ to the set of finite length binary strings. Let $L\left(C\right(X\left)\right)$ be the length of code $C\left(X\right)$. The objective is to find a uniquely decodable code that minimizes the expected code-length, that is,

$$\mathrm{Minimize}{\mathbb{E}}_P\left[L\left(C\left(X\right)\right)\right]$$

over all uniquely decodable codes C. Kraft and McMillan independently proved the following relation between uniquely decodable codes and their code-lengths.

Kraft-McMilan Theorem [26]: If C a uniquely decodable code, then

$${\sum }_{x\in \mathcal{X}}{2}^{-L\left(C\right(x\left)\right)}\le 1.$$

(11)

Conversely, given a length sequence that satisfies the above inequality, there exists a uniquely decodable code C with the given length sequence.

Thus, one can confine the search space for C to codes satisfying the Kraft–McMillan inequality (11).

Theorem 5.3.1 of [26]: If C is a uniquely decodable code, then ${\mathbb{E}}_P\left[L\left(C\left(X\right)\right)\right]\ge H\left(P\right)$.

 Proof. 

Choose $\psi \left(x\right)=2^{L\left(C\right(x\left)\right)}$, where $L\left(C\right(x\left)\right)$ is the length of code $C\left(x\right)$ assigned to alphabet x. Since C is uniquely decodable, from (11), we have ${\sum }_{x\in \mathcal{X}}\psi {\left(x\right)}^{-1}\le 1$. Now, an application of Proposition 2 with $k=1$ yields the desired result. □

 Theorem 1. 

Let $X^n:=X_1,\dots ,X_n$ be an i.i.d. sequence from ${\mathcal{X}}^n$ following the product distribution $P_n\left(X^n\right)={\prod }_{i=1}^{n}P\left(X_i\right)$. Let $Q_n\left(X^n\right)={\prod }_{i=1}^{n}Q\left(X_i\right)$, where Q is another probability distribution. Let $C_n$ be a code such that $L\left(C_n\left(X^n\right)\right)=\lceil -log{Q}_n\left(X^n\right)\rceil$. Then, $C_n$ satisfies Kraft–McMillan inequality and

$$\underset{n\to \infty }{lim}\frac{{\mathbb{E}}_{P_n}\left[L\left(C_n\left(X^n\right)\right)\right]}{n}=H\left(P\right)+I(P,Q).$$

 Proof. 

Choose ${\psi }_n\left(x^n\right)=2^{L\left(C_n\left(x^n\right)\right)}$, where $L(C_n\left(x^n\right)$ is the length of code $C_n\left(x^n\right)$ assigned to sequence $x^n$. Then, we have

$$\begin{array}{c}\hfill {\psi }_n\left(x^n\right)=2^{L\left(C_n\left(x^n\right)\right)}=2^{\lceil -log{Q}_n\left(x^n\right)\rceil }\le 2·2^{-log{Q}_n\left(x^n\right)}=2/Q_n\left(x^n\right).\end{array}$$

An application of Proposition 3 with $c_n=2$ yields ${lim\; sup}_{n\to \infty }{\mathbb{E}}_{P_n}\left[L\left(C_n\left(X^n\right)\right)\right]/n\le H\left(P\right)+I(P,Q)$. Furthermore, we also have

$$\begin{array}{c}\hfill {\psi }_n\left(x^n\right)=2^{L\left(C_n\left(x^n\right)\right)}=2^{\lceil -log{Q}_n\left(x^n\right)\rceil }\ge 2^{-log{Q}_n\left(x^n\right)}=1/Q_n\left(x^n\right).\end{array}$$

An application of Proposition 4 with $a_n=1$ gives ${lim\; inf}_{n\to \infty }{\mathbb{E}}_{P_n}\left[L\left(C_n\left(X^n\right)\right)\right]/n\ge H\left(P\right)+I(P,Q)$. □

3.2. Campbell Coding Problem

Campbell’s coding problem is similar to Shannon’s source coding problem except that, instead of minimizing the expected code-length, one is interested in minimizing the normalized cumulant of code lengths, that is,

$$\begin{array}{c}\hfill \mathrm{Minimize}\frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho L\left(C\right(X\left)\right)}],\end{array}$$

over all uniquely decodable codes C, and $\rho >0$. This problem was shown to be equivalent to minimizing buffer overflow probability by Humblet in [27]. A lower bound for the normalized cumulants in terms of Rényi entropy was provided by Campbell [4].

Lemma 1 of [4]: Let C be a uniquely decodable code. Then,

$$\begin{array}{c}\hfill \frac{1}{\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L\left(C\right(X\left)\right)}\right]\ge H_{\alpha }\left(P\right),\end{array}$$

(12)

where $\alpha =1/(1+\rho ).$

 Proof. 

Apply Proposition 1 with $\psi \left(x\right)=2^{L\left(C\right(x\left)\right)}$ and $k=1$. □

Notice that, if we ignore the integer constraint of the length function, then

$$\begin{array}{c}\hfill L\left(C\left(x\right)\right)=log\frac{Z_{P,\alpha }}{P{\left(x\right)}^{\alpha }},\end{array}$$

(13)

with $Z_{P,\alpha }$ as in Proposition 1, satisfies (11) and achieves the lower bound in (12). Campbell also showed that the lower bound in (12) can be achieved by encoding long sequences of symbols with code-lengths close to (13).

Theorem 1 of [4]: If $C_n$ is a uniquely decodable code such that

$$\begin{array}{c}\hfill L\left(C_n\left(x^n\right)\right)=⌈log\frac{Z_{P_n,\alpha }}{P_n{\left(x^n\right)}^{\alpha }}⌉,\end{array}$$

(14)

then

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L\left(C_n\left(X^n\right)\right)}\right]=H_{\alpha }\left(P\right).$$

 Proof. 

Choose ${\psi }_n\left(x^n\right)=2^{L\left(C_n\left(x^n\right)\right)}$. Then, from (14), we have

$$\frac{Z_{P_n,\alpha }}{P_n{\left(x^n\right)}^{\alpha }}\le {\psi }_n\left(x^n\right)<2·\frac{Z_{P_n,\alpha }}{P_n{\left(x^n\right)}^{\alpha }}·$$

The result follows by applying Propositions 3 and 4 with $c_n=2$, $a_n=1$ and $Q=P$. □

Mismatch Case:

Redundancy in the mismatched case of the Campbell’s problem was studied in [5,6]. Sundaresan showed that the difference in the normalized cumulant from the minimum when encoding according to an arbitrary uniquely decodable code is measured by $I_{\alpha }$-divergence up-to a factor of 1 [5]. We provide a more general version of this result in the following.

 Proposition 5. 

Let X be a random variable that assumes values from set $\mathcal{X}$ according to a probability distribution P. Let $\rho \in (-1,0)\cup (0,\infty )$ and $L:\mathcal{X}\to {\mathbb{Z}}_{+}$ be an arbitrary length function that satisfies (11). Define

$$\begin{array}{c}\hfill R_c(P,L,\rho ):=\frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho L\left(X\right)}]-\underset{K}{min}\frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho K\left(X\right)}],\end{array}$$

(15)

where the minimum is over all length functions K satisfying (11). Then, there exists a probability distribution $Q_L$ such that

$$\begin{array}{c}\hfill I_{\alpha }(P,Q_L)-log\eta -1\le R_c(P,L,\rho )\le I_{\alpha }(P,Q_L)-log\eta ,\end{array}$$

(16)

where $\eta ={\sum }_x{2}^{-L\left(x\right)}$.

 Proof. 

Since K satisfies (11), an application of Proposition 1 with $\psi \left(x\right)=2^{K\left(x\right)}$ gives us $\frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho K\left(X\right)}]\ge H_{\alpha }\left(P\right)$. Since $K\left(x\right)=\lceil log(Z_{P,\alpha }/P{\left(x\right)}^{\alpha })\rceil$ satisfies (11) and $\psi \left(x\right)=2^{K\left(x\right)}<2·Z_{P,\alpha }/P{\left(x\right)}^{\alpha }$, applying Proposition 3 with $n=1,c_1=2$, and $Q=P$, we have

$$\frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho K\left(X\right)}]\le H_{\alpha }\left(P\right)+1,$$

that is, the minimum in (15) is between $H_{\alpha }\left(P\right)$ and $H_{\alpha }\left(P\right)+1$. Hence,

$$\begin{array}{c}\hfill \frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho L\left(X\right)}]-H_{\alpha }\left(P\right)-1\le R_c(P,L,\rho )\le \frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho L\left(X\right)}]-H_{\alpha }\left(P\right).\end{array}$$

(17)

Let us now define a probability distribution $Q_L$ as

$$Q_L\left(x\right)=\frac{2^{-L\left(x\right)/\alpha }}{{\sum }_{x^{\prime }}{2}^{-L\left(x^{\prime }\right)/\alpha }}·$$

Then,

$$2^{L\left(x\right)}=Q_L{\left(x\right)}^{-\alpha }{Z}_{Q_L,\alpha }·\frac{1}{{\sum }_{x^{\prime }}{2}^{-L\left(x^{\prime }\right)}}=Q_L{\left(x\right)}^{-\alpha }{Z}_{Q_L,\alpha }·\frac{1}{\eta },$$

where $\eta ={\sum }_{x^{\prime }}{2}^{-L\left(x^{\prime }\right)}$. Applying Propositions 3 and 4 with $n=1$, ${\psi }_1\left(x\right)=2^{L\left(x\right)}$, $a_1=c_1=1/\eta$, we obtain

$$\frac{1}{\rho }log{\mathbb{E}}_P[2^{\rho L\left(X\right)}]=H_{\alpha }\left(P\right)+I_{\alpha }(P,Q_L)-log\eta .$$

(18)

Combining (17) and (18), we obtain the desired result. □

We remark that the bound in (16) can be loose when $\eta$ is small. For example, for a source with two symbols, say x and y, with code lengths $L\left(x\right)=L\left(y\right)=100$, we have $R_c(P,L,\rho )\ge I_{\alpha }(P,Q_L)+98$. However, if one imposes the constraint $1/2\le \eta \le 1$, then (16) simplifies to

$$|R_c(P,L,\rho )-I_{\alpha }(P,Q_L)|\le 1,$$

which is (Theorem 8 of [5]). $I_{\alpha }(P,Q_L)$ is, in a sense, the penalty when $Q_L$ does not match the true distribution P. In view of this, a result analogous to Proposition 5 also holds for the Shannon source coding problem.

3.3. Arıkan’s Guessing Problem

Let $\mathcal{X}$ be a set of objects with $\left|\mathcal{X}\right|=m$. Bob thinks of an object X (a random variable) from $\mathcal{X}$ according to a probability distribution P. Alice guesses it by asking questions of the form “Is $X=x$?”. The objective is to minimize average number of guesses required for Alice to guess X correctly. By a guessing strategy (or guessing function), we mean a one-one map $G:\mathcal{X}\to \{1,\dots ,m\}$, where $G\left(x\right)$ is to be interpreted as the number of questions required to guess x correctly. Arıkan studied the ${\rho }^{\mathrm{th}}$ moment of number of guesses and found upper and lower bounds in terms of Rényi entropy.

Theorem 1 of [13]: Let G be any guessing function. Then, for $\rho \in (-1,0)\cup (0,\infty )$,

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[G{\left(X\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-log(1+lnm).$$

 Proof. 

Let G be any guessing function. Let $\psi \left(x\right)=G\left(x\right)$. Then, we have ${\sum }_{x\in \mathcal{X}}\psi {\left(x\right)}^{-1}={\sum }_{x\in \mathcal{X}}1/G\left(x\right)={\sum }_{i=1}^{m}1/i\le 1+lnm$. An application of Proposition 1 with $k=1+lnm$ yields the desired result. □

Arıkan showed that an optimal guessing function guesses according to the decreasing order of P-probabilities with ties broken using an arbitrary but fixed rule [13]. He also showed that normalized cumulant of an optimal guessing function is bounded above by the Rényi entropy. Next, we present a proof of this using our general framework.

Proposition 4 of [13]: If $G^{*}$ is an optimal guessing function, then, for $\rho \in (-1,0)\cup (0,\infty )$,

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[G^{*}{\left(X\right)}^{\rho }\right]\le H_{\alpha }\left(P\right).$$

 Proof. 

Let us rearrange the probabilities $\left\{P\right(x),x\in \mathcal{X}\}$ in non-increasing order, say

$$p_1\ge p_2\ge \cdots \ge p_m.$$

Then, the optimal guessing function $G^{*}$ is given by $G^{*}\left(x\right)=i$ if $P\left(x\right)=p_i$. Let us index the elements in set $\mathcal{X}$ as $\{x_1,x_2,\dots ,x_m\}$, according to the decreasing order of their probabilities. Then, for $i\in \{1,\cdots ,m\}$, we have

$$\frac{Z_{P,\alpha }}{P{\left(x_i\right)}^{\alpha }}=\frac{{\sum }_{j=1}^m{p}_j^{\alpha }}{p_i^{\alpha }}\ge i=G^{*}\left(x_i\right).$$

(19)

That is, $G^{*}\left(x\right)\le \frac{Z_{P,\alpha }}{P{\left(x\right)}^{\alpha }}$ for $x\in \mathcal{X}$. Now, an application of Proposition 3 with $n=1$, $Q=P$, ${\psi }_1\left(x\right)=G^{*}\left(x\right)$ and $c_1=1$, gives us

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[G^{*}{\left(X\right)}^{\rho }\right]=\frac{1}{\rho }log{\mathbb{E}}_P\left[\psi {\left(X\right)}^{\rho }\right]\le H_{\alpha }\left(P\right)+I_{\alpha }(P,P)+log1=H_{\alpha }\left(P\right).$$

□

Arıkan also proved that the upper bound of Rényi entropy can be achieved by guessing long sequences of letters in an i.i.d. fashion.

Proposition 5 of [13] Let $X_1,X_2,\dots ,X_n$ be a sequence of i.i.d. guesses. Let $G_n^{*}(X_1,\dots ,X_n)$ be an optimal guessing function. Then, for $\rho \in (-1,0)\cup (0,\infty )$,

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_n^{*}{(X_1,X_2,\dots ,X_n)}^{\rho }\right]=H_{\alpha }\left(P\right).$$

 Proof. 

Let $G_n^{*}$ be the optimal guessing function from ${\mathcal{X}}^n$ to $\{1,2,\dots ,m^n\}$. An application of Corollary 1 with ${\psi }_n\left(x^n\right)=G_n^{*}\left(x^n\right)$ and $k_n=1+nlnm$ yields

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_n^{*}{\left(X^n\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-\underset{n\to \infty }{lim\; sup}\frac{log(1+nlnm)}{n}=H_{\alpha }\left(P\right).\end{array}$$

(20)

As in the proof of the previous result, we know that $G^{*}\left(x^n\right)\le \frac{Z_{P_n,\alpha }}{P_n{\left(x^n\right)}^{\alpha }}$ for $x^n\in {\mathcal{X}}^n$. Hence, an application of Proposition 3 with $Q_n=P_n$, ${\psi }_n\left(x^n\right)=G_n^{*}\left(x^n\right)$, and $c_n=1$ yields

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_n^{*}{\left(X^n\right)}^{\rho }\right]\le H_{\alpha }\left(P\right).\end{array}$$

(21)

Combining (20) and (21), we obtain the desired result. □

Henceforth, we shall denote the optimal guessing function corresponding to a probability distribution P by $G_P$.

Mismatch Case:

Suppose Alice does not know the true underlying probability distribution P, and guesses according to some guessing function G. The following proposition tells us that the penalty for deviating from the optimal guessing function can be measured by $I_{\alpha }$-divergence.

 Proposition 6. 

Let G be an arbitrary guessing function. Then, for $\rho \in (-1,0)\cup (0,\infty )$, there exists a probability distribution $Q_G$ on $\mathcal{X}$ such that

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[G{\left(X\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)+I_{\alpha }(P,Q_G)-log(1+lnm).$$

 Proof. 

Let G be a guessing function. Define a probability distribution $Q_G$ on $\mathcal{X}$ as

$$Q_G\left(x\right)=\frac{G{\left(x\right)}^{-1/\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}G{\left(x^{\prime }\right)}^{-1/\alpha }}.$$

(22)

Then, we have

$$\frac{Z_{Q_G,\alpha }}{Q_G{\left(x\right)}^{\alpha }}=G\left(x\right){\sum }_{x^{\prime }\in \mathcal{X}}\frac{1}{G\left(x^{\prime }\right)}\le G\left(x\right)·(1+lnm)·$$

Now, an application of Proposition 4 with $n=1$, ${\psi }_1\left(x\right)=G\left(x\right)$, and $a_1=1/(1+lnm)$ yields the desired result. □

A converse result is the following.

Proposition 1 of [5]: Let $G_Q$ be an optimal guessing function associated with Q. Then, for $\rho \in (-1,0)\cup (0,\infty )$,

$$\begin{array}{c}\hfill \frac{1}{\rho }log{\mathbb{E}}_P\left[G_Q{\left(X\right)}^{\rho }\right]\le H_{\alpha }\left(P\right)+I_{\alpha }(P,Q),\end{array}$$

where the expectation is with respect to P.

 Proof. 

Let us rearrange the probabilities $\left(Q\right(x),x\in \mathcal{X})$ in non-increasing order, say

$$q_1\ge q_2\ge \cdots \ge q_m.$$

By definition, $G_Q\left(x\right)=i$ if $Q\left(x\right)=q_i$. Then, as in (19), we have $G_Q\left(x\right)\le Z_{Q,\alpha }/Q{\left(x\right)}^{\alpha }$$\mathrm{for}x\in \mathcal{X}$. Hence, an application of Proposition 3 with $n=1$, ${\psi }_1\left(x\right)=G_Q\left(x\right)$, and $c_1=1$ proves the result. □

Observe that, given a guessing function G, if we apply the above proposition for $Q=Q_G$, where $Q_G$ is as in (22), then we obtain

$$\begin{array}{c}\hfill \frac{1}{\rho }log{\mathbb{E}}_{P_n}\left[G_{Q_G}{\left(X\right)}^{\rho }\right]\le H_{\alpha }\left(P\right)+I_{\alpha }(P,Q_G).\end{array}$$

Thus, the above two propositions can be combined to state the following, which is analogous to Proposition 5 (refer Section 3.2).

Theorem 6 of [5]: Let G be an arbitrary guessing function and $G_P$ be the optimal guessing function for P. For $\rho \in (-1,0)\cup (0,\infty )$, let

$$R_g(P,G,\rho ):=\frac{1}{\rho }log{\mathbb{E}}_P\left[G{\left(X\right)}^{\rho }\right]-\frac{1}{\rho }log{\mathbb{E}}_P\left[G_P{\left(X\right)}^{\rho }\right].$$

Then, there exists a probability distribution $Q_G$ such that

$$|R_g(P,G,\rho )-I_{\alpha }(P,Q_G)|\le log(1+lnm).$$

3.4. Memoryless Guessing

In memoryless guessing, the setup is similar to that of Arıkan’s guessing problem except that this time the guesser Alice comes up with guesses independent of her previous guesses. Let ${\widehat{X}}_1,{\widehat{X}}_2,\dots$ be Alice’s sequence of independent guesses according to a distribution $\widehat{P}$. The guessing function in this problem is defined as

$$\begin{array}{c}\hfill G_{\widehat{P}}\left(X\right):=inf\{i\ge 1:{\widehat{X}}_i=X\},\end{array}$$

that is, the number of guesses until a successful guess. Sundaresan [28], inspired by Arıkan’s result, showed that the minimum expected number of guesses required is $exp\{H_{\frac{1}{2}}\left(P\right)\}$, and the distribution that achieves this is surprisingly not the underlying distribution P, but the “tilted distribution” ${\widehat{P}}^{*}\left(x\right):=\sqrt{P\left(x\right)}/{\sum }_y\sqrt{P\left(y\right)}$.

Unlike in Arıkan’s guessing problem, Huleihel et al. [19] minimized what are called factorial moments, defined for $\rho \in {\mathbb{Z}}_{+}$ as

$$\begin{array}{c}\hfill V_{\widehat{P},\rho }\left(X\right)=\frac{1}{\rho !}{\prod }_{l=0}^{\rho -1}\left(G_{\widehat{P}}\left(X\right)+l\right).\end{array}$$

Huleihel et al. [19] (c.f. [20]) studied the following problem.

$$\begin{array}{c}\hfill \mathrm{Minimize}{\mathbb{E}}_P\left[V_{\widehat{P},\rho }\left(X\right)\right],\end{array}$$

over all $\widehat{P}\in \mathcal{P}$, where $\mathcal{P}$ is the probability simplex that is, $\mathcal{P}=\{{\left(P\left(x\right)\right)}_{x\in \mathcal{X}}:P\left(x\right)\ge 0,$${\sum }_{x}P\left(x\right)=1\}$. Let ${\widehat{P}}^{*}$ be the optimal solution of the above problem.

Theorem 1 of [19]: For any integer $\rho >0$, we have

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[V_{{\widehat{P}}^{*},\rho }\left(X\right)\right]=H_{\alpha }\left(P\right)$$

and ${\widehat{P}}^{*}\left(x\right)=P{\left(x\right)}^{\alpha }/Z_{P,\alpha }.$

 Proof. 

From [19], we know that

$$\begin{array}{c}\hfill {\mathbb{E}}_P\left[V_{\widehat{P},\rho }\left(X\right)\right]={\mathbb{E}}_P\left[\widehat{P}{\left(X\right)}^{-\rho }\right].\end{array}$$

(23)

Now, the result follows from Proposition 1 with $\psi \left(x\right)=\widehat{P}{\left(x\right)}^{-1}$ and $k=1$. Indeed, since $\widehat{P}$ is a probability distribution, we have ${\sum }_{x\in \mathcal{X}}\psi {\left(x\right)}^{-1}={\sum }_{x\in \mathcal{X}}\widehat{P}\left(x\right)=1$. Hence, $\frac{1}{\rho }log{\mathbb{E}}_P\left[V_{\widehat{P},\rho }\left(X\right)\right]\ge H_{\alpha }\left(P\right)$, and the lower bound is attained by ${\widehat{P}}^{*}\left(x\right)=P{\left(x\right)}^{\alpha }/Z_{P,\alpha }$. □

For a sequence of guesses, the above theorem can be stated in the following way. Let ${\widehat{X}}^n=({\widehat{X}}_1,\dots ,{\widehat{X}}_n$), where ${\widehat{X}}_i$’s are i.i.d. guesses, drawn from ${\mathcal{X}}^n$ with distribution ${\widehat{P}}_n$—the n-fold product distribution of $\widehat{P}$ on ${\mathcal{X}}^n$. If the true underlying distribution is $P_n$, then

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[V_{{\widehat{P}}_n^{*}\rho }\left(X^n\right)\right]=H_{\alpha }\left(P\right),\end{array}$$

where ${\widehat{P}}_n^{*}\left(x\right)=P_n{\left(x\right)}^{\alpha }/Z_{P_n,\alpha }$. For the mismatched case, we have the following result.

 Proposition 7. 

If the true underlying probability distribution is P, but Alice assumes it as Q and guesses according to its optimal one, namely ${\widehat{Q}}^{*}\left(x\right)=Q{\left(x\right)}^{\alpha }/Z_{Q,\alpha }$, then

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[V_{{\widehat{Q}}^{*},\rho }\left(X\right)\right]=H_{\alpha }\left(P\right)+I_{\alpha }(P,Q).$$

 Proof. 

Due to (23), the result follows easily by taking $n=1$, ${\psi }_1\left(x\right)={\widehat{Q}}^{*}{\left(x\right)}^{-1}$, $c_1=1,$ $a_1=1$ in Propositions 3 and 4. □

3.5. Tasks Partitioning Problem

Encoding of Tasks problem studied by Bunte and Lapidoth [8] can be phrased in the following way. Let $\mathcal{X}$ be a finite set of tasks. A task X is randomly drawn from $\mathcal{X}$ according to a probability distribution P, which may correspond to the frequency of occurrences of tasks. Suppose these tasks are associated with M keys. Typically, $M<\left|\mathcal{X}\right|$. Due to a limited availability of keys, more than one task may be associated with a single key. When a task needs to be performed, the key associated with it is pressed. Consequently, all tasks associated with this key will be performed. The objective in this problem is to minimize the number of redundant tasks performed. Usual coding techniques suggest assigning tasks with high probability to individual keys and leaving the low probability tasks unassigned. It may just be the case that some tasks may have a higher frequency of occurrence than others. However, for an individual, all tasks can be equally important. If $M\ge \left|\mathcal{X}\right|$, then one can perform tasks without any redundancy. However, Bunte and Lapidoth [8] showed that, even when $M<\left|\mathcal{X}\right|$, one can accomplish the tasks with much less redundancy on average, provided the underlying probability distribution is different from the uniform distribution.

Let $\mathcal{A}=\{A_1,A_2,\dots ,A_M\}$ be a partition of $\mathcal{X}$ that corresponds to the assignment of tasks to M keys. Let $A\left(x\right)$ be the cardinality of the subset containing x in the partition. We shall call A the partition function associated with partition $\mathcal{A}$. We shall assume that $\rho >0$ throughout this section, though some of the results hold even when $\rho \in (-1,0)$.

Theorem I.1 of [8]: The following results hold.

(a)

For any partition of $\mathcal{X}$ of size M with partition function A, we have

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[A{\left(X\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-logM.$$

(b)

If $M>log\left|\mathcal{X}\right|+2$, then there exists a partition of $\mathcal{X}$ of size at most M with partition function A such that

$$1\le {\mathbb{E}}_P\left[\widehat{A}{\left(X\right)}^{\rho }\right]\le 1+2^{\rho (H_{\alpha }\left(P\right)-log\tilde{M})},$$

where

$$\begin{array}{c}\hfill \tilde{M}:=(M-log|\mathcal{X}|-2)/4.\end{array}$$

(24)

 Proof. 

Part (a): Let $\psi \left(x\right)=A\left(x\right)$. Then, we have ${\sum }_{x\in \mathcal{X}}\psi {\left(x\right)}^{-1}={\sum }_{x\in \mathcal{X}}A{\left(x\right)}^{-1}=M$ (Prop. III-1 of [8]). Now, an application of Proposition 1 with $k=M$ gives us the desired result.

Part (b): For the proof of this part, we refer to [8]. □

Bunte and Lapidoth also proved the following limit results.

Theorem I.2 of [8]: Then, for every $n\ge 1$, there exists a partition ${\mathcal{A}}_n$ of ${\mathcal{X}}^n$ of size at most $M^n$ with an associated partition function $A_n$ such that

$$\underset{n\to \infty }{lim}{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]=\left\{\begin{array}{cc}1\hfill & \mathrm{if}logM>H_{\alpha }\left(P\right)\hfill \\ \infty \hfill & \mathrm{if}logM<H_{\alpha }\left(P\right),\hfill \end{array}\right.$$

where $X^n:=(X_1,\cdots ,X_n)$.

It should be noted that, in a general set-up of the tasks partitioning problem, it is not necessary that the partition size is of the form $M^n$; it can be some $M_n$ (a function of n). Consequently, we have the following result.

 Proposition 8. 

Let $\left\{M_n\right\}$ be a sequence of positive integers such that $M_n\ge nlog\left|\mathcal{X}\right|+3$, and

$$\gamma :=\underset{n\to \infty }{lim}\frac{log{M}_n}{n}$$

exists. Then, there exists a sequence of partitions of ${\mathcal{X}}^n$ of size at most $M_n$ with partition functions $A_n$ such that

(a) 

$$\underset{n\to \infty }{lim}{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]=1\mathrm{if}\gamma >H_{\alpha }\left(P\right),$$

(b) 

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right)-\gamma \mathrm{if}\gamma <H_{\alpha }\left(P\right).$$

 Proof. 

Let

$$\begin{array}{c}\hfill {\tilde{M}}_n:=(M_n-nlog\left|\mathcal{X}\right|-2)/4.\end{array}$$

(25)

We first claim that ${lim}_{n\to \infty }\frac{log{\tilde{M}}_n}{n}=\gamma$. Indeed, since $\frac{log(1/4)}{n}\le \frac{log{\tilde{M}}_n}{n}<\frac{log{M}_n}{n}$, when $\gamma =0$, we have ${lim}_{n\to \infty }\frac{log{\tilde{M}}_n}{n}=0$. On the other hand, when $\gamma >0$, we can find an $n_{\gamma }$ such that $M_n\ge 2^{\gamma n/2}\forall n\ge n_{\gamma }$. Thus, we have ${lim}_{n\to \infty }\frac{n}{M_n}=0$. Consequently,

$$\underset{n\to \infty }{lim}\frac{log{\tilde{M}}_n}{n}=\underset{n\to \infty }{lim}\frac{log{M}_n}{n}+\underset{n\to \infty }{lim}\frac{1}{n}log\left(1-\frac{(nlog|\mathcal{X}|+2)}{M_n}\right)-\underset{n\to \infty }{lim}\frac{2}{n}=\gamma .$$

This proves the claim. From Theorem I.1 of [11], for any $n\ge 1$ and $M_n>nlog\left|\mathcal{X}\right|+2$, there exists a partition ${\mathcal{A}}_n$ of ${\mathcal{X}}^n$ of size at most $M_n$ such that the associated partition function $A_n$ satisfies

$$\begin{array}{cc}\hfill {\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]& \le 1+2^{\rho (H_{\alpha }\left(P_n\right)-log{\tilde{M}}_n)}=1+2^{n\rho \left(H_{\alpha }\left(P\right)-\frac{log{\tilde{M}}_n}{n}\right)}.\hfill \end{array}$$

Part (a): When $\gamma >H_{\alpha }\left(p\right)$, let us choose $ϵ=(\gamma -H_{\alpha }\left(P\right))/2>0$. Then, there exists an $n_{ϵ}$ such that $\frac{log{\tilde{M}}_n}{n}\ge \gamma -ϵ\forall n\ge n_{ϵ}$. Thus, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]\le 1+2^{n\rho (H_{\alpha }\left(P\right)-\gamma +ϵ)}=1+2^{-n\rho (\gamma -H_{\alpha }\left(P\right))/2}\forall n\ge n_{ϵ}\end{array}$$

Consequently, ${lim\; sup}_{n\to \infty }{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]\le 1$. We also note that $A_n\left(x^n\right)\ge 1$ for all $x^n\in {\mathcal{X}}^n$.

Thus, ${lim\; inf}_{n\to \infty }{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]\ge 1$.

Part (b): For any $ϵ>0$, there exists an $n_{ϵ}$ such that $\frac{log{\tilde{M}}_n}{n}\ge \gamma -ϵ\forall n\ge n_{ϵ}$. Thus, we have

$$\begin{array}{cc}\hfill {\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]& \le 1+2^{n\rho (H_{\alpha }\left(P\right)-\gamma +ϵ)}\le 2^{1+n\rho (H_{\alpha }\left(P\right)-\gamma +ϵ)}\forall n\ge n_{ϵ}\hfill \end{array}$$

Hence, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]\le H_{\alpha }\left(P\right)-\gamma +ϵ\forall ϵ>0$$

Furthermore, an invocation of Corollary 1 with ${\psi }_n\left(x^n\right)=A_n\left(x^n\right)$ and $k_n={\sum }_{x^n\in {\mathcal{X}}^n}1/A_n\left(x^n\right)=M_n$ gives us

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-\underset{n\to \infty }{lim\; sup}\frac{log{M}_n}{n}=H_{\alpha }\left(P\right)-\gamma .$$

□

 Remark 1. 

It is interesting to note that, when $\gamma <H_{\alpha }\left(P\right)$, in addition to the fact that ${lim}_{n\to \infty }{\mathbb{E}}_{P_n}$$\left[A_n{\left(X^n\right)}^{\rho }\right]=\infty$, we also have ${\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]\approx 2^{n\rho (H_{\alpha }\left(P\right)-\gamma )}$ for large values of n.

Mismatch Case:

Let us now suppose that one does not know the true underlying probability distribution P, but arbitrarily partitions $\mathcal{X}$. Then, the penalty due to such a partition can be measured by the $I_{\alpha }$-divergence as stated in the following theorem.

 Proposition 9. 

Let $\mathcal{A}$ be a partition of $\mathcal{X}$ of size M with partition function A. Then, there exists a probability distribution $Q_A$ on $\mathcal{X}$ such that

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[A{\left(X\right)}^{\rho }\right]=H_{\alpha }\left(P\right)+I_{\alpha }(P,Q_A)-logM.$$

 Proof. 

Define a probability distribution $Q_A=\{Q_A\left(x\right),x\in \mathcal{X}\}$ as

$$Q_A\left(x\right):=\frac{A{\left(x\right)}^{-1/\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}A{\left(x^{\prime }\right)}^{-1/\alpha }}·$$

Then,

$$\frac{Z_{Q_A,\alpha }}{Q_A{\left(x\right)}^{\alpha }}=A\left(x\right){\sum }_{x^{\prime }\in \mathcal{X}}\frac{1}{A\left(x^{\prime }\right)}=A\left(x\right)·M,$$

where the last equality follows due to Proposition III.1 of [8]. Rearranging terms, we have $A\left(x\right)=\frac{Z_{Q_A,\alpha }}{M·Q_A{\left(x\right)}^{\alpha }}$. Hence, an application of Propositions 3 and 4 with $n=1$, ${\psi }_1\left(x\right)=A\left(x\right)$, $c_1=1/M$, $a_1=1/M$, and $Q=Q_A$ yields the desired result. □

A converse result is the following.

 Proposition 10. 

Let X be a random task from $\mathcal{X}$ following distribution P and $\rho \in (0,\infty )$. Let Q be another distribution on $\mathcal{X}$. If $M>log\left|\mathcal{X}\right|+2$, then there exists a partition ${\mathcal{A}}_Q$ (with an associated partition function $A_Q$) of $\mathcal{X}$ of size at most M such that

$${\mathbb{E}}_P\left[A_Q{\left(X\right)}^{\rho }\right]\le 1+2^{\rho (H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)-log\tilde{M})},$$

where $\tilde{M}$ is as in (24).

 Proof. 

Similar to proof of Theorem I.1 of [8]. □

4. Ordered Tasks Partitioning Problem

In Bunte–Lapidoth’s tasks partitioning problem [8], one is interested in the average number of tasks associated with a key. However, in some scenarios, it might be more important to minimize the average number of redundant tasks performed, before the intended task. To achieve this, tasks associated with a key should be performed in a decreasing order of their probabilities. With such a strategy in place, this problem draws parallel with Arıkan’s guessing problem [13].

Let $\mathcal{A}=\{{\mathcal{A}}_1,{\mathcal{A}}_2,\dots ,{\mathcal{A}}_M\}$ be a partition of $\mathcal{X}$ that corresponds to the assignment of tasks to M keys. Let $N\left(x\right)$ be the number of redundant tasks performed until and including the intended task x. We refer to $N(·)$ as the count function associated with partition $\mathcal{A}$. We suppress the dependence of N on $\mathcal{A}$ for the sake of notational convenience. If X denotes the intended task, then we are interested in the ${\rho }^{\mathrm{th}}$ moment of number of tasks performed, that is, ${\mathbb{E}}_P\left[N{\left(X\right)}^{\rho }\right]$, where $\rho >0$.

 Lemma 1. 

For any count function associated with a partition of size M, we have

$$\sum _{x\in \mathcal{X}}\frac{1}{N\left(x\right)}\le M\left[1+ln\left(\frac{\left|\mathcal{X}\right|}{M}\right)\right].$$

(26)

 Proof. 

For a partition $\mathcal{A}=\{A_1,A_2,\cdots ,A_M\}$ of $\mathcal{X}$, observe that

$$\sum _{x\in \mathcal{X}}\frac{1}{N\left(x\right)}=\left(1+\frac{1}{2}+\cdots +\frac{1}{|A_1|}\right)+\cdots +\left(1+\frac{1}{2}+\cdots +\frac{1}{|A_M|}\right).$$

(27)

Since $1+\frac{1}{2}+\cdots +\frac{1}{|A_k|}\le 1+ln\left|A_k\right|$, for any $k\in \{1,\cdots ,M\}$, we have

$$\begin{array}{cc}\hfill \sum _{x\in \mathcal{X}}\frac{1}{N\left(x\right)}& \le M+ln\left(\right|A_1|\cdots |A_M\left|\right)=M[1+ln(|A_1|\cdots |A_M{\left|\right)}^{1/M}]\hfill \\ \hfill & \stackrel{\left(a\right)}{\le }M\left[1+ln\left(\frac{|A_1|+\cdots +|A_M|}{M}\right)\right]=M\left[1+ln\left(\frac{\left|\mathcal{X}\right|}{M}\right)\right],\hfill \end{array}$$

(28)

where (a) follows due to the AM–GM inequality. □

 Proposition 11. 

Let X be a random task from $\mathcal{X}$ following distribution P. Then, the following hold:

(a)

For any partition of $\mathcal{X}$ of size M, we have

$$\begin{array}{c}\hfill \frac{1}{\rho }log{\mathbb{E}}_P\left[N{\left(X\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-log\left\{M[1+ln\left(\left|\mathcal{X}\right|/M\right)]\right\}\end{array}$$

(29)

(b)

Let $M>log\left|\mathcal{X}\right|+2$. Then, there exists a partition of $\mathcal{X}$ of size at most M with count function N such that

$$1\le {\mathbb{E}}_P\left[N{\left(X\right)}^{\rho }\right]\le 1+2^{\rho (H_{\alpha }\left(P\right)-log\tilde{M})},$$

where $\tilde{M}$ is as in (24).

 Proof. 

Part (a): Applying Proposition 1 with $k=M\left[1+ln\left(\left|\mathcal{X}\right|/M\right)\right]$ and $\psi \left(x\right)=N\left(x\right)$, we obtain the desired result.

Part (b): If A and N are, respectively, the partition and count functions of a partition $\mathcal{A}$, then we have $1\le N\left(x\right)\le A\left(x\right)$ for $x\in \mathcal{X}$. Once we observe this, the proof is same as Theorem I.1 (b) of [8]. □

 Proposition 12. 

Let $\left\{M_n\right\}$ be a sequence of positive integers such that $M_n\ge nlog\left|\mathcal{X}\right|+3$, and $\gamma :={lim}_{n\to \infty }log{M}_n/n$ exists. Then, there exists a sequence of partitions of ${\mathcal{X}}^n$ of size at most $M_n$ with count functions $N_n$ such that

(a) 

$$\underset{n\to \infty }{lim}{\mathbb{E}}_{P_n}\left[N_n{\left(X^n\right)}^{\rho }\right]=1\mathrm{if}\gamma >H_{\alpha }\left(P\right),$$

(b) 

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[N_n{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right)-\gamma \mathrm{if}\gamma <H_{\alpha }\left(P\right).$$

 Proof. 

Similar to proof of Proposition 8. □

Remark 2. 

(a) 

If we choose the trivial partition, namely ${\mathcal{A}}_n=\left\{{\mathcal{X}}^n\right\}$, then the ordered tasks partitioning problem simplifies to Arıkan’s guessing problem, that is, we have $M_n=1$, $N_n\left(x^n\right)=G_n\left(x^n\right)$ and (26) simplifies to

$$\sum _{x^n\in {\mathcal{X}}^n}\frac{1}{G_n\left(x^n\right)}\le 1+nln\left|\mathcal{X}\right|.$$

Hence, all results pertaining to the Arıken’s guessing problem can be derived from the ordered tasks partitioning problem.

(b) 

Structurally, ordered tasks partitioning problem differs from the Bunte–Lapidoth’s problem only due the factor $1+ln\left(\right|\mathcal{X}|/M)$ in (28). While this factor matters for one-shot results, for a sequence of i.i.d. tasks, this factor vanishes asymptotically.

Mismatch Case:

Let us now suppose that one does not know the true underlying probability distribution P, but arbitrarily partitions $\mathcal{X}$ and executes tasks within each subset of this partition in an arbitrary order. Then, the penalty due to such a partition and ordering can be measured by the $I_{\alpha }$-divergence as stated in the following propositions.

 Proposition 13. 

Let $\mathcal{A}$ be a partition of $\mathcal{X}$ of size M with count function N. Then, there exists a probability distribution $Q_N$ on $\mathcal{X}$ such that

$$\frac{1}{\rho }log{\mathbb{E}}_P\left[N{\left(X\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)+I_{\alpha }(P,Q_N)-log\left\{M\left[1+ln\left(\left|\mathcal{X}\right|/M\right)\right]\right\}.$$

 Proof. 

Define a probability distribution $Q_N=\{Q_N\left(x\right),x\in \mathcal{X}\}$ as

$$Q_N\left(x\right):=\frac{N{\left(x\right)}^{-1/\alpha }}{{\sum }_{x^{\prime }\in \mathcal{X}}N{\left(x^{\prime }\right)}^{-1/\alpha }}·$$

Then, by Lemma 1, we have

$$\begin{array}{ccc}\hfill \frac{Z_{Q_N,\alpha }}{Q_N{\left(x\right)}^{\alpha }}& =& N\left(x\right){\sum }_{x^{\prime }\in \mathcal{X}}\frac{1}{N\left(x^{\prime }\right)}\le N\left(x\right)·M[1+ln(\left|\mathcal{X}\right|/M)].\hfill \end{array}$$

Now, an application of Proposition 4 with $n=1$, ${\psi }_1\left(x\right)=N\left(x\right)$, $Q=Q_N$, and $a_1=1/M[1+ln(\left|\mathcal{X}\right|/M\left)\right]$ yields the desired result. □

A converse result is the following.

 Proposition 14. 

Let X be a random task from $\mathcal{X}$ following distribution P. Let Q be another distribution on $\mathcal{X}$. If $M>log\left|\mathcal{X}\right|+2$, then there exists partition ${\mathcal{A}}_Q$ (with an associated count function $N_Q$) of $\mathcal{X}$ of size at most M such that

$${\mathbb{E}}_P\left[N_Q{\left(X\right)}^{\rho }\right]\le 1+2^{\rho (H_{\alpha }\left(P\right)+I_{\alpha }(P,Q)-log\tilde{M})}\mathrm{if}\rho \in (0,\infty ),$$

where $\tilde{M}$ is as in (24).

 Proof. 

Identical to the proof of Proposition 11(b). □

5. Operational Connection among the Problems

In this section, we establish an operational relationship among the five problems (refer Figure 1) that we studied in the previous section. The relationship we are interested in is “Does knowing an optimal or asymptotically optimal solution in one problem helps us find the same in another?” In fact, we end up showing that, under suitable conditions, all the five problems form an equivalence class with respect to the above-mentioned relation.

Figure 1. Relationships established among the five problems. A directed arrow from problem A to problem B means knowing optimal or asymptotically optimal solution of A helps us find the same in B.

In this section, we assume $\rho >0$. First, we make the following observations:

• Among the five problems discussed in the previous section, only Arıkan’s guessing and Huleihel et al.’s memoryless guessing have a unique optimal solution; others only have asymptotically optimal solutions.
• Optimal solution of Huleihel et al.’s memoryless guessing problem is the $\alpha$-scaled measure of the underlying probability distribution P. Hence, knowledge about the optimal solution of this problem implies knowledge about an optimal (or asymptotically optimal) solution of all other problems.
• Among the Bunte–Lapidoth’s and ordered tasks problems, an asymptotically optimal solution of one yields that of the other. The partitioning lemma (Prop. III-2 of [8]) is the key result in these two problems, as it guarantees the existence of the asymptotically optimal partitions in both these problems.

5.1. Campbell’s Coding and Arıkan’s Guessing

An attempt to find a close relationship between these two problems was made, for example, by Hanawal and Sundaresan (Section II of [17]). Here, we show the equivalence between asymptotically optimal solutions of these two problems.

 Proposition 15. 

An asymptotically optimal solution exists for Campbell’s source coding problem if and only if an asymptotically optimal solution exists for Arıkan’s guessing problem.

 Proof. 

Let $\left\{G_n^{*}\right\}$ be an asymptotically optimal sequence of guessing functions, that is,

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_n^{*}{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right).$$

Define

$$Q_{G_n^{*}}\left(x^n\right):=c_n^{-1}·{G_n^{*}\left(x^n\right)}^{-1},$$

(30)

where $c_n$ is the normalization constant. Notice that

$$\begin{array}{c}\hfill c_n=\sum _{x^n}{G_n^{*}\left(x^n\right)}^{-1}\le 1+nln\left|\mathcal{X}\right|.\end{array}$$

(31)

Let us now define

$$L_{G_n^{*}}\left(x^n\right):=\lceil -log{Q}_{G_n^{*}}\left(x^n\right)\rceil .$$

Then, by (Proposition 1 of [17]),

$$L_{G_n^{*}}\left(x^n\right)\le log{G}_n^{*}\left(x^n\right)+1+log{c}_n.$$

Hence,

$$\begin{array}{c}\hfill 2^{\rho L_{G_n^{*}}\left(x^n\right)}\le 2^{\rho }·c_n^{\rho }·G_n^{*}{\left(x^n\right)}^{\rho }\le 2^{\rho }·{(1+nln|\mathcal{X}\left|\right)}^{\rho }·G_n^{*}{\left(x^n\right)}^{\rho }.\end{array}$$

Thus, we have

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L_{G_n^{*}}\left(X^n\right)}\right]\le \underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_n^{*}{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right).$$

We observe that

$$\begin{array}{c}\hfill \sum _{x^n\in {\mathcal{X}}^n}{2}^{-L_{G_n^{*}}\left(x^n\right)}=\sum _{x^n\in {\mathcal{X}}^n}{2}^{-\lceil -log{Q}_{G_n^{*}}\left(x^n\right)\rceil }\le \sum _{x^n\in {\mathcal{X}}^n}{2}^{log{Q}_{G_n^{*}}\left(x^n\right)}=1.\end{array}$$

Consequently, from (12), we have

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L_{G_n^{*}}\left(X^n\right)}\right]\ge H_{\alpha }\left(P\right).$$

Thus, $\left\{L_{G_n^{*}}\right\}$ is an asymptotically optimal sequence of length functions for Campbell’s coding problem.

Conversely, given an asymptotically optimal sequence of length functions $\left\{L_n^{*}\right\}$ for Campbell’s coding problem, define

$$Q_{L_n^{*}}\left(x^n\right):=\frac{2^{-L_n^{*}\left(x^n\right)}}{{\sum }_{y^n}{2}^{-L_n^{*}\left(x^n\right)}}.$$

Let $G_{L_n^{*}}$ be the guessing function on ${\mathcal{X}}^n$ that guesses according to the decreasing order of $Q_{L_n^{*}}$-probabilities. Then, by (Proposition 2 of [17]),

$$log{G}_{L_n^{*}}\left(x^n\right)\le L_n^{*}\left(x^n\right).$$

Thus,

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_{L_n^{*}}{\left(X^n\right)}^{\rho }\right]\le \underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L_n^{*}\left(X^n\right)}\right]=H_{\alpha }\left(P\right).$$

Furthermore, from Theorem 1 of [13], we have

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_{L_n^{*}}{\left(X^n\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-\underset{n\to \infty }{lim\; sup}\frac{1}{n}log(1+nln|\mathcal{X}\left|\right)=H_{\alpha }\left(P\right).$$

This completes the proof. □

5.2. Arıkan’s Guessing and Bunte–Lapidoth’s Tasks Partitioning Problem

Bracher et al. found a close connection between Arıkan’s guessing problem and Bunte–Lapidoth’s tasks partitioning problem in the context of distributed storage [29]. In this section, we establish a different relation between these problems.

 Proposition 16. 

An asymptotically optimal solution of Arıkan’s guessing problem gives rise to an asymptotically optimal solution of tasks partitioning problem.

 Proof. 

Let $\left\{G_n^{*}\right\}$ be an asymptotically optimal sequence of guessing functions. Define

$$Q_{G_n^{*}}\left(x^n\right)=d_n^{-1}{G}_n^{*}{\left(x^n\right)}^{-1/\alpha },$$

where $d_n$ is the normalization constant. Let $A_{G_n^{*}}$ be the partition function satisfying $A_{G_n^{*}}\left(x^n\right)\le \lceil {\beta }_n·Z_{Q_{G_n^{*}},\alpha }/Q_{G_n^{*}}{\left(x^n\right)}^{\alpha }\rceil$ guaranteed by (Proposition III-2 of [8]), where

$${\beta }_n=\frac{2}{M_n-nlog\left|\mathcal{X}\right|-2}·$$

Thus, we have

$$\begin{array}{ccc}\hfill A_{G_n^{*}}{\left(x^n\right)}^{\rho }\le {\lceil {\beta }_n·Z_{Q_{G_n^{*}},\alpha }/Q_{G_n^{*}}{\left(x^n\right)}^{\alpha }\rceil }^{\rho }& \stackrel{\left(a\right)}{\le }& \lceil {\beta }_n·(1+nln|\mathcal{X}\left|\right)·G_n^{*}\left(x^n\right){\rceil }^{\rho }\hfill \\ & \stackrel{\left(b\right)}{\le }& 1+2^{\rho }{\beta }_n^{\rho }·{(1+nln|\mathcal{X}\left|\right)}^{\rho }·{G_n^{*}\left(x^n\right)}^{\rho },\hfill \end{array}$$

where (a) holds because $Z_{Q_{G_n^{*}},\alpha }=d_n^{-\alpha }{\sum }_{x^n\in {\mathcal{X}}^n}{G}_n^{*}{\left(x^n\right)}^{-1}\le d_n^{-\alpha }(1+nln|\mathcal{X}\left|\right)$; and (b) hold because ${\lceil x\rceil }^{\rho }\le 1+2^{\rho }{x}^{\rho }$ for $x>0$. Hence,

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{P_n}\left[A_{G_n^{*}}{\left(X^n\right)}^{\rho }\right]& \le & 1+2^{\rho }{\beta }_n^{\rho }·{(1+nln|\mathcal{X}\left|\right)}^{\rho }·{\mathbb{E}}_{P_n}\left[G_n^{*}{\left(X^n\right)}^{\rho }\right]\hfill \\ & \stackrel{\left(c\right)}{\le }& 1+2^{2\rho }{\left(\frac{1+nln\left|\mathcal{X}\right|}{M_n-nlog\left|\mathcal{X}\right|-2}\right)}^{\rho }·2^{n\rho H_{\alpha }\left(P\right)}\hfill \\ & =& 1+2^{n\rho \left(H_{\alpha }\left(P\right)+\frac{log(1+nln|\mathcal{X}\left|\right)}{n}-\frac{log{\tilde{M}}_n}{n}\right)},\hfill \end{array}$$

where ${\tilde{M}}_n$ is as in (25); and inequality (c) follows from Proposition 4 of [13] proved in Section 3. Thus, if $M_n$ is such that $M_n\ge nlog\left|\mathcal{X}\right|+3$ and if $\gamma :={lim}_{n\to \infty }(log{M}_n)/n$ exists and $\gamma >H_{\alpha }\left(P\right)$, then we have

$$\underset{n\to \infty }{lim\; sup}{\mathbb{E}}_{P_n}\left[A_{G_n^{*}}{\left(X^n\right)}^{\rho }\right]\le 1.$$

Since ${\mathbb{E}}_{P_n}\left[A_{G_n^{*}}{\left(X^n\right)}^{\rho }\right]\ge 1$, we have ${lim\; inf}_{n\to \infty }{\mathbb{E}}_{P_n}\left[A_{G_n^{*}}{\left(X^n\right)}^{\rho }\right]\ge 1$. When $\gamma <H_{\alpha }\left(P\right)$, arguing along the lines of proof of Proposition 8(b), it can be shown than

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[A_{G_n^{*}}{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right)-\gamma .$$

□

Reverse implication of the above result does not hold always due to the additional parameter $M_n$ in the tasks partitioning problem. For example, if $M_n={\left|\mathcal{X}\right|}^n$ and $A_n\left(x^n\right)=1$ for every $x^n$, the partition does not provide any information about the underlying distribution. As a consequence, we will not be able to conclude anything about the optimal (or asymptotically optimal) solutions of other problems. However, if $M_n$ is such that $log{M}_n$ increases sub-linearly, then it does help us find asymptotically optimal solutions of other problems.

 Proposition 17. 

An asymptotically optimal sequence of partition functions $\left\{A_n\right\}$ with partition sizes $\left\{M_n\right\}$ for the tasks partitioning problem gives rise to an asymptotically optimal solution for the guessing problem if $M_n\ge nlog\left|\mathcal{X}\right|+3$ and ${lim}_{n\to \infty }(log{M}_n)/n=0.$

 Proof. 

By hypothesis,

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right).$$

For every $A_n$, define the probability distribution

$$Q_{A_n}\left(x^n\right):=c_n^{-1}{A}_n{\left(x^n\right)}^{-1},$$

where $c_n:={\sum }_{x^n}{A}_n{\left(x^n\right)}^{-1}=M_n$. Let $G_{A_n}^{*}$ be the guessing function that guesses according to the decreasing order of $Q_{A_n}$-probabilities. Then, by (Proposition 2 of [17]), we have

$$\begin{array}{cc}\hfill G_{A_n}^{*}\left(x^n\right)\le & Q_{A_n}{\left(x^n\right)}^{-1}=c_n{A}_n\left(x^n\right)=M_n{A}_n\left(x^n\right).\end{array}$$

Hence,

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_{A_n}^{*}{\left(X^n\right)}^{\rho }\right]\le \underset{n\to \infty }{lim\; sup}\frac{1}{n}log{M}_n+\underset{n\to \infty }{lim\; sup}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[A_n{\left(X^n\right)}^{\rho }\right]=H_{\alpha }\left(P\right).$$

Furthermore, an application of Theorem 1 of [13] gives us

$$\underset{n\to \infty }{lim\; inf}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[G_{A_n}^{*}{\left(X^n\right)}^{\rho }\right]\ge H_{\alpha }\left(P\right)-\underset{n\to \infty }{lim\; sup}\frac{1}{n}log(1+nln|\mathcal{X}\left|\right)=H_{\alpha }\left(P\right).$$

This completes the proof. □

5.3. Huleihel et al.’s Memoryless Guessing and Campbell’s Coding

We already know that, if one knows the optimal solution of Huleihel et al.’s memoryless guessing problem, that is, the $\alpha$-scaled measure of the underlying probability distribution P, one has knowledge about the optimal (or asymptotically optimal) solution of Campbell’s coding problem. In this section, we prove a converse statement. We first prove the following lemma.

 Lemma 2. 

Let $L_n^{*}$ denote the length function corresponding to an optimal solution for Campbell’s coding problem on the alphabet set ${\mathcal{X}}^n$ endowed with the product distribution $P_n$. Then, ${\sum }_{x^n\in {\mathcal{X}}^n}{2}^{-L_n^{*}\left(x^n\right)}\ge 1/2$.

 Proof. 

Suppose ${\sum }_{x^n\in {\mathcal{X}}^n}{2}^{-L_n^{*}\left(x^n\right)}<1/2$. Then, we must have $L_n^{*}\left(x^n\right)\ge 2$ for every $x^n\in {\mathcal{X}}^n$. Define ${\widehat{L}}_n\left(x^n\right):=L_n^{*}\left(x^n\right)-1$. We observe that ${\sum }_{x^n\in {\mathcal{X}}^n}{2}^{-{\widehat{L}}_n\left(x^n\right)}<1$, that is, the length function ${\widehat{L}}_n(·)$ satisfies (11). Hence, there exists a code $C_n$ for ${\mathcal{X}}^n$ such that $L\left(C_n\left(x^n\right)\right)={\widehat{L}}_n\left(x^n\right)$. Then, for $\rho >0$, we have $log{\mathbb{E}}_{P_n}[2^{\rho L_n^{*}\left(X^n\right)}]>log{\mathbb{E}}_{P_n}[2^{\rho {\widehat{L}}_n\left(X^n\right)}]$—a contradiction. □

 Proposition 18. 

An asymptotically optimal solution for Huleihel et al.’s memoryless guessing problem exists if an asymptotically optimal solution exists for Campbell’s coding problem.

 Proof. 

Let $\{L_n^{*},n\ge 1\}$ denote a sequence of asymptotically optimal length functions of Campbell’s coding problem, that is,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}[2^{\rho L_n^{*}\left(X^n\right)}]=H_{\alpha }\left(P\right).\end{array}$$

(32)

Let us define

$$Q_{L_n^{*}}\left(x^n\right):=\frac{2^{-L_n^{*}\left(x^n\right)/\alpha }}{{\sum }_{{\overline{x}}^n\in {\mathcal{X}}^n}{2}^{-L_n^{*}\left({\overline{x}}^n\right)/\alpha }}.$$

Then, we have

$$\begin{array}{cc}\hfill I_{\alpha }(P_n,Q_{L_n^{*}})& =\frac{\alpha }{1-\alpha }\log \left({\sum }_{x^n\in {\mathcal{X}}^n}{P}_n\left(x^n\right){\left[{\sum }_{{\widehat{x}}^n\in {\mathcal{X}}^n}{\left(\frac{Q_{L_n^{*}}\left({\widehat{x}}^n\right)}{Q_{L_n^{*}}\left(x^n\right)}\right)}^{\alpha }\right]}^{\frac{1-\alpha }{\alpha }}\right)-H_{\alpha }\left(P_n\right)\hfill \\ \hfill & =\frac{\alpha }{1-\alpha }\log \left({\sum }_{x^n\in {\mathcal{X}}^n}{P}_n\left(x^n\right){\left[{\sum }_{{\widehat{x}}^n\in {\mathcal{X}}^n}\frac{2^{-L_n^{*}\left({\widehat{x}}^n\right)}}{2^{-L_n^{*}\left(x^n\right)}}\right]}^{\rho }\right)-n{H}_{\alpha }\left(P\right)\hfill \\ \hfill & =\frac{1}{\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L_n^{*}\left(X^n\right)}\right]+log{\zeta }_n-n{H}_{\alpha }\left(P\right),\hfill \end{array}$$

where ${\zeta }_n={\sum }_{{\widehat{x}}^n\in {\mathcal{X}}^n}{2}^{-L_n^{*}\left({\widehat{x}}^n\right)}$. Hence,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}{I}_{\alpha }(P_n,Q_{L_n^{*}})=\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[2^{\rho L_n^{*}\left(X^n\right)}\right]+\underset{n\to \infty }{lim}\frac{1}{n}log{\zeta }_n-H_{\alpha }\left(P\right)\stackrel{\left(a\right)}{=}0,\end{array}$$

where (a) holds because ${\zeta }_n\in [1/2,1]$ (refer Lemma 2). If we assume the underlying probability distribution to be $Q_{L_n^{*}}$ instead of $P_n$, and perform memoryless guessing according to the escort distribution of $Q_{L_n^{*}}$, namely ${\widehat{Q}}_n^{*}\left(x^n\right)=Q_{L_n^{*}}{\left(x^n\right)}^{\alpha }/Z_{Q_{L_n^{*}},\alpha }$, due to Proposition 7, we have

$$\underset{n\to \infty }{lim}\frac{1}{n\rho }log{\mathbb{E}}_{P_n}\left[V_{{\widehat{Q}}_n^{*},\rho }\left(X^n\right)\right]=H_{\alpha }\left(P\right)+\underset{n\to \infty }{lim}\frac{1}{n}{I}_{\alpha }(P_n,Q_{L_n^{*}})=H_{\alpha }\left(P\right).$$

□

6. Summary and Conclusions

This paper was motivated by the need to find a unified framework for the problems on source coding, guessing and the tasks partitioning. To that end, we formulated a general moment minimization problem in the IID-lossless case and observed that the optimal value of its objective function is bounded below by Rényi entropy. We then re-established all achievable lower bounds in each of the above-mentioned problems using the generalized framework. It was interesting to note that the optimal solution did not depend on the moment function $\psi$, but only on the underlying probability distribution P and order of the moment $\rho$ (refer Proposition 1). We also presented a unified framework for the mismatched version of the above-mentioned problems. This framework not only led to refinement of the known theorems, but also helped us identify a few new results. We went on to extend the tasks partitioning problem by asking a more practical question and solved it using the unified theory. Finally, we established an equivalence among these problems, in the sense that an asymptotically optimal solution of one problem yields the asymptotically optimal solution of all other problems. Although the relationship between source coding and guessing is well-known in the literature [30,31,32,33], their connection to the tasks partitioning problem, and the connection in the mismatched version of the problems are new.

Our unified framework also has the potential to act as a general tool-set and provide insights for similar problems in information theory. For example, in Section 4, this framework enabled us to solve a more general tasks partitioning problem, namely, the ordered tasks partitioning problem using this framework. The presented that a unified approach can also be extended and explored further in several ways. This includes (a) Extension to general alphabet set: The guessing problem was originally studied for countably infinite alphabet set by Massey [12]. Courtade and Verdú have studied the source coding problem for a countably infinite alphabet set with a cumulant generating function of code word lengths as a design criterion [31]. It would be interesting to see if memory-less guessing and tasks partitioning problems can also be formulated for countably infinite alphabet sets and relationships among the problems can be extended. (b) More general sources: Relationship between source coding and guessing is very well-known in the literature. Relationship between guessing and source coding in the ‘with distortion’ case for finite alphabet was established by Merhav and Arıkan [14] and for a countably infinite alphabet by Hanawal and Sundaresan [34]. Relationship between Guessing and Campbell’s coding in the universal case was established by Sundaresan [35]. It would be interesting to see if these can be extended to memoryless guessing and tasks partitioning also, possibly in an unified manner. (c) Applications: Arıkan showed an application of the guessing problem in a sequential decoding problem [13]. Humblet showed that the cumulant of code-lengths arises in minimizing the probability of buffer overflow in source coding problems [27]. Rezaee et al. [36], Salamatian et al. [20], and Sundaresan [28] show the application of guessing in the security aspect of password protected systems. Our unified framework has the potential to help solve problems that arise in real life situations and fall in this framework.