#
Are Guessing, Source Coding and Tasks Partitioning Birds of A Feather?^{ †}

^{1}

^{2}

^{3}

^{4}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

- (a)
- a general framework for the problems on source coding, guessing and tasks partitioning;
- (b)
- lower and upper bounds for the general framework of these problems both in matched and mis-matched cases;
- (c)
- a unified approach to derive bounds for the mismatched version of these problems;
- (d)
- a generalized tasks partitioning problem; and
- (e)
- establishing operational commonality among the problems.

## 2. A General Minimization Problem

**Proposition 1.**

**Proof.**

**Proposition 2.**

**Proof.**

**Corollary 1.**

- (a)
- For $\rho \in (-1,0)\cup (0,\infty )$,$$\underset{n\to \infty}{lim\; inf}\frac{1}{n\rho}log{\mathbb{E}}_{{P}_{n}}\left[{\psi}_{n}{\left({X}^{n}\right)}^{\rho}\right]\ge {H}_{\alpha}\left(P\right)-\underset{n\to \infty}{lim\; sup}\frac{log{k}_{n}}{n}.$$
- (b)
- $$\underset{n\to \infty}{lim\; inf}\frac{1}{n}{\mathbb{E}}_{{P}_{n}}[log{\psi}_{n}\left({X}^{n}\right)]\ge H\left(P\right)-\underset{n\to \infty}{lim\; sup}\frac{log{k}_{n}}{n},$$

**Proof.**

#### A General Framework for Mismatched Cases

**Proposition 3.**

- (a)
- for $\rho \ne 0$, we have$$\mathit{sgn}\left(\rho \right)\xb7{\mathbb{E}}_{{P}_{n}}\left[{\psi}_{n}{\left({X}^{n}\right)}^{\rho}\right]\le \mathit{sgn}\left(\rho \right)\xb7{2}^{n\rho [{H}_{\alpha}\left(P\right)+{I}_{\alpha}(P,Q)+{n}^{-1}log{c}_{n}]},$$
- (b)
- for $\rho \ne 0$, we have$$\underset{n\to \infty}{lim\; sup}\frac{1}{n\rho}log{\mathbb{E}}_{{P}_{n}}\left[{\psi}_{n}{\left({X}^{n}\right)}^{\rho}\right]\le {H}_{\alpha}\left(P\right)+{I}_{\alpha}(P,Q)+\underset{n\to \infty}{lim\; sup}\frac{1}{n}log{c}_{n},$$
- (c)
- for $\rho =0$, we have$${\mathbb{E}}_{{P}_{n}}[log{\psi}_{n}\left({X}^{n}\right)]\le n[H\left(P\right)+I(P,Q)+{n}^{-1}log{c}_{n}],$$
- (d)
- for $\rho =0$, we have$$\underset{n\to \infty}{lim\; sup}\frac{1}{n}{\mathbb{E}}_{{P}_{n}}[log{\psi}_{n}\left({X}^{n}\right)]\le H\left(P\right)+I(P,Q)+\underset{n\to \infty}{lim\; sup}\frac{1}{n}log{c}_{n}.$$

**Proof.**

**Proposition 4.**

- (a)
- for $\rho \ne 0$, we have$$\mathit{sgn}\left(\rho \right)\xb7{\mathbb{E}}_{{P}_{n}}\left[{\psi}_{n}{\left({X}^{n}\right)}^{\rho}\right]\ge \mathit{sgn}\left(\rho \right)\xb7{2}^{n\rho ({H}_{\alpha}\left(P\right)+{I}_{\alpha}(P,Q)+{n}^{-1}log{a}_{n})},$$
- (b)
- for $\rho \ne 0$, we have$$\underset{n\to \infty}{lim\; inf}\frac{1}{n\rho}log{\mathbb{E}}_{{P}_{n}}\left[{\psi}_{n}{\left({X}^{n}\right)}^{\rho}\right]\ge {H}_{\alpha}\left(P\right)+{I}_{\alpha}(P,Q)+\underset{n\to \infty}{lim\; inf}\frac{1}{n}log{a}_{n},$$
- (c)
- for $\rho =0$, we have$${\mathbb{E}}_{{P}_{n}}[log{\psi}_{n}\left({X}^{n}\right)]\ge n(H\left(P\right)+I(P,Q)+{n}^{-1}log{a}_{n}),$$
- (d)
- for $\rho =0$, we have$$\underset{n\to \infty}{lim\; inf}\frac{1}{n}{\mathbb{E}}_{{P}_{n}}[log{\psi}_{n}\left({X}^{n}\right)]\ge H\left(P\right)+I(P,Q)+\underset{n\to \infty}{lim\; inf}\frac{1}{n}log{a}_{n}.$$

**Proof.**

## 3. Problem Statements and Known Results

#### 3.1. Source Coding Problem

**Proof.**

**Theorem 1.**

**Proof.**

#### 3.2. Campbell Coding Problem

**Proof.**

**Proof.**

**Mismatch Case**:

**Proposition 5.**

**Proof.**

#### 3.3. Arıkan’s Guessing Problem

**Proof.**

**Proof.**

**Proof.**

**Mismatch Case**:

**Proposition 6.**

**Proof.**

**Proof.**

#### 3.4. Memoryless Guessing

**Proof.**

**Proposition 7.**

**Proof.**

#### 3.5. Tasks Partitioning Problem

- (a)
- For any partition of $\mathcal{X}$ of size M with partition function A, we have$$\frac{1}{\rho}log{\mathbb{E}}_{P}\left[A{\left(X\right)}^{\rho}\right]\ge {H}_{\alpha}\left(P\right)-logM.$$
- (b)
- If $M>log\left|\mathcal{X}\right|+2$, then there exists a partition of $\mathcal{X}$ of size at most M with partition function A such that$$1\le {\mathbb{E}}_{P}\left[\widehat{A}{\left(X\right)}^{\rho}\right]\le 1+{2}^{\rho ({H}_{\alpha}\left(P\right)-log\tilde{M})},$$$$\begin{array}{c}\hfill \tilde{M}:=(M-log|\mathcal{X}|-2)/4.\end{array}$$

**Proof.**

**Proposition 8.**

- (a)
- $$\underset{n\to \infty}{lim}{\mathbb{E}}_{{P}_{n}}\left[{A}_{n}{\left({X}^{n}\right)}^{\rho}\right]=1\phantom{\rule{2.em}{0ex}}\mathrm{if}\phantom{\rule{4.pt}{0ex}}\gamma >{H}_{\alpha}\left(P\right),$$
- (b)
- $$\underset{n\to \infty}{lim}\frac{1}{n\rho}log{\mathbb{E}}_{{P}_{n}}\left[{A}_{n}{\left({X}^{n}\right)}^{\rho}\right]={H}_{\alpha}\left(P\right)-\gamma \phantom{\rule{2.em}{0ex}}\mathrm{if}\phantom{\rule{4.pt}{0ex}}\gamma <{H}_{\alpha}\left(P\right).$$

**Proof.**

**Remark 1.**

**Mismatch Case**:

**Proposition 9.**

**Proof.**

**Proposition 10.**

**Proof.**

## 4. Ordered Tasks Partitioning Problem

**Lemma 1.**

**Proof.**

**Proposition 11.**

- (a)
- For any partition of $\mathcal{X}$ of size M, we have$$\begin{array}{c}\hfill \frac{1}{\rho}log{\mathbb{E}}_{P}\left[N{\left(X\right)}^{\rho}\right]\ge {H}_{\alpha}\left(P\right)-log\left\{M[1+ln\left(\right)open="("\; close=")">\left|\mathcal{X}\right|/M\right]\}\end{array}$$
- (b)
- Let $M>log\left|\mathcal{X}\right|+2$. Then, there exists a partition of $\mathcal{X}$ of size at most M with count function N such that$$1\le {\mathbb{E}}_{P}\left[N{\left(X\right)}^{\rho}\right]\le 1+{2}^{\rho ({H}_{\alpha}\left(P\right)-log\tilde{M})},$$

**Proof.**

**Proposition 12.**

- (a)
- $$\underset{n\to \infty}{lim}{\mathbb{E}}_{{P}_{n}}\left[{N}_{n}{\left({X}^{n}\right)}^{\rho}\right]=1\phantom{\rule{2.em}{0ex}}\mathrm{if}\phantom{\rule{4.pt}{0ex}}\gamma >{H}_{\alpha}\left(P\right),$$
- (b)
- $$\underset{n\to \infty}{lim}\frac{1}{n\rho}log{\mathbb{E}}_{{P}_{n}}\left[{N}_{n}{\left({X}^{n}\right)}^{\rho}\right]={H}_{\alpha}\left(P\right)-\gamma \phantom{\rule{2.em}{0ex}}\mathrm{if}\phantom{\rule{4.pt}{0ex}}\gamma <{H}_{\alpha}\left(P\right).$$

**Proof.**

**Remark 2.**

- (a)
- If we choose the trivial partition, namely ${\mathcal{A}}_{n}=\left\{{\mathcal{X}}^{n}\right\}$, then the ordered tasks partitioning problem simplifies to Arıkan’s guessing problem, that is, we have ${M}_{n}=1$, ${N}_{n}\left({x}^{n}\right)={G}_{n}\left({x}^{n}\right)$ and (26) simplifies to$$\sum _{{x}^{n}\in {\mathcal{X}}^{n}}\frac{1}{{G}_{n}\left({x}^{n}\right)}\le 1+nln\left|\mathcal{X}\right|.$$Hence, all results pertaining to the Arıken’s guessing problem can be derived from the ordered tasks partitioning problem.
- (b)
- Structurally, ordered tasks partitioning problem differs from the Bunte–Lapidoth’s problem only due the factor $1+ln\left(\right|\mathcal{X}|/M)$ in (28). While this factor matters for one-shot results, for a sequence of i.i.d. tasks, this factor vanishes asymptotically.

**Mismatch Case**:

**Proposition 13.**

**Proof.**

**Proposition 14.**

**Proof.**

## 5. Operational Connection among the Problems

- Among the five problems discussed in the previous section, only Arıkan’s guessing and Huleihel et al.’s memoryless guessing have a unique optimal solution; others only have asymptotically optimal solutions.
- Optimal solution of Huleihel et al.’s memoryless guessing problem is the $\alpha $-scaled measure of the underlying probability distribution P. Hence, knowledge about the optimal solution of this problem implies knowledge about an optimal (or asymptotically optimal) solution of all other problems.
- Among the Bunte–Lapidoth’s and ordered tasks problems, an asymptotically optimal solution of one yields that of the other. The partitioning lemma (Prop. III-2 of [8]) is the key result in these two problems, as it guarantees the existence of the asymptotically optimal partitions in both these problems.

#### 5.1. Campbell’s Coding and Arıkan’s Guessing

**Proposition 15.**

**Proof.**

#### 5.2. Arıkan’s Guessing and Bunte–Lapidoth’s Tasks Partitioning Problem

**Proposition 16.**

**Proof.**

**Proposition 17.**

**Proof.**

#### 5.3. Huleihel et al.’s Memoryless Guessing and Campbell’s Coding

**Lemma 2.**

**Proof.**

**Proposition 18.**

**Proof.**

## 6. Summary and Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Shannon, C.E. A Mathematical Theory of Communication. Bell Syst. Tech. J.
**1948**, 27, 379–423. [Google Scholar] [CrossRef] [Green Version] - Rényi, A. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, Berkeley, CA, USA, 20 June–30 July 1960; pp. 547–561. [Google Scholar]
- Aczél, J.; Daróczy, Z. On Measures of Information and Their Characterizations. In Mathematicas in Science and Engineering; Academic Press: Cambridge, MA, USA, 1975; Volume 115. [Google Scholar]
- Campbell, L.L. A coding theorem and Rényi’s entropy. Inf. Control
**1965**, 8, 423–429. [Google Scholar] [CrossRef] [Green Version] - Sundaresan, R. Guessing Under Source Uncertainty. IEEE Trans. Inf. Theory
**2007**, 53, 269–287. [Google Scholar] [CrossRef] [Green Version] - Blumer, A.C.; McEliece, R.J. The Rényi redundancy of generalized Huffman codes. IEEE Trans. Inf. Theory
**1988**, 34, 1242–1249. [Google Scholar] [CrossRef] [Green Version] - Sundaresan, R. A measure of discrimination and its geometric properties. In Proceedings of the IEEE International Symposium on Information Theory, Lausanne, Switzerland, 30 June–5 July 2002; p. 264. [Google Scholar]
- Bunte, C.; Lapidoth, A. Encoding Tasks and Rényi Entropy. IEEE Trans. Inf. Theory
**2014**, 60, 5065–5076. [Google Scholar] [CrossRef] [Green Version] - Kumar, M.A.; Sundaresan, R. Minimization problems based on relative α-entropy I: Forward Projection. IEEE Trans. Inf. Theory
**2015**, 61, 5063–5080. [Google Scholar] [CrossRef] [Green Version] - Lutwak, E.; Yang, D.; Zhang, G. Cramér-Rao and moment-entropy inequalities for Rényi entropy and generalized Fisher information. IEEE Trans. Inf. Theory
**2005**, 51, 473–478. [Google Scholar] [CrossRef] - Ashok Kumar, M.; Sundaresan, R. Minimization Problems Based on Relative α-Entropy II: Reverse Projection. IEEE Trans. Inf. Theory
**2015**, 61, 5081–5095. [Google Scholar] [CrossRef] [Green Version] - Massey, J.L. Guessing and entropy. In Proceedings of the 1994 IEEE International Symposium on Information Theory, Trondheim, Norway, 27 June–1 July 1994; p. 204. [Google Scholar]
- Arikan, E. An inequality on guessing and its application to sequential decoding. IEEE Trans. Inf. Theory
**1996**, 42, 99–105. [Google Scholar] [CrossRef] [Green Version] - Arikan, E.; Merhav, N. Guessing subject to distortion. IEEE Trans. Inf. Theory
**1998**, 44, 1041–1056. [Google Scholar] [CrossRef] - Pfister, C.; Sullivan, W. Renyi entropy, guesswork moments, and large deviations. IEEE Trans. Inf. Theory
**2004**, 50, 2794–2800. [Google Scholar] [CrossRef] [Green Version] - Malone, D.; Sullivan, W. Guesswork and entropy. IEEE Trans. Inf. Theory
**2004**, 50, 525–526. [Google Scholar] [CrossRef] - Hanawal, M.K.; Sundaresan, R. Guessing Revisited: A Large Deviations Approach. IEEE Trans. Inf. Theory
**2011**, 57, 70–78. [Google Scholar] [CrossRef] [Green Version] - Christiansen, M.M.; Duffy, K.R. Guesswork, Large Deviations, and Shannon Entropy. IEEE Trans. Inf. Theory
**2013**, 59, 796–802. [Google Scholar] [CrossRef] [Green Version] - Huleihel, W.; Salamatian, S.; Médard, M. Guessing with limited memory. In Proceeding of the 2017 IEEE International Symposium on Information Theory (ISIT), Aachen, Germany, 25–30 June 2017; pp. 2253–2257. [Google Scholar]
- Salamatian, S.; Huleihel, W.; Beirami, A.; Cohen, A.; Médard, M. Why Botnets Work: Distributed Brute-Force Attacks Need No Synchronization. IEEE Trans. Inf. Forensics Secur.
**2019**, 14, 2288–2299. [Google Scholar] [CrossRef] [Green Version] - Arikan, E.; Merhav, N. Joint source-channel coding and guessing with application to sequential decoding. IEEE Trans. Inf. Theory
**1998**, 44, 1756–1769. [Google Scholar] [CrossRef] [Green Version] - Csiszár, I.; Shields, P. Information Theory and Statistics: A Tutorial, Foundations and Trends in Communications and Information Theory; Now Publishers: Delft, The Netherlands, 2004. [Google Scholar]
- Tsallis, C.; Mendes, R.S.; Plastino, A.R. The role of constraints within generalized non-extensive statistics. Phys. A
**1998**, 261, 534–554. [Google Scholar] [CrossRef] - Dupuis, P.; Ellis, R.S. A Weak Convergence Approach to the Theory of Large Deviations; John Wiley & Sons: Hoboken, NJ, USA, 1997. [Google Scholar]
- Shayevitz, O. On Rényi measures and hypothesis testing. In Proceedings of the 2011 IEEE International Symposium on Information Theory Proceedings, St. Petersburg, Russia, 31 July–5 August 2011; pp. 894–898. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; John Wiley & Sons: Hoboken, NJ, USA, 2012. [Google Scholar]
- Humblet, P. Generalization of Huffman coding to minimize the probability of buffer overflow. IEEE Trans. Inf. Theory
**1981**, 27, 230–232. [Google Scholar] [CrossRef] [Green Version] - Hanawal, M.K.; Sundaresan, R. Randomised Attacks on Passwords. DRDO-IISc Programme on Advanced Research in Mathematical Engineering. 2010. Available online: https://ece.iisc.ac.in/rajeshs/reprints/TR-PME-2010-11.pdf (accessed on 14 August 2022).
- Bracher, A.; Hof, E.; Lapidoth, A. Guessing Attacks on Distributed-Storage Systems. IEEE Trans. Inf. Theory
**2019**, 65, 6975–6998. [Google Scholar] [CrossRef] [Green Version] - Salamatian, S.; Liu, L.; Beirami, A.; Médard, M. Mismatched guesswork and one-to-one codes. In Proceedings of the 2019 IEEE Information Theory Workshop (ITW), Gotland, Sweden, 25–28 August 2019; pp. 1–5. [Google Scholar]
- Courtade, T.A.; Verdú, S. Cumulant generating function of codeword lengths in optimal lossless compression. In Proceedings of the 2014 IEEE International Symposium on Information Theory, Honolulu, HI, USA, 29 June –4 July 2014; pp. 2494–2498. [Google Scholar] [CrossRef]
- Kosut, O.; Sankar, L. Asymptotics and non-asymptotics for universal fixed-to-variable source coding. IEEE Trans. Inf. Theory
**2017**, 63, 3757–3772. [Google Scholar] [CrossRef] - Beirami, A.; Calderbank, R.; Christiansen, M.M.; Duffy, K.R.; Médard, M. A characterization of guesswork on swiftly tilting curves. IEEE Trans. Inf. Theory
**2018**, 60, 2850–2871. [Google Scholar] - Hanawal, M.K.; Sundaresan, R. Guessing and Compression Subject to Distortion; IndraStra Global: Sheridan, WY, USA, 2010. [Google Scholar]
- Sundaresan, R. Guessing Based On Length Functions. In Proceedings of the 2007 IEEE International Symposium on Information Theory, Cambridge, MA, USA, 1–6 July 2007; pp. 716–719. [Google Scholar] [CrossRef] [Green Version]
- Rezaee, A.; Beirami, A.; Makhdoumi, A.; Médard, M.; Duffy, K. Guesswork subject to a total entropy budget. In Proceedings of the 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA, 3–6 October 2017; pp. 1008–1015. [Google Scholar] [CrossRef]

**Figure 1.**Relationships established among the five problems. A directed arrow from problem A to problem B means knowing optimal or asymptotically optimal solution of A helps us find the same in B.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ashok Kumar, M.; Sunny, A.; Thakre, A.; Kumar, A.; Dinesh Manohar, G.
Are Guessing, Source Coding and Tasks Partitioning Birds of A Feather? *Entropy* **2022**, *24*, 1695.
https://doi.org/10.3390/e24111695

**AMA Style**

Ashok Kumar M, Sunny A, Thakre A, Kumar A, Dinesh Manohar G.
Are Guessing, Source Coding and Tasks Partitioning Birds of A Feather? *Entropy*. 2022; 24(11):1695.
https://doi.org/10.3390/e24111695

**Chicago/Turabian Style**

Ashok Kumar, M., Albert Sunny, Ashish Thakre, Ashisha Kumar, and G. Dinesh Manohar.
2022. "Are Guessing, Source Coding and Tasks Partitioning Birds of A Feather?" *Entropy* 24, no. 11: 1695.
https://doi.org/10.3390/e24111695