Bottleneck Problems: An Information and Estimation-Theoretic View ^†

Abstract

Information bottleneck (IB) and privacy funnel (PF) are two closely related optimization problems which have found applications in machine learning, design of privacy algorithms, capacity problems (e.g., Mrs. Gerber’s Lemma), and strong data processing inequalities, among others. In this work, we first investigate the functional properties of IB and PF through a unified theoretical framework. We then connect them to three information-theoretic coding problems, namely hypothesis testing against independence, noisy source coding, and dependence dilution. Leveraging these connections, we prove a new cardinality bound on the auxiliary variable in IB, making its computation more tractable for discrete random variables. In the second part, we introduce a general family of optimization problems, termed “bottleneck problems”, by replacing mutual information in IB and PF with other notions of mutual information, namely f-information and Arimoto’s mutual information. We then argue that, unlike IB and PF, these problems lead to easily interpretable guarantees in a variety of inference tasks with statistical constraints on accuracy and privacy. While the underlying optimization problems are non-convex, we develop a technique to evaluate bottleneck problems in closed form by equivalently expressing them in terms of lower convex or upper concave envelope of certain functions. By applying this technique to a binary case, we derive closed form expressions for several bottleneck problems.

1. Introduction

Optimization formulations that involve information-theoretic quantities (e.g., mutual information) have been instrumental in a variety of learning problems found in machine learning. A notable example is the information bottleneck ($\mathsf{IB}$) method [1]. Suppose Y is a target variable and X is an observable correlated variable with joint distribution $P_{XY}$. The goal of $\mathsf{IB}$ is to learn a “compact” summary (aka bottleneck) T of X that is maximally “informative” for inferring Y. The bottleneck variable T is assumed to be generated from X by applying a random function F to X, i.e., $T=F\left(X\right)$, in such a way that it is conditionally independent of Y given X, that we denote by

(1)

The $\mathsf{IB}$ quantifies this goal by measuring the “compactness” of T using the mutual information $I(X;T)$ and, similarly, “informativeness” by $I(Y;T)$. For a given level of compactness $R\ge 0$, $\mathsf{IB}$ extracts the bottleneck variable T that solves the constrained optimization problem

$$\mathsf{IB}\left(R\right)≔\sup I(Y;T)\mathrm{subject}\mathrm{to}I(X;T)\le R,$$

(2)

where the supremum is taken over all randomized functions $T=F\left(X\right)$ satisfying Y X T.

The optimization problem that underlies the information bottleneck has been studied in the information theory literature as early as the 1970’s—see [2,3,4,5]—as a technique to prove impossibility results in information theory and also to study the common information between X and Y. Wyner and Ziv [2] explicitly determined the value of $\mathsf{IB}\left(R\right)$ for the special case of binary X and Y—a result widely known as Mrs. Gerber’s Lemma [2,6]. More than twenty years later, the information bottleneck function was studied by Tishby et al. [1] and re-formulated in a data analytic context. Here, the random variable X represents a high-dimensional observation with a corresponding low-dimensional feature Y. $\mathsf{IB}$ aims at specifying a compressed description of image which is maximally informative about feature Y. This framework led to several applications in clustering [7,8,9] and quantization [10,11].

A closely-related framework to $\mathsf{IB}$ is the privacy funnel ($\mathsf{PF}$) problem [12,13,14]. In the $\mathsf{PF}$ framework, a bottleneck variable T is sought to maximally preserve “information” contained in X while revealing as little about Y as possible. This framework aims to capture the inherent trade-off between revealing X perfectly and leaking a sensitive attribute Y. For instance, suppose a user wishes to share an image X for some classification tasks. The image might carry information about attributes, say Y, that the user might consider as sensitive, even when such information is of limited use for the tasks, e.g., location, or emotion. The $\mathsf{PF}$ framework seeks to extract a representation of X from which the original image can be recovered with maximal accuracy while minimizing the privacy leakage with respect to Y. Using mutual information for both privacy leakage and informativeness, the privacy funnel can be formulated as

$$\mathsf{PF}\left(r\right)≔\inf I(Y;T)\mathrm{subject}\mathrm{to}I(X;T)\ge r,$$

(3)

where the infumum is taken over all randomized function $T=F\left(X\right)$ and r is the parameter specifying the level of informativeness. It is evident from the formulations (2) and (3) that $\mathsf{IB}$ and $\mathsf{PF}$ are closely related. In fact, we shall see later that they correspond to the upper and lower boundaries of a two-dimensional compact convex set. This duality has led to design of greedy algorithms [12,15] for estimating $\mathsf{PF}$ based on the agglomerative information bottleneck [9] algorithm. A similar formulation has recently been proposed in [16] as a tool to train a neural network for learning a private representation of data X; see [17,18] for other closely-related formulations. Solving $\mathsf{IB}$ and $\mathsf{PF}$ optimization problems analytically is challenging. However, recent machine learning applications, and deep learning algorithms in particular, have reignited the study of both $\mathsf{IB}$ and $\mathsf{PF}$ (see Related Work).

In this paper, we first give a cohesive overview of the existing results surrounding the $\mathsf{IB}$ and the $\mathsf{PF}$ formulations. We then provide a comprehensive analysis of $\mathsf{IB}$ and $\mathsf{PF}$ from an information-theoretic perspective, as well as a survey of several formulations connected to the $\mathsf{IB}$ and $\mathsf{PF}$ that have been introduced in the information theory and machine learning literature. Moreover, we overview connections with coding problems such as remote source-coding [19], testing against independence [20], and dependence dilution [21]. Leveraging these connections, we prove a new cardinality bound for the bottleneck variable in $\mathsf{IB}$, leading to more tractable optimization problem for $\mathsf{IB}$. We then consider a broad family of optimization problems by going beyond mutual information in formulations (2) and (3). We propose two candidates for this task: Arimoto’s mutual information [22] and f-information [23]. By replacing $I(Y;T)$ and/or $I(X;T)$ with either of these measures, we generate a family of optimization problems that we referred to as the bottleneck problems. These problems are shown to better capture the underlying trade-offs intended by $\mathsf{IB}$ and $\mathsf{PF}$ (see also the short version [24]). More specifically, our main contributions are listed next.

• Computing $\mathsf{IB}$ and $\mathsf{PF}$ are notoriously challenging when X takes values in a set with infinite cardinality (e.g., X is drawn from a continuous probability distribution). We consider three different scenarios to circumvent this difficulty. First, we assume that X is a Gaussian perturbation of Y, i.e., $X=Y+N^{\mathsf{G}}$ where $N^{\mathsf{G}}$ is a noise variable sampled from a Gaussian distribution independent of Y. Building upon the recent advances in entropy power inequality in [25], we derive a sharp upper bound for $\mathsf{IB}\left(R\right)$. As a special case, we consider jointly Gaussian $(X,Y)$ for which the upper bound becomes tight. This then provides a significantly simpler proof for the fact that in this special case the optimal bottleneck variable T is also Gaussian than the original proof given in [26]. In the second scenario, we assume that Y is a Gaussian perturbation of X, i.e., $Y=X+N^{\mathsf{G}}$. This corresponds to a practical setup where the feature Y might be perfectly obtained from a noisy observation of X. Relying on the recent results in strong data processing inequality [27], we obtain an upper bound on $\mathsf{IB}\left(R\right)$ which is tight for small values of R. In the last scenario, we compute second-order approximation of $\mathsf{PF}\left(r\right)$ under the assumption that T is obtained by Gaussian perturbation of X, i.e., $T=X+N^{\mathsf{G}}$. Interestingly, the rate of increase of $\mathsf{PF}\left(r\right)$ for small values of r is shown to be dictated by an asymmetric measure of dependence introduced by Rényi [28].
• We extend the Witsenhausen and Wyner’s approach [3] for analytically computing $\mathsf{IB}$ and $\mathsf{PF}$. This technique converts solving the optimization problems in $\mathsf{IB}$ and $\mathsf{PF}$ to determining the convex and concave envelopes of a certain function, respectively. We apply this technique to binary X and Y and derive a closed form expression for $\mathsf{PF}\left(r\right)$– we call this result Mr. Gerber’s Lemma.
• Relying on the connection between $\mathsf{IB}$ and noisy source coding [19] (see [29,30]), we show that the optimal bottleneck variable T in optimization problem (2) takes values in a set $\mathcal{T}$ with cardinality $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|$. Compared to the best cardinality bound previously known (i.e., $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|+1$), this result leads to a reduction in the search space’s dimension of the optimization problem (2) from ${\mathbb{R}}^{{\left|\mathcal{X}\right|}^2}$ to ${\mathbb{R}}^{\left|\mathcal{X}\right|\left(\right|\mathcal{X}|-1)}$. Moreover, we show that this does not hold for $\mathsf{PF}$, indicating a fundamental difference in optimizations problems (2) and (3).
• Following [14,31], we study the deterministic $\mathsf{IB}$ and $\mathsf{PF}$ (denoted by $\mathsf{dIB}$ and $\mathsf{dPF}$) in which T is assumed to be a deterministic function of X, i.e., $T=f\left(X\right)$ for some function f. By connecting $\mathsf{dIB}$ and $\mathsf{dPF}$ with entropy-constrained scalar quantization problems in information theory [32], we obtain bounds on them explicitly in terms of $\left|\mathcal{X}\right|$. Applying these bounds to $\mathsf{IB}$, we obtain that $\frac{\mathsf{IB}\left(R\right)}{I(X;Y)}$ is bounded by one from above and by $\min \{\frac{R}{H\left(X\right)},\frac{e^R-1}{\left|\mathcal{X}\right|}\}$ from below.
• By replacing $I(Y;T)$ and/or $I(X;T)$ in (2) and (3) with Arimoto’s mutual information or f-information, we generate a family of bottleneck problems. We then argue that these new functionals better describe the trade-offs that were intended to be captured by $\mathsf{IB}$ and $\mathsf{PF}$. The main reason is three-fold: First, as illustrated in Section 2.3, mutual information in $\mathsf{IB}$ and $\mathsf{PF}$ are mainly justified when $n\gg 1$ independent samples $(X_1,Y_1),\dots ,(X_n,Y_n)$ of $P_{XY}$ are considered. However, Arimoto’s mutual information allows for operational interpretation even in the single-shot regime (i.e., for $n=1$). Second, $I(Y;T)$ in $\mathsf{IB}$ and $\mathsf{PF}$ is meant to be a proxy for the efficiency of reconstructing Y given observation T. However, this can be accurately formalized by probability of correctly guessing Y given T (i.e., Bayes risk) or minimum mean-square error (MMSE) in estimating Y given T. While $I(Y;T)$ bounds these two measures, we show that they are precisely characterized by Arimoto’s mutual information and f-information, respectively. Finally, when $P_{XY}$ is unknown, mutual information is known to be notoriously difficult to estimate. Nevertheless, Arimoto’s mutual information and f-information are easier to estimate: While mutual information can be estimated with estimation error that scales as $O(\log n/\sqrt{n})$ [33], Diaz et al. [34] showed that this estimation error for Arimoto’s mutual information and f-information is $O(1/\sqrt{n})$.
  We also generalize our computation technique that enables us to analytically compute these bottleneck problems. Similar as before, this technique converts computing bottleneck problems to determining convex and concave envelopes of certain functions. Focusing on binary X and Y, we derive closed form expressions for some of the bottleneck problems.

1.1. Related Work

The $\mathsf{IB}$ formulation has been extensively applied in representation learning and clustering [7,8,35,36,37,38]. Clustering based on $\mathsf{IB}$ results in algorithms that cluster data points in terms of the similarity of $P_{Y|X}$. When data points lie in a metric space, usually geometric clustering is preferred where clustering is based upon the geometric (e.g., Euclidean) distance. Strouse and Schwab [31,39] proposed the deterministic $\mathsf{IB}$ (denoted by $\mathsf{dIB}$) by enforcing that $P_{T|X}$ is a deterministic mapping: $\mathsf{dIB}\left(R\right)$ denotes the supremum of $I(Y;f(X\left)\right)$ over all functions $f:\mathcal{X}\to \mathcal{T}$ satisfying $H\left(f\right(X\left)\right)\le R$. This optimization problem is closely related to the problem of scalar quantization in information theory: designing a function $f:\mathcal{X}\to \left[M\right]≔\{1,\dots ,M\}$ with a pre-determined output alphabet with f optimizing some objective functions. This objective might be maximizing or minimizing $H\left(f\right(X\left)\right)$ [40] or maximizing $I(Y;f(X\left)\right)$ for a random variable Y correlated with X [32,41,42,43]. Since $H\left(f\right(X\left)\right)\le \log M$ for $f:\mathcal{X}\to \left[M\right]$, the latter problem provides lower bounds for $\mathsf{dIB}$ (and thus for $\mathsf{IB}$). In particular, one can exploit [44] (Theorem 1) to obtain $I(X;Y)-\mathsf{dIB}\left(R\right)\le O\left(e^{-2R/\left|\mathcal{Y}\right|-1}\right)$ provided that $\min \left\{\right|\mathcal{X}|,2^R\}>2\left|\mathcal{Y}\right|$. This result establishes a linear gap between $\mathsf{dIB}$ and $I(X;Y)$ irrespective of $\left|\mathcal{X}\right|$.

The connection between quantization and $\mathsf{dIB}$ further allows us to obtain multiplicative bounds. For instance, if $Y\sim \mathsf{Bernoulli}\left(\frac{1}{2}\right)$ and $X=Y+N^{\mathsf{G}}$, where $N^{\mathsf{G}}\sim \mathcal{N}(0,1)$ is independent of Y, then it is well-known in information theory literature that $I(Y;f\left(X\right))\ge \frac{2}{\pi }I(X;Y)$ for all non-constant $f:\mathcal{X}\to \{0,1\}$ (see, e.g., [45] (Section 2.11)), thus $\mathsf{dIB}\left(R\right)\ge \frac{2}{\pi }I(X;Y)$ for $R\le 1$. We further explore this connection to provide multiplicative bounds on $\mathsf{dIB}\left(R\right)$ in Section 2.5.

The study of $\mathsf{IB}$ has recently gained increasing traction in the context of deep learning. By taking T to be the activity of the hidden layer(s), Tishby and Zaslavsky [46] (see also [47]) argued that neural network classifiers trained with cross-entropy loss and stochastic gradient descent (SGD) inherently aims at solving the $\mathsf{IB}$ optimization problems. In fact, it is claimed that the graph of the function $R\mapsto \mathsf{IB}\left(R\right)$ (the so-called the information plane) characterizes the learning dynamic of different layers in the network: shallow layers correspond to maximizing $I(Y;T)$ while deep layers’ objective is minimizing $I(X;T)$. While the generality of this claim was refuted empirically in [48] and theoretically in [49,50], it inspired significant follow-up studies. These include (i) modifying neural network training in order to solve the $\mathsf{IB}$ optimization problem [51,52,53,54,55]; (ii) creating connections between $\mathsf{IB}$ and generalization error [56], robustness [51], and detection of out-of-distribution data [57]; and (iii) using $\mathsf{IB}$ to understand specific characteristic of neural networks [55,58,59,60].

In both $\mathsf{IB}$ and $\mathsf{PF}$, mutual information poses some limitations. For instance, it may become infinity in deterministic neural networks [48,49,50] and also may not lead to proper privacy guarantee [61]. As suggested in [55,62], one way to address this issue is to replace mutual information with other statistical measures. In the privacy literature, several measures with strong privacy guarantee have been proposed including Rényi maximal correlation [21,63,64], probability of correctly recovering [65,66], minimum mean-squared estimation error (MMSE) [67,68], ${\chi }^2$-information [69] (a special case of f-information to be described in Section 3), Arimoto’s and Sibson’s mutual information [61,70]—to be discussed in Section 3, maximal leakage [71], and local differential privacy [72]. All these measures ensure interpretable privacy guarantees. For instance, it is shown in [67,68] that if ${\chi }^2$-information between Y and T is sufficiently small, then no functions of Y can be efficiently reconstructed given T; thus providing an interpretable privacy guarantee.

Another limitation of mutual information is related to its estimation difficulty. It is known that mutual information can be estimated from n samples with the estimation error that scales as $O(\log n/\sqrt{n})$ [33]. However, as shown by Diaz et al. [34], the estimation error for most of the above measures scales as $O(1/\sqrt{n})$. Furthermore, the recently popular variational estimators for mutual information, typically implemented via deep learning methods [73,74,75], presents some fundamental limitations [76]: the variance of the estimator might grow exponentially with the ground truth mutual information and also the estimator might not satisfy basic properties of mutual information such as data processing inequality or additivity. McAllester and Stratos [77] showed that some of these limitations are inherent to a large family of mutual information estimators.

1.2. Notation

We use capital letters, e.g., X, for random variables and calligraphic letters for their alphabets, e.g., $\mathcal{X}$. If X is distributed according to probability mass function (pmf) $P_X$, we write $X\sim P_X$. Given two random variables X and Y, we write $P_{XY}$ and $P_{Y|X}$ as the joint distribution and the conditional distribution of Y given X. We also interchangeably refer to $P_{Y|X}$ as a channel from X to Y. We use $H\left(X\right)$ to denote both entropy and differential entropy of X, i.e., we have

$$H\left(X\right)=-\sum _{x\in \mathcal{X}}{P}_X\left(x\right)\log P_X\left(x\right)$$

if X is a discrete random variable taking values in $\mathcal{X}$ with probability mass function (pmf) $P_X$ and

$$H\left(X\right)=-\int \log f_X\left(x\right)\log f_X\left(x\right)\mathrm{d}x,$$

where X is an absolutely continuous random variable with probability density function (pdf) $f_X$. If X is a binary random variable with $P_X\left(1\right)=p$, we write $X\sim \mathsf{Bernoulli}\left(p\right)$. In this case, its entropy is called binary entropy function and denoted by $h_{\mathsf{b}}\left(p\right)≔-p\log p-(1-p)\log (1-p)$. We use superscript $\mathsf{G}$ to describe a standard Gaussian random variable, i.e., $N^{\mathsf{G}}\sim \mathcal{N}(0,1)$. Given two random variables X and Y, their (Shannon’s) mutual information is denoted by $I(X;Y)≔H\left(Y\right)-H\left(Y\right|X)$. We let $\mathcal{P}\left(\mathcal{X}\right)$ denote the set of all probability distributions on the set $\mathcal{X}$. Given an arbitrary $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$ and a channel $P_{Y|X}$, we let $Q_X{P}_{Y|X}$ denote the resulting output distribution on $\mathcal{Y}$. For any $a\in [0,1]$, we use $\overline{a}$ to denote $1-a$ and for any integer $k\in \mathbb{N}$, $\left[k\right]≔\{1,2,\dots ,k\}$.

Throughout the paper, we assume a pair of (discrete or continuous) random variables $(X,Y)\sim P_{XY}$ are given with a fixed joint distribution $P_{XY}$, marginals $P_X$ and $P_Y$, and conditional distribution $P_{Y|X}$. We then use $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$ to denote an arbitrary distribution with $Q_Y=Q_X{P}_{Y|X}\in \mathcal{P}\left(\mathcal{Y}\right)$.

2. Information Bottleneck and Privacy Funnel: Definitions and Functional Properties

In this section, we review the information bottleneck and its closely related functional, the privacy funnel. We then prove some analytical properties of these two functionals and develop a convex analytic approach which enables us to compute closed-form expressions for both these two functionals in some simple cases.

To precisely quantify the trade-off between these two conflicting goals, the $\mathsf{IB}$ optimization problem (2) was proposed [1]. Since any randomized function $T=F\left(X\right)$ can be equivalently characterized by a conditional distribution, the optimization problem (2) can be instead expressed as

(4)

where R and $\tilde{R}$ denote the level of desired compression and informativeness, respectively. We use $\mathsf{IB}\left(R\right)$ and $\widetilde{\mathsf{IB}}\left(\tilde{R}\right)$ to denote $\mathsf{IB}(P_{XY},R)$ and $\widetilde{\mathsf{IB}}(P_{XY},\tilde{R})$, respectively, when the joint distribution is clear from the context. Notice that if $\mathsf{IB}(P_{XY},R)=\tilde{R}$, then $\widetilde{\mathsf{IB}}(P_{XY},\tilde{R})=R$.

Now consider the setup where data X is required to be disclosed while maintaining the privacy of a sensitive attribute, represented by Y. This goal was formulated by $\mathsf{PF}$ in (3). As before, replacing randomized function $T=F\left(X\right)$ with conditional distribution $P_{T|X}$, we can equivalently express (3) as

(5)

where $\tilde{r}$ and r denote the level of desired privacy and informativeness, respectively. The case $\tilde{r}=0$ is particularly interesting in practice and specifies perfect privacy, see e.g., [13,78]. As before, we write $\widetilde{\mathsf{PF}}\left(\tilde{r}\right)$ and $\mathsf{PF}\left(r\right)$ for $\widetilde{\mathsf{PF}}(P_{XY},\tilde{r})$ and $\mathsf{PF}(P_{XY},r)$ when $P_{XY}$ is clear from the context.

The following properties of $\mathsf{IB}$ and $\mathsf{PF}$ follow directly from their definitions. The proof of this result (and any other results in this section) is given in Appendix A.

Theorem 1.

For a given $P_{XY}$, the mappings $\mathsf{IB}\left(R\right)$ and $\mathsf{PF}\left(r\right)$ have the following properties:

• $\mathsf{IB}\left(0\right)=\mathsf{PF}\left(0\right)=0$.
• $\mathsf{IB}\left(R\right)=I(X;Y)$ for any $R\ge H\left(X\right)$ and $\mathsf{PF}\left(r\right)=I(X;Y)$ for $r\ge H\left(X\right)$.
• $0\le \mathsf{IB}\left(R\right)\le \min \{R,I(X;Y\left)\right\}$ for any $R\ge 0$ and $\mathsf{PF}\left(r\right)\ge \max \{r-H(X\left|Y\right),0\}$ for any $r\ge 0$.
• $R\mapsto \mathsf{IB}\left(R\right)$ is continuous, strictly increasing, and concave on the range $(0,I(X;Y\left)\right)$.
• $r\mapsto \mathsf{PF}\left(r\right)$ is continuous, strictly increasing, and convex on the range $(0,I(X;Y\left)\right)$.
• If $P_{Y|X}\left(y\right|x)>0$ for all $x\in \mathcal{X}$ and $y\in \mathcal{Y}$, then both $R\mapsto \mathsf{IB}\left(R\right)$ and $r\mapsto \mathsf{PF}\left(r\right)$ are continuously differentiable over $(0,H(X\left)\right)$.
• $R\mapsto \frac{\mathsf{IB}\left(R\right)}{R}$ is non-increasing and $r\mapsto \frac{\mathsf{PF}\left(r\right)}{r}$ is non-decreasing.
• We have

  

According to this theorem, we can always restrict both R and r in (4) and (5), respectively, to $[0,H(X\left)\right]$ as $\mathsf{IB}\left(R\right)=\mathsf{PF}\left(r\right)=I(X;Y)$ for all $r,R\ge H\left(X\right)$.

Define $\mathcal{M}=\mathcal{M}\left(P_{XY}\right)\subset {\mathbb{R}}^2$ as

(6)

It can be directly verified that $\mathcal{M}$ is convex. According to this theorem, $R\mapsto \mathsf{IB}\left(R\right)$ and $r\mapsto \mathsf{PF}\left(r\right)$ correspond to the upper and lower boundary of $\mathcal{M}$, respectively. The convexity of $\mathcal{M}$ then implies the concavity and convexity of $\mathsf{IB}$ and $\mathsf{PF}$. Figure 1 illustrates the set $\mathcal{M}$ for the simple case of binary X and Y.

Figure 1. Examples of the set $\mathcal{M}$, defined in (6). The upper and lower boundaries of this set correspond to information bottleneck ($\mathsf{IB}$) and privacy funnel ($\mathsf{PF}$), respectively. It is worth noting that, while $\mathsf{IB}$(R) = 0 only at R = 0, $\mathsf{PF}$(r) = 0 holds in general for r belonging to a non-trivial interval (only for $\left|\mathcal{X}\right|$ > 2). Moreover, note that in general neither upper nor lower boundaries are smooth. A sufficient condition for smoothness is $P_{X|Y}\left(y\right|x)$ > 0 (see Theorem 1), thus both $\mathsf{IB}$ and $\mathsf{PF}$ are smooth in the binary case.

While both $\mathsf{IB}\left(0\right)=0$ and $\mathsf{PF}\left(0\right)=0$, their behavior in the neighborhood around zero might be completely different. As illustrated in Figure 1, $\mathsf{IB}\left(R\right)>0$ for all $R>0$, whereas $\mathsf{PF}\left(r\right)=0$ for $r\in [0,r_0]$ for some $r_0>0$. When such $r_0>0$ exists, we say perfect privacy occurs: there exists a variable T satisfying Y X T such that $I(Y;T)=0$ while $I(X;T)>0$; making T a representation of X having perfect privacy (i.e., no information leakage about Y). A necessary and sufficient condition for the existence of such T is given in [21] (Lemma 10) and [13] (Theorem 3), described next.

Theorem 2

(Perfect privacy). Let $(X,Y)\sim P_{XY}$ be given and $\mathcal{A}\subset {[0,1]}^{\left|\mathcal{Y}\right|}$ be the set of vectors $\{P_{Y|X}(·|x),x\in \mathcal{X}\}$. Then there exists $r_0>0$ such that $\mathsf{PF}\left(r\right)=0$ for $r\in [0,r_0]$ if and only if vectors in $\mathcal{A}$ are linearly independent.

In light of this theorem, we obtain that perfect privacy occurs if $\left|\mathcal{X}\right|>\left|\mathcal{Y}\right|$. It also follows from the theorem that for binary X, perfect privacy cannot occur (see Figure 1a).

Theorem 1 enables us to derive a simple bounds for $\mathsf{IB}$ and $\mathsf{PF}$. Specifically, the facts that $\frac{\mathsf{PF}\left(r\right)}{r}$ is non-decreasing and $\frac{\mathsf{IB}\left(R\right)}{R}$ is non-increasing immediately result in the the following linear bounds.

Theorem 3

(Linear lower bound). For $r,R\in (0,H(X\left)\right)$, we have

$$\underset{\genfrac{}{}{0pt}{}{Q_X\in \mathcal{P}\left(\mathcal{X}\right)}{Q_X\ne P_X}}{\inf }\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)}\le \frac{\mathsf{PF}\left(r\right)}{r}\le \frac{I(X;Y)}{H\left(X\right)}\le \frac{\mathsf{IB}\left(R\right)}{R}\le \underset{\genfrac{}{}{0pt}{}{Q_X\in \mathcal{P}\left(\mathcal{X}\right)}{Q_X\ne P_X}}{\sup }\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)}\le 1.$$

(7)

In light of this theorem, if $\mathsf{PF}\left(r\right)=r$, then $I(X;Y)=H\left(X\right)$, implying $X=g\left(Y\right)$ for a deterministic function g. Conversely, if $X=g\left(Y\right)$ then $\mathsf{PF}\left(r\right)=r$ because for all T forming the Markov relation Y g(Y) T, we have $I(Y;T)=I\left(g\right(Y);T)$. On the other hand, we have $\mathsf{IB}\left(R\right)=R$ if and only if there exists a variable $T^{\ast }$ satisfying $I(X;T^{\ast })=I(Y;T^{\ast })$ and thus the following double Markov relations

It can be verified (see [79] (Problem 16.25)) that this double Markov condition is equivalent to the existence of a pair of functions f and g such that $f\left(X\right)=g\left(Y\right)$ and (X,Y) f(X) $T^{\ast }$. One special case of this setting, namely where g is an identity function, has been recently studied in details in [53] and will be reviewed in Section 2.5. Theorem 3 also enables us to characterize the “worst” joint distribution $P_{XY}$ with respect to $\mathsf{IB}$ and $\mathsf{PF}$. As demonstrated in the following lemma, if $P_{Y|X}$ is an erasure channel then $\frac{\mathsf{PF}\left(r\right)}{r}=\frac{\mathsf{IB}\left(R\right)}{R}=\frac{I(X;Y)}{H\left(X\right)}$.

Lemma 1.

• Let $P_{XY}$ be such that $\mathcal{Y}=\mathcal{X}\cup \{\perp \}$, $P_{Y|X}\left(x\right|x)=1-\delta$, and $P_{Y|X}(\perp |x)=\delta$ for some $\delta >0$. Then

  $$\frac{\mathsf{PF}\left(r\right)}{r}=\frac{\mathsf{IB}\left(R\right)}{R}=1-\delta .$$
• Let $P_{XY}$ be such that $\mathcal{X}=\mathcal{Y}\cup \{\perp \}$, $P_{X|Y}\left(y\right|y)=1-\delta$, and $P_{X|Y}(\perp |y)=\delta$ for some $\delta >0$. Then

  $$\mathsf{PF}\left(r\right)=\max \{r-H(X\left|Y\right),0\}.$$

The bounds in Theorem 3 hold for all r and R in the interval $[0,H(X\left)\right]$. We can, however, improve them when r and R are sufficiently small. Let ${\mathsf{PF}}^{\prime }\left(0\right)$ and ${\mathsf{IB}}^{\prime }\left(0\right)$ denote the slope of $\mathsf{PF}(·)$ and $\mathsf{IB}(·)$ at zero, i.e., ${\mathsf{PF}}^{\prime }\left(0\right)≔{\lim }_{r\to 0^{+}}\frac{\mathsf{PF}\left(r\right)}{r}$ and ${\mathsf{IB}}^{\prime }\left(0\right)≔{\lim }_{R\to 0^{+}}\frac{\mathsf{IB}\left(R\right)}{R}$.

Theorem 4.

Given $(X,Y)\sim P_{XY}$, we have

$$\begin{array}{cc}\hfill \underset{\genfrac{}{}{0pt}{}{Q_X\in \mathcal{P}\left(\mathcal{X}\right)}{Q_X\ne P_X}}{\inf }\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)}={\mathsf{PF}}^{\prime }\left(0\right)& \le \underset{\begin{array}{c}x\in \mathcal{X}:\\ P_X\left(x\right)>0\end{array}}{\min }\frac{D_{\mathsf{KL}}(P_{Y|X}(·|x)\parallel P_Y(·))}{-\log P_X\left(x\right)}\hfill \\ \hfill & \le \underset{\begin{array}{c}x\in \mathcal{X}:\\ P_X\left(x\right)>0\end{array}}{\max }\frac{D_{\mathsf{KL}}(P_{Y|X}(·|x)\parallel P_Y(·))}{-\log P_X\left(x\right)}\le {\mathsf{IB}}^{\prime }\left(0\right)=\underset{\genfrac{}{}{0pt}{}{Q_X\in \mathcal{P}\left(\mathcal{X}\right)}{Q_X\ne P_X}}{\sup }\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)}.\hfill \end{array}$$

This theorem provides the exact values of ${\mathsf{PF}}^{\prime }\left(0\right)$ and ${\mathsf{IB}}^{\prime }\left(0\right)$ and also simple bounds for them. While the exact expressions for ${\mathsf{PF}}^{\prime }\left(0\right)$ and ${\mathsf{IB}}^{\prime }\left(0\right)$ are usually difficult to compute, a simple plug-in estimator is proposed in [80] for ${\mathsf{IB}}^{\prime }\left(0\right)$. This estimator can be readily adapted to estimate ${\mathsf{PF}}^{\prime }\left(0\right)$. Theorem 4 reveals a profound connection between $\mathsf{IB}$ and the strong data processing inequality (SDPI) [81]. More precisely, thanks to the pioneering work of Anantharam et al. [82], it is known that the supremum of $\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)}$ over all $Q_X\ne P_X$ is equal the supremum of $\frac{I(Y;T)}{I(X;T)}$ over all $P_{T|X}$ satisfying Y X T and hence ${\mathsf{IB}}^{\prime }\left(0\right)$ specifies the strengthening of the data processing inequality of mutual information. This connection may open a new avenue for new theoretical results for $\mathsf{IB}$, especially when X or Y are continuous random variables. In particular, the recent non-multiplicative SDPI results [27,83] seem insightful for this purpose.

In many practical cases, we might have n i.i.d. samples $(X_1,Y_1),\dots ,(X_n,Y_n)$ of $(X,Y)\sim P_{XY}$. We now study how $\mathsf{IB}$ behaves in n. Let $X^n≔(X_1,\dots ,X_n)$ and $Y^n≔(Y_1,\dots ,Y_n)$. Due to the i.i.d. assumption, we have $P_{X^n{Y}^n}(x^n,y^n)={\prod }_{i=1}^n{P}_{XY}(x_i,y_i)$. This can also be described by independently feeding $X_i$, $i\in \left[n\right]$, to channel $P_{Y|X}$ producing $Y_i$. The following theorem, demonstrated first in [3] (Theorem 2.4), gives a formula for $\mathsf{IB}$ in terms of n.

Theorem 5

(Additivity). We have

$$\frac{1}{n}\mathsf{IB}(P_{X^n{Y}^n},nR)=\mathsf{IB}(P_{XY},R).$$

This theorem demonstrates that an optimal channel $P_{T^n|X^n}$ for i.i.d. samples $(X^n,Y^n)\sim P_{XY}$ is obtained by the Kronecker product of an optimal channel $P_{T|X}$ for $(X,Y)\sim P_{XY}$. This, however, may not hold in general for $\mathsf{PF}$, that is, we might have $\mathsf{PF}(P_{X^n{Y}^n},nr)<n\mathsf{PF}(P_{XY},r)$, see [13] (Proposition 1) for an example.

2.1. Gaussian $\mathsf{IB}$ and $\mathsf{PF}$

In this section, we turn our attention to a special, yet important, case where $X=Y+\sigma N^{\mathsf{G}}$, where $\sigma >0$ and $N^{\mathsf{G}}\sim \mathcal{N}(0,1)$ is independent of Y. This setting subsumes the popular case of jointly Gaussian $(X,Y)$ whose information bottleneck functional was computed in [84] for the vector case (i.e., $(X,Y)$ are jointly Gaussian random vectors).

Lemma 2.

Let ${\left\{Y_i\right\}}_{i=1}^n$ be n i.i.d. copies of $Y\sim P_Y$ and $X_i=Y_i+\sigma N_i^{\mathsf{G}}$ where $\left\{N_i^{\mathsf{G}}\right\}$ are i.i.d samples of $\mathcal{N}(0,1)$ independent of Y. Then, we have

$$\frac{1}{n}\mathsf{IB}(P_{X^n{Y}^n},nR)\le H\left(X\right)-\frac{1}{2}\log \left[2\pi e{\sigma }^2+e^{2\left(H\right(Y)-R)}\right].$$

It is worth noting that this result was concurrently proved in [85]. The main technical tool in the proof of this lemma is a strong version of the entropy power inequality [25] (Theorem 2) which holds even if $X_i$, $Y_i$, and $N_i$ are random vectors (as opposed to scalar). Thus, one can readily generalize Lemma 2 to the vector case. Note that the upper bound established in this lemma holds without any assumptions on $P_{T|X}$. This upper bound provides a significantly simpler proof for the well-known fact that for the jointly Gaussian $(X,Y)$, the optimal channel $P_{T|X}$ is Gaussian. This result was first proved in [26] and used in [84] to compute an expression of $\mathsf{IB}$ for the Gaussian case.

Corollary 1.

If $(X,Y)$ are jointly Gaussian with correlation coefficient ρ, then we have

$$\mathsf{IB}\left(R\right)=\frac{1}{2}\log \frac{1}{1-{\rho }^2+{\rho }^2{e}^{-2R}}.$$

(8)

Moreover, the optimal channel $P_{T|X}$ is given by $P_{T|X}(·|x)=\mathcal{N}(0,{\tilde{\sigma }}^2)$ for ${\tilde{\sigma }}^2={\sigma }_Y^2\frac{e^{-2R}}{{\rho }^2(1-e^{-2R})}$ where ${\sigma }_Y^2$ is the variance of Y.

In Lemma 2, we assumed that X is a Gaussian perturbation of Y. However, in some practical scenarios, we might have Y as a Gaussian perturbation of X. For instance, let X represent an image and Y be a feature of the image that can be perfectly obtained from a noisy observation of X. Then, the goal is to compress the image with a given compression rate while retaining maximal information about the feature. The following lemma, which is an immediate consequence of [27] (Theorem 1), gives an upper bound for $\mathsf{IB}$ in this case.

Lemma 3.

Let $X^n$ be n i.i.d. copies of a random variable X satisfying $\mathbb{E}\left[X^2\right]\le 1$ and $Y_i$ be the result of passing $X_i$, $i\in \left[n\right]$, through a Gaussian channel $Y=X+\sigma N^{\mathsf{G}}$, where $\sigma >0$ and $N^{\mathsf{G}}\sim \mathcal{N}(0,1)$ is independent of X. Then, we have

$$\frac{1}{n}\mathsf{IB}(P_{X^n{Y}^n},nR)\le R-\mathsf{\Psi }(R,\sigma ),$$

(9)

where

$$\mathsf{\Psi }(R,\sigma )≔\underset{x\in [0,\frac{1}{2}]}{\max }2\mathsf{Q}\left(\sqrt{\frac{1}{x{\sigma }^2}}\right)\left(R-h_{\mathsf{b}}\left(x\right)-\frac{x}{2}\log \left(1+\frac{1}{x{\sigma }^2}\right)\right),$$

(10)

$\mathsf{Q}\left(t\right)≔{\int }_t^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{t^2}{2}}dt$ is the Gaussian complimentary CDF and $h_{\mathsf{b}}\left(a\right)≔-a\log \left(a\right)-(1-a)\log (1-a)$ for $a\in (0,1)$ is the binary entropy function. Moreover, we have

$$\frac{1}{n}\mathsf{IB}(P_{X^n{Y}^n},nR)\le R-e^{-\frac{1}{R{\sigma }^2}\log \frac{1}{R}+\mathsf{\Theta }\left(\log \frac{1}{R}\right)}.$$

(11)

Note that that Lemma 3 holds for any arbitrary X (provided that $\mathbb{E}\left[X^2\right]\le 1$) and hence (9) bounds information bottleneck functionals for a wide family of $P_{XY}$. However, the bound is loose in general for large values of R. For instance, if $(X,Y)$ are jointly Gaussian (implying $Y=X+\sigma N^{\mathsf{G}}$ for some $\sigma >0$), then the right-hand side of (9) does not reduce to (8). To show this, we numerically compute the upper bound (9) and compare it with the Gaussian information bottleneck (8) in Figure 2.

Figure 2. Comparison of (8), the exact value of $\mathsf{IB}$ for jointly Gaussian X and Y (i.e., $Y=X+\sigma N^{\mathsf{G}}$ with X and $N^{\mathsf{G}}$ being both standard Gaussian $\mathcal{N}(0,1)$), with the general upper bound (9) for ${\sigma }^2=0.5$. It is worth noting that while the Gaussian $\mathsf{IB}$ converges to $I(X;Y)\approx 0.8$, the upper bound diverges.

The privacy funnel functional is much less studied even for the simple case of jointly Gaussian. Solving the optimization in $\mathsf{PF}$ over $P_{T|X}$ without any assumptions is a difficult challenge. A natural assumption to make is that $P_{T|X}(·|x)$ is Gaussian for each $x\in \mathcal{X}$. This leads to the following variant of $\mathsf{PF}$

$${\mathsf{PF}}^{\mathsf{G}}\left(r\right)≔\underset{\begin{array}{c}\sigma \ge 0,\\ I(X;T_{\sigma })\ge r\end{array}}{\inf }I(Y;T_{\sigma }),$$

where

$$T_{\sigma }≔X+\sigma N^{\mathsf{G}},$$

and $N^{\mathsf{G}}\sim \mathcal{N}(0,1)$ is independent of X. This formulation is tractable and can be computed in closed form for jointly Gaussian $(X,Y)$ as described in the following example.

Example 1.

Let X and Y be jointly Gaussian with correlation coefficient ρ. First note that since mutual information is invariant to scaling, we may assume without loss of generality that both X and Y are zero mean and unit variance and hence we can write $X=\rho Y+\sqrt{1-{\rho }^2}{M}^{\mathsf{G}}$ where $M^{\mathsf{G}}\sim \mathcal{N}(0,1)$ is independent of Y. Consequently, we have

$$I(X;T_{\sigma })=\frac{1}{2}\log \left(1+\frac{1}{{\sigma }^2}\right),$$

(12)

and

$$I(Y;T_{\sigma })=\frac{1}{2}\log \left(1+\frac{{\rho }^2}{1-{\rho }^2+{\sigma }^2}\right).$$

(13)

In order to ensure $I(X;T_{\sigma })\ge r$, we must have $\sigma \le {\left(e^{2r}-1\right)}^{-\frac{1}{2}}$. Plugging this choice of σ into (13), we obtain

$${\mathsf{PF}}^{\mathsf{G}}\left(r\right)=\frac{1}{2}\log \left(\frac{1}{1-{\rho }^2\left(1-e^{-2r}\right)}\right).$$

(14)

This example indicates that for jointly Gaussian $(X,Y)$, we have ${\mathsf{PF}}^{\mathsf{G}}\left(r\right)=0$ if and only if $r=0$ (thus perfect privacy does not occur) and the constraint $I(X;T_{\sigma })=r$ is satisfied by a unique $\sigma$. These two properties in fact hold for all continuous variables X and Y with finite second moments as demonstrated in Lemma A1 in Appendix A. We use these properties to derive a second-order approximation of ${\mathsf{PF}}^{\mathsf{G}}\left(r\right)$ when r is sufficiently small. For the following theorem, we use $\mathsf{var}\left(U\right)$ to denote the variance of the random variable U and $\mathsf{var}\left(U\right|V)≔\mathbb{E}\left[{(U-\mathbb{E}\left[U\right|V])}^2\right|V]$. We use ${\sigma }_X^2=\mathsf{var}\left(X\right)$ for short.

Theorem 6.

For any pair of continuous random variables $(X,Y)$ with finite second moments, we have as $r\to 0$

$${\mathsf{PF}}^{\mathsf{G}}\left(r\right)=\eta (X,Y)r+\Delta (X,Y)r^2+o\left(r^2\right),$$

where $\eta (X,Y)≔\frac{\mathsf{var}\left(\mathbb{E}\right[X\left|Y\right])}{{\sigma }_X^2}$ and

$$\Delta (X,Y)≔\frac{2}{{\sigma }_X^4}\left[\mathbb{E}\left[{\mathsf{var}}^2\left(X\right|Y)\right]-{\sigma }_X^2\mathbb{E}\left[\mathsf{var}\left(X\right|Y)\right]\right].$$

It is worth mentioning that the quantity $\eta (X,Y)$ was first defined by Rényi [28] as an asymmetric measure of correlation between X and Y. In fact, it can be shown that $\eta (X,Y)={\sup }_f{\rho }^2(X,f\left(Y\right)),$ where supremum is taken over all measurable functions f and $\rho (·,·)$ denotes the correlation coefficient. As a simple illustration of Theorem 6, consider jointly Gaussian X and Y with correlation coefficient $\rho$ for which ${\mathsf{PF}}^{\mathsf{G}}$ was computed in Example 1. In this case, it can be easily verified that $\eta (X,Y)={\rho }^2$ and $\Delta (X,Y)=-2{\sigma }_X^2{\rho }^2(1-{\rho }^2)$. Hence, for jointly Gaussian $(X,Y)$ with correlation coefficient $\rho$ and unit variance, we have ${\mathsf{PF}}^{\mathsf{G}}\left(r\right)={\rho }^{2}r-2{\rho }^2(1-{\rho }^2)r^2+o\left(r^2\right)$. In Figure 3, we compare the approximation given in Theorem 6 for this particular case.

Figure 3. Second-order approximation of ${\mathsf{PF}}^{\mathsf{G}}$ according to Theorem 6 for jointly Gaussian X and Y with correlation coefficient $\rho =0.8$. For this particular case, the exact expression of ${\mathsf{PF}}^{\mathsf{G}}$ is computed in (14).

2.2. Evaluation of $\mathsf{IB}$ and $\mathsf{PF}$

The constrained optimization problems in the definitions of $\mathsf{IB}$ and $\mathsf{PF}$ are usually challenging to solve numerically due to the non-linearity in the constraints. In practice, however, both $\mathsf{IB}$ and $\mathsf{PF}$ are often approximated by their corresponding Lagrangian optimizations

$${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)≔\underset{P_{T|X}}{\sup }I(Y;T)-\beta I(X;T)=H\left(Y\right)-\beta H\left(X\right)-\underset{P_{T|X}}{\inf }\left[H\left(Y\right|T)-\beta H(X\left|T\right)\right],$$

(15)

and

$${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)≔\underset{P_{T|X}}{\inf }I(Y;T)-\beta I(X;T)=H\left(Y\right)-\beta H\left(X\right)-\underset{P_{T|X}}{\sup }\left[H\left(Y\right|T)-\beta H(X\left|T\right)\right],$$

(16)

where $\beta \in {\mathbb{R}}_{+}$ is the Lagrangian multiplier that controls the tradeoff between compression and informativeness in for $\mathsf{IB}$ and the privacy and informativeness in $\mathsf{PF}$. Notice that for the computation of ${\mathcal{L}}_{\mathsf{IB}}$, we can assume, without loss of generality, that $\beta \in [0,1]$ since otherwise the maximizer of (15) is trivial. It is worth noting that ${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)$ and ${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)$ in fact correspond to lines of slope $\beta$ supporting $\mathcal{M}$ from above and below, thereby providing a new representation of $\mathcal{M}$.

Let $(X^{\prime },Y^{\prime })$ be a pair of random variables with $X^{\prime }\sim Q_X$ for some $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$ and $Y^{\prime }$ is the output of $P_{Y|X}$ when the input is $X^{\prime }$ (i.e., $Y^{\prime }\sim Q_X{P}_{Y|X}$). Define

$$F_{\beta }\left(Q_X\right)≔H\left(Y^{\prime }\right)-\beta H\left(X^{\prime }\right).$$

This function, in general, is neither convex nor concave in $Q_X$. For instance, $F\left(0\right)$ is concave and $F\left(1\right)$ is convex in $P_X$. The lower convex envelope of $F_{\beta }\left(Q_X\right)$ is defined as the largest convex function smaller than $F_{\beta }\left(Q_X\right)$. Similarly, the upper concave envelope of $F_{\beta }\left(Q_X\right)$ is defined as the smallest concave function larger than $F_{\beta }\left(Q_X\right)$. Let ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$ and ${\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]$ denote the lower convex and upper concave envelopes of $F_{\beta }\left(Q_X\right)$, respectively. If $F_{\beta }\left(Q_X\right)$ is convex at $P_X$, that is ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X}=F_{\beta }\left(P_X\right)$, then $F_{\beta }\left(Q_X\right)$ remains convex at $P_X$ for all ${\beta }^{\prime }\ge \beta$ because

$$\begin{array}{cc}\hfill {\mathcal{K}}_{\cup }\left[F_{{\beta }^{\prime }}\left(Q_X\right)\right]& ={\mathcal{K}}_{\cup }[F_{\beta }\left(Q_X\right)-({\beta }^{\prime }-\beta )H\left(X^{\prime }\right)]\hfill \\ \hfill & \ge {\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]+{\mathcal{K}}_{\cup }[-({\beta }^{\prime }-\beta )H\left(X^{\prime }\right)]\hfill \\ \hfill & ={\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]-({\beta }^{\prime }-\beta )H\left(X^{\prime }\right),\hfill \end{array}$$

where the last equality follows from the fact that $-({\beta }^{\prime }-\beta )H\left(X\right)$ is convex. Hence, at $P_X$ we have

$${\mathcal{K}}_{\cup }\left[F_{{\beta }^{\prime }}\left(Q_X\right)\right]{|}_{P_X}\ge {\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X}-({\beta }^{\prime }-\beta )H\left(X\right)=F_{\beta }\left(P_X\right)-({\beta }^{\prime }-\beta )H\left(X\right)=F_{{\beta }^{\prime }}\left(P_X\right).$$

Analogously, if $F_{\beta }\left(Q_X\right)$ is concave at $P_X$, that is ${\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X}=F_{\beta }\left(P_X\right)$, then $F_{\beta }\left(Q_X\right)$ remains concave at $P_X$ for all ${\beta }^{\prime }\le \beta$.

Notice that, according to (15) and (16), we can write

$${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)=H\left(Y\right)-\beta H\left(X\right)-{\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X},$$

(17)

and

$${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)=H\left(Y\right)-\beta H\left(X\right)-{\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X}.$$

(18)

In light of the above arguments, we can write

$${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)=0,$$

for all $\beta >{\beta }_{\mathsf{IB}}$ where ${\beta }_{\mathsf{IB}}$ is the smallest $\beta$ such that $F_{\beta }\left(P_X\right)$ touches ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$. Similarly,

$${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)=0,$$

for all $\beta <{\beta }_{\mathsf{PF}}$ where ${\beta }_{\mathsf{PF}}$ is the largest $\beta$ such that $F_{\beta }\left(P_X\right)$ touches ${\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]$. In the following theorem, we show that ${\beta }_{\mathsf{IB}}$ and ${\beta }_{\mathsf{PF}}$ are given by the values of ${\mathsf{IB}}^{\prime }\left(0\right)$ and ${\mathsf{PF}}^{\prime }\left(0\right)$, respectively, given in Theorem 4. A similar formulae ${\beta }_{\mathsf{IB}}$ and ${\beta }_{\mathsf{PF}}$ were given in [86].

Proposition 1.

We have,

$${\beta }_{\mathsf{IB}}=\underset{Q_X\ne P_X}{\sup }\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)},$$

and

$${\beta }_{\mathsf{PF}}=\underset{Q_X\ne P_X}{\inf }\frac{D_{\mathsf{KL}}(Q_Y\parallel P_Y)}{D_{\mathsf{KL}}(Q_X\parallel P_X)}.$$

Kim et al. [80] have recently proposed an efficient algorithm to estimate ${\beta }_{\mathsf{IB}}$ from samples of $P_{XY}$ involving a simple optimization problem. This algorithm can be readily adapted for estimating ${\beta }_{\mathsf{PF}}$. Proposition 1 implies that in optimizing the Lagrangians (17) and (18), we can restrict the Lagrange multiplier $\beta$, that is

$${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)=H\left(Y\right)-\beta H\left(X\right)-{\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X},\mathrm{for}\beta \in [0,{\beta }_{\mathsf{IB}}],$$

(19)

and

$${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)=H\left(Y\right)-\beta H\left(X\right)-{\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]{|}_{P_X},\mathrm{for}\beta \in [{\beta }_{\mathsf{PF}},\infty ).$$

(20)

Remark 1.

As demonstrated by Kolchinsky et al. [53], the boundary points 0 and ${\beta }_{\mathsf{IB}}$ are required for the computation of ${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)$. In fact, when Y is a deterministic function of X, then only $\beta =0$ and $\beta ={\beta }_{\mathsf{IB}}$ are required to compute the $\mathsf{IB}$ and other values of β are vacuous. The same argument can also be used to justify the inclusion of ${\beta }_{\mathsf{PF}}$ in computing ${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)$. Note also that since $F_{\beta }\left(Q_X\right)$ becomes convex for $\beta >{\beta }_{\mathsf{IB}}$, computing ${\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]$ becomes trivial for such values of β.

Remark 2.

Observe that the lower convex envelope of any function f can be obtained by taking Legendre-Fenchel transformation (aka. convex conjugate) twice. Hence, one can use the existing linear-time algorithms for approximating Legendre-Fenchel transformation (e.g., [87,88]) for approximating ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$.

Once ${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)$ and ${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)$ are computed, we can derive $\mathsf{IB}$ and $\mathsf{PF}$ via standard results in optimization (see [3] (Section IV) for more details):

$$\mathsf{IB}\left(R\right)=\underset{\beta \in [0,{\beta }_{\mathsf{IB}}]}{\inf }\beta R+{\mathcal{L}}_{\mathsf{IB}}\left(\beta \right),$$

(21)

and

$$\mathsf{PF}\left(r\right)=\underset{\beta \in [{\beta }_{\mathsf{PF}},\infty ]}{\sup }\beta r+{\mathcal{L}}_{\mathsf{PF}}\left(\beta \right).$$

(22)

Following the convex analysis approach outlined by Witsenhausen and Wyner [3], $\mathsf{IB}$ and $\mathsf{PF}$ can be directly computed from ${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)$ and ${\mathcal{L}}_{\mathsf{PF}}\left(\beta \right)$ by observing the following. Suppose for some $\beta$, ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$ (resp. ${\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]$) at $P_X$ is obtained by a convex combination of points $F_{\beta }\left(Q^i\right)$, $i\in \left[k\right]$ for some $Q^1,\dots ,Q^k$ in $\mathcal{P}\left(\mathcal{X}\right)$, integer $k\ge 2$, and weights ${\lambda }_i\ge 0$ (with ${\sum }_i{\lambda }_i=1$). Then ${\sum }_i{\lambda }_i{Q}^i=P_X$, and $T^{\ast }$ with properties $P_{T^{\ast }}\left(i\right)={\lambda }_i$ and $P_{X|T^{\ast }=i}=Q^i$ attains the minimum (resp. maximum) of $H\left(Y\right|T)-\beta H(X\left|T\right)$. Hence, $(I(X;T^{\ast }),I(Y;T^{\ast }))$ is a point on the upper (resp. lower) boundary of $\mathcal{M}$; implying that $\mathsf{IB}\left(R\right)=I(Y;T^{\ast })$ for $R=I(X;T^{\ast })$ (resp. $\mathsf{PF}\left(r\right)=I(Y;T^{\ast })$ for $r=I(X;T^{\ast })$). If for some $\beta$, ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$ at $P_X$ coincides with $F_{\beta }\left[P_X\right]$, then this corresponds to ${\mathcal{L}}_{\mathsf{IB}}\left(\beta \right)=0$. The same holds for ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$. Thus, all the information about the functional $\mathsf{IB}$ (resp. $\mathsf{PF}$) is contained in the subset of the domain of ${\mathcal{K}}_{\cup }\left[F_{\beta }\left(Q_X\right)\right]$ (resp. ${\mathcal{K}}_{\cap }\left[F_{\beta }\left(Q_X\right)\right]$) over which it differs from $F_{\beta }\left(Q_X\right)$. We will revisit and generalize this approach later in Section 3.

We can now instantiate this for the binary symmetric case. Suppose X and Y are binary variables and $P_{Y|X}$ is binary symmetric channel with crossover probability $\delta$, denoted by $\mathsf{BSC}\left(\delta \right)$ and defined as

$$\mathsf{BSC}\left(\delta \right)=\left[\begin{array}{cc}1-\delta & \delta \\ \delta & 1-\delta \end{array}\right],$$

(23)

for some $\delta \ge 0$. To describe the result in a compact fashion, we introduce the following notation: we let $h_{\mathsf{b}}:[0,1]\to [0,1]$ denote the binary entropy function, i.e., $h_{\mathsf{b}}\left(p\right)=-p\log p-(1-p)\log (1-p)$. Since this function is strictly increasing $[0,\frac{1}{2}]$, its inverse exists and is denoted by $h_{\mathsf{b}}^{-1}:[0,1]\to [0,\frac{1}{2}]$. Moreover, $a\ast b≔a(1-b)+b(1-a)$ for $a,b\in [0,1]$.

Lemma 4

(Mr. and Mrs. Gerber’s Lemma). For $X\sim \mathsf{Bernoulli}\left(p\right)$ for $p\le \frac{1}{2}$ and $P_{Y|X}=\mathsf{BSC}\left(\delta \right)$ for $\delta \ge 0$, we have

$$\mathsf{IB}\left(R\right)=h_{\mathsf{b}}(p\ast \delta )-h_{\mathsf{b}}\left(\delta \ast h_{\mathsf{b}}^{-1}\left(h_{\mathsf{b}}\left(p\right)-R\right)\right),$$

(24)

and

$$\mathsf{PF}\left(r\right)=h_{\mathsf{b}}(p\ast \delta )-\alpha h_{\mathsf{b}}\left(\delta \ast \frac{p}{z}\right)-\overline{\alpha }{h}_{\mathsf{b}}\left(\delta \right),$$

(25)

where $r=h_{\mathsf{b}}\left(p\right)-\alpha h_{\mathsf{b}}\left(\frac{p}{z}\right)$, $z=\max \left(\alpha ,2p\right)$, and $\alpha \in [0,1]$.

The result in (24) was proved by Wyner and Ziv [2] and is widely known as Mrs. Gerber’s Lemma in information theory. Due to the similarity, we refer to (25) as Mr. Gerber’s Lemma. As described above, to prove (24) and (25) it suffices to derive the convex and concave envelopes of the mapping $F_{\beta }:[0,1]\to \mathbb{R}$ given by

$$F_{\beta }\left(q\right)≔F_{\beta }\left(Q_X\right)=h_{\mathsf{b}}(q\ast \delta )-\beta h_{\mathsf{b}}\left(q\right),$$

(26)

where $q\ast \delta ≔q\overline{\delta }+\delta \overline{q}$ is the output distribution of $\mathsf{BSC}\left(\delta \right)$ when the input distribution is $\mathsf{Bernoulli}\left(q\right)$ for some $q\in (0,1)$. It can be verified that ${\beta }_{\mathsf{IB}}\le {(1-2\delta )}^2$. This function is depicted in Figure 4 depending of the values of $\beta \le {(1-2\delta )}^2$.

Figure 4. The mapping $q\mapsto F_{\beta }\left(q\right)=H\left(Y^{\prime }\right)-\beta H\left(X^{\prime }\right)$ where $X^{\prime }\sim \mathsf{Bernoulli}\left(q\right)$ and $Y^{\prime }$ is the result of passing $X^{\prime }$ through BSC(0.1), see (26).

2.3. Operational Meaning of $\mathsf{IB}$ and $\mathsf{PF}$

In this section, we illustrate several information-theoretic settings which shed light on the operational interpretation of both $\mathsf{IB}$ and $\mathsf{PF}$. The operational interpretation of $\mathsf{IB}$ has recently been extensively studied in information-theoretic settings in [29,30]. In particular, it was shown that $\mathsf{IB}$ specifies the rate-distortion region of noisy source coding problem [19,89] under the logarithmic loss as the distortion measure and also the rate region of the lossless source coding with side information at the decoder [90]. Here, we state the former setting (as it will be useful for our subsequent analysis of cardinality bound) and also provide a new information-theoretic setting in which $\mathsf{IB}$ appears as the solution. Then, we describe another setting, the so-called dependence dilution, whose achievable rate region has an extreme point specified by $\mathsf{PF}$. This in fact delineate an important difference between $\mathsf{IB}$ and $\mathsf{PF}$: while $\mathsf{IB}$ describes the entire rate-region of an information-theoretic setup, $\mathsf{PF}$ specifies only a corner point of a rate region. Other information-theoretic settings related to $\mathsf{IB}$ and $\mathsf{PF}$ include CEO problem [91] and source coding for the Gray-Wyner network [92].

2.3.1. Noisy Source Coding

Suppose Alice has access only to a noisy version X of a source of interest Y. She wishes to transmit a rate-constrained description from her observation (i.e., X) to Bob such that he can recover Y with small average distortion. More precisely, let $(X^n,Y^n)$ be n i.i.d. samples of $(X,Y)\sim P_{XY}$. Alice encodes her observation $X^n$ through an encoder $\varphi :{\mathcal{X}}^n\to \{1,\dots ,K_n\}$ and sends $\varphi \left(X^n\right)$ to Bob. Upon receiving $\varphi \left(X^n\right)$, Bob reconstructs a “soft” estimate of $Y^n$ via a decoder $\psi :\{1,\dots ,K_n\}\to {\widehat{\mathcal{Y}}}^n$ where $\widehat{\mathcal{Y}}=\mathcal{P}\left(\mathcal{Y}\right)$. That is, the reproduction sequence ${\widehat{y}}^n$ consists of n probability measures on $\mathcal{Y}$. For any source and reproduction sequences $y^n$ and ${\widehat{y}}^n$, respectively, the distortion is defined as

$$d(y^n,{\widehat{y}}^n)≔\frac{1}{n}\sum _{i=1}^{n}d(y_i,{\widehat{y}}_i),$$

where

$$d(y,\widehat{y})≔\log \frac{1}{\widehat{y}\left(y\right)}.$$

(27)

We say that a pair of rate-distortion $(\mathsf{R},\mathsf{D})$ is achievable if there exists a pair $(\varphi ,\psi )$ of encoder and decoder such that

$$\underset{n\to \infty }{lim\; sup}\mathbb{E}\left[d(Y^n,\psi \left(\varphi \left(X^n\right)\right))\right]\le \mathsf{D},\mathrm{and}\underset{n\to \infty }{lim\; sup}\frac{1}{n}\log K_n\le \mathsf{R}.$$

(28)

The noisy rate-distortion function ${\mathsf{R}}^{\mathsf{noisy}}\left(\mathsf{D}\right)$ for a given $\mathsf{D}\ge 0$, is defined as the minimum rate $\mathsf{R}$ such that $(\mathsf{R},\mathsf{D})$ is an achievable rate-distortion pair. This problem arises naturally in many data analytic problems. Some examples include feature selection of a high-dimensional dataset, clustering, and matrix completion. This problem was first studied by Dobrushin and Tsybakov [19], who showed that ${\mathsf{R}}^{\mathsf{noisy}}\left(\mathsf{D}\right)$ is analogous to the classical rate-distortion function

(29)

It can be easily verified that $\mathbb{E}\left[d(Y,\widehat{Y})\right]=H\left(Y\right|\widehat{Y})$ and hence (after relabeling $\widehat{Y}$ as T)

(30)

where $R=H\left(Y\right)-\mathsf{D}$, which is equal to $\widetilde{\mathsf{IB}}$ defined in (4). For more details in connection between noisy source coding and $\mathsf{IB}$, the reader is referred to [29,30,91,93]. Notice that one can study an essentially identical problem where the distortion constraint (28) is replaced by

$$\underset{n\to \infty }{\lim }\frac{1}{n}I(Y^n;\psi \left(\varphi \left(X^n\right)\right))]\ge R,\mathrm{and}\underset{n\to \infty }{lim\; sup}\frac{1}{n}\log K_n\le \mathsf{R}.$$

This problem is addressed in [94] for discrete alphabets $\mathcal{X}$ and $\mathcal{Y}$ and extended recently in [95] for any general alphabets.

2.3.2. Test against Independence with Communication Constraint

As mentioned earlier, the connection between $\mathsf{IB}$ and noisy source coding, described above, was known and studied in [29,30]. Here, we provide a new information-theoretic setting which provides yet another operational meaning for $\mathsf{IB}$. Given n i.i.d. samples $(X_1,Y_1),\dots ,(X_n,Y_n)$ from joint distribution Q, we wish to test whether $X_i$ are independent of $Y_i$, that is, Q is a product distribution. This task is formulated by the following hypothesis test:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill H_0:& Q=P_{XY},\hfill \\ \hfill H_1:& Q=P_X{P}_Y,\hfill \end{array}\end{array}$$

(31)

for a given joint distribution $P_{XY}$ with marginals $P_X$ and $P_Y$. Ahlswede and Csiszár [20] investigated this problem under a communication constraint: While Y observations (i.e., $Y_1,\dots ,Y_n$) are available, the X observations need to be compressed at rate R, that is, instead of $X^n$, only $\varphi \left(X^n\right)$ is present where $\varphi :{\mathcal{X}}^n\to \{1,\dots ,K_n\}$ satisfies

$$\frac{1}{n}\log K_n\le R.$$

For the type I error probability not exceeding a fixed $\epsilon \in (0,1)$, Ahlswede and Csiszár [20] derived the smallest possible type 2 error probability, defined as

$${\beta }_R(n,\epsilon )=\underset{\genfrac{}{}{0pt}{}{\varphi :{\mathcal{X}}^n\to \left[K\right]}{\frac{1}{n}\log K_n\le R}}{\min }\underset{A\subset \left[K_n\right]\times {\mathcal{Y}}^n}{\min }\left\{(P_{\varphi \left(X^n\right)}\times P_{Y^n})\left(A\right):P_{\varphi \left(X^n\right)\times Y^n}\left(A\right)\ge 1-\epsilon \right\}.$$

The following gives the asymptotic expression of ${\beta }_R(n,\epsilon )$ for every $\epsilon \in (0,1)$. For the proof, refer to [20] (Theorem 3).

Theorem 7

([20]). For every $R\ge 0$ and $\epsilon \in (0,1)$, we have

$$\underset{n\to \infty }{\lim }-\frac{1}{n}\log {\beta }_R(n,\epsilon )=\mathsf{IB}\left(R\right).$$

In light of this theorem, $\mathsf{IB}\left(R\right)$ specifies the exponential rate at which the type II error probability of the hypothesis test (31) decays as the number of samples increases.

2.3.3. Dependence Dilution

Inspired by the problems of information amplification [96] and state masking [97], Asoodeh et al. [21] proposed the dependence dilution setup as follows. Consider a source sequences $X^n$ of n i.i.d. copies of $X\sim P_X$. Alice observes the source $X^n$ and wishes to encode it via the encoder

$$f_n:{\mathcal{X}}^n\to \{1,2,\dots ,2^{nR}\},$$

for some $R>0$. The goal is to ensure that any user observing $f_n\left(X^n\right)$ can construct a list, of fixed size, of sequences in ${\mathcal{X}}^n$ that contains likely candidates of the actual sequence $X^n$ while revealing negligible information about a correlated source $Y^n$. To formulate this goal, consider the decoder

$$g_n:\{1,2,\dots ,2^{nR}\}\to 2^{{\mathcal{X}}^n},$$

where $2^{{\mathcal{X}}^n}$ denotes the power set of ${\mathcal{X}}^n$. A dependence dilution triple $(R,\mathsf{\Gamma },\Delta )\in {\mathbb{R}}_{+}^3$ is said to be achievable if, for any $\delta >0$, there exists a pair of encoder and decoder $(f_n,g_n)$ such that for sufficiently large n

$$Pr\left(X^n\notin g_n\left(J\right)\right)<\delta ,$$

(32)

having fixed size $|g_n\left(J\right)|=2^{n\left(H\right(X)-\mathsf{\Gamma })},$ where $J=f_n\left(X^n\right)$ and simultaneously

$$\frac{1}{n}I(Y^n;J)\le \Delta +\delta .$$

(33)

Notice that without side information J, the decoder can only construct a list of size $2^{nH\left(X\right)}$ which contains $X^n$ with probability close to one. However, after J is observed and the list $g_n\left(J\right)$ is formed, the decoder’s list size can be reduced to $2^{n\left(H\right(X)-\mathsf{\Gamma })}$ and thus reducing the uncertainty about $X^n$ by $n\mathsf{\Gamma }\in [0,nH(X\left)\right]$. This observation can be formalized to show (see [96] for details) that the constraint (32) is equivalent to

$$\frac{1}{n}I(X^n;J)\ge \mathsf{\Gamma }-\delta ,$$

(34)

which lower bounds the amount of information J carries about $X^n$. Built on this equivalent formulation, Asoodeh et al. [21] (Corollary 15) derived a necessary condition for the achievable dependence dilution triple.

Theorem 8

([21]). Any achievable dependence dilution triple $(R,\mathsf{\Gamma },\Delta )$ satisfies

$$\left\{\begin{array}{cc}R\hfill & \ge \mathsf{\Gamma }\hfill \\ \mathsf{\Gamma }\hfill & \le I(X;T)\hfill \\ \Delta \hfill & \ge I(Y;T)-I(X;T)+\mathsf{\Gamma },\hfill \end{array}\right.$$

for some auxiliary random variable T satisfying Y X T and taking $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|+1$ values.

According to this theorem, $\mathsf{PF}\left(\mathsf{\Gamma }\right)$ specifies the best privacy performance of the dependence dilution setup for the maximum amplification rate $\mathsf{\Gamma }$. While this informs the operational interpretation of $\mathsf{PF}$, Theorem 8 only provides an outer bound for the set of achievable dependence dilution triple $(R,\mathsf{\Gamma },\Delta )$. It is, however, not clear that $\mathsf{PF}$ characterizes the rate region of an information-theoretic setup.

The fact that $\mathsf{IB}$ fully characterizes the rate-region of an source coding setup has an important consequence: the cardinality of the auxiliary random variable T in $\mathsf{IB}$ can be improved to $\left|\mathcal{X}\right|$ instead of $\left|\mathcal{X}\right|+1$.

2.4. Cardinality Bound

Recall that in the definition of $\mathsf{IB}$ in (4), no assumption was imposed on the auxiliary random variable T. A straightforward application of Carathéodory-Fenchel-Eggleston theorem (see e.g., [98] (Section III) or [79] (Lemma 15.4)) reveals that $\mathsf{IB}$ is attained for T taking values in a set $\mathcal{T}$ with cardinality $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|+1$. Here, we improve this bound and show $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|$ is sufficient.

Theorem 9.

For any joint distribution $P_{XY}$ and $R\in (0,H(X\left)\right]$, information bottleneck $\mathsf{IB}\left(R\right)$ is achieved by T taking at most $\left|\mathcal{X}\right|$ values.

The proof of this theorem hinges on the operational characterization of $\mathsf{IB}$ as the lower boundary of the rate-distortion region of noisy source coding problem discussed in Section 2.3. Specifically, we first show that the extreme points of this region is achieved by T taking $\left|\mathcal{X}\right|$ values. We then make use of a property of the noisy source coding problem (namely, time-sharing) to argue that all points of this region (including the boundary points) can be attained by such T. It must be mentioned that this result was already claimed by Harremoës and Tishby in [99] without proof.

In many practical scenarios, feature X has a large alphabet. Hence, the bound $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|$, albeit optimal, still can make the information bottleneck function computationally intractable over large alphabets. However, label Y usually has a significantly smaller alphabet. While it is in general impossible to have a cardinality bound for T in terms of $\left|\mathcal{Y}\right|$, one can consider approximating $\mathsf{IB}$ assuming T takes N values. The following result, recently proved by Hirche and Winter [100], is in this spirit.

Theorem 10

([100]). For any $(X,Y)\sim P_{XY}$, we have

$$\mathsf{IB}(R,N)\le \mathsf{IB}\left(R\right)\le \mathsf{IB}(R,N)+\delta \left(N\right),$$

where $\delta \left(N\right)=4N^{-\frac{1}{\left|\mathcal{Y}\right|}}\left[\log \frac{\left|\mathcal{Y}\right|}{4}+\frac{1}{\left|\mathcal{Y}\right|}\log N\right]$ and $\mathsf{IB}(R,N)$ denotes the information bottleneck functional (4) with the additional constraint that $\left|\mathcal{T}\right|\le N$.

Recall that, unlike $\mathsf{PF}$, the graph of $\mathsf{IB}$ characterizes the rate region of a Shannon-theoretic coding problem (as illustrated in Section 2.3), and hence any boundary points can be constructed via time-sharing of extreme points of the rate region. This lack of operational characterization of $\mathsf{PF}$ translates into a worse cardinality bound than that of $\mathsf{IB}$. In fact, for $\mathsf{PF}$ the cardinality bound $\left|\mathcal{T}\right|\le \left|\mathcal{X}\right|+1$ cannot be improved in general. To demonstrate this, we numerically solve the optimization in $\mathsf{PF}$ assuming that $\left|\mathcal{T}\right|=\left|\mathcal{X}\right|$ when both X and Y are binary. As illustrated in Figure 5, this optimization does not lead to a convex function, and hence, cannot be equal to $\mathsf{PF}$.

Figure 5. The set $\left\{\right(I(X;T),I(Y;T)\left)\right\}$ with $P_X=\mathsf{Bernoulli}\left(0.9\right)$, $P_{Y|X=0}=[0.9,0.1]$, $P_{Y|X=1}=[0.85,0.15]$, and T restricted to be binary. While the upper boundary of this set is concave, the lower boundary is not convex. This implies that, unlike $\mathsf{IB}$, $\mathsf{PF}\left(r\right)$ cannot be attained by binary variables T.

2.5. Deterministic Information Bottleneck

As mentioned earlier, $\mathsf{IB}$ formalizes an information-theoretic approach to clustering high-dimensional feature X into cluster labels T that preserve as much information about the label Y as possible. The clustering label is assigned by the soft operator $P_{T|X}$ that solves the $\mathsf{IB}$ formulation (4) according to the rule: $X=x$ is likely assigned label $T=t$ if $D_{\mathsf{KL}}(P_{Y|x}\parallel P_{Y|t})$ is small where $P_{Y|t}={\sum }_x{P}_{Y|x}{P}_{X|t}$. That is, clustering is assigned based on the similarity of conditional distributions. As in many practical scenarios, a hard clustering operator is preferred, Strouse and Schwab [31] suggested the following variant of $\mathsf{IB}$, termed as deterministic information bottleneck $\mathsf{dIB}$

$$\mathsf{dIB}(P_{XY},R)≔\underset{\begin{array}{c}f:\mathcal{X}\to \mathcal{T},\\ H\left(f\right(X\left)\right)\le R\end{array}}{\sup }I(Y;f\left(X\right)),$$

(35)

where the maximization is taken over all deterministic functions f whose range is a finite set $\mathcal{T}$. Similarly, one can define

$$\mathsf{dPF}(P_{XY},r)≔\underset{\begin{array}{c}f:\mathcal{X}\to \mathcal{T},\\ H\left(f\right(X\left)\right)\ge r\end{array}}{\inf }I(Y;f\left(X\right)).$$

(36)

One way to ensure that $H\left(f\right(X\left)\right)\le R$ for a deterministic function f is to restrict the cardinality of the range of f: if $f:\mathcal{X}\to \left[e^R\right]$ then $H\left(f\right(X\left)\right)$ is necessarily smaller than R. Using this insight, we derive a lower for $\mathsf{dIB}(P_{XY},R)$ in the following lemma.

Lemma 5.

For any given $P_{XY}$, we have

$$\mathsf{dIB}(P_{XY},R)\ge \frac{e^R-1}{\left|\mathcal{X}\right|}I(X;Y),$$

and

$$\mathsf{dPF}(P_{XY},r)\le \frac{e^r-1}{\left|\mathcal{X}\right|}I(X;Y)+Pr(X\ge e^r)\log \frac{1}{Pr(X\ge e^r)}.$$

Note that both R and r are smaller than $H\left(X\right)$ and thus the multiplicative factors of $I(X;Y)$ in the lemma are smaller than one. In light of this lemma, we can obtain

$$\frac{e^R-1}{\left|\mathcal{X}\right|}I(X;Y)\le \mathsf{IB}\left(R\right)\le I(X;Y),$$

and

$$\mathsf{PF}\left(r\right)\le \frac{e^r-1}{\left|\mathcal{X}\right|}I(X;Y)+Pr(X\ge e^r)\log \frac{1}{Pr(X\ge e^r)}.$$

In most of practical setups, $\left|\mathcal{X}\right|$ might be very large, making the above lower bound for $\mathsf{IB}$ vacuous. In the following lemma, we partially address this issue by deriving a bound independent of $\mathcal{X}$ when Y is binary.

Lemma 6.

Let $P_{XY}$ be a joint distribution of arbitrary X and binary $Y\sim \mathsf{Bernoulli}\left(q\right)$ for some $q\in (0,1)$. Then, for any $R\ge \log 5$ we have

$$\mathsf{dIB}(P_{XY},R)\ge I(X;Y)-2\alpha h_{\mathsf{b}}\left(\frac{I(X;Y)}{2\alpha (e^R-4)}\right),$$

where $\alpha =\max \{\log \frac{1}{q},\log \frac{1}{1-q}\}$.

3. Family of Bottleneck Problems

In this section, we introduce a family of bottleneck problems by extending $\mathsf{IB}$ and $\mathsf{PF}$ to a large family of statistical measures. Similar to $\mathsf{IB}$ and $\mathsf{PF}$, these bottleneck problems are defined in terms of boundaries of a two-dimensional convex set induced by a joint distribution $P_{XY}$. Recall that $R\mapsto \mathsf{IB}(P_{XY},R)$ and $r\mapsto \mathsf{PF}(P_{XY},r)$ are the upper and lower boundary of the set $\mathcal{M}$ defined in (6) and expressed here again for convenience

(37)

Since $P_{XY}$ is given, $H\left(X\right)$ and $H\left(Y\right)$ are fixed. Thus, in characterizing $\mathcal{M}$ it is sufficient to consider only $H\left(X\right|T)$ and $H\left(Y\right|T)$. To generalize $\mathsf{IB}$ and $\mathsf{PF}$, we must therefore generalize $H\left(X\right|T)$ and $H\left(Y\right|T)$.

Given a joint distribution $P_{XY}$ and two non-negative real-valued functions $\mathsf{\Phi }:\mathcal{P}\left(\mathcal{X}\right)\to {\mathbb{R}}^{+}$ and $\mathsf{\Psi }:\mathcal{P}\left(\mathcal{Y}\right)\to {\mathbb{R}}^{+}$, we define

$$\mathsf{\Phi }\left(X\right|T)≔\mathbb{E}\left[\mathsf{\Phi }\left(P_{X|T}\right)\right]=\sum _{t\in \mathcal{T}}{P}_T\left(t\right)\mathsf{\Phi }\left(P_{X|T=t}\right),$$

(38)

and

$$\mathsf{\Psi }\left(Y\right|T)≔\mathbb{E}\left[\mathsf{\Psi }\left(P_{Y|T}\right)\right]=\sum _{t\in \mathcal{T}}{P}_T\left(t\right)\mathsf{\Psi }\left(P_{Y|T=t}\right).$$

(39)

When $X\sim P_X$ and $Y\sim P_Y$, we interchangeably write $\mathsf{\Phi }\left(X\right)$ for $\mathsf{\Phi }\left(P_X\right)$ and $\mathsf{\Phi }\left(Y\right)$ for $\mathsf{\Psi }\left(P_Y\right)$.

These definitions provide natural generalizations for Shannon’s entropy and mutual information. Moreover, as we discuss later in Section 3.2 and Section 3.3, it also can be specialized to represent a large family of popular information-theoretic and statistical measures. Examples include information and estimation theoretic quantities such as Arimoto’s conditional entropy of order $\alpha$ for $\mathsf{\Phi }\left(Q_X\right)=\left|\right|Q_X{\left|\right|}_{\alpha }$, probability of correctly guessing for $\mathsf{\Phi }\left(Q_X\right)=\left|\right|Q_X{\left|\right|}_{\infty }$, maximal correlation for binary case, and f-information for $\mathsf{\Phi }\left(Q_X\right)$ given by f-divergence. We are able to generate a family of bottleneck problems using different instantiations of $\mathsf{\Phi }\left(X\right|T)$ and $\mathsf{\Psi }\left(Y\right|T)$ in place of mutual information in $\mathsf{IB}$ and $\mathsf{PF}$. As we argue later, these problems better capture the essence of “informativeness” and “privacy”; thus providing analytical and interpretable guarantees similar in spirit to $\mathsf{IB}$ and $\mathsf{PF}$.

Computing these bottleneck problems in general boils down to the following optimization problems

(40)

and

(41)

Consider the set

(42)

Note that if both $\mathsf{\Phi }$ and $\mathsf{\Psi }$ are continuous (with respect to the total variation distance), then ${\mathcal{M}}_{\mathsf{\Phi },\mathsf{\Psi }}$ is compact. Moreover, it can be easily verified that ${\mathcal{M}}_{\mathsf{\Phi },\mathsf{\Psi }}$ is convex. Hence, its upper and lower boundaries are well-defined and are characterized by the graphs of ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$, respectively. As mentioned earlier, these functional are instrumental for computing the general bottleneck problem later. Hence, before we delve into the examples of bottleneck problems, we extend the approach given in Section 2.2 to compute ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$.

3.1. Evaluation of ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$

Analogous to Section 2.2, we first introduce the Lagrangians of ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$ as

$${\mathcal{L}}_{\mathsf{\Phi },\mathsf{\Psi }}^{\mathsf{U}}\left(\beta \right)≔\underset{P_{T|X}}{\sup }\mathsf{\Psi }\left(Y\right|T)-\beta \mathsf{\Phi }\left(X\right|T),$$

(43)

and

$${\mathcal{L}}_{\mathsf{\Phi },\mathsf{\Psi }}^{\mathsf{L}}\left(\beta \right)≔\underset{P_{T|X}}{\inf }\mathsf{\Psi }\left(Y\right|T)-\beta \mathsf{\Phi }\left(X\right|T),$$

(44)

where $\beta \ge 0$ is the Lagrange multiplier, respectively. Let $(X^{\prime },Y^{\prime })$ be a pair of random variable with $X^{\prime }\sim Q_X$ and $Y^{\prime }$ is the result of passing $X^{\prime }$ through the channel $P_{Y|X}$. Letting

$$F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)≔\mathsf{\Psi }\left(Y^{\prime }\right)-\beta \mathsf{\Phi }\left(X^{\prime }\right),$$

(45)

we obtain that

$${\mathcal{L}}_{\mathsf{\Phi },\mathsf{\Psi }}^{\mathsf{U}}\left(\beta \right)={\mathcal{K}}_{\cap }\left[F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]{|}_{P_X}\mathrm{and}{\mathcal{L}}_{\mathsf{\Phi },\mathsf{\Psi }}^{\mathsf{L}}\left(\beta \right)={\mathcal{K}}_{\cup }\left[F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]{|}_{P_X},$$

(46)

recalling that ${\mathcal{K}}_{\cap }$ and ${\mathcal{K}}_{\cup }$ are the upper concave and lower convex envelop operators. Once we compute ${\mathcal{L}}_{\mathsf{\Phi },\mathsf{\Psi }}^{\mathsf{U}}$ and ${\mathcal{L}}_{\mathsf{\Phi },\mathsf{\Psi }}^{\mathsf{L}}$ for all $\beta \ge 0$, we can use the standard results in optimizations theory (similar to (21) and (22)) to recover ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$. However, we can instead extend the approach Witsenhausen and Wyner [3] described in Section 2.2. Suppose for some $\beta$, ${\mathcal{K}}_{\cap }\left[F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]$ (resp. ${\mathcal{K}}_{\cup }\left[F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]$) at $P_X$ is obtained by a convex combination of points $F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q^i\right)$, $i\in \left[k\right]$ for some $Q^1,\dots ,Q^k$ in $\mathcal{P}\left(\mathcal{X}\right)$, integer $k\ge 2$, and weights ${\lambda }_i\ge 0$ (with ${\sum }_i{\lambda }_i=1$). Then ${\sum }_i{\lambda }_i{Q}^i=P_X$, and $T^{\ast }$ with properties $P_{T^{\ast }}\left(i\right)={\lambda }_i$ and $P_{X|T^{\ast }=i}=Q^i$ attains the maximum (resp. minimum) of $\mathsf{\Psi }\left(Y\right|T)-\beta \mathsf{\Phi }(X\left|T\right)$, implying that $(\mathsf{\Phi }\left(X\right|T^{\ast }),\mathsf{\Psi }\left(Y\right|T^{\ast }))$ is a point on the upper (resp. lower) boundary of $M_{\mathsf{\Phi },\mathsf{\Psi }}$. Consequently, such $T^{\ast }$ satisfies ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}\left(\zeta \right)=\mathsf{\Psi }\left(Y\right|T^{\ast })$ for $\zeta =\mathsf{\Phi }\left(X\right|T^{\ast })$ (resp. ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}\left(\zeta \right)=\mathsf{\Psi }\left(Y\right|T^{\ast })$ for $\zeta =\mathsf{\Phi }\left(X\right|T^{\ast })$). The algorithm to compute ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$ is then summarized in the following three steps:

• Construct the functional $F_{\beta }^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)≔\mathsf{\Psi }\left(Y^{\prime }\right)-\beta \mathsf{\Phi }\left(X^{\prime }\right)$ for $X^{\prime }\sim Q_X$ and $Y^{\prime }\sim Q_X{P}_{Y|X}$ and all $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$ and $\beta \ge 0$.
• Compute ${\mathcal{K}}_{\cap }\left[F^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]{|}_{P_X}$ and ${\mathcal{K}}_{\cup }\left[F^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]{|}_{P_X}$ evaluated at $P_X$.
• If for distributions $Q^1,\dots ,Q^k$ in $\mathcal{P}\left(X\right)$ for some $k\ge 1$, we have ${\mathcal{K}}_{\cap }\left[F^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]{|}_{P_X}={\sum }_{i=1}^k{\lambda }_i{F}^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q^i\right)$ or ${\mathcal{K}}_{\cup }\left[F^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q_X\right)\right]{|}_{P_X}={\sum }_{i=1}^k{\lambda }_i{F}^{\mathsf{\Phi },\mathsf{\Psi }}\left(Q^i\right)$ for some ${\lambda }_i\ge 0$ satisfying ${\sum }_{i=1}^k{\lambda }_i=1$, then then $P_{X|T=i}=Q_i$, $i\in \left[k\right]$ and $P_T\left(i\right)={\lambda }_i$ give the optimal $T^{\ast }$ in ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$, respectively.

We will apply this approach to analytically compute ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$ (and the corresponding bottleneck problems) for binary cases in the following sections.

3.2. Guessing Bottleneck Problems

Let $P_{XY}$ be given with marginals $P_X$ and $P_Y$ and the corresponding channel $P_{Y|X}$. Let also $Q_X\in \mathcal{P}\left(\mathcal{X}\right)$ be an arbitrary distribution on $\mathcal{X}$ and $Q_Y=Q_X{P}_{Y|X}$ be the output distribution of $P_{Y|X}$ when fed with $Q_X$. Any channel $P_{T|X}$, together with the Markov structure Y X T, generates unique $P_{X|T}$ and $P_{Y|T}$. We need the following basic definition from statistics.

Definition 1.

Let U be a discrete and V be an arbitrary random variables supported on $\mathcal{U}$ and $\mathcal{V}$ with $\left|\mathcal{U}\right|<\infty$, respectively. Then ${\mathsf{P}}_{\mathsf{c}}\left(U\right)$ the probability of correctly guessing U and ${\mathsf{P}}_{\mathsf{c}}\left(U\right|V)$ the probability of correctly guessing U given V are given by

$${\mathsf{P}}_{\mathsf{c}}\left(U\right)≔\underset{u\in \mathcal{U}}{\max }{P}_U\left(u\right),$$

and

$${\mathsf{P}}_{\mathsf{c}}\left(U\right|V)≔\underset{g}{\max }Pr(U=g\left(V\right))=\mathbb{E}\left[\underset{u\in \mathcal{U}}{\max }{P}_{U|V}\left(u\right|V)\right].$$

Moreover, the multiplicative gain of the observation V in guessing U is defined as (the reason for ∞ in the notation becomes clear later)

$$I_{\infty }(U;V)≔\log \frac{{\mathsf{P}}_{\mathsf{c}}\left(U\right|V)}{{\mathsf{P}}_{\mathsf{c}}\left(U\right)}.$$

As the names suggest, ${\mathsf{P}}_{\mathsf{c}}\left(U\right|V)$ and ${\mathsf{P}}_{\mathsf{c}}\left(U\right)$ characterize the optimal efficiency of guessing U with or without the observation V, respectively. Intuitively, $I_{\infty }(U;V)$ quantifies how useful the observation V is in estimating U: If it is small, then it means it is nearly as hard for an adversary observing V to guess U as it is without V. This observation motivates the use of $I_{\infty }(Y;T)$ as a measure of privacy in lieu of $I(Y;T)$ in $\mathsf{PF}$.

It is worth noting that $I_{\infty }(U;V)$ is not symmetric in general, i.e., $I_{\infty }(U;V)\ne I_{\infty }(V;U)$. Since observing T can only improve, we have ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)\ge {\mathsf{P}}_{\mathsf{c}}\left(Y\right)$; thus $I_{\infty }(Y;T)\ge 0$. However, $I_{\infty }(Y;T)=0$ does not necessarily imply independent of Y and T; instead, it means T is useless in estimating Y. As an example, consider $Y\sim \mathsf{Bernoulli}\left(p\right)$ and $P_{T|Y=0}=\mathsf{Bernoulli}\left(\delta \right)$ and $P_{T|Y=1}=\mathsf{Bernoulli}\left(\eta \right)$ with $\delta ,\eta \le \frac{1}{2}<p$. Then ${\mathsf{P}}_{\mathsf{c}}\left(Y\right)=p$ and

$${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)=\max \{\overline{\delta }\overline{p},\eta p\}+\overline{\eta }p.$$

Thus, if $\overline{\delta }\overline{p}\le \eta p$, then ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)={\mathsf{P}}_{\mathsf{c}}\left(Y\right)$. This then implies that $I_{\infty }(Y;T)=0$ whereas Y and T are clearly dependent; i.e., $I(Y;T)>0$. While in general $I(Y;T)$ and $I_{\infty }(Y;T)$ are not related, it can be shown that $I(Y;T)\le I_{\infty }(Y;T)$ if Y is uniform (see [65] (Proposition 1)). Hence, only with this uniformity assumption, $I_{\infty }(Y;T)$ implies the independence.

Consider $\mathsf{\Psi }\left(Q_X\right)=-{\sum }_{x\in \mathcal{X}}{Q}_X\left(x\right)\log \left(Q_X\left(x\right)\right)$ and $\mathsf{\Psi }\left(Q_Y\right)={\parallel Q_Y\parallel }_{\infty }$. Clearly, we have $\mathsf{\Phi }\left(X\right|T)=H(X\left|T\right)$. Note that

$$\mathsf{\Psi }\left(Y\right|T)=\sum _{t\in \mathcal{T}}{P}_T\left(t\right)\left|\right|P_{Y|T=t}{\left|\right|}_{\infty }={\mathsf{P}}_{\mathsf{c}}\left(Y\right|T),$$

(47)

thus both measures $H\left(X\right|T)$ and ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$ are special cases of the models described in the previous section. In particular, we can define the corresponding ${\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}$ and ${\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}$. We will see later that $I(X;T)$ and ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$ correspond to Arimoto’s mutual information of orders 1 and ∞, respectively. Define

(48)

This bottleneck functional formulated an interpretable guarantee:

$$\begin{gathered} {\mathsf{IB}}^{(\infty ,1)}\left(R\right)\mathrm{characterizes} \mathrm{the} \mathrm{best} \mathrm{error} \mathrm{probability} \mathrm{in} \mathrm{recovering} Y \\ \mathrm{among} \mathrm{all} R-\mathrm{bit} \mathrm{summaries} \mathrm{of} X \end{gathered}$$

Recall that the functional $\mathsf{PF}\left(r\right)$ aims at extracting maximum information of X while protecting privacy with respect to Y. Measuring the privacy in terms of ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$, this objective can be better formulated by

(49)

with the interpretable privacy guarantee:

$$\begin{gathered} {\mathsf{PF}}^{(\infty ,1)}\left(r\right)\mathrm{characterizes}\mathrm{the}\mathrm{smallest}\mathrm{probability}\mathrm{of}\mathrm{revealing}\mathrm{private}\mathrm{feature}Y \\ \mathrm{among}\mathrm{all}\mathrm{representations}\mathrm{of}X\mathrm{preserving}\mathrm{at}\mathrm{least}r\mathrm{bits}\mathrm{information}\mathrm{of}X \end{gathered}$$

Notice that the variable T in the formulations of ${\mathsf{IB}}^{(\infty ,1)}$ and ${\mathsf{PF}}^{(\infty ,1)}$ takes values in a set $\mathcal{T}$ of arbitrary cardinality. However, a straightforward application of the Carathéodory-Fenchel-Eggleston theorem (see e.g., [79] (Lemma 15.4)) reveals that the cardinality of $\mathcal{T}$ can be restricted to $\left|\mathcal{X}\right|+1$ without loss of generality. In the following lemma, we prove more basic properties of ${\mathsf{IB}}^{(\infty ,1)}$ and ${\mathsf{PF}}^{(\infty ,1)}$.

Lemma 7.

For any $P_{XY}$ with Y supported on a finite set $\mathcal{Y}$, we have

• ${\mathsf{IB}}^{(\infty ,1)}\left(0\right)={\mathsf{PF}}^{(\infty ,1)}\left(0\right)=0$.
• ${\mathsf{IB}}^{(\infty ,1)}\left(R\right)=I_{\infty }(X;Y)$ for any $R\ge H\left(X\right)$ and ${\mathsf{PF}}^{(\infty ,1)}\left(r\right)=I_{\infty }(X;Y)$ for $r\ge H\left(X\right)$.
• $R\mapsto exp\left({\mathsf{IB}}^{(\infty ,1)}\left(R\right)\right)$ is strictly increasing and concave on the range $(0,I_{\infty }(X;Y))$.
• $r\mapsto exp\left({\mathsf{PF}}^{(\infty ,1)}\left(r\right)\right)$ is strictly increasing, and convex on the range $(0,I_{\infty }(X;Y))$.

The proof follows the same lines as Theorem 1 and hence omitted. Lemma 7 in particular implies that inequalities $I(X;T)\le R$ and $I(X;T)\ge r$ in the definition of ${\mathsf{IB}}^{(\infty ,1)}$ and ${\mathsf{PF}}^{(\infty ,1)}$ can be replaced by $I(X;T)=R$ and $I(X;T)=r$, respectively. It can be verified that $I^{\infty }$ satisfies the data-processing inequality, i.e., $I^{\infty }(Y;T)\le I^{\infty }(Y;X)$ for the Markov chain Y X T. Hence, both ${\mathsf{IB}}^{(\infty ,1)}$ and ${\mathsf{PF}}^{(\infty ,1)}$ must be smaller than $I_{\infty }(Y;X)$. The properties listed in Lemma 7 enable us to derive a slightly tighter upper bound for ${\mathsf{PF}}^{(\infty ,1)}$ as demonstrated in the following.

Lemma 8.

For any $P_{XY}$ with Y supported on a finite set $\mathcal{Y}$, we have

$${\mathsf{PF}}^{(\infty ,1)}\left(r\right)\le \log \left[1+\frac{r}{H\left(X\right)}\left(e^{I_{\infty }(Y;X)}-1\right)\right],$$

and

$$\log \left[1+\frac{R}{H\left(X\right)}\left(e^{I_{\infty }(Y;X)}-1\right)\right]\le {\mathsf{IB}}^{(\infty ,1)}\left(R\right)\le I_{\infty }(Y;X).$$

The proof of this lemma (and any other results in this section) is given in Appendix B. This lemma shows that the gap between $I_{\infty }(Y;X)$ and ${\mathsf{IB}}^{(\infty ,1)}\left(R\right)$ when R is sufficiently close to $H\left(X\right)$ behaves like

$$I_{\infty }(Y;X)-{\mathsf{IB}}^{(\infty ,1)}\left(R\right)\le I_{\infty }(Y;X)-\log \left[1+\frac{R}{H\left(X\right)}\left(e^{I_{\infty }(Y;X)}-1\right)\right]\approx \left(1-e^{-I_{\infty }(Y;X)}\right)\left(1-\frac{R}{H\left(X\right)}\right).$$

Thus, ${\mathsf{IB}}^{(\infty ,1)}\left(R\right)$ approaches $I_{\infty }(Y;X)$ as $R\to H\left(X\right)$ at least linearly.

In the following theorem, we apply the technique delineated in Section 3.1 to derive closed form expressions for ${\mathsf{IB}}^{(\infty ,1)}$ and ${\mathsf{PF}}^{(\infty ,1)}$ for the binary symmetric case, thereby establishing similar results as Mr and Mrs. Gerber’s Lemma.

Theorem 11.

For $X\sim \mathsf{Bernoulli}\left(p\right)$ and $P_{Y|X}=\mathsf{BSC}\left(\delta \right)$ with $p,\delta \le \frac{1}{2}$, we have

$${\mathsf{PF}}^{(\infty ,1)}\left(r\right)=\log \left[\frac{\overline{\delta }-(h_{\mathsf{b}}\left(p\right)-r)(\frac{1}{2}-\delta )}{1-\delta \ast p}\right],$$

(50)

and

$${\mathsf{IB}}^{(\infty ,1)}\left(R\right)=\log \left[\frac{1-\delta \ast h_{\mathsf{b}}^{-1}(h_{\mathsf{b}}\left(p\right)-R)}{1-\delta \ast p}\right],$$

(51)

where $\overline{\delta }=1-\delta$.

As described in Section 3.1, to compute ${\mathsf{IB}}^{(\infty ,1)}$ and ${\mathsf{PF}}^{(\infty ,1)}$ it suffices to derive the convex and concave envelopes of the mapping $F_{\beta }^{(\infty ,1)}\left(q\right)≔{\mathsf{P}}_{\mathsf{c}}\left(Y^{\prime }\right)+\beta H\left(X^{\prime }\right)$ where $X^{\prime }\sim \mathsf{Bernoulli}\left(q\right)$ and $Y^{\prime }$ is the result of passing $X^{\prime }$ through $\mathsf{BSC}\left(\delta \right)$, i.e., $Y^{\prime }\sim \mathsf{Bernoulli}(\delta \ast q)$. In this case, ${\mathsf{P}}_{\mathsf{c}}\left(Y^{\prime }\right)=\max \{\delta \ast q,1-\delta \ast q\}$ and $F_{\beta }^{(\infty ,1)}$ can be expressed as

$$q\mapsto F_{\beta }^{(\infty ,1)}\left(q\right)=\max \{\delta \ast q,1-\delta \ast q\}+\beta h_{\mathsf{b}}\left(q\right).$$

(52)

This function is depicted in Figure 6.

Figure 6. The mapping $q\mapsto F_{\beta }^{(\infty ,1)}\left(q\right)={\mathsf{P}}_{\mathsf{c}}\left(Y^{\prime }\right)+\beta H\left(X^{\prime }\right)$ where $X^{\prime }\sim \mathsf{Bernoulli}\left(q\right)$ and $Y^{\prime }\sim \mathsf{Bernoulli}\left(q\right)\mathsf{BSC}\left(0.1\right)$.

The detailed derivation of convex and concave envelope of $F_{\beta }^{(\infty ,1)}$ is given in Appendix B. The proof of this theorem also reveals the following intuitive statements. If $X\sim \mathsf{Bernoulli}\left(p\right)$ and $P_{Y|X}=\mathsf{BSC}\left(\delta \right)$, then among all random variables T satisfying Y X T and $H\left(X\right|T)\le \lambda$, the minimum ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$ is given by $\overline{\delta }-\lambda (0.5-\delta )$. Notice that, without any information constraint (i.e., $\lambda =0$), ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)={\mathsf{P}}_{\mathsf{c}}\left(Y\right|X)=\overline{\delta }$. Perhaps surprisingly, this shows that the mutual information constraint has a linear effect on the privacy of Y. Similarly, to prove (51), we show that among all R-bit representations T of X, the best achievable accuracy ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$ is given by $1-\delta \ast h_{\mathsf{b}}^{-1}(h_{\mathsf{b}}\left(p\right)-R)$. This can be proved by combining Mrs. Gerber’s Lemma (cf. Lemma 4) and Fano’s inequality as follows. For all T such that $H\left(X\right|T)\ge \lambda$, the minimum of $H\left(Y\right|T)$ is given by $h_{\mathsf{b}}(\delta \ast h_{\mathsf{b}}^{-1}\left(\lambda \right))$. Since by Fano’s inequality, $H\left(Y\right|T)\le h_{\mathsf{b}}(1-{\mathsf{P}}_{\mathsf{c}}\left(Y\right|T))$, we obtain $\delta \ast h_{\mathsf{b}}^{-1}\left(\lambda \right)\le 1-{\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$ which leads to the same result as above. Nevertheless, in Appendix B we give another proof based on the discussion of Section 3.1.

3.3. Arimoto Bottleneck Problems

The bottleneck framework proposed in the last section benefited from interpretable guarantees brought forth by the quantity $I_{\infty }$. In this section, we define a parametric family of statistical quantities, the so-called Arimoto’s mutual information, which includes both Shannon’s mutual information and $I_{\infty }$ as extreme cases.

Definition 2

([22]). Let $U\sim P_U$ and $V\sim P_V$ be two random variables supported over finite sets $\mathcal{U}$ and $\mathcal{V}$, respectively. Their Arimoto’s mutual information of order $\alpha >1$ is defined as

$$I_{\alpha }(U;V)=H_{\alpha }\left(U\right)-H_{\alpha }\left(U\right|V),$$

(53)

where

$$H_{\alpha }\left(U\right)≔\frac{\alpha }{1-\alpha }\log \left|\right|P_U{\left|\right|}_{\alpha },$$

(54)

is the Rényi entropy of order α and

$$H_{\alpha }\left(U\right|V)≔\frac{\alpha }{1-\alpha }\log \sum _{v\in \mathcal{V}}{P}_V\left(v\right)\left|\right|P_{U|V=v}{\left|\right|}_{\alpha },$$

(55)

is the Arimoto’s conditional entropy of order α.

By continuous extension, one can define $I-\alpha (U;V)$ for $\alpha =1$ and $\alpha =\infty$ as $I(U;V)$ and $I_{\infty }(U;V)$, respectively. That is,

$$\underset{\alpha \to 1^{+}}{\lim }{I}_{\alpha }(U;V)=I(U;V),\mathrm{and}\underset{\alpha \to \infty }{\lim }{I}_{\alpha }(U;V)=I_{\infty }(U;V).$$

(56)

Arimoto’s mutual information was first introduced by Arimoto [22] and then later revisited by Liese and Vajda in [101] and more recently by Verdú in [102]. More in-depth analysis and properties of $I_{\alpha }$ can be found in [103]. It is shown in [71] (Lemma 1) that $I_{\alpha }(U;V)$ for $\alpha \in [1,\infty ]$ quantifies the minimum loss in recovering U given V where the loss is measured in terms of the so-called $\alpha$-loss. This loss function reduces to logarithmic loss (27) and ${\mathsf{P}}_{\mathsf{c}}\left(U\right|V)$ for $\alpha =1$ and $\alpha =\infty$, respectively. This sheds light on the utility and/or privacy guarantee promised by a constraint on Arimoto’s mutual information. It is now natural to use $I_{\alpha }$ for defining a family of bottleneck problems.

Definition 3.

Given a pair of random variables $(X,Y)\sim P_{XY}$ over finite sets $\mathcal{X}$ and $\mathcal{Y}$ and $\alpha ,\gamma \in [1,\infty ]$, we define ${\mathsf{IB}}^{(\alpha ,\gamma )}$ and ${\mathsf{PF}}^{(\alpha ,\gamma )}$ as

(57)

and

(58)

Of course, ${\mathsf{IB}}^{(1,1)}\left(R\right)=\mathsf{IB}\left(R\right)$ and ${\mathsf{PF}}^{(1,1)}\left(r\right)=\mathsf{PF}\left(r\right)$. It is known that Arimoto’s mutual information satisfies the data-processing inequality [103] (Corollary 1), i.e., $I_{\alpha }(Y;T)\le I_{\alpha }(Y;X)$ for the Markov chain Y X T. On the other hand, $I_{\gamma }(X;T)\le H_{\gamma }\left(X\right)$. Thus, both ${\mathsf{IB}}^{(\alpha ,\gamma )}\left(R\right)$ and ${\mathsf{PF}}^{(\alpha ,\gamma )}\left(r\right)$ equal $I_{\alpha }(Y;X)$ for $R,r\ge H_{\gamma }\left(X\right)$. Note also that $H_{\alpha }\left(Y\right|T)=\frac{\alpha }{1-\alpha }\log \mathsf{\Psi }\left(Y\right|T)$ where $\mathsf{\Psi }\left(Y\right|T)$ (see (39)) corresponding to the function $\mathsf{\Psi }\left(Q_Y\right)=\left|\right|Q_Y{\left|\right|}_{\alpha }$. Consequently, ${\mathsf{IB}}^{(\alpha ,\gamma )}$ and ${\mathsf{PF}}^{(\alpha ,\gamma )}$ are characterized by the lower and upper boundary of ${\mathcal{M}}_{\mathsf{\Phi },\mathsf{\Psi }}$, defined in (37), with respect to $\mathsf{\Phi }\left(Q_X\right)=\left|\right|Q_X{\left|\right|}_{\gamma }$ and $\mathsf{\Psi }\left(Q_Y\right)=\left|\right|Q_Y{\left|\right|}_{\alpha }$. Specifically, we have

$${\mathsf{IB}}^{(\alpha ,\gamma )}\left(R\right)=H_{\alpha }\left(Y\right)+\frac{\alpha }{\alpha -1}\log {\mathsf{U}}_{\mathsf{\Phi },\mathsf{\Psi }}\left(\zeta \right),$$

(59)

where $\zeta =e^{-(1-\frac{1}{\gamma })(H_{\gamma }\left(X\right)-R)}$, and

$${\mathsf{PF}}^{(\alpha ,\gamma )}\left(r\right)=H_{\alpha }\left(Y\right)+\frac{\alpha }{\alpha -1}\log {\mathsf{L}}_{\mathsf{\Phi },\mathsf{\Psi }}\left(\zeta \right),$$

(60)

where $\zeta =e^{-(1-\frac{1}{\gamma })(H_{\gamma }\left(X\right)-r)}$ and $\mathsf{\Phi }\left(Q_X\right)={\parallel Q_X\parallel }_{\gamma }$ and $\mathsf{\Psi }\left(Q_Y\right)={\parallel Q_Y\parallel }_{\alpha }$. This paves the way to apply the technique described in Section 2.2 to compute ${\mathsf{IB}}^{(\alpha ,\gamma )}$ and ${\mathsf{PF}}^{(\alpha ,\gamma )}$. Doing so requires the upper concave and lower convex envelope of the mapping $Q_X\mapsto \parallel Q_Y{\parallel }_{\alpha }-\beta {\parallel Q_X\parallel }_{\gamma }$ for some $\beta \ge 0$, where $Q_Y\sim Q_X{P}_{Y|X}$. In the following theorem, we drive these envelopes and give closed form expressions for ${\mathsf{IB}}^{(\alpha ,\gamma )}$ and ${\mathsf{PF}}^{(\alpha ,\gamma )}$ for a special case where $\alpha =\gamma \ge 2$.

Theorem 12.

Let $X\sim \mathsf{Bernoulli}\left(p\right)$ and $P_{Y|X}=\mathsf{BSC}\left(\delta \right)$ with $p,\delta \le \frac{1}{2}$. We have for $\alpha \ge 2$

$${\mathsf{PF}}^{(\alpha ,\alpha )}\left(r\right)=\frac{\alpha }{1-\alpha }\log \left[\frac{{\parallel p\ast \delta \parallel }_{\alpha }}{{\parallel q\ast \delta \parallel }_{\alpha }}\right],$$

where ${\parallel a\parallel }_{\alpha }≔{\parallel [a,\overline{a}]\parallel }_{\alpha }$ for $a\in [0,1]$ and $q\le p$ solves

$$\frac{\alpha }{1-\alpha }\log \left[\frac{{\parallel p\parallel }_{\alpha }}{{\parallel q\parallel }_{\alpha }}\right]=r.$$

Moreover,

$${\mathsf{IB}}^{(\alpha ,\alpha )}\left(R\right)=\frac{\alpha }{\alpha -1}\log \left[\frac{\overline{\lambda }{\parallel \delta \parallel }_{\alpha }+\lambda {\parallel \frac{q}{z}\ast \delta \parallel }_{\alpha }}{{\parallel p\ast \alpha \parallel }_{\alpha }}\right],$$

where $z=\max \{2p,\lambda \}$ and $\lambda \in [0,1]$ solves

$$\frac{\alpha }{\alpha -1}\log \frac{\overline{\lambda }+\lambda {\parallel \frac{p}{z}\parallel }_{\alpha }}{{\parallel p\parallel }_{\alpha }}=R.$$

By letting $\alpha \to \infty$, this theorem indicates that for X and Y connected through $\mathsf{BSC}\left(\delta \right)$ and all variables T forming Y X T, we have

$${\mathsf{P}}_{\mathsf{c}}\left(X\right|T)\ge \lambda ⟹{\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)\ge \delta \ast \lambda ,$$

(61)

which can be shown to be achieved $T^{\ast }$ generated by the following channel (see Figure 7)

$$P_{T^{\ast }|X}=\left[\begin{array}{cc}\frac{\lambda -p}{\overline{p}}& \frac{\overline{\lambda }}{\overline{p}}\\ 0& 1\end{array}\right].$$

(62)

Note that, by assumption, $p\le \frac{1}{2}$, and hence the event $\{X=1\}$ is less likely than $\{X=0\}$. Therefore, (61) demonstrates that to ensure correct recoverability of X with probability at lest $\lambda$, the most private approach (with respect to Y) is to obfuscate the higher-likely event $\{X=0\}$ with probability $\frac{\overline{\lambda }}{\overline{p}}$. As demonstrated in (61) the optimal privacy guarantee is linear in the utility parameter in the binary symmetric case. This is in fact a special case of the larger result recently proved in [65] (Theorem 1): the infimum of ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)$ over all variables T such that ${\mathsf{P}}_{\mathsf{c}}\left(X\right|T)\ge \lambda$ is piece-wise linear in $\lambda$, on equivalently, the mapping $e^r\mapsto exp\left({\mathsf{PF}}^{(\infty ,\infty )}\left(r\right)\right)$ is piece-wise linear.

Figure 7. The structure of the optimal $P_{T|X}$ for ${\mathsf{PF}}^{(\infty ,\infty )}$ when $P_{Y|X}=\mathsf{BSC}\left(\delta \right)$ and $X\sim \mathsf{Bernoulli}\left(p\right)$ with $\delta ,p\in [0,\frac{1}{2}]$. If the accuracy constraint is ${\mathsf{P}}_{\mathsf{c}}\left(X\right|T)\ge \lambda$ (or equivalently $I_{\infty }\left(X\right|T)\ge \log \frac{\lambda }{\overline{p}}$), then the parameter of optimal $P_{T|X}$ is given by $\eta =\frac{\overline{\lambda }}{\overline{p}}$, leading to ${\mathsf{P}}_{\mathsf{c}}\left(Y\right|T)=\delta \ast \lambda$.

Computing ${\mathsf{PF}}^{(\alpha ,\gamma )}$ analytically for every $\alpha ,\gamma >1$ seems to be challenging, however, the following lemma provides bounds for ${\mathsf{PF}}^{(\alpha ,\gamma )}$ and ${\mathsf{IB}}^{(\alpha ,\gamma )}$ in terms of ${\mathsf{PF}}^{(\infty ,\infty )}$ and ${\mathsf{IB}}^{(\infty ,\infty )}$, respectively.

Lemma 9.

For any pair of random variables $(X,Y)$ over finite alphabets and $\alpha ,\gamma >1$, we have

$$\frac{\alpha }{\alpha -1}{\mathsf{PF}}^{(\infty ,\infty )}\left(f\left(r\right)\right)-\frac{\alpha }{\alpha -1}{H}_{\infty }\left(Y\right)+H_{\alpha }\left(Y\right)\le {\mathsf{PF}}^{(\alpha ,\gamma )}\left(r\right)\le {\mathsf{PF}}^{(\infty ,\infty )}\left(g\left(r\right)\right)+H_{\alpha }\left(Y\right)-H_{\infty }\left(Y\right),$$

and

$$\frac{\alpha }{\alpha -1}{\mathsf{IB}}^{(\infty ,\infty )}\left(f\left(R\right)\right)-\frac{\alpha }{\alpha -1}{H}_{\infty }\left(Y\right)+H_{\alpha }\left(Y\right)\le {\mathsf{IB}}^{(\alpha ,\gamma )}\left(R\right)\le {\mathsf{IB}}^{(\infty ,\infty )}\left(g\left(R\right)\right)+H_{\alpha }\left(Y\right)-H_{\infty }\left(Y\right),$$

where $f\left(a\right)=\max \{a-H_{\gamma }\left(X\right)+H_{\infty }\left(X\right),0\}$ and $g\left(b\right)=\frac{\gamma -1}{\gamma }b+H_{\infty }\left(X\right)-\frac{\gamma -1}{\gamma }{H}_{\gamma }\left(X\right)$.

The previous lemma can be directly applied to derive upper and lower bounds for ${\mathsf{PF}}^{(\alpha ,\gamma )}$ and ${\mathsf{IB}}^{(\alpha ,\gamma )}$ given ${\mathsf{PF}}^{(\infty ,\infty )}$ and ${\mathsf{IB}}^{(\infty ,\infty )}$.

3.4. f-Bottleneck Problems

In this section, we describe another instantiation of the general framework introduced in terms of functions $\mathsf{\Phi }$ and $\mathsf{\Psi }$ that enjoys interpretable estimation-theoretic guarantee.

Definition 4.

Let $f:(0,\infty )\to \mathbb{R}$ be a convex function with $f\left(1\right)=0$. Furthermore, let U and V be two real-valued random variables supported over $\mathcal{U}$ and $\mathcal{V}$, respectively. Their f-information is defined by

$$I_f(U;V)≔D_f(P_{UV}\parallel P_U{P}_V),$$

(63)

where $D_f(·\parallel ·)$ is the f-divergence [104] between distributions and defined as

$$D_f(P\parallel Q)≔{\mathbb{E}}_Q\left[f\left(\frac{dP}{dQ}\right)\right].$$

Due to convexity of f, we have $D_f(P\parallel Q)\ge f\left(1\right)=0$ and hence f-information is always non-negative. If, furthermore, f is strictly convex at 1, then equality holds if and only $P=Q$. Csiszár introduced f-divergence in [104] and applied it to several problems in statistics and information theory. More recent developments about the properties of f-divergence and f-information can be found in [23] and the references therein. Any convex function f with the property $f\left(1\right)=0$ results in an f-information. Popular examples include $f\left(t\right)=t\log t$ corresponding to Shannon’s mutual information, $f\left(t\right)=|t-1|$ corresponding to T-information [83], and also $f\left(t\right)=t^2-1$ corresponding to ${\chi }^2$-information [69] for. It is worth mentioning that if we allow $\alpha$ to be in $(0,1)$ in Definition 2 (similar to [101]), then the resulting Arimoto’s mutual information can be shown to be an f-information in the binary case for a certain function f, see [101] (Theorem 8).

Let $(X,Y)\sim P_{XY}$ be given with marginals $P_X$ and $P_Y$. Consider functions $\mathsf{\Phi }$ and $\mathsf{\Psi }$ on $\mathcal{P}\left(\mathcal{X}\right)$ and $\mathcal{P}\left(\mathcal{Y}\right)$ defined as

$$\mathsf{\Phi }\left(Q_X\right)≔D_f(Q_X\parallel P_X)\mathrm{and}\mathsf{\Psi }\left(Q_Y\right)≔D_f(Q_Y\parallel P_Y).$$

Given a conditional distribution $P_{T|X}$, it is easy to verify that $\mathsf{\Phi }\left(X\right|T)=I_f(X;T)$ and $\mathsf{\Psi }\left(Y\right|T)=I_f(Y;T)$. This in turn implies that f-information can be utilized in (40) and (41) to define general bottleneck: Let $f:(0,\infty )\to \mathbb{R}$ and $g:(0,\infty )\to \mathbb{R}$ be two convex functions satisfying $f\left(1\right)=g\left(1\right)=0$. Then we define

(64)

and

(65)

In light of the discussion in Section 3.1, the optimization problems in ${\mathsf{IB}}^{(f,g)}$ and ${\mathsf{IB}}^{(f,g)}$ can be analytically solved by determining the upper concave and lower convex envelope of the mapping

$$Q_X\mapsto {\mathcal{F}}_{\beta }^{(f,g)}≔D_f(Q_Y\parallel P_Y)-\beta D_g(Q_X\parallel P_X),$$

(66)

where $\beta \ge 0$ is the Lagrange multiplier and $Q_Y=Q_X{P}_{Y|X}$.

Consider the function $f_{\alpha }\left(t\right)=\frac{t^{\alpha }-1}{\alpha -1}$ with $\alpha \in (1,\infty )\cup (1,\infty )$. The corresponding f-divergence is sometimes called Hellinger divergence of order $\alpha$, see e.g., [105]. Note that Hellinger divergence of order 2 reduces to ${\chi }^2$-divergence. Calmon et al. [68] and Asoodeh et al. [67] showed that if $I_{f_2}(Y;T)\le \epsilon$ for some $\epsilon \in (0,1)$, then the minimum mean-squared error (MMSE) of reconstructing any zero-mean unit-variance function of Y given T is lower bounded by $1-\epsilon$, i.e., no function of Y can be reconstructed with small MMSE given an observation of T. This result serves a natural justification for $I_{f_2}$ as an operational measure of both privacy and utility in a bottleneck problem.

Unfortunately, our approach described in Section 3.1 cannot be used to compute ${\mathsf{IB}}^{(f_2,f_2)}$ or ${\mathsf{PF}}^{(f_2,f_2)}$ in the binary symmetric case. The difficulty lies in the fact that the function ${\mathcal{F}}_{\beta }^{f_2,f_2}$, defined in (66), for the binary symmetric case is either convex or concave on its entire domain depending on the value of $\beta$. Nevertheless, one can consider Hellinger divergence of order $\alpha$ with $\alpha \ne 2$ and then apply our approach to compute ${\mathsf{IB}}^{(f_{\alpha },f_{\alpha })}$ or ${\mathsf{PF}}^{(f_{\alpha },f_{\alpha })}$. Since $D_{f_2}(P\parallel Q)\le {(1+(\alpha -1)D_{f_{\alpha }}(P\parallel Q))}^{1/(\alpha -1)}-1$ (see [106] (Corollary 5.6)), one can justify $I_{f_{\alpha }}$ as a measure of privacy and utility in a similar way as $I_{f_2}$.

We end this section by a remark about estimating the measures studied in this section. While we consider information-theoretic regime where the underlying distribution $P_{XY}$ is known, in practice only samples $(x_i,y_i)$ are given. Consequently, the de facto guarantees of bottleneck problems might be considerably different from those shown in this work. It is therefore essential to asses the guarantees of bottleneck problems when accessing only samples. To do so, one must derive bounds on the discrepancy between ${\mathsf{P}}_{\mathsf{c}}$, $I_{\alpha }$, and $I_f$ computed on the empirical distribution and the true (unknown) distribution. These bounds can then be used to shed light on the de facto guarantee of the bottleneck problems. Relying on [34] (Theorem 1), one can obtain that the gaps between the measures ${\mathsf{P}}_{\mathsf{c}}$, $I_{\alpha }$, and $I_f$ computed on empirical distributions and the true one scale as $O(1/\sqrt{n})$ where n is the number of samples. This is in contrast with mutual information for which the similar upper bound scales as $O(\log n/\sqrt{n})$ as shown in [33]. Therefore, the above measures appear to be easier to estimate than mutual information.

4. Summary and Concluding Remarks

Following the recent surge in the use of information bottleneck ($\mathsf{IB}$) and privacy funnel ($\mathsf{PF}$) in developing and analyzing machine learning models, we investigated the functional properties of these two optimization problems. Specifically, we showed that $\mathsf{IB}$ and $\mathsf{PF}$ correspond to the upper and lower boundary of a two-dimensional convex set $\mathcal{M}=\left\{\right(I(X;T),I(Y;T)):$ Y X T} where $(X,Y)\sim P_{XY}$ represents the observable data X and target feature Y and the auxiliary random variable T varies over all possible choices satisfying the Markov relation Y X T. This unifying perspective on $\mathsf{IB}$ and $\mathsf{PF}$ allowed us to adapt the classical technique of Witsenhausen and Wyner [3] devised for computing $\mathsf{IB}$ to be applicable for $\mathsf{PF}$ as well. We illustrated this by deriving a closed form expression for $\mathsf{PF}$ in the binary case—a result reminiscent of the Mrs. Gerber’s Lemma [2] in information theory literature. We then showed that both $\mathsf{IB}$ and $\mathsf{PF}$ are closely related to several information-theoretic coding problems such as noisy random coding, hypothesis testing against independence, and dependence dilution. While these connections were partially known in previous work (see e.g., [29,30]), we show that they lead to an improvement on the cardinality of T for computing $\mathsf{IB}$. We then turned our attention to the continuous setting where X and Y are continuous random variables. Solving the optimization problems in $\mathsf{IB}$ and $\mathsf{PF}$ in this case without any further assumptions seems a difficult challenge in general and leads to theoretical results only when $(X,Y)$ is jointly Gaussian. Invoking recent results on the entropy power inequality [25] and strong data processing inequality [27], we obtained tight bounds on $\mathsf{IB}$ in two different cases: (1) when Y is a Gaussian perturbation of X and (2) when X is a Gaussian perturbation of Y. We also utilized the celebrated I-MMSE relationship [107] to derive a second-order approximation of $\mathsf{PF}$ when T is considered to be a Gaussian perturbation of X.

In the second part of the paper, we argue that the choice of (Shannon’s) mutual information in both $\mathsf{IB}$ and $\mathsf{PF}$ does not seem to carry specific operational significance. It does, however, have a desirable practical consequence: it leads to self-consistent equations [1] that can be solved iteratively (without any guarantee to convergence though). In fact, this property is unique to mutual information among other existing information measures [99]. Nevertheless, we argued that other information measures might lead to better interpretable guarantee for both $\mathsf{IB}$ and $\mathsf{PF}$. For instance, statistical accuracy in $\mathsf{IB}$ and privacy leakage in $\mathsf{PF}$ can be shown to be precisely characterized by probability of correctly guessing (aka Bayes risk) or minimum mean-squared error (MMSE). Following this observation, we introduced a large family of optimization problems, which we call bottleneck problems, by replacing mutual information in $\mathsf{IB}$ and $\mathsf{PF}$ with Arimoto’s mutual information [22] or f-information [23]. Invoking results from [33,34], we also demonstrated that these information measures are in general easier to estimate from data than mutual information. Similar to $\mathsf{IB}$ and $\mathsf{PF}$, the bottleneck problems were shown to be fully characterized by boundaries of a two-dimensional convex set parameterized by two real-valued non-negative functions $\mathsf{\Phi }$ and $\mathsf{\Psi }$. This perspective enabled us to generalize the technique used to compute $\mathsf{IB}$ and $\mathsf{PF}$ for evaluating bottleneck problems. Applying this technique to the binary case, we derived closed form expressions for several bottleneck problems.