Information Extraction Under Privacy Constraints ^†

Abstract

A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables $(X,Y)$ governed by a given joint distribution, an agent observes Y and wants to convey to a potentially public user as much information about Y as possible while limiting the amount of information revealed about X. To this end, the so-called rate-privacy function is investigated to quantify the maximal amount of information (measured in terms of mutual information) that can be extracted from Y under a privacy constraint between X and the extracted information, where privacy is measured using either mutual information or maximal correlation. Properties of the rate-privacy function are analyzed and its information-theoretic and estimation-theoretic interpretations are presented for both the mutual information and maximal correlation privacy measures. It is also shown that the rate-privacy function admits a closed-form expression for a large family of joint distributions of $(X,Y)$. Finally, the rate-privacy function under the mutual information privacy measure is considered for the case where $(X,Y)$ has a joint probability density function by studying the problem where the extracted information is a uniform quantization of Y corrupted by additive Gaussian noise. The asymptotic behavior of the rate-privacy function is studied as the quantization resolution grows without bound and it is observed that not all of the properties of the rate-privacy function carry over from the discrete to the continuous case.

1. Introduction

With the emergence of user-customized services, there is an increasing desire to balance between the need to share data and the need to protect sensitive and private information. For example, individuals who join a social network are asked to provide information about themselves which might compromise their privacy. However, they agree to do so, to some extent, in order to benefit from the customized services such as recommendations and personalized searches. As another example, a participatory technology for estimating road traffic requires each individual to provide her start and destination points as well as the travel time. However, most participating individuals prefer to provide somewhat distorted or false information to protect their privacy. Furthermore, suppose a software company wants to gather statistical information on how people use its software. Since many users might have used the software to handle some personal or sensitive information -for example, a browser for anonymous web surfing or a financial management software- they may not want to share their data with the company. On the other hand, the company cannot legally collect the raw data either, so it needs to entice its users. In all these situations, a tradeoff in a conflict between utility advantage and privacy breach is required and the question is how to achieve this tradeoff. For example, how can a company collect high-quality aggregate information about users while strongly guaranteeing to its users that it is not storing user-specific information?

To deal with such privacy considerations, Warner [3] proposed the randomized response model in which each individual user randomizes her own data using a local randomizer (i.e., a noisy channel) before sharing the data to an untrusted data collector to be aggregated. As opposed to conditional security, see, e.g., [4,5,6], the randomized response model assumes that the adversary can have unlimited computational power and thus it provides unconditional privacy. This model, in which the control of private data remains in the users’ hands, has been extensively studied since Warner. As a special case of the randomized response model, Duchi et al. [7], inspired by the well-known privacy guarantee called differential privacy introduced by Dwork et al. [8,9,10], introduced locally differential privacy (LDP). Given a random variable $X\in \mathcal{X}$, another random variable $Z\in \mathcal{Z}$ is said to be the ε-LDP version of X if there exists a channel $Q:X\to Z$ such that $\frac{Q\left(B\right|x)}{Q\left(B\right|x^{\prime })}\le exp\left(\epsilon \right)$ for all measurable $B\subset \mathcal{Z}$ and all $x,x^{\prime }\in \mathcal{X}$. The channel Q is then called as the ε-LDP mechanism. Using Jensen’s inequality, it is straightforward to see that any ε-LDP mechanism leaks at most ε bits of private information, i.e., the mutual information between X and Z satisfies $I(X,Z)\le \epsilon$.

There have been numerous studies on the tradeoff between privacy and utility for different examples of randomized response models with different choices of utility and privacy measures. For instance, Duchi et al. [7] studied the optimal ε-LDP mechanism $\mathcal{M}:X\to Z$ which minimizes the risk of estimation of a parameter θ related to $P_X$. Kairouz et al. [11] studied an optimal ε-LDP mechanism in the sense of mutual information, where an individual would like to release an ε-LDP version Z of X that preserves as much information about X as possible. Calmon et al. [12] proposed a novel privacy measure (which includes maximal correlation and chi-square correlation) between X and Z and studied the optimal privacy mechanism (according to their privacy measure) which minimizes the error probability $Pr(\widehat{X}\left(Z\right)\ne X)$ for any estimator $\widehat{X}:Z\to X$.

In all above examples of randomized response models, given a private source, denoted by X, the mechanism generates Z which can be publicly displayed without breaching the desired privacy level. However, in a more realistic model of privacy, we can assume that for any given private data X, nature generates Y, via a fixed channel $P_{Y|X}$. Now we aim to release a public display Z of Y such that the amount of information in Y is preserved as much as possible while Z satisfies a privacy constraint with respect to X. Consider two communicating agents Alice and Bob. Alice collects all her measurements from an observation into a random variable Y and ultimately wants to reveal this information to Bob in order to receive a payoff. However, she is worried about her private data, represented by X, which is correlated with Y. For instance, X might represent her precise location and Y represents measurement of traffic load of a route she has taken. She wants to reveal these measurements to an online road monitoring system to received some utility. However, she does not want to reveal too much information about her exact location. In such situations, the utility is measured with respect to Y and privacy is measured with respect to X. The question raised in this situation then concerns the maximum payoff Alice can get from Bob (by revealing Z to him) without compromising her privacy. Hence, it is of interest to characterize such competing objectives in the form of a quantitative tradeoff. Such a characterization provides a controllable balance between utility and privacy.

This model of privacy first appears in Yamamoto’s work [13] in which the rate-distortion-equivocation function is defined as the tradeoff between a distortion-based utility and privacy. Recently, Sankar et al. [14], using the quantize-and-bin scheme [15], generalized Yamamoto’s model to study privacy in databases from an information-theoretic point of view. Calmon and Fawaz [16] and Monedero et al. [17] also independently used distortion and mutual information for utility and privacy, respectively, to define a privacy-distortion function which resembles the classical rate-distortion function. More recently, Makhdoumi et al. [18] proposed to use mutual information for both utility and privacy measures and defined the privacy funnel as the corresponding privacy-utility tradeoff, given by

$$t_R(X;Y):=\underset{\begin{array}{c}{P}_{Z|Y}:X-\circ -Y-\circ -Z\\ I(Y;Z)\ge R\end{array}}{min}I(X;Z)$$

(1)

where $X-\circ -Y-\circ -Z$ denotes that $X,Y$ and Z form a Markov chain in this order. Leveraging well-known algorithms for the information bottleneck problem [19], they provided a locally optimal greedy algorithm to evaluate $t_R(X;Y)$. Asoodeh et al. [1], independently, defined the rate-privacy function, $g_{\epsilon }(X;Y)$, as the maximum achievable $I(Y;Z)$ such that Z satisfies $I(X;Z)\le \epsilon$, which is a dual representation of the privacy funnel (1), and showed that for discrete X and Y, $g_0(X;Y)>0$ if and only if X is weakly independent of Y (cf, Definition 9). Recently, Calmon et al. [20] proved an equivalent result for $t_R(X;Y)$ using a different approach. They also obtained lower and upper bounds for $t_R(X;Y)$ which can be easily translated to bounds for $g_{\epsilon }(X;Y)$ (cf. Lemma 1). In this paper, we develop further properties of $g_{\epsilon }(X;Y)$ and also determine necessary and sufficient conditions on $P_{XY}$, satisfying some symmetry conditions, for $g_{\epsilon }(X;Y)$ to achieve its upper and lower bounds.

The problem treated in this paper can also be contrasted with the better-studied concept of secrecy following the pioneering work of Wyner [21]. While in secrecy problems the aim is to keep information secret only from wiretappers, in privacy problems the aim is to keep the private information (not necessarily all the information) secret from everyone including the intended receiver.

1.1. Our Model and Main Contributions

Using mutual information as measure of both utility and privacy, we formulate the corresponding privacy-utility tradeoff for discrete random variables X and Y via the rate-privacy function, $g_{\epsilon }(X;Y)$, in which the mutual information between Y and displayed data (i.e., the mechanism’s output), Z, is maximized over all channels $P_{Z|Y}$ such that the mutual information between Z and X is no larger than a given ε. We also formulate a similar rate-privacy function ${\widehat{g}}_{\epsilon }(X;Y)$ where the privacy is measured in terms of the squared maximal correlation, ${\rho }_m^2$, between, X and Z. In studying $g_{\epsilon }(X;Y)$ and ${\widehat{g}}_{\epsilon }(X;Y)$, any channel $Q:Y\to Z$ that satisfies $I(X;Z)\le \epsilon$ and ${\rho }_m^2(X;Z)\le \epsilon$, preserves the desired level of privacy and is hence called a privacy filter. Interpreting $I(Y;Z)$ as the number of bits that a privacy filter can reveal about Y without compromising privacy, we present the rate-privacy function as a formulation of the problem of maximal privacy-constrained information extraction from Y.

We remark that using maximal correlation as a privacy measure is by no means new as it appears in other works, see, e.g., [22,23] and [12] for different utility functions. We do not put any likelihood constraints on the privacy filters as opposed to the definition of LDP. In fact, the optimal privacy filters that we obtain in this work induce channels $P_{Z|X}$ that do not satisfy the LDP property.

The quantity $g_{\epsilon }(X;Y)$ is related to a notion of the reverse strong data processing inequality as follows. Given a joint distribution $P_{XY}$, the strong data processing coefficient was introduced in [24,25], as the smallest $s(X;Y)\le 1$ such that $I(X;Z)\le s(X;Y)I(Y;Z)$ for all $P_{Z|Y}$ satisfying the Markov condition $X--\circ -Y-\circ -Z$. In the rate-privacy function, we instead seek an upper bound on the maximum achievable rate at which Y can display information, $I(Y;Z)$, while meeting the privacy constraint $I(X;Z)\le \epsilon$. The connection between the rate-privacy function and the strong data processing inequality is further studied in [20] to mirror all the results of [25] in the context of privacy.

The contributions of this work are as follows:

• We study lower and upper bounds of $g_{\epsilon }(X;Y)$. The lower bound, in particular, establishes a multiplicative bound on $I(Y;Z)$ for any optimal privacy filter. Specifically, we show that for a given $(X,Y)$ and $\epsilon >0$ there exists a channel $Q:Y\to Z$ such that $I(X;Z)\le \epsilon$ and

  $$I(Y;Z)\ge \lambda (X;Y)\epsilon$$

  (2)

  where $\lambda (X;Y)\ge 1$ is a constant depending on the joint distribution $P_{XY}$. We then give conditions on $P_{XY}$ such that the upper and lower bounds are tight. For example, we show that the lower bound is achieved when Y is binary and the channel from Y to X is symmetric. We show that this corresponds to the fact that both $Y=0$ and $Y=1$ induce distributions $P_{X|Y}(·|0)$ and $P_{X|Y}(·|1)$ which are equidistant from $P_X$ in the sense of Kullback-Leibler divergence. We then show that the upper bound is achieved when Y is an erased version of X, or equivalently, $P_{Y|X}$ is an erasure channel.
• We propose an information-theoretic setting in which $g_{\epsilon }(X;Y)$ appears as a natural upper-bound for the achievable rate in the so-called "dependence dilution" coding problem. Specifically, we examine the joint-encoder version of an amplification-masking tradeoff, a setting recently introduced by Courtade [26] and we show that the dual of $g_{\epsilon }(X;Y)$ upper bounds the masking rate. We also present an estimation-theoretic motivation for the privacy measure ${\rho }_m^2(X;Z)\le \epsilon$. In fact, by imposing ${\rho }_m^2(X;Y)\le \epsilon$, we require that an adversary who observes Z cannot efficiently estimate $f\left(X\right)$, for any function f. This is reminiscent of semantic security [27] in the cryptography community. An encryption mechanism is said to be semantically secure if the adversary’s advantage for correctly guessing any function of the privata data given an observation of the mechanism’s output (i.e., the ciphertext) is required to be negligible. This, in fact, justifies the use of maximal correlation as a measure of privacy. The use of mutual information as privacy measure can also be justified using Fano’s inequality. Note that $I(X;Z)\le \epsilon$ can be shown to imply that $Pr(\widehat{X}\left(Z\right)\ne X)\ge \frac{H\left(X\right)-1-\epsilon }{log\left(\right|\mathcal{X}\left|\right)}$ and hence the probability of adversary correctly guessing X is lower-bounded.
• We also study the rate of increase $g_0^{\prime }(X;Y)$ of $g_{\epsilon }(X;Y)$ at $\epsilon =0$ and show that this rate can characterize the behavior of $g_{\epsilon }(X;Y)$ for any $\epsilon \ge 0$ provided that $g_0(X;Y)=0$. This again has connections with the results of [25]. Letting

  $$\Gamma \left(R\right):=\underset{\genfrac{}{}{0pt}{}{P_{Z|Y}:X-\circ -Y-\circ -Z}{I(Y;Z)\le R}}{max}I(X;Z)$$

  one can easily show that ${\Gamma }^{\prime }\left(0\right)={lim}_{R\to 0}\frac{\Gamma \left(R\right)}{R}=s(X;Y),$ and hence the rate of increase of $\Gamma \left(R\right)$ at $R=0$ characterizes the strong data processing coefficient. Note that here we have $\Gamma \left(0\right)=0$.
• Finally, we generalize the rate-privacy function to the continuous case where X and Y are both continuous and show that some of the properties of $g_{\epsilon }(X;Y)$ in the discrete case do not carry over to the continuous case. In particular, we assume that the privacy filter belongs to a family of additive noise channels followed by an M-level uniform scalar quantizer and give asymptotic bounds as $M\to \infty$ for the rate-privacy function.

1.2. Organization

The rest of the paper is organized as follows. In Section 2, we define and study the rate-privacy function for discrete random variables for two different privacy measures, which, respectively, lead to the information-theoretic and estimation-theoretic interpretations of the rate-privacy function. In Section 3, we provide such interpretations for the rate-privacy function in terms of quantities from information and estimation theory. Having obtained lower and upper bounds of the rate-privacy function, in Section 4 we determine the conditions on $P_{XY}$ such that these bounds are tight. The rate-privacy function is then generalized and studied in Section 5 for continuous random variables.

2. Utility-Privacy Measures: Definitions and Properties

Consider two random variables X and Y, defined over finite alphabets $\mathcal{X}$ and $\mathcal{Y}$, respectively, with a fixed joint distribution $P_{XY}$. Let X represent the private data and let Y be the observable data, correlated with X and generated by the channel $P_{Y|X}$ predefined by nature, which we call the observation channel. Suppose there exists a channel $P_{Z|Y}$ such that Z, the displayed data made available to public users, has limited dependence with X. Such a channel is called the privacy filter. This setup is shown in Figure 1. The objective is then to find a privacy filter which gives rise to the highest dependence between Y and Z. To make this goal precise, one needs to specify a measure for both utility (dependence between Y and Z) and also privacy (dependence between X and Z).

2.1. Mutual Information as Privacy Measure

Adopting mutual information as a measure of both privacy and utility, we are interested in characterizing the following quantity, which we call the rate-privacy function (since mutual information is adopted for utility, the privacy-utility tradeoff characterizes the optimal rate for a given privacy level, where rate indicates the precision of the displayed data Z with respect to the observable data Y for a privacy filter, which suggests the name),

$$g_{\epsilon }(X;Y):=\underset{P_{Z|Y}\in {\mathcal{D}}_{\epsilon }\left(P\right)}{sup}I(Y;Z)$$

(3)

where $(X,Y)$ has fixed distribution $P_{XY}=P$ and

$${\mathcal{D}}_{\epsilon }\left(P\right):=\{P_{Z|Y}:X-\circ -Y-\circ -Z,I(X;Z)\le \epsilon \}$$

(here $X-\circ -Y-\circ -Z$ means that $X,Y,$ and Z form a Markov chain in this order). Equivalently, we call $g_{\epsilon }(X;Y)$ the privacy-constrained information extraction function, as Z can be thought of as the extracted information from Y under privacy constraint $I(X;Z)\le \epsilon$.

Note that since $I(Y;Z)$ is a convex function of $P_{Z|Y}$ and furthermore the constraint set ${\mathcal{D}}_{\epsilon }\left(P\right)$ is convex, [28, Theorem 32.2] implies that we can restrict ${\mathcal{D}}_{\epsilon }\left(P\right)$ in (3) to $\{P_{Z|Y}:X-\circ -Y-\circ -Z,I(X;Z)=\epsilon \}$ whenever $\epsilon \le I(X;Y)$ . Note also that since for finite $\mathcal{X}$ and $\mathcal{Y}$, $P_{Z|Y}\to I(Y;Z)$ is a continuous map, therefore ${\mathcal{D}}_{\epsilon }\left(P\right)$ is compact and the supremum in (3) is indeed a maximum. In this case, using the Support Lemma [29], one can readily show that it suffices that the random variable Z is supported on an alphabet $\mathcal{Z}$ with cardinality $\left|\mathcal{Z}\right|\le \left|\mathcal{Y}\right|+1$. Note further that by the Markov condition $X-\circ -Y-\circ -Z$, we can always restrict $\epsilon \ge 0$ to only $0\le \epsilon <I(X;Y)$, because $I(X;Z)\le I(X;Y)$ and hence for $\epsilon \ge I(X;Y)$ the privacy constraint is removed and thus by setting $Z=Y$, we obtain $g_{\epsilon }(X;Y)=H\left(Y\right)$.

As mentioned earlier, a dual representation of $g_{\epsilon }(X;Y)$, the so called privacy funnel, is introduced in [18,20], defined in (1), as the least information leakage about X such that the communication rate is greater than a positive constant; $I(Y;Z)\ge R$ for some $R>0$. Note that if $t_R(X;Y)=\epsilon$ then $g_{\epsilon }(X;Y)=R$.

Given ${\epsilon }_1<{\epsilon }_2$ and a joint distribution $P=P_X\times P_{Y|X}$, we have ${\mathcal{D}}_{{\epsilon }_1}\left(P\right)\subset {\mathcal{D}}_{{\epsilon }_2}\left(P\right)$ and hence $\epsilon \to g_{\epsilon }(X;Y)$ is non-decreasing, i.e., $g_{{\epsilon }_1}(X;Y)\le g_{{\epsilon }_2}(X;Y)$. Using a similar technique as in [30, Lemma 1], Calmon et al. [20] showed that the mapping $R\mapsto \frac{t_R(X;Y)}{R}$ is non-decreasing for $R>0$. This, in fact, implies that $\epsilon \mapsto \frac{g_{\epsilon }(X;Y)}{\epsilon }$ is non-increasing for $\epsilon >0$. This observation leads to a lower bound for the rate privacy function $g_{\epsilon }(X;Y)$ as described in the following lemma.

Lemma 1 

([20]). For a given joint distribution P defined over $\mathcal{X}\times \mathcal{Y}$, the mapping $\epsilon \mapsto \frac{g_{\epsilon }(X;Y)}{\epsilon }$ is non-increasing on $\epsilon \in (0,\infty )$ and $g_{\epsilon }(X;Y)$ lies between two straight lines as follows:

$$\epsilon \frac{H\left(Y\right)}{I(X;Y)}\le g_{\epsilon }(X;Y)\le H\left(Y\right|X)+\epsilon$$

(4)

for $\epsilon \in (0,I(X;Y\left)\right)$.

Using a simple calculation, the lower bound in (4) can be shown to be achieved by the privacy filter depicted in Figure 2 with the erasure probability

$$\delta =1-\frac{\epsilon }{I(X;Y)}$$

(5)

In light of Lemma 1, the possible range of the map $\epsilon \mapsto g_{\epsilon }(X;Y)$ is as depicted in Figure 3.

We next show that $\epsilon \mapsto g_{\epsilon }(X;Y)$ is concave and continuous.

Lemma 2. 

For any given pair of random variables $(X,Y)$ over $\mathcal{X}\times \mathcal{Y}$, the mapping $\epsilon \mapsto g_{\epsilon }(X;Y)$ is concave for $\epsilon \ge 0$.

Proof. 

It suffices to show that for any $0\le {\epsilon }_1<{\epsilon }_2<{\epsilon }_3\le I(X;Y)$, we have

$$\frac{g_{{\epsilon }_3}(X;Y)-g_{{\epsilon }_1}(X;Y)}{{\epsilon }_3-{\epsilon }_1}\le \frac{g_{{\epsilon }_2}(X;Y)-g_{{\epsilon }_1}(X;Y)}{{\epsilon }_2-{\epsilon }_1}$$

(6)

which, in turn, is equivalent to

$$\left(\frac{{\epsilon }_2-{\epsilon }_1}{{\epsilon }_3-{\epsilon }_1}\right)g_{{\epsilon }_3}(X;Y)+\left(\frac{{\epsilon }_3-{\epsilon }_2}{{\epsilon }_3-{\epsilon }_1}\right)g_{{\epsilon }_1}(X;Y)\le g_{{\epsilon }_2}(X;Y)$$

(7)

Let $P_{Z_1|Y}:Y\to Z_1$ and $P_{Z_3|Y}:Y\to Z_3$ be two optimal privacy filters in ${\mathcal{D}}_{{\epsilon }_1}\left(P\right)$ and ${\mathcal{D}}_{{\epsilon }_3}\left(P\right)$ with disjoint output alphabets ${\mathcal{Z}}_1$ and ${\mathcal{Z}}_3$, respectively.

We introduce an auxiliary binary random variable $U\sim \mathsf{Bernoulli}\left(\lambda \right)$, independent of $(X,Y)$, where $\lambda :=\frac{{\epsilon }_2-{\epsilon }_1}{{\epsilon }_3-{\epsilon }_1}$ and define the following random privacy filter $P_{Z_{\lambda }|Y}$: We pick $P_{Z_3|Y}$ if $U=1$ and $P_{Z_1|Y}$ if $U=0$, and let $Z_{\lambda }$ be the output of this random channel which takes values in ${\mathcal{Z}}_1\cup {\mathcal{Z}}_3$. Note that $(X,Y)-\circ -Z-\circ -U$. Then we have

$$\begin{array}{ccc}\hfill I(X;Z_{\lambda })& =& I(X;Z_{\lambda },U)=I(X;Z_{\lambda }|U)=\lambda I(X;Z_3)+(1-\lambda )I(X;Z_1),\hfill \\ & \le & {\epsilon }_2\hfill \end{array}$$

which implies that $P_{Z_{\lambda }|Y}\in {\mathcal{D}}_{{\epsilon }_2}\left(P\right)$. On the other hand, we have

$$\begin{array}{ccc}\hfill g_{{\epsilon }_2}(X;Y)\ge I(Y;Z_{\lambda })& =& I(Y;Z_{\lambda },U)=I(Y;Z_{\lambda }|U)=\lambda I(Y;Z_3)+(1-\lambda )I(Y;Z_1)\hfill \\ & =& \left(\frac{{\epsilon }_2-{\epsilon }_1}{{\epsilon }_3-{\epsilon }_1}\right)g_{{\epsilon }_3}(X;Y)+\left(\frac{{\epsilon }_3-{\epsilon }_2}{{\epsilon }_3-{\epsilon }_1}\right)g_{{\epsilon }_1}(X;Y)\hfill \end{array}$$

which, according to (7), completes the proof. ☐

Remark 1. 

By the concavity of $\epsilon \mapsto g_{\epsilon }(X;Y)$, we can show that $g_{\epsilon }(X;Y)$ is a strictly increasing function of $\epsilon \le I(X;Y)$. To see this, assume there exists ${\epsilon }_1<{\epsilon }_2\le I(X;Y)$ such that $g_{{\epsilon }_1}(X;Y)=g_{{\epsilon }_2}(X;Y)$. Since $\epsilon \mapsto g_{\epsilon }(X;Y)$ is concave, then it follows that for all $\epsilon \ge {\epsilon }_2$, $g_{\epsilon }(X;Y)=g_{{\epsilon }_2}(X;Y)$ and since for $\epsilon =I(X;Y)$, $g_{I(X;Y)}(X;Y)=H\left(Y\right)$, implying that for any $\epsilon \ge {\epsilon }_2$, we must have $g_{\epsilon }(X;Y)=H\left(Y\right)$ which contradicts the upper bound shown in (4).

Corollary 3. 

For any given pair of random variables $(X,Y)$ over $\mathcal{X}\times \mathcal{Y}$, the mapping $\epsilon \mapsto g_{\epsilon }(X;Y)$ is continuous for $\epsilon \ge 0$.

Proof. 

Concavity directly implies that the mapping $\epsilon \mapsto g_{\epsilon }(X;Y)$ is continuous on $(0,\infty )$ (see for example [31, Theorem 3.2]). Continuity at zero follows from the continuity of mutual information. ☐

Remark 2. 

Using the concavity of the map $\epsilon \mapsto g_{\epsilon }(X;Y)$, we can provide an alternative proof for the lower bound in (4). Note that point $\left(I\right(X;Y),H(Y\left)\right)$ is always on the curve $g_{\epsilon }(X;Y)$, and hence by concavity, the straight line $\epsilon \mapsto \epsilon \frac{H\left(Y\right)}{I(X;Y)}$ is always below the lower convex envelop of $g_{\epsilon }(X;Y)$, i.e., the chord connecting $(0,g_0(X;Y))$ to $\left(I\right(X;Y),H(Y\left)\right)$, and hence $g_{\epsilon }(X;Y)\ge \epsilon \frac{H\left(Y\right)}{I(X;Y)}$. In fact, this chord yields a better lower bound for $g_{\epsilon }(X;Y)$ on $\epsilon \in [0,I(X;Y]$ as

$$g_{\epsilon }(X;Y)\ge \epsilon \frac{H\left(Y\right)}{I(X;Y)}+g_0(X;Y)\left[1-\frac{\epsilon }{I(X;Y)}\right]$$

(8)

which reduces to the lower bound in (4) only if $g_0(X;Y)=0$.

2.2. Maximal Correlation as Privacy Measure

By adopting the mutual information as the privacy measure between the private and the displayed data, we make sure that only limited bits of private information is revealed during the process of transferring Y. In order to have an estimation theoretic guarantee of privacy, we propose alternatively to measure privacy using a measure of correlation, the so-called maximal correlation.

Given the collection $\mathcal{C}$of all pairs of random variables $(U,V)\in \mathcal{U}\times \mathcal{V}$ where $\mathcal{U}$ and $\mathcal{V}$ are general alphabets, a mapping $T:\mathcal{C}\to [0,1]$ defines a measure of correlation [32] if $T(U,V)=0$ if and only if U and V are independent (in short, $U\perp \perp V$) and $T(U,V)$ attains its maximum value if $X=f\left(Y\right)$ or $Y=g\left(X\right)$ almost surely for some measurable real-valued functions f and g. There are many different examples of measures of correlation including the Hirschfeld-Gebelein-Rényi maximal correlation [32,33,34], the information measure [35], mutual information and f-divergence [36].

Definition 4 

([34]). Given random variables X and Y, the maximal correlation ${\rho }_m(X;Y)$ is defined as follows (recall that the correlation coefficient between U and V, is defined as $\rho (U;V):=\frac{\text{cov}(U;V)}{{\sigma }_U{\sigma }_V}$, where $\text{cov}(U;V),{\sigma }_U$ and ${\sigma }_V$ are the covariance between U and V, the standard deviations of U and V, respectively):

$${\rho }_m(X;Y):=\underset{f,g}{sup}\rho (f\left(X\right),g\left(Y\right))=\underset{\left(f\right(X),g(Y\left)\right)\in \mathcal{S}}{sup}\mathbb{E}\left[f\left(X\right)g\left(Y\right)\right]$$

where $\mathcal{S}$ is the collection of pairs of real-valued random variables $f\left(X\right)$ and $g\left(Y\right)$ such that $\mathbb{E}f\left(X\right)=\mathbb{E}g\left(Y\right)=0$ and $\mathbb{E}{f}^2\left(X\right)=\mathbb{E}{g}^2\left(Y\right)=1$. If $\mathcal{S}$ is empty (which happens precisely when at least one of X and Y is constant almost surely) then one defines ${\rho }_m(X;Y)$ to be 0. Rényi [34] derived an equivalent characterization of maximal correlation as follows:

$${\rho }_m^2(X;Y)=\underset{f:\mathbb{E}f\left(X\right)=0,\mathbb{E}{f}^2\left(X\right)=1}{sup}\mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Y]\right].$$

(9)

Measuring privacy in terms of maximal correlation, we propose

$${\widehat{g}}_{\epsilon }(X;Y):=\underset{P_{Z|Y}\in {\widehat{\mathcal{D}}}_{\epsilon }\left(P\right)}{sup}I(Y;Z)$$

as the corresponding rate-privacy tradeoff, where

$${\widehat{\mathcal{D}}}_{\epsilon }\left(P\right):=\{P_{Z|Y}:X-\circ -Y-\circ -Z,{\rho }_m^2(X;Z)\le \epsilon ,P_{XY}=P\}$$

Again, we equivalently call ${\widehat{g}}_{\epsilon }(X;Y)$ as the privacy-constrained information extraction function, where here the privacy is guaranteed by ${\rho }_m^2(X;Z)\le \epsilon$.

Setting $\epsilon =0$ corresponds to the case where X and Z are required to be statistically independent, i.e., absolutely no information leakage about the private source X is allowed. This is called perfect privacy. Since the independence of X and Z is equivalent to $I(X;Z)={\rho }_m(X;Z)=0$, we have ${\widehat{g}}_0(X;Y)=g_0(X;Y)$. However, for $\epsilon >0$, both $g_{\epsilon }(X;Y)\le {\widehat{g}}_{\epsilon }(X;Y)$ and $g_{\epsilon }(X;Y)\ge {\widehat{g}}_{\epsilon }(X;Y)$ might happen in general. For general $\epsilon \ge 0$, it directly follows using [23, Proposition 1] that

$${\widehat{g}}_{\epsilon }(X;Y)\le g_{{\epsilon }^{\prime }}(X;Y)$$

where ${\epsilon }^{\prime }:=log(k\epsilon +1)$ and $k:=\left|\mathcal{X}\right|-1$

Similar to $g_{\epsilon }(X;Y)$, we see that for ${\epsilon }_1\le {\epsilon }_2$, ${\widehat{\mathcal{D}}}_{{\epsilon }_1}\left(P\right)\subset {\widehat{\mathcal{D}}}_{{\epsilon }_2}\left(P\right)$ and hence $\epsilon \to {\widehat{g}}_{\epsilon }(X;Y)$ is non-decreasing. The following lemma is a counterpart of Lemma 1 for ${\widehat{g}}_{\epsilon }(X;Y)$.

Lemma 5. 

For a given joint distribution $P_{XY}$ defined over $\mathcal{X}\times \mathcal{Y}$, $\epsilon \mapsto \frac{{\widehat{g}}_{\epsilon }(X;Y)}{\epsilon }$ is non-increasing on $(0,\infty )$.

Proof. 

Like Lemma 1, the proof is similar to the proof of [30, Lemma 1]. We, however, give a brief proof for the sake of completeness.

For a given channel $P_{Z|Y}\in {\widehat{\mathcal{D}}}_{\epsilon }\left(P\right)$ and $\delta \ge 0$, we can define a new channel with an additional symbol e as follows

$$P_{Z^{\prime }|Y}\left(z^{\prime }\right|y)=\left\{\begin{array}{cc}(1-\delta )P_{Z|Y}\left(z^{\prime }\right|y)\hfill & \text{if}{z}^{\prime }\ne e\hfill \\ \delta \hfill & \text{if}{z}^{\prime }=e\hfill \end{array}\right.$$

(10)

It is easy to check that $I(Y;Z^{\prime })=(1-\delta )I(Y;Z)$ and also ${\rho }_m^2(X;Z^{\prime })=(1-\delta ){\rho }_m^2(X;Z)$; see [37, Page 8], which implies that $P_{Z^{\prime }|Y}\in {\widehat{\mathcal{D}}}_{{\epsilon }^{\prime }}\left(P\right)$ where ${\epsilon }^{\prime }=(1-\delta )\epsilon$. Now suppose that $P_{Z|Y}$ achieves ${\widehat{g}}_{\epsilon }(X;Y)$, that is, ${\widehat{g}}_{\epsilon }(X;Y)=I(Y;Z)$ and ${\rho }_m^2(X;Z)=\epsilon$. We can then write

$$\frac{{\widehat{g}}_{\epsilon }(X;Y)}{\epsilon }=\frac{I(Y;Z)}{\epsilon }=\frac{I(Y;Z^{\prime })}{{\epsilon }^{\prime }}\le \frac{g_{{\epsilon }^{\prime }}(X;Y)}{{\epsilon }^{\prime }}$$

Therefore, for ${\epsilon }^{\prime }\le \epsilon$ we have $\frac{g_{{\epsilon }^{\prime }}(X;Y)}{{\epsilon }^{\prime }}\ge \frac{g_{\epsilon }(X;Y)}{\epsilon }$. ☐

Similar to the lower bound for $g_{\epsilon }(X;Y)$ obtained from Lemma 1, we can obtain a lower bound for ${\widehat{g}}_{\epsilon }(X;Y)$ using Lemma 5. Before we get to the lower bound, we need a data processing lemma for maximal correlation. The following lemma proves a version of strong data processing inequality for maximal correlation from which the typical data processing inequality follows, namely, ${\rho }_m(X;Z)\le min\{{\rho }_m(Y;Z),{\rho }_m(X;Y)\}$ for $X,Y$ and Z satisfying $X-\circ -Y-\circ -Z$.

Lemma 6. 

For random variables X and Y with a joint distribution $P_{XY}$, we have

$$\underset{\begin{array}{c}X-\circ -Y-\circ -Z\\ {\rho }_m(Y;Z)\ne 0\end{array}}{sup}\frac{{\rho }_m(X;Z)}{{\rho }_m(Y;Z)}={\rho }_m(X;Y)$$

Proof. 

For arbitrary zero-mean and unit variance measurable functions $f\in {\mathcal{L}}^2\left(\mathcal{X}\right)$ and $g\in {\mathcal{L}}^2\left(\mathcal{Z}\right)$ and $X-\circ -Y-\circ -Z$, we have

$$\mathbb{E}\left[f\left(X\right)g\left(Z\right)\right]=\mathbb{E}\left[\mathbb{E}\left[f\right(X\left)\right|Y\left]\mathbb{E}\right[g\left(Z\right)\left|Y\right]\right]\le {\rho }_m(X;Y){\rho }_m(Y;Z)$$

where the inequality follows from the Cauchy-Schwartz inequality and (9). Thus we obtain ${\rho }_m(X;Z)\le {\rho }_m(X;Y){\rho }_m(Y;Z)$.

This bound is tight for the special case of $X\to Y\to X^{\prime }$, where $P_{X^{\prime }|Y}$ is the backward channel associated with $P_{Y|X}$. In the following, we shall show that ${\rho }_m(X;Y){\rho }_m(Y;X^{\prime })={\rho }_m(X;X^{\prime })$.

To this end, first note that the above implies that ${\rho }_m(X;Y){\rho }_m(Y;X^{\prime })\ge {\rho }_m(X;X^{\prime })$. Since $P_{XY}=P_{X^{\prime }Y}$, it follows that ${\rho }_m(X;Y)={\rho }_m(Y;X^{\prime })$ and hence the above implies that ${\rho }_m^2(X;Y)\ge {\rho }_m(X;X^{\prime })$. One the other hand, we have

$$\mathbb{E}\left[{\left[\mathbb{E}\left[f\left(X\right)\right|Y]\right]}^2\right]=\mathbb{E}\left[\mathbb{E}\left[f\left(X\right)\right|Y]\mathbb{E}\left[f\left(X^{\prime }\right)\right|Y]\right]=\mathbb{E}\left[\mathbb{E}\left[f\left(X\right)f\left(X^{\prime }\right)\right|Y]\right]=\mathbb{E}\left[f\left(X\right)f\left(X^{\prime }\right)\right]$$

which together with (9) implies that

$${\rho }_m^2(X;Y)\le \underset{f:\mathbb{E}f\left(X\right)=0,\mathbb{E}{f}^2\left(X\right)=1}{sup}\mathbb{E}\left[f\left(X\right)f\left(X^{\prime }\right)\right]\le {\rho }_m(X;X^{\prime })$$

Thus, ${\rho }_m^2(X;Y)={\rho }_m(X;X^{\prime })$ which completes the proof. ☐

Now a lower bound of ${\widehat{g}}_{\epsilon }(X;Y)$ can be readily obtained.

Corollary 7. 

For a given joint distribution $P_{XY}$ defined over $\mathcal{X}\times \mathcal{Y}$, we have for any $\epsilon >0$

$${\widehat{g}}_{\epsilon }(X;Y)\ge \frac{H\left(Y\right)}{{\rho }_m^2(X;Y)}min\{\epsilon ,{\rho }_m^2(X;Y)\}$$

Proof. 

By Lemma 6, we know that for any Markov chain $X-\circ -Y-\circ -Z$, we have ${\rho }_m(X;Z)\le {\rho }_m(X;Y)$ and hence for $\epsilon \ge {\rho }_m^2(X;Y)$, the privacy constraint ${\rho }_m^2(X;Z)\le \epsilon$ is not restrictive and hence ${\widehat{g}}_{\epsilon }(X;Y)=H\left(Y\right)$ by setting $Y=Z$. For $0<\epsilon \le {\rho }_m^2(X;Y)$, Lemma 5 implies that

$$\frac{{\widehat{g}}_{\epsilon }(X;Y)}{\epsilon }\ge \frac{H\left(Y\right)}{{\rho }_m^2(X;Y)}$$

from which the result follows. ☐

A loose upper bound of ${\widehat{g}}_{\epsilon }(X;Y)$ can be obtained using an argument similar to the one used for $g_{\epsilon }(X;Y)$. For the Markov chain $X-\circ -Y-\circ -Z$, we have

$$\begin{array}{ccc}\hfill I(Y;Z)& =& I(X;Z)+I(Y;Z|X)\le I(X;Z)+H(Y\left|X\right)\hfill \\ & \stackrel{\left(a\right)}{\le }& log\left(k{\rho }_m^2(X;Z)+1\right)+H\left(Y\right|X)\hfill \end{array}$$

(11)

where $k:=\left|\mathcal{X}\right|-1$ and $\left(a\right)$ comes from [23, Proposition 1]. We can, therefore, conclude from (11) and Corollary 7 that

$$\epsilon \frac{H\left(Y\right)}{{\rho }_m^2(X;Y)}\le {\widehat{g}}_{\epsilon }(X;Y)\le log\left(k\epsilon +1\right)+H\left(Y\right|X)$$

(12)

Similar to Lemma 2, the following lemma shows that the ${\widehat{g}}_{\epsilon }(X;Y)$ is a concave function of ε.

Lemma 8. 

For any given pair of random variables $(X,Y)$ with distribution P over $\mathcal{X}\times \mathcal{Y}$, the mapping $\epsilon \mapsto {\widehat{g}}_{\epsilon }(X;Y)$ is concave for $\epsilon \ge 0$.

Proof. 

The proof is similar to that of Lemma 2 except that here for two optimal filters $P_{Z_1|Y}:Y\to Z_1$ and $P_{Z_3|Y}:Y\to Z_3$ in ${\widehat{\mathcal{D}}}_{{\epsilon }_1}\left(P\right)$ and ${\widehat{\mathcal{D}}}_{{\epsilon }_3}\left(P\right)$, respectively, and the random channel $P_{Z_{\lambda }|Y}:Y\to Z$ with output alphabet ${\mathcal{Z}}_1\cup {\mathcal{Z}}_3$ constructed using a coin flip with probability γ, we need to show that $P_{Z_{\lambda }|Y}\in {\widehat{\mathcal{D}}}_{{\epsilon }_2}\left(P\right)$, where $0\le {\epsilon }_1<{\epsilon }_2<{\epsilon }_3\le {\rho }_m^2(X;Y)$. To show this, consider $f:\mathcal{X}\to \mathbb{R}$ such that $\mathbb{E}\left[f\right(X\left)\right]=0$ and $\mathbb{E}\left[f^2\left(X\right)\right]=1$ and let U be a binary random variable as in the proof of Lemma 2. We then have

$$\begin{array}{ccc}\hfill \mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_{\lambda }]\right]& =& \mathbb{E}\left[\mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_{\lambda }]\right|U]\right]\hfill \\ & \stackrel{\left(a\right)}{=}& \gamma \mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_3]\right]+(1-\gamma )\mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_1]\right]\hfill \end{array}$$

(13)

where $\left(a\right)$ comes from the fact that U is independent of X. We can then conclude from (13) and the alternative characterization of maximal correlation (9) that

$$\begin{array}{ccc}\hfill {\rho }_m^2(X;Z_{\lambda })& =& \underset{f:\mathbb{E}\left[f\left(X\right)\right]=0,\mathbb{E}\left[f^2\left(X\right)\right]=1}{sup}\mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_{\lambda }]\right]\hfill \\ & =& \underset{f:\mathbb{E}\left[f\left(X\right)\right]=0,\mathbb{E}\left[f^2\left(X\right)\right]=1}{sup}\left[\gamma \mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_3]\right]+(1-\gamma )\mathbb{E}\left[{\mathbb{E}}^2\left[f\left(X\right)\right|Z_1]\right]\right]\hfill \\ & \le & \gamma {\rho }_m^2(X;Z_3)+(1-\gamma ){\rho }_m^2(X;Z_1)\le \gamma {\epsilon }_3+(1-\gamma ){\epsilon }_1\hfill \end{array}$$

from which we can conclude that $P_{Z_{\lambda }|Y}\in {\widehat{\mathcal{D}}}_{{\epsilon }_2}\left(P\right)$. ☐

2.3. Non-Trivial Filters For Perfect Privacy

As it becomes clear later, requiring that $g_0(X;Y)=0$ is a useful assumption for the analysis of $g_{\epsilon }(X;Y)$. Thus, it is interesting to find a necessary and sufficient condition on the joint distribution $P_{XY}$ which results in $g_0(X;Y)=0$.

Definition 9 

([38]). The random variable X is said to be weakly independent of Y if the rows of the transition matrix $P_{X|Y}$, i.e., the set of vectors $\{P_{X|Y}(·|y),y\in \mathcal{Y}\}$, are linearly dependent.

The following lemma provides a necessary and sufficient condition for $g_0(X;Y)>0$.

Lemma 10. 

For a given $(X,Y)$ with a given joint distribution $P_{XY}=P_Y\times P_{X|Y}$, $g_0(X;Y)>0$ (and equivalently ${\widehat{g}}_0(X;Y)>0$) if and only if X is weakly independent of Y.

Proof. 

⇒ direction:

Assuming that $g_0(X;Y)>0$ implies that there exists a random variable Z over an alphabet $\mathcal{Z}$ such that the Markov condition $X-\circ -Y-\circ -Z$ is satisfied and $Z\perp \perp X$ while $I(Y;Z)>0$. Hence, for any $z_1$ and $z_2$ in $\mathcal{Z}$, we must have $P_{X|Z}\left(x\right|z_1)=P_{X|Z}\left(x\right|z_2)$ for all $x\in \mathcal{X}$, which implies that

$$\sum _{y\in \mathcal{Y}}{P}_{X|Y}\left(x\right|y)P_{Y|Z}\left(y\right|z_1)=\sum _{y\in \mathcal{Y}}{P}_{X|Y}\left(x\right|y)P_{Y|Z}\left(y\right|z_2)$$

and hence

$$\sum _{y\in \mathcal{Y}}{P}_{X|Y}\left(x\right|y)\left[P_{Y|Z}\left(y\right|z_1)-P_{Y|Z}\left(y\right|z_2)\right]=0$$

Since Y is not independent of Z, there exist $z_1$ and $z_2$ such that $P_{Y|Z}\left(y\right|z_1)\ne P_{Y|Z}\left(y\right|z_2)$ and hence the above shows that the set of vectors $P_{X|Y}(·|y)$, $y\in \mathcal{Y}$ is linearly dependent.

⇐ direction:

Berger and Yeung [38, Appendix II], in a completely different context, showed that if X being weakly independent of Y, one can always construct a binary random variable Z correlated with Y which satisfies $X-\circ -Y-\circ -Z$ and $X\perp \perp Z$, and hence $g_0(X;Y)>0$. ☐

Remark 3. 

Lemma 10 first appeared in [1]. However, Calmon et al. [20] studied (1), the dual version of $g_{\epsilon }(X;Y)$, and showed an equivalent result for $t_R(X;Y)$. In fact, they showed that for a given $P_{XY}$, one can always generate Z such that $I(X;Z)=0$, $I(Y;Z)>0$ and $X-\circ -Y-\circ -Z$, or equivalently $g_0(X;Y)>0$, if and only if the smallest singular value of the conditional expectation operator $f\mapsto \mathbb{E}\left[f\right(X\left)\right|Y]$ is zero. This condition can, in fact, be shown to be equivalent to the condition in Lemma 10.

Remark 4. 

It is clear that, according to Definition 9, X is weakly independent of Y if $\left|\mathcal{Y}\right|>\left|\mathcal{X}\right|$. Hence, Lemma 10 implies that $g_0(X;Y)>0$ if Y has strictly larger alphabet than X.

In light of the above remark, in the most common case of $\left|\mathcal{Y}\right|=\left|\mathcal{X}\right|$, one might have $g_0(X;Y)=0$, which corresponds to the most conservative scenario as no privacy leakage implies no broadcasting of observable data. In such cases, the rate of increase of $g_{\epsilon }(X;Y)$ at $\epsilon =0$, that is $g_0^{\prime }(X;Y):=\frac{\text{d}}{\text{d}\epsilon }{g}_{\epsilon }{(X;Y)|}_{\epsilon =0}$, which corresponds to the initial efficiency of privacy-constrained information extraction, proves to be very important in characterizing the behavior of $g_{\epsilon }(X;Y)$ for all $\epsilon \ge 0$. This is because, for example, by concavity of $\epsilon \mapsto g_{\epsilon }(X;Y)$, the slope of $g_{\epsilon }(X;Y)$ is maximized at $\epsilon =0$ and so

$$g_0^{\prime }(X;Y)=\underset{\epsilon \to 0}{lim}\frac{g_{\epsilon }(X;Y)}{\epsilon }=\underset{\epsilon >0}{sup}\frac{g_{\epsilon }(X;Y)}{\epsilon }$$

and hence $g_{\epsilon }(X;Y)\le \epsilon g_0^{\prime }(X;Y)$ for all $\epsilon \le I(X;Y)$ which, together with (4), implies that $g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)}$ if $g_0^{\prime }(X;Y)\le \frac{H\left(Y\right)}{I(X;Y)}$. In the sequel, we always assume that X is not weakly independent of Y, or equivalently $g_0(X;Y)=0$. For example, in light of Lemma 10 and Remark 4, we can assume that $\left|\mathcal{Y}\right|\le \left|\mathcal{X}\right|$.

It is easy to show that, X is weakly independent of binary Y if and only if X and Y are independent (see, e.g., [38, Remark 2]). The following corollary, therefore, immediately follows from Lemma 10.

Corollary 11. 

Let Y be a non-degenerate binary random variable correlated with X. Then $g_0(X;Y)=0$.

3. Operational Interpretations of the Rate-Privacy Function

In this section, we provide a scenario in which $g_{\epsilon }(X;Y)$ appears as a boundary point of an achievable rate region and thus giving an information-theoretic operational interpretation for $g_{\epsilon }(X;Y)$. We then proceed to present an estimation-theoretic motivation for ${\widehat{g}}_{\epsilon }(X;Y)$.

3.1. Dependence Dilution

Inspired by the problems of information amplification [39] and state masking [40], Courtade [26] proposed the information-masking tradeoff problem as follows. The tuple $(R_u,R_v,{\Delta }_A,{\Delta }_M)\in {\mathbb{R}}^4$ is said to be achievable if for two given separated sources $U\in \mathcal{U}$ and $V\in \mathcal{V}$ and any $\epsilon >0$ there exist mappings $f:{\mathcal{U}}^n\to \{1,2,\cdots ,2^{n{R}_u}\}$ and $g:{\mathcal{V}}^n\to \{1,2,\cdots ,2^{n{R}_v}\}$ such that $I(U^n;f\left(U^n\right),g\left(V^n\right))\le n({\Delta }_M+\epsilon )$ and $I(V^n;f\left(U^n\right),g\left(V^n\right))\ge n({\Delta }_A-\epsilon )$. In other words, $(R_u,R_v,{\Delta }_A,{\Delta }_M)$ is achievable if there exist indices K and J of rates $R_u$ and $R_v$ given $U^n$ and $V^n$, respectively, such that the receiver in possession of $(K,J)$ can recover at most $n{\Delta }_M$ bits about $U^n$ and at least $n{\Delta }_A$ about $V^n$. The closure of the set of all achievable tuple $(R_u,R_v,{\Delta }_A,{\Delta }_M)$ is characterized in [26]. Here, we look at a similar problem but for a joint encoder. In fact, we want to examine the achievable rate of an encoder observing both $X^n$ and $Y^n$ which masks $X^n$ and amplifies $Y^n$ at the same time, by rates ${\Delta }_M$ and ${\Delta }_A$, respectively.

We define a $(2^{nR},n)$ dependence dilution code by an encoder

$$f_n:{\mathcal{X}}^n\times {\mathcal{Y}}^n\to \{1,2,\cdots ,2^{nR}\}$$

and a list decoder

$$g_n:\{1,2,\cdots ,2^{nR}\}\to 2^{{\mathcal{Y}}^n}$$

where $2^{{\mathcal{Y}}^n}$ denotes the power set of ${\mathcal{Y}}^n$. A dependence dilution triple$(R,{\Delta }_A,{\Delta }_M)\in {\mathbb{R}}_{+}^3$ is said to be achievable if, for any $\delta >0$, there exists a $(2^{nR},n)$ dependence dilution code that for sufficiently large n satisfies the utility constraint:

$$Pr\left(Y^n\notin g_n\left(J\right)\right)<\delta$$

(14)

having a fixed list size

$$|g_n\left(J\right)|=2^{n(H\left(Y\right)-{\Delta }_A)},\forall J\in \{1,2,\cdots ,2^{nR}\}$$

(15)

where $J:=f_n(X^n,Y^n)$ is the encoder’s output, and satisfies the privacy constraint:

$$\frac{1}{n}I(X^n;J)\le {\Delta }_M+\delta$$

(16)

Intuitively speaking, upon receiving J, the decoder is required to construct list $g_n\left(J\right)\subset {\mathcal{Y}}^n$ of fixed size which contains likely candidates of the actual sequence $Y^n$. Without any observation, the decoder can only construct a list of size $2^{nH\left(Y\right)}$ which contains $Y^n$ with probability close to one. However, after J is observed and the list $g_n\left(J\right)$ is formed, the decoder’s list size can be reduced to $2^{n(H\left(Y\right)-{\Delta }_A)}$ and thus reducing the uncertainty about $Y^n$ by $0\le n{\Delta }_A\le nH\left(Y\right)$. This observation led Kim et al. [39] to show that the utility constraint (14) is equivalent to the amplification requirement

$$\frac{1}{n}I(Y^n;J)\ge {\Delta }_A-\delta$$

(17)

which lower bounds the amount of information J carries about $Y^n$. The following lemma gives an outer bound for the achievable dependence dilution region.

Theorem 12. 

Any achievable dependence dilution triple $(R,{\Delta }_A,{\Delta }_M)$ satisfies

$$\left\{\begin{array}{c}R\ge {\Delta }_A\hfill \\ {\Delta }_A\le I(Y;U)\hfill \\ {\Delta }_M\ge I(X;U)-I(Y;U)+{\Delta }_A\hfill \end{array}\right.$$

for some auxiliary random variable $U\in \mathcal{U}$ with a finite alphabet and jointly distributed with X and Y.

Before we prove this theorem, we need two preliminary lemmas. The first lemma is an extension of Fano’s inequality for list decoders and the second one makes use of a single-letterization technique to express $I(X^n;J)-I(Y^n;J)$ in a single-letter form in the sense of Csiszár and Körner [29].

Lemma 13 

([39,41]). Given a pair of random variables $(U,V)$ defined over $\mathcal{U}\times \mathcal{V}$ for finite $\mathcal{V}$ and arbitrary $\mathcal{U}$, any list decoder $g:\mathcal{U}\to 2^{\mathcal{V}}$, $U\mapsto g\left(U\right)$ of fixed list size m (i.e., $\left|g\right(u\left)\right|=m,\forall u\in \mathcal{U}$), satisfies

$$H\left(V\right|U)\le h_b\left(p_e\right)+p_{e}log\left|\mathcal{V}\right|+(1-p_e)logm$$

where $p_e:=Pr(V\notin g\left(U\right))$ and $h_b:[0,1]\to [0,1]$ is the binary entropy function.

This lemma, applied to J and $Y^n$ in place of U and V, respectively, implies that for any list decoder with the property (14), we have

$$H\left(Y^n\right|J)\le log\left|g_n\left(J\right)\right|+n{\epsilon }_n$$

(18)

where ${\epsilon }_n:=\frac{1}{n}+(log|\mathcal{Y}|-\frac{1}{n}log|g_n\left(J\right)\left|\right)p_e$ and hence ${\epsilon }_n\to 0$ as $n\to \infty$.

Lemma 14. 

Let $(X^n,Y^n)$ be n i.i.d. copies of a pair of random variables $(X,Y)$. Then for a random variable J jointly distributed with $(X^n,Y^n)$, we have

$$I(X^n;J)-I(Y^n;J)=\sum _{i=1}^n[I(X_i;U_i)-I(Y_i;U_i)]$$

where $U_i:=(J,X_{i+1}^n,Y^{i-1})$.

Proof. 

Using the chain rule for the mutual information, we can express $I(X^n;J)$ as follows

$$\begin{array}{ccc}\hfill I(X^n;J)& =& \sum _{i=1}^{n}I(X_i;J|X_{i+1}^n)=\sum _{i=1}^{n}I(X_i;J,X_{i+1}^n)\hfill \\ & =& \sum _{i=1}^n[I(X_i;J,X_{i+1}^n,Y^{i-1})-I(X_i;Y^{i-1}|J,X_{i+1}^n)]\hfill \\ & =& \sum _{i=1}^{n}I(X_i;U_i)-\sum _{i=1}^{n}I(X_i;Y^{i-1}|J,X_{i+1}^n)\hfill \end{array}$$

(19)

Similarly, we can expand $I(Y^n;J)$ as

$$\begin{array}{ccc}\hfill I(Y^n;J)& =& \sum _{i=1}^{n}I(Y_i;J|Y^{i-1})=\sum _{i=1}^{n}I(Y_i;J,Y^{i-1})\hfill \\ & =& \sum _{i=1}^n[I(Y_i;J,X_{i+1}^n,Y^{i-1})-I(Y_i;X_{i+1}^n|J,Y^{i-1})]\hfill \\ & =& \sum _{i=1}^{n}I(Y_i;U_i)-\sum _{i=1}^{n}I(Y_i;X_{i+1}^n|J,Y^{i-1})\hfill \end{array}$$

(20)

Subtracting (20) from (19), we get

$$\begin{array}{ccc}\hfill I(X^n;J)-I(Y^n;J)& =& \sum _{i=1}^n[I(X_i;U_i)-I(Y_i;U_i)]-\sum _{i=1}^n[I(X_i;Y^{i-1}|J,X_{i+1}^n)-I(X_{i+1}^n;Y_i|J,Y^{i-1})]\hfill \\ & \stackrel{\left(a\right)}{=}& \sum _{i=1}^n[I(X_i;U_i)-I(Y_i;U_i)]\hfill \end{array}$$

where $\left(a\right)$ follows from the Csiszár sum identity [42]. ☐

Proof of Theorem 12. 

The rate R can be bounded as

$$\begin{array}{ccc}\hfill nR& \ge & H\left(J\right)\ge I(Y^n;J)\hfill \\ & =& nH\left(Y\right)-H\left(Y^n\right|J)\hfill \\ & \stackrel{\left(a\right)}{\ge }& nH\left(Y\right)-log|g_n\left(J\right)|-n{\epsilon }_n\hfill \end{array}$$

(21)

$$\begin{array}{ccc}& \stackrel{\left(b\right)}{=}& n{\Delta }_A-n{\epsilon }_n\hfill \end{array}$$

(22)

where $\left(a\right)$ follows from Fano’s inequality (18) with ${\epsilon }_n\to 0$ as $n\to \infty$ and $\left(b\right)$ is due to (15). We can also upper bound ${\Delta }_A$ as

$$\begin{array}{ccc}\hfill {\Delta }_A& \stackrel{\left(a\right)}{=}& H\left(Y^n\right)-log\left|g_n\left(J\right)\right|\hfill \\ & \stackrel{\left(b\right)}{\le }& H\left(Y^n\right)-H\left(Y^n\right|J)+n{\epsilon }_n\hfill \\ & =& \sum _{i=1}^{n}H\left(Y_i\right)-H\left(Y_i\right|Y^{i-1},J)+n{\epsilon }_n\hfill \\ & \le & \sum _{i=1}^{n}H\left(Y_i\right)-H\left(Y_i\right|Y^{i-1},X_{i+1}^n,J)+n{\epsilon }_n\hfill \\ & =& \sum _{i=1}^{n}I(Y_i;U_i)+n{\epsilon }_n\hfill \end{array}$$

(23)

where $\left(a\right)$ follows from (15), $\left(b\right)$ follows from (18), and in the last equality the auxiliary random variable $U_i:=(Y^{i-1},X_{i+1}^n,J)$ is introduced.

We shall now lower bound $I(X^n;J)$:

$$\begin{array}{ccc}\hfill n({\Delta }_M+\delta )& \ge & I(X^n;J)\hfill \\ & \stackrel{\left(a\right)}{=}& I(Y^n;J)+\sum _{i=1}^n[I(X_i;U_i)-I(Y_i;U_i)]\hfill \\ & \stackrel{\left(b\right)}{\ge }& n{\Delta }_A+\sum _{i=1}^n[I(X_i;U_i)-I(Y_i;U_i)]-n{\epsilon }_n\hfill \end{array}$$

(24)

where $\left(a\right)$ follows from Lemma 14 and $\left(b\right)$ is due to Fano’s inequality and (15) (or equivalently from (17)).

Combining (22), (23) and (24), we can write

$$\begin{array}{ccc}\hfill R& \ge & {\Delta }_A-{\epsilon }_n\hfill \\ \hfill {\Delta }_A& \le & I(Y_Q;U_Q|Q)+{\epsilon }_n=I(Y_Q;U_Q,Q)+{\epsilon }_n\hfill \\ \hfill {\Delta }_M& \ge & {\Delta }_A+I(X_Q;U_Q|Q)-I(Y_Q;U_Q|Q)-{\epsilon }_n^{\prime }\hfill \\ & =& {\Delta }_A+I(X_Q;U_Q,Q)-I(Y_Q;U_Q,Q)-{\epsilon }_n^{\prime }\hfill \end{array}$$

where ${\epsilon }_n^{\prime }:={\epsilon }_n+\delta$ and Q is a random variable distributed uniformly over $\{1,2,\cdots ,n\}$ which is independent of $(X,Y)$ and hence $I(Y_Q;U_Q|Q)=\frac{1}{n}{\sum }_{i=1}^{n}I(Y_i;U_i)$. The results follow by denoting $U:=(U_Q,Q)$ and noting that $Y_Q$ and $X_Q$ have the same distributions as Y and X, respectively. ☐

If the encoder does not have direct access to the private source $X^n$, then we can define the encoder mapping as $f_n:{\mathcal{Y}}^n\to \{1,2,\cdots ,s^{nR}\}$. The following corollary is an immediate consequence of Theorem 12.

Corollary 15. 

If the encoder does not see the private source, then for all achievable dependence dilution triple $(R,{\Delta }_A,{\Delta }_M)$, we have

$$\left\{\begin{array}{c}R\ge {\Delta }_A\hfill \\ {\Delta }_A\le I(Y;U)\hfill \\ {\Delta }_M\ge I(X;U)-I(Y;U)+{\Delta }_A\hfill \end{array}\right.$$

for some joint distribution $P_{XYU}=P_{XY}{P}_{U|Y}$ where the auxiliary random variable $U\in \mathcal{U}$ satisfies $\left|\mathcal{U}\right|\le \left|\mathcal{Y}\right|+1$.

Remark 5. 

If source Y is required to be amplified (according to (17)) at maximum rate, that is, ${\Delta }_A=I(Y;U)$ for an auxiliary random variable U which satisfies $X-\circ -Y-\circ -U$, then by Corollary 15, the best privacy performance one can expect from the dependence dilution setting is

$${\Delta }_M^{*}=\underset{\begin{array}{c}U:X-\circ -Y-\circ -U\\ I(Y;U)\ge {\Delta }_A\end{array}}{min}I(X;U)$$

(25)

which is equal to the dual of $g_{\epsilon }(X;Y)$ evaluated at ${\Delta }_A$, $t_{{\Delta }_A}(X;Y)$, as defined in (1).

The dependence dilution problem is closely related to the discriminatory lossy source coding problem studied in [15]. In this problem, an encoder f observes $(X^n,Y^n)$ and wants to describe this source to a decoder, g, such that g recovers $Y^n$ within distortion level D and $I(f(X^n,Y^n);X^n)\le n{\Delta }_M$. If the distortion level is Hamming measure, then the distortion constraint and the amplification constraint are closely related via Fano’s inequality. Moreover, dependence dilution problem reduces to a secure lossless (list decoder of fixed size 1) source coding problem by setting ${\Delta }_A=H\left(H\right)$, which is recently studied in [43].

3.2. MMSE Estimation of Functions of Private Information

In this section, we provide a justification for the privacy guarantee ${\rho }_m^2(X;Z)\le \epsilon$. To this end, we recall the definition of the minimum mean squared error estimation.

Definition 16. 

Given random variables U and V, $\mathsf{mmse}\left(U\right|V)$ is defined as the minimum error of an estimate, $g\left(V\right)$, of U based on V, measured in the mean-square sense, that is

$$\mathsf{mmse}\left(U\right|V):=\underset{g\in {\mathcal{L}}^2\left(\mathcal{V}\right)}{inf}\mathbb{E}\left[{\left(U-g\left(V\right)\right)}^2\right]=\mathbb{E}\left[{\left(U-\mathbb{E}\left[U\right|V]\right)}^2\right]=\mathbb{E}\left[\mathsf{var}\left(U\right|V)\right]$$

(26)

where $\mathsf{var}\left(U\right|V)$ denotes the conditional variance of U given V.

It is easy to see that $\mathsf{mmse}\left(U\right|V)=0$ if and only if $U=f\left(V\right)$ for some measurable function f and $\mathsf{mmse}\left(U\right|V)=\mathsf{var}(U)$ if and only if $U\perp \perp V$. Hence, unlike for the case of maximal correlation, a small value of $\mathsf{mmse}\left(U\right|V)$ implies a strong dependence between U and V. Hence, although it is not a "proper" measure of correlation, in a certain sense it measures how well one random variable can be predicted from another one.

Given a non-degenerate measurable function $f:\mathcal{X}\to \mathbb{R}$, consider the following constraint on $\mathsf{mmse}\left(f\right(X\left)\right|Y)$

$$(1-\epsilon )\mathsf{var}\left(f\right(X\left)\right)\le \mathsf{mmse}\left(f\right(X\left)\right|Z)\le \mathsf{var}(f\left(X\right)).$$

(27)

This guarantees that no adversary knowing Z can efficiently estimate $f\left(X\right)$. First consider the case where f is an identity function, i.e., $f\left(x\right)=x$. In this case, a direct calculation shows that

$$\begin{array}{ccc}\hfill \mathsf{mmse}\left(X\right|Z)& \stackrel{\left(a\right)}{=}& \mathbb{E}\left[{(X-\mathbb{E}\left[X\right|Z])}^2\right]=\mathbb{E}\left[X^2\right]-\mathbb{E}\left[{\left(\mathbb{E}\left[X\right|Z]\right)}^2\right]\hfill \\ & =& \mathsf{var}\left(X\right)(1-{\rho }^2(X;\mathbb{E}\left[X\right|Z]))\hfill \\ & \stackrel{\left(b\right)}{\ge }& \mathsf{var}\left(X\right)(1-{\rho }_m^2(X;Z))\hfill \end{array}$$

where $\left(a\right)$ follows from (26) and $\left(b\right)$ is due to the definition of maximal correlation. Having imposed ${\rho }_m^2(X;Z)\le \epsilon$, we, can therefore conclude that the MMSE of estimating X given Z satisfies

$$(1-\epsilon )\mathsf{var}\left(X\right)\le \mathsf{mmse}\left(X\right|Z)\le \mathsf{var}(X)$$

(28)

which shows that ${\rho }_m^2(X;Z)\le \epsilon$ implies (27) for $f\left(x\right)=x$. However, in the following we show that the constraint ${\rho }_m^2(X;Z)\le \epsilon$ is, indeed, equivalent to (27) for any non-degenerate measurable $f:\mathcal{X}\to \mathbb{R}$.

Definition 17 

([44]). A joint distribution $P_{UV}$ satisfies a Poincaré inequality with constant $c\le 1$ if for all $f:\mathcal{U}\to \mathbb{R}$

$$c·\mathsf{var}\left(f\right(U\left)\right)\le \mathsf{mmse}\left(f\right(U\left)\right|V)$$

and the Poincaré constant for $P_{UV}$ is defined as

$$\vartheta (U;V):=\underset{f}{inf}\frac{\mathsf{mmse}\left(f\right(U\left)\right|V)}{\mathsf{var}\left(f\right(U\left)\right)}$$

The privacy constraint (27) can then be viewed as

$$\vartheta (X;Z)\ge 1-\epsilon .$$

(29)

Theorem 18 

([44]). For any joint distribution $P_{UV}$, we have

$$\vartheta (U;V)=1-{\rho }_m^2(U;V)$$

In light of Theorem 18 and (29), the privacy constraint (27) is equivalent to ${\rho }_m^2(X;Z)\le \epsilon$, that is,

$${\rho }_m^2(X;Z)\le \epsilon ⟺(1-\epsilon )\mathsf{var}\left(f\left(X\right)\right)\le \mathsf{mmse}\left(f\left(X\right)\right|Z)\le \mathsf{var}\left(f\left(X\right)\right)$$

for any non-degenerate measurable functions $f:\mathcal{X}\to \mathbb{R}$.

Hence, ${\widehat{g}}_{\epsilon }(X;Y)$ characterizes the maximum information extraction from Y such that no (non-trivial) function of X can be efficiently estimated, in terms of MMSE (27), given the extracted information.

4. Observation Channels for Minimal and Maximal ${\mathbf{g}}_{\epsilon }(\mathbf{X};\mathbf{Y})$

In this section, we characterize the observation channels which achieve the lower or upper bounds on the rate-privacy function in (4). We first derive general conditions for achieving the lower bound and then present a large family of observation channels $P_{Y|X}$ which achieve the lower bound. We also give a family of $P_{Y|X}$ which attain the upper bound on $g_{\epsilon }(X;Y)$.

4.1. Conditions for Minimal $g_{\epsilon }(X;Y)$

Assuming that $g_0(X;Y)=0$, we seek a set of conditions on $P_{XY}$ such that $g_{\epsilon }(X;Y)$ is linear in ε, or equivalently, $g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)}$. In order to do this, we shall examine the slope of $g_{\epsilon }(X;Y)$ at zero. Recall that by concavity of $g_{\epsilon }(X;Y)$, it is clear that $g_0^{\prime }(X;Y)\ge \frac{H\left(Y\right)}{I(X;Y)}$. We strengthen this bound in the following lemmas. For this, we need to recall the notion of Kullback-Leibler divergence. Given two probability distribution P and Q supported over a finite alphabet $\mathcal{U}$,

$$D\left(P\right|\left|Q\right):=\sum _{u\in \mathcal{U}}P\left(u\right)log\left(\frac{P\left(u\right)}{Q\left(u\right)}\right)$$

(30)

Lemma 19. 

For a given joint distribution $P_{XY}=P_Y\times P_{X|Y}$, if $g_0(X;Y)=0$, then for any $\epsilon \ge 0$

$$g_0^{\prime }(X;Y)\ge \underset{y\in \mathcal{Y}}{max}\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)}$$

Proof. 

The proof is given in Appendix A. ☐

Remark 6. 

Note that if for a given joint distribution $P_{XY}$, there exists $y_0\in \mathcal{Y}$ such that $D\left(P_{X|Y}(·|y_0)\right|\left|P_X(·)\right)=0$, it implies that $P_{X|Y}(·|y_0)=P_X\left(x\right)$. Consider the binary random variable $Z\in \{1,\text{e}\}$ constructed according to the distribution $P_{Z|Y}\left(1\right|y_0)=1$ and $P_{Z|Y}\left(\text{e}\right|y)=1$ for $y\in \mathcal{Y}\backslash \left\{y_0\right\}$. We can now claim that Z is independent of X, because $P_{X|Z}\left(x\right|1)=P_{X|Y}\left(x\right|y_0)=P_X\left(x\right)$ and

$$\begin{array}{ccc}\hfill P_{X|Z}\left(x\right|\mathrm{e})& =& \sum _{y\ne y_0}{P}_{X|Y}\left(x\right|y)P_{Y|Z}\left(y\right|\mathrm{e})=\sum _{y\ne y_0}{P}_{X|Y}\left(x\right|y)\frac{P_Y\left(y\right)}{1-P_Y\left(y_0\right)}\hfill \\ & =& \frac{1}{1-P_Y\left(y_0\right)}\sum _{y\ne y_0}{P}_{XY}(x,y)=P_X\left(x\right)\hfill \end{array}$$

Clearly, Z and Y are not independent, and hence $g_0(X;Y)>0$. This implies that the right-hand side of inequality in Lemma 19 can not be infinity.

In order to prove the main result, we need the following simple lemma.

Lemma 20. 

For any joint distribution $P_{XY}$, we have

$$\frac{H\left(Y\right)}{I(X;Y)}\le \underset{y\in \mathcal{Y}}{max}\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X\left(x\right)\right)}$$

where equality holds if and only if there exists a constant $c>0$ such that $-log{P}_Y\left(y\right)=cD(P_{X|Y}(·|y)\left|\right|P_X\left(x\right))$ for all $y\in \mathcal{Y}$.

Proof. 

It is clear that

$$\frac{H\left(Y\right)}{I(X;Y)}=\frac{-{\sum }_{y\in \mathcal{Y}}{P}_Y\left(y\right)log{P}_Y\left(y\right)}{{\sum }_{y\in \mathcal{Y}}{P}_Y\left(y\right)D(P_{X|Y}(·|y)\left|\right|P_X\left(x\right))}\le \underset{y\in \mathcal{Y}}{max}\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X\left(x\right)\right)}$$

where the inequality follows from the fact that for any three sequences of positive numbers ${\left\{a_i\right\}}_{i=1}^n$, ${\left\{b_i\right\}}_{i=1}^n$ and ${\left\{{\lambda }_i\right\}}_{i=1}^n$ we have $\frac{{\sum }_{i=1}^n{\lambda }_i{a}_i}{{\sum }_{i=1}^n{\lambda }_i{b}_i}\le {max}_{1\le i\le n}\frac{a_i}{b_i}$ where equality occurs if and only if $\frac{a_i}{b_i}=c$ for all $1\le i\le n$. ☐

Now we are ready to state the main result of this subsection.

Theorem 21. 

For a given $(X,Y)$ with joint distribution $P_{XY}=P_Y\times P_{X|Y}$, if $g_0(X;Y)=0$ and $\epsilon \mapsto g_{\epsilon }(X;Y)$ is linear for $0\le \epsilon \le I(X;Y)$, then for any $y\in \mathcal{Y}$

$$\frac{H\left(Y\right)}{I(X;Y)}=\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)}$$

Proof. 

Note that the fact that $g_0(X;Y)=0$ and $g_{\epsilon }(X;Y)$ is linear in ε is equivalent to $g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)}$. It is, therefore, immediate from Lemmas 19 and 20 that we have

$$\begin{array}{ccc}\hfill g_0^{\prime }(X;Y)& \stackrel{\left(a\right)}{=}& \frac{H\left(Y\right)}{I(X;Y)}\stackrel{\left(b\right)}{\le }\underset{y\in \mathcal{Y}}{max}\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X\left(x\right)\right)}\hfill \\ & \stackrel{\left(c\right)}{\le }& g_0^{\prime }(X;Y)\hfill \end{array}$$

(31)

where $\left(a\right)$ follows from the fact that $g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)}$ and $\left(b\right)$ and $\left(c\right)$ are due to Lemmas 20 and 19, respectively. The inequality in (31) shows that

$$\frac{H\left(Y\right)}{I(X;Y)}=\underset{y\in \mathcal{Y}}{max}\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X\left(x\right)\right)}$$

(32)

According to Lemma 20, (32) implies that the ratio of $\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X\left(x\right)\right)}$ does not depend on $y\in \mathcal{Y}$ and hence the result follows. ☐

This theorem implies that if there exists $y=y_1$ and $y=y_2$ such that $\frac{log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X\left(x\right)\right)}$ results in two different values, then $\epsilon \mapsto g_{\epsilon }(X,Y)$ cannot achieve the lower bound in (4), or equivalently

$$g_{\epsilon }(X;Y)>\epsilon \frac{H\left(Y\right)}{I(X;Y)}$$

This, therefore, gives a necessary condition for the lower bound to be achievable. The following corollary simplifies this necessary condition.

Corollary 22. 

For a given joint distribution $P_{XY}=P_Y\times P_{X|Y}$, if $g_0(X;Y)=0$ and $\epsilon \mapsto g_{\epsilon }(X;Y)$ is linear, then the following are equivalent:

(i) 

Y is uniformly distributed,

(ii) 

$D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)$ is constant for all $y\in \mathcal{Y}$.

Proof. 

$\left(i\right)\Rightarrow \left(ii\right)$:

From Theorem 21, we have for all $y\in \mathcal{Y}$

$$\frac{H\left(Y\right)}{I(X;Y)}=\frac{-log\left(P_Y\left(y\right)\right)}{D\left(P_{X|Y}(·|y)\left|\right|P_X(·)\right)}$$

(33)

Letting $D:=D\left(P_{X|Y}(·|y)\left|\right|P_X(·)\right)$ for any $y\in \mathcal{Y}$, we have ${\sum }_y{P}_Y\left(y\right)D=I(X;Y)$ and hence $D=I(X;Y)$, which together with (33) implies that $H\left(Y\right)=-log\left(P_Y\left(y\right)\right)$ for all $y\in \mathcal{Y}$ and hence Y is uniformly distributed.

$\left(ii\right)\Rightarrow \left(i\right)$:

When Y is uniformly distributed, we have from (33) that $I(X;Y)=D\left(P_{X|Y}(·|y)\left|\right|P_X(·)\right)$ which implies that $D\left(P_{X|Y}(·|y)\left|\right|P_X(·)\right)$ is constant for all $y\in \mathcal{Y}$. ☐

Example 1. 

Suppose $P_{Y|X}$ is a binary symmetric channel (BSC) with crossover probability $0<\alpha <1$ and $P_X=\mathsf{Bernoulli}\left(0.5\right)$. In this case, $P_{X|Y}$ is also a BSC with input distribution $P_Y=\mathsf{Bernoulli}\left(0.5\right)$. Note that Corollary 11 implies that $g_0(X;Y)=0$. We will show that $g_{\epsilon }(X;Y)$ is linear as a function of $\epsilon \ge 0$ for a larger family of symmetric channels (including BSC) in Corollary 24. Hence, the BSC with uniform input nicely illustrates Corollary 22, because $D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)=1-h\left(\alpha \right)$ for $y\in \{0,1\}$.

Example 2. 

Now suppose $P_{X|Y}$ is a binary asymmetric channel such that $P_{X|Y}(·|0)=\mathsf{Bernoulli}\left({\alpha }_0\right)$, $P_{X|Y}(·|1)=\mathsf{Bernoulli}\left({\alpha }_1\right)$ for some $0<{\alpha }_0,{\alpha }_1<1$ and input distribution $P_Y=\mathsf{Bernoulli}\left(p\right)$, $0<p\le 0.5$. It is easy to see that if ${\alpha }_0+{\alpha }_1=1$ then $D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)$ does not depend on y and hence we can conclude from Corollary 22 (noticing that $g_0(X;Y)=0$) that in this case for any $p<0.5$, $g_{\epsilon }(X;Y)$ is not linear and hence for $0<\epsilon <I(X;Y)$

$$g_{\epsilon }(X;Y)>\epsilon \frac{H\left(Y\right)}{I(X;Y)}$$

In Theorem 21, we showed that when $g_{\epsilon }(X;Y)$ achieves its lower bound, illustrated in (4), the slope of the mapping $\epsilon \mapsto g_{\epsilon }(X;Y)$ at zero is equal to $\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)}$ for any $y\in \mathcal{Y}$. We will show in the next section that the reverse direction is also true at least for a large family of binary-input symmetric output channels, for instance when $P_{Y|X}$ is a BSC, and thus showing that in this case,

$$g_0^{\prime }(X;Y)=\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)},\forall y\in \mathcal{Y}⟺g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)},0\le \epsilon \le I(X;Y)$$

4.2. Special Observation Channels

In this section, we apply the results of last section to different joint distributions $P_{XY}$. In the first family of channels from X to Y, we look at the case where Y is binary and the reverse channel $P_{X|Y}$ has symmetry in a particular sense, which will be specified later. One particular case of this family of channels is when $P_{X|Y}$ is a BSC. As a family of observation channels which achieves the upper bound of $g_{\epsilon }(X;Y)$, stated in (4), we look at the class of erasure channels from $X\to Y$, i.e., Y is an erasure version of X.

4.2.1. Observation Channels With Symmetric Reverse

The first example of $P_{XY}$ that we consider for binary Y is the so-called Binary Input Symmetric Output (BISO) $P_{X|Y}$, see for example [45,46]. Suppose $\mathcal{Y}=\{0,1\}$ and $\mathcal{X}=\{0,\pm 1,\pm 2,\cdots ,\pm k\}$, and for any $x\in \mathcal{X}$ we have $P_{X|Y}\left(x\right|1)=P_{X|Y}(-x|0)$. This clearly implies that $p_0:=P_{X|Y}\left(0\right|0)=P_{X|Y}\left(0\right|1)$. We notice that with this definition of symmetry, we can always assume that the output alphabet $\mathcal{X}=\{\pm 1,\pm 2,\cdots ,\pm k\}$ has even number of elements because we can split $X=0$ into two outputs, $X=0^{+}$ and $X=0^{-}$, with $P_{X|Y}\left(0^{-}\right|0)=P_{X|Y}\left(0^{+}\right|0)=\frac{p_0}{2}$ and $P_{X|Y}\left(0^{-}\right|1)=P_{X|Y}\left(0^{+}\right|1)=\frac{p_0}{2}$. The new channel is clearly essentially equivalent to the original one, see [46] for more details. This family of channels can also be characterized using the definition of quasi-symmetric channels [47, Definition 4.17]. A channel $\mathsf{W}$ is BISO if (after making $\left|\mathcal{X}\right|$ even) the transition matrix $P_{X|Y}$ can be partitioned along its columns into binary-input binary-output sub-arrays in which rows are permutations of each other and the column sums are equal. It is clear that binary symmetric channels and binary erasure channels are both BISO. The following lemma gives an upper bound for $g_{\epsilon }(X,Y)$ when $P_{X|Y}$ belongs to such a family of channels.

Lemma 23. 

If the channel $P_{X|Y}$ is BISO, then for $\epsilon \in [0,I(X;Y\left)\right]$,

$$\epsilon \frac{H\left(Y\right)}{I(X;Y)}\le g_{\epsilon }(X;Y)\le H\left(Y\right)-\frac{I(X;Y)-\epsilon }{C\left(P_{X|Y}\right)}$$

where $C\left(P_{X|Y}\right)$ denotes the capacity of $P_{X|Y}$.

Proof. 

The lower bound has already appeared in (4). To prove the upper bound note that by Markovity $X-\circ -Y-\circ -Z$, we have for any $x\in \mathcal{X}$ and $z\in \mathcal{Z}$

$$P_{X|Z}\left(x\right|z)=P_{X|Y}\left(x\right|0)P_{Y|Z}\left(0\right|z)+P_{X|Y}\left(x\right|1)P_{Y|Z}\left(1\right|z)$$

(34)

Now suppose ${\mathcal{Z}}_0:=\{z:P_{Y|Z}\left(0\right|z)\le P_{Y|Z}\left(1\right|z)\}$ and similarly ${\mathcal{Z}}_1:=\{z:P_{Y|Z}\left(1\right|z)\le P_{Y|Z}\left(0\right|z)\}$. Then (34) allows us to write for $z\in {\mathcal{Z}}_0$

$$P_{X|Z}\left(x\right|z)=P_{X|Y}\left(x\right|0)h_b^{-1}\left(H\left(Y\right|Z=z)\right)+P_{X|Y}\left(x\right|1)(1-h_b^{-1}\left(H\left(Y\right|Z=z)\right))$$

(35)

where $h_b^{-1}:[0,1]\to [0,0.5]$ is the inverse of binary entropy function, and for $z\in {\mathcal{Z}}_1$,

$$P_{X|Z}\left(x\right|z)=P_{X|Y}\left(x\right|0)(1-h_b^{-1}\left(H\left(Y\right|Z=z)\right))+P_{X|Y}\left(x\right|1)h_b^{-1}\left(H\left(Y\right|Z=z)\right)$$

(36)

Letting $P\otimes h_b^{-1}\left(H\left(Y\right|z)\right)$ and $\tilde{P}\otimes h_b^{-1}\left(H\left(Y\right|z)\right)$ denote the right-hand sides of (35) and (36), respectively, we can, hence, write

$$\begin{array}{ccc}\hfill H\left(X\right|Z)& =& \sum _{z\in \mathcal{Z}}{P}_Z\left(z\right)H\left(X\right|Z=z)\hfill \\ & \stackrel{\left(a\right)}{=}& \sum _{z\in {\mathcal{Z}}_0}{P}_Z\left(z\right)H(P\otimes h_b^{-1}\left(H\left(Y\right|Z=z)\right))+\sum _{z\in {\mathcal{Z}}_1}{P}_Z\left(z\right)H(\tilde{P}\otimes h_b^{-1}\left(H\left(Y\right|Z=z)\right))\hfill \\ & \stackrel{\left(b\right)}{\le }& \sum _{z\in {\mathcal{Z}}_0}{P}_Z\left(z\right)\left[(1-H\left(Y\right|Z=z))H(P\otimes h_b^{-1}\left(0\right))+H\left(Y\right|Z=z)H(P\otimes h_b^{-1}\left(1\right))\right]\hfill \\ & & +\sum _{z\in {\mathcal{Z}}_1}{P}_Z\left(z\right)\left[(1-H\left(Y\right|Z=z))H(\tilde{P}\otimes h_b^{-1}\left(0\right))+H\left(Y\right|Z=z)H(\tilde{P}\otimes h_b^{-1}\left(1\right))\right]\hfill \\ & \stackrel{\left(c\right)}{=}& \sum _{z\in {\mathcal{Z}}_0}{P}_Z\left(z\right)\left[(1-H\left(Y\right|Z=z))H\left(X\right|Y)+H\left(Y\right|Z=z)H\left(X_{\text{unif}}\right)\right]\hfill \\ & & +\sum _{z\in {\mathcal{Z}}_1}{P}_Z\left(z\right)\left[(1-H\left(Y\right|Z=z))H\left(X\right|Y)+H\left(Y\right|Z=z)H\left(X_{\text{unif}}\right)\right]\hfill \\ & =& H\left(X\right|Y)[1-H\left(Y\right|Z)]+H\left(Y\right|Z)H\left(X_{\text{unif}}\right)\hfill \end{array}$$

where $H\left(X_{\text{unif}}\right)$ denotes the entropy of X when Y is uniformly distributed. Here, $\left(a\right)$ is due to (35) and (36), $\left(b\right)$ follows form convexity of $u\mapsto H(P\otimes h_b^{-1}\left(u\right)))$ for all $u\in [0,1]$[48] and Jensen’s inequality. In $\left(c\right)$, we used the symmetry of channel $P_{X|Y}$ to show that $H\left(X\right|Y=0)=H(X|Y=1)=H\left(X\right|Y)$. Hence, we obtain

$$H\left(Y\right|Z)\ge \frac{H\left(X\right|Z)-H(X\left|Y\right)}{H\left(X_{\text{unif}}\right)-H\left(X\right|Y)}=\frac{I(X;Y)-I(X;Z)}{C\left(P_{X|Y}\right)}$$

where the equality follows from the fact that for BISO channel (and in general for any quasi-symmetric channel) the uniform input distribution is the capacity-achieving distribution [47, Lemma 4.18]. Since $g_{\epsilon }(X;Y)$ is attained when $I(X;Z)=\epsilon$, the conclusion immediately follows. ☐

This lemma then shows that the larger the gap between $I(X;Y)$ and $I(X;Y^{\prime })$ is for $Y^{\prime }\sim \mathsf{Bernoulli}\left(0.5\right)$, the more $g_{\epsilon }(X;Y)$ deviates from its lower bound. When $Y\sim \mathsf{Bernoulli}\left(0.5\right)$, then $C\left(P_{Y|X}\right)=I(X;Y)$ and $H\left(Y\right)=1$ and hence Lemma 23 implies that

$$\frac{\epsilon }{I(X;Y)}\le g_{\epsilon }(X;Y)\le 1-\frac{I(X;Y)-\epsilon }{I(X;Y)}=\frac{\epsilon }{I(X;Y)}$$

and hence we have proved the following corollary.

Corollary 24. 

If the channel $P_{X|Y}$ is BISO and $Y\sim \mathsf{Bernoulli}\left(0.5\right)$, then for any $\epsilon \ge 0$

$$g_{\epsilon }(X;Y)=\frac{1}{I(X;Y)}min\{\epsilon ,I(X;Y)\}$$

This corollary now enables us to prove the reverse direction of Theorem 21 for the family of BISO channels.

Theorem 25. 

If $P_{X|Y}$ is a BISO channel, then the following statements are equivalent:

(i) 

$g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)}$ for $0\le \epsilon \le I(X;Y)$.

(ii) 

The initial efficiency of privacy-constrained information extraction is

$$g_0^{\prime }(X;Y)=\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)},\forall y\in \mathcal{Y}$$

Proof. 

(i)⇒ (ii).

This follows from Theorem 21.

(ii)⇒ (i).

Let $Y\sim \mathsf{Bernoulli}\left(p\right)$ for $0<p<1$, and, as before, $\mathcal{X}=\{\pm 1,\pm 2,\cdots ,\pm k\}$, so that $P_{X|Y}$ is determined by a $2\times \left(2k\right)$ matrix. We then have

$$\frac{-log{P}_Y\left(0\right)}{D\left(P_{X|Y}(·|0)\right|\left|P_X(·)\right)}=\frac{log(1-p)}{H\left(X\right|Y)+{\sum }_{x=-k}^k{P}_{X|Y}\left(x\right|0)log\left(P_X\left(x\right)\right)}$$

(37)

and

$$\frac{-log{P}_Y\left(1\right)}{D\left(P_{X|Y}(·|1)\right|\left|P_X(·)\right)}=\frac{log\left(p\right)}{H\left(X\right|Y)+{\sum }_{x=-k}^k{P}_{X|Y}\left(x\right|1)log\left(P_X\left(x\right)\right)}.$$

(38)

The hypothesis implies that (37) is equal to (38), that is,

$$\frac{log(1-p)}{H\left(X\right|Y)+{\sum }_{x=-k}^k{P}_{X|Y}\left(x\right|0)log\left(P_X\left(x\right)\right)}=\frac{log\left(p\right)}{H\left(X\right|Y)+{\sum }_{x=-k}^k{P}_{X|Y}\left(x\right|1)log\left(P_X\left(x\right)\right)}$$

(39)

It is shown in Appendix B that (39) holds if and only if $p=0.5$. Now we can invoke Corollary 24 to conclude that $g_{\epsilon }(X;Y)=\epsilon \frac{H\left(Y\right)}{I(X;Y)}$. ☐

This theorem shows that for any BISO $P_{X|Y}$ channel with uniform input, the optimal privacy filter is an erasure channel depicted in Figure 2. Note that if $P_{X|Y}$ is a BSC with uniform input $P_Y=\mathsf{Bernoulli}\left(0.5\right)$, then $P_{Y|X}$ is also a BSC with uniform input $P_X=\mathsf{Bernoulli}\left(0.5\right)$. The following corollary specializes Corollary 24 for this case.

Corollary 26. 

For the joint distribution $P_X{P}_{Y|X}=\mathsf{Bernoulli}\left(0.5\right)\times \text{BSC}\left(\alpha \right)$, the binary erasure channel with erasure probability (shown in Figure 4)

$$\delta (\epsilon ,\alpha ):=1-\frac{\epsilon }{I(X;Y)}$$

(40)

for $0\le \epsilon \le I(X;Y)$, is the optimal privacy filter in (3). In other words, for $\epsilon \ge 0$

$$g_{\epsilon }(X;Y)=\frac{1}{I(X;Y)}min\{\epsilon ,I(X;Y)\}$$

Moreover, for a given $0<\alpha <\frac{1}{2}$, $P_X=\mathsf{Bernoulli}\left(0.5\right)$ is the only distribution for which $\epsilon \mapsto g_{\epsilon }(X;Y)$ is linear. That is, for $P_X{P}_{Y|X}=\mathsf{Bernoulli}\left(p\right)\times \text{BSC}\left(\alpha \right)$, $0<p<0.5$, we have

$$g_{\epsilon }(X;Y)>\epsilon \frac{H\left(Y\right)}{I(X;Y)}$$

Proof. 

As mentioned earlier, since $P_X=\mathsf{Bernoulli}\left(0.5\right)$ and $P_{Y|X}$ is $\text{BSC}\left(\alpha \right)$, it follows that $P_{X|Y}$ is also a BSC with uniform input and hence from Corollary 24, we have $g_{\epsilon }(X;Y)=\frac{\epsilon }{I(X;Y)}$. As in this case $g_{\epsilon }(X;Y)$ achieves the lower bound given in Lemma 1, we conclude from Figure 2 that BEC($\delta (\epsilon ,\alpha )$), where $\delta (\epsilon ,\alpha )=1-\frac{\epsilon }{I(X;Y)}$, is an optimal privacy filter. The fact that $P_X=\mathsf{Bernoulli}\left(0.5\right)$ is the only input distribution for which $\epsilon \mapsto g_{\epsilon }(X;Y)$ is linear follows from the proof of Theorem 25. In particular, we saw that a necessary and sufficient condition for $g_{\epsilon }(X;Y)$ being linear is that the ratio $\frac{-log{P}_Y\left(y\right)}{D\left(P_{X|Y}(·|y)\right|\left|P_X(·)\right)}$ is constant for all $y\in \mathcal{Y}$. As shown before, this is equivalent to $Y\sim \mathsf{Bernoulli}\left(0.5\right)$. For the binary symmetric channel, this is equivalent to $X\sim \mathsf{Bernoulli}\left(0.5\right)$. ☐

The optimal privacy filter for BSC(α) and uniform X is shown in Figure 4. In fact, this corollary immediately implies that the general lower-bound given in (4) is tight for the binary symmetric channel with uniform X.

4.2.2. Erasure Observation Channel

Combining (8) and Lemma 1, we have for $\epsilon \le I(X;Y)$

$$\epsilon \frac{H\left(Y\right)}{I(X;Y)}+g_0(X;Y)\left[1-\frac{\epsilon }{I(X;Y)}\right]\le g_{\epsilon }(X;Y)\le H\left(Y\right|X)+\epsilon$$

(41)

In the following we show that the above upper and lower bound coincide when $P_{Y|X}$ is an erasure channel, i.e., $P_{Y|X}\left(x\right|x)=1-\delta$ and $P_{Y|X}\left(\mathrm{e}\right|x)=\delta$ for all $x\in \mathcal{X}$ and $0\le \delta \le 1$.

Lemma 27. 

For any given $(X,Y)$, if $P_{Y|X}$ is an erasure channel (as defined above), then

$$g_{\epsilon }(X;Y)=H\left(Y\right|X)+min\{\epsilon ,I(X;Y)\}$$

for any $\epsilon \ge 0$.

Proof. 

It suffices to show that if $P_{Y|X}$ is an erasure channel, then $g_0(X;Y)=H\left(Y\right|X)$. This follows, since if $g_0(X;Y)=H\left(Y\right|X)$, then the lower bound in (41) becomes $H\left(Y\right|X)+\epsilon$ and thus $g_{\epsilon }(X;Y)=H\left(Y\right|X)+\epsilon$.

Let $\left|\mathcal{X}\right|=m$ and $\mathcal{Y}=\mathcal{X}\cup \left\{\text{e}\right\}$ where e denotes the erasure symbol. Consider the following privacy filter to generate $Z\in \mathcal{Y}$:

$$P_{Z|Y}\left(z\right|y)=\left\{\begin{array}{cc}\frac{1}{m}\hfill & \text{if}y\ne \text{e},z\ne \text{e},\hfill \\ 1\hfill & \text{if}y=z=\text{e}.\hfill \end{array}\right.$$

For any $x\in \mathcal{X}$, we have

$$P_{Z|X}\left(z\right|x)=P_{Z|Y}\left(z\right|x)P_{Y|X}\left(x\right|x)+P_{Z|Y}\left(z\right|\mathrm{e})P_{Y|X}\left(\mathrm{e}\right|x)=\left[\frac{1-\delta }{m}\right]1_{\{z\ne \mathrm{e}\}}+\delta 1_{\{z=\mathrm{e}\}}$$

which implies $Z\perp \perp X$ and thus $I(X;Z)=0$. On the other hand, $P_Z\left(z\right)=\left[\frac{1-\delta }{m}\right]1_{\{z\ne \text{e}\}}+\delta 1_{\{z=\text{e}\}}$, and therefore we have

$$\begin{array}{ccc}\hfill g_0(X;Y)& \ge & I(Y;Z)=H\left(Z\right)-H\left(Z\right|Y)=H\left(\frac{1-\delta }{m},\cdots ,\frac{1-\delta }{m},\delta \right)-(1-\delta )log\left(m\right)\hfill \\ & =& h\left(\delta \right)=H\left(Y\right|X)\hfill \end{array}$$

It then follows from Lemma 1 that $g_0(X;Y)=H\left(Y\right|X)$, which completes the proof. ☐

Example 3. 

In light of this lemma, we can conclude that if $P_{Y|X}=\text{BEC}\left(\delta \right)$, then the optimal privacy filter is a combination of an identity channel and a BSC($\alpha (\epsilon ,\delta )$), as shown in Figure 5, where $0\le \alpha (\epsilon ,\delta )\le \frac{1}{2}$ is the unique solution of

$$(1-\delta )[h_b(\alpha *p)-h_b\left(\alpha \right)]=\epsilon$$

(42)

where $X\sim \mathsf{Bernoulli}\left(p\right)$, $p\le 0.5$ and $a*b=a(1-b)+b(1-a)$. Note that it is easy to check that $I(X;Z)=(1-\delta )[h_b(\alpha *p)-h_b\left(\alpha \right)]$. Therefore, in order for this channel to be a valid privacy filter, the crossover probability, $\alpha (\epsilon ,\delta )$, must be chosen such that $I(X;Z)=\epsilon$. We note that for fixed $0<\delta <1$ and $0<p<0.5$, the map $\alpha \mapsto (1-\delta )[h_b(\alpha *p)-h_b\left(\alpha \right)]$ is monotonically decreasing on $[0,\frac{1}{2}]$ ranging over $[0,(1-\delta )h_b\left(p\right)]$ and since $\epsilon \le I(X;Y)=(1-\delta )h_b\left(p\right)$, the solution of the above equation is unique.

Combining Lemmas 1 and 27 with Corollary 26, we can show the following extremal property of the BEC and BSC channels, which is similar to other existing extremal properties of the BEC and the BSC, see, e.g., [46] and [45]. For $X\sim \mathsf{Bernoulli}\left(0.5\right)$, we have for any channel $P_{Y|X}$,

$$g_{\epsilon }(X;Y)\ge \frac{\epsilon }{I(X;Y)}=g_{\epsilon }\left(\text{BSC}\left(\widehat{\alpha }\right)\right)$$

where $g_{\epsilon }\left(\text{BSC}\left(\alpha \right)\right)$ is the rate-privacy function corresponding to $P_{XY}=\mathsf{Bernoulli}\left(0.5\right)\times \text{BSC}\left(\alpha \right)$ and $\widehat{\alpha }:=h_b^{-1}\left(H\left(X\right|Y)\right)$. Similarly, if $X\sim \mathsf{Bernoulli}\left(p\right)$, we have for any channel $P_{Y|X}$ with $H\left(Y\right|X)\le 1$,

$$g_{\epsilon }(X;Y)\le H\left(Y\right|X)+\epsilon =g_{\epsilon }\left(\text{BEC}\left(\widehat{\delta }\right)\right)$$

where $g_{\epsilon }\left(\text{BEC}\left(\delta \right)\right)$ is the rate-privacy function corresponding to $P_{XY}=\mathsf{Bernoulli}\left(p\right)\times \text{BEC}\left(\delta \right)$ and $\widehat{\delta }:=h_b^{-1}\left(H\left(Y\right|X)\right)$.

5. Rate-Privacy Function for Continuous Random Variables

In this section we extend the rate-privacy function $g_{\epsilon }(X;Y)$ to the continuous case. Specifically, we assume that the private and observable data are continuous random variables and that the filter is composed of two stages: first Gaussian noise is added and then the resulting random variable is quantized using an M-bit accuracy uniform scalar quantizer (for some positive integer $M\in \mathbb{N}$). These filters are of practical interest as they can be easily implemented. This section is divided in two subsections, in the first we discuss general properties of the rate-privacy function and in the second we study the Gaussian case in more detail. Some observations on ${\widehat{g}}_{\epsilon }(X;Y)$ for continuous X and Y are also given.

5.1. General Properties of the Rate-Privacy Function

Throughout this section we assume that the random vector $(X,Y)$ is absolutely continuous with respect to the Lebesgue measure on ${\mathbb{R}}^2$. Additionally, we assume that its joint density $f_{X,Y}$ satisfies the following.

(a)

There exist constants $C_1>0$, $p>1$ and bounded function $C_2:\mathbb{R}\to \mathbb{R}$ such that

$$f_Y\left(y\right)\le C_1{\left|y\right|}^{-p}$$

and also for $x\in \mathbb{R}$

$$f_{Y|X}\left(y\right|x)\le C_2\left(x\right){\left|y\right|}^{-p}$$

(b)

$\mathbb{E}\left[X^2\right]$ and $\mathbb{E}\left[Y^2\right]$ are both finite,

(c)

the differential entropy of $(X,Y)$ satisfies $h(X,Y)>-\infty$,

(d)

$H(\lfloor Y\rfloor )<\infty$, where $\lfloor a\rfloor$ denotes the largest integer ℓ such that $\ell \le a$.

Note that assumptions (b) and (c) together imply that $h(X,Y)$, $h\left(X\right)$ and $h\left(Y\right)$ are finite, i.e., the maps $x\mapsto f_X\left(x\right)|log{f}_X\left(x\right)|,y\mapsto f_Y\left(y\right)|log{f}_Y\left(y\right)|$ and $(x,y)\mapsto f_{X,Y}(x,y)|log\left(f_{X,Y}(x,y)\right)|$ are integrable. We also assume that X and Y are not independent, since otherwise the problem to characterize $g_{\epsilon }(X;Y)$ becomes trivial by assuming that the displayed data Z can equal the observable data Y.

We are interested in filters of the form ${\mathcal{Q}}_M(Y+\gamma N)$ where $\gamma \ge 0$, $N\sim N(0,1)$ is a standard normal random variable which is independent of X and Y, and for any positive integer M, ${\mathcal{Q}}_M$ denotes the M-bit accuracy uniform scalar quantizer, i.e., for all $x\in \mathbb{R}$

$${\mathcal{Q}}_M\left(x\right)=\frac{1}{2^M}⌊2^{M}x⌋$$

Let $Z_{\gamma }=Y+\gamma N$ and $Z_{\gamma }^M={\mathcal{Q}}_M\left(Z_{\gamma }\right)={\mathcal{Q}}_M(Y+\gamma N)$. We define, for any $M\in \mathbb{N}$,

$$g_{\epsilon ,M}(X;Y):=\underset{\begin{array}{c}\gamma \ge 0,\\ I(X;Z_{\gamma }^M)\le \epsilon \end{array}}{sup}I(Y;Z_{\gamma }^M)$$

(43)

and similarly

$$g_{\epsilon }(X;Y):=\underset{\begin{array}{c}\gamma \ge 0,\\ I(X;Z_{\gamma })\le \epsilon \end{array}}{sup}I(Y;Z_{\gamma })$$

(44)

The next theorem shows that the previous definitions are closely related.

Theorem 28. 

Let $\epsilon >0$ be fixed. Then ${\displaystyle \underset{M\to \infty }{lim}{g}_{\epsilon ,M}(X;Y)=g_{\epsilon }(X;Y)}$.

Proof. 

See Appendix C. ☐

In the limit of large M, $g_{\epsilon }(X;Y)$ approximates $g_{\epsilon ,M}(X;Y)$. This becomes relevant when $g_{\epsilon }(X;Y)$ is easier to compute than $g_{\epsilon ,M}(X;Y)$, as demonstrated in the following subsection. The following theorem summarizes some general properties of $g_{\epsilon }(X;Y)$.

Theorem 29. 

The function $\epsilon \mapsto g_{\epsilon }(X;Y)$ is non-negative, strictly-increasing, and satisfies

$$\underset{\epsilon \to 0}{lim}{g}_{\epsilon }(X;Y)=0and{g}_{I(X;Y)}(X;Y)=\infty$$

Proof. 

See Appendix C. ☐

As opposed to the discrete case, in the continuous case $g_{\epsilon }(X;Y)$ is no longer bounded. In the following section we show that $\epsilon \mapsto g_{\epsilon }(X;Y)$ can be convex, in contrast to the discrete case where it is always concave.

We can also define ${\widehat{g}}_{\epsilon ,M}(X;Y)$ and ${\widehat{g}}_{\epsilon }(X;Y)$ for continuous X and Y, similar to (43) and (44), but where the privacy constraints are replaced by ${\rho }_m^2(X;Z_{\gamma }^M)\le \epsilon$ and ${\rho }_m^2(X;Z_{\gamma })\le \epsilon$, respectively. It is clear to see from Theorem 29 that ${\widehat{g}}_0(X;Y)=g_0(X;Y)=0$ and ${\widehat{g}}_{{\rho }^2(X;Y)}(X;Y)=\infty$. However, although we showed that $g_{\epsilon }(X;Y)$ is indeed the asymptotic approximation of $g_{\epsilon ,M}(X;Y)$ for M large enough, it is not clear that the same statement holds for ${\widehat{g}}_{\epsilon }(X;Y)$ and ${\widehat{g}}_{\epsilon ,M}(X;Y)$.

5.2. Gaussian Information

The rate-privacy function for Gaussian Y has an interesting interpretation from an estimation theoretic point of view. Given the private and observable data $(X,Y)$, suppose an agent is required to estimateY based on the output of the privacy filter. We wish to know the effect of imposing a privacy constraint on the estimation performance.

The following lemma shows that $g_{\epsilon }(X;Y)$ bounds the best performance of the predictability of Y given the output of the privacy filter. The proof provided for this lemma does not use the Gaussianity of the noise process, so it holds for any noise process.

Lemma 30. 

For any given private data X and Gaussian observable data Y, we have for any $\epsilon \ge 0$

$$\underset{\begin{array}{c}\gamma \ge 0,\\ I(X;Z_{\gamma })\le \epsilon \end{array}}{inf}\mathsf{mmse}\left(Y\right|Z_{\gamma })\ge \mathsf{var}\left(Y\right)2^{-2g_{\epsilon }(X;Y)}$$

Proof. 

It is a well-known fact from rate-distortion theory that for a Gaussian Y and its reconstruction $\widehat{Y}$

$$I(Y;\widehat{Y})\ge \frac{1}{2}log\frac{\mathsf{var}\left(Y\right)}{\mathbb{E}[{(Y-\widehat{Y})}^2]}$$

and hence by setting $\widehat{Y}=\mathbb{E}\left[Y\right|Z_{\gamma }]$, where $Z_{\gamma }$ is an output of a privacy filter, and noting that $I(Y;\widehat{Y})\le I(Y;Z_{\gamma })$, we obtain

$$\mathsf{mmse}\left(Y\right|Z_{\gamma })\ge \mathsf{var}\left(Y\right)2^{-2I(Y;Z_{\gamma })}$$

(45)

from which the result follows immediately. ☐

According to Lemma 30, the quantity ${\lambda }_{\epsilon }\left(X\right):=2^{-2g_{\epsilon }(X;Y)}$ is a parameter that bounds the difficulty of estimating Gaussian Y when observing an additive perturbation Z with privacy constraint $I(X;Z)\le \epsilon$. Note that $0<{\lambda }_{\epsilon }\left(X\right)\le 1$, and therefore, provided that the privacy threshold is not trivial (i.e, $\epsilon <I(X;Y)$), the mean squared error of estimating Y given the privacy filter output is bounded away from zero, however the bound decays exponentially at rate of $g_{\epsilon }(X;Y)$.

To finish this section, assume that X and Y are jointly Gaussian with correlation coefficient ρ. The value of $g_{\epsilon }(X;Y)$ can be easily obtained in closed form as demonstrated in the following theorem.

Theorem 31. 

Let $(X,Y)$ be jointly Gaussian random variables with correlation coefficient ρ. For any $\epsilon \in [0,I(X;Y\left)\right)$ we have

$$g_{\epsilon }(X;Y)=\frac{1}{2}log\left(\frac{{\rho }^2}{2^{-2\epsilon }+{\rho }^2-1}\right)$$

Proof. 

One can always write $Y=aX+N_1$ where $a^2={\rho }^2\frac{\mathsf{var}\left(Y\right)}{\mathsf{var}\left(X\right)}$ and $N_1$ is a Gaussian random variable with mean 0 and variance ${\sigma }^2=(1-{\rho }^2)\mathsf{var}\left(Y\right)$ which is independent of $(X,Y)$. On the other hand, we have $Z_{\gamma }=Y+\gamma N$ where N is the standard Gaussian random variable independent of $(X,Y)$ and hence $Z_{\gamma }=aX+N_1+\gamma N$. In order for this additive channel to be a privacy filter, it must satisfy

$$I(X;Z_{\gamma })\le \epsilon$$

which implies

$$\frac{1}{2}log\left(\frac{\mathsf{var}\left(Y\right)+{\gamma }^2}{{\sigma }^2+{\gamma }^2}\right)\le \epsilon$$

and hence

$${\gamma }^2\ge \frac{2^{-2\epsilon }+{\rho }^2-1}{1-2^{-2\epsilon }}\mathsf{var}\left(Y\right)=:{\gamma }^{*}$$

Since $\gamma \mapsto I(Y;Z_{\gamma })$ is strictly decreasing (cf., Appendix C), we obtain

$$\begin{array}{ccc}\hfill g_{\epsilon }(X;Y)& =& I(Y;Z_{{\gamma }^{*}})=\frac{1}{2}log\left(1+\frac{\mathsf{var}\left(Y\right)}{{\gamma }^2}\right)\hfill \\ & =& \frac{1}{2}log\left(1+\frac{1-2^{-2\epsilon }}{2^{-2\epsilon }+{\rho }^2-1}\right)\hspace{1em}☐\hfill \end{array}$$

(46)

According to (46), we conclude that the optimal privacy filter for jointly Gaussian $(X,Y)$ is an additive Gaussian channel with signal to noise ratio ${\displaystyle \frac{1-2^{-2\epsilon }}{2^{-2\epsilon }+{\rho }^2-1}}$, which shows that if perfect privacy is required, then the displayed data is independent of the observable data Y, i.e., $g_0(X;Y)=0$.

Remark 7. 

We could assume that the privacy filter adds non-Gaussian noise to the observable data and define the rate-privacy function accordingly. To this end, we define

$$g_{\epsilon }^f(X;Y):=\underset{\genfrac{}{}{0pt}{}{\gamma \ge 0,}{I(X;Z_{\gamma }^f)}}{sup}I(Y;Z_{\gamma }^f)$$

where $Z_{\gamma }^f=Y+\gamma M_f$ and $M_f$ is a noise process that has stable distribution with density f and is independent of $(X,Y)$. In this case, we can use a technique similar to Oohama [49] to lower bound $g_{\epsilon }^f(X;Y)$ for jointly Gaussian $(X,Y)$. Since X and Y are jointly Gaussian, we can write $X=aY+bN$ where $a^2={\rho }^2\frac{\mathsf{var}\left(X\right)}{\mathsf{var}\left(Y\right)}$, $b=\sqrt{(1-{\rho }^2)\mathsf{var}X}$, and N is a standard Gaussian random variable that is independent of Y. We can apply the conditional entropy power inequality (cf., [42, Page 22]) for a random variable Z that is independent of N, to obtain

$$2^{2h\left(X\right|Z)}\ge 2^{2h\left(aY\right|Z)}+2^{2h\left(N\right)}=a^2{2}^{2h\left(Y\right|Z)}+2\pi e(1-{\rho }^2)\mathsf{var}\left(X\right)$$

and hence

$$2^{-2I(X;Z)}{2}^{2h\left(X\right)}\ge a^2{2}^{2h\left(Y\right)}{2}^{-2I(Y;Z)}+2\pi e(1-{\rho }^2)\mathsf{var}\left(X\right)$$

Assuming $Z=Z_{\gamma }^f$ and taking infimum from both sides of above inequality over γ such that $I(X;Z_{\gamma }^f)\le \epsilon$, we obtain

$$g_{\epsilon }^f(X;Y)\ge \frac{1}{2}log\left(\frac{{\rho }^2}{2^{-2\epsilon }+{\rho }^2-1}\right)=g_{\epsilon }(X;Y)$$

which shows that for Gaussian $(X,Y)$, Gaussian noise is the worst stable additive noise in the sense of privacy-constrained information extraction.

We can also calculate ${\widehat{g}}_{\epsilon }(X;Y)$ for jointly Gaussian $(X,Y)$.

Theorem 32. 

Let $(X,Y)$ be jointly Gaussian random variables with correlation coefficient ρ. For any $\epsilon \in [0,{\rho }^2)$ we have that

$${\widehat{g}}_{\epsilon }(X;Y)=\frac{1}{2}log\left(\frac{{\rho }^2}{{\rho }^2-\epsilon }\right)$$

Proof. 

Since for the correlation coefficient between Y and $Z_{\gamma }$ we have for any $\gamma \ge 0$,

$${\rho }^2(Y;Z_{\gamma })=\frac{\mathsf{var}\left(Y\right)}{\mathsf{var}\left(Y\right)+{\gamma }^2}$$

we can conclude that

$${\rho }^2(X;Z_{\gamma })=\frac{{\rho }^2\mathsf{var}\left(Y\right)}{\mathsf{var}\left(Y\right)+{\gamma }^2}$$

Since ${\rho }_m^2(X;Z)={\rho }^2(X;Z)$ (see, e.g., [34]), the privacy constraint ${\rho }_m^2(X;Z)\le \epsilon$ implies that

$$\frac{{\rho }^2\mathsf{var}\left(Y\right)}{\mathsf{var}\left(Y\right)+{\gamma }^2}\le \epsilon$$

and hence

$${\gamma }^2\ge \frac{({\rho }^2-\epsilon )\mathsf{var}\left(Y\right)}{\epsilon }=:{\widehat{\gamma }}_{\epsilon }^2$$

By monotonicity of the map $\gamma \mapsto I(Y;Z_{\gamma })$, we have

$${\widehat{g}}_{\epsilon }(X;Y)=I(Y;Z_{{\widehat{\gamma }}_{\epsilon }})=\frac{1}{2}log\left(1+\frac{\mathsf{var}\left(Y\right)}{{\widehat{\gamma }}_{\epsilon }^2}\right)=\frac{1}{2}log\left(\frac{{\rho }^2}{{\rho }^2-\epsilon }\right)\hspace{1em}☐$$

Theorems 31 and 32 show that unlike to the discrete case (cf. Lemmas 2 and 8), $\epsilon \mapsto g_{\epsilon }(X;Y)$ and $\epsilon \mapsto {\widehat{g}}_{\epsilon }(X;Y)$ are convex.

6. Conclusions

In this paper, we studied the problem of determining the maximal amount of information that one can extract by observing a random variable Y, which is correlated with another random variable X that represents sensitive or private data, while ensuring that the extracted data Z meets a privacy constraint with respect to X. Specifically, given two correlated discrete random variables X and Y, we introduced the rate-privacy function as the maximization of $I(Y;Z)$ over all stochastic ”privacy filters” $P_{Z|Y}$ such that $pm(X;Z)\le ϵ$, where $pm(·;·)$ is a privacy measure and $ϵ\ge 0$ is a given privacy threshold. We considered two possible privacy measure functions, $pm(X;Z)=I(X;Z)$ and $pm(X;Z)={\rho }_m^2(X;Z)$ where ${\rho }_m$ denotes maximal correlation, resulting in the rate-privacy functions $g_{ϵ}(X;Y)$ and ${\widehat{g}}_{ϵ}(X;Y)$, respectively. We analyzed these two functions, noting that each function lies between easily evaluated upper and lower bounds, and derived their monotonicity and concavity properties. We next provided an information-theoretic interpretation for $g_{ϵ}(X;Y)$ and an estimation-theoretic characterization for ${\widehat{g}}_{ϵ}(X;Y)$. In particular, we demonstrated that the dual function of $g_{ϵ}(X;Y)$ is a corner point of an outer bound on the achievable region of the dependence dilution coding problem. We also showed that ${\widehat{g}}_{ϵ}(X;Y)$ constitutes the largest amount of information that can be extracted from Y such that no meaningful MMSE estimation of any function of X can be realized by just observing the extracted information Z. We then examined conditions on $P_{XY}$ under which the lower bound on $g_{ϵ}(X;Y)$ is tight, hence determining the exact value of $g_{ϵ}(X;Y)$. We also showed that for any given Y, if the observation channel $P_{Y|X}$ is an erasure channel, then $g_{ϵ}(X;Y)$ attains its upper bound. Finally, we extended the notions of the rate-privacy functions $g_{ϵ}(X;Y)$ and ${\widehat{g}}_{ϵ}(X;Y)$ to the continuous case where the observation channel consists of an additive Gaussian noise channel followed by uniform scalar quantization.