Next Article in Journal
Throughput Capacity of Selfish Wireless Ad Hoc Networks with General Node Density
Next Article in Special Issue
Communication-Theoretic Model of Power Talk for a Single-Bus DC Microgrid
Previous Article in Journal
CoSpa: A Co-training Approach for Spam Review Identification with Support Vector Machine
Previous Article in Special Issue
Ultra-Reliable Link Adaptation for Downlink MISO Transmission in 5G Cellular Networks
Article Menu

Export Article

Information 2016, 7(1), 15; doi:10.3390/info7010015

Article
Information Extraction Under Privacy Constraints
Department of Mathematics and Statistics, Queen’s University, Kingston, Canada
Parts of the results in this paper were presented at the 52nd Allerton Conference on Communications, Control and Computing [1] and the 14th Canadian Workshop on Information Theory [2].
*
Author to whom correspondence should be addressed.
Academic Editors: Mikael Skoglund, Lars K. Rasmussen and Tobias Oechtering
Received: 1 November 2015 / Accepted: 3 March 2016 / Published: 10 March 2016

Abstract

:
A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables ( X , Y ) governed by a given joint distribution, an agent observes Y and wants to convey to a potentially public user as much information about Y as possible while limiting the amount of information revealed about X. To this end, the so-called rate-privacy function is investigated to quantify the maximal amount of information (measured in terms of mutual information) that can be extracted from Y under a privacy constraint between X and the extracted information, where privacy is measured using either mutual information or maximal correlation. Properties of the rate-privacy function are analyzed and its information-theoretic and estimation-theoretic interpretations are presented for both the mutual information and maximal correlation privacy measures. It is also shown that the rate-privacy function admits a closed-form expression for a large family of joint distributions of ( X , Y ) . Finally, the rate-privacy function under the mutual information privacy measure is considered for the case where ( X , Y ) has a joint probability density function by studying the problem where the extracted information is a uniform quantization of Y corrupted by additive Gaussian noise. The asymptotic behavior of the rate-privacy function is studied as the quantization resolution grows without bound and it is observed that not all of the properties of the rate-privacy function carry over from the discrete to the continuous case.
Keywords:
data privacy; equivocation; rate-privacy function; information theory; minimum mean-squared error estimation; additive channels; mutual information; maximal correlation

1. Introduction

With the emergence of user-customized services, there is an increasing desire to balance between the need to share data and the need to protect sensitive and private information. For example, individuals who join a social network are asked to provide information about themselves which might compromise their privacy. However, they agree to do so, to some extent, in order to benefit from the customized services such as recommendations and personalized searches. As another example, a participatory technology for estimating road traffic requires each individual to provide her start and destination points as well as the travel time. However, most participating individuals prefer to provide somewhat distorted or false information to protect their privacy. Furthermore, suppose a software company wants to gather statistical information on how people use its software. Since many users might have used the software to handle some personal or sensitive information -for example, a browser for anonymous web surfing or a financial management software- they may not want to share their data with the company. On the other hand, the company cannot legally collect the raw data either, so it needs to entice its users. In all these situations, a tradeoff in a conflict between utility advantage and privacy breach is required and the question is how to achieve this tradeoff. For example, how can a company collect high-quality aggregate information about users while strongly guaranteeing to its users that it is not storing user-specific information?
To deal with such privacy considerations, Warner [3] proposed the randomized response model in which each individual user randomizes her own data using a local randomizer (i.e., a noisy channel) before sharing the data to an untrusted data collector to be aggregated. As opposed to conditional security, see, e.g., [4,5,6], the randomized response model assumes that the adversary can have unlimited computational power and thus it provides unconditional privacy. This model, in which the control of private data remains in the users’ hands, has been extensively studied since Warner. As a special case of the randomized response model, Duchi et al. [7], inspired by the well-known privacy guarantee called differential privacy introduced by Dwork et al. [8,9,10], introduced locally differential privacy (LDP). Given a random variable X X , another random variable Z Z is said to be the ε-LDP version of X if there exists a channel Q : X Z such that Q ( B | x ) Q ( B | x ) exp ( ε ) for all measurable B Z and all x , x X . The channel Q is then called as the ε-LDP mechanism. Using Jensen’s inequality, it is straightforward to see that any ε-LDP mechanism leaks at most ε bits of private information, i.e., the mutual information between X and Z satisfies I ( X , Z ) ε .
There have been numerous studies on the tradeoff between privacy and utility for different examples of randomized response models with different choices of utility and privacy measures. For instance, Duchi et al. [7] studied the optimal ε-LDP mechanism M : X Z which minimizes the risk of estimation of a parameter θ related to P X . Kairouz et al. [11] studied an optimal ε-LDP mechanism in the sense of mutual information, where an individual would like to release an ε-LDP version Z of X that preserves as much information about X as possible. Calmon et al. [12] proposed a novel privacy measure (which includes maximal correlation and chi-square correlation) between X and Z and studied the optimal privacy mechanism (according to their privacy measure) which minimizes the error probability Pr ( X ^ ( Z ) X ) for any estimator X ^ : Z X .
In all above examples of randomized response models, given a private source, denoted by X, the mechanism generates Z which can be publicly displayed without breaching the desired privacy level. However, in a more realistic model of privacy, we can assume that for any given private data X, nature generates Y, via a fixed channel P Y | X . Now we aim to release a public display Z of Y such that the amount of information in Y is preserved as much as possible while Z satisfies a privacy constraint with respect to X. Consider two communicating agents Alice and Bob. Alice collects all her measurements from an observation into a random variable Y and ultimately wants to reveal this information to Bob in order to receive a payoff. However, she is worried about her private data, represented by X, which is correlated with Y. For instance, X might represent her precise location and Y represents measurement of traffic load of a route she has taken. She wants to reveal these measurements to an online road monitoring system to received some utility. However, she does not want to reveal too much information about her exact location. In such situations, the utility is measured with respect to Y and privacy is measured with respect to X. The question raised in this situation then concerns the maximum payoff Alice can get from Bob (by revealing Z to him) without compromising her privacy. Hence, it is of interest to characterize such competing objectives in the form of a quantitative tradeoff. Such a characterization provides a controllable balance between utility and privacy.
This model of privacy first appears in Yamamoto’s work [13] in which the rate-distortion-equivocation function is defined as the tradeoff between a distortion-based utility and privacy. Recently, Sankar et al. [14], using the quantize-and-bin scheme [15], generalized Yamamoto’s model to study privacy in databases from an information-theoretic point of view. Calmon and Fawaz [16] and Monedero et al. [17] also independently used distortion and mutual information for utility and privacy, respectively, to define a privacy-distortion function which resembles the classical rate-distortion function. More recently, Makhdoumi et al. [18] proposed to use mutual information for both utility and privacy measures and defined the privacy funnel as the corresponding privacy-utility tradeoff, given by
t R ( X ; Y ) : = min P Z | Y : X -∘- Y -∘- Z I ( Y ; Z ) R I ( X ; Z )
where X -∘- Y -∘- Z denotes that X , Y and Z form a Markov chain in this order. Leveraging well-known algorithms for the information bottleneck problem [19], they provided a locally optimal greedy algorithm to evaluate t R ( X ; Y ) . Asoodeh et al. [1], independently, defined the rate-privacy function, g ε ( X ; Y ) , as the maximum achievable I ( Y ; Z ) such that Z satisfies I ( X ; Z ) ε , which is a dual representation of the privacy funnel (1), and showed that for discrete X and Y, g 0 ( X ; Y ) > 0 if and only if X is weakly independent of Y (cf, Definition 9). Recently, Calmon et al. [20] proved an equivalent result for t R ( X ; Y ) using a different approach. They also obtained lower and upper bounds for t R ( X ; Y ) which can be easily translated to bounds for g ε ( X ; Y ) (cf. Lemma 1). In this paper, we develop further properties of g ε ( X ; Y ) and also determine necessary and sufficient conditions on P X Y , satisfying some symmetry conditions, for g ε ( X ; Y ) to achieve its upper and lower bounds.
The problem treated in this paper can also be contrasted with the better-studied concept of secrecy following the pioneering work of Wyner [21]. While in secrecy problems the aim is to keep information secret only from wiretappers, in privacy problems the aim is to keep the private information (not necessarily all the information) secret from everyone including the intended receiver.

1.1. Our Model and Main Contributions

Using mutual information as measure of both utility and privacy, we formulate the corresponding privacy-utility tradeoff for discrete random variables X and Y via the rate-privacy function, g ε ( X ; Y ) , in which the mutual information between Y and displayed data (i.e., the mechanism’s output), Z, is maximized over all channels P Z | Y such that the mutual information between Z and X is no larger than a given ε. We also formulate a similar rate-privacy function g ^ ε ( X ; Y ) where the privacy is measured in terms of the squared maximal correlation, ρ m 2 , between, X and Z. In studying g ε ( X ; Y ) and g ^ ε ( X ; Y ) , any channel Q : Y Z that satisfies I ( X ; Z ) ε and ρ m 2 ( X ; Z ) ε , preserves the desired level of privacy and is hence called a privacy filter. Interpreting I ( Y ; Z ) as the number of bits that a privacy filter can reveal about Y without compromising privacy, we present the rate-privacy function as a formulation of the problem of maximal privacy-constrained information extraction from Y.
We remark that using maximal correlation as a privacy measure is by no means new as it appears in other works, see, e.g., [22,23] and [12] for different utility functions. We do not put any likelihood constraints on the privacy filters as opposed to the definition of LDP. In fact, the optimal privacy filters that we obtain in this work induce channels PZ|X that do not satisfy the LDP property.
The quantity g ε ( X ; Y ) is related to a notion of the reverse strong data processing inequality as follows. Given a joint distribution P X Y , the strong data processing coefficient was introduced in [24,25], as the smallest s ( X ; Y ) 1 such that I ( X ; Z ) s ( X ; Y ) I ( Y ; Z ) for all P Z | Y satisfying the Markov condition X - -∘- Y -∘- Z . In the rate-privacy function, we instead seek an upper bound on the maximum achievable rate at which Y can display information, I ( Y ; Z ) , while meeting the privacy constraint I ( X ; Z ) ε . The connection between the rate-privacy function and the strong data processing inequality is further studied in [20] to mirror all the results of [25] in the context of privacy.
The contributions of this work are as follows:
  • We study lower and upper bounds of g ε ( X ; Y ) . The lower bound, in particular, establishes a multiplicative bound on I ( Y ; Z ) for any optimal privacy filter. Specifically, we show that for a given ( X , Y ) and ε > 0 there exists a channel Q : Y Z such that I ( X ; Z ) ε and
    I ( Y ; Z ) λ ( X ; Y ) ε
    where λ ( X ; Y ) 1 is a constant depending on the joint distribution P X Y . We then give conditions on P X Y such that the upper and lower bounds are tight. For example, we show that the lower bound is achieved when Y is binary and the channel from Y to X is symmetric. We show that this corresponds to the fact that both Y = 0 and Y = 1 induce distributions P X | Y ( · | 0 ) and P X | Y ( · | 1 ) which are equidistant from P X in the sense of Kullback-Leibler divergence. We then show that the upper bound is achieved when Y is an erased version of X, or equivalently, P Y | X is an erasure channel.
  • We propose an information-theoretic setting in which g ε ( X ; Y ) appears as a natural upper-bound for the achievable rate in the so-called "dependence dilution" coding problem. Specifically, we examine the joint-encoder version of an amplification-masking tradeoff, a setting recently introduced by Courtade [26] and we show that the dual of g ε ( X ; Y ) upper bounds the masking rate. We also present an estimation-theoretic motivation for the privacy measure ρ m 2 ( X ; Z ) ε . In fact, by imposing ρ m 2 ( X ; Y ) ε , we require that an adversary who observes Z cannot efficiently estimate f ( X ) , for any function f. This is reminiscent of semantic security [27] in the cryptography community. An encryption mechanism is said to be semantically secure if the adversary’s advantage for correctly guessing any function of the privata data given an observation of the mechanism’s output (i.e., the ciphertext) is required to be negligible. This, in fact, justifies the use of maximal correlation as a measure of privacy. The use of mutual information as privacy measure can also be justified using Fano’s inequality. Note that I ( X ; Z ) ε can be shown to imply that Pr ( X ^ ( Z ) X ) H ( X ) - 1 - ε log ( | X | ) and hence the probability of adversary correctly guessing X is lower-bounded.
  • We also study the rate of increase g 0 ( X ; Y ) of g ε ( X ; Y ) at ε = 0 and show that this rate can characterize the behavior of g ε ( X ; Y ) for any ε 0 provided that g 0 ( X ; Y ) = 0 . This again has connections with the results of [25]. Letting
    Γ ( R ) : = max P Z | Y : X -∘- Y -∘- Z I ( Y ; Z ) R I ( X ; Z )
    one can easily show that Γ ( 0 ) = lim R 0 Γ ( R ) R = s ( X ; Y ) , and hence the rate of increase of Γ ( R ) at R = 0 characterizes the strong data processing coefficient. Note that here we have Γ ( 0 ) = 0 .
  • Finally, we generalize the rate-privacy function to the continuous case where X and Y are both continuous and show that some of the properties of g ε ( X ; Y ) in the discrete case do not carry over to the continuous case. In particular, we assume that the privacy filter belongs to a family of additive noise channels followed by an M-level uniform scalar quantizer and give asymptotic bounds as M for the rate-privacy function.

1.2. Organization

The rest of the paper is organized as follows. In Section 2, we define and study the rate-privacy function for discrete random variables for two different privacy measures, which, respectively, lead to the information-theoretic and estimation-theoretic interpretations of the rate-privacy function. In Section 3, we provide such interpretations for the rate-privacy function in terms of quantities from information and estimation theory. Having obtained lower and upper bounds of the rate-privacy function, in Section 4 we determine the conditions on P X Y such that these bounds are tight. The rate-privacy function is then generalized and studied in Section 5 for continuous random variables.

2. Utility-Privacy Measures: Definitions and Properties

Consider two random variables X and Y, defined over finite alphabets X and Y, respectively, with a fixed joint distribution P X Y . Let X represent the private data and let Y be the observable data, correlated with X and generated by the channel P Y | X predefined by nature, which we call the observation channel. Suppose there exists a channel P Z | Y such that Z, the displayed data made available to public users, has limited dependence with X. Such a channel is called the privacy filter. This setup is shown in Figure 1. The objective is then to find a privacy filter which gives rise to the highest dependence between Y and Z. To make this goal precise, one needs to specify a measure for both utility (dependence between Y and Z) and also privacy (dependence between X and Z).

2.1. Mutual Information as Privacy Measure

Adopting mutual information as a measure of both privacy and utility, we are interested in characterizing the following quantity, which we call the rate-privacy function (since mutual information is adopted for utility, the privacy-utility tradeoff characterizes the optimal rate for a given privacy level, where rate indicates the precision of the displayed data Z with respect to the observable data Y for a privacy filter, which suggests the name),
g ε ( X ; Y ) : = sup P Z | Y D ε ( P ) I ( Y ; Z )
where ( X , Y ) has fixed distribution P X Y = P and
D ε ( P ) : = { P Z | Y : X -∘- Y -∘- Z , I ( X ; Z ) ε }
(here X -∘- Y -∘- Z means that X , Y , and Z form a Markov chain in this order). Equivalently, we call g ε ( X ; Y ) the privacy-constrained information extraction function, as Z can be thought of as the extracted information from Y under privacy constraint I ( X ; Z ) ε .
Note that since I ( Y ; Z ) is a convex function of P Z | Y and furthermore the constraint set D ε ( P ) is convex, [28, Theorem 32.2] implies that we can restrict D ε ( P ) in (3) to { P Z | Y : X -∘- Y -∘- Z ,I(X;Z)=ε} whenever ε I ( X ; Y ) . Note also that since for finite X and Y, P Z | Y I ( Y ; Z ) is a continuous map, therefore D ε ( P ) is compact and the supremum in (3) is indeed a maximum. In this case, using the Support Lemma [29], one can readily show that it suffices that the random variable Z is supported on an alphabet Z with cardinality | Z | | Y | + 1 . Note further that by the Markov condition X -∘- Y -∘- Z, we can always restrict ε 0 to only 0 ε < I ( X ; Y ) , because I ( X ; Z ) I ( X ; Y ) and hence for ε I ( X ; Y ) the privacy constraint is removed and thus by setting Z = Y , we obtain g ε ( X ; Y ) = H ( Y ) .
As mentioned earlier, a dual representation of g ε ( X ; Y ) , the so called privacy funnel, is introduced in [18,20], defined in (1), as the least information leakage about X such that the communication rate is greater than a positive constant; I ( Y ; Z ) R for some R > 0 . Note that if t R ( X ; Y ) = ε then g ε ( X ; Y ) = R .
Given ε 1 < ε 2 and a joint distribution P = P X × P Y | X , we have D ε 1 ( P ) D ε 2 ( P ) and hence ε g ε ( X ; Y ) is non-decreasing, i.e., g ε 1 ( X ; Y ) g ε 2 ( X ; Y ) . Using a similar technique as in [30, Lemma 1], Calmon et al. [20] showed that the mapping R t R ( X ; Y ) R is non-decreasing for R > 0 . This, in fact, implies that ε g ε ( X ; Y ) ε is non-increasing for ε > 0 . This observation leads to a lower bound for the rate privacy function g ε ( X ; Y ) as described in the following lemma.
Lemma 1 
([20]). For a given joint distribution P defined over X × Y , the mapping ε g ε ( X ; Y ) ε is non-increasing on ε ( 0 , ) and g ε ( X ; Y ) lies between two straight lines as follows:
ε H ( Y ) I ( X ; Y ) g ε ( X ; Y ) H ( Y | X ) + ε
for ε ( 0 , I ( X ; Y ) ) .
Using a simple calculation, the lower bound in (4) can be shown to be achieved by the privacy filter depicted in Figure 2 with the erasure probability
δ = 1 - ε I ( X ; Y )
In light of Lemma 1, the possible range of the map ε g ε ( X ; Y ) is as depicted in Figure 3.
We next show that ε g ε ( X ; Y ) is concave and continuous.
Lemma 2. 
For any given pair of random variables ( X , Y ) over X × Y , the mapping ε g ε ( X ; Y ) is concave for ε 0 .
Proof. 
It suffices to show that for any 0 ε 1 < ε 2 < ε 3 I ( X ; Y ) , we have
g ε 3 ( X ; Y ) - g ε 1 ( X ; Y ) ε 3 - ε 1 g ε 2 ( X ; Y ) - g ε 1 ( X ; Y ) ε 2 - ε 1
which, in turn, is equivalent to
ε 2 - ε 1 ε 3 - ε 1 g ε 3 ( X ; Y ) + ε 3 - ε 2 ε 3 - ε 1 g ε 1 ( X ; Y ) g ε 2 ( X ; Y )
Let P Z 1 | Y : Y Z 1 and P Z 3 | Y : Y Z 3 be two optimal privacy filters in D ε 1 ( P ) and D ε 3 ( P ) with disjoint output alphabets Z 1 and Z 3 , respectively.
We introduce an auxiliary binary random variable U Bernoulli ( λ ) , independent of ( X , Y ) , where λ : = ε 2 - ε 1 ε 3 - ε 1 and define the following random privacy filter P Z λ | Y : We pick P Z 3 | Y if U = 1 and P Z 1 | Y if U = 0 , and let Z λ be the output of this random channel which takes values in Z 1 Z 3 . Note that ( X , Y ) -∘-Z-∘-U. Then we have
I ( X ; Z λ ) = I ( X ; Z λ , U ) = I ( X ; Z λ | U ) = λ I ( X ; Z 3 ) + ( 1 - λ ) I ( X ; Z 1 ) , ε 2
which implies that P Z λ | Y D ε 2 ( P ) . On the other hand, we have
g ε 2 ( X ; Y ) I ( Y ; Z λ ) = I ( Y ; Z λ , U ) = I ( Y ; Z λ | U ) = λ I ( Y ; Z 3 ) + ( 1 - λ ) I ( Y ; Z 1 ) = ε 2 - ε 1 ε 3 - ε 1 g ε 3 ( X ; Y ) + ε 3 - ε 2 ε 3 - ε 1 g ε 1 ( X ; Y )
which, according to (7), completes the proof. ☐
Remark 1. 
By the concavity of ε g ε ( X ; Y ) , we can show that g ε ( X ; Y ) is a strictly increasing function of ε I ( X ; Y ) . To see this, assume there exists ε 1 < ε 2 I ( X ; Y ) such that g ε 1 ( X ; Y ) = g ε 2 ( X ; Y ) . Since ε g ε ( X ; Y ) is concave, then it follows that for all ε ε 2 , g ε ( X ; Y ) = g ε 2 ( X ; Y ) and since for ε = I ( X ; Y ) , g I ( X ; Y ) ( X ; Y ) = H ( Y ) , implying that for any ε ε 2 , we must have g ε ( X ; Y ) = H ( Y ) which contradicts the upper bound shown in (4).
Corollary 3. 
For any given pair of random variables ( X , Y ) over X × Y , the mapping ε g ε ( X ; Y ) is continuous for ε 0 .
Proof. 
Concavity directly implies that the mapping ε g ε ( X ; Y ) is continuous on ( 0 , ) (see for example [31, Theorem 3.2]). Continuity at zero follows from the continuity of mutual information. ☐
Remark 2. 
Using the concavity of the map ε g ε ( X ; Y ) , we can provide an alternative proof for the lower bound in (4). Note that point ( I ( X ; Y ) , H ( Y ) ) is always on the curve g ε ( X ; Y ) , and hence by concavity, the straight line ε ε H ( Y ) I ( X ; Y ) is always below the lower convex envelop of g ε ( X ; Y ) , i.e., the chord connecting ( 0 , g 0 ( X ; Y ) ) to ( I ( X ; Y ) , H ( Y ) ) , and hence g ε ( X ; Y ) ε H ( Y ) I ( X ; Y ) . In fact, this chord yields a better lower bound for g ε ( X ; Y ) on ε [ 0 , I ( X ; Y ] as
g ε ( X ; Y ) ε H ( Y ) I ( X ; Y ) + g 0 ( X ; Y ) 1 - ε I ( X ; Y )
which reduces to the lower bound in (4) only if g 0 ( X ; Y ) = 0 .

2.2. Maximal Correlation as Privacy Measure

By adopting the mutual information as the privacy measure between the private and the displayed data, we make sure that only limited bits of private information is revealed during the process of transferring Y. In order to have an estimation theoretic guarantee of privacy, we propose alternatively to measure privacy using a measure of correlation, the so-called maximal correlation.
Given the collection Cof all pairs of random variables ( U , V ) U × V where U and V are general alphabets, a mapping T : C [ 0 , 1 ] defines a measure of correlation [32] if T ( U , V ) = 0 if and only if U and V are independent (in short, U V ) and T ( U , V ) attains its maximum value if X = f ( Y ) or Y = g ( X ) almost surely for some measurable real-valued functions f and g. There are many different examples of measures of correlation including the Hirschfeld-Gebelein-Rényi maximal correlation [32,33,34], the information measure [35], mutual information and f-divergence [36].
Definition 4 
([34]). Given random variables X and Y, the maximal correlation ρ m ( X ; Y ) is defined as follows (recall that the correlation coefficient between U and V, is defined as ρ ( U ; V ) : = cov ( U ; V ) σ U σ V , where cov ( U ; V ) , σ U and σ V are the covariance between U and V, the standard deviations of U and V, respectively):
ρ m ( X ; Y ) : = sup f , g ρ ( f ( X ) , g ( Y ) ) = sup ( f ( X ) , g ( Y ) ) S E [ f ( X ) g ( Y ) ]
where S is the collection of pairs of real-valued random variables f ( X ) and g ( Y ) such that E f ( X ) = E g ( Y ) = 0 and E f 2 ( X ) = E g 2 ( Y ) = 1 . If S is empty (which happens precisely when at least one of X and Y is constant almost surely) then one defines ρ m ( X ; Y ) to be 0. Rényi [34] derived an equivalent characterization of maximal correlation as follows:
ρ m 2 ( X ; Y ) = sup f : E f ( X ) = 0 , E f 2 ( X ) = 1 E E 2 [ f ( X ) | Y ] .
Measuring privacy in terms of maximal correlation, we propose
g ^ ε ( X ; Y ) : = sup P Z | Y D ^ ε ( P ) I ( Y ; Z )
as the corresponding rate-privacy tradeoff, where
D ^ ε ( P ) : = { P Z | Y : X -∘- Y -∘- Z , ρ m 2 ( X ; Z ) ε , P X Y = P }
Again, we equivalently call g^ε(X;Y) as the privacy-constrained information extraction function, where here the privacy is guaranteed by ρ m 2 ( X ; Z ) ε .
Setting ε = 0 corresponds to the case where X and Z are required to be statistically independent, i.e., absolutely no information leakage about the private source X is allowed. This is called perfect privacy. Since the independence of X and Z is equivalent to I ( X ; Z ) = ρ m ( X ; Z ) = 0 , we have g ^ 0 ( X ; Y ) = g 0 ( X ; Y ) . However, for ε > 0 , both g ε ( X ; Y ) g ^ ε ( X ; Y ) and g ε ( X ; Y ) g ^ ε ( X ; Y ) might happen in general. For general ε 0 , it directly follows using [23, Proposition 1] that
g ^ ε ( X ; Y ) g ε ( X ; Y )
where ε : = log ( k ε + 1 ) and k : = | X | - 1
Similar to gε(X;Y), we see that for ε 1 ε 2 , D ^ ε 1 ( P ) D ^ ε 2 ( P ) and hence ε g ^ ε ( X ; Y ) is non-decreasing. The following lemma is a counterpart of Lemma 1 for g ^ ε ( X ; Y ) .
Lemma 5. 
For a given joint distribution P X Y defined over X × Y , ε g ^ ε ( X ; Y ) ε is non-increasing on ( 0 , ) .
Proof. 
Like Lemma 1, the proof is similar to the proof of [30, Lemma 1]. We, however, give a brief proof for the sake of completeness.
For a given channel PZ|Y D ^ ε ( P ) and δ 0 , we can define a new channel with an additional symbol e as follows
P Z | Y ( z | y ) = ( 1 - δ ) P Z | Y ( z | y ) if z e δ if z = e
It is easy to check that I ( Y ; Z ) = ( 1 - δ ) I ( Y ; Z ) and also ρ m 2 ( X ; Z ) = ( 1 - δ ) ρ m 2 ( X ; Z ) ; see [37, Page 8], which implies that P Z | Y D ^ ε ( P ) where ε = ( 1 - δ ) ε . Now suppose that P Z | Y achieves g ^ ε ( X ; Y ) , that is, g ^ ε ( X ; Y ) = I ( Y ; Z ) and ρ m 2 ( X ; Z ) = ε . We can then write
g ^ ε ( X ; Y ) ε = I ( Y ; Z ) ε = I ( Y ; Z ) ε g ε ( X ; Y ) ε
Therefore, for ε ε we have g ε ( X ; Y ) ε g ε ( X ; Y ) ε . ☐
Similar to the lower bound for g ε ( X ; Y ) obtained from Lemma 1, we can obtain a lower bound for g ^ ε ( X ; Y ) using Lemma 5. Before we get to the lower bound, we need a data processing lemma for maximal correlation. The following lemma proves a version of strong data processing inequality for maximal correlation from which the typical data processing inequality follows, namely, ρ m ( X ; Z ) min { ρ m ( Y ; Z ) , ρ m ( X ; Y ) } for X , Y and Z satisfying X -∘- Y -∘- Z .
Lemma 6. 
For random variables X and Y with a joint distribution P X Y , we have
sup X -∘- Y -∘- Z ρ m ( Y ; Z ) 0 ρ m ( X ; Z ) ρ m ( Y ; Z ) = ρ m ( X ; Y )
Proof. 
For arbitrary zero-mean and unit variance measurable functions f L 2 ( X ) and g L 2 ( Z ) and X -∘- Y -∘- Z, we have
E [ f ( X ) g ( Z ) ] = E E [ f ( X ) | Y ] E [ g ( Z ) | Y ] ρ m ( X ; Y ) ρ m ( Y ; Z )
where the inequality follows from the Cauchy-Schwartz inequality and (9). Thus we obtain ρ m ( X ; Z ) ρ m ( X ; Y ) ρ m ( Y ; Z ) .
This bound is tight for the special case of X Y X , where P X | Y is the backward channel associated with P Y | X . In the following, we shall show that ρ m ( X ; Y ) ρ m ( Y ; X ) = ρ m ( X ; X ) .
To this end, first note that the above implies that ρ m ( X ; Y ) ρ m ( Y ; X ) ρ m ( X ; X ) . Since P X Y = P X Y , it follows that ρ m ( X ; Y ) = ρ m ( Y ; X ) and hence the above implies that ρ m 2 ( X ; Y ) ρ m ( X ; X ) . One the other hand, we have
E [ [ E [ f ( X ) | Y ] ] 2 ] = E [ E [ f ( X ) | Y ] E [ f ( X ) | Y ] ] = E [ E [ f ( X ) f ( X ) | Y ] ] = E [ f ( X ) f ( X ) ]
which together with (9) implies that
ρ m 2 ( X ; Y ) sup f : E f ( X ) = 0 , E f 2 ( X ) = 1 E [ f ( X ) f ( X ) ] ρ m ( X ; X )
Thus, ρ m 2 ( X ; Y ) = ρ m ( X ; X ) which completes the proof. ☐
Now a lower bound of g ^ ε ( X ; Y ) can be readily obtained.
Corollary 7. 
For a given joint distribution P X Y defined over X × Y , we have for any ε > 0
g ^ ε ( X ; Y ) H ( Y ) ρ m 2 ( X ; Y ) min { ε , ρ m 2 ( X ; Y ) }
Proof. 
By Lemma 6, we know that for any Markov chain X -∘- Y -∘- Z , we have ρ m ( X ; Z ) ρ m ( X ; Y ) and hence for ε ρ m 2 ( X ; Y ) , the privacy constraint ρ m 2 ( X ; Z ) ε is not restrictive and hence g ^ ε ( X ; Y ) = H ( Y ) by setting Y = Z . For 0 < ε ρ m 2 ( X ; Y ) , Lemma 5 implies that
g ^ ε ( X ; Y ) ε H ( Y ) ρ m 2 ( X ; Y )
from which the result follows. ☐
A loose upper bound of g ^ ε ( X ; Y ) can be obtained using an argument similar to the one used for g ε ( X ; Y ) . For the Markov chain X -∘- Y -∘- Z, we have
I ( Y ; Z ) = I ( X ; Z ) + I ( Y ; Z | X ) I ( X ; Z ) + H ( Y | X ) ( a ) log k ρ m 2 ( X ; Z ) + 1 + H ( Y | X )
where k : = | X | - 1 and ( a ) comes from [23, Proposition 1]. We can, therefore, conclude from (11) and Corollary 7 that
ε H ( Y ) ρ m 2 ( X ; Y ) g ^ ε ( X ; Y ) log k ε + 1 + H ( Y | X )
Similar to Lemma 2, the following lemma shows that the g ^ ε ( X ; Y ) is a concave function of ε.
Lemma 8. 
For any given pair of random variables ( X , Y ) with distribution P over X × Y , the mapping ε g ^ ε ( X ; Y ) is concave for ε 0 .
Proof. 
The proof is similar to that of Lemma 2 except that here for two optimal filters P Z 1 | Y : Y Z 1 and P Z 3 | Y : Y Z 3 in D ^ ε 1 ( P ) and D ^ ε 3 ( P ) , respectively, and the random channel P Z λ | Y : Y Z with output alphabet Z 1 Z 3 constructed using a coin flip with probability γ, we need to show that P Z λ | Y D ^ ε 2 ( P ) , where 0 ε 1 < ε 2 < ε 3 ρ m 2 ( X ; Y ) . To show this, consider f : X R such that E [ f ( X ) ] = 0 and E [ f 2 ( X ) ] = 1 and let U be a binary random variable as in the proof of Lemma 2. We then have
E [ E 2 [ f ( X ) | Z λ ] ] = E E [ E 2 [ f ( X ) | Z λ ] | U ] = ( a ) γ E [ E 2 [ f ( X ) | Z 3 ] ] + ( 1 - γ ) E [ E 2 [ f ( X ) | Z 1 ] ]
where ( a ) comes from the fact that U is independent of X. We can then conclude from (13) and the alternative characterization of maximal correlation (9) that
ρ m 2 ( X ; Z λ ) = sup f : E [ f ( X ) ] = 0 , E [ f 2 ( X ) ] = 1 E [ E 2 [ f ( X ) | Z λ ] ] = sup f : E [ f ( X ) ] = 0 , E [ f 2 ( X ) ] = 1 γ E [ E 2 [ f ( X ) | Z 3 ] ] + ( 1 - γ ) E [ E 2 [ f ( X ) | Z 1 ] ] γ ρ m 2 ( X ; Z 3 ) + ( 1 - γ ) ρ m 2 ( X ; Z 1 ) γ ε 3 + ( 1 - γ ) ε 1
from which we can conclude that P Z λ | Y D ^ ε 2 ( P ) . ☐

2.3. Non-Trivial Filters For Perfect Privacy

As it becomes clear later, requiring that g 0 ( X ; Y ) = 0 is a useful assumption for the analysis of g ε ( X ; Y ) . Thus, it is interesting to find a necessary and sufficient condition on the joint distribution P X Y which results in g 0 ( X ; Y ) = 0 .
Definition 9 
([38]). The random variable X is said to be weakly independent of Y if the rows of the transition matrix P X | Y , i.e., the set of vectors { P X | Y ( · | y ) , y Y } , are linearly dependent.
The following lemma provides a necessary and sufficient condition for g 0 ( X ; Y ) > 0 .
Lemma 10. 
For a given ( X , Y ) with a given joint distribution P X Y = P Y × P X | Y , g 0 ( X ; Y ) > 0 (and equivalently g ^ 0 ( X ; Y ) > 0 ) if and only if X is weakly independent of Y.
Proof. 
⇒ direction:
Assuming that g 0 ( X ; Y ) > 0 implies that there exists a random variable Z over an alphabet Z such that the Markov condition X -∘- Y -∘- Z is satisfied and Z X while I ( Y ; Z ) > 0 . Hence, for any z 1 and z 2 in Z, we must have P X | Z ( x | z 1 ) = P X | Z ( x | z 2 ) for all x X , which implies that
y Y P X | Y ( x | y ) P Y | Z ( y | z 1 ) = y Y P X | Y ( x | y ) P Y | Z ( y | z 2 )
and hence
y Y P X | Y ( x | y ) P Y | Z ( y | z 1 ) - P Y | Z ( y | z 2 ) = 0
Since Y is not independent of Z, there exist z 1 and z 2 such that P Y | Z ( y | z 1 ) P Y | Z ( y | z 2 ) and hence the above shows that the set of vectors P X | Y ( · | y ) , y Y is linearly dependent.
⇐ direction:
Berger and Yeung [38, Appendix II], in a completely different context, showed that if X being weakly independent of Y, one can always construct a binary random variable Z correlated with Y which satisfies X -∘- Y -∘- Z and X Z , and hence g 0 ( X ; Y ) > 0 . ☐
Remark 3. 
Lemma 10 first appeared in [1]. However, Calmon et al. [20] studied (1), the dual version of g ε ( X ; Y ) , and showed an equivalent result for t R ( X ; Y ) . In fact, they showed that for a given P X Y , one can always generate Z such that I ( X ; Z ) = 0 , I ( Y ; Z ) > 0 and X -∘- Y -∘- Z, or equivalently g 0 ( X ; Y ) > 0 , if and only if the smallest singular value of the conditional expectation operator f E [ f ( X ) | Y ] is zero. This condition can, in fact, be shown to be equivalent to the condition in Lemma 10.
Remark 4. 
It is clear that, according to Definition 9, X is weakly independent of Y if | Y | > | X | . Hence, Lemma 10 implies that g 0 ( X ; Y ) > 0 if Y has strictly larger alphabet than X.
In light of the above remark, in the most common case of |Y|=|X|, one might have g 0 ( X ; Y ) = 0 , which corresponds to the most conservative scenario as no privacy leakage implies no broadcasting of observable data. In such cases, the rate of increase of g ε ( X ; Y ) at ε = 0 , that is g 0 ( X ; Y ) : = d d ε g ε ( X ; Y ) | ε = 0 , which corresponds to the initial efficiency of privacy-constrained information extraction, proves to be very important in characterizing the behavior of g ε ( X ; Y ) for all ε 0 . This is because, for example, by concavity of ε g ε ( X ; Y ) , the slope of g ε ( X ; Y ) is maximized at ε = 0 and so
g 0 ( X ; Y ) = lim ε 0 g ε ( X ; Y ) ε = sup ε > 0 g ε ( X ; Y ) ε
and hence g ε ( X ; Y ) ε g 0 ( X ; Y ) for all ε I ( X ; Y ) which, together with (4), implies that g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) if g 0 ( X ; Y ) H ( Y ) I ( X ; Y ) . In the sequel, we always assume that X is not weakly independent of Y, or equivalently g 0 ( X ; Y ) = 0 . For example, in light of Lemma 10 and Remark 4, we can assume that | Y | | X | .
It is easy to show that, X is weakly independent of binary Y if and only if X and Y are independent (see, e.g., [38, Remark 2]). The following corollary, therefore, immediately follows from Lemma 10.
Corollary 11. 
Let Y be a non-degenerate binary random variable correlated with X. Then g 0 ( X ; Y ) = 0 .

3. Operational Interpretations of the Rate-Privacy Function

In this section, we provide a scenario in which g ε ( X ; Y ) appears as a boundary point of an achievable rate region and thus giving an information-theoretic operational interpretation for g ε ( X ; Y ) . We then proceed to present an estimation-theoretic motivation for g ^ ε ( X ; Y ) .

3.1. Dependence Dilution

Inspired by the problems of information amplification [39] and state masking [40], Courtade [26] proposed the information-masking tradeoff problem as follows. The tuple ( R u , R v , Δ A , Δ M ) R 4 is said to be achievable if for two given separated sources U U and V V and any ε > 0 there exist mappings f : U n { 1 , 2 , , 2 n R u } and g : V n { 1 , 2 , , 2 n R v } such that I ( U n ; f ( U n ) , g ( V n ) ) n ( Δ M + ε ) and I ( V n ; f ( U n ) , g ( V n ) ) n ( Δ A - ε ) . In other words, ( R u , R v , Δ A , Δ M ) is achievable if there exist indices K and J of rates R u and R v given U n and V n , respectively, such that the receiver in possession of ( K , J ) can recover at most n Δ M bits about U n and at least n Δ A about V n . The closure of the set of all achievable tuple ( R u , R v , Δ A , Δ M ) is characterized in [26]. Here, we look at a similar problem but for a joint encoder. In fact, we want to examine the achievable rate of an encoder observing both X n and Y n which masks X n and amplifies Y n at the same time, by rates Δ M and Δ A , respectively.
We define a ( 2 n R , n ) dependence dilution code by an encoder
f n : X n × Y n { 1 , 2 , , 2 n R }
and a list decoder
g n : { 1 , 2 , , 2 n R } 2 Y n
where 2 Y n denotes the power set of Y n . A dependence dilution triple ( R , Δ A , Δ M ) R + 3 is said to be achievable if, for any δ > 0 , there exists a ( 2 n R , n ) dependence dilution code that for sufficiently large n satisfies the utility constraint:
Pr Y n g n ( J ) < δ
having a fixed list size
| g n ( J ) | = 2 n ( H ( Y ) - Δ A ) , J { 1 , 2 , , 2 n R }
where J : = f n ( X n , Y n ) is the encoder’s output, and satisfies the privacy constraint:
1 n I ( X n ; J ) Δ M + δ
Intuitively speaking, upon receiving J, the decoder is required to construct list g n ( J ) Y n of fixed size which contains likely candidates of the actual sequence Y n . Without any observation, the decoder can only construct a list of size 2 n H ( Y ) which contains Y n with probability close to one. However, after J is observed and the list g n ( J ) is formed, the decoder’s list size can be reduced to 2 n ( H ( Y ) - Δ A ) and thus reducing the uncertainty about Y n by 0 n Δ A n H ( Y ) . This observation led Kim et al. [39] to show that the utility constraint (14) is equivalent to the amplification requirement
1 n I ( Y n ; J ) Δ A - δ
which lower bounds the amount of information J carries about Y n . The following lemma gives an outer bound for the achievable dependence dilution region.
Theorem 12. 
Any achievable dependence dilution triple ( R , Δ A , Δ M ) satisfies
R Δ A Δ A I ( Y ; U ) Δ M I ( X ; U ) - I ( Y ; U ) + Δ A
for some auxiliary random variable U U with a finite alphabet and jointly distributed with X and Y.
Before we prove this theorem, we need two preliminary lemmas. The first lemma is an extension of Fano’s inequality for list decoders and the second one makes use of a single-letterization technique to express I ( X n ; J ) - I ( Y n ; J ) in a single-letter form in the sense of Csiszár and Körner [29].
Lemma 13 
([39,41]). Given a pair of random variables ( U , V ) defined over U × V for finite V and arbitrary U, any list decoder g : U 2 V , U g ( U ) of fixed list size m (i.e., | g ( u ) | = m , u U ), satisfies
H ( V | U ) h b ( p e ) + p e log | V | + ( 1 - p e ) log m
where p e : = Pr ( V g ( U ) ) and h b : [ 0 , 1 ] [ 0 , 1 ] is the binary entropy function.
This lemma, applied to J and Y n in place of U and V, respectively, implies that for any list decoder with the property (14), we have
H ( Y n | J ) log | g n ( J ) | + n ε n
where ε n : = 1 n + ( log | Y | - 1 n log | g n ( J ) | ) p e and hence ε n 0 as n .
Lemma 14. 
Let ( X n , Y n ) be n i.i.d. copies of a pair of random variables ( X , Y ) . Then for a random variable J jointly distributed with ( X n , Y n ) , we have
I ( X n ; J ) - I ( Y n ; J ) = i = 1 n [ I ( X i ; U i ) - I ( Y i ; U i ) ]
where U i : = ( J , X i + 1 n , Y i - 1 ) .
Proof. 
Using the chain rule for the mutual information, we can express I ( X n ; J ) as follows
I ( X n ; J ) = i = 1 n I ( X i ; J | X i + 1 n ) = i = 1 n I ( X i ; J , X i + 1 n ) = i = 1 n [ I ( X i ; J , X i + 1 n , Y i - 1 ) - I ( X i ; Y i - 1 | J , X i + 1 n ) ] = i = 1 n I ( X i ; U i ) - i = 1 n I ( X i ; Y i - 1 | J , X i + 1 n )
Similarly, we can expand I ( Y n ; J ) as
I ( Y n ; J ) = i = 1 n I ( Y i ; J | Y i - 1 ) = i = 1 n I ( Y i ; J , Y i - 1 ) = i = 1 n [ I ( Y i ; J , X i + 1 n , Y i - 1 ) - I ( Y i ; X i + 1 n | J , Y i - 1 ) ] = i = 1 n I ( Y i ; U i ) - i = 1 n I ( Y i ; X i + 1 n | J , Y i - 1 )
Subtracting (20) from (19), we get
I ( X n ; J ) - I ( Y n ; J ) = i = 1 n [ I ( X i ; U i ) - I ( Y i ; U i ) ] - i = 1 n [ I ( X i ; Y i - 1 | J , X i + 1 n ) - I ( X i + 1 n ; Y i | J , Y i - 1 ) ] = ( a ) i = 1 n [ I ( X i ; U i ) - I ( Y i ; U i ) ]
where ( a ) follows from the Csiszár sum identity [42]. ☐
Proof of Theorem 12. 
The rate R can be bounded as
n R H ( J ) I ( Y n ; J ) = n H ( Y ) - H ( Y n | J ) ( a ) n H ( Y ) - log | g n ( J ) | - n ε n
= ( b ) n Δ A - n ε n
where ( a ) follows from Fano’s inequality (18) with ε n 0 as n and ( b ) is due to (15). We can also upper bound Δ A as
Δ A = ( a ) H ( Y n ) - log | g n ( J ) | ( b ) H ( Y n ) - H ( Y n | J ) + n ε n = i = 1 n H ( Y i ) - H ( Y i | Y i - 1 , J ) + n ε n i = 1 n H ( Y i ) - H ( Y i | Y i - 1 , X i + 1 n , J ) + n ε n = i = 1 n I ( Y i ; U i ) + n ε n
where ( a ) follows from (15), ( b ) follows from (18), and in the last equality the auxiliary random variable U i : = ( Y i - 1 , X i + 1 n , J ) is introduced.
We shall now lower bound I ( X n ; J ) :
n ( Δ M + δ ) I ( X n ; J ) = ( a ) I ( Y n ; J ) + i = 1 n [ I ( X i ; U i ) - I ( Y i ; U i ) ] ( b ) n Δ A + i = 1 n [ I ( X i ; U i ) - I ( Y i ; U i ) ] - n ε n
where ( a ) follows from Lemma 14 and ( b ) is due to Fano’s inequality and (15) (or equivalently from (17)).
Combining (22), (23) and (24), we can write
R Δ A - ε n Δ A I ( Y Q ; U Q | Q ) + ε n = I ( Y Q ; U Q , Q ) + ε n Δ M Δ A + I ( X Q ; U Q | Q ) - I ( Y Q ; U Q | Q ) - ε n = Δ A + I ( X Q ; U Q , Q ) - I ( Y Q ; U Q , Q ) - ε n
where ε n : = ε n + δ and Q is a random variable distributed uniformly over { 1 , 2 , , n } which is independent of ( X , Y ) and hence I ( Y Q ; U Q | Q ) = 1 n i = 1 n I ( Y i ; U i ) . The results follow by denoting U : = ( U Q , Q ) and noting that Y Q and X Q have the same distributions as Y and X, respectively. ☐
If the encoder does not have direct access to the private source X n , then we can define the encoder mapping as f n : Y n { 1 , 2 , , s n R } . The following corollary is an immediate consequence of Theorem 12.
Corollary 15. 
If the encoder does not see the private source, then for all achievable dependence dilution triple ( R , Δ A , Δ M ) , we have
R Δ A Δ A I ( Y ; U ) Δ M I ( X ; U ) - I ( Y ; U ) + Δ A
for some joint distribution P X Y U = P X Y P U | Y where the auxiliary random variable U U satisfies | U | | Y | + 1 .
Remark 5. 
If source Y is required to be amplified (according to (17)) at maximum rate, that is, Δ A = I ( Y ; U ) for an auxiliary random variable U which satisfies X -∘- Y -∘- U , then by Corollary 15, the best privacy performance one can expect from the dependence dilution setting is
Δ M * = min U : X -∘- Y -∘- U I ( Y ; U ) Δ A I ( X ; U )
which is equal to the dual of g ε ( X ; Y ) evaluated at Δ A , t Δ A ( X ; Y ) , as defined in (1).
The dependence dilution problem is closely related to the discriminatory lossy source coding problem studied in [15]. In this problem, an encoder f observes ( X n , Y n ) and wants to describe this source to a decoder, g, such that g recovers Y n within distortion level D and I ( f ( X n , Y n ) ; X n ) n Δ M . If the distortion level is Hamming measure, then the distortion constraint and the amplification constraint are closely related via Fano’s inequality. Moreover, dependence dilution problem reduces to a secure lossless (list decoder of fixed size 1) source coding problem by setting Δ A = H ( H ) , which is recently studied in [43].

3.2. MMSE Estimation of Functions of Private Information

In this section, we provide a justification for the privacy guarantee ρ m 2 ( X ; Z ) ε . To this end, we recall the definition of the minimum mean squared error estimation.
Definition 16. 
Given random variables U and V, mmse ( U | V ) is defined as the minimum error of an estimate, g ( V ) , of U based on V, measured in the mean-square sense, that is
mmse ( U | V ) : = inf g L 2 ( V ) E [ U - g ( V ) 2 ] = E [ U - E [ U | V ] 2 ] = E [ var ( U | V ) ]
where var ( U | V ) denotes the conditional variance of U given V.
It is easy to see that mmse ( U | V ) = 0 if and only if U = f ( V ) for some measurable function f and mmse ( U | V ) = var ( U ) if and only if U V . Hence, unlike for the case of maximal correlation, a small value of mmse ( U | V ) implies a strong dependence between U and V. Hence, although it is not a "proper" measure of correlation, in a certain sense it measures how well one random variable can be predicted from another one.
Given a non-degenerate measurable function f : X R , consider the following constraint on mmse ( f ( X ) | Y )
( 1 - ε ) var ( f ( X ) ) mmse ( f ( X ) | Z ) var ( f ( X ) ) .
This guarantees that no adversary knowing Z can efficiently estimate f ( X ) . First consider the case where f is an identity function, i.e., f ( x ) = x . In this case, a direct calculation shows that
mmse ( X | Z ) = ( a ) E [ ( X - E [ X | Z ] ) 2 ] = E [ X 2 ] - E [ ( E [ X | Z ] ) 2 ] = var ( X ) ( 1 - ρ 2 ( X ; E [ X | Z ] ) ) ( b ) var ( X ) ( 1 - ρ m 2 ( X ; Z ) )
where ( a ) follows from (26) and ( b ) is due to the definition of maximal correlation. Having imposed ρ m 2 ( X ; Z ) ε , we, can therefore conclude that the MMSE of estimating X given Z satisfies
( 1 - ε ) var ( X ) mmse ( X | Z ) var ( X )
which shows that ρ m 2 ( X ; Z ) ε implies (27) for f ( x ) = x . However, in the following we show that the constraint ρ m 2 ( X ; Z ) ε is, indeed, equivalent to (27) for any non-degenerate measurable f : X R .
Definition 17 
([44]). A joint distribution P U V satisfies a Poincaré inequality with constant c 1 if for all f : U R
c · var ( f ( U ) ) mmse ( f ( U ) | V )
and the Poincaré constant for P U V is defined as
ϑ ( U ; V ) : = inf f mmse ( f ( U ) | V ) var ( f ( U ) )
The privacy constraint (27) can then be viewed as
ϑ ( X ; Z ) 1 - ε .
Theorem 18 
([44]). For any joint distribution P U V , we have
ϑ ( U ; V ) = 1 - ρ m 2 ( U ; V )
In light of Theorem 18 and (29), the privacy constraint (27) is equivalent to ρ m 2 ( X ; Z ) ε , that is,
ρ m 2 ( X ; Z ) ε ( 1 - ε ) var ( f ( X ) ) mmse ( f ( X ) | Z ) var ( f ( X ) )
for any non-degenerate measurable functions f : X R .
Hence, g ^ ε ( X ; Y ) characterizes the maximum information extraction from Y such that no (non-trivial) function of X can be efficiently estimated, in terms of MMSE (27), given the extracted information.

4. Observation Channels for Minimal and Maximal g ε ( X ; Y )

In this section, we characterize the observation channels which achieve the lower or upper bounds on the rate-privacy function in (4). We first derive general conditions for achieving the lower bound and then present a large family of observation channels P Y | X which achieve the lower bound. We also give a family of P Y | X which attain the upper bound on g ε ( X ; Y ) .

4.1. Conditions for Minimal g ε ( X ; Y )

Assuming that g 0 ( X ; Y ) = 0 , we seek a set of conditions on P X Y such that g ε ( X ; Y ) is linear in ε, or equivalently, g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) . In order to do this, we shall examine the slope of g ε ( X ; Y ) at zero. Recall that by concavity of g ε ( X ; Y ) , it is clear that g 0 ( X ; Y ) H ( Y ) I ( X ; Y ) . We strengthen this bound in the following lemmas. For this, we need to recall the notion of Kullback-Leibler divergence. Given two probability distribution P and Q supported over a finite alphabet U,
D ( P | | Q ) : = u U P ( u ) log P ( u ) Q ( u )
Lemma 19. 
For a given joint distribution P X Y = P Y × P X | Y , if g 0 ( X ; Y ) = 0 , then for any ε 0
g 0 ( X ; Y ) max y Y - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( · ) )
Proof. 
The proof is given in Appendix A. ☐
Remark 6. 
Note that if for a given joint distribution P X Y , there exists y 0 Y such that D ( P X | Y ( · | y 0 ) | | P X ( · ) ) = 0 , it implies that P X | Y ( · | y 0 ) = P X ( x ) . Consider the binary random variable Z { 1 , e } constructed according to the distribution P Z | Y ( 1 | y 0 ) = 1 and P Z | Y ( e | y ) = 1 for y Y \ { y 0 } . We can now claim that Z is independent of X, because P X | Z ( x | 1 ) = P X | Y ( x | y 0 ) = P X ( x ) and
P X | Z ( x | e ) = y y 0 P X | Y ( x | y ) P Y | Z ( y | e ) = y y 0 P X | Y ( x | y ) P Y ( y ) 1 - P Y ( y 0 ) = 1 1 - P Y ( y 0 ) y y 0 P X Y ( x , y ) = P X ( x )
Clearly, Z and Y are not independent, and hence g 0 ( X ; Y ) > 0 . This implies that the right-hand side of inequality in Lemma 19 can not be infinity.
In order to prove the main result, we need the following simple lemma.
Lemma 20. 
For any joint distribution P X Y , we have
H ( Y ) I ( X ; Y ) max y Y - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) )
where equality holds if and only if there exists a constant c > 0 such that - log P Y ( y ) = c D ( P X | Y ( · | y ) | | P X ( x ) ) for all y Y .
Proof. 
It is clear that
H ( Y ) I ( X ; Y ) = - y Y P Y ( y ) log P Y ( y ) y Y P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) ) max y Y - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) )
where the inequality follows from the fact that for any three sequences of positive numbers { a i } i = 1 n , { b i } i = 1 n and { λ i } i = 1 n we have i = 1 n λ i a i i = 1 n λ i b i max 1 i n a i b i where equality occurs if and only if a i b i = c for all 1 i n . ☐
Now we are ready to state the main result of this subsection.
Theorem 21. 
For a given ( X , Y ) with joint distribution P X Y = P Y × P X | Y , if g 0 ( X ; Y ) = 0 and ε g ε ( X ; Y ) is linear for 0 ε I ( X ; Y ) , then for any y Y
H ( Y ) I ( X ; Y ) = - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( · ) )
Proof. 
Note that the fact that g 0 ( X ; Y ) = 0 and g ε ( X ; Y ) is linear in ε is equivalent to g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) . It is, therefore, immediate from Lemmas 19 and 20 that we have
g 0 ( X ; Y ) = ( a ) H ( Y ) I ( X ; Y ) ( b ) max y Y - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) ) ( c ) g 0 ( X ; Y )
where ( a ) follows from the fact that g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) and ( b ) and ( c ) are due to Lemmas 20 and 19, respectively. The inequality in (31) shows that
H ( Y ) I ( X ; Y ) = max y Y - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) )
According to Lemma 20, (32) implies that the ratio of - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) ) does not depend on y Y and hence the result follows. ☐
This theorem implies that if there exists y = y 1 and y = y 2 such that log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( x ) ) results in two different values, then ε g ε ( X , Y ) cannot achieve the lower bound in (4), or equivalently
g ε ( X ; Y ) > ε H ( Y ) I ( X ; Y )
This, therefore, gives a necessary condition for the lower bound to be achievable. The following corollary simplifies this necessary condition.
Corollary 22. 
For a given joint distribution P X Y = P Y × P X | Y , if g 0 ( X ; Y ) = 0 and ε g ε ( X ; Y ) is linear, then the following are equivalent:
(i) 
Y is uniformly distributed,
(ii) 
D ( P X | Y ( · | y ) | | P X ( · ) ) is constant for all y Y .
Proof. 
( i ) ( i i ) :
From Theorem 21, we have for all y Y
H ( Y ) I ( X ; Y ) = - log ( P Y ( y ) ) D P X | Y ( · | y ) | | P X ( · )
Letting D : = D P X | Y ( · | y ) | | P X ( · ) for any y Y , we have y P Y ( y ) D = I ( X ; Y ) and hence D = I ( X ; Y ) , which together with (33) implies that H ( Y ) = - log ( P Y ( y ) ) for all y Y and hence Y is uniformly distributed.
( i i ) ( i ) :
When Y is uniformly distributed, we have from (33) that I ( X ; Y ) = D P X | Y ( · | y ) | | P X ( · ) which implies that D P X | Y ( · | y ) | | P X ( · ) is constant for all y Y . ☐
Example 1. 
Suppose P Y | X is a binary symmetric channel (BSC) with crossover probability 0 < α < 1 and P X = Bernoulli ( 0.5 ) . In this case, P X | Y is also a BSC with input distribution P Y = Bernoulli ( 0.5 ) . Note that Corollary 11 implies that g 0 ( X ; Y ) = 0 . We will show that g ε ( X ; Y ) is linear as a function of ε 0 for a larger family of symmetric channels (including BSC) in Corollary 24. Hence, the BSC with uniform input nicely illustrates Corollary 22, because D ( P X | Y ( · | y ) | | P X ( · ) ) = 1 - h ( α ) for y { 0 , 1 } .
Example 2. 
Now suppose P X | Y is a binary asymmetric channel such that P X | Y ( · | 0 ) = Bernoulli ( α 0 ) , P X | Y ( · | 1 ) = Bernoulli ( α 1 ) for some 0 < α 0 , α 1 < 1 and input distribution P Y = Bernoulli ( p ) , 0 < p 0.5 . It is easy to see that if α 0 + α 1 = 1 then D ( P X | Y ( · | y ) | | P X ( · ) ) does not depend on y and hence we can conclude from Corollary 22 (noticing that g 0 ( X ; Y ) = 0 ) that in this case for any p < 0.5 , g ε ( X ; Y ) is not linear and hence for 0 < ε < I ( X ; Y )
g ε ( X ; Y ) > ε H ( Y ) I ( X ; Y )
In Theorem 21, we showed that when g ε ( X ; Y ) achieves its lower bound, illustrated in (4), the slope of the mapping ε g ε ( X ; Y ) at zero is equal to - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( · ) ) for any y Y . We will show in the next section that the reverse direction is also true at least for a large family of binary-input symmetric output channels, for instance when P Y | X is a BSC, and thus showing that in this case,
g 0 ( X ; Y ) = - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( · ) ) , y Y g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) , 0 ε I ( X ; Y )

4.2. Special Observation Channels

In this section, we apply the results of last section to different joint distributions P X Y . In the first family of channels from X to Y, we look at the case where Y is binary and the reverse channel P X | Y has symmetry in a particular sense, which will be specified later. One particular case of this family of channels is when P X | Y is a BSC. As a family of observation channels which achieves the upper bound of g ε ( X ; Y ) , stated in (4), we look at the class of erasure channels from X Y , i.e., Y is an erasure version of X.

4.2.1. Observation Channels With Symmetric Reverse

The first example of P XY that we consider for binary Y is the so-called Binary Input Symmetric Output (BISO) P X | Y , see for example [45,46]. Suppose Y = { 0 , 1 } and X = { 0 , ± 1 , ± 2 , , ± k } , and for any x X we have P X | Y ( x | 1 ) = P X | Y ( - x | 0 ) . This clearly implies that p 0 : = P X | Y ( 0 | 0 ) = P X | Y ( 0 | 1 ) . We notice that with this definition of symmetry, we can always assume that the output alphabet X = { ± 1 , ± 2 , , ± k } has even number of elements because we can split X = 0 into two outputs, X = 0 + and X = 0 - , with P X | Y ( 0 - | 0 ) = P X | Y ( 0 + | 0 ) = p 0 2 and P X | Y ( 0 - | 1 ) = P X | Y ( 0 + | 1 ) = p 0 2 . The new channel is clearly essentially equivalent to the original one, see [46] for more details. This family of channels can also be characterized using the definition of quasi-symmetric channels [47, Definition 4.17]. A channel W is BISO if (after making | X | even) the transition matrix P X | Y can be partitioned along its columns into binary-input binary-output sub-arrays in which rows are permutations of each other and the column sums are equal. It is clear that binary symmetric channels and binary erasure channels are both BISO. The following lemma gives an upper bound for g ε ( X , Y ) when P X | Y belongs to such a family of channels.
Lemma 23. 
If the channel P X | Y is BISO, then for ε [ 0 , I ( X ; Y ) ] ,
ε H ( Y ) I ( X ; Y ) g ε ( X ; Y ) H ( Y ) - I ( X ; Y ) - ε C ( P X | Y )
where C ( P X | Y ) denotes the capacity of P X | Y .
Proof. 
The lower bound has already appeared in (4). To prove the upper bound note that by Markovity X -∘- Y -∘- Z, we have for any x X and z Z
P X | Z ( x | z ) = P X | Y ( x | 0 ) P Y | Z ( 0 | z ) + P X | Y ( x | 1 ) P Y | Z ( 1 | z )
Now suppose Z0:={z:PY|Z(0|z)PY|Z(1|z)} and similarly Z 1 : = { z : P Y | Z ( 1 | z ) P Y | Z ( 0 | z ) } . Then (34) allows us to write for z Z 0
P X | Z ( x | z ) = P X | Y ( x | 0 ) h b - 1 ( H ( Y | Z = z ) ) + P X | Y ( x | 1 ) ( 1 - h b - 1 ( H ( Y | Z = z ) ) )
where h b - 1 : [ 0 , 1 ] [ 0 , 0.5 ] is the inverse of binary entropy function, and for z Z 1 ,
P X | Z ( x | z ) = P X | Y ( x | 0 ) ( 1 - h b - 1 ( H ( Y | Z = z ) ) ) + P X | Y ( x | 1 ) h b - 1 ( H ( Y | Z = z ) )
Letting P h b - 1 ( H ( Y | z ) ) and P ˜ h b - 1 ( H ( Y | z ) ) denote the right-hand sides of (35) and (36), respectively, we can, hence, write
H ( X | Z ) = z Z P Z ( z ) H ( X | Z = z ) = ( a ) z Z 0 P Z ( z ) H ( P h b - 1 ( H ( Y | Z = z ) ) ) + z Z 1 P Z ( z ) H ( P ˜ h b - 1 ( H ( Y | Z = z ) ) ) ( b ) z Z 0 P Z ( z ) ( 1 - H ( Y | Z = z ) ) H ( P h b - 1 ( 0 ) ) + H ( Y | Z = z ) H ( P h b - 1 ( 1 ) ) + z Z 1 P Z ( z ) ( 1 - H ( Y | Z = z ) ) H ( P ˜ h b - 1 ( 0 ) ) + H ( Y | Z = z ) H ( P ˜ h b - 1 ( 1 ) ) = ( c ) z Z 0 P Z ( z ) ( 1 - H ( Y | Z = z ) ) H ( X | Y ) + H ( Y | Z = z ) H ( X unif ) + z Z 1 P Z ( z ) ( 1 - H ( Y | Z = z ) ) H ( X | Y ) + H ( Y | Z = z ) H ( X unif ) = H ( X | Y ) [ 1 - H ( Y | Z ) ] + H ( Y | Z ) H ( X unif )
where H ( X unif ) denotes the entropy of X when Y is uniformly distributed. Here, ( a ) is due to (35) and (36), ( b ) follows form convexity of u H ( P h b - 1 ( u ) ) ) for all u [ 0 , 1 ] [48] and Jensen’s inequality. In ( c ) , we used the symmetry of channel P X | Y to show that H ( X | Y = 0 ) = H ( X | Y = 1 ) = H ( X | Y ) . Hence, we obtain
H ( Y | Z ) H ( X | Z ) - H ( X | Y ) H ( X unif ) - H ( X | Y ) = I ( X ; Y ) - I ( X ; Z ) C ( P X | Y )
where the equality follows from the fact that for BISO channel (and in general for any quasi-symmetric channel) the uniform input distribution is the capacity-achieving distribution [47, Lemma 4.18]. Since g ε ( X ; Y ) is attained when I ( X ; Z ) = ε , the conclusion immediately follows. ☐
This lemma then shows that the larger the gap between I ( X ; Y ) and I ( X ; Y ) is for Y Bernoulli ( 0.5 ) , the more g ε ( X ; Y ) deviates from its lower bound. When Y Bernoulli ( 0.5 ) , then C ( P Y | X ) = I ( X ; Y ) and H ( Y ) = 1 and hence Lemma 23 implies that
ε I ( X ; Y ) g ε ( X ; Y ) 1 - I ( X ; Y ) - ε I ( X ; Y ) = ε I ( X ; Y )
and hence we have proved the following corollary.
Corollary 24. 
If the channel P X | Y is BISO and Y Bernoulli ( 0.5 ) , then for any ε 0
g ε ( X ; Y ) = 1 I ( X ; Y ) min { ε , I ( X ; Y ) }
This corollary now enables us to prove the reverse direction of Theorem 21 for the family of BISO channels.
Theorem 25. 
If P X | Y is a BISO channel, then the following statements are equivalent:
(i) 
g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) for 0 ε I ( X ; Y ) .
(ii) 
The initial efficiency of privacy-constrained information extraction is
g 0 ( X ; Y ) = - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( · ) ) , y Y
Proof. 
(i)⇒ (ii).
This follows from Theorem 21.
(ii)⇒ (i).
Let Y Bernoulli ( p ) for 0 < p < 1 , and, as before, X = { ± 1 , ± 2 , , ± k } , so that P X | Y is determined by a 2 × ( 2 k ) matrix. We then have
- log P Y ( 0 ) D ( P X | Y ( · | 0 ) | | P X ( · ) ) = log ( 1 - p ) H ( X | Y ) + x = - k k P X | Y ( x | 0 ) log ( P X ( x ) )
and
- log P Y ( 1 ) D ( P X | Y ( · | 1 ) | | P X ( · ) ) = log ( p ) H ( X | Y ) + x = - k k P X | Y ( x | 1 ) log ( P X ( x ) ) .
The hypothesis implies that (37) is equal to (38), that is,
log ( 1 - p ) H ( X | Y ) + x = - k k P X | Y ( x | 0 ) log ( P X ( x ) ) = log ( p ) H ( X | Y ) + x = - k k P X | Y ( x | 1 ) log ( P X ( x ) )
It is shown in Appendix B that (39) holds if and only if p = 0.5 . Now we can invoke Corollary 24 to conclude that g ε ( X ; Y ) = ε H ( Y ) I ( X ; Y ) . ☐
This theorem shows that for any BISO P X | Y channel with uniform input, the optimal privacy filter is an erasure channel depicted in Figure 2. Note that if P X | Y is a BSC with uniform input P Y = Bernoulli ( 0.5 ) , then P Y | X is also a BSC with uniform input P X = Bernoulli ( 0.5 ) . The following corollary specializes Corollary 24 for this case.
Corollary 26. 
For the joint distribution P X P Y | X = Bernoulli ( 0.5 ) × BSC ( α ) , the binary erasure channel with erasure probability (shown in Figure 4)
δ ( ε , α ) : = 1 - ε I ( X ; Y )
for 0 ε I ( X ; Y ) , is the optimal privacy filter in (3). In other words, for ε 0
g ε ( X ; Y ) = 1 I ( X ; Y ) min { ε , I ( X ; Y ) }
Moreover, for a given 0 < α < 1 2 , P X = Bernoulli ( 0.5 ) is the only distribution for which ε g ε ( X ; Y ) is linear. That is, for P X P Y | X = Bernoulli ( p ) × BSC ( α ) , 0 < p < 0.5 , we have
g ε ( X ; Y ) > ε H ( Y ) I ( X ; Y )
Proof. 
As mentioned earlier, since P X = Bernoulli ( 0.5 ) and P Y | X is BSC ( α ) , it follows that P X | Y is also a BSC with uniform input and hence from Corollary 24, we have g ε ( X ; Y ) = ε I ( X ; Y ) . As in this case g ε ( X ; Y ) achieves the lower bound given in Lemma 1, we conclude from Figure 2 that BEC( δ ( ε , α ) ), where δ ( ε , α ) = 1 - ε I ( X ; Y ) , is an optimal privacy filter. The fact that P X = Bernoulli ( 0.5 ) is the only input distribution for which ε g ε ( X ; Y ) is linear follows from the proof of Theorem 25. In particular, we saw that a necessary and sufficient condition for g ε ( X ; Y ) being linear is that the ratio - log P Y ( y ) D ( P X | Y ( · | y ) | | P X ( · ) ) is constant for all y Y . As shown before, this is equivalent to Y Bernoulli ( 0.5 ) . For the binary symmetric channel, this is equivalent to X Bernoulli ( 0.5 ) . ☐
The optimal privacy filter for BSC(α) and uniform X is shown in Figure 4. In fact, this corollary immediately implies that the general lower-bound given in (4) is tight for the binary symmetric channel with uniform X.

4.2.2. Erasure Observation Channel

Combining (8) and Lemma 1, we have for ε I ( X ; Y )
ε H ( Y ) I ( X ; Y ) + g 0 ( X ; Y ) 1 - ε I ( X ; Y ) g ε ( X ; Y ) H ( Y | X ) + ε
In the following we show that the above upper and lower bound coincide when P Y | X is an erasure channel, i.e., P Y | X ( x | x ) = 1 - δ and P Y | X ( e | x ) = δ for all x X and 0 δ 1 .
Lemma 27. 
For any given ( X , Y ) , if P Y | X is an erasure channel (as defined above), then
g ε ( X ; Y ) = H ( Y | X ) + min { ε , I ( X ; Y ) }
for any ε 0 .
Proof. 
It suffices to show that if P Y | X is an erasure channel, then g 0 ( X ; Y ) = H ( Y | X ) . This follows, since if g 0 ( X ; Y ) = H ( Y | X ) , then the lower bound in (41) becomes H ( Y | X ) + ε and thus g ε ( X ; Y ) = H ( Y | X ) + ε .
Let | X | = m and Y = X { e } where e denotes the erasure symbol. Consider the following privacy filter to generate Z Y :
P Z | Y ( z | y ) = 1 m if y e , z e , 1 if y = z = e .
For any x X , we have
P Z | X ( z | x ) = P Z | Y ( z | x ) P Y | X ( x | x ) + P Z | Y ( z | e ) P Y | X ( e | x ) = 1 - δ m 1 { z e } + δ 1 { z = e }
which implies Z X and thus I ( X ; Z ) = 0 . On the other hand, P Z ( z ) = 1 - δ m 1 { z e } + δ 1 { z = e } , and therefore we have
g 0 ( X ; Y ) I ( Y ; Z ) = H ( Z ) - H ( Z | Y ) = H 1 - δ m , , 1 - δ m , δ - ( 1 - δ ) log ( m ) = h ( δ ) = H ( Y | X )
It then follows from Lemma 1 that g 0 ( X ; Y ) = H ( Y | X ) , which completes the proof. ☐
Example 3. 
In light of this lemma, we can conclude that if P Y | X = BEC ( δ ) , then the optimal privacy filter is a combination of an identity channel and a BSC( α ( ε , δ ) ), as shown in Figure 5, where 0 α ( ε , δ ) 1 2 is the unique solution of
( 1 - δ ) [ h b ( α * p ) - h b ( α ) ] = ε
where X Bernoulli ( p ) , p 0.5 and a * b = a ( 1 - b ) + b ( 1 - a ) . Note that it is easy to check that I ( X ; Z ) = ( 1 - δ ) [ h b ( α * p ) - h b ( α ) ] . Therefore, in order for this channel to be a valid privacy filter, the crossover probability, α ( ε , δ ) , must be chosen such that I ( X ; Z ) = ε . We note that for fixed 0 < δ < 1 and 0 < p < 0.5 , the map α ( 1 - δ ) [ h b ( α * p ) - h b ( α ) ] is monotonically decreasing on [ 0 , 1 2 ] ranging over [ 0 , ( 1 - δ ) h b ( p ) ] and since ε I ( X ; Y ) = ( 1 - δ ) h b ( p ) , the solution of the above equation is unique.
Combining Lemmas 1 and 27 with Corollary 26, we can show the following extremal property of the BEC and BSC channels, which is similar to other existing extremal properties of the BEC and the BSC, see, e.g., [46] and [45]. For X Bernoulli ( 0.5 ) , we have for any channel P Y | X ,
g ε ( X ; Y ) ε I ( X ; Y ) = g ε ( BSC ( α ^ ) )
where g ε ( BSC ( α ) ) is the rate-privacy function corresponding to P X Y = Bernoulli ( 0.5 ) × BSC ( α ) and α ^ : = h b - 1 ( H ( X | Y ) ) . Similarly, if X Bernoulli ( p ) , we have for any channel P Y | X with H ( Y | X ) 1 ,
g ε ( X ; Y ) H ( Y | X ) + ε = g ε ( BEC ( δ ^ ) )
where g ε ( BEC ( δ ) ) is the rate-privacy function corresponding to P X Y = Bernoulli ( p ) × BEC ( δ ) and δ ^ : = h b - 1 ( H ( Y | X ) ) .

5. Rate-Privacy Function for Continuous Random Variables

In this section we extend the rate-privacy function g ε ( X ; Y ) to the continuous case. Specifically, we assume that the private and observable data are continuous random variables and that the filter is composed of two stages: first Gaussian noise is added and then the resulting random variable is quantized using an M-bit accuracy uniform scalar quantizer (for some positive integer M N ). These filters are of practical interest as they can be easily implemented. This section is divided in two subsections, in the first we discuss general properties of the rate-privacy function and in the second we study the Gaussian case in more detail. Some observations on g ^ ε ( X ; Y ) for continuous X and Y are also given.

5.1. General Properties of the Rate-Privacy Function

Throughout this section we assume that the random vector ( X , Y ) is absolutely continuous with respect to the Lebesgue measure on R 2 . Additionally, we assume that its joint density f X , Y satisfies the following.
(a)
There exist constants C 1 > 0 , p > 1 and bounded function C 2 : R R such that
f Y ( y ) C 1 | y | - p
and also for x R
f Y | X ( y | x ) C 2 ( x ) | y | - p
(b)
E [ X 2 ] and E [ Y 2 ] are both finite,
(c)
the differential entropy of ( X , Y ) satisfies h ( X , Y ) > - ,
(d)
H ( Y ) < , where a denotes the largest integer such that a .
Note that assumptions (b) and (c) together imply that h(X,Y), h ( X ) and h ( Y ) are finite, i.e., the maps x f X ( x ) | log f X ( x ) | , y f Y ( y ) | log f Y ( y ) | and ( x , y ) f X , Y ( x , y ) | log ( f X , Y ( x , y ) ) | are integrable. We also assume that X and Y are not independent, since otherwise the problem to characterize g ε ( X ; Y ) becomes trivial by assuming that the displayed data Z can equal the observable data Y.
We are interested in filters of the form Q M ( Y + γ N ) where γ 0 , N N ( 0 , 1 ) is a standard normal random variable which is independent of X and Y, and for any positive integer M, Q M denotes the M-bit accuracy uniform scalar quantizer, i.e., for all x R
Q M ( x ) = 1 2 M 2 M x
Let Z γ = Y + γ N and Z γ M = Q M ( Z γ ) = Q M ( Y + γ N ) . We define, for any M N ,
g ε , M ( X ; Y ) : = sup γ 0 , I ( X ; Z γ M ) ε I ( Y ; Z γ M )
and similarly
g ε ( X ; Y ) : = sup γ 0 , I ( X ; Z γ ) ε I ( Y ; Z γ )
The next theorem shows that the previous definitions are closely related.
Theorem 28. 
Let ε > 0 be fixed. Then lim M g ε , M ( X ; Y ) = g ε ( X ; Y ) .
Proof. 
See Appendix C. ☐
In the limit of large M, g ε ( X ; Y ) approximates g ε , M ( X ; Y ) . This becomes relevant when g ε ( X ; Y ) is easier to compute than g ε , M ( X ; Y ) , as demonstrated in the following subsection. The following theorem summarizes some general properties of g ε ( X ; Y ) .
Theorem 29. 
The function ε g ε ( X ; Y ) is non-negative, strictly-increasing, and satisfies
lim ε 0 g ε ( X ; Y ) = 0 a n d g I ( X ; Y ) ( X ; Y ) =
Proof. 
See Appendix C. ☐
As opposed to the discrete case, in the continuous case g ε ( X ; Y ) is no longer bounded. In the following section we show that ε g ε ( X ; Y ) can be convex, in contrast to the discrete case where it is always concave.
We can also define g^ε,M(X;Y) and g ^ ε ( X ; Y ) for continuous X and Y, similar to (43) and (44), but where the privacy constraints are replaced by ρ m 2 ( X ; Z γ M ) ε and ρ m 2 ( X ; Z γ ) ε , respectively. It is clear to see from Theorem 29 that g ^ 0 ( X ; Y ) = g 0 ( X ; Y ) = 0 and g ^ ρ 2 ( X ; Y ) ( X ; Y ) = . However, although we showed that g ε ( X ; Y ) is indeed the asymptotic approximation of g ε , M ( X ; Y ) for M large enough, it is not clear that the same statement holds for g ^ ε ( X ; Y ) and g ^ ε , M ( X ; Y ) .

5.2. Gaussian Information

The rate-privacy function for Gaussian Y has an interesting interpretation from an estimation theoretic point of view. Given the private and observable data ( X , Y ) , suppose an agent is required to estimateY based on the output of the privacy filter. We wish to know the effect of imposing a privacy constraint on the estimation performance.
The following lemma shows that g ε ( X ; Y ) bounds the best performance of the predictability of Y given the output of the privacy filter. The proof provided for this lemma does not use the Gaussianity of the noise process, so it holds for any noise process.
Lemma 30. 
For any given private data X and Gaussian observable data Y, we have for any ε 0
inf γ 0 , I ( X ; Z γ ) ε mmse ( Y | Z γ ) var ( Y ) 2 - 2 g ε ( X ; Y )
Proof. 
It is a well-known fact from rate-distortion theory that for a Gaussian Y and its reconstruction Y ^
I ( Y ; Y ^ ) 1 2 log var ( Y ) E [ ( Y - Y ^ ) 2 ]
and hence by setting Y ^ = E [ Y | Z γ ] , where Z γ is an output of a privacy filter, and noting that I ( Y ; Y ^ ) I ( Y ; Z γ ) , we obtain
mmse ( Y | Z γ ) var ( Y ) 2 - 2 I ( Y ; Z γ )
from which the result follows immediately. ☐
According to Lemma 30, the quantity λ ε ( X ) : = 2 - 2 g ε ( X ; Y ) is a parameter that bounds the difficulty of estimating Gaussian Y when observing an additive perturbation Z with privacy constraint I ( X ; Z ) ε . Note that 0 < λ ε ( X ) 1 , and therefore, provided that the privacy threshold is not trivial (i.e, ε < I ( X ; Y ) ), the mean squared error of estimating Y given the privacy filter output is bounded away from zero, however the bound decays exponentially at rate of g ε ( X ; Y ) .
To finish this section, assume that X and Y are jointly Gaussian with correlation coefficient ρ. The value of g ε ( X ; Y ) can be easily obtained in closed form as demonstrated in the following theorem.
Theorem 31. 
Let ( X , Y ) be jointly Gaussian random variables with correlation coefficient ρ. For any ε [ 0 , I ( X ; Y ) ) we have
g ε ( X ; Y ) = 1 2 log ρ 2 2 - 2 ε + ρ 2 - 1
Proof. 
One can always write Y = a X + N 1 where a 2 = ρ 2 var ( Y ) var ( X ) and N 1 is a Gaussian random variable with mean 0 and variance σ 2 = ( 1 - ρ 2 ) var ( Y ) which is independent of ( X , Y ) . On the other hand, we have Z γ = Y + γ N where N is the standard Gaussian random variable independent of ( X , Y ) and hence Z γ = a X + N 1 + γ N . In order for this additive channel to be a privacy filter, it must satisfy
I ( X ; Z γ ) ε
which implies
1 2 log var ( Y ) + γ 2 σ 2 + γ 2 ε
and hence
γ 2 2 - 2 ε + ρ 2 - 1 1 - 2 - 2 ε var ( Y ) = : γ *
Since γ I ( Y ; Z γ ) is strictly decreasing (cf., Appendix C), we obtain
g ε ( X ; Y ) = I ( Y ; Z γ * ) = 1 2 log 1 + var ( Y ) γ 2 = 1 2 log 1 + 1 - 2 - 2 ε 2 - 2 ε + ρ 2 - 1
According to (46), we conclude that the optimal privacy filter for jointly Gaussian ( X , Y ) is an additive Gaussian channel with signal to noise ratio 1 - 2 - 2 ε 2 - 2 ε + ρ 2 - 1 , which shows that if perfect privacy is required, then the displayed data is independent of the observable data Y, i.e., g 0 ( X ; Y ) = 0 .
Remark 7. 
We could assume that the privacy filter adds non-Gaussian noise to the observable data and define the rate-privacy function accordingly. To this end, we define
g ε f ( X ; Y ) : = sup γ 0 , I ( X ; Z γ f ) I ( Y ; Z γ f )
where Z γ f = Y + γ M f and M f is a noise process that has stable distribution with density f and is independent of ( X , Y ) . In this case, we can use a technique similar to Oohama [49] to lower bound g ε f ( X ; Y ) for jointly Gaussian ( X , Y ) . Since X and Y are jointly Gaussian, we can write X = a Y + b N where a 2 = ρ 2 var ( X ) var ( Y ) , b = ( 1 - ρ 2 ) var X , and N is a standard Gaussian random variable that is independent of Y. We can apply the conditional entropy power inequality (cf., [42, Page 22]) for a random variable Z that is independent of N, to obtain
2 2 h ( X | Z ) 2 2 h ( a Y | Z ) + 2 2 h ( N ) = a 2 2 2 h ( Y | Z ) + 2 π e ( 1 - ρ 2 ) var ( X )
and hence
2 - 2 I ( X ; Z ) 2 2 h ( X ) a 2 2 2 h ( Y ) 2 - 2 I ( Y ; Z ) + 2 π e ( 1 - ρ 2 ) var ( X )
Assuming Z = Z γ f and taking infimum from both sides of above inequality over γ such that I ( X ; Z γ f ) ε , we obtain
g ε f ( X ; Y ) 1 2 log ρ 2 2 - 2 ε + ρ 2 - 1 = g ε ( X ; Y )
which shows that for Gaussian ( X , Y ) , Gaussian noise is the worst stable additive noise in the sense of privacy-constrained information extraction.
We can also calculate g ^ ε ( X ; Y ) for jointly Gaussian ( X , Y ) .
Theorem 32. 
Let ( X , Y ) be jointly Gaussian random variables with correlation coefficient ρ. For any ε [ 0 , ρ 2 ) we have that
g ^ ε ( X ; Y ) = 1 2 log ρ 2 ρ 2 - ε
Proof. 
Since for the correlation coefficient between Y and Z γ we have for any γ 0 ,
ρ 2 ( Y ; Z γ ) = var ( Y ) var ( Y ) + γ 2
we can conclude that
ρ 2 ( X ; Z γ ) = ρ 2 var ( Y ) var ( Y ) + γ 2
Since ρ m 2 ( X ; Z ) = ρ 2 ( X ; Z ) (see, e.g., [34]), the privacy constraint ρ m 2 ( X ; Z ) ε implies that
ρ 2 var ( Y ) var ( Y ) + γ 2 ε
and hence
γ 2 ( ρ 2 - ε ) var ( Y ) ε = : γ ^ ε 2
By monotonicity of the map γ I ( Y ; Z γ ) , we have
g ^ ε ( X ; Y ) = I ( Y ; Z γ ^ ε ) = 1 2 log 1 + var ( Y ) γ ^ ε 2 = 1 2 log ρ 2 ρ 2 - ε
Theorems 31 and 32 show that unlike to the discrete case (cf. Lemmas 2 and 8), ε g ε ( X ; Y ) and ε g ^ ε ( X ; Y ) are convex.

6. Conclusions

In this paper, we studied the problem of determining the maximal amount of information that one can extract by observing a random variable Y, which is correlated with another random variable X that represents sensitive or private data, while ensuring that the extracted data Z meets a privacy constraint with respect to X. Specifically, given two correlated discrete random variables X and Y, we introduced the rate-privacy function as the maximization of I ( Y ; Z ) over all stochastic ”privacy filters” P Z | Y such that p m ( X ; Z ) ϵ , where p m ( · ; · ) is a privacy measure and ϵ 0 is a given privacy threshold. We considered two possible privacy measure functions, p m ( X ; Z ) = I ( X ; Z ) and p m ( X ; Z ) = ρ m 2 ( X ; Z ) where ρ m denotes maximal correlation, resulting in the rate-privacy functions g ϵ ( X ; Y ) and g ^ ϵ ( X ; Y ) , respectively. We analyzed these two functions, noting that each function lies between easily evaluated upper and lower bounds, and derived their monotonicity and concavity properties. We next provided an information-theoretic interpretation for g ϵ ( X ; Y ) and an estimation-theoretic characterization for g ^ ϵ ( X ; Y ) . In particular, we demonstrated that the dual function of g ϵ ( X ; Y ) is a corner point of an outer bound on the achievable region of the dependence dilution coding problem. We also showed that g ^ ϵ ( X ; Y ) constitutes the largest amount of information that can be extracted from Y such that no meaningful MMSE estimation of any function of X can be realized by just observing the extracted information Z. We then examined conditions on P X Y under which the lower bound on g ϵ ( X ; Y ) is tight, hence determining the exact value of g ϵ ( X ; Y ) . We also showed that for any given Y, if the observation channel P Y | X is an erasure channel, then g ϵ ( X ; Y ) attains its upper bound. Finally, we extended the notions of the rate-privacy functions g ϵ ( X ; Y ) and g ^ ϵ ( X ; Y ) to the continuous case where the observation channel consists of an additive Gaussian noise channel followed by uniform scalar quantization.

Acknowledgments

This work was supported in part by Natural Sciences and Engineering Council (NSERC) of Canada.

Author Contributions

All authors of this paper contributed equally. All authors have read and approved the final manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Proof of Lemma 19

Given a joint distribution P X Y defined over X × Y where X = { 1 , 2 , , m } and Y = { 1 , 2 , , n } with n m , we consider a privacy filter specified by the following distribution for δ > 0 and Z = { k , e }
P Z | Y ( k | y ) = δ 1 { y = k }
P Z | Y ( e | y ) = 1 - δ 1 { y = k }
where 1 { · } denotes the indicator function. The system of X -∘- Y -∘- Z in this case is depicted in Figure 6 for the case of k = 1 .
We clearly have P Z ( k ) = δ P Y ( k ) and P Z ( e ) = 1 - δ P Y ( k ) , and hence
P X | Z ( x | k ) = P X Z ( x , k ) δ P Y ( k ) = P X Y Z ( x , k , k ) δ P Y ( k ) = δ P X Y ( x , k ) δ P Y ( k ) = P X | Y ( x | k )
and also,
P X | Z ( x | e ) = P X Z ( x , e ) 1 - δ P Y ( k ) = y P X Y Z ( x , y , e ) 1 - δ P Y ( k ) = y k P X Y Z ( x , y ) + ( 1 - δ ) P X Y ( x , k ) 1 - δ P Y ( k ) = P X ( x ) - δ P X Y ( x , k ) 1 - δ P Y ( k )
It, therefore, follows that for k { 1 , 2 , , n }
H ( X | Z = k ) = H ( X | Y = k )
and
H ( X | Z = e ) = H P X ( 1 ) - δ P X Y ( 1 , k ) 1 - δ P Y ( k ) , , P X ( m ) - δ P X Y ( m , k ) 1 - δ P Y ( k ) = : h X ( δ )
We then write
I ( X ; Z ) = H ( X ) - H ( X | Z ) = H ( X ) - δ P Y ( k ) H ( X | Y = k ) - ( 1 - δ P Y ( k ) ) h X ( δ )
and hence,
d d δ I ( X ; Z ) = - P Y ( k ) H ( X | Y = k ) + P Y ( k ) h X ( δ ) - ( 1 - δ P Y ( k ) ) h X ( δ )
where
h X ( δ ) = d d δ h X ( δ ) = - x = 1 m P X ( x ) P Y ( k ) - P X Y ( x , k ) [ 1 - δ P Y ( k ) ] 2 log P X ( x ) - δ P X Y ( x , y ) 1 - δ P Y ( k )
Using the first-order approximation of mutual information for δ = 0 , we can write
I ( X ; Z ) = d d δ I ( X ; Z ) | δ = 0 δ + o ( δ ) = δ x = 1 m P X Y ( x , k ) log P X Y ( x , k ) P X ( x ) P Y ( k ) + o ( δ ) = δ P Y ( k ) D ( P X | Y ( · | k ) | | P X ( · ) ) + o ( δ )
Similarly, we can write
I ( Y ; Z ) = h ( Z ) - y = 1 n P Y ( y ) h ( Z | Y = y ) = h ( Z ) - P Y ( k ) h ( δ ) = h ( δ P Y ( k ) ) - P Y ( k ) h ( δ ) = - δ P Y ( k ) log ( P Y ( k ) ) - Ψ ( 1 - δ P Y ( k ) ) + P Y ( k ) Ψ ( 1 - δ )
where Ψ ( x ) : = x log x which yields
d d δ I ( Y ; Z ) = - Ψ ( P Y ( k ) ) + P Y ( k ) log 1 - δ P Y ( k ) 1 - δ
From the above, we obtain
I ( Y ; Z ) = d d δ I ( Y ; Z ) | δ = 0 δ + o ( δ ) = - δ Ψ ( P Y ( k ) ) + o ( δ )
Clearly from (A3), in order for the filter P Z | Y specified in (A1) and (A2) to belong to D ε ( P X Y ) , we must have
ε δ = P Y ( k ) D ( P X | Y ( · | k ) | | P X ( · ) ) + o ( δ ) δ
and hence from (A4), we have
I ( Y ; Z ) = - Ψ ( P Y ( k ) ) P Y ( k ) D ( P X | Y ( · | k ) | | P X ( · ) ) ε + o ( δ )
This immediately implies that
g 0 ( X ; Y ) = lim ε 0 g ε ( X ; Y ) ε - Ψ ( P Y ( k ) ) P Y ( k ) D ( P X | Y ( · | k ) | | P X ( · ) ) = - log ( P Y ( k ) ) D P X | Y ( · | k ) | | P X ( · )
where we have used the assumption g 0 ( X , Y ) = 0 in the first equality.

Appendix B. Completion of Proof of Theorem 25

To prove that the equality (39) has only one solution p = 1 2 , we first show the following lemma.
Lemma 33. 
Let P and Q be two distributions over X = { ± 1 , ± 2 , , ± k } which satisfy P ( x ) = Q ( - x ) . Let R λ : = λ P + ( 1 - λ ) Q for λ ( 0 , 1 ) . Then
D ( P | | R 1 - λ ) D ( P | | R λ ) < log ( 1 - λ ) log ( λ )
for λ ( 0 , 1 2 ) and
D ( P | | R 1 - λ ) D ( P | | R λ ) > log ( 1 - λ ) log ( λ )
for λ ( 1 2 , 1 ) .
Note that it is easy to see that the map λ D(P||Rλ) is convex and strictly decreasing and hence D ( P | | R λ ) > D ( P | | R 1 - λ ) when λ ( 0 , 1 2 ) and D ( P | | R λ ) < D ( P | | R 1 - λ ) when λ ( 1 2 , 1 ) . Inequality (A6) and (A7) strengthen these monotonic behavior and show that D ( P | | R λ ) > log ( λ ) log ( 1 - λ ) D ( P | | R 1 - λ ) and D ( P | | R λ ) < log ( λ ) log ( 1 - λ ) D ( P | | R 1 - λ ) for λ ( 0 , 1 2 ) and λ ( 1 2 , 1 ) , respectively.
Proof. 
Without loss of generality, we can assume that P ( x ) > 0 for all x X . Let X + : = { x X | P ( X ) > P ( - x ) } , X - : = { x X | P ( X ) < P ( - x ) } and X 0 : = { x X | P ( X ) = P ( - x ) } . We notice that when x X + , then - x X - , and hence | X + | = | X - | = m for a 0 < m k . After relabelling if needed, we can therefore assume that X + = { 1 , 2 , , m } and X - = { - m , , - 2 , - 1 } . We can write
D ( P | | R λ ) = x = - k k log P ( x ) λ P ( x ) + ( 1 - λ ) Q ( x ) = x = - k k log P ( x ) λ P ( x ) + ( 1 - λ ) P ( - x ) = ( a ) x = 1 m P ( x ) log P ( x ) λ P ( x ) + ( 1 - λ ) P ( - x ) + P ( - x ) log P ( - x ) λ P ( - x ) + ( 1 - λ ) P ( x ) = ( b ) x = 1 m P ( x ) log 1 λ + ( 1 - λ ) ζ x + P ( x ) ζ x log 1 λ + ( 1 - λ ) ζ x = ( c ) x = 1 m P ( x ) Ξ ( λ , ζ x ) log 1 λ
where ( a ) follows from the fact that for x X 0 , log P ( x ) R λ ( x ) = 0 for any λ ( 0 , 1 ) , and in ( b ) and ( c ) we introduced ζ x : = P ( - x ) P ( x ) and
Ξ ( λ , ζ ) : = 1 log 1 λ log 1 λ + ( 1 - λ ) ζ + ζ log 1 λ + ( 1 - λ ) ζ
Similarly, we can write
D ( P | | R 1 - λ ) = x = - k k log P ( x ) ( 1 - λ ) P ( x ) + λ Q ( x ) = x = - k k log P ( x ) ( 1 - λ ) P ( x ) + λ P ( - x ) = x = 1 m P ( x ) log P ( x ) ( 1 - λ ) P ( x ) + λ P ( - x ) + P ( - x ) log P ( - x ) ( 1 - λ ) P ( - x ) + λ P ( x ) = x = 1 m P ( x ) log 1 1 - λ + λ ζ x + P ( x ) ζ x log 1 1 - λ + λ ζ x = x = 1 m P ( x ) Ξ ( 1 - λ , ζ x ) log 1 1 - λ
which implies that
D ( P | | R λ ) - log ( λ ) - D ( P | | R 1 - λ ) - log ( 1 - λ ) = x = 1 m P ( x ) Ξ ( λ , ζ x ) - Ξ ( 1 - λ , ζ x )
Hence, in order to show (A6), it suffices to verify that
Φ ( λ , ζ ) : = Ξ ( λ , ζ ) - Ξ ( 1 - λ , ζ ) > 0
for any λ ( 0 , 1 2 ) and ζ ( 1 , ) . Since log ( λ ) log ( 1 - λ ) is always positive for λ ( 0 , 1 2 ) , it suffices to show that
h ( ζ ) : = Φ ( λ , ζ ) log ( 1 - λ ) log ( λ ) > 0
for λ ( 0 , 1 2 ) and ζ ( 1 , ) . We have
h ( ζ ) = A ( λ , ζ ) B ( λ , ζ )
where
A ( λ , ζ ) : = 1 + ζ ( 1 - λ + λ ζ ) 2 ( λ + ( 1 - λ ) ζ ) 2 ζ
and
B ( λ , ζ ) : = λ 2 ( 1 + λ ( λ - 2 ) ( ζ - 1 ) 2 + ζ ( ζ - 1 ) ) log ( λ ) - ( 1 - λ ) 2 ( λ 2 ( ζ - 1 ) 2 + ζ ) log ( 1 - λ ) .
We have
2 ζ 2 B ( λ , ζ ) = 2 λ 2 ( 1 - λ ) 2 log λ 1 - λ < 0
because λ ( 0 , 1 2 ) and hence λ < 1 - λ . This implies that the map ζ B ( λ , ζ ) is concave for any λ ( 0 , 1 2 ) and ζ ( 1 , ) . Moreover, since ζ B ( λ , ζ ) is a quadratic polynomial with negative leading coefficient, it is clear that lim ζ B ( λ , ζ ) = - . Consider now g ( λ ) : = B ( λ , 1 ) = λ 2 log ( λ ) - ( 1 - λ ) 2 log ( 1 - λ ) . We have lim λ 0 g ( λ ) = g ( 1 2 ) = 0 and g ( λ ) = 2 log λ 1 - λ < 0 for λ ( 0 , 1 2 ) . It implies that λ g ( λ ) is concave over ( 0 , 1 2 ) and hence g ( λ ) > 0 over ( 0 , 1 2 ) which implies that B ( λ , 1 ) > 0 . This together with the fact that ζ B ( λ , ζ ) is concave and it approaches to - as ζ imply that there exists a real number c = c ( λ ) > 1 such that B ( λ , ζ ) > 0 for all ζ ( 1 , c ) and B ( λ , ζ ) < 0 for all ζ ( c , ) . Since A ( λ , ζ ) > 0 , it follows from (A10) that ζ h ( ζ ) is convex over ( 1 , c ) and concave over ( c , ) . Since h ( 1 ) = h ( 1 ) = 0 and lim ζ h ( ζ ) = , we can conclude that h ( ζ ) > 0 over ( 1 , ) . That is, Φ ( λ , ζ ) > 0 and thus Ξ ( λ , ζ ) - Ξ ( 1 - λ , ζ ) > 0 , for λ ( 0 , 1 2 ) and ζ ( 1 , ) .
The inequality (A7) can be proved by (A6) and switching λ to 1 - λ . ☐
Letting P ( · ) = P X | Y ( · | 1 ) and Q ( · ) = P X | Y ( · | 0 ) and λ = Pr ( Y = 1 ) = p , we have R p ( x ) = P X ( x ) = p P ( x ) + ( 1 - p ) Q ( x ) and R 1 - p = P X ( - x ) = ( 1 - p ) P ( x ) + p Q ( x ) . Since D ( P X | Y ( · | 0 ) | | P X ( · ) ) = D ( P | | R 1 - p ) , we can conclude from Lemma 33 that
D ( P X | Y ( · | 0 ) | | P X ( · ) ) - log ( 1 - p ) < D ( P X | Y ( · | 1 ) | | P X ( · ) ) - log ( p )
over p ( 0 , 1 2 ) and
D ( P X | Y ( · | 0 ) | | P X ( · ) ) - log ( 1 - p ) > D ( P X | Y ( · | 1 ) | | P X ( · ) ) - log ( p )
over p ( 1 2 , 1 ) , and hence equation (39) has only solution p = 1 2 .

Appendix C. Proof of Theorems 28 and 29

The proof of Theorem 29 does not depend on the proof of Theorem 28, so, there is no harm in proving the former theorem first. The following version of the data-processing inequality will be required.
Lemma 34. 
Let X and Y be absolutely continuous random variables such that X, Y and ( X , Y ) have finite differential entropies. If V is an absolutely continuous random variable independent of X and Y, then
I ( X ; Y + V ) I ( X ; Y )
with equality if and only if X and Y are independent.
Proof. 
Since X -∘- Y -∘- ( Y + V ) , the data processing inequality implies that I ( X ; Y + V ) I ( X ; Y ) . It therefore suffices to show that this inequality is tight if and only X and Y are independent. It is known that data processing inequality is tight if and only if X -∘-(Y+V) -∘-Y. This is equivalent to saying that for any measurable set A R and for P Y + V almost all z, Pr ( X A | Y + V = z , Y = y ) = Pr ( X A | Y + V = z ) . On the other hand, due to the independence of V and ( X , Y ) , we have Pr ( X A | Y + V = z , Y = y ) = Pr ( X A | Y = z - v ) . Hence, the equality holds if and only if Pr ( X A | Y + V = z ) = Pr ( X A | Y = z - v ) which implies that X and Y must be independent. ☐
Lemma 35. 
In the notation of Section 5.1, the function γ I ( Y ; Z γ ) is strictly-decreasing and continuous. Additionally, it satisfies
I ( Y ; Z γ ) 1 2 log 1 + var ( Y ) γ 2
with equality if and only if Y is Gaussian. In particular, I ( Y ; Z γ ) 0 as γ .
Proof. 
Recall that, by assumption b), var ( Y ) is finite. The finiteness of the entropy of Y follows from assumption, the corresponding statement for Y + γ N follows from a routine application of the entropy power inequality [50, Theorem 17.7.3] and the fact that var ( Y + γ N ) = var ( Y ) + γ 2 < , and for ( Y , Y + γ N ) the same conclusion follows by the chain rule for differential entropy. The data processing inequality, as stated in Lemma 34, implies
I ( Y ; Z γ + δ ) I ( Y ; Y + γ N ) = I ( Y ; Z γ )
Clearly Y and Y + γ N are not independent, therefore the inequality is strict and thus γ I ( Y , Z γ ) is strictly-decreasing.
Continuity will be studied for γ = 0 and γ > 0 separately. Recall that h ( γ N ) = 1 2 log ( 2 π e γ 2 ) . In particular, lim γ 0 h ( γ N ) = - . The entropy power inequality shows then that lim γ 0 I ( Y ; Y + γ N ) = . This coincides with the convention I ( Y ; Z 0 ) = I ( Y ; Y ) = . For γ > 0 , let ( γ n ) n 1 be a sequence of positive numbers such that γ n γ . Observe that
I ( Y ; Z γ n ) = h ( Y + γ n N ) - h ( γ n N ) = h ( Y + γ n N ) - 1 2 log ( 2 π e γ n 2 )
Since lim n 1 2 log ( 2 π e γ n 2 ) = 1 2 log ( 2 π e γ 2 ) , we only have to show that h ( Y + γ n N ) h ( Y + γ N ) as n to establish the continuity at γ. This, in fact, follows from de Bruijn’s identity (cf., [50, Theorem 17.7.2]).
Since the channel from Y to Z γ is an additive Gaussian noise channel, we have I ( Y ; Z γ ) 1 2 log 1 + var ( Y ) γ 2 with equality if and only if Y is Gaussian. The claimed limit as γ 0 is clear. ☐
Lemma 36. 
The function γ I ( X ; Z γ ) is strictly-decreasing and continuous. Moreover, I ( X ; Z γ ) 0 when γ .
Proof. 
The proof of the strictly-decreasing behavior of γ I ( X ; Z γ ) is proved as in the previous lemma.
To prove continuity, let γ 0 be fixed. Let ( γ n ) n 1 be any sequence of positive numbers converging to γ. First suppose that γ > 0 . Observe that
I ( X ; Z γ n ) = h ( Y + γ n N ) - h ( Y + γ n N | X )
for all n 1 . As shown in Lemma 35, h ( Y + γ n N ) h ( Y + γ N ) as n . Therefore, it is enough to show that h ( Y + γ n N | X ) h ( Y + γ N | X ) as n . Note that by de Bruijn’s identity, we have h ( Y + γ n N | X = x ) h ( Y + γ N | X = x ) as n for all x R . Note also that since
h ( Z γ n | X = x ) 1 2 log 2 π e var ( Z γ n | x )
we can write
h ( Z γ n | X ) E 1 2 log ( 2 π e var ( Z γ n | X ) ) 1 2 log 2 π e E [ var ( Z γ n | X ) ]
and hence we can apply dominated convergence theorem to show that h ( Y + γ n N | X ) h ( Y + γ N | X ) as n .
To prove the continuity at γ = 0 , we first note that Linder and Zamir [51, Page 2028] showed that h ( Y + γ n N | X = x ) h ( Y | X = x ) as n , then, as before, by dominated convergence theorem we can show that h ( Y + γ n N | X ) h ( Y | X ) . Similarly [51] implies that h ( Y + γ n N ) h ( Y ) . This concludes the proof of the continuity of γ I ( X ; Z γ ) .
Furthermore, by the data processing inequality and previous lemma,
0 I ( X ; Z γ ) I ( Y ; Z γ ) 1 2 log 1 + var ( Y ) γ 2
and hence we conclude that lim γ I ( X ; Z γ ) = 0 . ☐
Proof of Theorem 29. 
The nonnegativity of g ε ( X ; Y ) follows directly from definition.
By Lemma 36, for every 0 < ε I ( X ; Y ) there exists a unique γ ε [ 0 , ) such that I ( X ; Z γ ε ) = ε , so g ε ( X ; Y ) = I ( Y ; Z γ ε ) . Moreover, ε γ ε is strictly decreasing. Since γ I ( Y ; Z γ ) is strictly-decreasing, we conclude that ε g ε ( X ; Y ) is strictly increasing.
The fact that ε γ ε is strictly decreasing, also implies that γ ε as ε 0 . In particular,
lim ε 0 g ε ( X ; Y ) = lim ε 0 I ( Y ; Z γ ε ) = lim γ ε I ( Y ; Z γ ε ) = lim γ I ( Y ; Z γ ) = 0
By the data processing inequality we have that I ( X ; Z γ ) I ( X ; Y ) for all γ 0 , i.e., any filter satisfies the privacy constraint for ε = I ( X ; Y ) . Thus, g I ( X ; Y ) ( X ; Y ) I ( Y ; Y ) = . ☐
In order to prove Theorem 28, we first recall the following theorem by Rényi [52].
Theorem 37 
([52]). If U is an absolutely continuous random variable with density f U ( x ) and if H ( U ) < , then
lim n H ( n - 1 n U ) - log ( n ) = - R f U ( x ) log f U ( x ) d x
provided that the integral on the right hand side exists.
We will need the following consequence of the previous theorem.
Lemma 38. 
If U is an absolutely continuous random variable with density f U ( x ) and if H ( U ) < , then H ( Q M ( U ) ) - M H ( Q M + 1 ( U ) ) - ( M + 1 ) for all M 1 and
lim n H ( Q M ( U ) ) - M = - R f U ( x ) log f U ( x ) d x
provided that the integral on the right hand side exists.
The previous lemma follows from the fact that Q M + 1 ( U ) is constructed by refining the quantization partition for Q M ( U ) .
Lemma 39. 
For any γ 0 ,
lim M I ( X ; Z γ M ) = I ( X ; Z γ ) and lim M I ( Y ; Z γ M ) = I ( Y ; Z γ )
Proof. 
Observe that
I ( X ; Z γ M ) = I ( X ; Q M ( Y + γ N ) ) = H ( Q M ( Y + γ N ) ) - H ( Q M ( Y + γ N ) | X ) = [ H ( Q M ( Y + γ N ) ) - M ] - R f X ( x ) [ H ( Q M ( Y + γ N ) | X = x ) - M ] d x
By the previous lemma, the integrand is decreasing in M, and thus we can take the limit with respect to M inside the integral. Thus,
lim M I ( X ; Z γ M ) = h ( Y + γ N ) - h ( Y + γ N | X ) = I ( X ; Z γ )
The proof for I ( Y ; Z γ M ) is analogous. ☐
Lemma 40. 
Fix M N . Assume that f Y ( y ) C | y | - p for some positive constant C and p > 1 . For integer k and γ 0 , let
p k , γ : = Pr Q M ( Y + γ N ) = k 2 M
Then
p k , γ C 2 ( p - 1 ) M + p k p + 1 { γ > 0 } γ 2 M + 1 k 2 π e - k 2 / 2 2 M + 3 γ 2
Proof. 
The case γ = 0 is trivial, so we assume that γ > 0 . For notational simplicity, let r a = a 2 M for all a Z . Assume that k 0 . Observe that
p k , γ = - - f γ N ( n ) f Y ( y ) 1 r k , r k + 1 ( y + n ) d y d n = - e - n 2 / 2 γ 2 2 π γ 2 Pr Y r k , r k + 1 - n d n
We will estimate the above integral by breaking it up into two pieces.
First, we consider
- r k 2 e - n 2 / 2 γ 2 2 π γ 2 Pr Y r k , r k + 1 - n d n
When n r k 2 , then r k - n r k / 2 . By the assumption on the density of Y,
Pr Y r k , r k + 1 - n C 2 M r k 2 - p
(The previous estimate is the only contribution when γ = 0 .) Therefore,
- r k 2 e - n 2 / 2 γ 2 2 π γ 2 Pr Y r k , r k + 1 - n d n C 2 M r k 2 - p - r k 2 e - n 2 / 2 γ 2 2 π γ 2 d n C 2 ( p - 1 ) M + p k p
Using the trivial bound Pr Y r k , r k + 1 - n 1 and well known estimates for the error function, we obtain that
r k 2 e - n 2 / 2 γ 2 2 π γ 2 Pr Y r k , r k + 1 - n d n < 1 2 π 2 γ r k e - r k 2 / 8 γ 2 = γ 2 M + 1 k 2 π e - k 2 / 2 2 M + 3 γ 2
Therefore,
p k , γ C 2 ( p - 1 ) M + p k p + γ 2 M + 1 k 2 π e - k 2 / 2 2 M + 3 γ 2
The proof for k < 0 is completely analogous. ☐
Lemma 41. 
Fix M N . Assume that f Y ( y ) C | y | - p for some positive constant C and p > 1 . The mapping γ H ( Q M ( Y + γ N ) ) is continuous.
Proof. 
Let ( γ n ) n 1 be a sequence of non-negative real numbers converging to γ 0 . First, we will prove continuity at γ 0 > 0 . Without loss of generality, assume that γ n > 0 for all n N . Define γ * = inf { γ n | n 1 } and γ * = sup { γ n | n 1 } . Clearly 0 < γ * γ * < . Recall that
p k , γ = R e - z 2 / 2 γ 2 2 π γ 2 Pr Y k 2 M , k + 1 2 M - z d z
Since, for all n N and z R ,
e - z 2 / 2 γ n 2 2 π γ n 2 Pr Y k 2 M , k + 1 2 M - z e - z 2 / 2 ( γ * ) 2 2 π γ * 2
the dominated convergence theorem implies that
lim n p k , γ n = p k , γ 0
The previous lemma implies that for all n 0 and | k | > 0 ,
p k , γ n C 2 ( p - 1 ) M + p k p + γ n 2 M + 1 k 2 π e - k 2 / 2 2 M + 3 γ n 2
Thus, for k large enough, p k , γ n A k p for a suitable positive constant A that does not depend on n. Since the function x - x log ( x ) is increasing in [ 0 , 1 / 2 ] , there exists K > 0 such that for | k | > K
- p k , γ n log ( p k , γ n ) A k p log ( A - 1 k p )
Since | k | > K A k p log ( A - 1 k p ) < , for any ϵ > 0 there exists K ϵ such that
| k | > K ϵ A k p log ( A - 1 k p ) < ϵ
In particular, for all n 0 ,
H ( Q ( Y + γ n N ) ) - | k | K ϵ - p k , γ n log ( p k , γ n ) = | k | > K ϵ - p k , γ n log ( p k , γ n ) < ϵ
Therefore, for all n 1 ,
H ( Q ( Y + γ n N ) ) - H ( Q ( Y + γ 0 N ) ) | k | > K ϵ - p k , γ n log ( p k , γ n ) + | k | K ϵ p k , γ 0 log ( p k , γ 0 ) - p k , γ n log ( p k , γ n ) + | k | > K ϵ - p k , γ 0 log ( p k , γ 0 ) ϵ + | k | K ϵ p k , γ 0 log ( p k , γ 0 ) - p k , γ n log ( p k , γ n ) + ϵ
By continuity of the function x - x log ( x ) on [ 0 , 1 ] and equation (A11), we conclude that
lim sup n H ( Q ( Y + γ n N ) ) - H ( Q ( Y + γ 0 N ) ) 3 ϵ
Since ϵ is arbitrary,
lim n H ( Q ( Y + γ n N ) ) = H ( Q ( Y + γ 0 N ) )
as we wanted to prove.
To prove continuity at γ 0 = 0 , observe that equation (A11) holds in this case as well. The rest is analogous to the case γ 0 > 0 . ☐
Lemma 42. 
The functions γ I ( X ; Z γ M ) and γ I ( Y ; Z γ M ) are continuous for each M N .
Proof. 
Since H ( Q M ( Y + γ N ) | Y = y ) and H ( Q M ( Y + γ N ) | X = x ) for x , y R are bounded by M, and f Y | X ( y | x ) satisfies assumption (b), the conclusion follows from the dominated convergence theorem. ☐
Proof of Theorem 28. 
For every M N , let Γ ϵ M : = { γ 0 | I ( X ; Z γ M ) ϵ } . The Markov chain X Y Z γ Z γ M + 1 Z γ M and the data processing inequality imply that
I ( X ; Z γ ) I ( X ; Z γ M + 1 ) I ( X ; Z γ M )
and, in particular,
ϵ = I ( X ; Z γ ϵ ) I ( X ; Z γ ϵ M + 1 ) I ( X ; Z γ ϵ M )
where γ ϵ is as defined in the proof of Theorem 29. This implies then that
γ ϵ Γ ϵ M + 1 Γ ϵ M
and thus
I ( Y ; Z γ ϵ M ) g ϵ , M ( X ; Y )
Taking limits in both sides, Lemma 39 implies
g ϵ ( X ; Y ) = I ( Y ; Z γ ϵ ) lim inf M g ϵ , M ( X ; Y )
Observe that
g ϵ , M ( X ; Y ) = sup γ Γ ϵ M I ( Y ; Z γ M ) sup γ Γ ϵ M I ( Y ; Z γ ) = I ( Y ; Z γ ϵ , m i n M )
where inequality follows from Markovity and γ ϵ , min M : = inf Γ ϵ M γ . By equation (A12), γ ϵ Γ ϵ M + 1 Γ ϵ M and in particular γ ϵ , min M γ ϵ , min M + 1 γ ϵ . Thus, { γ ε , min M } is an increasing sequence in M and bounded from above and, hence, has a limit. Let γ ϵ , min = lim M γ ϵ , min M . Clearly
γ ϵ , min γ ϵ
By the previous lemma we know that I ( X ; Z γ M ) is continuous, so Γ ϵ M is closed for all M N . Thus, we have that γ ϵ , min M = min Γ ϵ M γ and in particular γ ϵ , min M Γ ϵ M . By the inclusion Γ ϵ M + 1 Γ ϵ M , we have then that γ ϵ , min M + n Γ ϵ M for all n N . By closedness of Γ ϵ M we have then that γ ϵ , min Γ ϵ M for all M N . In particular,
I ( X ; Z γ ϵ , min M ) ϵ
for all M N . By Lemma 39,
I ( X ; Z γ ϵ , min ) ϵ = I ( X ; Z γ ϵ )
and by the monotonicity of γ I ( X ; Z γ ) , we obtain that γ ϵ γ ϵ , min . Combining the previous inequality with (A15) we conclude that γ ϵ , min = γ ϵ . Taking limits in the inequality (A14)
lim sup M g ϵ , M ( X ; Y ) lim sup M I ( Y ; Z γ ϵ , min M ) = I ( Y ; Z γ ϵ , min )
Plugging γ ϵ , min = γ ϵ in above we conclude that
lim sup M g ϵ , M ( X ; Y ) I ( Y ; Z γ ϵ ) = g ϵ ( X ; Y )
and therefore lim M g ϵ , M ( X ; Y ) = g ϵ ( X ; Y ) . ☐

References

  1. Asoodeh, S.; Alajaji, F.; Linder, T. Notes on information-theoretic privacy. In Proceedings of the 52nd Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, 30 September–3 October 2014; pp. 1272–1278.
  2. Asoodeh, S.; Alajaji, F.; Linder, T. On maximal correlation, mutual information and data privacy. In Proceedings of the IEEE 14th Canadian Workshop on Information Theory (CWIT), St. John’s, NL, Canada, 6–9 July 2015; pp. 27–31.
  3. Warner, S.L. Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias. J. Am. Stat. Assoc. 1965, 60, 63–69. [Google Scholar] [CrossRef] [PubMed]
  4. Blum, A.; Ligett, K.; Roth, A. A learning theory approach to non-interactive database privacy. In Proceedings of the Fortieth Annual ACM Symposium on the Theory of Computing, Victoria, BC, Canada, 17–20 May 2008; pp. 1123–1127.
  5. Dinur, I.; Nissim, K. Revealing information while preserving privacy. In Proceedings of the Twenty-Second Symposium on Principles of Database Systems, San Diego, CA, USA, 9–11 June 2003; pp. 202–210.
  6. Rubinstein, P.B.; Bartlett, L.; Huang, J.; Taft, N. Learning in a large function space: Privacy-preserving mechanisms for SVM learning. J. Priv. Confid. 2012, 4, 65–100. [Google Scholar]
  7. Duchi, J.C.; Jordan, M.I.; Wainwright, M.J. Privacy aware learning. 2014; arXiv: 1210.2085. [Google Scholar]
  8. Dwork, C.; McSherry, F.; Nissim, K.; Smith, A. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography (TCC’06), New York, NY, USA, 5–7 March 2006; pp. 265–284.
  9. Dwork, C. Differential privacy: A survey of results. In Theory and Applications of Models of Computation, Proceedings of the 5th International Conference, TAMC 2008, Xi’an, China, 25–29 April 2008; Agrawal, M., Du, D., Duan, Z., Li, A., Eds.; Springer: Berlin/Heidelberg, Germany, 2008. Lecture Notes in Computer Science. Volume 4978, pp. 1–19. [Google Scholar]
  10. Dwork, C.; Lei, J. Differential privacy and robust statistics. In Proceedings of the 41st Annual ACM Symposium on the Theory of Computing, Bethesda, MD, USA, 31 May–2 June 2009; pp. 437–442.
  11. Kairouz, P.; Oh, S.; Viswanath, P. Extremal mechanisms for local differential privacy. 2014; arXiv: 1407.1338v2. [Google Scholar]
  12. Calmon, F.P.; Varia, M.; Médard, M.; Christiansen, M.M.; Duffy, K.R.; Tessaro, S. Bounds on inference. In Proceedings of the 51st Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, 2–4 October 2013; pp. 567–574.
  13. Yamamoto, H. A source coding problem for sources with additional outputs to keep secret from the receiver or wiretappers. IEEE Trans. Inf. Theory 1983, 29, 918–923. [Google Scholar] [CrossRef]
  14. Sankar, L.; Rajagopalan, S.; Poor, H. Utility-privacy tradeoffs in databases: An information-theoretic approach. IEEE Trans. Inf. Forensics Secur. 2013, 8, 838–852. [Google Scholar] [CrossRef]
  15. Tandon, R.; Sankar, L.; Poor, H. Discriminatory lossy source coding: side information privacy. IEEE Trans. Inf. Theory 2013, 59, 5665–5677. [Google Scholar] [CrossRef]
  16. Calmon, F.; Fawaz, N. Privacy against statistical inference. In Proceedings of the 50th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, 1–5 October 2012; pp. 1401–1408.
  17. Rebollo-Monedero, D.; Forne, J.; Domingo-Ferrer, J. From t-closeness-like privacy to postrandomization via information theory. IEEE Trans. Knowl. Data Eng. 2010, 22, 1623–1636. [Google Scholar] [CrossRef]
  18. Makhdoumi, A.; Salamatian, S.; Fawaz, N.; Médard, M. From the information bottleneck to the privacy funnel. In Proceedings of the IEEE Information Theory Workshop (ITW), Hobart, Australia, 2–5 November 2014; pp. 501–505.
  19. Tishby, N.; Pereira, F.C.; Bialek, W. The information bottleneck method. 2000; arXiv: physics/0004057. [Google Scholar]
  20. Calmon, F.P.; Makhdoumi, A.; Médard, M. Fundamental limits of perfect privacy. In Proceedings of the IEEE Int. Symp. Inf. Theory (ISIT), Hong Kong, China, 14–19 June 2015; pp. 1796–1800.
  21. Wyner, A.D. The Wire-Tap Channel. Bell Syst. Tech. J. 1975, 54, 1355–1387. [Google Scholar] [CrossRef]
  22. Makhdoumi, A.; Fawaz, N. Privacy-utility tradeoff under statistical uncertainty. In Proceedings of the 51st Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, 2–4 October 2013; pp. 1627–1634.
  23. Li, C.T.; El Gamal, A. Maximal correlation secrecy. 2015; arXiv: 1412.5374. [Google Scholar]
  24. Ahlswede, R.; Gács, P. Spreading of sets in product spaces and hypercontraction of the Markov operator. Ann. Probab. 1976, 4, 925–939. [Google Scholar] [CrossRef]
  25. Anantharam, V.; Gohari, A.; Kamath, S.; Nair, C. On maximal correlation, hypercontractivity, and the data processing inequality studied by Erkip and Cover. 2014; arXiv:1304.6133v1. [Google Scholar]
  26. Courtade, T. Information masking and amplification: The source coding setting. In Proceedings of the IEEE Int. Symp. Inf. Theory (ISIT), Boston, MA, USA, 1–6 July 2012; pp. 189–193.
  27. Goldwasser, S.; Micali, S. Probabilistic encryption. J. Comput. Syst. Sci. 1984, 28, 270–299. [Google Scholar] [CrossRef]
  28. Rockafellar, R.T. Convex Analysis; Princeton Univerity Press: Princeton, NJ, USA, 1997. [Google Scholar]
  29. Csiszár, I.; Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems; Cambridge University Press: Cambridge, UK, 2011. [Google Scholar]
  30. Shulman, N.; Feder, M. The uniform distribution as a universal prior. IEEE Trans. Inf. Theory 2004, 50, 1356–1362. [Google Scholar] [CrossRef]
  31. Rudin, W. Real and Complex Analysis, 3rd ed.; McGraw Hill: New York, NY, USA, 1987. [Google Scholar]
  32. Gebelein, H. Das statistische Problem der Korrelation als Variations- und Eigenwert-problem und sein Zusammenhang mit der Ausgleichungsrechnung. Zeitschrift f ur Angewandte Mathematik und Mechanik 1941, 21, 364–379. (In German) [Google Scholar] [CrossRef]
  33. Hirschfeld, H.O. A connection between correlation and contingency. Camb. Philos. Soc. 1935, 31, 520–524. [Google Scholar] [CrossRef]
  34. Rényi, A. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica 1959, 10, 441–451. [Google Scholar] [CrossRef]
  35. Linfoot, E.H. An informational measure of correlation. Inf. Control 1957, 1, 85–89. [Google Scholar] [CrossRef]
  36. Csiszár, I. Information-type measures of difference of probability distributions and indirect observation. Studia Scientiarum Mathematicarum Hungarica 1967, 2, 229–318. [Google Scholar]
  37. Zhao, L. Common Randomness, Efficiency, and Actions. Ph.D. Thesis, Stanford University, Stanford, CA, USA, 2011. [Google Scholar]
  38. Berger, T.; Yeung, R. Multiterminal source encoding with encoder breakdown. IEEE Trans. Inf. Theory 1989, 35, 237–244. [Google Scholar] [CrossRef]
  39. Kim, Y.H.; Sutivong, A.; Cover, T. State mplification. IEEE Trans. Inf. Theory 2008, 54, 1850–1859. [Google Scholar] [CrossRef]
  40. Merhav, N.; Shamai, S. Information rates subject to state masking. IEEE Trans. Inf. Theory 2007, 53, 2254–2261. [Google Scholar] [CrossRef]
  41. Ahlswede, R.; Körner, J. Source coding with side information and a converse for degraded broadcast channels. IEEE Trans. Inf. Theory 1975, 21, 629–637. [Google Scholar] [CrossRef]
  42. Kim, Y.H.; El Gamal, A. Network Information Theory; Cambridge University Press: Cambridge, UK, 2012. [Google Scholar]
  43. Asoodeh, S.; Alajaji, F.; Linder, T. Lossless secure source coding, Yamamoto’s setting. In Proceedings of the 53rd Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, 30 September–2 October 2015.
  44. Raginsky, M. Logarithmic Sobolev inequalities and strong data processing theorems for discrete channels. In Proceedings of the IEEE Int. Sym. Inf. Theory (ISIT), Istanbul, Turkey, 7–12 July 2013; pp. 419–423.
  45. Geng, Y.; Nair, C.; Shamai, S.; Wang, Z.V. On broadcast channels with binary inputs and symmetric outputs. IEEE Trans. Inf. Theory 2013, 59, 6980–6989. [Google Scholar] [CrossRef]
  46. Sutskover, I.; Shamai, S.; Ziv, J. Extremes of information combining. IEEE Trans. Inf. Theory 2005, 51, 1313–1325. [Google Scholar] [CrossRef]
  47. Alajaji, F.; Chen, P.N. Information Theory for Single User Systems, Part I. Course Notes, Queen’s University. Available online: http://www.mast.queensu.ca/math474/it-lecture-notes.pdf (accessed on 4 March 2015).
  48. Chayat, N.; Shamai, S. Extension of an entropy property for binary input memoryless symmetric channels. IEEE Trans.Inf. Theory 1989, 35, 1077–1079. [Google Scholar] [CrossRef]
  49. Oohama, Y. Gaussian multiterminal source coding. IEEE Trans. Inf. Theory 1997, 43, 2254–2261. [Google Scholar] [CrossRef]
  50. Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley: New York, NY, USA, 2006. [Google Scholar]
  51. Linder, T.; Zamir, R. On the asymptotic tightness of the Shannon lower bound. IEEE Trans. Inf. Theory 2008, 40, 2026–2031. [Google Scholar] [CrossRef]
  52. Rényi, A. On the dimension and entropy of probability distributions. cta Mathematica Academiae Scientiarum Hungarica 1959, 10, 193–215. [Google Scholar] [CrossRef]
Figure 1. Information-theoretic privacy.
Figure 1. Information-theoretic privacy.
Information 07 00015 g001
Figure 2. Privacy filter that achieves the lower bound in (4) where Z δ is the output of an erasure privacy filter with erasure probability specified in (5).
Figure 2. Privacy filter that achieves the lower bound in (4) where Z δ is the output of an erasure privacy filter with erasure probability specified in (5).
Information 07 00015 g002
Figure 3. The region of g ε ( X ; Y ) in terms of ε specified by (4).
Figure 3. The region of g ε ( X ; Y ) in terms of ε specified by (4).
Information 07 00015 g003
Figure 4. Optimal privacy filter for P Y | X = B S C ( α ) with uniform X where δ ( ε , α ) is specified in (40).
Figure 4. Optimal privacy filter for P Y | X = B S C ( α ) with uniform X where δ ( ε , α ) is specified in (40).
Information 07 00015 g004
Figure 5. Optimal privacy filter for P Y | X = B E C ( δ ) where δ ( ε , α ) is specified in (42).
Figure 5. Optimal privacy filter for P Y | X = B E C ( δ ) where δ ( ε , α ) is specified in (42).
Information 07 00015 g005
Figure 6. The privacy filter associated with (A1) and (A2) with k = 1 . We have P Z | Y ( · | 1 ) = Bernoulli ( δ ) and P Z | Y ( · | y ) = Bernoulli ( 0 ) for y { 2 , 3 , , n } .
Figure 6. The privacy filter associated with (A1) and (A2) with k = 1 . We have P Z | Y ( · | 1 ) = Bernoulli ( δ ) and P Z | Y ( · | y ) = Bernoulli ( 0 ) for y { 2 , 3 , , n } .
Information 07 00015 g006
Information EISSN 2078-2489 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top