The Double-Sided Information Bottleneck Function ^†

Abstract

A double-sided variant of the information bottleneck method is considered. Let $(\mathsf{X},\mathsf{Y})$ be a bivariate source characterized by a joint pmf ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$. The problem is to find two independent channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ (setting the Markovian structure $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}\to \mathsf{V}$), that maximize $I(\mathsf{U};\mathsf{V})$ subject to constraints on the relevant mutual information expressions: $I(\mathsf{U};\mathsf{X})$ and $I(\mathsf{V};\mathsf{Y})$. For jointly Gaussian $\mathsf{X}$ and $\mathsf{Y}$, we show that Gaussian channels are optimal in the low-SNR regime but not for general SNR. Similarly, it is shown that for a doubly symmetric binary source, binary symmetric channels are optimal when the correlation is low and are suboptimal for high correlations. We conjecture that Z and S channels are optimal when the correlation is 1 (i.e., $\mathsf{X}=\mathsf{Y}$) and provide supporting numerical evidence. Furthermore, we present a Blahut–Arimoto type alternating maximization algorithm and demonstrate its performance for a representative setting. This problem is closely related to the domain of biclustering.

1. Introduction

The information bottleneck (IB) method [1] plays a central role in advanced lossy source compression. The analysis of classical source coding algorithms is mainly approached via the rate-distortion theory, where a fidelity measure must be defined. However, specifying an appropriate distortion measure in many real-world applications is challenging and sometimes infeasible. The IB framework introduces an essentially different concept, where another variable is provided, which carries the relevant information in the data to be compressed. The quality of the reconstructed sequence is measured via the mutual information metric between the reconstructed data and the relevance variables. Thus, the IB method provides a universal fidelity measure.

In this work, we extend and generalize the IB method by imposing an additional bottleneck constraint on the relevant variable and considering noisy observation of the source. In particular, let $(\mathsf{X},\mathsf{Y})$ be a bivariate source characterized by a fixed joint probability law ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$ and consider all Markov chains $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}\to \mathsf{V}$. The Double-Sided Information Bottleneck (DSIB) function is defined as [2]:

$$R_{{\mathsf{P}}_{\mathsf{X}\mathsf{Y}}}(C_u,C_v)\triangleq maxI(\mathsf{U};\mathsf{V}),$$

(1)

where the maximization is over all ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ satisfying $I(\mathsf{U};\mathsf{X})\le C_u$ and $I(\mathsf{V};\mathsf{Y})\le C_v$. This problem is illustrated in Figure 1. In our study, we aim to determine the maximum value and the achieving conditional distributions $({\mathsf{P}}_{\mathsf{U}|\mathsf{X}},{\mathsf{P}}_{\mathsf{V}|\mathsf{Y}})$ (test channels) of (1) for various fixed sources ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$ and constraints $C_u$ and $C_v$.

Figure 1. Block diagram of the Double-Sided Information Bottleneck function.

The problem we consider originates from the domain of clustering. Clustering is applied to organize similar entities in unsupervised learning [3]. It has numerous practical applications in data science, such as: joint word-document clustering, gene expression [4], and pattern recognition. The data in those applications are arranged as a contingency table. Usually, clustering is performed on one dimension of the table, but sometimes it is helpful to apply clustering on both dimensions of the contingency table [5], for example, when there is a strong correlation between the rows and the columns of the table or when high-dimensional sparse structures are handled. The input and output of a typical biclustering algorithm are illustrated in Figure 2. Consider an $S\times T$ data matrix $(a_{st})$. Find partitions ${\mathcal{B}}_k\subseteq \{1,\cdots ,S\}$ and ${\mathcal{C}}_l\subseteq \{1,\cdots ,T\}$, $k=1,\cdots ,K$, $l=1,\cdots ,L$ such that all elements of the “biclusters” [6] ${(a_{st})}_{s\in {\mathcal{B}}_k,t\in {\mathcal{C}}_l}$ are homogeneous. The measure of homogeneity depends on the application.

Figure 2. Illustration of a typical biclustering algorithm.

This problem can also be motivated by a remote source coding setting. Consider a latent random variable $\mathsf{W}$, which satisfies $\mathsf{U}\leftarrow \mathsf{X}\leftarrow \mathsf{W}\to \mathsf{Y}\to \mathsf{V}$ and represents a source of information. We have two users that observe noisy versions of $\mathsf{W}$, i.e., $\mathsf{X}$ and $\mathsf{Y}$. Those users try to compress the observed noisy data so that their reconstructed versions, $\mathsf{U}$ and $\mathsf{V}$, will be comparable under the maximum mutual information metric. The problem we consider also bears practical applications. Imagine a distributed sensor network where the different edges measure a noisy version of a particular signal but are not allowed to communicate with each other. Each of the nodes performs compression of the received signal. Under the DSIB framework, we can find the optimal compression schemes that preserve the reconstructed symbols’ proximity subject to the mutual information measure.

Dhillon et al. [7] initiated an information-theoretic approach to biclustering. They have regarded the normalized non-negative contingency table as a joint probability distribution matrix of two random variables. Mutual information was proposed as a measure for optimal co-clustering. An optimization algorithm was presented that intertwines both row and column clustering at all stages. Distributed clustering from a proper information-theoretic perspective was first explicitly considered by Pichler et al. [2]. Consider the model illustrated in Figure 3. A bivariate memory-less source with joint law ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$ generates n i.i.d. copies $({\mathsf{X}}^n,{\mathsf{Y}}^n)$ of $(\mathsf{X},\mathsf{Y})$. Each sequence is observed at two different encoders, and each encoder generates a description of the observed sequence, $f_n({\mathsf{X}}^n)$ and $g_n({\mathsf{Y}}^n)$. The objective is to construct the mappings $f_n$ and $g_n$ such that the normalized mutual information between the descriptions would be maximal while the description coding has bounded rate constraints. Single-letter inner and outer bounds for a general ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$ were derived. An example of a doubly symmetric binary source (DSBS) was given, and several converse results were established. Furthermore, connections were made to the standard IB [1] and the multiple description CEO problems [8]. In addition, the equivalence of information-theoretic biclustering problem to hypothesis testing against independence with multiterminal data compression and a pattern recognition problem was established in [9,10], respectively.

Figure 3. Block diagram of the information-theoretic biclustering problem.

The DSIB problem addressed in our paper is, in fact, a single-letter version of the distributed clustering setup [2]. The inner bound in [2] coincides with our problem definition. Moreover, if the Markov condition $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}\to \mathsf{Z}$ is imposed on the multi-letter variant, then those problems are equivalent. A similar setting, but with a maximal correlation criterion between the reconstructed random variables, has been considered in [11,12]. Furthermore, it is sometimes the case that the optimal biclustering problem is more straightforward to solve than its standard, single-sided, clustering counterpart. For example, the Courtade–Kumar conjecture [13] for the standard single-sided clustering setting was ultimately proven for the biclustering setting [14]. A particular case, where $(\mathsf{X},\mathsf{Y})$ are drawn from DSBS distribution and the mappings $f_n$ and $g_n$ are restricted to be Boolean functions, was addressed in [14]. The bound $I(f_n({\mathsf{X}}^n);g_n({\mathsf{Y}}^n))\le I(\mathsf{X};\mathsf{Y})$ was established, which is tight if and only if $f_n$ and $g_n$ are dictator functions.

1.1. Related Work

Our work extends the celebrated standard (single-sided) IB (SSIB) method introduced by Tishby et al. [1]. Indeed, consider the problem illustrated in Figure 4. This single-sided counterpart of our work is essentially a remote source coding problem [15,16,17], choosing the distortion measure as the logarithmic loss. The random variable $\mathsf{U}$ represents the noisy version ($\mathsf{X}$) of the source ($\mathsf{Y}$) with a constrained number of bits ($I(\mathsf{U};\mathsf{X})\le C$), and the goal is to maximize the relevant information in $\mathsf{U}$ regarding $\mathsf{Y}$ (measured by the mutual information between $\mathsf{Y}$ and $\mathsf{U}$). In the standard IB setup, $I(\mathsf{U};\mathsf{X})$ is referred to as the complexity of $\mathsf{U}$, and $I(\mathsf{Y};\mathsf{U})$ is referred to as the relevance of $\mathsf{U}$.

Figure 4. Block diagram of the Single-Sided Information Bottleneck function.

For the particular case where $(\mathsf{U},\mathsf{X},\mathsf{Y})$ are discrete, an optimal ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ can be found by iteratively solving a set of self-consistent equations. A generalized Blahut–Arimoto algorithm [18,19,20,21] was proposed to solve those equations. The optimal test-channel ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ was characterized using a variation principle in [1]. A particular case of deterministic mappings from $\mathsf{X}$ to $\mathsf{U}$ was considered in [22], and algorithms that find those mappings were described.

Several representing scenarios have been considered for the SSIB problem. The setting where the pair $(\mathsf{X},\mathsf{Y})$ is a doubly symmetric binary source (DSBS) with transition probability p was addressed from various perspectives in [17,23,24]. Utilizing Mrs. Gerber’s Lemma (MGL) [25], one can show that the optimal test-channel for the DSBS setting is a BSC. The case where $(\mathbf{X},\mathbf{Y})$ are jointly multivariate Gaussians in the SSIB framework was first considered in [26]. It was shown that the optimal distribution of $(\mathbf{U},\mathbf{X},\mathbf{Y})$ is also jointly Gaussian. The optimality of the Gaussian test channel can be proven using EPI [27], or exploiting I-MMSE and Single Crossing Property [28]. Moreover, the proof can be easily extended to jointly Gaussian random vectors $(\mathbf{X},\mathbf{Y})$ under the I-MMSE framework [29].

In a more general scenario where $\mathsf{X}=\mathsf{Y}+\mathsf{Z}$ and only $\mathsf{Z}$ is fixed to be Gaussian, it was shown that discrete signaling with deterministic quantizers as test-channel sometimes outperforms Gaussian ${\mathsf{P}}_{\mathsf{X}}$ [30]. This exciting observation leads to a conjecture that discrete inputs are optimal for this general setting and may have a connection to the input amplitude constrained AWGN channels where it was already established that discrete input distributions are optimal [31,32,33]. One reason for the optimality of discrete distributions stems from the observation that constraining the compression rate limits the usable input amplitude. However, as far as we know, it remains an open problem.

There are various related problems considered in the literature that are equivalent to the SSIB; namely, they share a similar single-letter optimization problem. In the conditional entropy bound (CEB) function, studied in [17], given a fixed bivariate source $(\mathsf{X},\mathsf{Y})$ and an equality constraint on the conditional entropy of $\mathsf{X}$ given $\mathsf{U}$, the goal is to minimize the conditional entropy of $\mathsf{Y}$ given $\mathsf{U}$ over the set of $\mathsf{U}$ such that $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}$ constitute a Markov chain. One can show that CEB is equivalent to SSIB. The common reconstruction CR setting [34] is a source coding with a side-information problem, also known as Wyner–Ziv coding, as depicted in Figure 5; with an additional constraint, the encoder can reconstruct the same sequence as the decoder. Additional assumption of log-loss fidelity results in a single-letter rate-distortion region equivalent to the SSIB. In the problem of information combining (IC) [23,35], motivated by message combining in LDPC decoders, a source of information, ${\mathsf{P}}_{\mathsf{Y}}$, is observed through two test-channels ${\mathsf{P}}_{\mathsf{X}|\mathsf{Y}}$ and ${\mathsf{P}}_{\mathsf{Z}|\mathsf{Y}}$. The IC framework aims to design those channels in two extreme approaches. For the first, IC asks what those channels should be to make the output pair $(\mathsf{X},\mathsf{Z})$ maximally informative regarding $\mathsf{Y}$. On the contrary, IC also considers how to design ${\mathsf{P}}_{\mathsf{X}|\mathsf{Y}}$ and ${\mathsf{P}}_{\mathsf{Z}|\mathsf{Y}}$ to minimize the information in $(\mathsf{X},\mathsf{Z})$ regarding $\mathsf{Y}$. The problem of minimizing IC can be shown to be equivalent to the SSIB. In fact, if $(\mathsf{X},\mathsf{Y})$ is a DSBS, then by [23], ${\mathsf{P}}_{\mathsf{Z}|\mathsf{Y}}$ is a binary symmetric channel (BSC), recovering similar results from (Section IV.A of [17]).

Figure 5. Block diagram of Source Coding with Side Information.

The IB method has been extended to various network topologies. A multilayer extension of the IB method is depicted in Figure 6. This model was first considered in [36]. A multivariate source $(\mathsf{X},{\mathsf{Y}}_1,\cdots ,{\mathsf{Y}}_L)$ generates a sequence of n i.i.d. copies $({\mathsf{X}}^n,{\mathsf{Y}}_1^n,\cdots ,{\mathsf{Y}}_L^n)$. The receiver has access only to the sequence ${\mathsf{X}}^n$ while $({\mathsf{Y}}_1^n,\cdots ,{\mathsf{Y}}_L^n)$ are hidden. The decoder performs a consecutive L-stage compression of the observed sequence. The representation at step k must be maximally informative about the respective hidden sequence ${\mathsf{Y}}_k$, $k\in \{1,2,\cdots ,L\}$. This setup is highly motivated by the structure of deep neural networks. Specific results were established for the binary and Gaussian sources.

Figure 6. Block diagram of the Multi-Layer IB.

The model depicted in Figure 7 represents a multiterminal extension of the standard IB. A set of receivers observe noisy versions $({\mathsf{X}}_1,{\mathsf{X}}_2,\cdots ,{\mathsf{X}}_K)$ of some source of information $\mathsf{Y}$. The channel outputs $({\mathsf{X}}_1,{\mathsf{X}}_2,\cdots ,{\mathsf{X}}_K)$ are conditionally independent given $\mathsf{Y}$. The receivers are connected to the central processing unit through noiseless but limited-capacity backhaul links. The central processor aims to attain a good prediction $\widehat{\mathsf{Y}}$ of the source $\mathsf{Y}$ based on compressed representations of the noisy version of $\mathsf{Y}$ obtained from the receivers. The quality of prediction is measured via the mutual information merit between $\mathsf{Y}$ and $\widehat{\mathsf{Y}}$. The Distributive IB setting is essentially a CEO source coding problem under logarithmic loss (log-loss) distortion measure [37]. The case where $(\mathsf{X},{\mathsf{Y}}_1,\cdots ,{\mathsf{Y}}_K)$ are jointly Gaussian random variables was addressed in [20], and a Blahut–Arimoto-type algorithm was proposed. An optimized algorithm to design quantizers was proposed in [38].

Figure 7. Block diagram of the Distributive IB.

A cooperative multiterminal extension of the IB method was proposed in [39]. Let $({\mathsf{X}}_1^n,{\mathsf{X}}_2^n,{\mathsf{Y}}^n)$ be n i.i.d. copies of the multivariate source $({\mathsf{X}}_1,{\mathsf{X}}_2,\mathsf{Y})$. The sequences ${\mathsf{X}}_1^n$ and ${\mathsf{X}}_2^n$ are observed at encoders 1 and 2, respectively. Each encoder sends a representation of the observed sequence through a noiseless yet rate-limited link to the other encoder and the mutual decoder. The decoder attempts to reconstruct the latent representation sequence ${\mathsf{Y}}^n$ based on the received descriptions. As shown in Figure 8, this setup differs from the CEO setup [40] since the encoders can cooperate during the transmission. The set of all feasible rates of complexity and relevance were characterized, and specific regions for the binary and Gaussian sources were established. There are many additional variations of multi-user IB in the literature [20,26,35,36,37,39,40,41,42,43,44].

Figure 8. Block diagram of the Collaborative IB.

The IB problem connects to many timely aspects, such as capital investment [43], distributed learning [45], deep learning [46,47,48,49,50,51,52], and convolutional neural networks [53,54]. Moreover, it has been recently shown that the IB method can be used to reduce the data transfer rate and computational complexity in 5G LDPC decoders [55,56]. The IB method has also been connected with constructing good polar codes [57]. Due to the exponential output-alphabet growth of polarized channels, it becomes demanding to compute their capacities to identify the location of “frozen bits". Quantization is employed in order to reduce the computation complexity. The quality of the quantization scheme is assessed via mutual information preservation. It can be shown that the corresponding IB problem upper bounds the quantization technique. Quantization algorithms based upon the IB method were considered in [58,59,60]. Furthermore, a relationship between the KL means algorithm and the IB method has been discovered in [61].

A recent comprehensive tutorial on the IB method and related problems is given in [24]. Applications of IB problem in machine learning are detailed in [26,45,46,47,51,52,62].

1.2. Notations

Throughout the paper, random variables are denoted using a sans-serif font, e.g., $\mathsf{X}$, their realizations are denoted by the respective lower-case letters, e.g., x, and their alphabets are denoted by the respective calligraphic letters, e.g., $\mathcal{X}$. Let ${\mathcal{X}}^n$ stand for the set of all n-tuples of elements from $\mathcal{X}$. An element from ${\mathcal{X}}^n$ is denoted by $x^n=(x_1,x_2,\cdots ,x_n)$ and substrings are denoted by $x_i^j=(x_i,x_{i+1},\cdots ,x_j)$. The cardinality of a finite set, say $\mathcal{X}$, is denoted by $\left|\mathcal{X}\right|$. The probability mass function (pmf) of $\mathsf{X}$, the joint pmf of $\mathsf{X}$ and $\mathsf{Y}$, and the conditional pmf of $\mathsf{X}$ given $\mathsf{Y}$ are denoted by ${\mathsf{P}}_{\mathsf{X}}$, ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$, and ${\mathsf{P}}_{\mathsf{X}|\mathsf{Y}}$, respectively. The expectation of $\mathsf{X}$ is denoted by ${\mathbb{E}}_{}\left[\mathsf{X}\right]$. The probability of an event $\mathcal{E}$ is denoted as $\mathsf{P}\left(\mathcal{E}\right)$.

Let $\mathsf{X}$ and $\mathsf{Y}$ be an n-ary and m-ary random variables, respectively. The marginal probability vector is denoted by a lowercase boldface letter, i.e.,

$$\mathbf{q}\triangleq {\left(\mathsf{P}\left(\mathsf{X}=1\right),\mathsf{P}\left(\mathsf{X}=2\right),\cdots ,\mathsf{P}\left(\mathsf{X}=n\right)\right)}^T.$$

(2)

The probability vector of an n-ary uniform random variable is denoted by ${\mathbf{u}}_n$. We denote by T the transition matrix from $\mathsf{X}$ to $\mathsf{Y}$, i.e.,

$$T_{ij}\triangleq \mathsf{P}\left(\mathsf{Y}=i|\mathsf{X}=j\right),1\le i\le m,1\le j\le n.$$

(3)

The entropy of n-ary probability vector $\mathbf{q}$ is given by $h(\mathbf{q})$, where

$$h(\mathbf{q})\triangleq -\sum _{i=1}^n{q}_{i}log{q}_i.$$

(4)

Throughout this paper all logarithms are taken to base 2 unless stated otherwise. We denote the ones complemented with a bar, i.e., $\overline{x}=1-x$. The binary convolution of $x,y\in [0,1]$ is defined as $x\ast y\triangleq x\overline{y}+\overline{x}y$. The binary entropy function is defined by $h_b(p):[0,1]\to [0,1]$, i.e., $h_b(p)\triangleq -plogp-\overline{p}log\overline{p}$, and $h_b^{-1}(·)$ its inverse, restricted to $[0,1/2]$.

Let $\mathsf{X}$ and $\mathsf{Y}$ be a pair of random variables with joint pmf ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$ and marginal pmfs ${\mathsf{P}}_{\mathsf{X}}={\mathbf{q}}_x$ and ${\mathsf{P}}_{\mathsf{Y}}={\mathbf{q}}_y$. Furthermore, let T ($\overline{T}$) be the transition matrix from $\mathsf{X}$ ($\mathsf{Y}$) to $\mathsf{Y}$ ($\mathsf{X}$). The mutual information between $\mathsf{X}$ and $\mathsf{Y}$ is defined as:

$$I(\mathsf{X};\mathsf{Y})=I({\mathbf{q}}_{\mathsf{x}},T)=I({\mathbf{q}}_{\mathsf{y}},\overline{T})=\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{\mathsf{P}}_{\mathsf{X}\mathsf{Y}}(x,y)log\frac{{\mathsf{P}}_{\mathsf{X}\mathsf{Y}}(x,y)}{{\mathsf{P}}_{\mathsf{X}}(x){\mathsf{P}}_{\mathsf{Y}}(y)}.$$

(5)

1.3. Paper Outline

Section 2 gives a proper definition of the DSIB optimization problem, mentions various results directly related to this work, and provides some general preliminary results. The spotlight of Section 3 is on the binary $(\mathsf{X},\mathsf{Y})$, where we derive bounds on the respective DSIB function and show a complete characterization for extreme scenarios. The jointly Gaussian $(\mathsf{X},\mathsf{Y})$ is considered in Section 4, where an elegant representation of an objective function is presented, and complete characterization in the low-SNR regime is established. A Blahut–Arimoto-type alternating maximization algorithm will be presented in Section 5. Representative numerical evaluation of the bounds and the proposed algorithm will be provided in Section 6. Finally, a summary and possible future directions will be described in Section 7. The prolonged proofs are postponed to the Appendix A.

2. Problem Formulation and Basic Properties

The DSIB function is a multi-terminal extension of the standard IB [1]. First, we briefly remind the latter’s definition and give related results that will be utilized for its double-sided counterpart. Then, we provide a proper definition of the DSIB optimization problem and present some general preliminaries.

2.1. The Single-Sided Information Bottleneck (SSIB) Function

Definition 1

(SSIB). Let $(\mathsf{X},\mathsf{V})$ be a pair of random variables with $\left|\mathcal{X}\right|=n$, $\left|\mathcal{V}\right|=m$, and fixed ${\mathsf{P}}_{\mathsf{X}\mathsf{V}}$. Denote by $\mathbf{q}$ the marginal probability vector of $\mathsf{X}$, and let T be the transition matrix from $\mathsf{X}$ to $\mathsf{V}$, i.e.,

$$T_{ij}\triangleq \mathsf{P}\left(\mathsf{V}=i|\mathsf{X}=j\right),1\le i\le m,1\le j\le n.$$

Consider all random variables $\mathsf{U}$ satisfying the Markov chain $\mathsf{U}\to \mathsf{X}\to \mathsf{V}$. The SSIB function is defined as:

$$\begin{array}{cccccc}\hfill & {\widehat{R}}_T(\mathbf{q},C)\triangleq \hfill & \hfill & \underset{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}}{\mathit{maximize}}\hfill & \hfill & I(\mathsf{U};\mathsf{V})\hfill \\ \hfill & \hfill & \hfill & \mathit{subject}\mathit{to}\hfill & \hfill & I(\mathsf{X};\mathsf{U})\le C.\hfill \end{array}$$

(6)

Remark 1.

The SSIB problem defined in (6) is equivalent (has similar solution) to the CEB problem considered in [17].

Although the optimization problem in (6) is well defined, the auxiliary random variable $\mathsf{U}$ may have an unbounded alphabet. The following lemma provides an upper bound on the cardinality of $\mathcal{U}$, thus relaxing the optimization domain.

Lemma 1

(Lemma 2.2 of [17]). The optimization over $\mathsf{U}$ in (6) can be restricted to $\left|\mathcal{U}\right|\le n+1$.

Remark 2.

A tighter bound, namely $\left|\mathcal{U}\right|\le n$, was previously proved in [63] for the corresponding dual problem, namely, the IB Lagrangian. However, since ${\widehat{R}}_T(\mathbf{q},C)$ is generally not a strictly convex function of C, it cannot be directly applied for the primal problem (6).

Note that the SSIB optimization problem (6) is basically a convex function maximization over a convex set; thus, the maximum is attained on the boundary of the set.

Lemma 2

(Theorem 2.5 of [17]). The inequality constraint in (6) can be replaced by equality constraint, i.e., $I(\mathsf{X};\mathsf{U})=C$.

2.2. The Double-Sided Information Bottleneck (DSIB) Function

Definition 2

(DSIB). Let $(\mathsf{X},\mathsf{Y})$ be a pair of random variables with $\left|\mathcal{X}\right|=n$, $\left|\mathcal{Y}\right|=m$ and fixed ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$. Consider all the random variables $\mathsf{U}$ and $\mathsf{V}$ satisfying the Markov chain $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}\to \mathsf{V}$. The DSIB function $R:[0,H(\mathsf{X})]\times [0,H(\mathsf{Y})]\to {\mathbb{R}}_{+}$ is defined as:

$$\begin{array}{cccccc}\hfill & R_{{\mathsf{P}}_{\mathsf{X}\mathsf{Y}}4pt}(C_u,C_v)\triangleq \hfill & \hfill & \underset{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}},{\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}}{\mathit{maximize}}\hfill & \hfill & I(\mathsf{U};\mathsf{V})\hfill \\ \hfill & \hfill & \hfill & \mathit{subject}\mathit{to}\hfill & \hfill & I(\mathsf{X};\mathsf{U})\le C_u\mathit{and}I(\mathsf{Y};\mathsf{V})\le C_v.\hfill \end{array}$$

(7)

The achieving conditional distributions ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ will be termed as the optimal test-channels. Occasionally, we will drop the subscript denoting the particular choice of the bivariate source ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$.

Note that (7) can be expressed in the following equivalent form:

$$\begin{array}{ccccccc}\hfill & R(C_u,C_v)\triangleq \hfill & \hfill & \underset{{\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}}{\mathrm{maximize}}\hfill & \hfill & \underset{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}}{\mathrm{maximize}}\hfill & \hfill I(\mathsf{U};\mathsf{V}).\\ \hfill & \hfill & \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \mathrm{subject}\mathrm{to}\hfill \\ \hfill & \hfill & & I(\mathsf{Y};\mathsf{V})\le C_v\hfill & \hfill & I(\mathsf{X};\mathsf{U})\le C_u\hfill \end{array}$$

(8)

Evidently, we can define (8) using (6). Indeed, fix ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ so that it satisfies $I(\mathsf{Y};\mathsf{V})\le C_v$. Denote by $T_{\mathsf{V}|\mathsf{Y}}$ the transition matrix from $\mathsf{Y}$ to $\mathsf{V}$ and by $T_{\mathsf{Y}|\mathsf{X}}$ the transition matrix from $\mathsf{X}$ to $\mathsf{Y}$, respectively, i.e.,

$$\begin{array}{cc}\hfill {(T_{\mathsf{V}|\mathsf{Y}})}_{ik}& \triangleq \mathsf{P}\left(\mathsf{V}=i|\mathsf{Y}=k\right),1\le i\le \left|\mathcal{V}\right|,1\le k\le m,\hfill \\ \hfill {(T_{\mathsf{Y}|\mathsf{X}})}_{kj}& \triangleq \mathsf{P}\left(\mathsf{Y}=k|\mathsf{X}=j\right),1\le k\le m,1\le j\le n.\hfill \end{array}$$

Denote by ${\mathbf{q}}_{\mathsf{x}}$ and ${\mathbf{q}}_{\mathsf{y}}$ the marginal probability vectors of $\mathsf{X}$ and $\mathsf{Y}$, respectively, and consider the inner maximization term in (8). Since ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ and ${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}$ are fixed, then ${\mathsf{P}}_{\mathsf{X}\mathsf{V}}={\sum }_y{\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}(·|y){\mathsf{P}}_{\mathsf{X}\mathsf{Y}}(·,y)$ is also fixed. Denote by $T_{\mathsf{V}|\mathsf{X}}\triangleq T_{\mathsf{V}|\mathsf{Y}}{T}_{\mathsf{Y}|\mathsf{X}}$ the transition matrix from $\mathsf{X}$ to $\mathsf{V}$. Therefore, the inner maximization term in (8) is just the SSIB function with parameters $T_{\mathsf{V}|\mathsf{X}}$ and $C_u$, namely, ${\widehat{R}}_{T_{\mathsf{V}|\mathsf{X}}}({\mathbf{q}}_{\mathsf{x}},C_u)$. Hence, our problem can also be interpreted in the following two equivalent ways:

$$\begin{array}{cccccc}\hfill & R(C_u,C_v)\triangleq \hfill & \hfill & \underset{T_{\mathsf{V}|\mathsf{Y}}}{\mathrm{maximize}}\hfill & \hfill & {\widehat{R}}_{T_{\mathsf{V}|\mathsf{Y}}{T}_{\mathsf{Y}|\mathsf{X}}}({\mathbf{q}}_{\mathsf{x}},C_u)\hfill \\ \hfill & \hfill & \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & I({\mathbf{q}}_{\mathsf{y}},T_{\mathsf{V}|\mathsf{Y}})\le C_v;\hfill \end{array}$$

(9)

or, similarly, by interchanging the order of maximization in (8), it can be expressed as follows:

$$\begin{array}{cccccc}\hfill & R(C_u,C_v)\triangleq \hfill & \hfill & \underset{T_{\mathsf{U}|\mathsf{X}}}{\mathrm{maximize}}\hfill & \hfill & {\widehat{R}}_{T_{\mathsf{U}|\mathsf{X}}{T}_{\mathsf{X}|\mathsf{Y}}}({\mathbf{q}}_{\mathsf{y}},C_v)\hfill \\ \hfill & \hfill & \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & I({\mathbf{q}}_{\mathsf{x}},T_{\mathsf{U}|\mathsf{X}})\le C_u,\hfill \end{array}$$

(10)

where $T_{\mathsf{U}|\mathsf{X}}$ is the transition matrix from $\mathsf{X}$ to $\mathsf{U}$, and $T_{\mathsf{X}|\mathsf{Y}}$ is the transition matrix from $\mathsf{Y}$ to $\mathsf{X}$. This representation gives us a different perspective on our problem as an optimal compressed representation of the relevance random variable for the IB framework.

Remark 3.

Taking $C_v=\infty$ in (9) results in an deterministic channel from $\mathsf{Y}$ to $\mathsf{V}$, i.e., $\mathsf{V}=\mathsf{Y}$. Thus, the DSIB problem defined in (7) reduces to the SSIB problem (6).

The bound from Lemma 1 can be utilized to give cardinality bounding for the double-sided problem.

Proposition 1.

For the DSIB optimization problem defined in (7), it suffices to consider random variables $\mathsf{U}$ and $\mathsf{V}$ with cardinalities $\left|\mathcal{U}\right|\le n+1$ and $\left|\mathcal{V}\right|\le m+1$.

Proof. 

Let $T_{\mathsf{U}|\mathsf{X}}$ and $T_{\mathsf{V}|\mathsf{Y}}$ be two arbitrary transition matrices. By Lemma 1, there exists $T_{\tilde{\mathsf{U}}|\mathsf{X}}$ with $|\tilde{\mathcal{U}}|\le n+1$ such that $I(\tilde{\mathsf{U}};\mathsf{V})\ge I(\mathsf{U};\mathsf{V})$ and $I(\mathsf{X};\tilde{\mathsf{U}})\le C_u$. Similarly, $T_{\mathsf{V}|\mathsf{Y}}$ can be replaced with $T_{\tilde{\mathsf{V}}|\mathsf{Y}}$, $|\tilde{\mathcal{V}}|\le m+1$ such that $I(\tilde{\mathsf{U}};\tilde{\mathsf{V}})\ge I(\tilde{\mathsf{U}},\mathsf{V})\ge I(\mathsf{U};\mathsf{V})$, and $I(\mathsf{Y};\tilde{\mathsf{V}})\le C_v$. Therefore, there exists an optimal solution with $\left|\mathcal{U}\right|\le n+1$ and $\left|\mathcal{V}\right|\le m+1$.    □

In the following two sections, we will present the primary analytical outcomes of our study. First, we consider the scenario where our bivariate source is binary, specifically DSBS. Then, we handle the case where $\mathsf{X}$ and $\mathsf{Y}$ are jointly Gaussian.

3. Binary $(\mathsf{X},\mathsf{Y})$

Let $(\mathsf{X},\mathsf{Y})$ be a DSBS with parameter p, i.e.,

$${\mathsf{P}}_{\mathsf{X}\mathsf{Y}}(x,y)=\frac{1}{2}(p·𝟙(x\ne y)+(1-p)𝟙(x=y)).$$

(11)

We entitle the respective optimization problem (7) as the binary double-sided information bottleneck (BDSIB) and emphasize its dependence on the parameter p as $R(C_u,C_v,p)$.

The following proposition states that the cardinality bound from Lemma 1 can be tightened in the binary case.

Proposition 2.

Considering the optimization problem in (6) with $\mathsf{X}=\mathrm{Ber}(q)$ and $\left|\mathcal{Y}\right|=3$, binary $\mathsf{U}$ is optimal.

The proof of this proposition is postponed to Appendix A. Using similar justification for Proposition 1 combined with Proposition 2, we have the following strict cardinality formula for the BDSIB setting.

Proposition 3.

For the respective DSBS setting of (7), it suffices to consider random variables $\mathsf{U}$ and $\mathsf{V}$ with cardinalities $\left|\mathcal{U}\right|=\left|\mathcal{V}\right|=2$.

Note that the above statement is not required for the results in the rest of this section to hold and will be mainly applied to justify our conjectures via numerical simulations.

We next show that the specific objective function for the binary setting of (7), i.e, the mutual information between $\mathsf{U}$ and $\mathsf{V}$, has an elegant representation which will be useful in deriving lower and upper bounds.

Lemma 3.

The mutual information between $\mathsf{U}$ and $\mathsf{V}$ can be expressed as follows:

$$I(\mathsf{U};\mathsf{V})={\mathbb{E}}_{{\mathsf{P}}_{\mathsf{U}}\times {\mathsf{P}}_{\mathsf{V}}}\left[K(\mathsf{U},\mathsf{V},p)logK(\mathsf{U},\mathsf{V},p)\right],$$

(12)

where the expectation is taken over the product measure ${\mathsf{P}}_{\mathsf{U}}\times {\mathsf{P}}_{\mathsf{V}}$, $\mathsf{U}$ and $\mathsf{V}$ are binary random variables satisfying:

$$\mathsf{P}\left(\mathsf{U}=0\right)=\frac{{\alpha }_1-\frac{1}{2}}{{\alpha }_1-{\alpha }_0},\mathsf{P}\left(\mathsf{V}=0\right)=\frac{{\beta }_1-\frac{1}{2}}{{\beta }_1-{\beta }_0},$$

(13)

the kernel $K(u,v,p)$ is given by:

$$K(u,v,p)=2{\alpha }_u\ast {\beta }_v\ast p=1-(1-2p)(1-2{\alpha }_u)(1-2{\beta }_v),$$

(14)

and the reverse test-channels are defined by: ${\alpha }_u\triangleq \mathsf{P}\left(\mathsf{X}=1|\mathsf{U}=u\right)$, ${\beta }_v\triangleq \mathsf{P}\left(\mathsf{Y}=0|\mathsf{V}=v\right)$. Furthermore, since $|(1-2p)(1-2{\alpha }_u)(1-2{\beta }_v)|<1$, utilizing Taylor’s expansion of $log(1-x)$, we obtain:

$$I(\mathsf{U};\mathsf{V})=\sum _{n=2}^{\infty }\frac{{(1-2p)}^n{\mathbb{E}}_{}\left[{(1-2{\alpha }_{\mathsf{U}})}^n\right]{\mathbb{E}}_{}\left[{(1-2{\beta }_{\mathsf{V}})}^n\right]}{n(n-1)}.$$

(15)

The general cascade of test-channels and the DSBS, defined by ${\left\{{\alpha }_u\right\}}_{u=0}^1$, ${\left\{{\beta }_v\right\}}_{v=0}^1$ and p, is illustrated in Figure 9. The proof of Lemma 3 is postponed to Appendix B.

Figure 9. General test-channel construction of the BDSIB function.

We next examine some corner cases for which $R(C_u,C_v,p)$ is fully characterized.

3.1. Special Cases

A particular case where we have a complete analytical solution is when p tends to $1/2$.

Theorem 1.

Suppose $p=\frac{1}{2}-ϵ$, and consider $ϵ\to 0$. Then

$$R(C_u,C_v,ϵ)=2{ϵ}^{2}loge·{(1-2h_b^{-1}(1-C_u))}^2{(1-2h_b^{-1}(1-C_v))}^2+o({ϵ}^2),$$

(16)

and it is achieved by taking ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ as BSC test-channels satisfying the constraints with equality.

This theorem follows by considering the low SNR regime in Lemma 3 and is proved in Appendix D. For the lower bound we take ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ to be BSCs.

In Section 6 we will give a numerical evidence that BSC test-channels are in fact optimal provided that p is sufficiently large. However, for small p this is no longer the case and we believe the following holds.

Conjecture 1.

Let $\mathsf{X}=\mathsf{Y}$, i.e., $p=0$. The optimal test-channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$ that achieve $R(C_u,C_v,0)$ are Z-channel and S-channel respectively.

Remark 4.

Our results in the numerical section strongly support this conjecture. In fact they prove it within the resolution of the experiments, i.e., for optimizing over a dense set of test-channels rather then all test-channels. Nevertheless, we were not able to find an analytical proof for this result.

Remark 5.

Suppose $\mathsf{X}=\mathsf{Y}$, $I(\mathsf{X};\mathsf{U})=C_u$, and $I(\mathsf{X};\mathsf{V})=C_v$. Since $I(\mathsf{U};\mathsf{V})=I(\mathsf{U};\mathsf{X})+I(\mathsf{V};\mathsf{X})-I(\mathsf{X};\mathsf{U},\mathsf{V})$ (as $\mathsf{U}\to \mathsf{X}\to \mathsf{Y}\to \mathsf{V}$ form a Markov chain in this order) then maximizing $I(\mathsf{U};\mathsf{V})$ is equivalent to minimizing $I(\mathsf{X};\mathsf{U},\mathsf{V})$, namely, minimizing information combining as in [23,35]. Therefore, Conjecture 1 is equivalent to the conjecture that among all channels with $I(\mathsf{X};\mathsf{U})\ge C_u$ and $I(\mathsf{Y};\mathsf{V})\ge C_v$, Z and S are the worst channels for information combining.

This observation leads us the following additional conjecture.

Conjecture 2.

The test-channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$ that maximize $I(\mathsf{X};\mathsf{U},\mathsf{V})$ are both Z channels.

Remark 6.

Suppose now that p is arbitrary and assume that one of the channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ or ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ is restricted to be a binary memoryless symmetric (BMS) channel (Chapter 4 of [64]), then the maximal $I(\mathsf{U};\mathsf{V})$ is attained by BSC channels, as those are the symmetric channels minimizing $I(\mathsf{X};\mathsf{U},\mathsf{V})$ [23]. It is not surprising that once the BMS constraint is removed, symmetric channels are no longer optimal (see the discussion in (Section VI.C of [23])).

Consider now the case $\mathsf{X}=\mathsf{Y}$ ($p=0$) with an additional symmetry assumption $C_u=C_v$. The most reasonable apriori guess is that the optimal test-channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$ are the same up to some permutation of inputs and outputs. Surprisingly, this is not the case, unless they are BSC or Z channels, as the following negative result states.

Proposition 4.

Suppose $C_u=C_v$ and the transition matrix from $\mathsf{X}$ to $\mathsf{V}$, given by

$$T_{\mathsf{V}|\mathsf{X}}=\left(\begin{array}{cc}a& b\\ 1-a& 1-b\end{array}\right),$$

(17)

satisfies $I({\mathbf{u}}_2,T_{\mathsf{V}|\mathsf{X}})=C_v$. Consider the respective SSIB optimization problem

$${\widehat{R}}_{T_{\mathsf{V}|\mathsf{X}}}({\mathbf{u}}_2,C_u)=\underset{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}:I(\mathsf{U};\mathsf{X})\le C_u}{max}I(\mathsf{U};\mathsf{V}).$$

(18)

The optimal ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ that attains (6) with ${\mathbf{q}}_{\mathsf{X}}={\mathbf{u}}_2$ and $C=C_u$ does not equal to ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$ or any permutation of ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$, unless ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$ is a BSC or a Z channel.

The proof is based on [17] and is postponed to Appendix E.

As for the case of $\mathsf{X}\ne \mathsf{Y}$, i.e., $p\ne 0$, we have the following conjecture.

Conjecture 3.

For every $(C_u,C_v)\in [0,1]\times [0,1]$, there exists $\theta (C_u,C_v)$, such that for every $p>\theta (C_u,C_v)$ the achieving test-channels $P_{\mathsf{U}|\mathsf{X}}$ and $P_{\mathsf{V}|\mathsf{Y}}$ are BSC with parameters $\alpha =h_b^{-1}(1-C_u)$ and $\beta =h_b^{-1}(1-C_v)$ respectively.

We will provide supporting arguments for this conjecture via numerical simulations in Section 6.

3.2. Bounds

In this section we present our lower and upper bounds on the BDSIB function, then we compare them for various channel parameters. The proofs are postponed to Appendix F. For the simplicity of the following presentation we define

$$g_b(x)\triangleq \frac{1}{2(1-x)}{h}_b(x),x\in [0,1/2],$$

(19)

denote $g_b^{-1}(·)$ as its inverse restricted to $[0,1]$, and $\hslash (x)\triangleq -xlogx$.

Proposition 5.

The BDSIB function is bounded from below by

$$\begin{array}{c}R(C_u,C_v,p)\ge \hfill \\ \hfill max\left\{\begin{array}{cc}& 1-h_b(\alpha \ast \beta \ast p),\hfill \\ & 1-\frac{1}{2\overline{\delta }\overline{\zeta }}\left[\hslash (\delta \ast \zeta \ast p)+(1-2\zeta )·\hslash (\overline{\delta }\ast p)+(1-2\delta )·\hslash (\overline{\zeta }\ast p)+(1-2\delta )(1-2\zeta )·\hslash (p)\right],\hfill \end{array}\right.\end{array}$$

(20)

where $\alpha =h_b^{-1}(1-C_u)$, $\beta =h_b^{-1}(1-C_v)$, $\delta =g_b^{-1}(1-C_u)$, and $\zeta =g_b^{-1}(1-C_v)$.

All terms in the RSH of (20) are attained by taking test-channels that match the constraints with equality and plugging them in Lemma 3. In particular: the first term is achieved by BSC test channels with transition probabilities $\alpha$ and $\beta$; the second term is achieved by taking ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ be a $Z(\delta )$ channel and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ be an $S(\zeta )$ channel. The aforementioned test-channel configurations are illustrated in Figure 10.

Figure 10. Test-channel that achieve the lower bound of Proposition 5.

We compare the different lower bounds derived in Proposition 5 for various values of constraints. The achievable rate vs channel transition probability p is shown in Figure 11. Our first observation is that BSC test-channels outperform all other choices for almost all values of p. However, Figure 12 gives a closer look on small values of p. It is evident that the combination of Z and S test-channels outperforms any other schemes for small values of p. We have used this observation as one supporting evidence to Conjecture 1.

Figure 11. Comparison of the lower bounds.

Figure 12. Comparison of the lower bounds in high SNR regime.

We proceed to give an upper bound.

Proposition 6.

A general upper bound on BDSIB is given by

$$R(C_u,C_v,p)\le min\left\{\begin{array}{cc}\hfill & {(1-2p)}^2(1-2h_b^{-1}{(1-C_u)}^2(1-2h_b^{-1}{(1-C_v)}^2,\hfill \\ \hfill & min\{1-h_b(h_b^{-1}(1-C_u)\ast p),1-h_b(h_b^{-1}(1-C_v)\ast p)\}.\hfill \end{array}\right.$$

(21)

Note that the first term can be derived by applying Jensen’s inequality on (12), and the second term is a combination of the standard IB and the cut-set bound. We postpone the proof of Proposition 6 to Appendix F.

Remark 7.

Since $p=\frac{1}{2}-ϵ$, we have a factor 2 loss in the first term compared to the precise behavior we have found for $p\approx \frac{1}{2}$ in Theorem 1. This loss comes from the fact that the bound in (21) actually upper bounds the χ-squared mutual information between $\mathsf{U}$ and $\mathsf{V}$. It is well-known that for very small $I(\mathsf{X};\mathsf{Y})$ we have that $I(\mathsf{X};\mathsf{Y})\approx 1/2I_{{\chi }^2}(\mathsf{X};\mathsf{Y})$, see [65].

We compare the different upper bounds from Proposition 6 in Figure 13 for various bottleneck constraints, and in Figure 14 for various values of channel transition probabilities p. We observe that there are regions of C and p for which Jensen’s based bound outperforms the standard IB bound.

Figure 13. Comparison of the upper bounds for various values of $(C_u,C_v)$.

Figure 14. Comparison of the upper bounds for various values of p.

Finally, we compare the best lower and upper bounds from Propositions 5 and 6 for various values of channel parameters in Figure 15. We observe that the bounds are tighter for asymmetric constraints and high transition probabilities.

Figure 15. Capacity bounds for various values of p and $C=C_u=C_v$.

4. Gaussian $(\mathsf{X},\mathsf{Y})$

In this section we consider a specific setting where $(\mathsf{X},\mathsf{Y})$ is the normalized zero mean Gaussian bivariate source, namely,

$$\left(\begin{array}{c}\mathsf{X}\\ \mathsf{Y}\end{array}\right)\sim \mathcal{N}\left(\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)\right).$$

(22)

We establish achievability schemes and show that Gaussian test-channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ are optimal for vanishing SNR. Furthermore we show an elegant representation of the problem through probabilistic Hermite polynomials which are defined by

$$H_n(x)\triangleq {(-1)}^n{e}^{\frac{x^2}{2}}\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}{e}^{-\frac{x^2}{2}},n\in {\mathbb{N}}_0.$$

(23)

We denote the Gaussian DSIB function with explicit dependency on $\rho$ as $R(C_u,C_v,\rho )$.

Proposition 7.

Let $H_n(·)$ be the nth order probabilistic Hermite polynomial, then the objective function of (7) for the Gaussian setting is given by

$$I(\mathsf{U};\mathsf{V})={\mathbb{E}}_{\mathsf{U}\mathsf{V}}\left[log\left(\sum _{n=0}^{\infty }\frac{{\rho }^n}{n!}{\mathbb{E}}_{}\left[H_n(\mathsf{X})|\mathsf{U}\right]{\mathbb{E}}_{}\left[H_n(\mathsf{Y})|\mathsf{V}\right]\right)\right].$$

(24)

This representation follows by considering $I(\mathsf{U};\mathsf{V})=D({\mathsf{P}}_{\mathsf{U}\mathsf{V}}||{\mathsf{P}}_{\mathsf{U}}·{\mathsf{P}}_{\mathsf{V}})$ and expressing $\frac{{\mathsf{P}}_{\mathsf{U}\mathsf{V}}}{{\mathsf{P}}_{\mathsf{U}}·{\mathsf{P}}_{\mathsf{V}}}$ using Mehler Kernel [66]. Mehler Kernel decomposition is a special case of a much richer family of Lancaster distributions [67]. The proof of Proposition 7 is relegated to Appendix G.

Now we give two lower bounds on $R(C_u,C_v,\rho )$. Our first lower bound is established by choosing ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ to be additive Gaussian channels, satisfying the mutual information constraints with equality.

Proposition 8.

A lower bound on $R(C_u,C_v,\rho )$ is given by

$$R(C_u,C_v,\rho )\ge -\frac{1}{2}log\left(1-{\rho }^2\left(1-2^{-2C_u}\right)\left(1-2^{-2C_v}\right)\right).$$

(25)

The proof of this bound is developed in Appendix H.

Although it was shown in [26] that choosing the test-channel to be Gaussian is optimal for the single-sided variant, it is not the case for its double-sided extension. We will show this by examining a specific set of values for the rate constraints, $(C_u,C_v)=(1,1)$. Furthermore, we choose the test channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ to be deterministic quantizers.

Proposition 9.

Let $(C_u,C_v)=(1,1)$, then

$$R(1,1,\rho )\ge 1-h_2\left(\frac{arccos\rho }{\pi }\right).$$

(26)

The proof of this bound is developed in Appendix I.

We compare the bounds from Propositions 8 and 9 with $(C_u,C_v)=(1,1)$ in Figure 16. The most unexpected observation here is that the deterministic quantizers lower bound outperform the Gaussian test-channels for high values of $\rho$. The crossing point of those bounds is given by

$${\rho }_{\mathrm{cros}}=\frac{e}{\sqrt{1+e^2}}\to \sqrt{SN{R}_{\mathrm{cros}}}=\frac{{\rho }_{\mathrm{cros}}}{\sqrt{1-{\rho }_{\mathrm{cros}}^2}}=e.$$

(27)

Figure 16. Comparison of the lower bounds from Propositions 8 and 9.

We proceed to present our upper bound on $R(C_u,C_v,\rho )$. This bound is a combination of the cutset bound and the single-sided Gaussian IB.

Proposition 10.

An upper bound on (7) with Gaussian $(\mathsf{X},\mathsf{Y})$ setting (22) is given by

$$R(C_u,C_v,\rho )\le min\left\{-\frac{1}{2}log(1-{\rho }^2(1-2^{-2C_u})),-\frac{1}{2}log(1-{\rho }^2(1-2^{-2C_v}))\right\}.$$

(28)

We compare the best lower and upper bounds from Propositions 8–10 in Figure 17. We observe that the bounds become tighter as the constraint increases and in the low-SNR regime.

Figure 17. Capacity bounds for various values of p and $C=C_u=C_v=1$.

4.1. Low-SNR Regime

For $\rho \to 0$, the exact asymptotic behavior of the Gaussian (Proposition 8) and deterministic (Proposition 9) test-channels, respectively, for $C_u=C_v=1$ is given by:

$$\begin{array}{cc}\hfill \underset{\rho \to 0}{lim}-\frac{1}{2}log\left(1-{\rho }^2(1-2^{-2C_u})(1-2^{-2C_v})\right)& =\frac{9loge}{32}{\rho }^2+o({\rho }^2),\hfill \\ \hfill \underset{\rho \to 0}{lim}1-h_2\left(\frac{arccos\rho }{\pi }\right)& =\frac{2loge}{{\pi }^2}{\rho }^2+o({\rho }^2).\hfill \end{array}$$

Hence, the Gaussian choice outperforms the second lower bound for vanishing SNR. The following theorem establishes that Gaussian test-channels are optimal for low-SNR.

Theorem 2.

For small ρ, the GDSIB function is given by:

$$R(C_u,C_v,\rho )=\frac{{\rho }^{2}loge}{2}(1-2^{-2C_u})(1-2^{-2C_v})+o({\rho }^2).$$

(29)

The lower bound follows from Proposition 8. The upper bound is established by considering the kernel representation from Proposition 7 in the limit of vanishing $\rho$. The detailed proof is given in Appendix J.

4.2. Optimality of Symbol-by-Symbol Quantization When $\mathsf{X}=\mathsf{Y}$

Consider an extreme scenario for which $\mathsf{X}=\mathsf{Y}\sim \mathcal{N}(0,1)$. Taking the encoders ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{X}}$ as a symbol-by-symbol deterministic quantizers satisfying:

$$H(\mathsf{U})=H(\mathsf{V})=min\{C_u,C_v\},$$

we achieve the optimum

$$I(\mathsf{U};\mathsf{V})=min\{C_u,C_v\}.$$

5. Alternating Maximization Algorithm

Consider the DSIB problem for DSBS with parameter p analyzed in Section 3. The respective optimization problem involves simultaneous search of the maximum over the sets $\left\{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}\right\}$ and $\left\{{\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}\right\}$. An alternating maximization, namely, fixing ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$, then finding the respective optimal ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ and vice versa, is sub-optimal in general and may result in convergent to a saddle point. However, for the case $p=0$ with symmetric bottleneck constraints, Proposition 4 implies that such point exists only for the BSC and Z/S channels. This motivates us to believe that performing an alternating maximization procedure on (9) will not result in sub-optimal saddle point, but rather converge to the optimal solution also for the general discrete $(\mathsf{X},\mathsf{Y})$.

Thus, we propose an alternating maximization algorithm. The main idea is to fix ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ and then compute ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}^{\ast }$ that attains the inner term in (9). Then, using ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}^{\ast }$, we find the optimal ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}^{\ast }$ that attains the inner term in (10). Then, we repeat the procedure in alternating manner until convergence.

Note that inner terms of (9) and (10) are just the standard IB problem defined in (6). For completeness, we state here the main result from [1] and adjust it for our problem. Consider the respective Lagrangian of (6) given by:

$$L({\mathsf{P}}_{\mathsf{U}|\mathsf{X}},\lambda )=I(\mathsf{U};\mathsf{V})+\lambda (C-I(\mathsf{X};\mathsf{U})).$$

(30)

Lemma 4

(Theorem 4 of [1]). The optimal test-channel that maximizes (30) satisfies the equation:

$${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}(u|x)=\frac{{\mathsf{P}}_{\mathsf{U}}(u)}{Z(x,\beta )}{e}^{-\beta D({\mathsf{P}}_{\mathsf{V}|\mathsf{X}=x}\parallel {\mathsf{P}}_{\mathsf{V}|\mathsf{U}=u})},$$

(31)

where $\beta \triangleq 1/\lambda$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{U}}$ is given via Bayes’ rule, as follows:

$${\mathsf{P}}_{\mathsf{V}|\mathsf{U}}(v|u)=\frac{1}{{\mathsf{P}}_{\mathsf{U}}(u)}\sum _x{\mathsf{P}}_{\mathsf{V}|\mathsf{X}}(v|x){\mathsf{P}}_{\mathsf{U}|\mathsf{X}}(u|x){\mathsf{P}}_{\mathsf{X}}(x).$$

(32)

In a very similar manner to the Blahut–Arimoto algorithm [18], the self-consistent equations can be adapted into converging, alternating iterations over the convex sets $\left\{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}\right\}={\Delta }_n^{\otimes n}$, $\left\{{\mathsf{P}}_{\mathsf{U}}\right\}={\Delta }_n$, and $\left\{{\mathsf{P}}_{\mathsf{V}|\mathsf{U}}\right\}={\Delta }_n^{\otimes n}$, as stated in the following lemma.

Lemma 5

(Theorem 5 of [1]). The self-consistent equations are satisfied simultaneously at the minima of the functional:

$$F({\mathsf{P}}_{\mathsf{U}|\mathsf{X}},{\mathsf{P}}_{\mathsf{U}},{\mathsf{P}}_{\mathsf{Y}|\mathsf{U}})=I(\mathsf{U};\mathsf{X})+\beta {\mathbb{E}}_{}\left[D({\mathsf{P}}_{\mathsf{V}|\mathsf{X}}\parallel {\mathsf{P}}_{\mathsf{V}|\mathsf{U}})\right],$$

(33)

where the minimization is performed independently over the convex sets of $\left\{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}\right\}={\Delta }_n^{\otimes n}$, $\left\{{\mathsf{P}}_{\mathsf{U}}\right\}={\Delta }_n$, and $\left\{{\mathsf{P}}_{\mathsf{V}|\mathsf{U}}\right\}={\Delta }_n^{\otimes n}$. The minimization is performed by the converging alternation iterations as described in Algorithm 1.

Algorithm 1: IB iterative algorithm IBAM(args)

Next, we propose a combined algorithm to solve the optimization problem from (7). The main idea is to fix one of the test-channels, i.e., ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$, and then find the corresponding optimal opposite test-channel, i.e., ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$, using Algorithm 1. Then, we apply again Algorithm 1 by switching roles, i.e., fixing the opposite test-channel, i.e., ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$, and then identifying the optimal ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$. We repeat this procedure until convergence of the objective function $I(\mathsf{U};\mathsf{V})$. We summarize the proposed composite method in Algorithm 2.

Remark 8.

Note that every alternating step of the algorithm involves finding an optimal $({\beta }^{\ast },{\eta }^{\ast })$ that corresponds to the respective problem constraints $(C_u,C_v)$. We have chosen to implement this exploration step using a bisection-type method. This may result that the actual pair $(C_u,C_v)$ is ϵ-far away from the desired constraint.

Algorithm 2: DSIB iterative algorithm DSIBAM(args)

6. Numerical Results

In this section, we focus on the DSBS setting of Section 3. In the first part of this section, we will show using a brute-force method the existence of a sharp, phase-transition phenomena in the optimal test-channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ vs. DSBS parameter p. In the second part of this section, we will evaluate the alternating maximization algorithm proposed in Section 5; then, we compare its performance to the brute-force method.

6.1. Exhaustive Search

In this set of simulations, we again fix the transition matrix from $\mathsf{Y}$ to $\mathsf{V}$ characterized by the parameters:

$$T=\left(\begin{array}{cc}a& b\\ 1-a& 1-b\end{array}\right),$$

(34)

chosen such that $I(\mathsf{Y};\mathsf{V})=C_v$. This choice defines a path $b=f(a)$ in the $(a,b)$ plain. Then, for every such T we optimize $I(\mathsf{U};\mathsf{V})$ for different values of the DSBS parameter p. The results for a specific choice of $(C_u,C_v)=(0.4,0.6)$ vs. a for different values of p are plotted in Figure 18. Note that the region of a corresponds to the continuous conversion from a Z channel ($a=0$) to a BSC ($a=a_{max}$). We observe here a very sharp transition from the optimality of Z-S channels to BSC channels configuration for a small change in p. This kind of behavior continues to hold with a different choice of $(C_u=0.1,C_v=0.9)$, as can be seen in Figure 19.

Figure 18. Maximal $I(\mathsf{U};\mathsf{V})$ for fixed values $(C_u,C_v)=(0.4,0.6)$ and different values of p.

Figure 19. Maximal $I(\mathsf{U};\mathsf{V})$ for fixed values $(C_u,C_v)=(0.1,0.9)$ and different values of p.

Next, we would like to emphasize this sharp phase transition phenomena by plotting the optimal a that achieves the maximal $I(\mathsf{U};\mathsf{V})$ vs the DSBS parameter p. The results for various combinations of $C_u$ and $C_v$ are presented in Figure 20 and Figure 21. We observe that the curves are convex for $p\in [0,p_{th})$ and constant for $p>p_{th}$ with $a=a_{bsc}$. Furthermore, the derivative of $a(p)$ for $p\to p_{th}$ tends to ∞.

Figure 20. Optimal value of a for various values of $C_u$ and $C_v$.

Figure 21. Optimal value of a for various values of $C_u$ and $C_v=0.9$.

One may further claim that there is no sharp transition to the BSC test-channels ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ as p grows away from zero, but rather only approaches BSC. To convince the reader that the optimal test channels are exactly BSC, we performed an alternating maximization experiment. We fixed $p>0$, $C_u$ and $C_v$. Then we have chosen ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ as an almost BSC channel satisfying $I(\mathsf{Y};\mathsf{V})\le C_v$ and found the channel $P_{\mathsf{X}|\mathsf{U}}$ that maximizes $I(\mathsf{U};\mathsf{V})$ subject to $I(\mathsf{X};\mathsf{U})\le C_u$. Then, we fixed the channel ${\mathsf{P}}_{\mathsf{X}|\mathsf{U}}$ and found the ${\mathsf{P}}_{\mathsf{Y}|\mathsf{V}}$ that maximizes $I(\mathsf{U};\mathsf{V})$ subject to $I(\mathsf{Y};\mathsf{V})\le C_v$. We have repeated this alternating maximization procedure until it converges. The transition matrices were parameterized as follows:

$$T_{\mathsf{Y}|\mathsf{V}}=\left(\begin{array}{cc}{q}_0& q_1\\ 1-q_0& 1-q_1\end{array}\right),T_{\mathsf{X}|\mathsf{U}}=\left(\begin{array}{cc}{p}_0& p_1\\ 1-p_0& 1-p_1\end{array}\right).$$

(35)

The results for different values of p, $C_u$, and $C_v$ are shown in Figure 22, Figure 23 and Figure 24. We observe that $p_0$ and $q_0$ rapidly converge to their respective BSC values satisfying the mutual information constraints. Note that the last procedure is still an exhaustive search, but it is performed in alternating fashion between the sets $\left\{{\mathsf{P}}_{\mathsf{U}|\mathsf{X}}\right\}$ and $\left\{{\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}\right\}$.

Figure 22. Alternating maximization with exhaustive search for various p, $C_u$, $C_v$.

Figure 23. Alternating maximization with exhaustive search for various p, $C_u$, $C_v$.

Figure 24. Alternating maximization with exhaustive search for various p, $C_u$, $C_v$.

6.2. Alternating Maximization

In this section, we will evaluate the algorithm proposed in Section 5. We focus on the DSBS setting of Section 3 with various values of problem parameters.

First, we explore the convergence behavior of the proposed algorithm. Figure 25 shows the objective function $I(\mathsf{U};\mathsf{V})$ on every iteration step for representative fixed-channel transition parameters p and the constraints $C_u$ and $C_v$. We observe a slow convergence to a final value for $p=0$ and $C_u=C_v=0.2$, but once the constraints and the transition probability are increased, the algorithm converges much more rapidly. The non-monotonic behavior in some regimes is justified with the help of Remark 8. In Figure 26, we see the respective test-channel probabilities ${\alpha }_0+{\alpha }_1$, $1-{\alpha }_0$, ${\beta }_0+{\beta }_1$, and $1-{\beta }_1$. First, note that if ${\alpha }_0+{\alpha }_1=1$, then ${\mathsf{P}}_{\mathsf{X}|\mathsf{U}}$ is a BSC. Similarly, if ${\beta }_0+{\beta }_1=1$, then ${\mathsf{P}}_{\mathsf{Y}|\mathsf{V}}$ is a BSC. Second, if $1-{\alpha }_0=1$, then ${\mathsf{P}}_{\mathsf{X}|\mathsf{U}}$ is a Z-channel. Similarly, if $1-{\beta }_1=1$, then ${\mathsf{P}}_{\mathsf{Y}|\mathsf{V}}$ is an S-channel. We observe that for $p=0$, the test-channels ${\mathsf{P}}_{\mathsf{X}|\mathsf{U}}$ and ${\mathsf{P}}_{\mathsf{Y}|\mathsf{V}}$ converge to Z- and S-channels, respectively. As for all other settings, the test-channels converge to BSC channels.

Figure 25. Convergence of $I(\mathsf{U};\mathsf{V})$ for various values of p, $C_u$ and $C_v$.

Figure 26. Convergence of $I(\mathsf{U};\mathsf{V})$ p with: (a) $C_u=C_v=0.2$, $p=0$; (b) $C_u=C_v=0.7$, $p=0$; (c) $C_u=C_v=0.5$, $p=0.1$; (d) $C_u=0.65,C_v=0.4$, $p=0.1$.

Finally, we compare the outcome of Algorithm 2 to the optimal solution achieved by the brute-force method, namely, evaluating (12) for every ${\mathsf{P}}_{\mathsf{U}|\mathsf{X}}$ and ${\mathsf{P}}_{\mathsf{V}|\mathsf{Y}}$ that satisfy the problem constraints. The results for various values of channel parameters are shown in Figure 27. We observe that the proposed algorithm achieves the optimum for any DSBS parameter p and some representative constraints $C_u$ and $C_v$.

Figure 27. Comparison of the proposed alternating maximization algorithm and the brute-force search method for various problem parameters.

7. Concluding Remarks

In this paper, we have considered the Double-Sided Information Bottleneck problem. Cardinality bounds on the representation’s alphabets were obtained for an arbitrary discrete bivariate source. When $\mathsf{X}$ and $\mathsf{Y}$ are binary, we have shown that taking binary auxiliary random variables is optimal. For DSBS, we have shown that BSC test-channels are optimal when $p\to 0.5$. Furthermore, numerical simulations for arbitrary p indicate that Z -and S-channels are optimal for $p=0$. As for the Gaussian bivariate source, representation of $I(\mathsf{U};\mathsf{V})$ utilizing Hermite polynomials was given. In addition, the optimality of the Gaussian test-channels was demonstrated for vanishing SNR. Moreover, we have constructed a lower bound attained by deterministic quantizers that outperforms the jointly Gaussian choice at high SNR. Note that the solution for the n-letter problem $max\frac{1}{n}I(\mathsf{U};\mathsf{V})$ for $\mathsf{U}\to {\mathsf{X}}^n\to {\mathsf{Y}}^n\to \mathsf{V}$ under constraints $I(\mathsf{U};{\mathsf{X}}^n)\le n{C}_u$ and $I(\mathsf{V};{\mathsf{Y}}^n)\le n{C}_v$ does not tensorize in general. For ${\mathsf{X}}^n={\mathsf{Y}}^n\sim {\mathrm{Ber}}^{\otimes n}(0.5)$, we can easily achieve the cut-set bound $I(\mathsf{U};\mathsf{V})/n=min\{C_u,C_v\}$. In addition, if time-sharing is allowed, the results change drastically.

Finally, we have proposed an alternating maximization algorithm based on the standard IB [1]. For the DSBS, it was shown that the algorithm converges to the global optimal solution.