A Coding Theorem for f-Separable Distortion Measures

Abstract

In this work we relax the usual separability assumption made in rate-distortion literature and propose $\mathrm{f}$-separable distortion measures, which are well suited to model non-linear penalties. The main insight behind $\mathrm{f}$-separable distortion measures is to define an n-letter distortion measure to be an $\mathrm{f}$-mean of single-letter distortions. We prove a rate-distortion coding theorem for stationary ergodic sources with $\mathrm{f}$-separable distortion measures, and provide some illustrative examples of the resulting rate-distortion functions. Finally, we discuss connections between $\mathrm{f}$-separable distortion measures, and the subadditive distortion measure previously proposed in literature.

1. Introduction

Rate-distortion theory, a branch of information theory that studies models for lossy data compression, was introduced by Claude Shannon in [1]. The approach of [1] is to model the information source with distribution $P_X$ on $\mathcal{X}$, a reconstruction alphabet $\widehat{\mathcal{X}}$, and a distortion measure $\mathrm{d}:\mathcal{X}\times \widehat{\mathcal{X}}\to [0,\infty )$. When the information source produces a sequence of n realizations, the source $P_{X^n}$ is defined on ${\mathcal{X}}^n$ with reconstruction alphabet ${\widehat{\mathcal{X}}}^n$, where ${\mathcal{X}}^n$ and ${\widehat{\mathcal{X}}}^n$ are n-fold Cartesian products of $\mathcal{X}$ and $\widehat{\mathcal{X}}$. In that case, [1] extended the notion of a single-letter distortion measure to the n-letter distortion measure, ${\mathrm{d}}^n:{\mathcal{X}}^n\times {\widehat{\mathcal{X}}}^n\to [0,\infty )$, by taking an arithmetic average of single-letter distortions,

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)=\frac{1}{n}\sum _{i=1}^n\mathrm{d}(x_i,{\widehat{x}}_i).\end{array}$$

(1)

Distortion measures that satisfy (1) are referred to as separable (also additive, per-letter, averaging); the separability assumption has been ubiquitous throughout rate-distortion literature ever since its inception in [1].

On the one hand, the separability assumption is quite natural and allows for a tractable characterization of the fundamental trade-off between the rate of compression and the average distortion. For example, in the case when $X^n$ is a stationary and memoryless source the rate-distortion function, which captures this trade-off, admits a simple characterization:

$$\begin{array}{c}\hfill \mathcal{R}(d)=\underset{P_{\widehat{X}|X}:\mathbb{E}\left[\mathrm{d}(X,\widehat{X})\right]\le d}{\inf }I(X;\widehat{X}).\end{array}$$

(2)

On the other hand, the separability assumption is very restrictive as it only models distortion penalties that are linear functions of the per-letter distortions in the source reproduction. Real-world distortion measures, however, may be highly non-linear; it is desirable to have a theory that also accommodates non-linear distortion measures. To this end, we propose the following definition:

Definition 1 (f-separable distortion measure).

Let $\mathrm{f}(z)$ be a continuous, increasing function on $[0,\infty )$. An n-letter distortion measure ${\mathrm{d}}^n(·,·)$ is $\mathrm{f}$-separable with respect to a single-letter distortion $\mathrm{d}(·,·)$ if it can be written as

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)={\mathrm{f}}^{-1}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{f}\left(\mathrm{d}(x_i,{\widehat{x}}_i)\right)\right).\end{array}$$

(3)

For $\mathrm{f}(z)=z$ this is the classical separable distortion set up. By selecting $\mathrm{f}$ appropriately, it is possible to model a large class of non-linear distortion measures, see Figure 1 for illustrative examples.

Figure 1. The number of reconstruction errors for an information source with 100 bits vs. the penalty assessed by $\mathrm{f}$-separable distortion measures based on the Hamming single-letter distortion. The $\mathrm{f}(z)=z$ plot corresponds to the separable distortion. The $\mathrm{f}$-separable assumption accommodates all of the other plots, and many more, with the appropriate choice of the function $\mathrm{f}$.

In this work, we characterize the rate-distortion function for stationary and ergodic information sources with $\mathrm{f}$-separable distortion measures. In the special case of memoryless and stationary sources we obtain the following intuitive result:

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)=\underset{P_{\widehat{X}|X}:\mathbb{E}\left[\mathrm{f}\left(\mathrm{d}(X,\widehat{X})\right)\right]\le \mathrm{f}(d)}{\inf }I(X;\widehat{X}).\end{array}$$

(4)

A pleasing implication of this result is that much of rate-distortion theory (e.g., the Blahut-Arimoto algorithm) developed since [1] can be leveraged to work under the far more general $\mathrm{f}$-separable assumption.

The rest of this paper is structured as follows. The remainder of Section 1 overviews related work: Section 1.1 provides the intuition behind Definition 1, Section 1.2 reviews related work in other compression problems, and Section 1.3 connects $\mathrm{f}$-separable distortion measures with sub-additive distortion measures. Section 2 formally sets up the problem and demonstrates why convexity of the rate-distortion function does not always hold under the $\mathrm{f}$-separable assumption. Section 3 presents our main result, Theorem 1, as well as some illustrative examples. Additional discussion about problem formulation and sub-additive distortion measures is given in Section 4. We conclude the paper in Section 5.

1.1. Generalized $\mathrm{f}$-Mean and Rényi Entropy

To understand the intuition behind Definition 1, consider aggregating n numbers $(z_1,\cdots ,z_n)$ by defining a sequence of functions (indexed by n)

$$\begin{array}{c}\hfill {\mathrm{M}}_n(\mathbf{z})={\mathrm{f}}^{-1}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{f}(z_i)\right)\end{array}$$

(5)

where $\mathrm{f}$ is a continuous, increasing function on $[z_{\min },z_{\max }]$, $z_{\min }=\min {\left\{z_i\right\}}_{i=1}^n$, and $z_{\max }=\max {\left\{z_i\right\}}_{i=1}^n$. It is easy to see that (5) satisfies the following properties:

• ${\mathrm{M}}_n(\mathbf{z})$ is continuous and monotonically increasing in each $z_i$,
• ${\mathrm{M}}_n(\mathbf{z})$ is a symmetric function of each $z_i$,
• If $z_i=z$ for all i, then ${\mathrm{M}}_n(\mathbf{z})=z$,
• For any $m\le n$

  $$\begin{array}{c}\hfill {\mathrm{M}}_n(\mathbf{z})={\mathrm{M}}_n\left({\mathrm{M}}_m({\mathbf{z}}_1^m),\cdots ,{\mathrm{M}}_m({\mathbf{z}}_1^m),z_{m+1},\cdots ,z_n\right).\end{array}$$

  (6)

Moreover, it is shown in [2] that any sequence of functions ${\mathrm{M}}_n$ that satisfies these properties must have the form of Equation (5) for some continuous, increasing $\mathrm{f}$. The function ${\mathrm{M}}_n$ is referred to as “Kolmogorov mean”, “quasi-arithmetic mean”, or “generalized $\mathrm{f}$-mean”. The most prominent examples are the geometric mean, $\mathrm{f}(z)=\log z$, and the root-mean-square, $\mathrm{f}(z)=z^2$.

The main insight behind Definition 1 is to define an n-letter distortion measure to be an $\mathrm{f}$-mean of single-letter distortions. The $\mathrm{f}$-separable distortion measures include all n-letter distortion measures that satisfy the above properties, with the last property saying that the non-linear “shape” of distortion measure (cf. Figure 1) is independent of n.

Finally, we note that Rényi also arrived at his well-known family of entropies [3] by taking an $\mathrm{f}$-mean of the information random variable:

$$\begin{array}{cc}\hfill H_{\alpha }(X)& ={\mathrm{f}}_{\alpha }^{-1}\mathbb{E}\left[{\mathrm{f}}_{\alpha }({ı}_X(X))\right],\alpha \in (0,1)\cup (1,\infty )\hfill \end{array}$$

(7)

where the information at x is

$$\begin{array}{cc}\hfill {ı}_X(x)& =\log \frac{1}{P_X(x)}.\hfill \end{array}$$

(8)

Rényi [3] limited his consideration to functions of the form ${\mathrm{f}}_{\alpha }(z)=\exp \left\{(1-\alpha )z\right\}$ in order to ensure that entropy is additive for independent random variables.

1.2. Compression with Non-Linear Cost

Source coding with non-linear cost has already been explored in the variable-length lossless compression setting. Let $\ell (x)$ denote the length of the encoding of x by a given variable length code. Campbell [4,5] proposed minimizing a cost function of the form

$$\begin{array}{c}\hfill {\mathrm{f}}^{-1}\mathbb{E}\left[\mathrm{f}(\ell (X))\right],\end{array}$$

(9)

instead of the usual expected length. The main result of [4,5] is that for

$$\begin{array}{c}\hfill {\mathrm{f}}_t(z)=\exp \left\{tz\right\},t\in (-1,0)\cup (0,\infty ),\end{array}$$

(10)

the fundamental limit of such setup is Rényi entropy of order $\alpha =\frac{1}{t+1}$. For more general $\mathrm{f}$, this problem was handled by Kieffer [6], who showed that (9) has a fundamental limit for a large class of functions $\mathrm{f}$. That limit is Rényi entropy of order $\alpha =\frac{1}{t+1}$ with

$$\begin{array}{c}\hfill t=\underset{z\to \infty }{\lim }\frac{{\mathrm{f}}^{\prime \prime }(z)}{{\mathrm{f}}^{\prime }(z)}.\end{array}$$

(11)

More recently, a number of works [7,8,9] studied related source coding paradigms, such as guessing and task encoding. These works also focused on the exponential functions given in (10); in [7,8] Rényi entropy is shown to be a fundamental limit yet again.

1.3. Sub-Additive Distortion Measures

A notable departure from the separability assumption in rate-distortion theory is sub-additive distortion measures discussed in [10]. Namely, a distortion measure is sub-additive if

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)\le \frac{1}{n}\sum _{i=1}^n\mathrm{d}(x_i,{\widehat{x}}_i).\end{array}$$

(12)

In the present setting, an $\mathrm{f}$-separable distortion measure is sub-additive if $\mathrm{f}$ is concave:

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)={\mathrm{f}}^{-1}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{f}\left(\mathrm{d}(x_i,{\widehat{x}}_i)\right)\right)\le \frac{1}{n}\sum _{i=1}^n\mathrm{d}(x_i,{\widehat{x}}_i).\end{array}$$

(13)

Thus, the results for sub-additive distortion measures, such as the convexity of the rate-distortion function, are applicable to $\mathrm{f}$-separable distortion measures when $\mathrm{f}$ is concave.

2. Preliminaries

Let X be a random variable defined on $\mathcal{X}$ with distribution $P_X$, with reconstruction alphabet $\widehat{\mathcal{X}}$, and a distortion measure $\mathrm{d}:\mathcal{X}\times \widehat{\mathcal{X}}\to [0,\infty )$. Let $\mathcal{M}=\{1,\cdots ,M\}$ be the message set.

Definition 2 (Lossy source code).

A lossy source code $(\mathrm{g},\mathrm{c})$ is a pair of mappings,

$$\begin{array}{cc}\hfill \mathrm{g}& :\mathcal{X}\to \mathcal{M}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \mathrm{c}& :\mathcal{M}\to \widehat{\mathcal{X}}.\hfill \end{array}$$

(15)

A lossy source-code $(\mathrm{g},\mathrm{c})$ is an $(M,d)$-lossy source code on $(\mathcal{X},\widehat{\mathcal{X}},\mathrm{d})$ if

$$\begin{array}{c}\hfill \mathbb{E}\left[\mathrm{d}(X,\mathrm{c}(\mathrm{g}(X))\right]\le d.\end{array}$$

(16)

A lossy source code $(\mathrm{g},\mathrm{c})$ is an $(M,d,ϵ)$-lossy source code on $(\mathcal{X},\widehat{\mathcal{X}},\mathrm{d})$ if

$$\begin{array}{c}\hfill \mathbb{P}\left[\mathrm{d}(X,\mathrm{c}(\mathrm{g}(X))>d\right]\le ϵ.\end{array}$$

(17)

Definition 3.

An information source $\mathbf{X}$ is a stochastic process

$$\begin{array}{c}\hfill \mathbf{X}={\left\{X^n=(X_1,\cdots ,X_n)\right\}}_{n=1}^{\infty }.\end{array}$$

(18)

If $(\mathrm{g},\mathrm{c})$ is an $(M,d)$-lossy source code for $X^n$ on $({\mathcal{X}}^n,{\widehat{\mathcal{X}}}^n,{\mathrm{d}}^n)$, we say $(\mathrm{g},\mathrm{c})$ is an $(n,M,d)$-lossy source code. Likewise, an $(M,d,ϵ)$-lossy source code for $X^n$ on $({\mathcal{X}}^n,{\widehat{\mathcal{X}}}^n,{\mathrm{d}}^n)$ is an $(n,M,d,ϵ)$-lossy source code.

2.1. Rate-Distortion Function (Average Distortion)

Definition 4.

Let a sequence of distortion measures $\left\{{\mathrm{d}}^n\right\}$ be given. The rate-distortion pair $(R,d)$ is achievable if there exists a sequence of $(n,M_n,d_n)$-lossy source codes such that

$$\begin{array}{c}\hfill \underset{n\to \infty }{\lim \sup }\frac{1}{n}\log M_n\le R,and\underset{n\to \infty }{\lim \sup } d_n\le d.\end{array}$$

Our main object of study is the following rate-distortion function with respect to $\mathrm{f}$-separable distortion measures.

Definition 5.

Let $\left\{{\mathrm{d}}^n\right\}$ be a sequence of $\mathrm{f}$-separable distortion measures. Then,

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)=\inf \{R:(R,d)is achievable\}.\end{array}$$

(19)

If $\mathrm{f}$ is the identity, then we omit the subscript $\mathrm{f}$ and simply write $\mathcal{R}(d)$.

2.2. Rate-Distortion Function (Excess Distortion)

It is useful to consider the rate-distortion function for $\mathrm{f}$-separable distortion measures under the excess distortion paradigm.

Definition 6.

Let a sequence of distortion measures $\left\{{\mathrm{d}}^n\right\}$ be given. The rate-distortion pair $(R,d)$ is (excess distortion) achievable if for any $\gamma >0$ there exists a sequence of $(n,M_n,d+\gamma ,{ϵ}_n)$-lossy source codes such that

$$\begin{array}{c}\hfill \underset{n\to \infty }{\lim \sup }\frac{1}{n}\log M_n\le R,and\underset{n\to \infty }{\lim \sup } {ϵ}_n=0.\end{array}$$

Definition 7.

Let $\left\{{\mathrm{d}}^n\right\}$ be a sequence of $\mathrm{f}$-separable distortion measures. Then,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{f}}^{\prime }(d)=\inf \{R& :(R,d) is (excess distortion) achievable\}.\hfill \end{array}$$

(20)

Characterizing the $\mathrm{f}$-separable rate-distortion function is particularly simple under the excess distortion paradigm, as shown in the following lemma.

Lemma 1.

Let the single-letter distortion $\mathrm{d}$ and an increasing, continuous function $\mathrm{f}$ be given. Then,

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}^{\prime }(d)={\tilde{\mathcal{R}}}^{\prime }(\mathrm{f}(d))\end{array}$$

(21)

where ${\tilde{\mathcal{R}}}^{\prime }(d)$ is computed with respect to $\tilde{\mathrm{d}}(x,\widehat{x})=\mathrm{f}(\mathrm{d}(x,\widehat{x}))$.

Proof. 

Let $\left\{{\mathrm{d}}^n\right\}$ be a sequence of $\mathrm{f}$-separable distortions based on $\mathrm{d}(·,·)$ and let $\left\{{\tilde{\mathrm{d}}}^n\right\}$ be a sequence of separable distortion measures based on $\tilde{\mathrm{d}}(·,·)=\mathrm{f}(\mathrm{d}(·,·))$.

Since $\mathrm{f}$ is increasing and continuous at d, then for any $\gamma >0$ there exists $0<\tilde{\gamma }$ such that

$$\begin{array}{c}\hfill \mathrm{f}(d+\gamma )-\mathrm{f}(d)=\tilde{\gamma }.\end{array}$$

(22)

The reverse is also true by continuity of $\mathrm{f}$: for any $\tilde{\gamma }>0$ there exists $\gamma >0$ such that (22) is satisfied.

Any source code $({\mathrm{g}}_n,{\mathrm{c}}_n)$ is an $(n,M_n,d+\gamma ,{ϵ}_n)$-lossless code under $\mathrm{f}$-separable distortion ${\mathrm{d}}^n$ if and only if $({\mathrm{g}}_n,{\mathrm{c}}_n)$ is also an $(n,M_n,\mathrm{f}(d)+\tilde{\gamma },{ϵ}_n)$-lossless code under separable distortion ${\tilde{\mathrm{d}}}^n$. Indeed,

$$\begin{array}{cc}\hfill {ϵ}_n& \ge \mathbb{P}\left[{\mathrm{d}}^n(X^n,{\mathrm{c}}_n\left({\mathrm{g}}_n(X^n)\right))\ge d+\gamma \right]\hfill \end{array}$$

(23)

$$\begin{array}{c}=\mathbb{P}\left[{\mathrm{f}}^{-1}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{f}(\mathrm{d}(X_i,{\widehat{X}}_i))\right)\ge d+\gamma \right]\hfill \end{array}$$

(24)

$$\begin{array}{c}=\mathbb{P}\left[\frac{1}{n}\sum _{i=1}^n\mathrm{f}(\mathrm{d}(X_i,{\widehat{X}}_i))\ge \mathrm{f}(d+\gamma )\right]\hfill \end{array}$$

(25)

$$\begin{array}{c}=\mathbb{P}\left[{\tilde{\mathrm{d}}}^n(X^n,{\mathrm{c}}_n\left({\mathrm{g}}_n(X^n)\right))\ge \mathrm{f}(d)+\tilde{\gamma }\right]\hfill \end{array}$$

(26)

where ${\widehat{X}}^n={\mathrm{c}}_n\left({\mathrm{g}}_n(X^n)\right)$. It follows that $(R,d)$ is (excess distortion) achievable with respect to $\left\{{\mathrm{d}}^n\right\}$ if and only if $(R,\mathrm{f}(d))$ is (excess distortion) achievable with respect to $\{{\tilde{\mathrm{d}}}^n\}$. The lemma statement follows from this observation and Definition 6. ☐

2.3. $\mathrm{f}$-Separable Rate-Distortion Functions and Convexity

While it is a well-established result in rate-distortion theory that all separable rate-distortion functions are convex ([11], Lemma 10.4.1), this need not hold for $\mathrm{f}$-separable rate-distortion functions.

The convexity argument for separable distortion measures is based on the idea of time sharing; that is, suppose there exists an $(n_1,M_1,d_1)$-lossy source code of blocklength $n_1$ and an $(n_2,M_2,d_2)$-lossy source code of blocklength $n_2$. Then, there exists an $(n,M,d)$-lossy source code of blocklength n with $M=M_1{M}_2$ and $d=\frac{n_1}{n_1+n_2}{d}_1+\frac{n_1}{n_1+n_2}{d}_2$: such a code is just a concatenation of codes over blocklengths $n_1$ and $n_2$. The distortion d is achievable since

$$\begin{array}{c}\hfill {\mathrm{d}}^{n_1}(x^{n_1},{\widehat{x}}^{n_1})=\frac{1}{n_1}\sum _{i=1}^{n_1}\mathrm{d}(x_i,{\widehat{x}}_i)=d_1\end{array}$$

(27)

and letting $n=n_1+n_2$,

$$\begin{array}{c}\hfill {\mathrm{d}}^{n_2}\left(x_{n_1+1}^n,{\widehat{x}}_{n_1+1}^n\right)=\frac{1}{n_2}\sum _{i=n_1+1}^n\mathrm{d}(x_i,{\widehat{x}}_i)=d_2.\end{array}$$

(28)

Time sharing between the two schemes gives

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)=\frac{1}{n}\sum _{i=1}^n\mathrm{d}(x_i,{\widehat{x}}_i)=\frac{n_1}{n}{d}_1+\frac{n_2}{n}{d}_2.\end{array}$$

(29)

However, this bound on the distortion need not hold for $\mathrm{f}$-separable distortions. Consider $\mathrm{f}$ which is strictly convex and suppose

$$\begin{array}{c}\hfill {\mathrm{f}}^{-1}\left(\frac{1}{n_1}\sum _{i=1}^{n_1}\mathrm{f}\left(\mathrm{d}(x_i,{\widehat{x}}_i)\right)\right)=d_1,{\mathrm{f}}^{-1}\left(\frac{1}{n_2}\sum _{i=n_1+1}^n\mathrm{f}\left(\mathrm{d}(x_i,{\widehat{x}}_i)\right)\right)=d_2.\end{array}$$

(30)

We can write

$$\begin{array}{c}\hfill {\mathrm{f}}^{-1}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{f}\left(\mathrm{d}(x_i,{\widehat{x}}_i)\right)\right)={\mathrm{f}}^{-1}\left(\frac{n_1}{n}\mathrm{f}(d_1)+\frac{n_2}{n}\mathrm{f}(d_2)\right)>\frac{n_1}{n}{d}_1+\frac{n_2}{n}{d}_2.\end{array}$$

(31)

Thus, concatinating the two schemes together does not guarantee that the distortion assigned by the $\mathrm{f}$-separable distortion measure is bounded by d.

3. Main Result

In this section we make the following standard assumptions, see [12].

• $\mathbf{X}$ is a stationary and ergodic source.
• The single-letter distortion function $\mathrm{d}(·,·)$ and the continuous and increasing function $\mathrm{f}(·)$ are such that

  $$\begin{array}{c}\hfill \underset{\widehat{x}\in \widehat{\mathcal{X}}}{\inf }\mathbb{E}\left[\mathrm{f}(\mathrm{d}(X,\widehat{x}))\right]<\infty .\end{array}$$

  (32)
• For each $d>0$, there exists a countable subset $\left\{{\widehat{x}}_i\right\}$ of $\widehat{\mathcal{X}}$ and a countable measurable partition $\left\{E_i\right\}$ of $\mathcal{X}$ such that $\mathrm{d}(x,{\widehat{x}}_i)\le d$, $x\in E_i$ for each ${\widehat{x}}_i$, and

  $$\begin{array}{c}\hfill \sum _i{P}_{X_1}(E_i)\log \frac{1}{P_{X_1}(E_i)}<\infty .\end{array}$$

  (33)

Theorem 1.

Under the stated assumptions, the rate-distortion function is given by

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)=\tilde{\mathcal{R}}(\mathrm{f}(d))\end{array}$$

(34)

where

$$\begin{array}{c}\hfill \tilde{\mathcal{R}}(\mathrm{f}(d))=\underset{n\to \infty }{\lim }\underset{P_{{\widehat{X}}^n|X^n}:\frac{1}{n}{\sum }_{i=1}^n\mathbb{E}\tilde{\mathrm{d}}(X_i,{\widehat{X}}_i)\le \mathrm{f}(d)}{\inf }\frac{1}{n}I(X^n;{\widehat{X}}^n)\end{array}$$

(35)

is the rate-distortion function computed with respect to the separable distortion measure given by $\tilde{\mathrm{d}}(x,\widehat{x})=\mathrm{f}(\mathrm{d}(x,\widehat{x}))$.

For stationary memoryless sources (34) particularizes to

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)=\underset{P_{\widehat{X}|X}:\mathbb{E}\left[\mathrm{f}\left(\mathrm{d}(X,\widehat{X})\right)\right]\le \mathrm{f}(d)}{\inf }I(X;\widehat{X}).\end{array}$$

(36)

Proof. 

Equations (35) and (36) are widely known in literature (see, for example, [10,11,13]); it remains to show (34). Under the stated assumptions,

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)\stackrel{(a)}{\le }{\mathcal{R}}_{\mathrm{f}}^{\prime }(d)\stackrel{(b)}{=}{\tilde{\mathcal{R}}}^{\prime }(\mathrm{f}(d))\stackrel{(c)}{=}\tilde{\mathcal{R}}(\mathrm{f}(d))\end{array}$$

(37)

where (a) follows from assumption (2) and Theorem A1 in the Appendix A, (b) is shown in Lemma 1, and (c) is due to [14] (see also ([13], Theorem 5.9.1)). The other direction,

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)\ge \tilde{\mathcal{R}}(\mathrm{f}(d))\end{array}$$

(38)

is a consequence of the strong converse by Kieffer [12], see Lemma A1 in the Appendix A. ☐

An immediate application of Theorem 1 gives the $\mathrm{f}$-separable rate-distortion function for several well-known binary memoryless sources (BMS).

Example 1 (BMS, Hamming distortion).

Let $\mathbf{X}$ be the binary memoryless source. That is, $\mathcal{X}=\widehat{\mathcal{X}}=\{0,1\}$, $X_i$ is a Bernoulli$(p)$ random variable, and $\mathrm{d}(·,·)$ is the usual Hamming distortion measure. Then, for any continuous increasing $\mathrm{f}(·)$ and $p\le \frac{1}{2}$,

$${\mathcal{R}}_{\mathrm{f}}(d)=\left\{\begin{array}{cc}h(p)-h\left(\frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}\right),& \frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}<p\\ 0,& o.w.\end{array}\right\}$$

where

$$\begin{array}{c}\hfill h(p)=p\log \frac{1}{p}+(1-p)\log \frac{1}{1-p}\end{array}$$

(39)

is the binary entropy function. The result follows from a series of obvious equalities,

$$\begin{array}{cc}\hfill {\mathcal{R}}_{\mathrm{f}}(d)& =\underset{P_{\widehat{X}|X}:\mathbb{E}\left[\mathrm{f}\left(\mathrm{d}(X,\widehat{X})\right)\right]\le \mathrm{f}(d)}{\inf }I(X;\widehat{X})\hfill \end{array}$$

(40)

$$\begin{array}{c}=\underset{P_{\widehat{X}|X}:\frac{\mathbb{E}\left[\mathrm{f}\left(\mathrm{d}(X,\widehat{X})\right)\right]-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}\le \frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}}{\inf }I(X;\widehat{X})\hfill \end{array}$$

(41)

$$\begin{array}{c}=\underset{P_{\widehat{X}|X}:\mathbb{E}\left[\frac{\mathrm{f}\left(\mathrm{d}(X,\widehat{X})\right)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}\right]\le \frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}}{\inf }I(X;\widehat{X})\hfill \end{array}$$

(42)

$$\begin{array}{c}=\underset{P_{\widehat{X}|X}:\mathbb{E}\left[\mathrm{d}(X,\widehat{X})\right]\le \frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}}{\inf }I(X;\widehat{X})\hfill \end{array}$$

(43)

$$\begin{array}{c}=\mathcal{R}\left(\frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}\right).\hfill \end{array}$$

(44)

The rate-distortion function given in Example 1 is plotted in Figure 2 for different functions $\mathrm{f}$. The simple derivation in Example 1 could be applied to any source for which the single-letter distortion measure can take on only two values, as is shown in the next example.

Figure 2. ${\mathcal{R}}_{\mathrm{f}}(d)$ for the binary memoryless source with $p=0.5$. Compare these to the $\mathrm{f}$-separable distortion measures plotted for the binary source with Hamming distortion in Figure 1.

Example 2 (BMS, Erasure distortion).

Let $\mathbf{X}$ be the binary memoryless source and let the reconstruction alphabet have the erasure option. That is, $\mathcal{X}=\{0,1\}$, $\widehat{\mathcal{X}}=\{0,1,e\}$, and $X_i$ is a Bernoulli$\left(\frac{1}{2}\right)$ random variable. Let $\mathrm{d}(·,·)$ be the usual erasure distortion measure:

$$\mathrm{d}(x,\widehat{x})=\left\{\begin{array}{cc}0,& x=\widehat{x}\\ 1,& \widehat{x}=e\\ \infty ,& o.w.\end{array}\right\}.$$

The separable rate-distortion function for the erasure distortion is given by

$$\mathcal{R}(d)=1-d,$$

see ([11], Problem 10.7). Then, for any continuous increasing $\mathrm{f}(·)$,

$${\mathcal{R}}_{\mathrm{f}}(d)=1-\frac{\mathrm{f}(d)-\mathrm{f}(0)}{\mathrm{f}(1)-\mathrm{f}(0)}.$$

The rate-distortion function given in Example 2 is plotted in Figure 3 for different functions $\mathrm{f}$. Observe that for concave $\mathrm{f}$ (i.e., subadditive distortion) the resulting rate-distortion function is convex, which is consistent with [10]. However, for $\mathrm{f}$ that are not concave, the rate-distortion function is not always convex. Unlike in the conventional separable distortion measure, an $\mathrm{f}$-separable distortion measure is not convex in general.

Figure 3. ${\mathcal{R}}_{\mathrm{f}}(d)$ for the binary memoryless source with $p=0.5$ and erasure per-letter distortion.

Having a closed-form analytic expression for a separable distortion measure does not always mean that we could easily derive such an expression for an $\mathrm{f}$-separable distortion measure with the same per-letter distortion. For example, consider the Gaussian source with the mean-square-error (MSE) per-letter distortion. According to Theorem 1, letting $\mathrm{f}(z)=\sqrt{z}$ recovers the Gaussian source with the absolute value per-letter distortion. This setting, and variations on it, is a difficult problem in general [15]. However, we can recover the $\mathrm{f}$-separable rate-distortion function whenever the per-letter distortion $\mathrm{d}(·,·)$ composed with $\mathrm{f}(·)$ reconstructs the MSE distortion, see Figure 4.

Figure 4. ${\mathcal{R}}_{\mathrm{f}}(d)$ for the Gaussian memoryless source with mean zero and unit variance.

Theorem 1 shows that for well-behaved stationary ergodic sources, ${\mathcal{R}}_{\mathrm{f}}(d)$ admits a simple characterization. According to Lemma 1, the same characterization holds for the excess distortion paradigm without stationary and ergodic assumptions. The next example shows that, in general, ${\mathcal{R}}_{\mathrm{f}}(d)\ne \tilde{\mathcal{R}}(\mathrm{f}(d))$ within the average distortion paradigm. Thus, assumption (1) is necessary for Theorem 1 to hold.

Example 3 (Mixed Source).

Fix $\lambda \in (0,1)$ and let the source $\mathbf{X}$ be a mixture of two i.i.d. sources,

$$\begin{array}{c}\hfill P_{X^n}(x^n)=\lambda \prod _{i=1}^n{P}^1(x_i)+(1-\lambda )\prod _{i=1}^n{P}^2(x_i).\end{array}$$

(45)

We can alternatively express $\mathbf{X}$ as

$$\begin{array}{c}\hfill X^n=Z{X}_1^n+(1-Z)X_2^n\end{array}$$

(46)

where Z is a Bernoulli(λ) random variable. Then, the rate-distortion function for the mixture source (45) and continuous increasing $\mathrm{f}$ is given in Lemma A2 in the Appendix B. Namely,

$$\begin{array}{c}\hfill {\mathcal{R}}_{\mathrm{f}}(d)=\underset{(d_1,d_2):\lambda d_1+(1-\lambda )d_2\le d}{\min }\max \left\{{\mathcal{R}}_{\mathrm{f}}^1(d_1),{\mathcal{R}}_{\mathrm{f}}^2(d_2)\right\}\end{array}$$

(47)

where ${\mathcal{R}}_{\mathrm{f}}^1(d)$ and ${\mathcal{R}}_{\mathrm{f}}^2(d)$ are the rate-distortion functions for discrete memoryless soruces given by $P^1$ and $P^2$, respectively. Likewise,

$$\begin{array}{c}\hfill \tilde{\mathcal{R}}\left(\mathrm{f}(d)\right)=\underset{(d_1,d_2):\lambda d_1+(1-\lambda )d_2\le \mathrm{f}(d)}{\min }\max \left\{{\tilde{\mathcal{R}}}^1(d_1),{\tilde{\mathcal{R}}}^2(d_2)\right\}.\end{array}$$

(48)

As shown in Figure 5, Equations (47) and (48) are not equal in general.

Figure 5. Mixed binary source with $p_1=0.5$, $p_2=0.001$, and $\lambda =0.5$. Three examples of $\mathrm{f}$-separable rate-distortion functions are given. For $\mathrm{f}(z)=z$, the relation $\mathcal{R}(d)=\tilde{\mathcal{R}}(d)$ follows immediately. When $\mathrm{f}$ is not the identity, ${\mathcal{R}}_{\mathrm{f}}(d)\ne \tilde{\mathcal{R}}(\mathrm{f}(d))$ in general for non-ergodic sources.

4. Discussion

4.1. Sub-Additive Distortion Measures

Recall that an $\mathrm{f}$-separable distortion measure is sub-additive if $\mathrm{f}$ is concave (cf. Section 1.3). Clearly, not all $\mathrm{f}$-separable distortion measures are sub-additive, and not all sub-additive distortion measures are $\mathrm{f}$-separable. An examplar of a sub-additive distortion measure (which is not $\mathrm{f}$-separable) given in ([10], Chapter 5.2) is

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)=\frac{1}{n}{\left(\sum _{i=1}^n{\mathrm{d}}^q(x_i,{\widehat{x}}_i)\right)}^{1/q},q>1.\end{array}$$

(49)

The sub-additivity of (49) follows from the Minkowski inequality. Comparing (49) to a sub-additive, $\mathrm{f}$-separable distortion measure given by

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)={\left(\frac{1}{n}\sum _{i=1}^n{\mathrm{d}}^q(x_i,{\widehat{x}}_i)\right)}^{1/q},0\le q\le 1,\end{array}$$

(50)

we see that the discrepancy between (49) and (50) has to do not only with the different ranges of q but with the scaling factor as a function of n.

Consider a binary source with Hamming distortion and let $x^n={\mathbf{0}}^n,{\widehat{x}}^n={\mathbf{1}}^n$. Rewriting (49) we obtain

$$\begin{array}{c}\hfill {\mathrm{d}}^n(x^n,{\widehat{x}}^n)=\frac{1}{n^{(q-1)/q}}{\left(\frac{1}{n}\sum _{i=1}^n{\mathrm{d}}^q(x_i,{\widehat{x}}_i)\right)}^{1/q}\end{array}$$

(51)

and

$$\begin{array}{cc}\hfill \underset{n\to \infty }{\lim }{\mathrm{d}}^n(x^n,{\widehat{x}}^n)& =\underset{n\to \infty }{\lim }\frac{1}{n^{(q-1)/q}}{\left(\frac{1}{n}\sum _{i=1}^n{\mathrm{d}}^q(0,1)\right)}^{1/q}\hfill \end{array}$$

(52)

$$\begin{array}{c}=\underset{n\to \infty }{\lim }\frac{1}{n^{(q-1)/q}}{\left(\frac{1}{n}\sum _{i=1}^{n}1\right)}^{1/q}\hfill \end{array}$$

(53)

$$\begin{array}{c}=\underset{n\to \infty }{\lim }\frac{1}{n^{(q-1)/q}}=0.\hfill \end{array}$$

(54)

In the binary example, the limiting distortion of (49) is zero even when the reconstruction of $x^n$ gets every single symbol wrong. It is easy to observe that example (49) is similarly degenerate in many cases of interest. The distortion measure given by (50), on the other hand, is an example of a non-trivial sub-additive distortion measure, as can be seen in Figure 2 and Figure 3 for $q=\frac{1}{2}$.

4.2. A Consequence of Theorem 1

In light of the discussion in Section 1.1, an alert reader may consider modifying (16) to

$$\begin{array}{c}\hfill {\mathrm{f}}^{-1}\left(\mathbb{E}\left[\mathrm{f}\left(\mathrm{d}(X,\mathrm{c}(\mathrm{g}(X))\right)\right]\right)\le d,\end{array}$$

(55)

and studying the $(M,d)$-lossy source codes under this new paradigm. Call the corresponding rate-distortion function ${\mathcal{R}}^{\mathrm{f}}(d)$ and assume that n-letter distortion measures are separable. Thus, at block length n the constraint (55) is

$$\begin{array}{c}\hfill \mathbb{E}\left[\mathrm{f}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{d}(X_i,{\widehat{X}}_i)\right)\right]\le \mathrm{f}(d)\end{array}$$

(56)

where $\widehat{X}=\mathrm{c}(\mathrm{g}(X))$. This is equivalent to the following constraints:

$$\begin{array}{cc}\hfill \mathbb{E}\left[\mathrm{f}\left(\frac{1}{n}\sum _{i=1}^n{\mathrm{f}}^{-1}(\tilde{\mathrm{d}}(X_i,{\widehat{X}}_i))\right)\right]& \le \mathrm{f}(d)\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill \mathrm{and}\hspace{1em}\mathbb{E}\left[{\tilde{\mathrm{d}}}^n(X_i,{\widehat{X}}_i))\right]& \le \mathrm{f}(d)\hfill \end{array}$$

(58)

where ${\tilde{\mathrm{d}}}^n$ is an ${\mathrm{f}}^{-1}$-separable distortion measure. Putting these observations together with Theorem 1 yields

$$\begin{array}{c}\hfill {\mathcal{R}}^{\mathrm{f}}(d)={\tilde{\mathcal{R}}}_{{\mathrm{f}}^{-1}}(\mathrm{f}(d))=\mathcal{R}({\mathrm{f}}^{-1}(\mathrm{f}(d)))=\mathcal{R}(d).\end{array}$$

(59)

A consequence of Theorem 1 is that the rate distortion function remains unchanged under this new paradigm.

5. Conclusions

This paper proposes $\mathrm{f}$-separable distortion measures as a good model for non-linear distortion penalties. The rate-distortion function for $\mathrm{f}$-separable distortion measures is characterized in terms of separable rate-distortion function with respect to a new single-letter distortion measure, $\mathrm{f}(\mathrm{d}(·,·))$. This characterization is straightforward for the excess distortion paradigm, as seen in Lemma 1. The proof is more involved for the average distortion paradigm, as seen in Theorem 1. An important implication of Theorem 1 is that many prominant results in rate-distortion literature (e.g., Blahut-Arimoto algorithm) can be leveraged to work for $\mathrm{f}$-separable distortion measures.

Finally, we mention that a similar generalization is well-suited for channels with non-linear costs. That is, we say that ${\mathsf{b}}^n$ is an $\mathrm{f}$-separable cost function if it can be written as

$$\begin{array}{c}\hfill {\mathsf{b}}^n(x^n)={\mathrm{f}}^{-1}\left(\frac{1}{n}\sum _{i=1}^n\mathrm{f}\left(\mathsf{b}(x_i)\right)\right).\end{array}$$

(60)

With this generalization we can state the following result which is out of the scope of this special issue.

Theorem 2 (Channels with cost).

The capacity of a stationary memoryless channel given by $P_{Y|X}$ and $\mathrm{f}$-separable cost function based on single-letter function $\mathsf{b}(x)$ is

$$\begin{array}{c}\hfill C_{\mathrm{f}}(\beta )=\underset{P_X:\mathbb{E}\left[\mathrm{f}\left(\mathsf{b}(X)\right)\right]\le \mathrm{f}(\beta )}{\sup }I(X;Y).\end{array}$$

(61)