Bounds on the Sum-Rate of MIMO Causal Source Coding Systems with Memory under Spatio-Temporal Distortion Constraints

Abstract

In this paper, we derive lower and upper bounds on the OPTA of a two-user multi-input multi-output (MIMO) causal encoding and causal decoding problem. Each user’s source model is described by a multidimensional Markov source driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ noise process subject to three classes of spatio-temporal distortion constraints. To characterize the lower bounds, we use state augmentation techniques and a data processing theorem, which recovers a variant of rate distortion function as an information measure known in the literature as nonanticipatory $ϵ$-entropy, sequential or nonanticipative RDF. We derive lower bound characterizations for a system driven by an $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process, which we solve using the SDP algorithm for all three classes of distortion constraints. We obtain closed form solutions when the system’s noise is possibly non-Gaussian for both users and when only one of the users is described by a source model driven by a Gaussian noise process. To obtain the upper bounds, we use the best linear forward test channel realization that corresponds to the optimal test channel realization when the system is driven by a Gaussian noise process and apply a sequential causal DPCM-based scheme with a feedback loop followed by a scaled ECDQ scheme that leads to upper bounds with certain performance guarantees. Then, we use the linear forward test channel as a benchmark to obtain upper bounds on the OPTA, when the system is driven by an additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ non-Gaussian noise process. We support our framework with various simulation studies.

1. Problem Statement

We consider the two-user causal encoding and causal decoding setup illustrated in Figure 1. In this setup, users 1 and 2 are modeled by the following discrete-time time-invariant multidimensional Markov processes:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbf{x}}_{t+1}^1& =A_1{\mathbf{x}}_t^1+{\mathbf{w}}_t^1,t=0,1,\dots \hfill \\ \hfill {\mathbf{x}}_{t+1}^2& =A_2{\mathbf{x}}_t^2+{\mathbf{w}}_t^2,\hfill \end{array},\end{array}$$

(1)

where ${\mathbf{x}}_t^1\in {\mathbb{R}}^{p_1},{\mathbf{x}}_t^2\in {\mathbb{R}}^{p_2}$, with $p_1$ not necessarily equal to $p_2$, $(A_1,A_2)$ are known constant matrices of appropriate dimensions and $({\mathbf{w}}_t^1,{\mathbf{w}}_t^2)$ are additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ possibly non-Gaussian noise processes with zero mean and covariance matrix ${\mathrm{\Sigma }}_{{\mathbf{w}}^i}\succ 0,i=1,2$, independent of ${\mathbf{x}}_0^i,i=1,2$ and from each other for all $t\ge 0$. The initial states ${\mathbf{x}}_0^i,i=1,2$ are given by ${\mathbf{x}}_0^i\sim (0;{\mathrm{\Sigma }}_{{\mathbf{x}}_0^i}),i=1,2$. Finally, we restrict the eigenvalues of $(A_1,A_2)$ to be within the unit circle, which means that each user’s system model in (1) is asymptotically stable (i.e., asymptotically stationary).

The goal is to cast the performance of the setup in Figure 1 under various distortion metrics when the encoder compress information causally whereas the lossless compression between the encoder and the decoder is done in one shot assuming their clocks are synchronized.

First, we apply state space augmentation [1] to the state-space models in (1) to transform them into a single augmented state-space model as follows:

$$\begin{array}{c}\hfill {\mathbf{x}}_{t+1}=A{\mathbf{x}}_t+{\mathbf{w}}_t,\end{array}$$

(2)

where ${\mathbf{x}}_{t+1}={[{\mathbf{x}}_{t+1}^{1^{\mathrm{T}}},{\mathbf{x}}_{t+1}^{2^{\mathrm{T}}}]}^{{}^{\mathrm{T}}}\in {\mathbb{R}}^{p_1+p_2}$, A is a block diagonal matrix and ${\mathbf{w}}_t$ is an additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ possibly non-Gaussian noise process such that ${\mathbf{w}}_t\sim (0;{\mathrm{\Sigma }}_{\mathbf{w}})$ where $(A,{\mathrm{\Sigma }}_{\mathbf{w}})$ are of the form

$$\begin{array}{c}\hfill \left[\begin{array}{cc}{A}_1& \mathbf{0}\\ \mathbf{0}& A_2\end{array}\right]\in {\mathbb{R}}^{(p_1+p_2)\times (p_1+p_2)},\left[\begin{array}{cc}{\mathrm{\Sigma }}_{{\mathbf{w}}^1}& \mathbf{0}\\ \mathbf{0}& {\mathrm{\Sigma }}_{{\mathbf{w}}^2}\end{array}\right]\in {\mathbb{R}}^{(p1+p2)\times (p1+p2)}.\end{array}$$

(3)

We note that the operation in (3) can be mathematically denoted as $A=A_1\oplus A_2$ and similarly, ${\mathrm{\Sigma }}_{\mathbf{w}}={\mathrm{\Sigma }}_{{\mathbf{w}}^1}\oplus {\mathrm{\Sigma }}_{{\mathbf{w}}^2}$ (see the notation section for “⊕”).

System Operation: The encoder at each time instant t observes the augmented state ${\mathbf{x}}_t$ and generates the data packet ${\ell }_t\in \{1,\dots ,2^{R_t}\}$ of instantaneous rate $R_t$. At time t, the packet ${\ell }_t$ is sent over a noiseless channel with rate $R_t$. The decoder at each time t, receives ${\ell }_t$ to construct an estimate ${\mathbf{y}}_t$ of ${\mathbf{x}}_t$. We assume that the clocks of the encoder and decoder are synchronized. Formally, the encoder ($\mathcal{E}$) and the decoder ($\mathcal{D}$) are specified by a sequence of measurable functions $\{(f_t,g_t):t\in {\mathbb{N}}_0\}$ as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{E}:{\ell }_t& =f_t({\ell }^{t-1},x^t),{\ell }_{-\infty }^{-1}=\varnothing ,\hfill \\ \hfill \mathcal{D}:y_t& =g_t\left({\ell }^t\right).\hfill \end{array}\end{array}$$

(4)

1.1. Generalizations

It should be noted that the setup in Figure 1 can be generalized to any finite number of users. The only change will appear in the number of state-space equations and the dimension of the vectors and matrices in the augmented state-space representation of (2).

Next, we explain the setup of two users that are correlated (in states). In such scenario, users 1 and 2 are modeled by the following discrete-time time-invariant multidimensional Markov processes:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbf{x}}_{t+1}^1& =A_{11}{\mathbf{x}}_t^1+A_{12}{\mathbf{x}}_t^2+{\mathbf{w}}_t^1,t=0,1,\dots \hfill \\ \hfill {\mathbf{x}}_{t+1}^2& =A_{22}{\mathbf{x}}_t^2+A_{21}{\mathbf{x}}_t^1+{\mathbf{w}}_t^2,\hfill \end{array},\end{array}$$

(5)

where ${\mathbf{x}}_t^1\in {\mathbb{R}}^{p_1},{\mathbf{x}}_t^2\in {\mathbb{R}}^{p_2}$, with $p_1$ not necessarily equal to $p_2$, $(A_{11},A_{12},A_{21},A_{22})$ are known constant matrices of appropriate dimensions whereas all the other assumptions remain similar to the user models described in (1). The single augmented state-space model now is obtained as follows:

$$\begin{array}{c}\hfill {\mathbf{x}}_{t+1}=\widehat{A}{\mathbf{x}}_t+{\mathbf{w}}_t,\end{array}$$

(6)

where $\widehat{A}$ is a block matrix of the form

$$\begin{array}{c}\hfill \left[\begin{array}{cc}{A}_{11}& A_{12}\\ A_{21}& A_{22}\end{array}\right]\in {\mathbb{R}}^{(p_1+p_2)\times (p_1+p_2)},\end{array}$$

(7)

where $A_{11},A_{22}$ are square matrices but $(A_{12},A_{21})$ may be rectangular matrices (if $p_1\ne p_2$). We will not consider this case in our paper because it is straightforward by replacing everywhere matrix A with matrix $\widehat{A}$. Clearly, this case can be generalized to any finite number of users with appropriate modifications on the state-space models.

1.2. Distortion Constraints

In this work we consider three types of distortion constraints. These are articulated as follows:

(i)

a per-dimension (spatial) distortion constraint on the asymptotically averaged total (across the time) MMSE covariance matrix;

(ii)

an asymptotically averaged total (across the time and across the space) distortion constraint;

(iii)

a covariance matrix distortion constraint.

Next, we give the definition of each distortion constraint and explain some of their utility in multi-user systems.

A per-dimension (spatial) distortion constraint imposed on the covariance distortion matrix ${\mathrm{\Sigma }}_{\Delta }\triangleq {lim\; sup}_{n⟶\infty }\frac{1}{n+1}{\mathrm{\sum }}_{t=0}^n\mathbb{E}\left\{({\mathbf{x}}_t-{\mathbf{y}}_t){({\mathbf{x}}_t-{\mathbf{y}}_t)}^{\mathrm{T}}\right\}$, where ${\mathrm{\Sigma }}_{\Delta }⪰0$, is defined as follows:

$$\begin{array}{c}\hfill {\mathrm{\Sigma }}_{{\Delta }_{ii}}\le D_{ii},i=1,\dots ,p,\end{array}$$

(8)

where $D_{ii}\in [0,D_{ii}^{max}]$ are given diagonal entries of the positive semidefinite matrix $\widehat{D}⪰0$, with $\mathrm{trace}\left(\widehat{D}\right)\equiv D$, $D\in [0,D_{max}]$. Note that under this distortion constraint, it trivially holds that ${lim\; sup}_{n⟶\infty }\frac{1}{n+1}{\sum }_{t=0}^n\mathbb{E}\left\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\right\}\le D$.

Utility: The choice of per-dimension distortion constraints is arguably more realistic in various network systems. For instance, one use of such hard constraints can be found in multivariable feedback control systems also called centralized multi-input multi-output (MIMO) systems [2] (see Figure 2). In such networks, it may be the case that one wishes to minimize the temporally total performance criterion or satisfy some total fidelity constraint. However, in addition it is always required that the resource allocation to the different nodes (or variables) to never exceed certain performance thresholds when the demands in data transmission within the communication link allows only limited rate. Nonetheless, the problem is that variables interact. Some variables could be considered more important for certain applications according to the demands of the system or the quality of service, which is why they need hard constraints.

An asymptotically averaged total (across the time and space) distortion constraint is defined as follows:

$$\begin{array}{c}\hfill \underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}\sum _{t=0}^n\mathbb{E}\left\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\right\}\le D_{{}^{\mathrm{T}}},\end{array}$$

(9)

where $D_{{}^{\mathrm{T}}}\in [0,D_{{}^{\mathrm{T}}}^{max}]$, with $D_{{}^{\mathrm{T}}}$ not necessarily equal to D.

Utility: The asymptotically averaged total (across the time and space) distortion constraint ensure shared or allocated distortion arbitrarily among the transmit dimensions. The combination of the per-dimension distortion constraint with the averaged total distortion constraints ensure a total allocated distortion budget in the system that depends on the allowable (by design) distortion budget at each dimension (or user).

A covariance matrix distortion constraint is a generalization of the per-dimension distortion constraint defined by

$$\begin{array}{c}\hfill {\mathrm{\Sigma }}_{\Delta }⪯D_{\mathrm{cov}},\end{array}$$

(10)

where $D_{\mathrm{cov}}⪰0$.

Utility: During the recent years, there has been a shift from conventional $\mathrm{MSE}$ distortion constraints (scalar-valued target distortions) to covariance matrix distortions in the areas of multiterminal and distributed source coding [3,4,5,6,7] and signal processing [7,8,9]. Nonetheless, the argument for considering covariance distortion constraints despite its difficulty is its generality and the flexibility in formulating new problems. For instance, one practical example would be wireless adhoc microphones, that transmit to receiver(s) over an MIMO channel. In such setups, perhaps the receiver(s) need to do beam forming or some multi-channel Wiener filtering variants. In both cases, one needs to know the covariance matrix of e.g., the error signal (covariance distortion matrix) to perform the desired signal enhancement. For example, if the quality of one of the signals is too bad, this could harm the overall signal enhancement, and one therefore need to trade-off the bits correctly among the microphones. In an adhoc microphone array, the different signals are naturally correlated, which adds an interesting interplay between them that goes beyond MSE distortion.

1.3. Operational Interpretations

In this subsection, we use the three types of distortion constraints introduced in Section 1.2 to define the corresponding operational definitions for which we study lower and upper bounds in this paper.

Definition 1 

(Causal $\mathrm{RDF}$ subject to (8)). The operational causal $\mathrm{RDF}$ under per-dimension distortion constraints is defined as follows:

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^c\left(D\right)\triangleq \underset{\begin{array}{c}(f_t,g_t):t\in {\mathbb{N}}_0\\ {\mathrm{\Sigma }}_{{\Delta }_{ii}}\le D_{ii},i=1,\dots ,p\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}\sum _{t=0}^n{R}_t,\end{array}$$

(11)

where $D_{ii}\in [0,D_{ii,max}]$ and $D\in [0,D_{max}]$.

Definition 2 

(Causal $\mathrm{RDF}$ subject to (8) and (9)). The operational causal $\mathrm{RDF}$ under joint per-dimension and asymptotically averaged total distortion constraints is defined as follows:

$$\begin{array}{c}\hfill R_{\mathrm{joint}}^c\left(D^{*}\right)\triangleq \underset{\begin{array}{c}(f_t,g_t):t\in {\mathbb{N}}_0\\ {\mathrm{\Sigma }}_{{\Delta }_{ii}}\le D_{ii},i=1,\dots ,p,\\ {lim\; sup}_{n⟶\infty }\frac{1}{n+1}{\sum }_{t=0}^n\mathbb{E}\left\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\right\}\le D^{*}\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}\sum _{t=0}^n{R}_t,\end{array}$$

(12)

where $D^{*}=min\{D_{{}^{\mathrm{T}}},D\}$.

Interplay between Definitions 1 and 2. Clearly, Definition 1 is a lower bound to Definition 2 because its constraint set of feasible solutions is larger. Note that, in general, the asymptotically averaged total distortion constraint in (12) is active when $D_{{}^{\mathrm{T}}}\le D$, otherwise, it is a trivial constraint and (12) is equivalent to the optimization problem of (11). This observation will be shown via a simulation study in the sequel of the paper.

Definition 3 

(Causal $\mathrm{RDF}$ subject to (10)). The operational causal $\mathrm{RDF}$ under covariance matrix distortion constraints is defined as follows:

$$\begin{array}{c}\hfill R^c\left(D_{\mathrm{cov}}\right)\triangleq \underset{\begin{array}{c}(f_t,g_t):t\in {\mathbb{N}}_0\\ {\mathrm{\Sigma }}_{\Delta }⪯D_{\mathrm{cov}}\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}\sum _{t=0}^n{R}_t,\end{array}$$

(13)

where $D_{\mathrm{cov}}⪰0$.

Literature Review. In information theory, causal coding and causal decoding also termed zero-delay coding (see, e.g., [10,11,12,13,14]) does not rely on the traditional construction based on random codebooks that in turn allows asymptotically large source vector dimensions [15] to establish achievability of a certain (non-causal) rate-distortion performance. Indeed, the optimal rate-distortion performance for causal source coding (with the clocks of the encoder and decoder to be synchronized), is hard to compute and often bounds are derived in the literature. For example, lower and upper bounds on the operational causal $\mathrm{RDF}$ subject to solely the distortion constraint in (9) (or the more stringent per instant distortion constraint $\mathbb{E}\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\}\ge D_t,\forall t$) are already studied extensively for various special cases of the setup of Figure 1, see, e.g., [11,14,16,17] and the references therein. In this work, we study new problems related to the causal $\mathrm{RDF}$ for the general multi-user source coding setup of Figure 1 under various classes of distortion constraints that their utility (partly explained in Section 1.2) has not been studied in the literature so far. These bounds are established using tools from information theory, convex optimization and causal MMSE estimation.

1.4. Contributions

In this paper we obtain the following results for the setup of Figure 1.

• Characterization and computation of the true lower bounds on (11)–(13) when the users’ augmented source model is driven by a Gaussian noise process (Lemma 3, Theorem 2).
• Analytical lower bounds on (11)–(13) when the users’ augmented source model is driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ noise process (including both additive Gaussian and non-Gaussian noise) (Theorem 3). As a consequence, we also obtain analytical lower bounds when only one of the users’ source model is driven by a Gaussian noise process (Corollary 2).
• Characterizations and computation of achievable bounds on (11)–(13) when the users’ augmented source model is driven by a Gaussian noise process (Theorem 4).
• Characterizations of achievable bounds on (11)–(13), when the users’ augmented source model is driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ non-Gaussian noise process (Theorem 5).

Machinery and tools. The information theoretic rate distortion definitions that are used to obtain the lower bounds in this paper are derived using a data processing theorem (Theorem 1) that reveals the “suitable” information measure to use. The derivation of the steady-state characterization of the lower bounds in Lemma 3 is derived using inequalities from matrix algebra and a convexity argument that allows the use of Jensen’s inequality. To derive lower bounds beyond additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process, we use the fact that the characterizations of the lower bounds for the Gaussian case are in fact the characterizations obtained for the best linear coding policies (Lemma 4) hence these can serve as a benchmark to derive lower bounds beyond Gaussian noise process by leveraging certain trace/determinant inequalities and most importantly Minkowski’s determinant inequality ([18], Exercise 12.13) and EPI [19]. The upper bounds on the OPTA when the system’s noise is Gaussian, are derived using a causal sequential DPCM-based scheme with feedback loop which is equivalent to the scheme first derived in [14], followed by an ECDQ scheme that uses vector quantization. The upper bounds on the OPTA, when the system’s noise is additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ non-Gaussian are obtained using precisely the same trick that is used to obtain the lower bounds, i.e., we use the linear test channel realization that achieves similar upper bounds for the Gaussian case and then, using an SLB type concept (Theorem 5) we obtain the desired results.

The paper is structured as follows. In Section 2 we characterize and compute lower bounds on the OPTA of (11)–(13). In Section 3 we characterize and compute achievable coding scheme on the OPTA of (11)–(13). We draw conclusions and future directions in Section 4.

2. Lower Bounds

In this section, we first choose a suitable information measure that will be used to derive a lower bound on Definitions 1–3. This information measure is a variant of directed information subject to some conditional independence constraints. Then, we obtain lower bounds on Definitions 1–3 for jointly Gaussian Markov processes and for Markov processes driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ possibly non-Gaussian noise process.

First, we write the joint distribution of the communication system of Figure 1, i.e., from the two users described by the augmented state $\{{\mathbf{x}}_t:t\in {\mathbb{N}}_0^n\}$ to the augmented output of the $\mathrm{MMSE}$ decoder $\{{\mathbf{y}}_t:t\in {\mathbb{N}}_0^n\}$. In particular, the joint distribution induced by the joint process $\{({\mathbf{x}}_t,{\ell }_t,{\mathbf{y}}_t):t\in {\mathbb{N}}_0^n\}$ admits the following decomposition:

$$\begin{array}{cc}\hfill \mathbf{P}(d{x}^n,d{\ell }^n,d{y}^n)& ={\otimes }_{t=0}^n\mathbf{P}(d{y}_t,d{\ell }_t,d{x}_t|y^{t-1},{\ell }^{t-1},x^{t-1})\hfill \\ \hfill & ={\otimes }_{t=0}^n\mathbf{P}(d{y}_t|y^{t-1},x^t,{\ell }^t)\otimes \mathbf{P}(d{\ell }_t|{\ell }^{t-1},y^{t-1},x^t)\otimes \mathbf{P}(d{x}_t|x^{t-1},y^{t-1},{\ell }^{t-1})-a.s.\hfill \\ \hfill & ={\otimes }_{t=0}^n\mathbf{P}(d{y}_t|y^{t-1},{\ell }^t)\otimes \mathbf{P}(d{\ell }_t|{\ell }^{t-1},y^{t-1},x^t)\otimes \mathbf{P}(d{x}_t|x^{t-1})-a.s.,\hfill \end{array}$$

(14)

which means that the augmented state “source” process ${\mathbf{x}}_t$, and the decoder’s output process ${\mathbf{y}}_t$ satisfy the following conditional independence constraints:

$$\begin{array}{cc}\hfill \mathbf{P}(d{x}_t|x^{t-1},y^{t-1},{\ell }^{t-1})& =\mathbf{P}(d{x}_t|x^{t-1})-a.s.,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill \mathbf{P}(d{y}_t|y^{t-1},{\ell }^t,x^t)& =\mathbf{P}(d{y}_t|y^{t-1},{\ell }^t)-a.s.\hfill \end{array}$$

(16)

For (14) we state the following technical remark.

Remark 1 

(Trivial initial information). In (14) we assume that the joint distribution $\mathbf{P}(d{x}^{-1},d{\ell }^{-1},d{y}^{-1})$ generates trivial information. This means that $\mathbf{P}(d{x}_0|x^{-1},y^{-1},{\ell }^{-1})=\mathbf{P}\left(d{x}_0\right)$, $\mathbf{P}(d{y}_0|y^{-1},x^0,{\ell }^0)=\mathbf{P}(d{y}_0|x_0,{\ell }_0)$ and $\mathbf{P}(d{\ell }_0|{\ell }^{-1},y^{-1},x^0)=\mathbf{P}(d{\ell }_0|x_0)$.

We next prove a data processing theorem.

Theorem 1 

(Data processing theorem). Provided the decomposition of the joint distribution in (14) holds, the augmented state-space representation of the system in Figure 1 admits the following data processing inequalities:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill I({\mathbf{x}}^n;{\mathbf{y}}^n)\stackrel{\left(\mathbf{ii}\right)}{\le }I({\mathbf{x}}^n;{\ell }^n||{\mathbf{y}}^{n-1})\stackrel{\left(\mathbf{i}\right)}{\le }\sum _{t=0}^n{R}_t,\end{array}\end{array}$$

(17)

where

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill I({\mathbf{x}}^n;{\ell }^n||{\mathbf{y}}^{n-1})& \triangleq \sum _{t=0}^{n}I({\mathbf{x}}^t;{\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1}),\hfill \\ \hfill I({\mathbf{x}}^n;{\mathbf{y}}^n)& \triangleq \sum _{t=0}^{n}I({\mathbf{x}}^t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1}),\hfill \end{array}\end{array}$$

(18)

assuming $I({\mathbf{x}}^t;{\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})<\infty ,\forall t,$ and $I({\mathbf{x}}^t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1})<\infty ,\forall t$.

Proof. 

We first prove (i).

$$\begin{array}{cc}\hfill \sum _{t=0}^n{R}_t& \ge \sum _{t=0}^{n}H({\ell }_t|{\ell }^{t-1})\hfill \\ \hfill & \stackrel{\left(a\right)}{\ge }\sum _{t=0}^{n}H({\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})\hfill \\ \hfill & \stackrel{\left(b\right)}{\ge }\sum _{t=0}^n\left[H({\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})-H({\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1},{\mathbf{x}}^t)\right]\hfill \\ \hfill & \stackrel{\left(c\right)}{=}\sum _{t=0}^{n}I({\mathbf{x}}^t;{\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})\hfill \\ \hfill & \equiv I({\mathbf{x}}^n;{\ell }^n||{\mathbf{y}}^{n-1}),\hfill \end{array}$$

where $\left(a\right)$ follows because conditioning reduces entropy [19]; $\left(b\right)$ follows because of the non-negativity of the discrete entropy [19]; $\left(c\right)$ follows by definition.

Next, we prove (ii). This can be shown as follows:

$$\begin{array}{cc}\hfill I({\mathbf{x}}^t;{\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})-I({\mathbf{x}}^t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1})& \stackrel{\left(d\right)}{=}I({\mathbf{x}}^t;{\ell }_t,{\mathbf{y}}_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})-I({\mathbf{x}}^t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1})\hfill \\ \hfill & \stackrel{\left(e\right)}{=}I({\mathbf{x}}^t;{\ell }^t|{\mathbf{y}}^t)-I({\mathbf{x}}^t;{\ell }^{t-1}|{\mathbf{y}}^{t-1}),\hfill \\ \hfill & \stackrel{\left(f\right)}{=}I({\mathbf{x}}^t;{\ell }^t|{\mathbf{y}}^t)-I({\mathbf{x}}^{t-1};{\ell }^{t-1}|{\mathbf{y}}^{t-1}),\hfill \end{array}$$

(19)

where $\left(d\right)$ follows from an adaptation of ([20], Lemma 3.3) to processes, i.e., $I({\mathbf{x}}^t;{\ell }_t,{\mathbf{y}}_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})=I({\mathbf{x}}^t;{\ell }_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})+I({\mathbf{x}}^t;{\mathbf{y}}_t|{\ell }^t,{\mathbf{y}}^{t-1})$ and the second term is zero because of the conditional independence constraint (16); $\left(e\right)$ follows by the chain rule of conditional mutual information (again an adaptation of ([20], Lemma 3.3)) which decomposes the conditional mutual information in two different ways, i.e.,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill I({\mathbf{x}}^t;{\ell }^t,{\mathbf{y}}_t|{\mathbf{y}}^{t-1})& =I({\mathbf{x}}^t;{\ell }^{t-1}|{\mathbf{y}}^{t-1})+I({\mathbf{x}}^t;{\ell }_t,{\mathbf{y}}_t|{\ell }^{t-1},{\mathbf{y}}^{t-1})\hfill \\ \hfill & =I({\mathbf{x}}^t;{\ell }^t|{\mathbf{y}}^t)+I({\mathbf{x}}^t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1});\hfill \end{array}\end{array}$$

$\left(f\right)$ follows because an adaptation of ([20], Lemma 3.3) can be applied to $I({\mathbf{x}}^t;{\ell }^{t-1}|{\mathbf{y}}^{t-1})$ as follows

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill I({\mathbf{x}}^t;{\ell }^{t-1}|{\mathbf{y}}^{t-1})& =I({\mathbf{x}}_t,{\mathbf{x}}^{t-1};{\ell }^{t-1}|{\mathbf{y}}^{t-1})\hfill \\ \hfill & =I({\mathbf{x}}_t;{\ell }^{t-1}|{\mathbf{x}}^{t-1},{\mathbf{y}}^{t-1})+I({\mathbf{x}}^{t-1};{\ell }^{t-t}|{\mathbf{y}}^{t-1})\hfill \\ \hfill & \stackrel{\left(g\right)}{=}I({\mathbf{x}}^{t-1};{\ell }^{t-t}|{\mathbf{y}}^{t-1}),\forall t,\hfill \end{array}\end{array}$$

where $\left(g\right)$ follows because $I({\mathbf{x}}_t;{\ell }^{t-1}|{\mathbf{x}}^{t-1},{\mathbf{y}}^{t-1})=0$. This can be shown as follows.

$$\begin{array}{cc}\hfill I({\mathbf{x}}_t;{\ell }^{t-1}|{\mathbf{x}}^{t-1},{\mathbf{y}}^{t-1})& =h({\mathbf{x}}_t|{\mathbf{x}}^{t-1},{\mathbf{y}}^{t-1})-h({\mathbf{x}}_t|{\mathbf{x}}^{t-1},{\mathbf{y}}^{t-1},{\ell }^{t-1})\hfill \\ \hfill & =h({\mathbf{x}}_t|{\mathbf{x}}^{t-1},{\mathbf{y}}^{t-1})-h({\mathbf{x}}_t|{\mathbf{x}}^{t-1})\le 0,\forall t,\hfill \end{array}$$

(20)

where each $h(·)$ is assumed to be finite for any t, and (20) follows from the conditional independence constraint in (15). From the non-negativity of conditional mutual information [19], the result follows.

Finally, from (19) we have

$$\begin{array}{cc}\hfill \sum _{t=0}^n\left[I({\mathbf{x}}^t;{\ell }^t|{\mathbf{y}}^t)-I({\mathbf{x}}^{t-1};{\ell }^{t-1}|{\mathbf{y}}^{t-1})\right]& =I({\mathbf{x}}_0;{\ell }_0|{\mathbf{y}}_0)+I({\mathbf{x}}^1;{\ell }^1|{\mathbf{y}}^1)-I({\mathbf{x}}_0;{\ell }_0|{\mathbf{y}}_0)+\dots \hfill \\ \hfill & \dots +I({\mathbf{x}}^n;{\ell }^n|{\mathbf{y}}^n)-I({\mathbf{x}}^{n-1};{\ell }^{n-1}|{\mathbf{y}}^{n-1})\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill & =I({\mathbf{x}}^n;{\ell }^n|{\mathbf{y}}^n)\ge 0,\hfill \end{array}$$

(22)

where (22) follows by applying the method of differences in (21). The result follows because (22) is by definition non-negative. We note that if $I({\mathbf{x}}_0;{\ell }_0|{\mathbf{y}}_0)$ also appeared in the cancellations, then, this would have been the telescopic sum of the series. This completes the derivation. □

We note that Theorem 1 is different from the data processing theorem derived in ([21], Lemma 1) in that we assume the conditional independence constraint (15) instead of the conditional independence constraint $\mathbf{P}(d{x}_t|x^{t-1},y^{t-1},{\ell }^{t-1})=\mathbf{P}(d{x}_t|x^{t-1},y^{t-1})-a.s.$, i.e., the source process is not allowed to have access via feedback to the previous output symbols ${\mathbf{y}}^{t-1}$. This technical difference results into having the mutual information in (18) subject to conditional independence constraints, instead of the well-known directed information [22].

Before we introduce the information theoretic definitions that correspond to lower bounds on (11)–(13), we formally show the construction of $I({\mathbf{x}}^n;{\mathbf{y}}^n)$.

Source. The augmented source process $\{{\mathbf{x}}_t:t\in {\mathbb{N}}_0\}$ induces the sequence of conditional distributions $\{\mathbf{P}(d{x}_t|x^{t-1}),t\in {\mathbb{N}}_0^n\}$. Since no initial information is assumed, the distribution at $t=0$ is $\mathbf{P}\left(d{x}_0\right)$. In addition, by Bayes’ rule we obtain $\mathbf{P}\left(d{x}^n\right)\triangleq {\otimes }_{t=0}^n\mathbf{P}(d{x}_t|x^{t-1})$.

Reproduction or “test-channel”. The reproduction process $\{{\mathbf{y}}_t:t\in {\mathbb{N}}_0^n\}$ parameterized by ${\mathcal{X}}^t$ induces the sequence of conditional distributions, known as test-channels, by $\{\mathbf{P}(d{y}_t|y^{t-1},x^t),t\in {\mathbb{N}}_0^n\}$. At $t=0$, no initial state information is assumed, hence $\mathbf{P}(d{y}_0|y^{-1},x^0)=\mathbf{P}(d{y}_0|x_0)$. In addition, by Bayes’ rule we obtain $\overrightarrow{\mathbf{Q}}(d{y}^n|x^n)\triangleq {\otimes }_{t=0}^n\mathbf{P}(d{y}_t|y^{t-1},x^t)$.

From ([23], Remark 1), it can be shown that the sequence of conditional distributions $\{\mathbf{P}(d{x}_t|x^{t-1}):t\in {\mathbb{N}}_0^n\}$ and $\{\mathbf{P}(d{y}_t|y^{t-1},x^t):t\in {\mathbb{N}}_0^n\}$ uniquely define the family of conditional distributions on ${\mathcal{X}}^n$ and ${\mathcal{Y}}^n$ parameterized by $x^n\in {\mathcal{X}}^n$, respectively, given by the joint distribution

$$\begin{array}{c}\hfill \mathbf{P}(d{x}^n,d{y}^n)=\mathbf{P}\left(d{x}^n\right)\otimes \overrightarrow{\mathbf{Q}}(d{y}^n|x^n).\end{array}$$

(23)

In addition, from (23), we can uniquely define the ${\mathcal{Y}}^n—$marginal distribution by

$$\begin{array}{c}\hfill \mathbf{P}\left(d{y}^n\right)\triangleq {\int }_{{\mathcal{X}}^n}\mathbf{P}\left(d{x}^n\right)\otimes \overrightarrow{\mathbf{Q}}(d{y}^n|x^n),\end{array}$$

and the conditional distributions $\{\mathbf{P}(d{y}_t|y^{t-1}):t\in {\mathbb{N}}_0^n\}$.

Given the above construction of distributions, we can formally introduce the information measure using relative entropy as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill I({\mathbf{x}}^n;{\mathbf{y}}^n)& \stackrel{\left(a\right)}{\triangleq }\mathbb{D}(\mathbf{P}(d{x}^n,d{y}^n)||\mathbf{P}\left(d{x}^n\right)\times \mathbf{P}\left(d{y}^n\right))\in [0,\infty ]\hfill \\ \hfill & \stackrel{\left(b\right)}{=}{\int }_{{\mathcal{X}}^n\times {\mathcal{Y}}^n}log\left(\frac{d\overrightarrow{\mathbf{Q}}(·|x^n)}{d\mathbf{P}(·)}\left(y^n\right)\right)\mathbf{P}(d{x}^n,d{y}^n)\hfill \\ \hfill & \stackrel{\left(c\right)}{=}\sum _{t=0}^n\mathbb{E}\left\{log\left(\frac{d\mathbf{P}(·|{\mathbf{y}}^{t-1},{\mathbf{x}}^t)}{d\mathbf{P}(·|{\mathbf{y}}^{t-1})}\left({\mathbf{y}}_t\right)\right)\right\}\hfill \\ \hfill & \stackrel{\left(d\right)}{=}\sum _{t=0}^{n}I({\mathbf{x}}^t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1}),\hfill \end{array}\end{array}$$

(24)

where $\left(a\right)$ follows by definition of relative entropy between $\mathbf{P}(d{x}^n,d{y}^n)$ and the product distribution $\mathbf{P}\left(d{x}^n\right)\times \mathbf{P}\left(d{y}^n\right)$; $\left(b\right)$ is due to the Radon–Nikodym derivative theorem ([23], Appendix A and Appendix C); $\left(c\right)$ is due to chain rule of relative entropy; $\left(d\right)$ follows by definition.

We now state as a definition the lower bounds on (11)–(13).

Definition 4 

(Lower bounds). Using the previous construction of distributions and the information measure of (24), we can define the following lower bounds on (11)–(13).

(1) 

The sum-rate subject to per-dimension distortion constraint is defined as follows:

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^{LB}\left(D\right)\triangleq \underset{\begin{array}{c}\mathbf{P}(d{y}_t|y^{t-1},x^t):t=0,\dots ,\infty \\ {\mathrm{\Sigma }}_{{\Delta }_{ii}}\le D_{ii},i=1,\dots ,p\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}I({\mathbf{x}}^n;{\mathbf{y}}^n),\end{array}$$

(25)

with

$$\begin{array}{c}\hfill R_{\mathrm{pd},[0,n]}^{LB}\left(D\right)\triangleq \underset{\begin{array}{c}\mathbf{P}(d{y}_t|y^{t-1},x^t):t\in {\mathbb{N}}_0^n\\ {\mathrm{\Sigma }}_{{\Delta }_{ii},t}\le D_{ii},i=1,\dots ,p\end{array}}{inf}I({\mathbf{x}}^n;{\mathbf{y}}^n).\end{array}$$

(26)

where ${\mathrm{\Sigma }}_{\Delta ,t}\triangleq \frac{1}{n+1}{\sum }_{t=0}^n\mathbb{E}\left\{({\mathbf{x}}_t-{\mathbf{y}}_t){({\mathbf{x}}_t-{\mathbf{y}}_t)}^{\mathrm{T}}\right\}$, ${\mathrm{\Sigma }}_{{\Delta }_{ii},t}\triangleq \frac{1}{n+1}{\sum }_{t=0}^n{\left[\mathbb{E}\left\{({\mathbf{x}}_t-{\mathbf{y}}_t){({\mathbf{x}}_t-{\mathbf{y}}_t)}^{\mathrm{T}}\right\}\right]}_{ii}$ and $D\ge 0$.

(2) 

The sum-rate subject to joint distortion constraints is defined as follows:

$$\begin{array}{c}\hfill R_{\mathrm{joint}}^{LB}\left(D^{*}\right)\triangleq \underset{\begin{array}{c}\mathbf{P}(d{y}_t|y^{t-1},x^t):t=0,\dots ,\infty \\ {\mathrm{\Sigma }}_{{\Delta }_{ii}}\le D_{ii},i=1,\dots ,p\\ {lim\; sup}_{n⟶\infty }\frac{1}{n+1}{\sum }_{t=0}^n\mathbb{E}\left\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\right\}\le D_{{}^{\mathrm{T}}}\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}I({\mathbf{x}}^n;{\mathbf{y}}^n),\end{array}$$

(27)

$$\begin{array}{c}\hfill R_{\mathrm{joint},[0,n]}^{LB}\left(D^{*}\right)\triangleq \underset{\begin{array}{c}\mathbf{P}(d{y}_t|y^{t-1},x^t):t\in {\mathbb{N}}_0^n\\ {\mathrm{\Sigma }}_{{\Delta }_{ii},t}\le D_{ii},i=1,\dots ,p\\ \frac{1}{n+1}{\sum }_{t=0}^n\mathbb{E}\left\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\right\}\le D_{{}^{\mathrm{T}}}\end{array}}{inf}I({\mathbf{x}}^n;{\mathbf{y}}^n),\end{array}$$

(28)

where $D^{*}=min\{D,D_{{}^{\mathrm{T}}}\}$.

(3) 

The sum-rate subject to covariance matrix distortion constraints is defined as follows:

$$\begin{array}{c}\hfill R^{LB}\left(D_{\mathrm{cov}}\right)\triangleq \underset{\begin{array}{c}\mathbf{P}(d{y}_t|y^{t-1},x^t):t=0,\dots ,\infty \\ {\mathrm{\Sigma }}_{\Delta }⪯D_{\mathrm{cov}}\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}I({\mathbf{x}}^n;{\mathbf{y}}^n),\end{array}$$

(29)

$$\begin{array}{c}\hfill R_{[0,n]}^{LB}\left(D_{\mathrm{cov}}\right)\triangleq \underset{\begin{array}{c}\mathbf{P}(d{y}_t|y^{t-1},x^t):t\in {\mathbb{N}}_0^n\\ {\mathrm{\Sigma }}_{\Delta ,t}⪯D_{\mathrm{cov}}\end{array}}{inf}I({\mathbf{x}}^n;{\mathbf{y}}^n),\end{array}$$

(30)

where $D_{\mathrm{cov}}⪰0$.

Next, we stress some technical remarks related to the new information theoretic measures in Definition 4 that can be obtained using known results in the literature and some known lower bounds that use the same objective function with (26)–(30).

Remark 2 

(Comments on Definition 4). It can be shown that the infimization problems (26), (28) and (30), in contrast to their operational counterparts (11)–(13) are convex with respect to their test channel. This can be shown following, for instance, the techniques of [23]. By the structural properties of the test channel derived in ([24], Section 4), if the source is first-order Markov, i.e., with distribution $\mathbf{P}(d{x}_t|x_{t-1}),t\in {\mathbb{N}}_0^n$, the test channel distribution is of the form $\mathbf{P}(d{y}_t|y^{t-1},x_t),t\in {\mathbb{N}}_0^n$. Finally, combining this structural result, with ([25], Theorem 1.8.6), it can be shown that if ${\mathbf{x}}^n$ is Gaussian then a jointly Gaussian process $\{({\mathbf{x}}_t,{\mathbf{y}}_t):t\in {\mathbb{N}}_0\}$ achieves a smaller value of the information rates, and if ${\mathbf{x}}^n$ is Gaussian and Markov, then the infimum in (26), (28) and (30) can be restricted to test channel distributions which are Gaussian, of the form $\mathbf{P}(d{y}_t|y_{t-1},x_t)$.

We recall that when the distortion constraint set contains only (9), its finite time horizon counterpart or the per instant distortion constraint $\mathbb{E}\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\}\le D_t\forall t$, we end up having the well known nonanticipatory-ϵ entropy [26] also found in the literature as sequential or nonanticipative $\mathrm{RDF}$ [27,28]. Nonanticipatory-ϵ entropy received significant interest during the last twenty years in an anthology of papers (see, e.g., [11,16,24,29,30,31]) due to its utility in control related and delay-constrained applications. Moreover, the characterizations in (29) and (30) do not appear to be manageable to solve using standard techniques, and no closed-form statements are available for the general RDF in the literature. For this reason, we will seek only for non-tight bounds.

In view of the above, in the sequel we characterize and compute lower bounds on Definitions 1–3 for Gauss–Markov processes and for Markov models driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ noise processes.

2.1. Characterization and Computation of Jointly Gaussian Processes

In this section, we assume that the augmented joint process $\{({\mathbf{x}}_t,{\mathbf{y}}_t):t\in {\mathbb{N}}_0\}$ is jointly Gaussian. We use this assumption to first characterize and then to compute optimally (26), (28) and (30).

We first use the following helpful lemma. We exclude the proof because it is already derived in other papers, see, e.g., [14,24]. The only modification is the augmented joint processes $\{({\mathbf{x}}_t,{\mathbf{y}}_t):t\in {\mathbb{N}}_0^n\}$.

Lemma 1 

(Realization of $\{{\mathbf{P}}^{*}(d{y}_t|y^{t-1},x_t):t\in {\mathbb{N}}_0^n\}$). Consider the class of conditionally Gaussian test channels $\{{\mathbf{P}}^{*}(d{y}_t|y^{t-1},x_t):t\in {\mathbb{N}}_0^n\}$. Then, the following statements hold.

(1) 

Any candidate of $\{{\mathbf{P}}^{*}(d{y}_t|y^{t-1},x_t):t\in {\mathbb{N}}_0^n\}$ can be realized by the recursion

$$\begin{array}{cc}\hfill {\mathbf{y}}_t=& H_t\left({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t-1}\right)+{\widehat{\mathbf{x}}}_{t|t-1}+{\mathbf{v}}_t,{\widehat{\mathbf{x}}}_{0|-1}=given,t\in {\mathbb{N}}_0^n,\hfill \end{array}$$

(31)

where ${\widehat{\mathbf{x}}}_{t|t-1}\triangleq \mathbb{E}\{{\mathbf{x}}_t|{\mathbf{y}}^{t-1}\}$, $\{{\mathbf{v}}_t\in {\mathbb{R}}^{p_1+p_2}\sim \mathcal{N}(0;{\mathrm{\Sigma }}_{{\mathbf{v}}_t}):t\in {\mathbb{N}}_0^n\}$ is an independent Gaussian process independent of $\{{\mathbf{w}}_t:t\in {\mathbb{N}}_0^{n-1}\}$ and ${\mathbf{x}}_0$, and $\{H_t\in {\mathbb{R}}^{(p_1+p_2)\times (p_1+p_2)}:t\in {\mathbb{N}}_0^n\}$ are time-varying deterministic matrices.

Moreover, the innovations process $\{{\mathbf{I}}_t\in {\mathbb{R}}^{p_1+p_2}:t\in {\mathbb{N}}_0^n\}$ of (31) is the orthogonal process defined by

$$\begin{array}{c}\hfill {\mathbf{I}}_t\triangleq {\mathbf{y}}_t-\mathbb{E}\left\{{\mathbf{y}}_t|{\mathbf{y}}^{t-1}\right\}=H_t\left({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t-1}\right)+{\mathbf{v}}_t,\end{array}$$

where ${\mathbf{I}}_t\sim \mathcal{N}(0;{\mathrm{\Sigma }}_{{\mathbf{I}}_t})$, ${\mathrm{\Sigma }}_{{\mathbf{I}}_t}=H_t{\mathrm{\Sigma }}_{t|t-1}{H}_t^{\mathrm{T}}+{\mathrm{\Sigma }}_{{\mathbf{v}}_t}$ and ${\mathrm{\Sigma }}_{t|t-1}\triangleq \mathbb{E}\left\{({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t-1}){({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t-1})}^{\mathrm{T}}|{\mathbf{y}}^{t-1}\right\}=\mathbb{E}\left\{({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t-1}){({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t-1})}^{\mathrm{T}}\right\}$.

(2) 

Let ${\widehat{\mathbf{x}}}_{t|t}\triangleq \mathbb{E}\{{\mathbf{x}}_t|{\mathbf{y}}^t\}$ and ${\mathrm{\Sigma }}_{t|t}\triangleq \mathbb{E}\left\{({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t}){({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t})}^{\mathrm{T}}|{\mathbf{y}}^t\right\}=\mathbb{E}\left\{({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t}){({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t})}^{\mathrm{T}}\right\}$. Then, $\{{\widehat{\mathbf{x}}}_{t|t-1},{\mathrm{\Sigma }}_{t|t-1}:t\in {\mathbb{N}}_0^n\}$ satisfy the following vector-valued equations:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\widehat{\mathbf{x}}}_{t|t-1}=A{\widehat{\mathbf{x}}}_{t-1|t-1},\hfill \\ \hfill & {\mathrm{\Sigma }}_{t|t-1}=A{\mathrm{\Sigma }}_{t-1|t-1}{A}^{\mathrm{T}}+{\mathrm{\Sigma }}_{{\mathbf{w}}_t},\hfill \\ \hfill & {\widehat{\mathbf{x}}}_{t|t}={\widehat{\mathbf{x}}}_{t|t-1}+N_t{\mathbf{I}}_t,\hfill \\ \hfill & N_t={\mathrm{\Sigma }}_{t|t-1}{H}_t^{\mathrm{T}}{\mathrm{\Sigma }}_{{\mathbf{I}}_t}^{-1}\left(KalmanGain\right),\hfill \\ \hfill & {\mathrm{\Sigma }}_{t|t}={\mathrm{\Sigma }}_{t|t-1}-{\mathrm{\Sigma }}_{t|t-1}{H}_t^{\mathrm{T}}{\mathrm{\Sigma }}_{{\mathbf{I}}_t}^{-1}{H}_t{\mathrm{\Sigma }}_{t|t-1},\hfill \end{array}\end{array}$$

(32)

where ${\mathrm{\Sigma }}_{t|t}⪰0$ and ${\mathrm{\Sigma }}_{t|t-1}\succ 0$.

(3) 

Using $\mathrm{MMSE}$ estimation via the vector-valued KF recursions of (32), the following finite dimensional characterizations of $R_{\mathrm{pd},[0,n]}^{LB,G}\left(D\right),R_{\mathrm{joint},[0,n]}^{LB,G}\left(D^{*}\right),R_{\mathrm{cov},[0,n]}^{LB,G}\left(D_{\mathrm{cov}}\right)$ can be obtained:

$$\begin{array}{cc}\hfill R_{\mathrm{pd},[0,n]}^{LB,G}\left(D\right)& =\underset{\begin{array}{c}{H}_t\in {\mathbb{R}}^{(p_1+p_2)\times \left(p_1+p_2\right)},{\mathrm{\Sigma }}_{{\mathbf{v}}_t}⪰0,t\in {\mathbb{N}}_0^n\\ 0\le {\tilde{\mathrm{\Sigma }}}_{ii,t}\le D_{ii},i=1,\dots ,p\end{array}}{inf}\frac{1}{2}\sum _{t=0}^n{\left[log\frac{|{\mathrm{\Sigma }}_{t|t-1}|}{|{\mathrm{\Sigma }}_{t|t}|}\right]}^{+},\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill R_{\mathrm{joint},[0,n]}^{LB,G}\left(D^{*}\right)& =\underset{\begin{array}{c}{H}_t\in {\mathbb{R}}^{(p_1+p_2)\times \left(p_1+p_2\right)},{\mathrm{\Sigma }}_{{\mathbf{v}}_t}⪰0,t\in {\mathbb{N}}_0^n\\ 0\le {\tilde{\mathrm{\Sigma }}}_{ii,t}\le D_{ii},i=1,\dots ,p\\ \frac{1}{n+1}{\sum }_{t=0}^n\mathrm{trace}\left((I_{p_1+p_2}-H_t){\mathrm{\Sigma }}_{t|t-1}{(I_{p_1+p_2}-H_t)}^{\mathrm{T}}+{\mathrm{\Sigma }}_{{\mathbf{v}}_t}\right)\le D_{{}^{\mathrm{T}}}\end{array}}{inf}\frac{1}{2}\sum _{t=0}^n{\left[log\frac{|{\mathrm{\Sigma }}_{t|t-1}|}{|{\mathrm{\Sigma }}_{t|t}|}\right]}^{+},\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill R_{[0,n]}^{LB,G}\left(D_{\mathrm{cov}}\right)& =\underset{\begin{array}{c}{H}_t\in {\mathbb{R}}^{(p_1+p_2)\times \left(p_1+p_2\right)},{\mathrm{\Sigma }}_{{\mathbf{v}}_t}⪰0,t\in {\mathbb{N}}_0^n\\ \frac{1}{n+1}{\sum }_{t=0}^n\left((I_{p_1+p_2}-H_t){\mathrm{\Sigma }}_{t|t-1}{(I_{p_1+p_2}-H_t)}^{\mathrm{T}}+{\mathrm{\Sigma }}_{{\mathbf{v}}_t}\right)⪯D_{\mathrm{cov}}\end{array}}{inf}\frac{1}{2}\sum _{t=0}^n{\left[log\frac{|{\mathrm{\Sigma }}_{t|t-1}|}{|{\mathrm{\Sigma }}_{t|t}|}\right]}^{+},\hfill \end{array}$$

(35)

where ${\tilde{\mathrm{\Sigma }}}_{ii,t}\triangleq \frac{1}{n+1}{\sum }_{t=0}^n{\left[(I_{p_1+p_2}-H_t){\mathrm{\Sigma }}_{t|t-1}{(I_{p_1+p_2}-H_t)}^{\mathrm{T}}+{\mathrm{\Sigma }}_{{\mathbf{v}}_t}\right]}_{ii}\ge 0$, $D\in [0,D_{max}]$ and $D^{*}\in [0,D_{max}^{*}]$.

We note that one can directly study the finite-dimensional characterizations of Lemma 1, (3), and try to come up with numerical solutions. However, it is much more insightful to use instead the identification of the design parameters $\{(H_t,{\mathrm{\Sigma }}_{{\mathbf{v}}_t}):t\in {\mathbb{N}}_0^n\}$ of the test-channel realization in (31). This approach is already done in [14,24] hence we state it without a proof. Note, however, that compared to [14,24] that assume distortion constraints like (9) (or the per time instant counterpart of (9), i.e., $\mathbb{E}\left\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\right\}\le D_t,\forall t$), here we assume augmented state-space models and various spatio-temporal distortion constraints, namely, per-dimension, jointly per-dimension and averaged total distortion constraints, and covariance matrix distortion constraint.

Lemma 2 

(Alternative characterizations of (33)–(35) via system identification). Consider Lemma 1 and set ${\Delta }_t\triangleq {\mathrm{\Sigma }}_{t|t}$ and ${\mathrm{\Lambda }}_t\triangleq {\mathrm{\Sigma }}_{t|t-1}$. Then, the following statements hold.

(1) 

The test-channel distribution $\mathbf{P}(d{y}_t|y^{t-1},x_t)$ admits the following linear Markov additive noise realization:

$$\begin{array}{c}\hfill {\mathbf{y}}_t=H_t{\mathbf{x}}_t+(I_{p_1+p_2}-H_t)A{\mathbf{y}}_{t-1}+{\mathbf{v}}_t,{\mathbf{y}}_{-1}=given,t\in {\mathbb{N}}_0^n,\end{array}$$

(36)

where

$$\begin{array}{c}\hfill H_t\triangleq I_{p_1+p_2}-{\Delta }_t{\mathrm{\Lambda }}_t^{-1},{\mathrm{\Sigma }}_{{\mathbf{v}}_t}\triangleq {\Delta }_t{H}_t^{\mathrm{T}}⪰0.\end{array}$$

(37)

(2) 

The finite-dimensional characterizations of $R_{\mathrm{pd},[0,n]}^{LB,G}\left(D\right),R_{\mathrm{joint},[0,n]}^{LB,G}\left(D^{*}\right),R_{\mathrm{cov},[0,n]}^{LB,G}\left(D_{\mathrm{cov}}\right)$ can be simplified to the following alternative characterizations:

$$\begin{array}{cc}\hfill R_{\mathrm{pd},[0,n]}^{LB,G}\left(D\right)& =\underset{0\le {\Delta }_{ii,t}\le D_{ii},i=1,\dots ,p,t\in {\mathbb{N}}_0^n}{inf}\frac{1}{2}\sum _{t=0}^n{\left[log\frac{|{\mathrm{\Lambda }}_t|}{|{\Delta }_t|}\right]}^{+},\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill R_{\mathrm{joint},[0,n]}^{LB,G}\left(D^{*}\right)& =\underset{\begin{array}{c}0\le {\Delta }_{ii,t}\le D_{ii},i=1,\dots ,p,t\in {\mathbb{N}}_0^n,\\ \frac{1}{n+1}{\sum }_{t=0}^n\mathrm{trace}\left({\Delta }_t\right)\le D_{{}^T}\end{array}}{inf}\frac{1}{2}\sum _{t=0}^n{\left[log\frac{|{\mathrm{\Lambda }}_t|}{|{\Delta }_t|}\right]}^{+},\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill R_{[0,n]}^{LB,G}\left(D_{\mathrm{cov}}\right)& =\underset{\frac{1}{n+1}{\sum }_{t=0}^n\left({\Delta }_t\right)⪯D_{\mathrm{cov}}}{inf}\frac{1}{2}\sum _{t=0}^n{\left[log\frac{|{\mathrm{\Lambda }}_t|}{|{\Delta }_t|}\right]}^{+},\hfill \end{array}$$

(40)

where ${\Delta }_{ii,t}$ is defined precisely as ${\tilde{\mathrm{\Sigma }}}_{ii,t}$.

Next, we give some technical remarks related to Lemma 2.

Remark 3 

(Special case and technical remarks).

(1) 

Clearly, if in the forward test-channel realization with additive noise, we assume that the block diagonal matrix $A=\mathbf{0}$ (null matrix), then, we recover the classical forward test-channel realization for vector memoryless Gaussian source subject to a $\mathrm{MSE}$ distortion (see, e.g., ([32], Chapter 4.5), ([15], Chapter 9.7)) given by

$$\begin{array}{c}\hfill {\mathbf{y}}_t=H_t{\mathbf{x}}_t+{\mathbf{v}}_t,t\in {\mathbb{N}}_0^n,\end{array}$$

(41)

and the coefficients in (37) give

$$\begin{array}{c}\hfill H_t\triangleq I_{p_1+p_2}-{\Delta }_t{\mathrm{\Sigma }}_{\mathbf{w}}^{-1}⪰0,{\mathrm{\Sigma }}_{{\mathbf{v}}_t}\triangleq {\Delta }_t{H}_t^{\mathrm{T}}⪰0.\end{array}$$

(42)

Moreover, the characterizations in (38)–(40) change in that ${\mathrm{\Lambda }}_t={\mathrm{\Sigma }}_{\mathbf{w}}$. Clearly, (42) can be seen as reverse-waterfilling design parameters.

(2) 

Compared to(1), we note that $H_t$ in (37) should not be confused with a positive semidefinite matrix defined in the usual quadratic form [33] but instead it can possibly be a non-symmetric matrix which however contains only real non-negative eigenvalues. This observation is important because it means that in general the design variables $({\Delta }_t,{\mathrm{\Lambda }}_t)$ do not commute like in the classical reverse-waterfilling problems for memoryless multivariate Gaussian random variables or in $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ processes (see, e.g., [19,25]).

(3) 

For jointly Gaussian processes, the linear forward realization in (36) is the optimal realization among all realizations for this problem because the KF is the optimal causal MMSE estimator. Beyond Gaussian processes, and when the noise is zero-mean, uncorrelated and white (in our setup these properties hold), the optimal realization for Gaussian processes becomes the best linear realization (see, e.g., ([34], §3.2) or ([35], p. 130)) and similarly the corresponding characterizations in (38)–(40) are the best linear characterizations. By saying “best-linear” realization and characterizations, respectively, we mean that there may be non-linear realizations and hence non-linear-based characterizations that outperform the best linear.

(4) 

The characterization of (39) is different from the characterization obtained in ([16], Theorem 1, (25e)) that uses weighted distortion constraints. The former optimization problem imposes hard constraints whereas the latter imposes soft constraints via weights. Nonetheless, an interesting open question is whether there exists a set of weights, which will give the same per dimension distortion when imposed as a weighted total distortion constraint.

(5) 

It should also be stressed that the per-dimension constraints on the diagonal entries of ${\Delta }_t$ are not the same as having constraints on the eigenvalues of ${\Delta }_t$. This further means that even for this class of distortion constraints, it is still possible to have rate-distortion resource allocation (i.e., a type of reverse-waterfilling optimization).

Remark 4 

(Convexity). The optimization problems in (38) and (39) are convex because the objective function is linear and the constraints are affine and positive semi-definite. Thus, the problem can be solved numerically using convex programming software (see, e.g, [36]) or the more challenging $\mathrm{KKT}$ conditions that are first-order necessary conditions for global optimality ([37], Chapter 5.3). The latter, will give certain non-linear matrix Riccati equations that need to be solved in order to construct a reverse-waterfilling algorithm.

Remark 5 

(Existence of Solution). A sufficient condition for existence of a solution with a finite value in (38)–(40) is to consider the strict LMI constraint $0\prec {\Delta }_t⪯{\mathrm{\Lambda }}_t$ that ensures the objective function is bounded. The strict LMI ensures that ${\Delta }_t\succ 0$ which further means that $D>0$, $D_{{}^{\mathrm{T}}}>0$ and $D_{\mathrm{cov}}\succ 0$.

In what follows, we derive lower bounds on (11)–(13).

Lemma 3 

(Steady-state lower bounds on (11)–(13)). Suppose that the conditions of Remark 5 hold. Moreover, let $\Delta \triangleq \frac{1}{n+1}{\sum }_{t=0}^n{\Delta }_t$ for some finite n. Then, the following statements hold.

(1) 

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^c\left(D\right)\ge \underset{\begin{array}{c}0\prec \Delta ⪯\mathrm{\Lambda }\\ {\Delta }_{ii}\le D_{ii},i=1,\dots ,p\end{array}}{min}\frac{1}{2}log\frac{|\mathrm{\Lambda }|}{|\Delta |},\end{array}$$

(43)

where $\mathrm{\Lambda }=A\Delta A^{\mathrm{T}}+{\mathrm{\Sigma }}_{\mathbf{w}}$.

(2) 

$$\begin{array}{c}\hfill R_{\mathrm{joint}}^c\left(D^{*}\right)\ge \underset{\begin{array}{c}0\prec \Delta ⪯\mathrm{\Lambda }\\ {\Delta }_{ii}\le D_{ii},i=1,\dots ,p\\ \mathrm{trace}(\Delta )\le D\end{array}}{min}\frac{1}{2}log\frac{|\mathrm{\Lambda }|}{|\Delta |},\end{array}$$

(44)

for some $D^{*}=min\{D_{{}^{\mathrm{T}}},D\}$.

(3) 

$$\begin{array}{c}\hfill R^c\left(D_{\mathrm{cov}}\right)\ge \underset{\begin{array}{c}0\prec \Delta ⪯\mathrm{\Lambda }\\ 0\prec \Delta ⪯D_{\mathrm{cov}}\end{array}}{min}\frac{1}{2}log\frac{|\mathrm{\Lambda }|}{|\Delta |}.\end{array}$$

(45)

Proof. 

See Appendix A. □

It should be remarked that instead of the derivation based on a convexity argument in Lemma 3, one can assume that the optimal minimizer $\mathbf{P}(d{y}_t|y_{t-1},x_t)$ that achieves (43)–(45) is time-invariant and the output distribution $\mathbf{P}(d{y}_t|y_{t-1})$ is also time-invariant with a unique invariant distribution, see, e.g., ([14], Theorem 3). Moreover, the optimal linear forward test-channel that achieves the lower bounds in (43)–(45) correspond to the time-invariant realization (36), given by

$$\begin{array}{c}\hfill {\mathbf{y}}_t=H{\mathbf{x}}_t+(I_{p_1+p_2}-H)A{\mathbf{y}}_{t-1}+{\mathbf{v}}_t,\end{array}$$

(46)

whereas the corresponding time-invariant scaling coefficients in (37) are as follows

$$\begin{array}{c}\hfill H\triangleq I_{p_1+p_2}-\Delta {\mathrm{\Lambda }}^{-1},{\mathrm{\Sigma }}_{\mathbf{v}}\triangleq \Delta H^{\mathrm{T}}⪰0.\end{array}$$

(47)

From Lemma 3, the following corollary can be immediately obtained.

Corollary 1 

(Fixed design variable $\Delta$). If in Lemma 2 we assume that ${\Delta }_t=\Delta$, $\forall t$, then we obtain (43)–(45).

Proof. 

This is immediate from the derivation of Lemma 3. □

In what follows, we show that the lower bounds in Lemma 3 are semidefinite representable, thus, they can be readily computed.

Theorem 2 

(Computation of the lower bounds in Lemma 3). Consider the variable $Q_1\triangleq {\Delta }^{-1}-A{}^{\mathrm{T}}{\mathrm{\Sigma }}_{\mathbf{w}}^{-1}A$, where $\Delta \succ 0$. Then, the following semidefinite programming representations hold.

(1) 

For some $D\equiv \mathrm{trace}\left(\widehat{D}\right)>0$, the lower bound in (43), denoted hereinafter by $R_{\mathrm{pd}}^{LB}\left(D\right)$, is semidefinite representable as follows:

$$\begin{array}{cc}\hfill R_{\mathrm{pd}}^{LB}\left(D\right)& =\underset{\begin{array}{c}{Q}_1\succ 0\end{array}}{min}-\frac{1}{2}log|Q_1|+\frac{1}{2}log|{\mathrm{\Sigma }}_{\mathbf{w}}|.\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& 0\prec \Delta ⪯\mathrm{\Lambda }\hfill \\ \hfill & {\Delta }_{ii}\le D_{ii},i=1,\dots ,p\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & \left[\begin{array}{cc}\Delta -Q_1& \Delta A{}^{\mathrm{T}}\\ A\Delta & \mathrm{\Lambda }\end{array}\right]⪰0\hfill \end{array}$$

(49)

(2) 

For some $D^{*}=min\{D_{{}^{\mathrm{T}}},D\}>0$, the lower bound in (44), denoted hereinafter by $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)$, is semidefinite representable as follows:

$$\begin{array}{cc}\hfill R_{\mathrm{joint}}^{LB}\left(D^{*}\right)& =\underset{\begin{array}{c}{Q}_1\succ 0\end{array}}{min}-\frac{1}{2}log|Q_1|+\frac{1}{2}log|{\mathrm{\Sigma }}_{\mathbf{w}}|.\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& 0\prec \Delta ⪯\mathrm{\Lambda }\hfill \\ \hfill & {\Delta }_{ii}\le D_{ii},i=1,\dots ,p\hfill \\ \hfill & \mathrm{trace}(\Delta )\le D_{{}^{\mathrm{T}}}\hfill \\ \hfill & \left[\begin{array}{cc}\Delta -Q_1& \Delta A{}^{\mathrm{T}}\\ A\Delta & \mathrm{\Lambda }\end{array}\right]⪰0\hfill \end{array}$$

(50)

(3) 

For some $D_{\mathrm{cov}}\succ 0$, the lower bound in (45), denoted hereinafter by $R^{LB}\left(D_{\mathrm{cov}}\right)$, is semidefinite representable as follows:

$$\begin{array}{cc}\hfill R^{LB}\left(D_{\mathrm{cov}}\right)& =\underset{\begin{array}{c}{Q}_1\succ 0\end{array}}{min}-\frac{1}{2}log|Q_1|+\frac{1}{2}log|{\mathrm{\Sigma }}_{\mathbf{w}}|.\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& 0\prec \Delta ⪯\mathrm{\Lambda }\hfill \\ \hfill & \Delta ⪯D_{\mathrm{cov}}\hfill \\ \hfill & \left[\begin{array}{cc}\Delta -Q_1& \Delta A{}^{\mathrm{T}}\\ A\Delta & \mathrm{\Lambda }\end{array}\right]⪰0\hfill \end{array}$$

(51)

Proof. 

See Appendix B. □

Next, we stress some comments on the semidefinite representation of the lower bounds in Theorem 2.

Remark 6 

(Comments on Theorem 2).

(1) 

We note that a similar characterization to the characterizations derived in Theorem 2 (subject to the distortion constraint (9) or the per-instant distortion constraint $\mathbb{E}\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\}\le D_t,\forall t$, for a special case of the setup in Figure 1) is recently derived in ([16], Equation (27)). The log-determinant convex optimization problems in Theorem 2 are widely used in systems and control theories because they are able to deal efficiently with LMIs [38].

(2) 

Recently, the efficiency of SDP algorithm in solving linear and non-linear optimization problems attracted experts from the field of information theory who noticed its usefulness in solving distributed source coding problems (see, e.g., [3,39]). Such log-determinant problems when solved using the semidefinite programming method are known to have polynomial worst-case complexity (see, e.g., [40]). In addition, for an interior point method such as the SDP approach, the most computationally expensive step is the Cholesky factorization involved in the Newton steps.

(3) 

On the other hand, due to its complexity, the SDP approach for high dimensional systems is often time consuming whereas for very large scale systems is occasionally impossible to obtain numerical solutions. Hence, ideally one could preferably consider alternative methods to solve a problem sacrificing for instance the optimality of the SDP algorithm but gaining in scalability and reducing the complexity. The most computationally efficient way to compute such problems and, additionally, to gain some insight from the solution is via the well-known reverse-waterfilling algorithm ([19], Theorem 10.3.3), which is however very hard to construct and compute because one needs to employ and solve complicated KKT conditions [37]. Such effort was recently made for multivariate Gauss-Markov processes under per instant, averaged total and asymptotically averaged total distortion constraints in [24,41].

Next, we perform some numerical illustrations using the semidefinite representations of Theorem 2. We also compare (48) and (50), to the known expression obtained only for the asymptotically averaged total MSE distortion constraint in ([16], Equation (27)). We note that the SDP algorithm for (48)–(51) is implemented using the CVX platform [36].

Example 1 

(Comparison of $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)$, $R_{\mathrm{pd}}^{LB}\left(D\right)$ and ([16], Equation (27))). For the system in (1), we assume that user 1 is described by a ${\mathbb{R}}^2$-valued time-invariant Markov source driven by $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process with parameters $(A_1,{\mathrm{\Sigma }}_{{\mathbf{w}}^1})$:

$$\begin{array}{c}\hfill (A_1,{\mathrm{\Sigma }}_{{\mathbf{w}}^1})=\left(\left[\begin{array}{cc}0.5& 0.2\\ 0.3& 0.6\end{array}\right],\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\right),\end{array}$$

(52)

whereas, user 2 is described by a ${\mathbb{R}}^3$-valued time-invariant Markov source driven by $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process with parameters $(A_2,{\mathrm{\Sigma }}_{{\mathbf{w}}^2})$:

$$\begin{array}{c}\hfill (A_2,{\mathrm{\Sigma }}_{{\mathbf{w}}^2})=\left(\left[\begin{array}{ccc}0.5& 0.2& 0.1\\ 0.3& 0.6& 0.1\\ 0.7& 0.3& 0.4\end{array}\right],\left[\begin{array}{ccc}1& 0& 0\\ 0& 0.2& 0\\ 0& 0& 0.5\end{array}\right]\right).\end{array}$$

(53)

Clearly, the augmented state space model (2) generates $A=A_1\oplus A_2$ and ${\mathrm{\Sigma }}_{\mathbf{w}}={\mathrm{\Sigma }}_{\mathbf{w}}^1\oplus {\mathrm{\Sigma }}_{\mathbf{w}}^2$. For this example, we assume that $D_{{}^{\mathrm{T}}}=1.5$ and $D_{11}=0.1,D_{22}=0.01,D_{33}=0.6,D_{44}=0.15,D_{55}=0.1$, which implies that $D=0.96$. This means that $D^{*}=min\{D_{{}^{\mathrm{T}}},D\}=0.96$.

In Figure 3, we compare the numerical solutions of $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)$ and $R_{\mathrm{pd}}^{LB}\left(D\right)$ with ([16], Equation (27)), denoted hereinafter as $R^{LB}\left(D_{{}^{\mathrm{T}}}\right)$.

Based on this numerical study, we observe that for distortion levels between $(0,D^{*}=D]$, $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)\ge R_{\mathrm{pd}}^{LB}\left(D\right)$ whereas for values of $D_{{}^{\mathrm{T}}}$ greater than $D^{*}$ we observe that $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)=R_{\mathrm{pd}}^{LB}\left(D\right)$ because the asymptotically averaged total MSE distortion constraint is inactive. This observation verifies our comment in Section 1.2 regarding the connection of (11) and (12). Clearly, at high rates (or high resolution) we observe that $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)\approx R^{LB}\left(D_{{}^{\mathrm{T}}}\right)$.

Another interesting observation (illustrated in Figure 4) that can be made, is that if in the same example we allocate the total budget of per dimension distortion equally, i.e., $D_{ii}=D_{jj},\forall i\ne j$, we observe that for distortion levels between $(0,D^{*}=D]$, $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)=R^{LB}\left(D_{{}^{\mathrm{T}}}\right)\ge R_{\mathrm{pd}}^{LB}\left(D\right)$.

Example 2 

(Covariance matrix distortion constraint). For the system in (1) we assume that user 1 is described by a ${\mathbb{R}}^2$-valued time-invariant Markov source driven by $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process with parameters $(A_1,{\mathrm{\Sigma }}_{{\mathbf{w}}^1})$:

$$\begin{array}{c}\hfill (A_1,{\mathrm{\Sigma }}_{{\mathbf{w}}^1})=\left(\left[\begin{array}{cc}0.5& 0.2\\ 0.3& 0.6\end{array}\right],\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\right),\end{array}$$

(54)

whereas, user 2 is described by an $\mathbb{R}$-valued time-invariant Markov source driven by $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process with parameters $(A_2,{\mathrm{\Sigma }}_{{\mathbf{w}}^2})$:

$$\begin{array}{c}\hfill (A_2,{\mathrm{\Sigma }}_{{\mathbf{w}}^2})=(0.6,2).\end{array}$$

(55)

The augmented state space model (2) is generated by $A=A_1\oplus A_2$ and similarly ${\mathrm{\Sigma }}_{\mathbf{w}}={\mathrm{\Sigma }}_{\mathbf{w}}^1\oplus {\mathrm{\Sigma }}_{\mathbf{w}}^2$. For this example, we assume a covariance matrix distortion constraint given by:

$$\begin{array}{c}\hfill D_{\mathrm{cov}}=\left[\begin{array}{ccc}1.5& \gamma & \gamma \\ \gamma & 1& \gamma \\ \gamma & \gamma & 0.5\end{array}\right],\end{array}$$

(56)

where $\gamma >0$ is the positive correlation coefficient between the distortion matrix components (i.e., diagonal entries) and it is chosen such that $D_{\mathrm{cov}}\succ 0$.

In Figure 5 we demonstrate a comparison between $R^{LB}\left(D_{\mathrm{cov}}\right)$ evaluated for several different values of γ. One interesting observation that can be made is that higher distortion correlation in (56) leads to less bits with a ${\gamma }_{max}\approx 0.53$, beyond which the value of $R^{LB}\left(D_{\mathrm{cov}}\right)$ remains unchanged. Another interesting observation is that for negative correlation γ, the approximation via SDP does not give a number. However, this is not the case, in general (see, e.g., ([42], Example 1)).

Using the same simulation study, we can arrive to an interesting connection between the approximation in (51) and (48). In particular, if for instance in (56) we restrict the matrix distortion constraint only to the main diagonal elements (i.e., exactly like the the per-dimension constraints) then, we obtain the plot of Figure 6 which clearly demonstrates that $R^{LB}\left(D_{\mathrm{cov}}\right)=R_{\mathrm{pd}}^{LB}\left(D\right)$. In fact, restricting the covariance matrix distortion constraint of (56) to the per dimension distortion constraint, is as if we optimize via a solution space in which γ is allowed to have any value in $\mathbb{R}$. As a result, the feasible set of solutions is larger when the constraint set is subject to per-dimension distortion constraints rather than the covariance matrix distortion constraint.

2.2. Analytical Lower Bounds for Markov Sources Driven by Additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Noise Processes

In this subsection, we derive analytical lower bounds on (11)–(13)) when the source model describing the behavior of user 1 or user 2 is driven by possibly $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ non-Gaussian noise process.

We first give a lemma which will facilitate the derivation of our lower bounds. We only consider the case of RDFs subject to per-dimension distortion constraints because the other classes of distortion constraints follow similarly.

Lemma 4 

(Rate-distortion bounds). For the augmented source model describing the behavior of users 1, 2 in (3), the following inequalities hold assuming distortion constraints in the class of (8):

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^{LB}\left(D\right)\stackrel{\left(a\right)}{\le }{R}_{\mathrm{pd}}^{LB,linear}\left(D\right)\le R_{\mathrm{pd}}^c\left(D\right),\end{array}$$

(57)

where

$$\begin{array}{cc}& R_{\mathrm{pd}}^{LB,linear}\left(D\right)\triangleq \underset{\begin{array}{c}{\mathbf{P}}^{linear}(d{y}_t|y_{t-1},x_t):t\in {\mathbb{N}}_0\\ {\mathrm{\Sigma }}_{{\Delta }_{ii}}\le D_{ii},i=1,\dots ,p\end{array}}{inf}\underset{n⟶\infty }{lim\; sup}\frac{1}{n+1}\sum _{t=0}^{n}I({\mathbf{x}}_t;{\mathbf{y}}_t|{\mathbf{y}}_{t-1}),\hfill \end{array}$$

(58)

and $\left(a\right)$ holds with equality if the augmented state space model described in (2) is jointly Gaussian and the optimal minimizer, i.e., ${\mathbf{P}}^{*}(d{y}_t|y^{t-1},x^t)$ of $R_{\mathrm{pd}}^{LB}\left(D\right)$ is conditionally Gaussian. Equality $\left(a\right)$ holds trivially at $D_{max}$.

Proof. 

The RHS inequality follows from Theorem 1 and (43) whereas the LHS inequality follows from the fact that the constraint set of $R_{\mathrm{pd}}^{LB}\left(D\right)$ is larger than the constraint set of $R_{\mathrm{pd}}^{LB,linear}\left(D\right)$ which is restricted to linear coding policies. Now, under the specific augmented source model in (3), and using Lemma 2, (1), we obtain $R_{\mathrm{pd}}^{LB,linear}\left(D\right)$ defined as in (58) because these are the best linear coding policies since KF algorithm is the best linear causal $\mathrm{MSE}$ estimator beyond additive Gaussian noise processes (see the discussion in Remark 3, (3)). Clearly, if the augmented source in (3) is jointly Gaussian and the optimal realization of $R_{\mathrm{pd}}^{LB}\left(D\right)$ is conditionally Gaussian, then, the system model is jointly Gaussian and the optimal policies are linear given by the forward linear test channel realization obtained in (36) hence the LHS inequality holds with equality. □

Remark 7 

(Comments on Lemma 4). We note that Lemma 4 holds if we assume RDFs with distortion constraints in the class of (9) or (10).

The following theorem is a major result of this paper.

Theorem 3 

(Analytical lower bounds on (11)–(13)). Consider the source models of users $1,2$ in (1). Then, the following analytical lower bounds on (11)–(13) hold.

(1) 

For $D_{ii}>0,\forall i$, we obtain

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^c\left(D\right)\ge \frac{(p_1+p_2)}{2}log\left(|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}+\frac{(p_1+p_2)N\left(\mathbf{w}\right)}{D}\right),\end{array}$$

(59)

where $D\in \left(0,\frac{(p_1+p_2)N\left(\mathbf{w}\right)}{1-|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}}\right]$ with $N\left(\mathbf{w}\right)=\frac{1}{2\pi e}{2}^{\frac{2}{p_1+p_2}h\left(\mathbf{w}\right)}$ and $h\left(\mathbf{w}\right)>-\infty$.

(2) 

For $D_{{}^{\mathrm{T}}}>0$, and $D_{ii}>0,\forall i$, we obtain

$$\begin{array}{c}\hfill R_{\mathrm{joint}}^c\left(D^{*}\right)\ge \frac{(p_1+p_2)}{2}log\left(|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}+\frac{(p_1+p_2)N\left(\mathbf{w}\right)}{D^{*}}\right),\end{array}$$

(60)

where $D^{*}\in \left(0,\frac{(p_1+p_2)N\left(\mathbf{w}\right)}{1-|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}}\right]$, with $N\left(\mathbf{w}\right)$ defined as in(1)and $h\left(\mathbf{w}\right)>-\infty$.

(3) 

For $D_{\mathrm{cov}}\succ 0$ we obtain

$$\begin{array}{c}\hfill R^c\left(D_{\mathrm{cov}}\right)\ge \frac{(p_1+p_2)}{2}log\left(|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}+\frac{N\left(\mathbf{w}\right)}{|D_{\mathrm{cov}}{|}^{\frac{1}{p_1+p_2}}}\right),\end{array}$$

(61)

where $|D_{\mathrm{cov}}|\in \left(0,{\left(\frac{N\left(\mathbf{w}\right)}{1-|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}}\right)}^{(p_1+p_2)}\right]$, with $N\left(\mathbf{w}\right)$ defined as in(1)and $h\left(\mathbf{w}\right)>-\infty$.

Proof. 

See Appendix C. □

The following technical remarks can be made regarding Theorem 3.

Remark 8. 

(1) 

Note that if in Theorem 3 we allow $h\left(\mathbf{w}\right)=-\infty$, then, the analytical lower bound expressions take a negative finite value or $-\infty$, which cannot be the case ($\mathrm{RDF}$ is, by definition, non-negative). A way to include the case where $h\left(\mathbf{w}\right)$ is allowed to be $-\infty$ in our lower bound expressions, is to set the objective functions in (59)–(61) to be ${[log(·)]}^{+}$. This will mean that whenever $h\left(\mathbf{w}\right)=-\infty$, the analytical lower bound expression will be zero.

(2) 

The analytical lower bounds in (59)–(61) do not correspond to the best linear forward test channel realization of Lemma 3 (see (46)) which is also the optimal policy under the assumption of a MMSE decoder when the system’s noise is purely Gaussian (see Remark 3,(3)). Moreover, it is not clear what is the realization that achieves them the same way the bounds in Lemma 3 are achieved for Gaussian processes.

(3) 

If in Theorem 3 we assume that the users $1,2$ have source models described by Markov processes driven by additive Gaussian noise processes then from the EPI (see, e.g., ([43], Equation (7))) $N\left(\mathbf{w}\right)=|{\mathrm{\Sigma }}_{\mathbf{w}}{|}^{p_1+p_2}$ and (59)–(61) change accordingly.

(4) 

One can choose to further bound (61) using the inequality $|D_{\mathrm{cov}}{|}^{-\frac{1}{p_1+p_2}}\ge \frac{p_1+p_2}{\mathrm{trace}\left(D_{\mathrm{cov}}\right)}$ obtaining a further lower bound that coincides with the lower bound in (59) (see also the discussion in Example 2). Such lower bound will mean that we extend the set of feasible solutions that correspond to the initial problem statement (13), to be similar to the initial problem statement of (11) which cannot be the case, in general. Our bound in (61) encapsulates the off diagonal elements of the distortion covariance matrix distortion $D_{\mathrm{cov}}$ hence it is an appropriate lower bound for the specific problem.

In what follows, we give a numerical simulation where we compare the solution of $R_{\mathrm{joint}}^{LB}\left(D^{*}\right)$ (that corresponds to the lower bound achieved by the optimal coding policies when the system is driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise processes) computed via the SDP representation of (50), with the lower bound obtained in (60) when the system’s noise is also Gaussian.

Example 3 

(Comparison of (50) with (60) for jointly Gaussian processes). We consider the same input data assumed in Example 1 for users $1,2$. Then, we proceed to compute the true lower bound of (50) and the lower bound obtained in (60).

Our simulation study in Figure 7 shows that at high rates the performance of the two bounds is almost identical whereas to moderate and low rates we observe a gap that remains constant when $D_{{}^{\mathrm{T}}}\ge D$, i.e., when the asymptotically averaged total distortion constraint is inactive. The same behavior is expected for systems of larger dimension (larger scale optimization problems) with a possibility of an increased gap to moderate and low rates depending on the structure of the block diagonal matrices A and ${\mathrm{\Sigma }}_{\mathbf{w}}$.

Next, we state a corollary of Theorem 3.

Corollary 2 

(Analytical bounds when users $1,2$ are not specified by the same additive noise process). Consider the source models of users $1,2$ in (1). Moreover, assume that ${\mathbf{w}}_t^1\sim \mathcal{N}(0;{\mathrm{\Sigma }}_{{\mathbf{w}}^1})$ and ${\mathbf{w}}_t^2\sim (0;{\mathrm{\Sigma }}_{{\mathbf{w}}^2})$ with ${\mathrm{\Sigma }}_{{\mathbf{w}}^1}⪯I_{p_1}$ and ${\mathrm{\Sigma }}_{{\mathbf{w}}^2}⪯I_{p_2}$ and $h\left({\mathbf{w}}^2\right)>-\infty$. Then, the following analytical lower bounds on (11)–(13) hold.

(1) 

For $D_{ii}>0,\forall i$, we obtain

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^c\left(D\right)\ge \frac{(p_1+p_2)}{2}log\left(|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}+\frac{(p_1+p_2){|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}|}^{\frac{1}{p_1}}N\left({\mathbf{w}}^2\right)}{D}\right),\end{array}$$

(62)

where $D\in \left(0,\frac{(p_1+p_2)N\left(\mathbf{w}\right)}{1-|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}}\right]$ with $N\left({\mathbf{w}}^2\right)=\frac{1}{2\pi e}{2}^{\frac{2}{p_2}h\left({\mathbf{w}}^2\right)}$.

(2) 

For $D_{{}^{\mathrm{T}}}>0$, $D_{ii}>0,\forall i$, we obtain

$$\begin{array}{c}\hfill R_{\mathrm{joint}}^c\left(D^{*}\right)\ge \frac{(p_1+p_2)}{2}log\left(|A^T{A|}^{\frac{1}{p_1+p_2}}+\frac{(p_1+p_2){|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}|}^{\frac{1}{p_1}}N\left({\mathbf{w}}^2\right)}{D^{*}}\right),\end{array}$$

(63)

where $D^{*}\in \left(0,\frac{(p_1+p_2){|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}|}^{\frac{1}{p_1}}N\left({\mathbf{w}}^2\right)}{1-|A^T{A|}^{\frac{1}{p_1+p_2}}}\right]$.

(3) 

For $D_{\mathrm{cov}}\succ 0$ we obtain

$$\begin{array}{c}\hfill R^c\left(D_{\mathrm{cov}}\right)\ge \frac{(p_1+p_2)}{2}log\left(|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}+\frac{|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}{|}^{\frac{1}{p_1}}N\left({\mathbf{w}}^2\right)}{|D_{\mathrm{cov}}{|}^{\frac{1}{p_1+p_2}}}\right),\end{array}$$

(64)

where $|D_{\mathrm{cov}}|\in \left(0,{\left(\frac{|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}{|}^{\frac{1}{p_1}}N\left({\mathbf{w}}^2\right)}{1-|A^{\mathrm{T}}{A|}^{\frac{1}{p_1+p_2}}}\right)}^{(p_1+p_2)}\right]$.

Proof. 

All cases (1)–(3) follow almost identical steps with the derivation of Theorem 3. The only different but crucial step lies in (A6) where we then use the fact that

$$\begin{array}{c}\hfill |{\mathrm{\Sigma }}_{\mathbf{w}}{|}^{\frac{1}{p_1+p_2}}\stackrel{\left(a\right)}{=}|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}{|}^{\frac{1}{p_1+p_2}}|{\mathrm{\Sigma }}_{{\mathbf{w}}^2}{|}^{\frac{1}{p_1+p_2}}\stackrel{\left(b\right)}{\ge }|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}{|}^{\frac{1}{p_1}}|{\mathrm{\Sigma }}_{{\mathbf{w}}^2}{|}^{\frac{1}{p_2}}\stackrel{\left(c\right)}{\ge }{|{\mathrm{\Sigma }}_{{\mathbf{w}}^1}|}^{\frac{1}{p_1}}N\left({\mathbf{w}}^2\right),\end{array}$$

(65)

where $\left(a\right)$ follows from properties of block diagonal matrices ([33], Section 0.9.2); $\left(b\right)$ follows from the conditions of the corollary on the noise covariance matrices; $\left(c\right)$ follows from the EPI ([43], Equation (7)). □

One can deduce the following for Corollary 2.

Remark 9. 

Corollary 2 will give similar analytical lower bounds (with appropriate modifications) if instead of user 1, we assume that the source model of user 2 is driven by a Gaussian noise process. The additional assumption on the covariance matrix of the noise process in both users is imposed because otherwise we cannot guarantee that the key series of inequalities (65) will be satisfied.

3. Upper Bounds

In this section we explain the case of encoding the augmented vector-valued Markov source modeled by (3) using a sequential causal DPCM scheme with a feedback loop followed by an ECDQ. The scheme relies on the linear forward test channel realization of the bounds in Lemma 2. The precursor of the DPCM-based scheme with feedback loop is [14] whereas ECDQ is a classical source coding approach with standard performance guarantees in information theory (see, e.g., [44]). The ECDQ scheme is utilized to bound the rate performance of the DPCM scheme. This approach furnish with an achievable (upper) bound the operation causal RDFs in (11)–(13).

3.1. DPCM with Feedback Loop

First, we briefly describe the sequential causal DPCM scheme with feedback loop introduced in ([14], Figure 2) (see also [45]). Observe that because the augmented source is modeled as a first-order multidimensional Markov process, the sequential causal coding is precisely equivalent to a predictive coding paradigm (see, e.g., [14,46]).

At each time instant t, the encoder or innovations’ encoder performs the linear operation

$$\begin{array}{c}\hfill {\widehat{\mathbf{x}}}_t={\mathbf{x}}_t-A{\mathbf{y}}_{t-1},\end{array}$$

(66)

where at $t=0$ we assume initial data ${\widehat{\mathbf{x}}}_0={\mathbf{x}}_0$ and also ${\mathbf{y}}_{t-1}\triangleq \mathbb{E}\left\{{\mathbf{x}}_{t-1}|{\ell }^{t-1}\right\}$, i.e., an estimate of ${\mathbf{x}}_{t-1}$ given the previous quantized symbols ${\ell }^{t-1}$ (Note that the process ${\widehat{\mathbf{x}}}_t$ has a temporal correlation since it subtracts the error of ${\mathbf{x}}_t$ given all previous quantized symbols ${\ell }^{t-1}$ and not the infinite past of the source ${\mathbf{x}}_{-\infty }^t$. Hence, ${\widehat{\mathbf{x}}}_t$ is only an estimate of the true process and this causes a part of the sub-optimality of this scheme.). Then, by means of a ${\mathbb{R}}^{p_1+p_2}$-valued MMSE quantizer that operates at a rate $R_t$, we generate the quantized reconstruction ${\widehat{\mathbf{y}}}_t$ of the residual source ${\widehat{\mathbf{x}}}_t$ denoted by ${\widehat{\mathbf{y}}}_t={\mathbf{y}}_t-A{\mathbf{y}}_{t-1}$. Then, we send ${\ell }_t$ over the channel (the corresponding data packet to ${\widehat{\mathbf{y}}}_t$). At the decoder we receive ${\ell }_t$ and recover the quantized symbol ${\widehat{\mathbf{y}}}_t$ of ${\widehat{\mathbf{x}}}_t$.

Then, we generate the estimate ${\mathbf{y}}_t$ using the linear operation

$$\begin{array}{c}\hfill {\mathbf{y}}_t={\widehat{\mathbf{y}}}_t+A{\mathbf{y}}_{t-1}.\end{array}$$

(67)

Combining both (66) and (67), we obtain

$$\begin{array}{c}\hfill {\mathbf{x}}_t-{\mathbf{y}}_t={\widehat{\mathbf{x}}}_t-{\widehat{\mathbf{y}}}_t.\end{array}$$

(68)

From (68), we can immediately deduce that the error between ${\mathbf{x}}_t$ and ${\mathbf{y}}_t$ is equal to the quantization error introduced by ${\widehat{\mathbf{x}}}_t$ and ${\widehat{\mathbf{y}}}_t$ which means that the $\mathrm{MSE}$ distortion at each instant of time satisfy

$$\begin{array}{c}\hfill \mathbb{E}\{||{\mathbf{x}}_t-{\mathbf{y}}_t{||}_2^2\}=\mathbb{E}\{||{\widehat{\mathbf{x}}}_t-{\widehat{\mathbf{y}}}_t{||}_2^2\}.\end{array}$$

(69)

In addition, the covariance matrix ${\mathrm{\Sigma }}_{\Delta }$ yields

$$\begin{array}{c}\hfill \mathbb{E}\left\{({\mathbf{x}}_t-{\mathbf{y}}_t){({\mathbf{x}}_t-{\mathbf{y}}_t)}^{\mathrm{T}}\right\}=\mathbb{E}\left\{({\widehat{\mathbf{x}}}_t-{\widehat{\mathbf{y}}}_t){({\widehat{\mathbf{x}}}_t-{\widehat{\mathbf{y}}}_t)}^{\mathrm{T}}\right\}.\end{array}$$

(70)

A pictorial view of the DPCM scheme with feedback loop is given in Figure 8.

3.2. Bounding (11)–(13) via A DPCM-based ECDQ for Gaussian Noise Processes

In this subsection, we bound the rate performance of the DPCM scheme described in Section 3.1 in the infinite time horizon, using a scheme that utilizes the steady-state linear forward test-channel realization that achieves the lower bounds of Lemma 3. Essentially, what we do in this scheme is that we replace the quantization noise with an additive Gaussian noise with the same second moments (see e.g., [47] or [44] (Chapter 5) and the references therein).

Recall that the steady-state linear forward test-channel realization of the lower bounds in Lemma 3 is written as follows:

$$\begin{array}{c}\hfill {\mathbf{y}}_t=H{\mathbf{x}}_t+(I_{p_1+p_2}-H)A{\mathbf{y}}_{t-1}+{\mathbf{v}}_t,\end{array}$$

(71)

whereas the steady-state reverse-waterfilling parameters $(H,{\mathrm{\Sigma }}_{\mathbf{v}})$ are given by

$$\begin{array}{c}\hfill H\triangleq I_{p_1+p_2}-\Delta {\mathrm{\Lambda }}^{-1},{\mathrm{\Sigma }}_{\mathbf{v}}=H\Delta ⪰0.\end{array}$$

(72)

The forward test-channel realization of (71) is illustrated in Figure 9.

Before we proceed, we point out the following important technical remarks on the realization of (71) and the coefficients (72).

Remark 10 

(Observations (71) and (72)). The linear forward test channel realization with additive noise in (71) is equivalent to the steady-state realization in (46) because for both it can be shown that the MSE distortion constraint is achieved (i.e., ${\mathbf{v}}_t\sim \mathcal{N}(0;{\mathrm{\Sigma }}_{\mathbf{v}})$, ${\mathrm{\Sigma }}_{\mathbf{v}}=H\Delta =\Delta H^{\mathrm{T}}⪰0$). Moreover, this realization is equivalent but simpler to build compared to the forward test channel realization introduced in [14] in which non-singular matrices and diagonalization by congruence is assumed (see ([14], Theorem 4)).

In the test channel realization of Figure 9, a reverse-waterfilling in spatial dimension is possible when we assume asymptotically averaged total MSE distortion constraints similar to ([14], Theorem 40). This reverse-waterfilling is dictated by the rank of matrix H. To make this point clear, if H is full rank, then all spatial dimensions in the system are active whereas if H is rank deficient, then, some dimensions are inactive (these dimensions form the null space and the nullity of H) and for these dimensions the rate is zero hence they can be excluded from the realization of Figure 9. In the sequel, we will present simulations where we study the reverse-waterfilling in the spatial domain under a certain distortion constraint studied in this paper.

Pre/Post Filtered ECDQ with multiplicative factors for augmented multivariate Gauss–Markov sources and spatial reverse-waterfilling. First, we consider a $\mathrm{rank}\left(H\right)—$dimensional lattice quantizer $Q_{\mathrm{rank}\left(H\right)}(·)$ [48] such that

$$\begin{array}{c}\hfill \mathbb{E}\left\{{\mathbf{z}}_t{\mathbf{z}}_t^{\mathrm{T}}\right\}={\mathrm{\Sigma }}_{{\mathbf{v}}^c},{\mathrm{\Sigma }}_{{\mathbf{v}}^c}\succ 0,\end{array}$$

where ${\mathbf{z}}_t\in {\mathbb{R}}^{\mathrm{rank}\left(H\right)}$ is a random dither vector generated both at the encoder and the decoder independent of the input signals ${\widehat{\mathbf{x}}}_t$ and the previous realizations of the dither, uniformly distributed over the basic Voronoi cell of the $\mathrm{rank}\left(H\right)—$dimensional lattice quantizer $Q_{\mathrm{rank}\left(H\right)}(·)$ such that ${\mathbf{v}}_t^c\sim Unif(0;{\mathrm{\Sigma }}_{{\mathbf{v}}_t^c})$. At the encoder the lattice quantizer quantize $H{\widehat{\mathbf{x}}}_t+{\mathbf{z}}_t$, that is, $Q_{\mathrm{rank}\left(H\right)}(H{\widehat{\mathbf{x}}}_t+{\mathbf{z}}_t)$, where ${\widehat{\mathbf{x}}}_t$ is given by (66). Then, the encoder applies conditional entropy coding to the output of the quantizer and transmits the output of the entropy coder. At the decoder the coded bits are received and the output of the quantizer is reconstructed, i.e., $Q_{\mathrm{rank}\left(H\right)}(H{\widehat{\mathbf{x}}}_t+{\mathbf{z}}_t)$. Then, it generates an estimate by subtracting ${\mathbf{z}}_t$ from the quantizer’s output and multiplies the result by $I_{\mathrm{rank}\left(H\right)}$ ($I_{\mathrm{rank}\left(H\right)}$ denotes the identity matrix with dimensions according to the rank of H. This identity matrix can be excluded but we include it here for completeness.) as follows:

$$\begin{array}{c}\hfill {\mathbf{y}}_t=I_{\mathrm{rank}\left(H\right)}(Q_{\mathrm{rank}\left(H\right)}(H{\widehat{\mathbf{x}}}_t+{\mathbf{z}}_t)-{\mathbf{z}}_t),\end{array}$$

(73)

Performance. The coding rate at each instant of time of the conditional entropy of the $\mathrm{MMSE}$ quantizer is given by

$$\begin{array}{cc}\hfill H(Q_{\mathrm{rank}\left(H\right)}|{\mathbf{z}}_t)& =I(H{\widehat{\mathbf{x}}}_t;H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t^c)\hfill \\ \hfill & \stackrel{\left(a\right)}{=}I(H{\widehat{\mathbf{x}}}_t;H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)+\mathbb{D}({\mathbf{v}}_t^c||{\mathbf{v}}_t)-\mathbb{D}(H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t^c||H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }I(H{\widehat{\mathbf{x}}}_t;H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)+\mathbb{D}({\mathbf{v}}_t^c||{\mathbf{v}}_t)\hfill \\ \hfill & \stackrel{\left(c\right)}{\le }I(H{\widehat{\mathbf{x}}}_t;H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)\hfill \\ \hfill & \stackrel{\left(d\right)}{\le }I({\mathbf{x}}_t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1})+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)\hfill \end{array}$$

(74)

where ${\mathbf{v}}_t^c\in {\mathbb{R}}^{\mathrm{rank}\left(H\right)}$ is the (uniform) coding noise in the ECDQ scheme and ${\mathbf{v}}_t$ is the corresponding Gaussian counterpart; $\left(a\right)$ follows because the two random vectors ${\mathbf{v}}_t^c,{\mathbf{v}}_t$ have the same second moments hence we can use the identity $\mathbb{D}(\mathbf{x}||{\mathbf{x}}^{\prime })=h\left({\mathbf{x}}^{\prime }\right)-h\left(\mathbf{x}\right)$; $\left(b\right)$ follows because $\mathbb{D}(H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t^c||H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)\ge 0$; $\left(c\right)$ follows because the divergence of the coding noise from Gaussianity is less than or equal to $\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)$[47] where $G_{\mathrm{rank}\left(H\right)}$ is the dimensionless normalized second moment of the lattice ([44], Definition 3.2.2); $\left(d\right)$ follows from data processing properties, namely, $I({\mathbf{x}}_t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1})\stackrel{(*)}{=}I({\mathbf{x}}_t;{\mathbf{y}}_t|{\mathbf{y}}_{t-1})\stackrel{(**)}{=}I({\widehat{\mathbf{x}}}_t;{\widehat{\mathbf{y}}}_t)\stackrel{(***)}{\ge }I(H{\widehat{\mathbf{x}}}_t;H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)$ where $(*)$ follows from the realization of (71), $(**)$ follows from the fact that ${\widehat{\mathbf{x}}}_t$ and ${\widehat{\mathbf{y}}}_t$ (obtained by (67)) are independent of ${\mathbf{y}}_{t-1}$, and $(***)$ is a consequence of data processing inequality since $(H{\widehat{\mathbf{x}}}_t+{\mathbf{v}}_t)\leftrightarrow {\widehat{\mathbf{x}}}_t\leftrightarrow H{\widehat{\mathbf{x}}}_t$. Under the assumption that the clocks of the entropy encoder and entropy decoder in the ECDQ scheme are synchronized, then, the total coding rate is obtained as follows

$$\begin{array}{cc}\hfill \sum _{t=1}^n{R}_t\le \sum _{t=0}^n(H(Q_{\mathrm{rank}\left(H\right)}|{\mathbf{z}}_t)& \stackrel{\left(e\right)}{\le }\sum _{t=0}^{n}I({\mathbf{x}}_t;{\mathbf{y}}_t|{\mathbf{y}}^{t-1})+\frac{(n+1)\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)\hfill \\ \hfill & \stackrel{\left(f\right)}{=}\frac{1}{2}\sum _{t=0}^n{log}_2\frac{|{\mathrm{\Lambda }}_t|}{|{\Delta }_t|}+\frac{(n+1)\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right),\hfill \end{array}$$

(75)

where $\left(e\right)$ follows from (74); $\left(f\right)$ follows from the derivation of Lemma 2.

The previous analysis yields the following theorem.

Theorem 4 

(Achievability bound on (11)–(13)). Suppose that ${\Delta }_t=\Delta ,\forall t$ and assume that the users $1,2$ source models (1) are driven by Gaussian noise processes. Then, the augmented state space source model in (3) ensures the following achievability bounds on (11)–(13), as follows

$$\begin{array}{cc}& R_{\mathrm{pd}}^c\left(D\right)\le R_{\mathrm{pd}}^{LB}\left(D\right)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)\hfill \end{array}$$

(76)

$$\begin{array}{cc}& R_{\mathrm{joint}}^c\left(D^{*}\right)\le R_{\mathrm{joint}}^{LB}\left(D^{*}\right)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right),\hfill \end{array}$$

(77)

$$\begin{array}{cc}& R^c\left(D_{\mathrm{cov}}\right)\le R^{LB}\left(D_{\mathrm{cov}}\right)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right).\hfill \end{array}$$

(78)

Proof. 

Under the conditions of the Theorem and the ECDQ scheme that leads to (75), the RHS terms in (76)–(78) are all constants. Then, taking the limit in both sides of (76)–(78) and then the appropriate infimization (minimization) constraint sets, the result follows. □

We wish to point out the following for Theorem 4.

Remark 11 

(Comments on Theorem 4).

(1) 

The ECDQ that leads to (75) is not the same as the standard symmetric ECDQ scheme for scalar-valued processes, i.e., when the coefficient H breaks into two pre and post additive noise channel scalings that tune the MSE distortion (see, e.g., [44,47]). In our pre-post scaled ECDQ scheme we take asymmetric coefficients based on the realization of Figure 9. This leads to a coarser lattice than the one used for the unscaled ECDQ (for details see for instance [44]).

(2) 

Since the upper bound essentially relies on the obtained lower bound for all (76)–(78), this means that similar observations can be made. For instance, if $D<D_{{}^T}$, then, (77) recovers (76), i.e., the asymptotically averaged total distortion constraint is inactive. Moreover, we cannot claim tightness of the achievability bound in (78) because the lower bound is already non-tight.

Next, we give an example where we compare the RL gap for various distortion levels of the achievability bound obtained in Theorem 4 and the lower bound obtained in Theorem 2 for the operational causal RDF with joint distortion constraints.

Example 4 

(RL gap of achievability and lower bounds). In this example, we plot lower and upper bounds on the operational causal RDF subject to joint distortion constraints using the bounds derived in (50) and (77), respectively. We first consider the same input data assumed in Example 1 for users $1,2$. Then, we proceed to compute the lower bound via (50) and the achievability bound in (77). After the first numerical study, we consider another one for which we only change the Gaussian noise process for both users $1,2$, as follows

$$\begin{array}{c}\hfill ({\mathrm{\Sigma }}_{{\mathbf{w}}^1},{\mathrm{\Sigma }}_{{\mathbf{w}}^2})=\left(\left[\begin{array}{cc}1& 0.5\\ 0.5& 1\end{array}\right],\left[\begin{array}{ccc}1.4039& 0.6034& 0.5165\\ 0.6034& 0.9563& 0.7682\\ 0.5165& 0.7682& 0.6620\end{array}\right]\right).\end{array}$$

(79)

Using the data of Example 1, and the same $D_{{}^{\mathrm{T}}}$ and $D_{ii},i=1,2,3,4,5$, we obtain the plots of Figure 10. For this study, we have used a Schläfli lattice (for details on this lattice see, e.g., [48]). ${\tilde{D}}_5$ with a dimensionless normalized second moment of the lattice $G_5\approx 0.0756$ bits. In this example H is always full rank and the RL gap is constant at $0.9218$ bits/augmented vector.

For the second example, we obtain the plots of Figure 11. For this study, we have used the dimensionless normalized second moment of a Schläfli lattice ${\tilde{D}}_5$ for the full rank case and for the rank deficient cases a Schläfli lattice $D_4$ with a dimensionless normalized second moment of the lattice $G_4\approx 0.0766$ bits. Similar to the first study, when H is always full rank, the RL gap is $0.9218$ bits/augmented vector whereas for H rank deficient the RL gap is $0.7754$ bits/augmented vector.

3.3. Bounding (11)–(13) via a DPCM-based ECDQ for Non-Gaussian Noise Processes

Similar to Lemma 4 where linear policies are the benchmark to derive lower bounds on (11)–(13), in this subsection we derive upper bounds on (11)–(13) using the linear test channel realization in Figure 9 and the DPCM-based ECDQ scheme of Section 3.1 and Section 3.2.

Next, we state the following theorem.

Theorem 5 

(Achievability bound on (11)–(13) for additive non-Gaussian noise process). Suppose that ${\Delta }_t=\Delta ,\forall t$ and assume that the users $1,2$ source models (1) are driven by non-Gaussian noise processes. Then, the augmented state space source model in (3) ensures the following achievability bounds on (11)–(13), as follows

$$\begin{array}{cc}& R_{\mathrm{pd}}^c\left(D\right)\le R_{\mathrm{pd}}^{LB,linear}\left(D\right)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)+\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G)\hfill \end{array}$$

(80)

$$\begin{array}{cc}& R_{\mathrm{joint}}^c\left(D^{*}\right)\le R_{\mathrm{joint}}^{LB,linear}\left(D^{*}\right)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)+\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G),\hfill \end{array}$$

(81)

$$\begin{array}{cc}& R^c\left(D_{\mathrm{cov}}\right)\le R^{LB,linear}\left(D_{\mathrm{cov}}\right)+\frac{\mathrm{rank}\left(H\right)}{2}log\left(2\pi e{G}_{\mathrm{rank}\left(H\right)}\right)+\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G).\hfill \end{array}$$

(82)

where $\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G)$ is the KL divergence between the residual source ${\widehat{\mathbf{x}}}_t$ assuming linear policies and the Gaussian residual source ${\widehat{\mathbf{x}}}_t^G\sim \mathcal{N}(0;\mathrm{\Lambda })$ in Figure 9.

Proof. 

We only prove (80) because (81) and (82) follow similarly. In addition, in parts we sketch the derivation because it is clear from the previous results. From Lemmas 4 and 3, we can easily obtain the following lower bound (similar to SLB [32])

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^c\left(D\right)\ge R_{\mathrm{pd}}^{LB,linear}\left(D\right)-\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G),\end{array}$$

(83)

where $\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G)\ge 0$ is the discrepancy between the residual source ${\widehat{\mathbf{x}}}_t$ assuming linear policies and the optimal Gaussian residual source ${\widehat{\mathbf{x}}}_t^G\sim \mathcal{N}(0;\mathrm{\Lambda })$. From (83), we obtain

$$\begin{array}{c}\hfill R_{\mathrm{pd}}^{LB,linear}\left(D\right)\le R_{\mathrm{pd}}^c\left(D\right)+\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G).\end{array}$$

(84)

Then, applying the DPCM-based ECDQ scheme based on the linear forward test channel realization of Figure 9 discussed in Section 3.1 and Section 3.2 we obtain (76). Since the coding scheme is obtained because we have assumed linear policies, $R_{\mathrm{pd}}^{LB}\left(D\right)$ will be replaced by $R_{\mathrm{pd}}^{LB,linear}\left(D\right)$. This completes the derivation. □

Remark 12 

(Comments on Theorem 5). Clearly, Theorem 5 is a generalization of Theorem 4 under the assumption of the linear realization of Figure 9 with systems driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ non-Gaussian noise process. If in (80)–(82), we assume that the system is driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ Gaussian noise process, then, clearly $\mathbb{D}(\widehat{\mathbf{x}}||{\widehat{\mathbf{x}}}^G)=0$ and Theorem 5 recovers Theorem 4.

4. Conclusions and Future Research

In this paper, we derived bounds on the OPTA of a two-user MIMO causal encoding and causal decoding problem (assuming the clocks of the encoder and the decoder to be synchronized). In our setup, each one of the users is described by a multivariate Markov source driven by additive $\mathrm{i}.\mathrm{i}.\mathrm{d}.$ noise process (possibly non-Gaussian) subject to three classes of spatio-temporal distortion constraints.

Although not directly pursued in this paper, all the results can be easily generalized to any finite number of users in Figure 1. Moreover, as a future research we aim to study the case of separate encoding for each user which will be a generalized version of a multi-user (distributed) source coding setup. Finally, due to the lack of insight of our results (mainly because we employed the general SDP solvers to compute our bounds) it makes sense to consider more specific setups and try to solve them using KKT conditions and, then, identify structural properties of matrices $(A,{\mathrm{\Sigma }}_{\mathbf{w}})$ for which KKT conditions can be optimally solved.