On the Complexity Reduction of Coding WSS Vector Processes by Using a Sequence of Block Circulant Matrices

Abstract

In the present paper, we obtain a result on the rate-distortion function (RDF) of wide sense stationary (WSS) vector processes that allows us to reduce the complexity of coding those processes. To achieve this result, we propose a sequence of block circulant matrices. In addition, we use the proposed sequence to reduce the complexity of filtering WSS vector processes.

1. Introduction

Applications where Toeplitz matrices arise commonly involve computation of inverses, products, and eigenvalues. Although inverses and products of Toeplitz matrices are not in general Toeplitz, inverses and products of circulant matrices are circulant. Moreover, unlike for Toeplitz matrices, an eigenvalue decomposition is known for any circulant matrix. Hence, the substitution of Toeplitz matrices by circulant matrices leads to a notable reduction of the computational complexity in those applications.

A sequence of Toeplitz matrices $\left\{T_n\left(f\right)\right\}$ generated by a function f can be found in any information-theoretic or statistical signal processing application in which a wide sense stationary (WSS) process appears. In those applications, $\left\{T_n\left(f\right)\right\}$ is the sequence of correlation matrices of the WSS process and the function f is its power spectral density (PSD).

The substitution of the sequence $\left\{T_n\left(f\right)\right\}$ by a sequence of circulant matrices can be done when both sequences are asymptotically equivalent. In [1], Gray gave an example of such a sequence of circulant matrices $\left\{C_n\left(f\right)\right\}$, where each circulant matrix $C_n\left(f\right)$ depends on the entire sequence $\left\{T_n\left(f\right)\right\}$. However, in practical situations, what we usually know is ${\left\{T_n\left(f\right)\right\}}_{n\le n_0}$ for certain natural number $n_0$. For those practical situations, Pearl presented in [2] a different sequence of circulant matrices $\{{\widehat{C}}_n\left(f\right)\}$, which is also asymptotically equivalent to the sequence of Toeplitz matrices $\left\{T_n\left(f\right)\right\}$, but this satisfies the condition that each circulant matrix ${\widehat{C}}_n\left(f\right)$ only depends on the corresponding Toeplitz matrix $T_n\left(f\right)$.

In the present paper, the sequence $\{{\widehat{C}}_n\left(f\right)\}$ is generalized to block circulant matrices. This sequence of block circulant matrices is used to obtain a result on the rate-distortion function (RDF) of WSS vector processes that allows us to reduce the complexity of coding those processes. In addition, we use that sequence to reduce the complexity of filtering WSS vector processes.

The paper is organized as follows. In Section 2, we set up notation and review the mathematical definitions and results used in the rest of the paper. In Section 3, we define the sequence of block circulant matrices considered in this paper and show its main properties. Finally, in Section 4 and Section 5, we use this sequence of block circulant matrices to reduce the complexity of coding and filtering WSS vector processes, respectively.

2. Preliminaries

2.1. Notation

In this paper, $\mathbb{N}$, $\mathbb{Z}$, $\mathbb{R}$, and $\mathbb{C}$ denote the set of natural numbers (i.e., the set of positive integers), the set of integer numbers, the set of (finite) real numbers, and the set of (finite) complex numbers, respectively. If $m,n\in \mathbb{N}$, then ${\mathbb{C}}^{m\times n}$, $0_{m\times n}$, and $I_n$ are the set of all $m\times n$ complex matrices, the $m\times n$ zero matrix, and the $n\times n$ identity matrix, respectively. The symbol ∗ denotes conjugate transpose, E stands for expectation, $\mathrm{i}$ is the imaginary unit, $\mathrm{tr}$ denotes trace, δ stands for the Kronecker delta, ⊗ is the Kronecker product, ${\lambda }_k\left(A\right)$, $k\in \{1,\dots ,n\}$, are the eigenvalues of an $n\times n$ Hermitian matrix A arranged in decreasing order, and ${\sigma }_k\left(B\right)$, $k\in \{1,\dots ,n\}$, are the singular values of an $n\times n$ matrix B arranged in decreasing order.

Let $\{{\mathbf{x}}_n:n\in \mathbb{N}\}$ be a random N-dimensional vector process, i.e., ${\mathbf{x}}_n$ is a random (column) vector of dimension N for all $n\in \mathbb{N}$. We denote by ${\mathbf{x}}_{n:1}$ the random vector of dimension $nN$ given by

$${\mathbf{x}}_{n:1}:=\left(\begin{array}{c}{\mathbf{x}}_n\\ {\mathbf{x}}_{n-1}\\ {\mathbf{x}}_{n-2}\\ ⋮\\ {\mathbf{x}}_1\end{array}\right),n\in \mathbb{N}.$$

2.2. Block Toeplitz Matrices

We first review the concept of block Toeplitz matrix.

Definition 1.

An $m\times n$ block Toeplitz matrix with $M\times N$ blocks is an $mM\times nN$ matrix of the form

$$\left(\begin{array}{ccccc}{\mathsf{F}}_0& {\mathsf{F}}_{-1}& {\mathsf{F}}_{-2}& \cdots & {\mathsf{F}}_{1-n}\\ {\mathsf{F}}_1& {\mathsf{F}}_0& {\mathsf{F}}_{-1}& \cdots & {\mathsf{F}}_{2-n}\\ {\mathsf{F}}_2& {\mathsf{F}}_1& {\mathsf{F}}_0& \cdots & {\mathsf{F}}_{3-n}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ {\mathsf{F}}_{m-1}& {\mathsf{F}}_{m-2}& {\mathsf{F}}_{m-3}& \cdots & {\mathsf{F}}_0\end{array}\right),$$

where ${\mathsf{F}}_k\in {\mathbb{C}}^{M\times N}$ with $k\in \{1-n,\dots ,m-1\}$.

Consider a matrix-valued function of a real variable $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$, which is continuous and $2\pi$-periodic. For every $n\in \mathbb{N}$, we denote by $T_n\left(F\right)$ the $n\times n$ block Toeplitz matrix with $M\times N$ blocks given by

$$T_n\left(F\right):={\left({\mathsf{F}}_{j-k}\right)}_{j,k=1}^n,$$

where ${\left\{{\mathsf{F}}_k\right\}}_{k\in \mathbb{Z}}$ is the sequence of Fourier coefficients of F:

$${\mathsf{F}}_k=\frac{1}{2\pi }{\int }_0^{2\pi }{\mathrm{e}}^{-k\omega \mathrm{i}}F\left(\omega \right)d\omega \forall k\in \mathbb{Z}.$$

It should be mentioned that $T_n\left(F\right)$ is Hermitian for all $n\in \mathbb{N}$ if and only if $F\left(\omega \right)$ is Hermitian for all $\omega \in \mathbb{R}$ (see [3] (Theorem 4.4.1)). Furthermore, in this case, from [3] (Theorem 4.4.2) (it was previously given in [4] (p. 5674) but without a proof) and [5] (Corollary VI.1.6), we obtain

$$\underset{\omega \in [0,2\pi ]}{min}{\lambda }_N\left(F\left(\omega \right)\right)=inf\left(F\right)\le {\lambda }_{nN}\left(T_n\left(F\right)\right)\le {\lambda }_1\left(T_n\left(F\right)\right)\le sup\left(F\right)=\underset{\omega \in [0,2\pi ]}{max}{\lambda }_1\left(F\left(\omega \right)\right)\forall n\in \mathbb{N}.$$

(1)

2.3. Block Circulant Matrices

We first review the concept of a block circulant matrix.

Definition 2.

An $n\times n$ block circulant matrix with $M\times N$ blocks is an $n\times n$ block Toeplitz matrix with $M\times N$ blocks of the form

$$\left(\begin{array}{ccccc}{\mathsf{C}}_0& {\mathsf{C}}_{-1}& {\mathsf{C}}_{-2}& \cdots & {\mathsf{C}}_{1-n}\\ {\mathsf{C}}_{1-n}& {\mathsf{C}}_0& {\mathsf{C}}_{-1}& \cdots & {\mathsf{C}}_{2-n}\\ {\mathsf{C}}_{2-n}& {\mathsf{C}}_{1-n}& {\mathsf{C}}_0& \cdots & {\mathsf{C}}_{3-n}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ {\mathsf{C}}_{-1}& {\mathsf{C}}_{-2}& {\mathsf{C}}_{-3}& \cdots & {\mathsf{C}}_0\end{array}\right),$$

where ${\mathsf{C}}_k\in {\mathbb{C}}^{M\times N}$ with $k\in \{1-n,\dots ,0\}$.

The next result [6] (Lemma 3) characterizes block circulant matrices.

Lemma 1.

Let $C\in {\mathbb{C}}^{nM\times nN}$; then, the following statements are equivalent:

1.

C is an $n\times n$ block circulant matrix with $M\times N$ blocks.

2.

There exist $A_1,\dots ,A_n\in {\mathbb{C}}^{M\times N}$ such that

$$C=(V_n\otimes I_M)\mathrm{diag}(A_1,\dots ,A_n){(V_n\otimes I_N)}^{*},$$

where $\mathrm{diag}(A_1,\dots ,A_n)={\mathrm{diag}}_{1\le k\le n}\left(A_k\right):={\left({\delta }_{j,k}{A}_j\right)}_{j,k=1}^n$ and $V_n$ is the $n\times n$ Fourier unitary matrix

$${\left[V_n\right]}_{j,k}:=\frac{1}{\sqrt{n}}{\mathrm{e}}^{-\frac{2\pi (j-1)(k-1)}{n}\mathrm{i}},j,k\in \{1,\dots ,n\}.$$

2.4. Asymptotically Equivalent Sequences of Matrices

We now review the concept of asymptotically equivalent sequences of matrices introduced in [6] (Definition 2), which is an extension of the original concept given by Gray in [7] to sequences of non-square matrices.

Definition 3.

Consider two strictly increasing sequences of natural numbers $\{d_n^{\left(1\right)}\}$ and $\{d_n^{\left(2\right)}\}$. Let $A_n$ and $B_n$ be $d_n^{\left(1\right)}\times d_n^{\left(2\right)}$ matrices for all $n\in \mathbb{N}$. We say that the sequences $\left\{A_n\right\}$ and $\left\{B_n\right\}$ are asymptotically equivalent, and write $\left\{A_n\right\}\sim \left\{B_n\right\}$, if $\{\parallel A_n{\parallel }_2\}$ and $\{\parallel B_n{\parallel }_2\}$ are bounded, and

$$\underset{n\to \infty }{lim}\frac{\parallel A_n-B_n{\parallel }_F}{\sqrt{n}}=0,$$

where ${\parallel ·\parallel }_2$ and ${\parallel ·\parallel }_F$ are the spectral norm and the Frobenius norm, respectively (The definition and the main properties of these two matrix norms can be found, e.g., in [3] (Section 2.1)).

We finish this section by reviewing [6] (Lemma 4).

Lemma 2.

Let $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ be continuous and $2\pi$-periodic. Then,

$$\left\{T_n\left(F\right)\right\}\sim \left\{C_n\left(F\right)\right\},$$

where $C_n\left(F\right)$ is the $n\times n$ block circulant matrix with $M\times N$ blocks given by

$$C_n\left(F\right):=(V_n\otimes I_M)\mathrm{diag}\left(F\left(0\right),F\left(\frac{2\pi }{n}\right),\dots ,F\left(\frac{2\pi (n-1)}{n}\right)\right){(V_n\otimes I_N)}^{*}\forall n\in \mathbb{N}.$$

When $M=N=1$ and F is in the Wiener class (We recall that a function $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ is said to be in the Wiener class if it is continuous and $2\pi$-periodic, and it satisfies ${\sum }_{k=-\infty }^{\infty }{\parallel {\mathsf{F}}_k\parallel }_F<\infty$, see, e.g., [3] (Appendix B)), $\left\{C_n\left(F\right)\right\}$ is the sequence of circulant matrices defined by Gray in [1] (Equation (4.32)), see [3] (Lemma 5.2).

3. Sequence of Block Circulant Matrices Considered

We begin this section by presenting a sequence of block circulant matrices $\{{\widehat{C}}_n\left(F\right)\}$, where each block circulant matrix ${\widehat{C}}_n\left(F\right)$ only depends on the corresponding block Toeplitz matrix $T_n\left(F\right)$.

Definition 4.

Let $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ be continuous and $2\pi$-periodic. For every $n\in \mathbb{N}$, we define $\{{\widehat{C}}_n\left(F\right)\}$ as the $n\times n$ block circulant matrix with $M\times N$ blocks given by

$${\widehat{C}}_n\left(F\right):=(V_n\otimes I_M){\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right){(V_n\otimes I_N)}^{*}.$$

When $M=N=1$ and the matrices $T_n\left(F\right)$ are real and symmetric, $\{{\widehat{C}}_n\left(F\right)\}$ is the sequence of circulant matrices defined by Pearl in [2].

The following result gives an expression for the blocks of the block circulant matrix ${\widehat{C}}_n\left(F\right)$.

Lemma 3.

Consider $n\in \mathbb{N}$ and let $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ be continuous and $2\pi$-periodic.

1.

If $j,k\in \{1,\dots ,n\}$, then

$${[{\widehat{C}}_n\left(F\right)]}_{j,k}=\left(1-\frac{|j-k|}{n}\right){\mathsf{F}}_{j-k}+\frac{|j-k|}{n}{\mathsf{F}}_{j-k-n\mathrm{sgn}(j-k)},$$

where $\mathrm{sgn}$ is the sign function.

2.

If $T_n\left(F\right)$ is real, then ${\widehat{C}}_n\left(F\right)$ is real.

Proof. 

(1) Fix $j,k\in \{1,\dots ,n\}$. For convenience, we denote ${\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)$ by $D_n\left(F\right)$. We have

$$\begin{array}{ccc}\hfill {[{\widehat{C}}_n\left(F\right)]}_{j,k}& =& \sum _{h=1}^n{[V_n\otimes I_M]}_{j,h}{\left[D_n\left(F\right){(V_n\otimes I_N)}^{*}\right]}_{h,k}=\sum _{h=1}^n{\left[V_n\right]}_{j,h}{I}_M\sum _{l=1}^n{\left[D_n\left(F\right)\right]}_{h,l}{\left[{(V_n\otimes I_N)}^{*}\right]}_{l,k}\hfill \\ \hfill & =& \sum _{h=1}^n{\left[V_n\right]}_{j,h}{\left[D_n\left(F\right)\right]}_{h,h}{[V_n^{*}\otimes I_N^{*}]}_{h,k}=\sum _{h=1}^n{\left[V_n\right]}_{j,h}{\left[D_n\left(F\right)\right]}_{h,h}{\left[V_n^{*}\right]}_{h,k}{I}_N\hfill \\ \hfill & =& \sum _{h=1}^n{\left[V_n\right]}_{j,h}\overline{{\left[V_n\right]}_{k,h}}{\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{h,h}\hfill \\ \hfill & =& \sum _{h=1}^n{\left[V_n\right]}_{j,h}\overline{{\left[V_n\right]}_{k,h}}\sum _{l=1}^n{\left[{(V_n\otimes I_M)}^{*}\right]}_{h,l}{\left[T_n\left(F\right)(V_n\otimes I_N)\right]}_{l,h}\hfill \\ \hfill & =& \sum _{h=1}^n{\left[V_n\right]}_{j,h}\overline{{\left[V_n\right]}_{k,h}}\sum _{l=1}^n{\left[V_n^{*}\right]}_{h,l}{I}_M\sum _{s=1}^n{\left[T_n\left(F\right)\right]}_{l,s}{[V_n\otimes I_N]}_{s,h}\hfill \\ \hfill & =& \sum _{h=1}^n{\left[V_n\right]}_{j,h}\overline{{\left[V_n\right]}_{k,h}}\sum _{l=1}^n\overline{{\left[V_n\right]}_{l,h}}\sum _{s=1}^n{\mathsf{F}}_{l-s}{\left[V_n\right]}_{s,h}{I}_N=\sum _{h,l,s=1}^n{\left[V_n\right]}_{j,h}\overline{{\left[V_n\right]}_{k,h}}\overline{{\left[V_n\right]}_{l,h}}{\left[V_n\right]}_{s,h}{\mathsf{F}}_{l-s}\hfill \\ \hfill & =& \frac{1}{n^2}\sum _{h,l,s=1}^n{\mathrm{e}}^{-\frac{2\pi (j-1)(h-1)}{n}\mathrm{i}}{\mathrm{e}}^{\frac{2\pi (k-1)(h-1)}{n}\mathrm{i}}{\mathrm{e}}^{\frac{2\pi (l-1)(h-1)}{n}\mathrm{i}}{\mathrm{e}}^{-\frac{2\pi (s-1)(h-1)}{n}\mathrm{i}}{\mathsf{F}}_{l-s}\hfill \\ \hfill & =& \frac{1}{n^2}\sum _{h,l,s=1}^n{\mathrm{e}}^{\frac{2\pi (h-1)(k-j+l-s)}{n}\mathrm{i}}{\mathsf{F}}_{l-s}=\frac{1}{n^2}\sum _{l,s=1}^n\sum _{h=1}^n{\left({\mathrm{e}}^{\frac{2\pi (k-j+l-s)}{n}\mathrm{i}}\right)}^{h-1}{\mathsf{F}}_{l-s}\hfill \\ \hfill & =& \frac{1}{n^2}\left(\sum _{\begin{array}{c}l,s=1\\ k-j+l-s\in n\mathbb{Z}\end{array}}^{n}n{\mathsf{F}}_{l-s}+\sum _{\begin{array}{c}l,s=1\\ k-j+l-s\notin n\mathbb{Z}\end{array}}^n\frac{1-{\left({\mathrm{e}}^{\frac{2\pi (k-j+l-s)}{n}\mathrm{i}}\right)}^n}{1-{\mathrm{e}}^{\frac{2\pi (k-j+l-s)}{n}\mathrm{i}}}{\mathsf{F}}_{l-s}\right)=\frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ k-j+l-s\in n\mathbb{Z}\end{array}}^n{\mathsf{F}}_{l-s},\hfill \end{array}$$

where $\overline{z}$ denotes the conjugate of $z\in \mathbb{C}$.

If $j=k$, then

$${[{\widehat{C}}_n\left(F\right)]}_{j,k}=\frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ l-s=0\end{array}}^n{\mathsf{F}}_{l-s}=\frac{1}{n}\sum _{l=1}^n{\mathsf{F}}_0={\mathsf{F}}_0.$$

If $j>k$, then

$$\begin{array}{ccc}\hfill {[{\widehat{C}}_n\left(F\right)]}_{j,k}& =& \frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ k-j+l-s\in \{-n,0\}\end{array}}^n{\mathsf{F}}_{l-s}=\frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ l-s=j-k\end{array}}^n{\mathsf{F}}_{l-s}+\frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ l-s=j-k-n\end{array}}^n{\mathsf{F}}_{l-s}\hfill \\ & =& \frac{1}{n}(n-|j-k\left|\right)F_{j-k}+\frac{1}{n}(n-|j-k-n\left|\right)F_{j-k-n}=\left(1-\frac{|j-k|}{n}\right)F_{j-k}+\frac{j-k}{n}{F}_{j-k-n}.\hfill \end{array}$$

If $j<k$, then

$$\begin{array}{ccc}\hfill {[{\widehat{C}}_n\left(F\right)]}_{j,k}& =& \frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ k-j+l-s\in \{0,n\}\end{array}}^n{\mathsf{F}}_{l-s}=\frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ l-s=j-k\end{array}}^n{\mathsf{F}}_{l-s}+\frac{1}{n}\sum _{\begin{array}{c}l,s=1\\ l-s=j-k+n\end{array}}^n{\mathsf{F}}_{l-s}\hfill \\ & =& \frac{1}{n}(n-|j-k\left|\right)F_{j-k}+\frac{1}{n}(n-|j-k+n\left|\right)F_{j-k+n}=\left(1-\frac{|j-k|}{n}\right)F_{j-k}+\frac{k-j}{n}{F}_{j-k+n}.\hfill \end{array}$$

(2) It is a direct consequence of Assertion (1). ☐

When $M=N=1$ and the matrices $T_n\left(F\right)$ are real and symmetric, from Lemma 3, we obtain [2] (Equation (7)).

We now show that the block Toeplitz matrix $T_n\left(F\right)$ can be approximated by the block circulant matrix ${\widehat{C}}_n\left(F\right)$ for large n.

Lemma 4.

If $F:\mathbb{R}\to {\mathbb{C}}^{M\times N}$ is continuous and $2\pi$-periodic, then

1.

$\parallel {\widehat{C}}_n{\left(F\right)\parallel }_2\le {\parallel T_n\left(F\right)\parallel }_2\le {\sigma }_1\left(F\right)$ for all $n\in \mathbb{N}$, where ${\sigma }_1\left(F\right):={sup}_{\omega \in [0,2\pi ]}{\sigma }_1\left(F\left(\omega \right)\right)<\infty$.

2.

$\left\{T_n\left(F\right)\right\}\sim \{{\widehat{C}}_n\left(F\right)\}$.

Proof. 

(1) From [3] (Theorem 4.3) $\parallel T_n{\left(F\right)\parallel }_2\le {\sigma }_1\left(F\right)$ for all $n\in \mathbb{N}$. Consequently, to prove Assertion (1), we only need to show that $\parallel {\widehat{C}}_n{\left(F\right)\parallel }_2\le {\parallel T_n\left(F\right)\parallel }_2$ for all $n\in \mathbb{N}$. Fix $n\in \mathbb{N}$. As the spectral norm is unitarily invariant (see, e.g., [3] (Section 2.1)) and $V_n\otimes I_M$ and ${(V_n\otimes I_N)}^{*}$ are unitary matrices, we have

$$\begin{array}{cc}\hfill \parallel {\widehat{C}}_n{\left(F\right)\parallel }_2& ={∥{\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)∥}_2\hfill \\ & ={\max }_{1\le k\le n}\{{∥{\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}∥}_2\}.\hfill \end{array}$$

Consider $k_0\in \{1,\dots ,n\}$ and ${\mathrm{x}}_0\in {\mathbb{C}}^{N\times 1}$ satisfying

$$\parallel {\widehat{C}}_n{\left(F\right)\parallel }_2={∥{\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k_0,k_0}∥}_2=\frac{{∥{\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k_0,k_0}{\mathrm{x}}_0∥}_2}{\parallel {\mathrm{x}}_0{\parallel }_2}.$$

Let ${\widehat{\mathrm{x}}}_0\in {\mathbb{C}}^{nN\times 1}$ with ${\left[{\widehat{\mathrm{x}}}_0\right]}_{j,1}={\delta }_{j,k_0}{\mathrm{x}}_0$ for all $j\in \{1,\dots ,n\}$, then

$$\begin{array}{ccc}\hfill 0\le \parallel {\widehat{C}}_n{\left(F\right)\parallel }_2& \le & \frac{{∥{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N){\widehat{\mathrm{x}}}_0∥}_2}{{∥{\mathrm{x}}_0∥}_2}\hfill \\ & =& \frac{{∥{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N){\widehat{\mathrm{x}}}_0∥}_2}{{∥{\widehat{\mathrm{x}}}_0∥}_2}\le {∥{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)∥}_2={∥T_n\left(F\right)∥}_2.\hfill \end{array}$$

(2) From Lemma 1, ${(V_n\otimes I_M)}^{*}{C}_n\left(F\right)(V_n\otimes I_N)$ is an $n\times n$ block diagonal matrix with $M\times N$ blocks:

$${(V_n\otimes I_M)}^{*}{C}_n\left(F\right)(V_n\otimes I_N)={\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{C}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right).$$

Therefore,

$$C_n\left(F\right)=(V_n\otimes I_M){\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{C}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right){(V_n\otimes I_N)}^{*}.$$

Hence, since the Frobenius norm is unitarily invariant (see, e.g., [3] (Section 2.1)), one obtains

$$\begin{array}{c}\parallel C_n\left(F\right)-{\widehat{C}}_n{\left(F\right)\parallel }_F\hfill \\ =∥(V_n\otimes I_M)\left({\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{C}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)\right.\hfill \\ {\left.-{\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)\right){(V_n\otimes I_N)}^{*}∥}_F\hfill \\ ={∥\left({\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{C}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)-{\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)\right)∥}_F\hfill \\ ={∥{\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}(C_n\left(F\right)-T_n\left(F\right))(V_n\otimes I_N)\right]}_{k,k}\right)∥}_F\hfill \\ \le {∥{(V_n\otimes I_M)}^{*}(C_n\left(F\right)-T_n\left(F\right))(V_n\otimes I_N)∥}_F={∥C_n\left(F\right)-T_n\left(F\right)∥}_F={∥T_n\left(F\right)-C_n\left(F\right)∥}_F.\hfill \end{array}$$

Thus, from Lemma 2, we conclude that

$$0\le \frac{\parallel T_n\left(F\right)-{\widehat{C}}_n{\left(F\right)\parallel }_F}{\sqrt{n}}\le \frac{{∥T_n\left(F\right)-C_n\left(F\right)∥}_F+{\parallel C_n\left(F\right)-{\widehat{C}}_n\left(F\right)\parallel }_F}{\sqrt{n}}\le 2\frac{{∥T_n\left(F\right)-C_n\left(F\right)∥}_F}{\sqrt{n}}\to 0.$$

 ☐

We now show that ${\widehat{C}}_n\left(F\right)$ keeps several properties of $T_n\left(F\right)$ for the case in which $M=N$.

Lemma 5.

Consider $n\in \mathbb{N}$ and let $F:\mathbb{R}\to {\mathbb{C}}^{N\times N}$ be continuous and $2\pi$-periodic.

1.

If $T_n\left(F\right)$ is Hermitian, then ${\widehat{C}}_n\left(F\right)$ is Hermitian.

2.

If $T_n\left(F\right)$ is real and symmetric, then ${\widehat{C}}_n\left(F\right)$ is real and symmetric.

3.

If $T_n\left(F\right)$ is positive semidefinite, then ${\widehat{C}}_n\left(F\right)$ is positive semidefinite.

4.

If $T_n\left(F\right)$ is positive definite, then ${\widehat{C}}_n\left(F\right)$ is positive definite.

Proof. 

(1) As $T_n\left(F\right)$ is Hermitian, ${(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)$ is Hermitian. Therefore, ${\left[{(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}$ is Hermitian for all $k\in \{1,\dots ,n\}$, and, consequently, ${\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)$ is also Hermitian. Hence, ${\widehat{C}}_n\left(F\right)=(V_n\otimes I_N)$ ${\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right){(V_n\otimes I_N)}^{*}$ is Hermitian.

(2) If $T_n\left(F\right)$ is real and symmetric, then $T_n\left(F\right)$ is Hermitian. Applying Lemma 3 and Assertion (1), ${\widehat{C}}_n\left(F\right)$ is real and Hermitian, and hence, ${\widehat{C}}_n\left(F\right)$ is real and symmetric.

(3) For every $k\in \{1,\dots ,n\}$, let ${\mathrm{u}}_k\in {\mathbb{C}}^{nN\times N}$ with ${\left[{\mathrm{u}}_k\right]}_{j,1}={\delta }_{j,k}{I}_N$ for all $j\in \{1,\dots ,n\}$. If $\mathrm{x}\in {\mathbb{C}}^{nN\times 1},$ then

$$\begin{array}{ccc}\hfill {\mathrm{x}}^{*}{\widehat{C}}_n\left(F\right)\mathrm{x}& =& {\mathrm{y}}^{*}{\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N)\right]}_{k,k}\right)\mathrm{y}\hfill \\ & =& {\mathrm{y}}^{*}{\mathrm{diag}}_{1\le k\le n}\left({\mathrm{u}}_k^{*}{(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N){\mathrm{u}}_k\right)\mathrm{y}\hfill \\ & =& \sum _{k=1}^n{\left[\mathrm{y}\right]}_{k,1}^{*}{\mathrm{u}}_k^{*}{(V_n\otimes I_N)}^{*}{T}_n\left(F\right)(V_n\otimes I_N){\mathrm{u}}_k{\left[\mathrm{y}\right]}_{k,1}=\sum _{k=1}^n{\mathrm{z}}_k^{*}{T}_n\left(F\right){\mathrm{z}}_k,\hfill \end{array}$$

(2)

where $\mathrm{y}={(V_n\otimes I_N)}^{*}\mathrm{x}$ and ${\mathrm{z}}_k=(V_n\otimes I_N){\mathrm{u}}_k{\left[\mathrm{y}\right]}_{k,1}$ for all $k\in \{1,\dots ,n\}$. Since $T_n\left(F\right)$ is positive semidefinite, ${\mathrm{z}}_k^{*}{T}_n\left(F\right){\mathrm{z}}_k\ge 0$ for all $k\in \{1,\dots ,n\}$, and, thus, ${\mathrm{x}}^{*}{\widehat{C}}_n\left(F\right)\mathrm{x}\ge 0$.

(4) Consider $\mathrm{x}\in {\mathbb{C}}^{nN\times 1}$. Since $T_n\left(F\right)$ is positive definite, it is positive semidefinite, and from Assertion (3), we have ${\mathrm{x}}^{*}{\widehat{C}}_n\left(F\right)\mathrm{x}\ge 0$. Suppose that ${\mathrm{x}}^{*}{\widehat{C}}_n\left(F\right)\mathrm{x}=0$. As $T_n\left(F\right)$ is positive definite, applying Assertion (2) yields ${\mathrm{z}}_k=0_{nN\times 1}$ for all $k\in \{1,\dots ,n\}$. Consequently, ${\mathrm{u}}_k{\left[\mathrm{y}\right]}_{k,1}=0_{nN\times 1}$ for all $k\in \{1,\dots ,n\}$, and, therefore, ${\left[\mathrm{y}\right]}_{k,1}=0_{N\times 1}$ for all $k\in \{1,\dots ,n\}$. Hence, $\mathrm{y}=0_{nN\times 1}$, and thus, $\mathrm{x}=0_{nN\times 1}$. ☐

Finally, we show that Assertion (1) is also true if $T_n\left(F\right)$ is replaced by ${\widehat{C}}_n\left(F\right)$.

Lemma 6.

Let $F:\mathbb{R}\to {\mathbb{C}}^{N\times N}$ be continuous and $2\pi$-periodic. If $F\left(\omega \right)$ is Hermitian for all $\omega \in \mathbb{R},$ then

$$inf\left(F\right)\le {\lambda }_{nN}\left({\widehat{C}}_n\left(F\right)\right)\le {\lambda }_1\left({\widehat{C}}_n\left(F\right)\right)\le sup\left(F\right)\forall n\in \mathbb{N}.$$

Proof. 

Applying Lemma 5 ${\widehat{C}}_n\left(F\right)$ is Hermitian for all $n\in \mathbb{N}$. Fix $n\in \mathbb{N}$ and $\mathrm{x}\in {\mathbb{C}}^{nN\times 1}$. From Assertion (2), we have

$${\mathrm{x}}^{*}{\widehat{C}}_n\left(F\right)\mathrm{x}=\sum _{k=1}^n{\mathrm{z}}_k^{*}{T}_n\left(F\right){\mathrm{z}}_k.$$

Suppose that $\mathrm{x}$ is an eigenvector of ${\widehat{C}}_n\left(F\right)$ with ${\parallel \mathrm{x}\parallel }_2=1$. Consequently,

$${\lambda }_j({\widehat{C}}_n\left(F\right))={\lambda }_j({\widehat{C}}_n\left(F\right)){\mathrm{x}}^{*}\mathrm{x}={\mathrm{x}}^{*}{\lambda }_j({\widehat{C}}_n\left(F\right))\mathrm{x}={\mathrm{x}}^{*}{\widehat{C}}_n\left(F\right)\mathrm{x}=\sum _{k=1}^n{\mathrm{z}}_k^{*}{T}_n\left(F\right){\mathrm{z}}_k$$

for some $j\in \{1,\dots ,nN\}$. Let $F\left(\omega \right)=U\left(\omega \right)\mathrm{diag}({\lambda }_1\left(F\left(\omega \right)\right),\dots ,{\lambda }_N\left(F\left(\omega \right)\right)){\left(U\left(\omega \right)\right)}^{-1}$ be a unitary diagonalization (i.e., an eigenvalue decomposition where the eigenvector matrix $U\left(\omega \right)$ is unitary) of $F\left(\omega \right)$ for all $\omega \in [0,2\pi ]$. Then,

$$\begin{array}{cc}& {\lambda }_j({\widehat{C}}_n\left(F\right))\hfill \\ & =\sum _{k=1}^n{\left[{\mathrm{z}}_k^{*}{T}_n\left(F\right){\mathrm{z}}_k\right]}_{1,1}=\sum _{k=1}^n\sum _{h=1}^n{\left[{\mathrm{z}}_k^{*}\right]}_{1,h}{\left[T_n\left(F\right){\mathrm{z}}_k\right]}_{h,1}=\sum _{k=1}^n\sum _{h=1}^n{\left[{\mathrm{z}}_k^{*}\right]}_{1,h}\sum _{l=1}^n{\left[T_n\left(F\right)\right]}_{h,l}{\left[{\mathrm{z}}_k\right]}_{l,1}\hfill \\ & =\sum _{k=1}^n\sum _{h=1}^n{\left[{\mathrm{z}}_k\right]}_{h,1}^{*}\sum _{l=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }{\mathrm{e}}^{-(h-l)\omega \mathrm{i}}F\left(\omega \right)d\omega {\left[{\mathrm{z}}_k\right]}_{l,1}\hfill \\ & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{h=1}^n{\left[{\mathrm{z}}_k\right]}_{h,1}^{*}\sum _{l=1}^n{\mathrm{e}}^{(-h+l)\omega \mathrm{i}}F\left(\omega \right){\left[{\mathrm{z}}_k\right]}_{l,1}d\omega \hfill \\ & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }{\left(\sum _{h=1}^n{\mathrm{e}}^{h\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{h,1}\right)}^{*}F\left(\omega \right)\left(\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right)d\omega \hfill \\ & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }{\left({\left(U\left(\omega \right)\right)}^{*}\sum _{h=1}^n{\mathrm{e}}^{h\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{h,1}\right)}^{*}\mathrm{diag}({\lambda }_1\left(F\left(\omega \right)\right),\dots ,{\lambda }_N\left(F\left(\omega \right)\right))\left({\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right)d\omega \hfill \\ & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{s=1}^N{\left[{\left({\left(U\left(\omega \right)\right)}^{*}\sum _{h=1}^n{\mathrm{e}}^{h\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{h,1}\right)}^{*}\right]}_{1,s}{\lambda }_s\left(F\left(\omega \right)\right){\left[{\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right]}_{s,1}d\omega \hfill \\ & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{s=1}^N{\lambda }_s\left(F\left(\omega \right)\right){\left|{\left[{\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right]}_{s,1}\right|}^{2}d\omega .\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}& inf\left(F\right)\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{s=1}^N{\left|{\left[{\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right]}_{s,1}\right|}^{2}d\omega \le {\lambda }_j({\widehat{C}}_n\left(F\right))\hfill \\ & \le sup\left(F\right)\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{s=1}^N{\left|{\left[{\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right]}_{s,1}\right|}^{2}d\omega .\hfill \end{array}$$

Since

$$\frac{1}{2\pi }{\int }_0^{2\pi }{\mathrm{e}}^{m\omega \mathrm{i}}d\omega =\left\{\begin{array}{cc}1\hfill & \mathrm{if}m=0,\hfill \\ 0\hfill & \mathrm{if}m\in \mathbb{Z}\backslash \left\{0\right\},\hfill \end{array}\right.$$

we obtain

$$\begin{array}{ccc}& & \sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{s=1}^N{\left|{\left[{\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}\right]}_{s,1}\right|}^{2}d\omega \hfill \\ & & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }{\left({\left(U\left(\omega \right)\right)}^{*}\sum _{h=1}^n{\mathrm{e}}^{h\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{h,1}\right)}^{*}{\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}d\omega \hfill \\ & & =\sum _{k=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }\sum _{h=1}^n{\mathrm{e}}^{-h\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{h,1}^{*}U\left(\omega \right){\left(U\left(\omega \right)\right)}^{*}\sum _{l=1}^n{\mathrm{e}}^{l\omega \mathrm{i}}{\left[{\mathrm{z}}_k\right]}_{l,1}d\omega \hfill \\ & & =\sum _{k=1}^n\sum _{h=1}^n{\left[{\mathrm{z}}_k^{*}\right]}_{1,h}\sum _{l=1}^n\frac{1}{2\pi }{\int }_0^{2\pi }{\mathrm{e}}^{(-h+l)\omega \mathrm{i}}d\omega {\left[{\mathrm{z}}_k\right]}_{l,1}=\sum _{k=1}^n\sum _{h=1}^n{\left[{\mathrm{z}}_k^{*}\right]}_{1,h}{\left[{\mathrm{z}}_k\right]}_{h,1}=\sum _{k=1}^n{\mathrm{z}}_k^{*}{\mathrm{z}}_k\hfill \\ & & =\sum _{k=1}^n{\left[\mathrm{y}\right]}_{k,1}^{*}{\mathrm{u}}_k^{*}{(V_n\otimes I_N)}^{*}(V_n\otimes I_N){\mathrm{u}}_k{\left[\mathrm{y}\right]}_{k,1}=\sum _{k=1}^n{\left[{\mathrm{y}}^{*}\right]}_{1,k}{\mathrm{u}}_k^{*}{\mathrm{u}}_k{\left[\mathrm{y}\right]}_{k,1}=\sum _{k=1}^n{\left[{\mathrm{y}}^{*}\right]}_{1,k}{\left[\mathrm{y}\right]}_{k,1}\hfill \\ & & ={\mathrm{y}}^{*}\mathrm{y}={\mathrm{x}}^{*}(V_n\otimes I_N){(V_n\otimes I_N)}^{*}\mathrm{x}={\mathrm{x}}^{*}\mathrm{x}=1,\hfill \end{array}$$

which completes the proof. ☐

4. Coding WSS Vector Processes by Using the Sequence of Block Circulant Matrices Considered

Consider that $\{{\mathbf{x}}_n:n\in \mathbb{N}\}$ is a real zero-mean Gaussian N-dimensional vector process. From [8], we know that the RDF of the real zero-mean Gaussian vector ${\mathbf{x}}_{n:1}$ is given by

$$R_n\left(D\right)=\frac{1}{nN}\sum _{k=1}^{nN}max\left\{0,\frac{1}{2}ln\frac{{\lambda }_k\left(E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{\top }\right)\right)}{{\theta }_n}\right\},n\in \mathbb{N},$$

where ${\theta }_n$ is a real number satisfying

$$D=\frac{1}{nN}\sum _{k=1}^{nN}min\left\{{\theta }_n,{\lambda }_k\left(E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{\top }\right)\right)\right\}.$$

We recall that $R_n\left(D\right)$ represents the lowest possible required rate (measured in nats) for jointly encoding (compressing) n symbols from N sources with mean square error (MSE) distortion D.

We assume that $\{{\mathbf{x}}_n:n\in \mathbb{N}\}$ is WSS with continuous PSD X and $inf\left(X\right)>0$. Fix $D\in (0,inf(X\left)\right]$, and from Assertion (1), we obtain that ${\theta }_n=D$, and, consequently,

$$\begin{array}{cc}\hfill R_n\left(D\right)& =\frac{1}{nN}\sum _{k=1}^{nN}max\left\{0,\frac{1}{2}ln\frac{{\lambda }_k\left(E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{\top }\right)\right)}{D}\right\}\hfill \\ & =\frac{1}{nN}\sum _{k=1}^{nN}\frac{1}{2}ln\frac{{\lambda }_k\left(E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{\top }\right)\right)}{D}=\frac{1}{2nN}\sum _{k=1}^{nN}ln\frac{{\lambda }_k\left(T_n\left(X\right)\right)}{D}\forall n\in \mathbb{N}.\hfill \end{array}$$

For every $n\in \mathbb{N}$, let us compute the discrete Fourier transform (DFT) of the N sources as

$${\mathbf{y}}_{n:1}=\left(V_n^{*}\otimes I_N\right){\mathbf{x}}_{n:1}={\left(V_n\otimes I_N\right)}^{*}{\mathbf{x}}_{n:1}.$$

(3)

The correlation matrix of ${\mathbf{y}}_{n:1}$ is given by

$$\begin{array}{cc}\hfill E\left({\mathbf{y}}_{n:1}{\mathbf{y}}_{n:1}^{*}\right)& =E\left({\left(V_n\otimes I_N\right)}^{*}{\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{*}\left(V_n\otimes I_N\right)\right)={\left(V_n\otimes I_N\right)}^{*}E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{*}\right)\left(V_n\otimes I_N\right)\hfill \\ & ={\left(V_n\otimes I_N\right)}^{*}E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{\top }\right)\left(V_n\otimes I_N\right)={\left(V_n\otimes I_N\right)}^{*}{T}_n\left(X\right)\left(V_n\otimes I_N\right).\hfill \end{array}$$

It is clear from Assertion (3) that the lowest possible required rate for encoding ${\mathbf{y}}_{n:1}$ with MSE distortion D is the same as for ${\mathbf{x}}_{n:1}$, i.e., $R_n\left(D\right)$.

Let us compress the Gaussian vector ${\mathbf{y}}_{n:1}$ as if the samples ${\mathbf{y}}_{k_1}$ and ${\mathbf{y}}_{k_2}$ were uncorrelated for all $k_1,k_2\in \{1,\dots ,n\}$ with $k_1\ne k_2$, or, equivalently, as if its correlation matrix were ${\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_N)}^{*}{T}_n\left(X\right)(V_n\otimes I_N)\right]}_{k,k}\right)$ instead of ${(V_n\otimes I_N)}^{*}{T}_n\left(X\right)(V_n\otimes I_N)$. In this case, from Lemma 6, we obtain that the corresponding rate is given by

$${\widehat{R}}_n\left(D\right)=\frac{1}{2nN}\sum _{k=1}^{nN}ln\frac{{\lambda }_k({\widehat{C}}_n\left(X\right))}{D}\forall n\in \mathbb{N}.$$

We now study the rate loss ${\widehat{R}}_n\left(D\right)-R_n\left(D\right)$. We have

$$\begin{array}{cc}\hfill 0& \le {\widehat{R}}_n\left(D\right)-R_n\left(D\right)=\frac{1}{2nN}\sum _{k=1}^{nN}ln\frac{{\lambda }_k({\widehat{C}}_n\left(X\right))}{{\lambda }_k\left(T_n\left(X\right)\right)}\hfill \\ & =\frac{1}{2nN}ln\prod _{k=1}^{nN}\frac{{\lambda }_k({\widehat{C}}_n\left(X\right))}{{\lambda }_k\left(T_n\left(X\right)\right)}=\frac{1}{2nN}ln\frac{det({\widehat{C}}_n\left(X\right))}{det\left(T_n\left(X\right)\right)}=\frac{1}{2nN}lndet({\widehat{C}}_n\left(X\right){\left(T_n\left(X\right)\right)}^{-1})\hfill \\ & =\frac{1}{2nN}ln\prod _{k=1}^{nN}{\lambda }_k({\widehat{C}}_n\left(X\right){\left(T_n\left(X\right)\right)}^{-1})\le \frac{1}{2nN}ln\left({\left(\frac{1}{nN}\sum _{k=1}^{nN}{\lambda }_k({\widehat{C}}_n\left(X\right){\left(T_n\left(X\right)\right)}^{-1})\right)}^{nN}\right)\hfill \\ & =\frac{1}{2}ln\left(\frac{1}{nN}\sum _{k=1}^{nN}{\lambda }_k({\widehat{C}}_n\left(X\right){\left(T_n\left(X\right)\right)}^{-1})\right)=\frac{1}{2}ln\left(\frac{1}{nN}\mathrm{tr}({\widehat{C}}_n\left(X\right){\left(T_n\left(X\right)\right)}^{-1})\right)\hfill \\ & \le \frac{1}{2}ln\left(\frac{\sqrt{nN}}{nN}{\parallel {\widehat{C}}_n\left(X\right){\left(T_n\left(X\right)\right)}^{-1}\parallel }_F\right)=\frac{1}{2}ln\left(\frac{1}{\sqrt{nN}}{\parallel ({\widehat{C}}_n\left(X\right)-T_n\left(X\right)){\left(T_n\left(X\right)\right)}^{-1}+I_{nN}\parallel }_F\right)\hfill \\ & \le \frac{1}{2}ln\left(\frac{1}{\sqrt{nN}}\left(\sqrt{nN}+{\parallel ({\widehat{C}}_n\left(X\right)-T_n\left(X\right)){\left(T_n\left(X\right)\right)}^{-1}\parallel }_F\right)\right)\hfill \\ & \le \frac{1}{2}ln\left(\frac{1}{\sqrt{nN}}\left(\sqrt{nN}+\parallel {\widehat{C}}_n\left(X\right)-T_n{\left(X\right)\parallel }_F{\parallel {\left(T_n\left(X\right)\right)}^{-1}\parallel }_2\right)\right)\hfill \\ & =\frac{1}{2}ln\left(1+\frac{\parallel {\widehat{C}}_n\left(X\right)-T_n{\left(X\right)\parallel }_F}{\sqrt{nN}}{\parallel {\left(T_n\left(X\right)\right)}^{-1}\parallel }_2\right).\hfill \end{array}$$

Applying Assertion (1) yields

$$0\le {\widehat{R}}_n\left(D\right)-R_n\left(D\right)\le \frac{1}{2}ln\left(1+\frac{1}{\sqrt{N}inf\left(X\right)}\frac{\parallel {\widehat{C}}_n\left(X\right)-T_n{\left(X\right)\parallel }_F}{\sqrt{n}}\right),$$

and hence, from Lemma 4, we conclude that

$$\underset{n\to \infty }{lim}\left({\widehat{R}}_n\left(D\right)-R_n\left(D\right)\right)=0.$$

(4)

The obtained result (4) on the RDF of WSS vector processes shows that there is no rate loss for large enough n if we consider the correlation matrix of ${\mathbf{y}}_{n:1}$ as if it were a block diagonal matrix. Obviously, encoding all the samples ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_n$ separately involves a notably lower computational complexity than encoding them jointly. The complexity of coding a WSS vector process in this way is $O(nlogn)$ if the fast Fourier transform (FFT) algorithm is used.

5. Filtering WSS Vector Processes by Using the Sequence of Block Circulant Matrices Considered

Consider a zero-mean WSS M-dimensional vector process $\{{\mathbf{x}}_n:n\in \mathbb{N}\}$ with continuous power spectral density (PSD) X. Let $\{{\mathbf{y}}_n:n\in \mathbb{N}\}$ be a zero-mean WSS N-dimensional vector process with continuous PSD Y. Assume that those two processes are jointly WSS with continuous joint PSD Z.

For every $n\in \mathbb{N}$, if ${\widehat{\mathbf{x}}}_{n:1}$ is an estimation of ${\mathbf{x}}_{n:1}$ from ${\mathbf{y}}_{n:1}$ of the form

$${\widehat{\mathbf{x}}}_{n:1}=W{\mathbf{y}}_{n:1}$$

(5)

with $W\in {\mathbb{C}}^{nM\times nN}$, the MSE per sample is given by

$$\begin{array}{ccc}\hfill \frac{1}{n}\mathrm{MSE}\left(W\right)& :=& \frac{1}{n}E\left(\parallel {\mathbf{x}}_{n:1}-{\widehat{\mathbf{x}}}_{n:1}{\parallel }_2^2\right)=\frac{1}{n}E\left({\left({\mathbf{x}}_{n:1}-{\widehat{\mathbf{x}}}_{n:1}\right)}^{*}\left({\mathbf{x}}_{n:1}-{\widehat{\mathbf{x}}}_{n:1}\right)\right)\hfill \\ \hfill & =& \frac{1}{n}\mathrm{tr}\left(E\left(\left({\mathbf{x}}_{n:1}-{\widehat{\mathbf{x}}}_{n:1}\right){\left({\mathbf{x}}_{n:1}-{\widehat{\mathbf{x}}}_{n:1}\right)}^{*}\right)\right)\hfill \\ \hfill & =& \frac{1}{n}\mathrm{tr}\left(E\left({\mathbf{x}}_{n:1}{\mathbf{x}}_{n:1}^{*}\right)-E\left({\mathbf{x}}_{n:1}{\mathbf{y}}_{n:1}^{*}\right)W^{*}-WE\left({\mathbf{y}}_{n:1}{\mathbf{x}}_{n:1}^{*}\right)+WE\left({\mathbf{y}}_{n:1}{\mathbf{y}}_{n:1}^{*}\right)W^{*}\right)\hfill \\ \hfill & =& \frac{1}{n}\mathrm{tr}\left(T_n\left(X\right)-T_n\left(Z\right)W^{*}-W{\left(T_n\left(Z\right)\right)}^{*}+W{T}_n\left(Y\right)W^{*}\right).\hfill \end{array}$$

The minimum MSE (MMSE) is given by $\mathrm{MMSE}=\mathrm{MSE}\left(W_0\right)$, where

$$W_0=E\left({\mathbf{x}}_{n:1}{\mathbf{y}}_{n:1}^{*}\right){\left(E\left({\mathbf{y}}_{n:1}{\mathbf{y}}_{n:1}^{*}\right)\right)}^{-1}=T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}$$

whenever $det\left(T_n\left(Y\right)\right)\ne 0$ (or, equivalently, whenever $T_n\left(Y\right)$ is positive definite). The filter $W_0$ is known as the Wiener filter.

Consider the following filter:

$$\begin{array}{cc}\hfill W_C& :={\widehat{C}}_n\left(Z\right){\left({\widehat{C}}_n\left(Y\right)\right)}^{-1}\hfill \\ & =(V_n\otimes I_M){\mathrm{diag}}_{1\le k\le n}\left({\left[{(V_n\otimes I_M)}^{*}{T}_n\left(Z\right)(V_n\otimes I_N)\right]}_{k,k}{\left[{(V_n\otimes I_N)}^{*}{T}_n\left(Y\right)(V_n\otimes I_N)\right]}_{k,k}^{-1}\right){(V_n\otimes I_N)}^{*}.\hfill \end{array}$$

The filter $W_C$ is well defined because, from Lemma 5, ${\widehat{C}}_n\left(Y\right)$ is positive definite, and, therefore, it is invertible.

We now study the effect on the MSE when the optimal filter $W_0$ is substituted by $W_C$. We have

$$\begin{array}{cc}\hfill 0& \le \frac{\mathrm{MSE}\left(W_C\right)}{n}-\frac{\mathrm{MMSE}}{n}=\left|\frac{\mathrm{MSE}\left(W_C\right)}{n}-\frac{\mathrm{MMSE}}{n}\right|\hfill \\ & =\left|\frac{1}{n}\mathrm{tr}(T_n\left(X\right)-T_n\left(Z\right)W_C^{*}-W_C{\left(T_n\left(Z\right)\right)}^{*}+W_C\left(T_n\left(Y\right)\right)W_C^{*})-\frac{1}{n}\mathrm{tr}(T_n\left(X\right)-T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}{\left(T_n\left(Z\right)\right)}^{*})\right|\hfill \\ & =\left|\frac{1}{n}\mathrm{tr}(-T_n\left(Z\right)W_C^{*}-W_C{\left(T_n\left(Z\right)\right)}^{*}+W_C\left(T_n\left(Y\right)\right)W_C^{*}+T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}{\left(T_n\left(Z\right)\right)}^{*})\right|\hfill \\ & =\frac{1}{n}\left|\mathrm{tr}((T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-W_C)({\left(T_n\left(Z\right)\right)}^{*}-T_n\left(Y\right)W_C^{*}))\right|\hfill \\ & \le \frac{\sqrt{nM}}{n}{\parallel (T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-W_C)({\left(T_n\left(Z\right)\right)}^{*}-T_n\left(Y\right)W_C^{*})\parallel }_F\hfill \\ & \le \sqrt{\frac{M}{n}}\parallel T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-W_C{\parallel }_F{\parallel {\left(T_n\left(Z\right)\right)}^{*}-T_n\left(Y\right)W_C^{*}\parallel }_2\hfill \\ & \le \sqrt{\frac{M}{n}}\parallel T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-{\widehat{C}}_n\left(Z\right){({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_F{\parallel {\left(T_n\left(Z\right)\right)}^{*}-T_n\left(Y\right){({\widehat{C}}_n\left(Y\right))}^{-1}{({\widehat{C}}_n\left(Z\right))}^{*}\parallel }_2.\hfill \end{array}$$

Since

$$\begin{array}{c}\parallel T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-{\widehat{C}}_n\left(Z\right){({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_F\hfill \\ =\parallel T_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-{\widehat{C}}_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}+{\widehat{C}}_n\left(Z\right){\left(T_n\left(Y\right)\right)}^{-1}-{\widehat{C}}_n\left(Z\right){({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_F\hfill \\ \le \parallel (T_n\left(Z\right)-{\widehat{C}}_n\left(Z\right)){\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_F+{\parallel {\widehat{C}}_n\left(Z\right)({(T_n\left(Y\right))}^{-1}-{({\widehat{C}}_n\left(Y\right))}^{-1})\parallel }_F\hfill \\ \le \parallel T_n\left(Z\right)-{\widehat{C}}_n{\left(Z\right)\parallel }_F\parallel {\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_2+\parallel {\widehat{C}}_n{\left(Z\right)\parallel }_2{\parallel {\left(T_n\left(Y\right)\right)}^{-1}-{({\widehat{C}}_n\left(Y\right))}^{-1}\parallel }_F\hfill \\ \le \parallel T_n\left(Z\right)-{\widehat{C}}_n{\left(Z\right)\parallel }_F{\parallel {\left(T_n\left(Y\right)\right)}^{-1}\parallel }_2\hfill \\ +\parallel {\widehat{C}}_n{\left(Z\right)\parallel }_2{\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}{\widehat{C}}_n\left(Y\right){\left(T_n\left(Y\right)\right)}^{-1}-{({\widehat{C}}_n\left(Y\right))}^{-1}{T}_n\left(Y\right){\left(T_n\left(Y\right)\right)}^{-1}\parallel }_F\hfill \\ \le \parallel T_n\left(Z\right)-{\widehat{C}}_n{\left(Z\right)\parallel }_F\parallel {\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_2+\parallel {\widehat{C}}_n{\left(Z\right)\parallel }_2{\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}({\widehat{C}}_n\left(Y\right)-T_n\left(Y\right)){\left(T_n\left(Y\right)\right)}^{-1}\parallel }_F\hfill \\ \le \parallel T_n\left(Z\right)-{\widehat{C}}_n{\left(Z\right)\parallel }_F\parallel {\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_2+\parallel {\widehat{C}}_n{\left(Z\right)\parallel }_2\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_2{\parallel ({\widehat{C}}_n\left(Y\right)-T_n\left(Y\right)){\left(T_n\left(Y\right)\right)}^{-1}\parallel }_F\hfill \\ \le \parallel T_n\left(Z\right)-{\widehat{C}}_n{\left(Z\right)\parallel }_F\parallel {\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_2+\parallel {\widehat{C}}_n{\left(Z\right)\parallel }_2\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_2\parallel {\widehat{C}}_n\left(Y\right)-T_n{\left(Y\right)\parallel }_F{\parallel {\left(T_n\left(Y\right)\right)}^{-1}\parallel }_2\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill \parallel {\left(T_n\left(Z\right)\right)}^{*}-T_n\left(Y\right){({\widehat{C}}_n\left(Y\right))}^{-1}{({\widehat{C}}_n\left(Z\right))}^{*}{\parallel }_2& \le & \parallel {\left(T_n\left(Z\right)\right)}^{*}{\parallel }_2+{\parallel T_n\left(Y\right){({\widehat{C}}_n\left(Y\right))}^{-1}{({\widehat{C}}_n\left(Z\right))}^{*}\parallel }_2\hfill \\ & \le & \parallel T_n{\left(Z\right)\parallel }_2+\parallel T_n\left(Y\right){({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_2{\parallel {({\widehat{C}}_n\left(Z\right))}^{*}\parallel }_2\hfill \\ & \le & \parallel T_n{\left(Z\right)\parallel }_2+\parallel T_n{\left(Y\right)\parallel }_2\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_2{\parallel {\widehat{C}}_n\left(Z\right)\parallel }_2,\hfill \end{array}$$

we obtain

$$\begin{array}{ccc}\hfill 0& \le & \frac{\mathrm{MSE}\left(W_C\right)}{n}-\frac{\mathrm{MMSE}}{n}\hfill \\ & \le & \sqrt{M}\parallel {\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_2(\parallel T_n{\left(Z\right)\parallel }_2+\parallel T_n{\left(Y\right)\parallel }_2\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_2\parallel {\widehat{C}}_n\left(Z\right){\parallel }_2)\hfill \\ & & \left(\frac{\parallel {\widehat{C}}_n\left(Z\right)-T_n{\left(Z\right)\parallel }_F}{\sqrt{n}}+\parallel {\widehat{C}}_n{\left(Z\right)\parallel }_2{\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}\parallel }_2\frac{\parallel {\widehat{C}}_n\left(Y\right)-T_n{\left(Y\right)\parallel }_F}{\sqrt{n}}\right).\hfill \end{array}$$

Consequently, applying Assertion (1) and Lemma 4 yields

$$\begin{array}{ccc}\hfill 0\le \frac{\mathrm{MSE}\left(W_C\right)}{n}-\frac{\mathrm{MMSE}}{n}& \le & \sqrt{M}\parallel {\left(T_n\left(Y\right)\right)}^{-1}{\parallel }_2{\sigma }_1\left(Z\right)(1+sup\left(Y\right)\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}{\parallel }_2)\hfill \\ & & \left(\frac{\parallel {\widehat{C}}_n\left(Z\right)-T_n{\left(Z\right)\parallel }_F}{\sqrt{n}}+{\sigma }_1\left(Z\right){\parallel {({\widehat{C}}_n\left(Y\right))}^{-1}\parallel }_2\frac{\parallel {\widehat{C}}_n\left(Y\right)-T_n{\left(Y\right)\parallel }_F}{\sqrt{n}}\right).\hfill \end{array}$$

If we assume that $inf\left(Y\right)>0$, from Assertion (1), we obtain that $T_n\left(Y\right)$ is positive definite for all $n\in \mathbb{N}$. Moreover, applying Assertion (1) and Lemma 6 yields

$$0\le \frac{\mathrm{MSE}\left(W_C\right)}{n}-\frac{\mathrm{MMSE}}{n}\le \frac{\sqrt{M}{\sigma }_1\left(Z\right)}{inf\left(Y\right)}\left(1+\frac{sup\left(Y\right)}{inf\left(Y\right)}\right)\left(\frac{\parallel {\widehat{C}}_n\left(Z\right)-T_n{\left(Z\right)\parallel }_F}{\sqrt{n}}+\frac{{\sigma }_1\left(Z\right)}{inf\left(Y\right)}\frac{\parallel {\widehat{C}}_n\left(Y\right)-T_n{\left(Y\right)\parallel }_F}{\sqrt{n}}\right),$$

and, therefore, from Lemma 4, we conclude that

$$\underset{n\to \infty }{lim}\left(\frac{\mathrm{MSE}\left(W_C\right)}{n}-\frac{\mathrm{MMSE}}{n}\right)=0.$$

(6)

The obtained result (6) shows that there is no difference in the MSE for large enough n if we substitute the optimal filter $W_0$ by $W_C$. Obviously, the computational complexity of the operation (5) is notably reduced when applying this substitution and the FFT algorithm is used. Specifically, the complexity is reduced from $O\left(n^2\right)$ to $O(nlogn)$.

6. Conclusions

In this paper, we present a sequence of block circulant matrices and we apply it to reduce the complexity of coding and filtering WSS vector processes. Specifically, in both applications, the complexity is reduced from $O\left(n^2\right)$ to $O(nlogn)$, which is the complexity of performing an FFT.