Decrease in Computational Load and Increase in Accuracy for Filtering of Random Signals

Abstract

This paper describes methods for optimal filtering of random signals that involve large matrices. We developed a procedure that allows us to significantly decrease the computational load associated with numerically implementing the associated filter and increase its accuracy. The procedure is based on the reduction of a large covariance matrix to a collection of smaller matrices. This is done in such a way that the filter equation with large matrices is equivalently represented by a set of equations with smaller matrices. The filter we developed is represented by $\mathbf{x}={\sum }_{j=1}^p{\mathbf{M}}_j{\mathbf{y}}_j$ and minimizes the associated error over all matrices ${\mathbf{M}}_1,\dots ,{\mathbf{M}}_p$. As a result, the proposed optimal filter has two degrees of freedom that increase its accuracy. They are associated, first, with the optimal determination of matrices ${\mathbf{M}}_1,\dots ,{\mathbf{M}}_p$ and second, with an increase in the number p of components in the filter. The error analysis and results of numerical simulations are provided.

1. Introduction

Preliminaries

Although there is an extensive literature on studying methods for filtering of random signals, the problem we consider seems to be different from the known ones. The objective of this paper is to formulate and solve the filtering problem under scenarios that relate to a reduction of computation complexity and an increase in the filtering accuracy for large covariance matrices.

Let $\Omega$ be a set of outcomes in probability space $(\Omega ,\Sigma ,\mu )$ for which $\Sigma$ is a $\sigma$–algebra of measurable subsets of $\Omega$ and $\mu :\Sigma \to [0,1]$ is an associated probability measure. Let $\mathbf{x}\in L^2(\Omega ,{\mathbb{R}}^m)$ be a signal of interest and $\mathbf{y}\in L^2(\Omega ,{\mathbb{R}}^n)$ be an observable signal. Here, $L^2(\Omega ,{\mathbb{R}}^m)$ is the space of square-integrable functions defined on $\Omega$ with values in ${\mathbb{R}}^m$. We write $\mathbf{y}={[{\mathbf{y}}_1,\dots ,{\mathbf{y}}_p]}^T$ where ${\mathbf{y}}_i\in L^2(\Omega ,{\mathbb{R}}^{n_i})$, for $i=1,\dots ,p$ where $n_1+\dots +n_p=n$. W.l.o.g., we assume that all random vectors have zero mean. Let us denote

$$\begin{array}{c}\hfill {\parallel \mathbf{x}\parallel }_{\Omega }^2={\int }_{\Omega }\sum _{i=1}^m{\left[{\mathbf{x}}_{\left(i\right)}\left(\omega \right)\right]}^{2}d\mu \left(\omega \right),{\mathbf{E}}_{\mathbf{x}\mathbf{y}}={\int }_{\Omega }\mathbf{x}\left(\omega \right){\left[\mathbf{y}\left(\omega \right)\right]}^{T}d\mu \left(\omega \right),\end{array}$$

(1)

where ${\mathbf{E}}_{\mathbf{xy}}$ is the covariance matrix formed from $\mathbf{x}$ and $\mathbf{y}$. Let the filter under consideration be represented by a bounded linear transformation $\mathcal{M}:L^2(\Omega ,{\mathbb{R}}^n)\to L^2(\Omega ,{\mathbb{R}}^m)$ such that $\left[\mathcal{M}\right(\mathbf{y}\left)\right]\left(\omega \right)=\mathbf{M}\left[\mathbf{y}\right(\omega \left)\right]$, for each $\omega \in \Omega$, where $\mathbf{M}\in {\mathbb{R}}^{m\times n}$. Henceforth, we will simply write $\mathbf{M}$ in each case.

For a matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, we denote its Moore–Penrose inverse, also known as the pseudoinverse, by ${\mathbf{A}}^{†}\in {\mathbb{R}}^{n\times m}$. Matrices $\mathbf{I}$ and $\mathbf{0}$ represent the identity and zero matrices, respectively, with appropriate dimensions in each context. The Frobenius norm of matrix $\mathbf{A}$ is denoted by $\parallel \mathbf{A}\parallel$.

The well-known problem of finding the linear optimal filter is formulated as follows. Given large covariance matrices ${\mathbf{E}}_{\mathbf{xy}}$ and ${\mathbf{E}}_{\mathbf{yy}}$, find $\mathbf{M}$ that solves

$$\underset{\mathbf{M}}{min}{∥\mathbf{x}-\mathbf{M}\mathbf{y}∥}_{\Omega }^2.$$

(2)

The minimal Frobenius norm solution to (2) is given by (see, for example, [1,2,3,4,5])

$$\mathbf{M}={\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}.$$

(3)

2. Statement of the Problem

The problem we consider is as follows. Given large ${\mathbf{E}}_{\mathbf{xy}}$ and ${\mathbf{E}}_{\mathbf{yy}}$, find a computational procedure for solving

$$\underset{{\mathbf{M}}_1,\dots ,{\mathbf{M}}_p}{min}{∥\mathbf{x}-\sum _{j=1}^p{\mathbf{M}}_j{\mathbf{y}}_j∥}_{\Omega }^2.$$

(4)

that significantly decreases the computational complexity for the ${\mathbf{E}}_{\mathbf{yy}}^{†}$ evaluation compared to that needed for the filter represented in (3).

The problem in (4) can also be interpreted as a system-identification problem [6,7,8]. In this case, ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$ are the inputs of the system, $\mathbf{M}=[{\mathbf{M}}_1,\dots ,{\mathbf{M}}_p]$ is the input-output map of the system, and $\mathbf{x}$ is the system output. For example, in environmental monitoring, ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$ may represent random data coming from nodes measuring variations in temperature, light or pressure and $\mathbf{x}$ may represent a random signal obtained after the received data have been merged in order to estimate required data parameters to within a prescribed degree of accuracy. Similar scenarios arise in target localization and tracking [9].

As mentioned above, the covariance matrices are assumed to be known. This assumption is a well-known and significant limitation in problems dealing with the optimal filtering of random signals. See [10,11] in this regard. The covariances can be estimated in various ways, and particular techniques are given in many papers. We cite [12,13,14,15,16,17] as examples. Estimation of covariance matrices is an important problem that is not considered in this paper.

Motivation—Relation to Known Techniques

In (3), dimensions m and n, of $\mathbf{x}$ and $\mathbf{y}$, respectively, can be very large. In a number of important applied problems, such as those in biology, ecology, finance, sociology and medicine (see e.g., [18,19]), $m,n=O\left({10}^3\right)$ and greater. For example, measurements of gene expression can contain tens of hundreds of samples. Each sample can contain tens of thousands of genes. As a result, in such cases, associated covariance matrices in (3) are very large. Computation of a large pseudo-inverse matrix results in a quite slow computing procedure and can cause the computer to run out of memory. In particular, the associated well-known phenomenon of the curse of dimensionality [20] states that many problems become exponentially difficult in high dimensions. Therefore, our main motivation is to develop a method that decreases the computational load associated with the optimal filter in the case of high-dimensional random signals.

We also show that the proposed method allows us to increase the associated accuracy compared to that of the direct method.

Differences between the proposed procedure and known related methods are as follows. The recursive pseudo-inverse computation [21,22,23] avoids the direct recomputation of a large matrix pseudo-inverse using iterative computation of increasing matrix horizontal blocks. The pseudo-inverse of the first block of the sequence is assumed to be known. The computational cost remains high in high-dimensional settings. This method does not improve the associated accuracy as the proposed filter does when the degree p increases.

Randomized algorithms, such as those based on low-rank approximation via random sampling (e.g., Randomized SVD), provide approximate solutions to regression or filtering problems with fewer operations. Some references are given in [24,25,26]. These algorithms involve nondeterministic error, whereas the method in this paper is exact. The accuracy of the randomized algorithms strongly depends on the number of random samples and can be unstable. The algorithm proposed in this paper systematically improves accuracy by increasing the number of blocks p while maintaining error control.

Techniques based on sparse matrix factorization exploit matrix sparsity to reduce computational complexity [27,28,29]. Their effectiveness depends on the matrix being naturally sparse; otherwise, the advantage is lost. The approach in this paper does not require any matrix sparsity.

Further, in the case of high-dimensional signals, some other known dimensionality-reduction techniques can be used (as those, for example, in [2,3,5]) as an intermediate step for the filtering. At the same time, those techniques also involve large pseudo-inverse covariance matrices of the same size as that in (3). Additionally, when the dimensionality-reduction procedure is applied, intrinsic associated errors appear. The errors may involve, in particular, an undesirable distortion of the data. Thus, an application of the dimensionality-reduction techniques to the problem under consideration does not allow us to avoid computation of large pseudo-inverse covariance matrices. Therefore, we wish to develop techniques that reduce computation of large pseudo-inverses ${\mathbf{E}}_{\mathbf{yy}}^{†}$ and ${\mathbf{M}}^{†}$ without involving any additional associated error.

The following example illustrates the motivating problem. Simulations in this example and also in Example 2 of Section 4.1.7 were run on a Dell Latitude 7400 Business Laptop (manufactured in 2020) with CPU (central processing unit) parameters as follows: processor type, Intel Core i7 8th Gen, 8665U; processor speed, 1.90 GHz (TurboBoost up to 4.80 GHz) and memory, 16 GB DDR4 SDRAM.

Example 1. 

Let ${\mathbf{E}}_{\mathbf{yy}}\in {\mathbb{R}}^{q\times q}$ be a matrix whose entries are normally distributed with mean 0 and variance 1. In Table 1 and Table 2, we provide the time (in seconds) needed for Matlab R2024b to compute matrix ${\mathbf{E}}_{\mathbf{yy}}^{†}$ for different values of q and for two different commands to compute the pseudo-inverse, pinv and lsqminnorm.

Table 1. Time needed for Matlab to compute ${\mathbf{E}}_{\mathbf{yy}}^{†}\in {\mathbb{R}}^{q\times q}$ for different dimensions q using the command pinv.

q	500	1000	2000	3000	4000	5000	6000
Time	$0.01$	$0.41$	$3.95$	$14.02$	$33.07$	$63.11$	$115.23$

Table 2. Time needed for Matlab to compute ${\mathbf{E}}_{\mathbf{yy}}^{†}\in {\mathbb{R}}^{q\times q}$ for different dimensions q using the command lsqminnorm.

q	500	1000	2000	3000	4000	5000	6000
Time	$0.01$	$0.20$	$1.71$	$11.83$	$33.72$	$70.01$	$148.03$

We observe that computation of, e.g., a $5000\times 5000$ pseudo-inverse matrix by commands pinv and lsqminnorm requires about 63 or 70 s, respectively, while computation of the five pseudo-inverse matrices of sizes $1000\times 1000$ requires about $2.05=5\times 0.41$ s or $1=5\times 0.2$ s, respectively. Thus, a procedure that would allow us to reduce the computation of ${\mathbf{E}}_{\mathbf{yy}}^{†}\in {\mathbb{R}}^{5000\times 5000}$ to a computation of, say, five pseudo-inverse matrices of sizes $1000\times 1000$ would be faster. At the same time, such a procedure would additionally involve associated matrix multiplications that are also time-consuming. This observation is further developed in Section 4.1.2 and Section 4.1.7.

3. Contribution and Novelty

The main conceptual novelty of the approach developed in this paper is in the technique that allows us to decrease the computational load needed for the numerical realization of the proposed filter and reduce the associated memory usage. The procedure is based on the reduction of a large ${\mathbf{E}}_{\mathbf{yy}}$ matrix to a collection of smaller matrices. This is done such that the filter equation with large matrices is equivalently represented by a set of equations with smaller matrices. As a result, as shown in Section 4.1.7, the proposed p-th-degree filter requires computation of p pseudo-inverse matrices of sizes much smaller than the size of the large matrix ${\mathbf{E}}_{\mathbf{yy}}$. In this regard, see Section 4.1.2 for more details. The associated Matlab code is given in Section 4.1.7.

In Section 2, Section 3 and Section 4, we also show that the accuracy of the proposed filter increases with increases in the degree of the filter. Details are provided in Section 4.1.2. In particular, the algorithm for the optimal determination of ${\mathbf{M}}_1,\dots ,$${\mathbf{M}}_p$ is given in Section 4.1.7. Note that the algorithm is easy to implement numerically. The error analysis associated with the filter under consideration is provided in Section 4.1.3, Section 4.1.5 and Section 4.1.8.

Remark 1. 

Note that for every outcome $\omega \in \Omega$, realizations ${\mathbf{y}}_1\left(\omega \right),$$\dots ,{\mathbf{y}}_p\left(\omega \right)$ and $\mathbf{x}\left(\omega \right)$ of signals ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$ and $\mathbf{x}$, respectively, occur with certain probabilities. Thus, random signals ${\mathbf{y}}_j$, for $j=1,\dots ,p,$ and $\mathbf{x}$ are associated with infinite sets of realizations $V_j=\left\{{\mathbf{y}}_j\left(\omega \right)\right|\omega \in \Omega \}\subseteq {\mathbb{R}}^n$ and $X=\left\{\mathbf{x}\left(\omega \right)\right|\omega \in \Omega \}\subseteq {\mathbb{R}}^m$, respectively. For different values of ω, the values ${\mathbf{y}}_j\left(\omega \right)$, for $j=1,\dots ,p,$ and $\mathbf{x}\left(\omega \right)$ are, in general, different, and in many cases will span the entire spaces ${\mathbb{R}}_j^n$ and ${\mathbb{R}}^m$, respectively. Therefore, for each value of $\mathbf{x}$ and ${\mathbf{y}}_j$, for $j=1,\dots ,p$, the filter under consideration can be interpreted as map ${\mathbb{R}}^{n_1}\times \dots {\mathbb{R}}^{n_p}\to {\mathbb{R}}^m$ where probabilities are assigned to each column of ${\mathbb{R}}^{n_j}$ and ${\mathbb{R}}^m$. Importantly, the filter is invariant with respect to outcome $\omega \in \Omega$, and therefore is the same for all ${\mathbf{y}}_1\left(\omega \right)\dots ,{\mathbf{y}}_p\left(\omega \right)$ and $\mathbf{x}\left(\omega \right)$ with different outcomes $\omega \in \Omega$.

4. Solution of the Problem

We wish to assume in (4) that ${\sum }_{j=1}^p{\mathbf{M}}_j{\mathbf{y}}_j$ is never the zero vector. To this end, we need to extend the known definition of the linear independence of vectors, as follows.

Let ${\mathbf{M}}^{\left(1\right)}\in {\mathbb{R}}^{m\times n_1},\dots ,{\mathbf{M}}^{\left(p\right)}\in {\mathbb{R}}^{m\times n_p}$ be some matrices. For $j=1,\dots ,p$, let $\mathcal{N}\left({\mathbf{M}}^{\left(j\right)}\right)$ be the null space of matrix ${\mathbf{M}}^{\left(j\right)}$.

A linear combination in the generalized sense of vectors ${\mathbf{y}}_1,\dots ,$${\mathbf{y}}_p$ is a vector

$$\widehat{\mathbf{y}}={\mathbf{M}}^{\left(1\right)}{\mathbf{y}}_1\left(\omega \right)+\dots +{\mathbf{M}}^{\left(p\right)}{\mathbf{y}}_p\left(\omega \right).$$

(5)

The linear combination is considered nontrivial if ${\mathbf{y}}_j\left(\omega \right)\notin \mathcal{N}\left({\mathbf{M}}^{\left(j\right)}\right)$ for at least one of $j=1,\dots ,p.$ Note that if ${\mathbf{M}}^{\left(j\right)}\ne \mathbf{0}$, then it is still, of course, possible that ${\mathbf{y}}_j\left(\omega \right)\in \mathcal{N}\left({\mathbf{M}}^{\left(j\right)}\right)$. Thus, condition ${\mathbf{y}}_j\left(\omega \right)\notin \mathcal{N}\left({\mathbf{M}}^{\left(j\right)}\right)$ is more general than condition ${\mathbf{M}}^{\left(j\right)}\ne \mathbf{0}$.

Definition 1. 

Random vectors ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$ are called linearly independent in the generalized sense if there is no nontrivial linear combination in the generalized sense of these vectors equal to the zero vector; in other words, if

$${\mathbf{M}}^{\left(1\right)}{\mathbf{y}}_1\left(\omega \right)+\dots +{\mathbf{M}}^{\left(p\right)}{\mathbf{y}}_p\left(\omega \right)=\mathbf{0},$$

(6)

for almost all $\omega \in \Omega$, implies that ${\mathbf{y}}_j\left(\omega \right)\in \mathcal{N}\left({\mathbf{M}}^{\left(j\right)}\right)$, for each $j=1,\dots ,p$, and almost all $\omega \in \Omega$.

All random vectors considered below are assumed to be linearly independent in the generalized sense.

4.1. Solution of Problem in (4)

In (3), matrix $\mathbf{M}$ is the minimum Frobenius norm solution of the equation

$$\mathbf{M}{\mathbf{E}}_{\mathbf{yy}}={\mathbf{E}}_{\mathbf{xy}}.$$

(7)

Here, we exploit the specific structure of the original system (7) to develop a special block elimination procedure that allows us to reduce the original system of Equation (7) to a collection of independent smaller subsystems. Their solution implies less computational load than that needed for (3).

To this end, let us represent ${\mathbf{E}}_{\mathbf{yy}}$ and ${\mathbf{E}}_{\mathbf{xy}}$ in blocked forms as follows:

$${\mathbf{E}}_{\mathbf{yy}}=\left[\begin{array}{ccc}{\mathbf{E}}_{11}& \dots & {\mathbf{E}}_{1p}\\ ⋮& ⋮& ⋮\\ {\mathbf{E}}_{p1}& \dots & {\mathbf{E}}_{pp}\end{array}\right]\mathrm{and}{\mathbf{E}}_{\mathbf{xy}}=[{\mathbf{E}}_1\dots {\mathbf{E}}_p],$$

(8)

where ${\mathbf{E}}_{ij}={\mathbf{E}}_{{\mathbf{y}}_i{\mathbf{y}}_j}\in {\mathbb{R}}^{n_j\times n_j}$ and ${\mathbf{E}}_j={\mathbf{E}}_{{\mathbf{xy}}_j}\in {\mathbb{R}}^{m\times n_j}$, for $j,i=1,\dots ,p$. Then, (7) implies

$$[{\mathbf{M}}_1\dots {\mathbf{M}}_p]\left[\begin{array}{ccc}{\mathbf{E}}_{11}& \dots & {\mathbf{E}}_{1p}\\ ⋮& ⋮& ⋮\\ {\mathbf{E}}_{p1}& \dots & {\mathbf{E}}_{pp}\end{array}\right]=[{\mathbf{E}}_1\dots {\mathbf{E}}_p].$$

(9)

Thus, the solution of the problem in (7) is equivalent to the solution of Equation (9).

We provide a specific solution of Equation (9) in the following way. First, in Section 4.1.1, a generic basis for the solution is given. In Section 4.1.4, the solution of Equation (9) is considered for $p=3$. This is a preliminary step in the solution of Equation (9) for an arbitrary p provided in Section 4.1.6.

4.1.1. Generic Basis for the Solution of Problem (7): Case $p=2$ in (9). Second-Degree Filter

To provide a generic basis for the solution of Equation (9), let us consider the case $p=2$ in (4), i.e., the second-degree filter. Then, (9) becomes

$$\left[{\mathbf{M}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}\end{array}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2\right].$$

(10)

We denote

$${\mathbf{E}}_{22}^{\left(1\right)}={\mathbf{E}}_{22}-{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}\mathrm{and}{\mathbf{E}}_2^{\left(1\right)}={\mathbf{E}}_2-{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}.$$

(11)

Recall, it has been shown in [4] that for any random vectors g and h,

$${\mathbf{E}}_{\mathbf{gh}}={\mathbf{E}}_{\mathbf{gh}}{\mathbf{E}}_{\mathbf{hh}}^{†}{\mathbf{E}}_{\mathbf{hh}}.$$

(12)

Lemma 1. 

Let ${\mathbf{M}}_1$ and ${\mathbf{M}}_2$ satisfy Equation (10). Then, (10) decomposes into equations

$${\mathbf{M}}_1{\mathbf{E}}_{11}={\mathbf{E}}_1-{\mathbf{M}}_2{\mathbf{E}}_{21}and{\mathbf{M}}_2{\mathbf{E}}_{22}^{\left(1\right)}={\mathbf{E}}_2^{\left(1\right)}$$

(13)

Proof. 

Let us denote ${\overline{\mathbf{M}}}_1={\mathbf{M}}_1+{\mathbf{M}}_2{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}$. Then,

$$\left[{\overline{\mathbf{M}}}_1{\mathbf{M}}_2\right]=\left[{\mathbf{M}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}\mathbf{I}& \mathbf{0}\\ {\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}& \mathbf{I}\end{array}\right]$$

(14)

and

$$\left[{\mathbf{M}}_1{\mathbf{M}}_2\right]=\left[{\overline{\mathbf{M}}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}\mathbf{I}& \mathbf{0}\\ -{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}& \mathbf{I}\end{array}\right],$$

(15)

where by (12),

$${\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{11}={\mathbf{E}}_{21}.$$

(16)

Thus, (10) is represented as

$$\left[{\overline{\mathbf{M}}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}\mathbf{I}& \mathbf{0}\\ -{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}& \mathbf{I}\end{array}\right]\left[\begin{array}{cc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}\end{array}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2\right]$$

(17)

which implies

$$\left[{\overline{\mathbf{M}}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}\mathbf{I}& \mathbf{0}\\ -{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}& \mathbf{I}\end{array}\right]\left[\begin{array}{cc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}\end{array}\right]\left[\begin{array}{cc}\mathbf{I}& -{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}\\ \mathbf{0}& \mathbf{I}\end{array}\right]$$

(18)

$$=\left[{\mathbf{E}}_1{\mathbf{E}}_2\right]\left[\begin{array}{cc}\mathbf{I}& -{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}\\ \mathbf{0}& \mathbf{I}\end{array}\right].$$

(19)

On the basis of (19), we see that (7), for $p=2$, reduces to the equation

$$\left[{\overline{\mathbf{M}}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}{\mathbf{E}}_{11}& \mathbf{0}\\ \mathbf{0}& {\mathbf{E}}_{22}^{\left(1\right)}\end{array}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\right].$$

(20)

The latter implies (13).    □

Remark 2. 

The above proof is based on Equation (15), which implies the equivalence of Equations (10) and (13). At the same time, (13) can equivalently be obtained as follows. Let us multiply (10) from the right by $N_{21}=\left[\begin{array}{cc}\mathbf{I}& -{{\mathbf{E}}_{11}}^{†}{\mathbf{E}}_{12}\\ \mathbf{0}& \mathbf{I}\end{array}\right]$. Then,

$$\left[{\mathbf{M}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}\end{array}\right]{\mathbf{N}}_{21}=\left[{\mathbf{E}}_1{\mathbf{E}}_2\right]{\mathbf{N}}_{21},$$

(21)

i.e.,

$$\left[{\mathbf{M}}_1{\mathbf{M}}_2\right]\left[\begin{array}{cc}{\mathbf{E}}_{11}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}\end{array}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\right].$$

(22)

Then, (13) follows from (22).

Theorem 1. 

For $p=2,$ the minimal Frobenius norm solution of problem (7) is represented by

$${\mathbf{M}}_2={\tilde{\mathbf{M}}}_2={\mathbf{E}}_2^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†},{\mathbf{M}}_1={\tilde{\mathbf{M}}}_1=({\mathbf{E}}_1-{\tilde{\mathbf{M}}}_2{\mathbf{E}}_{21}){\mathbf{E}}_{11}^{†}$$

(23)

Proof. 

Matrices ${\mathbf{M}}_1$ and ${\mathbf{M}}_2$ that solve (10) satisfy Equation (13). By [2,3], their minimal Frobenius norm solutions are given by (23).    □

It follows from (23) that the second-degree filter requires computation of two pseudo-inverse matrices, ${\mathbf{E}}_{11}^{†}\in {\mathbb{R}}^{n_1\times n_1}$ and ${\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}\in {\mathbb{R}}^{n_2\times n_2}$, of sizes that are smaller than the size of ${\mathbf{E}}_{\mathbf{yy}}^{†}\in {\mathbb{R}}^{q\times q}$ (if, of course, $n_1\ne 0$ and $n_2\ne 0$).

4.1.2. Decrease in Computational Load

We wish to compare the computational load needed for a numerical realization of the second-degree filter given, for $p=2$, by (23), and that of the filter represented by (3).

By ([30], p. 254), the product of $m\times n$ and $n\times q$ matrices requires, for large $m,n$ and q, $2mnq$ flops. For large q, the Golub–Reinsch SVD used for computation of the $q\times q$ matrix pseudo-inverse requires $21q^3$ flops (see [30], p. 254).

In (11), for $n_1=n_2=q/2$, the evaluation of ${\mathbf{E}}_{22}^{\left(1\right)}$ implies the evaluation of ${\mathbf{E}}_{11}^{†}$, matrix multiplications in ${\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}$ and matrix subtraction. These operations require $21{(q/2)}^3$ flops, $2{(q/2)}^3+2{(q/2)}^3$ flops and $q^2/4$ flops, respectively. Similarly, computation of ${\mathbf{E}}_2^{\left(1\right)}$ requires $2m{q}^2/4+2q^3/8+mq/2$ flops. In (23), computation of ${\mathbf{M}}_2$ and ${\mathbf{M}}_1$ imply $21q^3/8+2m{q}^2/4$ flops and $2m{q}^2/4+2m{q}^2/4+mq/2$ flops, respectively.

Let us denote by $C\left(\mathbf{A}\right)$ a number of flops required to compute an operation $\mathbf{A}$. In total, the second-degree filter requires $C\left({\mathbf{E}}_{22}^{\left(1\right)}\right)+C\left({\mathbf{E}}_2^{\left(1\right)}\right)+C\left({\mathbf{M}}_2\right)+C\left({\mathbf{M}}_1\right)$ flops where

$$C\left({\mathbf{E}}_{22}^{\left(1\right)}\right)=q^2/4+25q^3/8,$$

(24)

$$C\left({\mathbf{E}}_2^{\left(1\right)}\right)=mq/2+2m{q}^2/4+2q^3/8,$$

(25)

$$C\left({\mathbf{M}}_2\right)=2m{q}^2/4+21q^3/8,$$

(26)

$$C\left({\mathbf{M}}_1\right)=mq/2+m{q}^2.$$

(27)

Thus, the number of flops needed for the numerical realization of the second-degree filter is equal to $mq+q^2/4+2m{q}^2+6q^3$.

The computational load required by the filter given by (3) is $2m{q}^2+21q^3$ flops. It is customary to choose $m\le q$. In this case, clearly, $mq+q^2/4+2m{q}^2+6q^3<2m{q}^2+21q^3$. In particular, if $m=q$, then the latter inequality becomes $5q^2/4+8q^3<23q^3.$ That is, for sufficiently large m and q, the proposed second-degree filter requires, roughly, up to $2.8$ times less flops than that by the filter given by (3). In simulations by Matlab provided in Example 2 below, the second-degree filter takes about half the time in seconds taken by the filter given by (3).

In Section 4.1.4 and Section 4.1.6, the procedure described in Section 4.1.1 is extended to the case of filters of higher degrees. For given n, this will imply, obviously, smaller sizes of blocks of matrices ${\mathbf{E}}_{\mathbf{yy}}$ and ${\mathbf{E}}_{\mathbf{xy}}$ than those for the second-degree filter, and as a result, a further decrease in the associated computational load.

4.1.3. Error Associated with the Second-Degree Filter Determined by (23)

Now, we wish to show that the error associated with the second-degree filter determined for $p=2$ by (23) is less than the error associated with the filter determined by (3).

Let us denote

$$ϵ=\underset{\mathbf{M}}{min}{∥\mathbf{x}-\mathbf{M}\mathbf{y}∥}_{\Omega }^2,{\epsilon }_2=\underset{{\mathbf{M}}_1,{\mathbf{M}}_2}{min}{∥\mathbf{x}-\left[{\mathbf{M}}_1{\mathbf{M}}_2\right]\left[\begin{array}{c}{\mathbf{y}}_1\\ {\mathbf{y}}_2\end{array}\right]∥}_{\Omega }^2.$$

(28)

Theorem 2. 

Let ${\mathbf{y}}_1$ and ${\mathbf{y}}_2$ be such that

$$\sum _{j=1}^2{∥{\mathbf{E}}_j^{(j-1)}{\left[{\left({\mathbf{E}}_{jj}^{(j-1)}\right)}^{1/2}\right]}^{†}∥}^2>{\parallel {\mathbf{E}}_{\mathbf{xy}}{\left({\mathbf{E}}_{\mathbf{yy}}^{1/2}\right)}^{†}\parallel }^2,$$

(29)

where ${\mathbf{E}}_1={\mathbf{E}}_1^{\left(0\right)}$ and ${\mathbf{E}}_{11}={\mathbf{E}}_{11}^{\left(0\right)}$. Then,

$${\epsilon }_2<ϵ.$$

(30)

Proof. 

It is known (see, for example, [1,2,3,4,5]) that, for M determined by (3), and for ${\tilde{\mathbf{M}}}_1$ and ${\tilde{\mathbf{M}}}_2$ determined by (23),

$$ϵ=\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}{\parallel }^2-\mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\},{\epsilon }_2={\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}\parallel }^2-\mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\},$$

(31)

where $\mathbf{y}=\left[\begin{array}{c}{\mathbf{y}}_1\\ {\mathbf{y}}_2\end{array}\right]$ and, bearing in mind (2), $\left[{\tilde{\mathbf{M}}}_1{\tilde{\mathbf{M}}}_2\right]={\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}$. Therefore, by (23),

$$\begin{array}{ccc}& & \mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\}\hfill \\ & =& \mathrm{tr}\left\{\left[{\tilde{\mathbf{M}}}_1{\tilde{\mathbf{M}}}_2\right]{\mathbf{E}}_{\mathbf{yx}}\right\}\hfill \\ & =& \mathrm{tr}\{{\tilde{\mathbf{M}}}_1{\mathbf{E}}_1^T+{\tilde{\mathbf{M}}}_2{\mathbf{E}}_2^T\}\hfill \\ & =& \mathrm{tr}\{({\mathbf{E}}_1{\mathbf{E}}_{11}^{†}-{\tilde{\mathbf{M}}}_2{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}){\mathbf{E}}_1^T+{\tilde{\mathbf{M}}}_2{\mathbf{E}}_2^T\}\hfill \\ & =& \mathrm{tr}\left\{{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T\right\}+\mathrm{tr}\left\{{\tilde{\mathbf{M}}}_2({\mathbf{E}}_2^T-{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\right\}\hfill \\ & =& \mathrm{tr}\left\{{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T\right\}+\mathrm{tr}\left\{({\mathbf{E}}_2-{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}){({\mathbf{E}}_{22}-{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12})}^{†}({\mathbf{E}}_2^T-{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\right\}\hfill \\ & =& \parallel {\mathbf{E}}_1{\mathbf{E}}_{11}^{1/2}{\parallel }^2+{\parallel {\mathbf{E}}_2^{\left(1\right)}{\left[{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{1/2}\right]}^{†}\parallel }^2.\hfill \end{array}$$

(32)

Then,

$${\epsilon }_2=\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}{\parallel }^2-(\parallel {\mathbf{E}}_1{\mathbf{E}}_{11}^{1/2}{\parallel }^2+\parallel {\mathbf{E}}_2^{\left(1\right)}{\left[{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{1/2}\right]}^{†}{\parallel }^2).$$

(33)

At the same time,

$$ϵ=\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}{\parallel }^2-{\parallel {\mathbf{E}}_{\mathbf{xy}}{\left({\mathbf{E}}_{\mathbf{yy}}^{1/2}\right)}^{†}\parallel }^2.$$

(34)

As a result, (30) follows from (29), (33) and (34).    □

Remark 3. 

The condition in (29) can, of course, be practically satisfied. The following Corollary 1 demonstrates the case.

Corollary 1. 

Let ${\mathbf{y}}_1=\mathbf{y}$ and ${\mathbf{y}}_2$ be a nonzero signal. Then, (30) is always true.

Proof. 

The proof follows directly from (29), (33) and (34).    □

4.1.4. Solution of Equation (9) for $p=3$

To clarify the solution device of Equation (9) for an arbitrary p, let us now consider a particular case with $p=3$. The solution is based on the development of the procedure represented by (21) and (22).

For $p=3$, Equation (9) is as follows:

$$\left[{\mathbf{M}}_1{\mathbf{M}}_2{\mathbf{M}}_3\right]\left[\begin{array}{ccc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}& {\mathbf{E}}_{13}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}& {\mathbf{E}}_{23}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}& {\mathbf{E}}_{33}\end{array}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2{\mathbf{E}}_3\right].$$

(35)

First, consider a reduction of (35) to the equation with the block-lower triangular matrix.

Step 1. By (12), for $i=2,3$, ${\mathbf{E}}_{i1}={\mathbf{E}}_{i1}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{11}$. Define

$${\mathbf{N}}_{3,1}=\left[\begin{array}{ccc}\mathbf{I}& -{{\mathbf{E}}_{11}}^{†}{\mathbf{E}}_{12}& -{{\mathbf{E}}_{11}}^{†}{\mathbf{E}}_{13}\\ \mathbf{0}& \mathbf{I}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{I}\end{array}\right].$$

(36)

Now, in (35), for $i=1,2,3$ and $j=2,3$, update block-matrices ${\mathbf{E}}_{ij}$ and ${\mathbf{E}}_j$ to ${\mathbf{E}}_{ij}^{\left(1\right)}$ and ${\mathbf{E}}_j^{\left(1\right)}$, respectively, so that

$$\left[\begin{array}{ccc}{\mathbf{E}}_{11}& \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& {\mathbf{E}}_{23}^{\left(1\right)}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}^{\left(1\right)}& {\mathbf{E}}_{33}^{\left(1\right)}\end{array}\right]=\left[\begin{array}{ccc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}& {\mathbf{E}}_{13}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}& {\mathbf{E}}_{23}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}& {\mathbf{E}}_{33}\end{array}\right]{\mathbf{N}}_{3,1}$$

(37)

and

$$\left[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}{\mathbf{E}}_3^{\left(1\right)}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2{\mathbf{E}}_3\right]{\mathbf{N}}_{3,1}$$

(38)

where ${\mathbf{E}}_{kj}^{\left(1\right)}={\mathbf{E}}_{kj}-{\mathbf{E}}_{k1}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{1j}$, for $k=2,3$ and $j=2,3$, and ${\mathbf{E}}_j^{\left(1\right)}={\mathbf{E}}_j-{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{1j}$, for $j=2,3$. Note that ${\mathbf{E}}_{12}^{\left(1\right)}={\mathbf{E}}_{13}^{\left(1\right)}=\mathbf{0}$ by (12).

Step 2. Set

$${\mathbf{N}}_{3,2}=\left[\begin{array}{ccc}\mathbf{I}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{I}& -{\left({{\mathbf{E}}_{22}}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{23}^{\left(1\right)}\\ \mathbf{0}& \mathbf{0}& \mathbf{I}\end{array}\right].$$

(39)

By (12), ${\mathbf{E}}_{32}^{\left(1\right)}$$={\mathbf{E}}_{32}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{22}^{\left(1\right)}$. Taking the latter into account, update block-matrices ${\mathbf{E}}_{ij}^{\left(1\right)}$ and ${\mathbf{E}}_j^{\left(1\right)}$ in the LHS of (38), for $i=2,3$ and $j=3$, so that

$$\left[\begin{array}{ccc}{\mathbf{E}}_{11}& \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \mathbf{0}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}^{\left(1\right)}& {\mathbf{E}}_{33}^{\left(2\right)}\end{array}\right]=\left[\begin{array}{ccc}{\mathbf{E}}_{11}& \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& {\mathbf{E}}_{23}^{\left(1\right)}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}^{\left(1\right)}& {\mathbf{E}}_{33}^{\left(1\right)}\end{array}\right]{\mathbf{N}}_{3,2}$$

(40)

and

$$\left[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}{\mathbf{E}}_3^{\left(2\right)}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}{\mathbf{E}}_3^{\left(1\right)}\right]{\mathbf{N}}_{3,2}$$

(41)

where ${\mathbf{E}}_{33}^{\left(2\right)}={\mathbf{E}}_{33}^{\left(1\right)}-{\mathbf{E}}_{32}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{23}^{\left(1\right)}$ and ${\mathbf{E}}_3^{\left(2\right)}={\mathbf{E}}_3^{\left(1\right)}-{\mathbf{E}}_2^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{23}^{\left(1\right)}$.

As a result, we finally obtain the updated matrix equation with the block-lower triangular matrix

$$\left[{\mathbf{M}}_1{\mathbf{M}}_2{\mathbf{M}}_3\right]\left[\begin{array}{ccc}{\mathbf{E}}_{11}& \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \mathbf{0}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}^{\left(1\right)}& {\mathbf{E}}_{33}^{\left(2\right)}\end{array}\right]=\left[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}{\mathbf{E}}_3^{\left(2\right)}\right].$$

(42)

Solution of Equation (42). Using a back substitution, Equation (42) is represented by the three equations, as follows:

$$\left\{\begin{array}{ccc}\hfill {\mathbf{M}}_3{\mathbf{E}}_{33}^{\left(2\right)}& =& {\mathbf{E}}_3^{\left(2\right)},\hfill \\ \hfill {\mathbf{M}}_2{\mathbf{E}}_{22}^{\left(1\right)}+{\mathbf{M}}_3{\mathbf{E}}_{32}^{\left(1\right)}& =& {\mathbf{E}}_2^{\left(1\right)},\hfill \\ \hfill {\mathbf{M}}_1{\mathbf{E}}_{11}+{\mathbf{M}}_2{\mathbf{E}}_{21}+{\mathbf{M}}_3{\mathbf{E}}_{31}& =& {\mathbf{E}}_1,\hfill \end{array}\right.$$

(43)

where ${\mathbf{E}}_{11},{\mathbf{E}}_{21},{\mathbf{E}}_{31}$ and ${\mathbf{E}}_1$ are the original blocks in (35).

It follows from (43) that the solution of Equation (9), for $p=3$, is represented by the minimal Frobenius norm solutions to the problems

$$\underset{{\mathbf{M}}_3}{min}{\parallel {\mathbf{E}}_3^{\left(2\right)}-{\mathbf{M}}_3{\mathbf{E}}_{33}^{\left(2\right)}\parallel }^2,$$

(44)

$$\underset{{\mathbf{M}}_2}{min}{\parallel ({\mathbf{E}}_2^{\left(1\right)}-{\tilde{\mathbf{M}}}_3{\mathbf{E}}_{32}^{\left(1\right)})-{\mathbf{M}}_2{\mathbf{E}}_{22}^{\left(1\right)}\parallel }^2,$$

(45)

$$\underset{{\mathbf{M}}_1}{min}{\parallel ({\mathbf{E}}_1-\sum _{j=2}^3{\tilde{\mathbf{M}}}_j{\mathbf{E}}_{j1})-{\mathbf{M}}_1{\mathbf{E}}_{11}\parallel }^2,$$

(46)

where ${\tilde{\mathbf{M}}}_3$ and ${\tilde{\mathbf{M}}}_2$ are solutions to the problems in (44) and (45), respectively. Matrices ${\tilde{\mathbf{M}}}_j$, for $j=1,2,3$, that solve (44)–(46) are as follows:

$${\tilde{\mathbf{M}}}_3={\mathbf{E}}_3^{\left(2\right)}{\left[{\mathbf{E}}_{33}^{\left(2\right)}\right]}^{†},$$

(47)

$${\tilde{\mathbf{M}}}_2=({\mathbf{E}}_2^{\left(1\right)}-{\tilde{\mathbf{M}}}_3{\mathbf{E}}_{32}^{\left(1\right)}){\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†},$$

(48)

$${\tilde{\mathbf{M}}}_1=({\mathbf{E}}_1-\sum _{j=2}^3{\tilde{\mathbf{M}}}_j{\mathbf{E}}_{j1}){\left({\mathbf{E}}_{11}\right)}^{†}.$$

(49)

Thus, the third-degree filter requires computation of three pseudo-inverse matrices, ${\mathbf{E}}_{11}^{†}\in {\mathbb{R}}^{n_1\times n_1}$, ${\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}\in {\mathbb{R}}^{n_2\times n_2}$ and ${\left[{\mathbf{E}}_{33}^{\left(2\right)}\right]}^{†}\in {\mathbb{R}}^{n_3\times n_3}$.

4.1.5. Error Associated with the Third-Degree Filter Determined by (47)–(49)

We wish to show that the error associated with the filter determined by (47)–(49), is less than the error $ϵ$ associated with the filter determined by (3).

To this end, let us denote

$${\epsilon }_3=\underset{{\mathbf{M}}_1,{\mathbf{M}}_2,{\mathbf{M}}_3}{min}{∥\mathbf{x}-\left[{\mathbf{M}}_1{\mathbf{M}}_2{\mathbf{M}}_3\right]\mathbf{y}∥}_{\Omega }^2,$$

(50)

where $\mathbf{y}={[{\mathbf{y}}_1^T,{\mathbf{y}}_2^T,{\mathbf{y}}_3^T]}^T$. As before, ${\mathbf{E}}_1={\mathbf{E}}_1^{\left(0\right)}$ and ${\mathbf{E}}_{11}={\mathbf{E}}_{11}^{\left(0\right)}$.

Theorem 3. 

Let ${\mathbf{y}}_1$, ${\mathbf{y}}_2$ and ${\mathbf{y}}_3$ be such that

$$\sum _{j=1}^3{∥{\mathbf{E}}_j^{(j-1)}{\left[{\left({\mathbf{E}}_{jj}^{(j-1)}\right)}^{1/2}\right]}^{†}∥}^2>{\parallel {\mathbf{E}}_{\mathbf{xy}}{\left({\mathbf{E}}_{\mathbf{yy}}^{1/2}\right)}^{†}\parallel }^2.$$

(51)

Then,

$${\epsilon }_3<ϵ.$$

(52)

Proof. 

$${\epsilon }_3={\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}\parallel }^2-\mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\},$$

(53)

and, bearing in mind an obvious extension of Remark (2), for $p=3$, $\left[{\tilde{\mathbf{M}}}_1{\tilde{\mathbf{M}}}_2{\tilde{\mathbf{M}}}_3\right]={\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}$. Therefore, by (47)–(49),

$$\begin{array}{c}\mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\}\hfill \\ =\mathrm{tr}\left\{\left[{\tilde{\mathbf{M}}}_1{\tilde{\mathbf{M}}}_2{\tilde{\mathbf{M}}}_3\right]{\mathbf{E}}_{\mathbf{yx}}\right\}\hfill \\ =\mathrm{tr}\{{\tilde{\mathbf{M}}}_1{\mathbf{E}}_1^T+{\tilde{\mathbf{M}}}_2{\mathbf{E}}_2^T+{\tilde{\mathbf{M}}}_3{\mathbf{E}}_3^T\}\hfill \\ =\mathrm{tr}\{({\mathbf{E}}_1{\mathbf{E}}_{11}^{†}-[{\tilde{\mathbf{M}}}_2{\mathbf{E}}_{21}+{\tilde{\mathbf{M}}}_3{\mathbf{E}}_{31}]{\mathbf{E}}_{11}^{†}){\mathbf{E}}_1^T+{\tilde{\mathbf{M}}}_2{\mathbf{E}}_2^T+{\tilde{\mathbf{M}}}_3{\mathbf{E}}_3^T\}\hfill \\ =\mathrm{tr}\left\{{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T\right\}+\mathrm{tr}\{{\tilde{\mathbf{M}}}_2({\mathbf{E}}_2^T-{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\hfill \\ +{\tilde{\mathbf{M}}}_3({\mathbf{E}}_3^T-{\mathbf{E}}_{31}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\}\hfill \\ =\mathrm{tr}\left\{{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T\right\}+\mathrm{tr}\{({\mathbf{E}}_2^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}-{\tilde{\mathbf{M}}}_3{\mathbf{E}}_{32}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}){\left({\mathbf{E}}_2^{\left(1\right)}\right)}^T\hfill \\ +{\tilde{\mathbf{M}}}_3({\mathbf{E}}_3^T-{\mathbf{E}}_{31}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\}\hfill \\ =\mathrm{tr}\{{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T+{\mathbf{E}}_2^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\left({\mathbf{E}}_2^{\left(1\right)}\right)}^T\hfill \\ +{\tilde{\mathbf{M}}}_3[{\mathbf{E}}_3^T-{\mathbf{E}}_{31}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T-{\mathbf{E}}_{32}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}+{\left({\mathbf{E}}_2^{\left(1\right)}\right)}^T]\}\hfill \\ =\parallel {\mathbf{E}}_1{\mathbf{E}}_{11}^{1/2}{\parallel }^2+{\parallel {\mathbf{E}}_2{\mathbf{E}}_{22}^{1/2}\parallel }^2+\mathrm{tr}\left\{{\mathbf{E}}_3^{\left(2\right)}{\left({\mathbf{E}}_{33}^{\left(2\right)}\right)}^{†}{\left({\mathbf{E}}_3^{\left(2\right)}\right)}^T\right\}\hfill \\ =\parallel {\mathbf{E}}_1{\mathbf{E}}_{11}^{1/2}{\parallel }^2+\parallel {\mathbf{E}}_2^{\left(1\right)}{\left[{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{1/2}\right]}^{†}{\parallel }^2+{\parallel {\mathbf{E}}_3^{\left(2\right)}{\left[{\left({\mathbf{E}}_{33}^{\left(2\right)}\right)}^{1/2}\right]}^{†}\parallel }^2.\hfill \end{array}$$

(54)

Thus, (51)–(52) follows from $ϵ$ in (28), (53) and (54).    □

Corollary 2. 

If ${\mathbf{y}}_1=\mathbf{y}$ and at least one of ${\mathbf{y}}_2$ or ${\mathbf{y}}_3$ is a nonzero vector, then (52) is always true.

Proof. 

The proof follows from (51), (54) and (34).    □

4.1.6. Solution of Equation (9) for Arbitrary p

Here, we extend the procedure considered in Section 4.1.4 to the case of an arbitrary p. First, we reduce the Equation (9) to an equation with the block-lower triangular matrix. We do this by the following steps, where each step is justified by an obvious extension of (1).

Step 1. First, in (9), for $k=1,\dots ,p$ and $j=2,\dots ,p$, we wish to update blocks ${\mathbf{E}}_{kj}$ and ${\mathbf{E}}_k$ to ${\mathbf{E}}_{kj}^{\left(1\right)}$ and ${\mathbf{E}}_k^{\left(1\right)}$, respectively. To this end, define

$${\mathbf{N}}_{p,1}=\left[\begin{array}{cccc}\mathbf{I}& -{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{12}& \dots & -{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{1p}\\ \mathbf{0}& \mathbf{I}& \dots & \mathbf{0}\\ \dots & \dots & \dots & \dots \\ \mathbf{0}& \dots & \mathbf{I}& \mathbf{0}\\ \mathbf{0}& \dots & \mathbf{0}& \mathbf{I}\end{array}\right].$$

(55)

Bearing in mind that by (12), ${\mathbf{E}}_{i1}={\mathbf{E}}_{i1}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{11}$, for $i=2,\dots ,p$, we obtain

$$\left[\begin{array}{cccc}{\mathbf{E}}_{11}& \mathbf{0}& \dots & \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \dots & {\mathbf{E}}_{2p}^{\left(1\right)}\\ \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p-1,1}& {\mathbf{E}}_{p-1,2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p-1,p}^{\left(1\right)}\\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& \dots & {\mathbf{E}}_{pp}^{\left(1\right)}\end{array}\right]=\left[\begin{array}{cccc}{\mathbf{E}}_{11}& {\mathbf{E}}_{12}& \dots & {\mathbf{E}}_{1p}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}& \dots & {\mathbf{E}}_{2p}\\ \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p-1,1}& {\mathbf{E}}_{p-1,2}& \dots & {\mathbf{E}}_{p-1,p}\\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}& \dots & {\mathbf{E}}_{pp}\end{array}\right]{\mathbf{N}}_{p,1}$$

(56)

and

$$[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\dots {\mathbf{E}}_p^{\left(1\right)}]=[{\mathbf{E}}_1{\mathbf{E}}_2\dots {\mathbf{E}}_p]{\mathbf{N}}_{p,1},$$

(57)

where, for $k=1,\dots ,p$ and $j=2,\dots ,p$,

$${\mathbf{E}}_{kj}^{\left(1\right)}={\mathbf{E}}_{kj}-{\mathbf{E}}_{k1}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{1j}$$

(58)

and

$${\mathbf{E}}_j^{\left(1\right)}={\mathbf{E}}_j-{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_{1j}.$$

(59)

Step 2. Now, we wish to update blocks ${\mathbf{E}}_{kj}^{\left(1\right)}$ and ${\mathbf{E}}_k^{\left(1\right)}$, for $k=2,\dots ,p$ and $j=3,\dots ,p$, of the matrices in the LHS of (56) and (57), respectively.

Set

$${\mathbf{N}}_{p,2}=\left[\begin{array}{ccccc}\mathbf{I}& \mathbf{0}& \mathbf{0}& \dots & \mathbf{0}\\ \mathbf{0}& \mathbf{I}& -{\left({{\mathbf{E}}_{22}}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{23}^{\left(1\right)}& \dots & -{\left({{\mathbf{E}}_{22}}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{2p}^{\left(1\right)}\\ \mathbf{0}& \mathbf{0}& \mathbf{I}& \dots & \mathbf{0}\\ \dots & \dots & \dots & \dots & \dots \\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \dots & \mathbf{I}\end{array}\right],$$

(60)

Because by (12), ${\mathbf{E}}_{i2}^{\left(1\right)}={\mathbf{E}}_{i2}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{22}^{\left(1\right)}$, for $i=3,\dots ,p$, we have

$$\left[\begin{array}{ccccc}{\mathbf{E}}_{11}& \mathbf{0}& \mathbf{0}& \dots & \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \mathbf{0}& \dots & \mathbf{0}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}^{\left(1\right)}& {\mathbf{E}}_{33}^{\left(2\right)}& \dots & {\mathbf{E}}_{3p}^{\left(2\right)}\\ \dots & \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& {\mathbf{E}}_{p3}^{\left(2\right)}& \dots & {\mathbf{E}}_{pp}^{\left(2\right)}\end{array}\right]=\left[\begin{array}{ccccc}{\mathbf{E}}_{11}& \mathbf{0}& \mathbf{0}& \dots & \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& {\mathbf{E}}_{23}^{\left(1\right)}& \dots & {\mathbf{E}}_{2p}^{\left(1\right)}\\ {\mathbf{E}}_{31}& {\mathbf{E}}_{32}^{\left(1\right)}& {\mathbf{E}}_{33}^{\left(1\right)}& \dots & {\mathbf{E}}_{3p}^{\left(1\right)}\\ \dots & \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& {\mathbf{E}}_{p3}^{\left(1\right)}& \dots & {\mathbf{E}}_{pp}^{\left(1\right)}\end{array}\right]{\mathbf{N}}_{p,2}$$

(61)

and

$$[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}{\mathbf{E}}_3^{\left(2\right)}\dots {\mathbf{E}}_p^{\left(2\right)}]=[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}{\mathbf{E}}_3^{\left(1\right)}\dots {\mathbf{E}}_p^{\left(1\right)}]{\mathbf{N}}_{p,2},$$

(62)

where, for $k=2,\dots ,p$ and $j=3,\dots ,p$,

$${\mathbf{E}}_{kj}^{\left(2\right)}={\mathbf{E}}_{kj}^{\left(1\right)}-{\mathbf{E}}_{k2}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{2j}^{\left(1\right)}$$

(63)

and

$${\mathbf{E}}_j^{\left(2\right)}={\mathbf{E}}_j^{\left(1\right)}-{\mathbf{E}}_2^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}{\mathbf{E}}_{2j}^{\left(1\right)}.$$

(64)

We continue this updating procedure up to the final Step $(p-1)$. In Step $(p-2)$, the following matrices are obtained:

$$\left[\begin{array}{ccccc}{\mathbf{E}}_{11}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \dots & \mathbf{0}& \mathbf{0}\\ \dots & \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p-1,1}& {\mathbf{E}}_{p-1,2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p-1,p-1}^{(p-2)}& {\mathbf{E}}_{p-1,p}^{(p-2)}\\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p,p-1}^{(p-2)}& {\mathbf{E}}_{pp}^{(p-2)}\end{array}\right]$$

(65)

and

$$[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\dots {\mathbf{E}}_{p-2}^{(p-3)}{\mathbf{E}}_{p-1}^{(p-2)}{\mathbf{E}}_p^{(p-2)}].$$

(66)

Step $(p-1)$. By (12),

$${\mathbf{E}}_{p-1,p}^{(p-2)}={\mathbf{E}}_{p-1,p}^{(p-2)}{\left({\mathbf{E}}_{p-1,p-1}^{\left(2\right)}\right)}^{†}{\mathbf{E}}_{p-1,p-1}^{(p-2)}.$$

(67)

In (65) and (66), we now update only two blocks, ${\mathbf{E}}_{p-1,p}^{(p-2)}$ and ${\mathbf{E}}_{pp}^{(p-2)}$, and the single block ${\mathbf{E}}_p^{(p-2)}$. To this end, define

$${\mathbf{N}}_{p,p-1}=\left[\begin{array}{ccccc}\mathbf{I}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{I}& \dots & \mathbf{0}& \mathbf{0}\\ \dots & \dots & \dots & \dots & \dots \\ \mathbf{0}& \mathbf{0}& \dots & \mathbf{I}& -{{\mathbf{E}}_{p-1,p-1}^{(p-2)}}^{†}{\mathbf{E}}_{p-1,p}^{(p-2)}\\ \mathbf{0}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{I}\end{array}\right],$$

(68)

and obtain

$$\left[\begin{array}{ccccc}{\mathbf{E}}_{11}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \dots & \mathbf{0}& \mathbf{0}\\ \dots & \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p-1,1}& {\mathbf{E}}_{p-1,2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p-1,p-1}^{(p-2)}& \mathbf{0}\\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p,p-1}^{(p-2)}& {\mathbf{E}}_{pp}^{(p-1)}\end{array}\right]=\left[\begin{array}{ccccc}{\mathbf{E}}_{11}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \dots & \mathbf{0}& \mathbf{0}\\ \dots & \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p-1,1}& {\mathbf{E}}_{p-1,2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p-1,p-1}^{(p-2)}& {\mathbf{E}}_{p-1,p}^{(p-2)}\\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p,p-1}^{(p-2)}& {\mathbf{E}}_{pp}^{(p-2)}\end{array}\right]{\mathbf{N}}_{p,p-1}$$

(69)

and

$$[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\dots {\mathbf{E}}_{p-2}^{(p-3)}{\mathbf{E}}_{p-1}^{(p-2)}{\mathbf{E}}_p^{(p-1)}]=[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\dots {\mathbf{E}}_{p-1}^{(p-2)}{\mathbf{E}}_p^{(p-2)}]{\mathbf{N}}_{p,p-1},$$

(70)

where

$${\mathbf{E}}_{pp}^{(p-1)}={\mathbf{E}}_{pp}^{(p-2)}-{\mathbf{E}}_{p,p-1}^{(p-2)}{\left({\mathbf{E}}_{p-1,p-1}^{(p-2)}\right)}^{†}{\mathbf{E}}_{p-1,p}^{(p-2)}$$

(71)

and

$${\mathbf{E}}_p^{(p-1)}={\mathbf{E}}_p^{(p-2)}-{\mathbf{E}}_{p-1}^{(p-2)}{\left({\mathbf{E}}_{p-1,p-1}^{(p-2)}\right)}^{†}{\mathbf{E}}_{p-1,p}^{(p-2)}.$$

(72)

As a result, on the basis of (69)–(72), Equation (9) is reduced to the equation with the the block-lower triangular matrix as follows

$$\widehat{\mathbf{M}}\left[\begin{array}{ccccc}{\mathbf{E}}_{11}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{0}\\ {\mathbf{E}}_{21}& {\mathbf{E}}_{22}^{\left(1\right)}& \dots & \mathbf{0}& \mathbf{0}\\ \dots & \dots & \dots & \dots & \dots \\ {\mathbf{E}}_{p-1,1}& {\mathbf{E}}_{p-1,2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p-1,p-1}^{(p-2)}& \mathbf{0}\\ {\mathbf{E}}_{p1}& {\mathbf{E}}_{p2}^{\left(1\right)}& \dots & {\mathbf{E}}_{p,p-1}^{(p-2)}& {\mathbf{E}}_{pp}^{(p-1)}\end{array}\right]=\widehat{\mathbf{E}},$$

(73)

where $\widehat{\mathbf{M}}=[{\mathbf{M}}_1{\mathbf{M}}_2\dots {\mathbf{M}}_{p-1}{\mathbf{M}}_p]$ and $\widehat{\mathbf{E}}=[{\mathbf{E}}_1{\mathbf{E}}_2^{\left(1\right)}\dots {\mathbf{E}}_{p-1}^{(p-2)}{\mathbf{E}}_p^{(p-1)}].$

4.1.7. Solution of Equation (9)

Using a back substitution, (73) is represented by p equations as follows:

$$\left\{\begin{array}{ccc}\hfill {\mathbf{M}}_p{\mathbf{E}}_{pp}^{(p-1)}& =& {\mathbf{E}}_p^{(p-1)},\hfill \\ \hfill {\mathbf{M}}_{p-1}{\mathbf{E}}_{p-1,p-1}^{(p-2)}+{\mathbf{M}}_p{\mathbf{E}}_{p,p-1}^{(p-2)}& =& {\mathbf{E}}_{p-1}^{(p-2)},\hfill \\ \hfill ⋮\\ \hfill {\mathbf{M}}_2{\mathbf{E}}_{22}^{\left(1\right)}+\dots +{\mathbf{M}}_{p-1}{\mathbf{E}}_{p-1,2}^{\left(1\right)}+{\mathbf{M}}_p{\mathbf{E}}_{p2}^{\left(1\right)}& =& {\mathbf{E}}_2^{\left(1\right)},\hfill \\ \hfill {\mathbf{M}}_1{\mathbf{E}}_{11}+\dots +{\mathbf{M}}_{p-1}{\mathbf{E}}_{p-1,1}+{\mathbf{M}}_p{\mathbf{E}}_{p1}& =& {\mathbf{E}}_1.\hfill \end{array}\right.$$

(74)

Therefore, the solution of Equation (9) is represented by solutions to the problems

$$\underset{{\mathbf{M}}_p}{min}{\parallel {\mathbf{E}}_p^{(p-1)}-{\mathbf{M}}_p{\mathbf{E}}_{pp}^{(p-1)}\parallel }^2,$$

(75)

$$\underset{{\mathbf{M}}_{p-1}}{min}{\parallel ({\mathbf{E}}_{p-1}^{(p-2)}-{\tilde{\mathbf{M}}}_p{\mathbf{E}}_{p,p-1}^{(p-2)})-{\mathbf{M}}_{p-1}{\mathbf{E}}_{p-1,p-1}^{(p-2)}\parallel }^2,$$

(76)

$$⋮$$

$$\underset{{\mathbf{M}}_2}{min}{\parallel ({\mathbf{E}}_2^{\left(1\right)}-\sum _{j=3}^p{\tilde{\mathbf{M}}}_j{\mathbf{E}}_{j2}^{\left(1\right)})-{\mathbf{M}}_2{\mathbf{E}}_{22}^{\left(1\right)}\parallel }^2,$$

(77)

$$\underset{{\mathbf{M}}_1}{min}{\parallel ({\mathbf{E}}_1-\sum _{j=2}^p{\tilde{\mathbf{M}}}_j{\mathbf{E}}_{j1})-{\mathbf{M}}_1{\mathbf{E}}_{11}\parallel }^2.$$

(78)

In (76)–(78), ${\tilde{\mathbf{M}}}_p,\dots ,{\tilde{\mathbf{M}}}_2$ are solutions to the problems in (75)–(77), respectively. On the basis of [31], the minimal Frobenius norm solutions ${\tilde{\mathbf{M}}}_p,\dots ,{\tilde{\mathbf{M}}}_1$ to (75)–(78) are given by

$${\tilde{\mathbf{M}}}_p={\mathbf{E}}_p^{\left(p-1\right)}{\left({\mathbf{E}}_{pp}^{\left(p-1\right)}\right)}^{†},$$

(79)

$${\tilde{\mathbf{M}}}_{p-1}=({\mathbf{E}}_{p-1}^{(p-2)}-{\tilde{\mathbf{M}}}_p{\mathbf{E}}_{p,p-1}^{(p-2)}){\left({\mathbf{E}}_{p-1,p-1}^{(p-2)}\right)}^{†},$$

(80)

$$⋮$$

$${\tilde{\mathbf{M}}}_2=({\mathbf{E}}_2^{\left(1\right)}-\sum _{j=3}^p{\tilde{\mathbf{M}}}_j{\mathbf{E}}_{j2}^{\left(1\right)}){\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†},$$

(81)

$${\tilde{\mathbf{M}}}_1=({\mathbf{E}}_1-\sum _{j=2}^p{\tilde{\mathbf{M}}}_j{\mathbf{E}}_{j1}){\mathbf{E}}_{11}^{†}.$$

(82)

Thus, the p-th-degree filter requires computation of p pseudo-inverse matrices, ${\mathbf{E}}_{11}^{†}\in {\mathbb{R}}^{n_1\times n_1}$, ${\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}\in {\mathbb{R}}^{n_2\times n_2}$, $\dots ,$${\left({\mathbf{E}}_{pp}^{\left(p-1\right)}\right)}^{†}\in {\mathbb{R}}^{n_p\times n_p}$.

Algorithm 1 is based on formulas (58), (59), (63), (64), (71), (72) and (79)–(82).

Algorithm 1: Solution of Equation (9)

    Note that the algorithm is quite simple. Its output is constructed from the successive computation of matrices ${\mathbf{E}}_{kj}$ and ${\mathbf{E}}_k$.

• In line 2, the algorithm updates matrices ${\mathbf{E}}_{kj}$ and ${\mathbf{E}}_k$ in ways similar to those for the particular cases given by (58), (59), (63), (64), (71), (72).
• In line 5, matrix ${\mathbf{M}}_p$ is calculated as in (79).
• In line 7, matrices ${\mathbf{M}}_{p-1},\dots ,{\mathbf{M}}_1$ are calculated as in (80)–(82).

Importantly, ${\mathbf{E}}_{ss}^{†}$ is the $n_s\times n_s$ matrix with $n_s<n$; therefore, it requires less computation than $n\times n$ pseudo-inverse ${\mathbf{E}}_{\mathbf{yy}}^{†}$.

Example 2. 

Here, we wish to numerically illustrate the above algorithm and  Matlab  code (see Appendix A).

Covariance matrices ${\mathbf{E}}_{\mathbf{yy}}$ and ${\mathbf{E}}_{\mathbf{xy}}$ are estimated by $s^{-1}\mathbf{Y}{\mathbf{Y}}^T$ and $s^{-1}\mathbf{X}{\mathbf{Y}}^T$, respectively, where $\mathbf{X}=[\mathbf{x}\left({\omega }_1\right)\dots \mathbf{x}\left({\omega }_s\right)]\in {\mathbb{R}}^{m\times s}$, $\mathbf{Y}=[\mathbf{y}\left({\omega }_1\right)\dots \mathbf{y}\left({\omega }_s\right)]\in {\mathbb{R}}^{n\times s}$, $\mathbf{x}\left({\omega }_j\right)\in {\mathbb{R}}^m$ and $\mathbf{y}\left({\omega }_j\right)\in {\mathbb{R}}^n$, for all $j=1,\dots ,s$, and s is the number of samples. Entries of $\mathbf{X}$ and $\mathbf{Y}$ are normally distributed with mean 0 and variance 1. We choose $m=1000$, $n=1000,\dots ,5000$, $p=1,2,4,5$ and $n_j=n/p$ for $j=1,\dots ,p$.

In Table 3 and Table 4 and Figure 1 and Figure 2, for $p=1,2,4,5$, the computation times (in seconds) needed by the above algorithm and the code (see Appendix A) for different values of dimension n are presented. The results represented in Table 3 and Figure 1 were obtained with the command  pinv  used for calculation of the pseudoinverses. The results in Table 4 and Figure 2 were obtained with the command  lsqminnorm.

Table 3. Numerical results for Example 2 obtained using the command pinv.

Dimension n
	1000	2000	3000	4000	5000	6000
Time in seconds
$p=1$	$0.77$	$3.98$	$14.01$	$33.54$	$63.51$	$115.79$
$p=2$	$0.93$	$2.02$	$7.58$	$17.00$	$34.82$	$57.97$
$p=4$	$0.71$	$1.83$	$4.05$	$8.86$	$18.74$	$37.03$
$p=5$	$0.51$	$1.57$	$3.70$	$8.44$	$15.13$	$28.02$

Table 4. Numerical results for Example 2 obtained using the command lsqminnorm.

Dimension n
	1000	2000	3000	4000	5000	6000
Time in seconds
$p=1$	$0.25$	$1.88$	$12.28$	$33.58$	$72.72$	$150.06$
$p=2$	$0.20$	$1.21$	$2.45$	$4.28$	$10.12$	$27.83$
$p=4$	$0.17$	$1.012$	$2.16$	$4.14$	$6.32$	$14.25$
$p=5$	$0.16$	$0.67$	$1.66$	$3.81$	$5.80$	$11.83$

Figure 1. Graphical representation of the results given in Table 3.

Figure 2. Graphical representation of the results given in Table 4.

In Figure 3, the values of the error of the proposed method are represented for $p=1,\dots ,5$ and $n=1000,\dots ,6000$. It follows from Figure 3 that the accuracy of the proposed method increases when p increases. Note that for $p=1$, the proposed filter coincides with the known filter represented by (3).

Figure 3. Graphical representation of the errors.

Recall that the simulations were performed on a Dell Latitude 7400 Business Laptop (manufactured in 2020). The time decreases with the increase in p. In other words, the computational load needed for the numerical realization of the proposed p-th-degree filter decreases with the increase in degree p. Note that the proposed p-th-degree filter requires computation of p pseudo-inverse matrices and also matrix multiplications determined by the above algorithm. As a result, the proposed method is faster than that of the known filter in (3) and also requires less memory usage. In particular, for $n=5000$ and $p=5,$ the proposed filter is around four times faster than the filter defined by (3).

4.1.8. The Error Associated with the Filter Represented by (79)–(82)

Let us denote by

$${\epsilon }_p=\underset{{\mathbf{M}}_1,\dots ,{\mathbf{M}}_p}{min}{∥\mathbf{x}-[{\mathbf{M}}_1,\dots ,{\mathbf{M}}_p]\mathbf{y}∥}_{\Omega }^2$$

(83)

the error associated with the filter determined by (79)–(82). We wish to analyze the error.

Theorem 4. 

The error associated with the filter determined by (79)–(82) is given by

$${\epsilon }_p={\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}\parallel }^2-\sum _{j=1}^p{∥{\mathbf{E}}_j^{(j-1)}{\left[{\left({\mathbf{E}}_{jj}^{(j-1)}\right)}^{1/2}\right]}^{†}∥}^2.$$

(84)

Let ${\mathbf{y}}_p$ be a nonzero vector. Then, the error decreases if the degree of the proposed filter p increases, i.e.,

$${\epsilon }_p<{\epsilon }_{p-1},\mathrm{for}p=2,3,\dots .$$

(85)

In particular, if ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$ are such that

$$\sum _{j=1}^p{∥{\mathbf{E}}_j^{(j-1)}{\left[{\left({\mathbf{E}}_{jj}^{(j-1)}\right)}^{1/2}\right]}^{†}∥}^2>{\parallel {\mathbf{E}}_{\mathbf{xy}}{\left({\mathbf{E}}_{\mathbf{yy}}^{1/2}\right)}^{†}\parallel }^2$$

(86)

then,

$${\epsilon }_p<ϵ,$$

(87)

where ϵ is the error associated with the filter given by (3).

Proof. 

$${\epsilon }_p={\parallel {\mathbf{E}}_{\mathbf{xx}}^{1/2}\parallel }^2-\mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\},$$

(88)

where, much as in the proofs of (2) and (3), $[{\tilde{\mathbf{M}}}_1\dots {\tilde{\mathbf{M}}}_p]={\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}$. Note that, as before, ${\mathbf{E}}_1={\mathbf{E}}_1^{\left(0\right)}$ and ${\mathbf{E}}_{11}={\mathbf{E}}_{11}^{\left(0\right)}$. Therefore, by (79)–(82),

$$\begin{array}{c}\mathrm{tr}\left\{{\mathbf{E}}_{\mathbf{xy}}{\mathbf{E}}_{\mathbf{yy}}^{†}{\mathbf{E}}_{\mathbf{yx}}\right\}\hfill \\ =\mathrm{tr}\left\{[{\tilde{\mathbf{M}}}_1\dots {\tilde{\mathbf{M}}}_p]{\mathbf{E}}_{\mathbf{yx}}\right\}\hfill \\ =\mathrm{tr}\{{\tilde{\mathbf{M}}}_1{\mathbf{E}}_1^T+\dots +{\tilde{\mathbf{M}}}_p{\mathbf{E}}_p^T\}\hfill \\ =\mathrm{tr}\{({\mathbf{E}}_1{\mathbf{E}}_{11}^{†}-[{\tilde{\mathbf{M}}}_2{\mathbf{E}}_{21}+\dots +{\tilde{\mathbf{M}}}_p{\mathbf{E}}_{p1}]{\mathbf{E}}_{11}^{†}){\mathbf{E}}_1^T\hfill \\ +{\tilde{\mathbf{M}}}_2{\mathbf{E}}_2^T+\dots +{\tilde{\mathbf{M}}}_p{\mathbf{E}}_p^T\}\hfill \\ =\mathrm{tr}\left\{{\mathbf{E}}_1{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T\right\}+\mathrm{tr}\{{\tilde{\mathbf{M}}}_2({\mathbf{E}}_2^T-{\mathbf{E}}_{21}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)+\dots \hfill \\ +{\tilde{\mathbf{M}}}_p({\mathbf{E}}_p^T-{\mathbf{E}}_{p1}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\}+\mathrm{tr}\{[{\mathbf{E}}_2^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}-{\tilde{\mathbf{M}}}_3{\mathbf{E}}_{32}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}-\hfill \\ \dots -{\tilde{\mathbf{M}}}_p{\mathbf{E}}_{p2}^{\left(1\right)}{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{†}]{\left({\mathbf{E}}_2^{\left(1\right)}\right)}^T+\dots +{\tilde{\mathbf{M}}}_p({\mathbf{E}}_p^T-{\mathbf{E}}_{p1}{\mathbf{E}}_{11}^{†}{\mathbf{E}}_1^T)\}\hfill \\ =\dots \hfill \\ =\parallel {\mathbf{E}}_1{\mathbf{E}}_{11}^{1/2}{\parallel }^2+\parallel {\mathbf{E}}_2{\mathbf{E}}_{22}^{1/2}{\parallel }^2+\dots +{\parallel {\mathbf{E}}_{p-1}{\mathbf{E}}_{p-1,p-1}^{1/2}\parallel }^2\hfill \\ +\mathrm{tr}\left\{{\mathbf{E}}_p^{\left(2\right)}{\left({\mathbf{E}}_{pp}^{\left(2\right)}\right)}^{†}{\left({\mathbf{E}}_p^{\left(2\right)}\right)}^T\right\}\hfill \\ =\parallel {\mathbf{E}}_1{\mathbf{E}}_{11}^{1/2}{\parallel }^2+\parallel {\mathbf{E}}_2^{\left(1\right)}{\left[{\left({\mathbf{E}}_{22}^{\left(1\right)}\right)}^{1/2}\right]}^{†}{\parallel }^2+\dots +{\parallel {\mathbf{E}}_p^{\left(2\right)}{\left[{\left({\mathbf{E}}_{pp}^{\left(2\right)}\right)}^{1/2}\right]}^{†}\parallel }^2.\hfill \end{array}$$

(89)

As a result, (88) and (89) imply (84). In turn, (84) implies (85). Inequalities (29) and (87) follow from (34), (88) and (89).    □

Remark 4. 

Of course, in practice, the condition in (29) can easily be satisfied by a variation of the dimensions of vectors ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$ and/or by their choice. The following Figure 3 illustrates this observation.

Corollary 3. 

If ${\mathbf{y}}_1=\mathbf{y}$ and at least one of ${\mathbf{y}}_2,\dots ,{\mathbf{y}}_p$ is not the zero vector, then (87) is always true.

Proof. 

The proof follows directly from (34) and (84).    □

5. Conclusions

In a number of important applications, such as those in biology and medicine, associated random signals are high-dimensional. Therefore, covariance matrices formed from those signals are high-dimensional as well. Large matrices also arise because of the filter structure. As a result, the numerical realization of a filter that processes high-dimensional signals might be very time-consuming, and in some cases, might cause the computer to run out of memory. In this paper, we have presented a method of optimal filter determination which targets the case of high-dimensional random signals. In particular, we have developed a procedure that significantly decreases the computational load needed to numerically realize the filter (see Section 4.1.7 and Example 2). The procedure is based on reduction of the filter equation with large matrices to an equivalent representation by a set of equations with smaller matrices. The associated algorithm and Matlab code have been provided. The algorithm is easy to implement numerically. We have also shown that filter accuracy is improved with an increase in filter degree, i.e. in the number of filter parameters (see (4)).

The results demonstrate that in settings with high-dimensional random signals, a number of existing applications may benefit from the use of the proposed technique. The Matlab code is given in Appendix A. In [32], the technique provided here is extended to the case of filtering that is optimal with respect to minimization of both matrices ${\mathbf{M}}_1,\dots ,{\mathbf{M}}_p$ and random vectors ${\mathbf{y}}_1,\dots ,{\mathbf{y}}_p$.