A Covariance-Free Strictly Complex-Valued Relevance Vector Machine for Reducing the Order of Linear Time-Invariant Systems

Abstract

Multiple-input multiple-output (MIMO) linear time-invariant (LTI) systems exhibit enormous computational costs for high-dimensional problems. To address this problem, we propose a novel approach for reducing the dimensionality of MIMO systems. The method leverages the Takenaka–Malmquist basis and incorporates the strictly complex-valued relevant vector machine (SCRVM). We refer to this method as covariance-free maximum likelihood (CoFML). The proposed method avoids the explicit computation of the covariance matrix. CoFML solves multiple linear systems to obtain the required posterior statistics for covariance. This is achieved by exploiting the preconditioning matrix and the matrix diagonal element estimation rule. We provide theoretical justification for this approximation and show why our method scales well in high-dimensional settings. By employing the CoFML algorithm, we approximate MIMO systems in parallel, resulting in significant computational time savings. The effectiveness of this method is demonstrated through three well-known examples.

1. Introduction

A reduction in dimensionality is crucial in multiple-input multiple-output (MIMO) system models due to the high computational costs associated with high-dimensional problems. A MIMO linear time-invariant (LTI) system with l outputs and k inputs can be represented by a matrix of transfer functions in the following form [1,2,3,4,5,6]:

$$\begin{array}{c}\hfill F\left(s\right)={\displaystyle \frac{B_{m-1}{s}^{m-1}+\cdots +B_0}{d_m{s}^m+d_1{s}^{m-1}+\cdots +1}}.\end{array}$$

(1)

Here, $B_i\in {\mathbb{R}}^{l\times k}$, m is the system order, and $d_i\in \mathbb{R}$, $i=0,1,2,\cdots ,m$.

The n-th approximate model is given by the following [1,2,3,4,5,6]:

$$\begin{array}{c}\hfill {\widehat{F}}_n\left(s\right)={\displaystyle \frac{{\widehat{B}}_1{s}^{n-1}+\cdots +{\widehat{B}}_n}{{\widehat{d}}_0{s}^n+{\widehat{d}}_1{s}^{n-1}+\cdots +1}},\end{array}$$

(2)

where ${\widehat{B}}_i\in {\mathbb{R}}^{l\times k}$, n is the reduced system order, and ${\widehat{d}}_i\in \mathbb{R}$, $i=0,1,2,\cdots ,n$.

Numerous techniques have been developed for model reduction in linear invariant systems. These techniques encompass a wide range of approaches, including linear matrix inequalities [7], error minimization [8], magnitude and phase criteria [9], balanced truncation [10,11], rational interpolation [12], the Krylov method [13], adaptive Fourier decomposition (AFD) [14], Routh approximations, and Padé-type model reductions [15,16]. However, those methods incur substantial computational costs and necessitate prior knowledge of the actual system, which is typically unknown in practice. Consequently, a persistent demand exists for rapid and efficient methods to reduce the order of LTI systems. To address this research gap, this study explores novel MIMO system model reduction approaches that are computationally efficient and do not rely on explicit knowledge of the underlying system. By leveraging the latest advancements in the field, we seek to develop effective techniques that can reliably reduce the order of LTI systems while preserving their essential dynamic characteristics.

AFD and SCRVM methods are effective in reducing LTI systems. These methods rely on the rational orthogonal basis, also called the Takenaka–Malmquist (TM) system [17]. In the case of the open right-half plane $\mathrm{\Pi }=\{s\in \mathbb{C}:\mathfrak{R}(s)>0\}$, the TM system is defined as follows:

$$\begin{array}{c}\hfill {\mathcal{B}}_k\left(s\right)={\displaystyle \frac{\sqrt{2\mathfrak{R}\left(a_k\right)}}{s+{\overline{a}}_k}}\prod _{l=1}^{k-1}{\displaystyle \frac{s-a_l}{s+{\overline{a}}_l}},k=1,2,\dots ,\end{array}$$

(3)

where $a_k\in \mathrm{\Pi }$ and $\mathfrak{R}(·)$ are the complex value real parts. The basis ${\left\{{\mathcal{B}}_k\right\}}_{k=0}^{\infty }$ (with $B_0=1$) forms a basis of the Hardy-2 space $H_2(\mathrm{\Pi })$ if and only if

$$\begin{array}{c}\hfill \sum _{k=1}^{\infty }{\displaystyle \frac{\sqrt{2\mathfrak{R}\left(a_k\right)}}{1+|a_k{|}^2}}=\infty .\end{array}$$

(4)

The shifted Cauchy kernel, denoted as $\mathcal{B}\left(s\right)=e_a\left(s\right)={\displaystyle \frac{\sqrt{2\mathfrak{R}\left(a\right)}}{s+\overline{a}}}$, is a component of the TM system. These systems are commonly used in model reduction due to their parameter-based linear model structure [18,19,20,21,22].

In recent years, the relevance vector machine (RVM) has emerged as a robust framework for solving sparse coding problems and providing uncertainty quantification [23,24,25]. RVM, also called sparse Bayesian learning (SBL), was proposed by Tipping [26] in 2001. RVM has achieved significant success in high-spectral image classification [27,28,29] and reconstruction [30]. It also has found applications in various fields such as direction-of-arrival estimation (DOA) [31,32,33], classification [25], compressive sensing [34], feature selection [35], signal processing [36,37,38], image reconstruction [30,39], financial prediction [40], and more.

The RVM offers computational efficiency by progressively “pruning” irrelevant vectors during the inference process, reducing computation time for covariance matrix inversion. This advantage is precious for high-dimensional problems. However, when dealing with large-scale datasets, RVM faces challenges due to the computational costs of the iterative process, scaling $O\left(T{M}^3\right)$ in time and $O\left(M^2\right)$ in space, where T is the iteration time, and M is the number of parameters. Additionally, for complex-valued data, the computational time increases to $O\left(T{\left(2M\right)}^3\right)$ with a storage space of $O\left({\left(2M\right)}^2\right)$. There exist several methods to reduce the costs of the iterative process, such as iteratively reweighted least-squares (IRLS) [41], approximate message passing (AMP) [42], and variational inference (VI) [43]. A popular method that does not require the computation of the inverse matrix, called inverse-free sparse Bayesian learning (IFSBL) [33,43], is often faster in practice. IFSBL bypasses matrix inversion via relaxed evidence lower bound (ELBO), employing a variational EM scheme for efficient, fast-convergence SBL. However, these methods need more scalability at very high dimensions M and more accurate recovery of the sparse codes. Researchers have proposed approximate inference algorithms and acceleration techniques like CoFEM [44], which have significant advantages in both scalability and accuracy over other SBL approaches. Unfortunately, CoFEM only handles real-valued data. This paper introduces a novel method called covariance-free maximum likelihood (CoFML) to address the complex-valued issue similar to CoFEM. CoFML, which omits explicit covariance matrix calculations, provides unbiased posterior estimates through linear system solutions and advanced numerical algebra, resulting in a fast, accurate, and sparsely approximated model.

The paper is organized as follows: Section 2 presents an innovative approach to enhance the strictly complex-valued relevant vector machine (SCRVM) algorithm’s computational efficiency in scenarios involving high-dimensional problems, which we call CoFML. Section 3 provides a theoretical analysis of CoFML. Numerical examples are presented in Section 4, followed by conclusions in Section 5.

2. Reduction in MIMO Systems

2.1. Strictly Complex-Valued Relvence Vector Machine Inference

We expand the MIMO LTI systems $F\left(s\right)$ in (1) row by row into L SISO LTI systems $F_l\left(s\right)$, with $F_l$ denoting the l-th system of $F\left(s\right)$, where $l=1,2,\dots ,L$ and $L=p\times q$. We define the single-input signal $\mathit{s}={[s_1,\cdots ,s_N]}^T$ and the single-output signal as ${\mathit{z}}_l={[z_{l1},\cdots ,z_{lN}]}^T$, and N is the number of examples. The transfer function of the l-th systems is represented as follows:

$$\begin{array}{c}\hfill {\mathit{z}}_l=F_l(\mathit{s};{\mathit{\theta }}_l)+{\mathit{v}}_l={\mathbf{\Omega }}_l{\mathit{\theta }}_l+{\mathit{v}}_l,\end{array}$$

(5)

where ${\mathit{\theta }}_l={[{\theta }_{l1},\cdots ,{\theta }_{lM}]}^T\in {\mathbb{C}}^{M\times 1}$, ${\mathbf{\Omega }}_l=$ $\left[\begin{array}{ccc}{e}_{a_1}\left(s_1\right)& \cdots & e_{a_M}\left(s_1\right)\\ ⋮& \ddots & ⋮\\ e_{a_1}\left(s_N\right)& \cdots & e_{a_M}\left(s_N\right)\end{array}\right]\in {\mathbb{C}}^{N\times M}$ and $e_{a_i}\left(s_j\right)={\displaystyle \frac{\sqrt{2\mathfrak{R}\left(a_i\right)}}{s_j+{\overline{a}}_i}},i=1,2,\cdots ,M,j=1,2,\cdots ,N$. The noise vector ${\mathit{v}}_l={[v_{l1},v_{l2},\cdots ,v_{lN}]}^T$ is assumed to be complex Gaussian-distributed, with $v_{ln}\sim \mathcal{N}\left(v_{ln}\right|0,{\sigma }^2{\mathit{I}}_2)$. Here, $0={[0,\cdots ,0]}^T\in {\mathbb{R}}^{N\times 1}$, $v_{ln}$ includes both the real and imaginary components of the complex noise, and ${\sigma }^2$ denotes the variance. Notably, those systems have the same denominator, so we have ${\mathbf{\Omega }}_1={\mathbf{\Omega }}_2=\dots ={\mathbf{\Omega }}_L$. By considering the real and imaginary parts separately and assuming independence among ${\mathit{z}}_l$, the likelihood function for the complete data set can be expressed as follows:

$$\begin{array}{cc}\hfill p\left({\mathit{Z}}_l\right|{\mathbf{\Theta }}_l,{\sigma }^2)& =\mathcal{N}\left({\mathit{Z}}_l\right|\mathit{K}{\mathbf{\Theta }}_l,{\sigma }^2{\mathit{I}}_{2N})\hfill \\ \hfill & ={\left(2\pi {\sigma }^2\right)}^{-N}exp\left\{-{\displaystyle \frac{1}{2}}{({\mathit{Z}}_l-\mathit{K}{\mathbf{\Theta }}_l)}^T{\sigma }^{-2}{\mathit{I}}_{2N}({\mathit{Z}}_l-\mathit{K}{\mathbf{\Theta }}_l)\right\},\hfill \end{array}$$

(6)

where $\mathit{K}=$$\left[\begin{array}{cc}\Re (\mathbf{\Omega })& -\Im (\mathbf{\Omega })\\ \Im (\mathbf{\Omega })& \Re (\mathbf{\Omega })\end{array}\right]$, ${\mathit{Z}}_l=$$\left[\begin{array}{c}\Re \left({\mathit{z}}_l\right)\\ \Im \left({\mathit{z}}_l\right)\end{array}\right]$, $\mathit{S}=$$\left[\begin{array}{c}\Re \left(\mathit{s}\right)\\ \Im \left(\mathit{s}\right)\end{array}\right]$, ${\mathbf{\Theta }}_l=$$\left[\begin{array}{c}\Re \left({\mathit{\theta }}_l\right)\\ \Im \left({\mathit{\theta }}_l\right)\end{array}\right]$, and $\Im (·)$ is the imaginary part of a complex number.

The above notation represents a $2N$-dimensional Gaussian distribution over ${\mathit{Z}}_l$ with a mean of $\mathit{K}{\mathbf{\Theta }}_l$ and variance of ${\sigma }^2{\mathit{I}}_{2N}$. For simplicity, the implicit conditioning on the set of input vectors $\mathit{S}$ is omitted in Equation (6) and subsequent expressions during inference.

The prior distribution for the coefficients ${\mathbf{\Theta }}_l$ is modeled as a zero-mean Gaussian distribution:

$$\begin{array}{c}\hfill p\left({\mathbf{\Theta }}_l\right|\mathit{\alpha })=\mathcal{N}\left({\mathbf{\Theta }}_l\right|\mathbf{0},\mathit{A}),\end{array}$$

(7)

where $0={[0,\cdots ,0]}^T\in {\mathbb{R}}^{2M\times 1}$, $\mathit{A}=\mathrm{diag}\left(\mathit{\alpha }\right)$, and $\mathit{\alpha }={[{\alpha }_1,\cdots ,{\alpha }_{2M}]}^T$. To enforce sparsity with equal sparsity patterns for the real and imaginary parts, we set ${\alpha }_l={\alpha }_{M+l}>0$ for $l=1,2,\cdots ,M$. For convenience, we define ${\mathit{A}}_1=\mathrm{diag}\left({[{\alpha }_1,\cdots ,{\alpha }_M]}^T\right)$, so that $\mathit{A}=\left[\begin{array}{cc}{\mathit{A}}_1& \mathit{O}\\ \mathit{O}& {\mathit{A}}_1\end{array}\right]$ where $\mathit{O}$ is the zero matrix. Additionally, we assign Gamma distributions [24] as hyperpriors for ${\alpha }_m$ and the noise variance ${\sigma }^2$:

$$\begin{array}{cc}\hfill p\left(\mathit{\alpha }\right)& =\prod _{m=1}^{2M}\mathrm{\Gamma }\left({\alpha }_m^{-1}\right|a,b),\hfill \\ \hfill p\left({\sigma }^{-2}\right)& =\mathrm{\Gamma }\left({\sigma }^2\right|c,d).\hfill \end{array}$$

We set $a=b=c=d={10}^{-4}$ [26] to ensure flat priors.

By using Bayes’ rule, the posterior covariance and mean are given by

$$\begin{array}{c}\hfill \mathbf{\Sigma }={({\sigma }^{-2}{\mathit{K}}^T\mathit{K}+\mathit{A})}^{-1},\end{array}$$

(8)

and

$$\begin{array}{c}\hfill {\mathit{U}}_l={\sigma }^{-2}\mathbf{\Sigma }{\mathit{K}}^T{\mathit{Z}}_l,\end{array}$$

(9)

respectively. To facilitate subsequent inference, we introduce the notation ${\mu }_{il}=U_{il}+j{U}_{i+Ml}$, where $i=1,2,\cdots ,M$. We define ${\mathit{\mu }}_l={({\mu }_{1l},{\mu }_{2l},\cdots ,{\mu }_{Ml})}^T$, leading to ${\mathit{U}}_l=\left[\begin{array}{c}\mathfrak{R}\left({\mathit{\mu }}_l\right)\\ \mathfrak{I}\left({\mathit{\mu }}_l\right)\end{array}\right]$.

To infer the SCRVM, we use the augmented vector $\underline{{\mathit{z}}_l}=\left[\begin{array}{c}{\mathit{z}}_l\\ {\mathit{z}}_{\mathit{l}}^{*}\end{array}\right]$, $\underline{{\mathit{\theta }}_l}=\left[\begin{array}{c}{\mathit{\theta }}_l\\ {\mathit{\theta }}_{\mathit{l}}^{*}\end{array}\right]$, $\underline{{\mathit{\mu }}_l}=\left[\begin{array}{c}{\mathit{\mu }}_l\\ {\mathit{\mu }}_{\mathit{l}}^{*}\end{array}\right]$ and the augmented matrix $\underline{\mathbf{\Omega }}=\left[\begin{array}{cc}\mathbf{\Omega }& \mathit{O}\\ \mathit{O}& {\mathbf{\Omega }}^{*}\end{array}\right]$ [45]. Using a simple relation of the composite augmented vector and matrix, given by ${\underline{\mathit{z}}}_l=\mathit{T}{\mathit{Z}}_l$, ${\underline{\mathit{\theta }}}_l=\mathit{T}{\mathbf{\Theta }}_l$, ${\underline{\mathit{\mu }}}_l=\mathit{T}{\mathit{U}}_l$, and $\underline{\mathbf{\Omega }}={\displaystyle \frac{1}{2}}\mathit{T}\mathit{K}{\mathit{T}}^H$, where $\mathit{T}=\left[\begin{array}{cc}\mathit{I}& j\mathit{I}\\ \mathit{I}& -j\mathit{I}\end{array}\right]\in {\mathbb{C}}^{2N\times 2N}$, and ${\mathit{T}}^H\mathit{T}=\mathit{T}{\mathit{T}}^H=2\mathit{I}$. Hence, we have ${\mathit{Z}}_l={\displaystyle \frac{1}{2}}{\mathit{T}}^H{\underline{\mathit{z}}}_l$, ${\mathbf{\Theta }}_l={\displaystyle \frac{1}{2}}{\mathit{T}}^H{\underline{\mathit{\theta }}}_l$, and ${\mathit{U}}_l={\displaystyle \frac{1}{2}}{\mathit{T}}^H{\underline{\mathit{\mu }}}_l$. By using these simple transformations, we obtain the following:

$$\begin{array}{cc}\hfill p\left({\mathit{\theta }}_l\right|{\mathit{z}}_l,\mathit{\alpha },{\sigma }^2)& ={\displaystyle \frac{1}{{\left(\pi \right)}^M{\left|{\mathbf{\Sigma }}_{{\mathit{\theta }}_{\mathit{l}}{\mathit{\theta }}_{\mathit{l}}}\right|}^{\frac{1}{2}}}}exp\{-{\displaystyle \frac{1}{2}}{({\underline{\mathit{\theta }}}_l-{\underline{\mathit{\mu }}}_l)}^H{\mathbf{\Sigma }}_{{\mathit{\theta }}_{\mathit{l}}{\mathit{\theta }}_{\mathit{l}}}^{-1}({\underline{\mathit{\theta }}}_l-{\underline{\mathit{\mu }}}_l)\},\hfill \end{array}$$

(10)

where

$$\begin{array}{cc}\hfill {\mathbf{\Sigma }}_{{\mathit{\theta }}_{\mathit{l}}{\mathit{\theta }}_{\mathit{l}}}^{-1}& ={\displaystyle \frac{1}{4}}\mathit{T}{\mathbf{\Sigma }}^{-1}{\mathit{T}}^H=\left[\begin{array}{cc}{\mathbf{\Lambda }}^{-1}& \mathit{O}\\ \mathit{O}& {\mathbf{\Lambda }}^{-*}\end{array}\right],\hfill \\ \hfill |{\mathbf{\Sigma }}_{{\mathit{\theta }}_{\mathit{l}}{\mathit{\theta }}_{\mathit{l}}}|& =2^{2M}|\mathbf{\Sigma }|,\hfill \end{array}$$

(11)

and

$$\begin{array}{c}\hfill {\underline{\mathit{\mu }}}_l=\mathit{T}{\mathit{U}}_l={\left(2{\sigma }^2\right)}^{-1}{\mathbf{\Sigma }}_{{\mathit{\theta }}_{\mathit{l}}{\mathit{\theta }}_{\mathit{l}}}{\mathbf{\Omega }}^H{\underline{\mathit{z}}}_l=\left[\begin{array}{c}{\mathit{\mu }}_l\\ {\mathit{\mu }}_{\mathit{l}}^{*}\end{array}\right],\end{array}$$

(12)

where $\mathbf{\Lambda }={\left({\left(2{\sigma }^2\right)}^{-1}{\mathbf{\Omega }}^H\mathbf{\Omega }+{\displaystyle \frac{{\mathit{A}}_{\mathbf{1}}}{2}}\right)}^{-1}$ and ${\mathit{\mu }}_l={\left(2{\sigma }^2\right)}^{-1}\mathbf{\Lambda }{\mathbf{\Omega }}^H{\mathit{z}}_l$.

To simplify the inference, we introduce the notation $c^2=2{\sigma }^2$ and $\mathit{E}={\displaystyle \frac{1}{2}}{\mathit{A}}_1$ (with $b_i={\displaystyle \frac{{\alpha }_i}{2}}$), yielding the following:

$$\begin{array}{c}\hfill \mathbf{\Lambda }={(c^{-2}{\mathbf{\Omega }}^H\mathbf{\Omega }+\mathit{E})}^{-1},\end{array}$$

(13)

and

$$\begin{array}{c}\hfill {\mathit{\mu }}_l=c^{-2}\mathbf{\Lambda }{\mathbf{\Omega }}^H{\mathit{z}}_l.\end{array}$$

(14)

Therefore, the distribution of Equation (10) can also be rewritten as follows:

$$p\left({\mathit{\theta }}_l\right|{\mathit{z}}_l,\mathit{\alpha },{\sigma }^2)={\displaystyle \frac{1}{{\left(\pi \right)}^M|\mathbf{\Lambda }|}}exp\{-{({\mathit{\theta }}_l-{\mathit{\mu }}_l)}^H{\mathbf{\Lambda }}^{-1}({\mathit{\theta }}_l-{\mathit{\mu }}_l)\}.$$

(15)

Then, we also have

$$p\left({\mathit{z}}_l\right|\mathit{\alpha },{\sigma }^2)={\left(\pi \right)}^{-N}{\left|\mathit{D}\right|}^{-1}exp\left\{-{\mathit{z}}_l^H{\mathit{D}}^{-1}{\mathit{z}}_l\right\},$$

(16)

where $\mathit{D}=2{\sigma }^2\mathit{I}+2\mathbf{\Omega }{A}_1^{-1}{\mathbf{\Omega }}^H=c^2\mathit{I}+\mathbf{\Omega }{\mathit{E}}^{-1}{\mathbf{\Omega }}^H$ is a Hermitian positive definite matrix [46].

So, we can represent the joint probability density function of MIMO LTI systems as follows:

$$\begin{array}{cc}\hfill \mathcal{L}(\mathit{\alpha },{\sigma }^2)& =\sum _{l=1}^{L}logp\left({\mathit{z}}_l\right|\mathit{\alpha },{\sigma }^2)\hfill \\ \hfill & =-(LNlog\pi +Llog|\mathit{D}|+\sum _{l=1}^L{\mathit{z}}_l^H{\mathit{D}}^{-1}{\mathit{z}}_l).\hfill \end{array}$$

(17)

Using the maximum likelihood method [24] to find its maximum value, we obtain the following:

$${\alpha }_i^{new}={\displaystyle \frac{{\gamma }_i}{\frac{1}{L}{\displaystyle \sum _{l=1}^L}{\left|{\mu }_{il}\right|}^2}},$$

(18)

and

$${\left({\sigma }^2\right)}^{new}={\displaystyle \frac{2N}{\frac{1}{L}{\displaystyle \sum _{l=1}^L}\left|\right|{\mathit{z}}_l-\mathbf{\Omega }{\mathit{\mu }}_l{\left|\right|}^2+Tr(\mathbf{\Lambda }{\mathbf{\Omega }}^H\mathbf{\Omega })}},$$

(19)

where ${\gamma }_i=2-{\alpha }_i{\mathrm{\Lambda }}_{ii}$, ${\mu }_{il}$ is the i-th elements of (14), and $Tr(·)$ is the trace of a matrix.

2.2. An Estimator for the Diagonal of Covariance Matrix

We adopt the technique in [47] to estimate the diagonal components of $\mathbf{\Lambda }$.

Proposition 1. 

Let ${\mathit{v}}_k={[v_{k,1},v_{k,2},\dots ,v_{k,M}]}^T\in {\mathbb{C}}^{M\times 1}(k=1,2,\cdots ,2M$) be random probe vectors, where each ${\mathit{v}}_k$ comprises independent and identically distributed elements such that $\mathbb{E}\left[{\sum }_{k=1}^{2M}{v}_{k,i}·v_{k,j}\right]=0$ for all $i\ne j=1,\dots ,M$. Moreover, it also assumes that ${\sum }_{k=1}^{2M}{v}_{k,i}·v_{k,j^{\prime }}$ is independent of ${\sum }_{k=1}^{2M}{v}_{k,i}·v_{k,j^{″}},j^{\prime }\ne j^{″}$. For each ${\mathit{v}}_k$. Let ${\mathit{r}}_k=\mathbf{\Lambda }{\mathit{v}}_k$, where Λ is (13) and ${\mathit{r}}_k={[r_{k,1},\cdots ,r_{k,M}]}^T$. Consider the estimator $\mathit{\tau }\in {\mathbb{R}}^{M\times 1}$, where, for each $j=1,\dots ,M$,

$$\begin{array}{c}\hfill {\tau }_i={\displaystyle \frac{{\displaystyle \sum _{k=1}^{2M}}{v}_{k,i}·r_{k,i}}{{\displaystyle \sum _{k=1}^{2M}}{v}_{k,i}^2}}.\end{array}$$

(20)

Then, the ${\tau }_i$ provides an unbiased estimate of ${\mathrm{\Lambda }}_{ii}$.

Proof. 

The expected value of ${\tau }_i$ is given by the following:

$$\mathbb{E}\left[{\tau }_i\right]={\mathrm{\Lambda }}_{i,i}+\sum _{j\ne i}{\mathrm{\Lambda }}_{i,j}·\left({\displaystyle \frac{\mathbb{E}\left[{\displaystyle \sum _{k=1}^{2M}}{v}_{k,j}·v_{k,i}\right]}{{\displaystyle \sum _{k=1}^{2M}}{v}_{k,i}^2}}\right),$$

because the $\mathbb{E}\left[{\sum }_{k=1}^{2M}{v}_{k,i}{v}_{k,j}\right]=0$ for all $i\ne j$, and we have $\mathbb{E}\left[{\tau }_i\right]={\mathrm{\Lambda }}_{i,i}$. The proof is completed.    □

In particular, we assume that ${\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_{2M}$ are independent Rademacher variables, which means $P(v_{k,i}=-1)=P(v_{k,i}=1)=0.5$. Then, we can simplify (20) to the following form:

$$\begin{array}{c}\hfill {\tau }_i={\displaystyle \frac{1}{2M}}\sum _{k=1}^{2M}{v}_{k,i}·r_{k,i}.\end{array}$$

(21)

Theorem 1. 

Let ${\tau }_i$ be the estimator given by Equation (21), satisfying the conditions of Proposition 1. Then, ${\tau }_i$ is one of the optimal estimators for ${\mathrm{\Lambda }}_{ii}$.

Proof. 

By using Proposition 1 and the calculated methods of [44] that confirm $\mathbb{E}\left[{\tau }_i\right]={\mathrm{\Lambda }}_{i,i}$, we can calculate the variance ${\varsigma }_i^2$ of ${\tau }_i$ as follows:

$$\begin{array}{cc}\hfill {\varsigma }_i^2& =\mathbb{E}\left[{\left({\tau }_i-\mathbb{E}\left[{\tau }_i\right]\right)}^2\right]\hfill \\ \hfill & =\mathbb{E}\left[{\left(\sum _{i^{\prime }\ne i}{\mathrm{\Lambda }}_{i,i^{\prime }}·{\displaystyle \frac{{\displaystyle \sum _{k=1}^{2M}}{v}_{k,i}·v_{k,i^{\prime }}}{{\displaystyle \sum _{k=1}^{2M}}{v}_{k,i}^2}}\right)}^2\right]\hfill \\ \hfill & =\mathbb{E}\left[\sum _{i^{\prime }\ne i}\sum _{i^{″}\ne i}{\mathrm{\Lambda }}_{i,i^{\prime }}·{\mathrm{\Lambda }}_{i,i^{″}}·{\displaystyle \frac{e_{i,i^{\prime }}}{e_{i,i}}}·{\displaystyle \frac{e_{i,i^{″}}}{e_{i,i}}}\right],\hfill \end{array}$$

where $e_{i,l}:={\sum }_{k=1}^{2M}{v}_{k,i}·v_{k,l}$. In the numerator, due to the independence of ${\sum }_{k=1}^{2M}{v}_{k,i}·v_{k,i^{\prime }}$ and ${\sum }_{k=1}^{2M}{v}_{k,i}·v_{k,i^{″}}$ when $i^{\prime }\ne i^{″}$, we observe that $\mathbb{E}\left[e_{i,i^{\prime }}·e_{i,i^{″}}\right]=\mathbb{E}\left[e_{i,i^{\prime }}\right]\mathbb{E}\left[e_{i,i^{″}}\right]=0$. Consequently, ${\tau }_i$ serves as an optimal estimator for ${\mathrm{\Lambda }}_{i,i}$. This completes the proof.    □

Then, we transform the inversion problem into the problem of solving a linear system; that is, solving the linear equation $\mathbf{C}\mathbf{y}=\mathbf{b}$, where $\mathbf{C}:={\mathbf{\Lambda }}^{-1}$ and $\mathbf{b}:={\mathit{v}}_k$. Furthermore, we can concurrently solve these systems by considering the matrix equation $\mathbf{CY}=\mathbf{B}$, where the inputs $\mathbf{C}\in {\mathbb{C}}^{M\times M}$ and $\mathbf{B}\in {\mathbb{C}}^{M\times (2M+L)}$ are defined as follows:

$$\begin{array}{cc}& \mathbf{C}:=c^{-2}{\mathbf{\Omega }}^H\mathbf{\Omega }+\mathit{E},\hfill \\ & \mathbf{B}:=[{\mathit{b}}_1,{\mathit{b}}_2,\dots ,{\mathit{b}}_{2M+L}]=\left[{\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_{2M},c^{-2}{\mathbf{\Omega }}^H{\mathit{z}}_1,\cdots ,c^{-2}{\mathbf{\Omega }}^H{\mathit{z}}_L\right].\hfill \end{array}$$

By labeling the columns of the solution matrix $\mathbf{Y}\in {\mathbb{C}}^{M\times (2M+L)}$ as

$$\mathit{Y}:=[{\mathit{y}}_1,{\mathit{y}}_2\dots ,{\mathit{y}}_{2M},\dots ,{\mathit{y}}_{2M+L}]=[{\mathit{r}}_1,\dots ,{\mathit{r}}_{2M},{\mathit{\mu }}_1,\dots ,{\mathit{\mu }}_L],$$

our desired quantities for CoFML, ${\mathit{\mu }}_l$ and $\mathit{\tau }$, can be calculated using (21). Then, we perform the update in (18) as

$${\alpha }_i^{new}={\displaystyle \frac{{\gamma }_i}{\frac{1}{L}{\displaystyle \sum _{l=1}^L}{\left|{\mu }_{jl}\right|}^2}},$$

(22)

where ${\gamma }_i=2-{\alpha }_i{\tau }_i$.

Then, we choose the conjugate gradient (CG) algorithm to solve the multiple linear system $\mathbf{CY}=\mathbf{B}$. This requires converting the matrix–vector multiplication of the CG algorithm for a single linear system into a matrix–matrix multiplication.

Lemma 1. 

Consider the CG algorithm applied to solve ${\mathbf{Cy}}_i={\mathit{b}}_i$ for $i=1,2,\cdots ,2M+L$, where $\mathbf{C}\in {\mathbb{C}}^{M\times M}$ is a positive definite matrix, and $\mathit{B}\in {\mathbb{C}}^{M\times (2M+L)}$. Here, ${\mathit{y}}_i$ and ${\mathit{b}}_i$ represent the i-th column of $\mathit{Y}$ and $\mathit{B}$, respectively. Let ${\mathit{y}}_i^0\in {\mathbb{C}}^{M\times 1}$ be the initial solution, ${\mathit{y}}_i^{*}$ be the exact solution, and ${\mathit{y}}_i^k$ denote the solution obtained by the CG algorithm at the k-th step. We can establish the following relationship:

$${∥{\mathit{y}}_i^k-{\mathit{y}}_i^{*}∥}_{\mathit{C}}\le 2{\left({\displaystyle \frac{\sqrt{K}-1}{\sqrt{K}+1}}\right)}^k{∥{\mathit{y}}_i^0-{\mathit{y}}_i^{*}∥}_{\mathit{C}},$$

(23)

where ${\left|\right|\mathit{x}\left|\right|}_{\mathbf{C}}:=\sqrt{{\mathit{x}}^H\mathbf{C}\mathit{x}}$ denotes the norm induced by the positive definite matrix $\mathbf{C}$ for any vector $\mathit{x}\in {\mathbb{C}}^{M\times 1}$, and $K={\mathit{cond}}_2\left(\mathbf{C}\right)$ [48]. From (23), we observe that when $\mathbf{C}$ is ill-conditioned ($K>1$), the convergence of the CG algorithm tends to be slow. However, in SBL iterations, many diagonal elements of ${\alpha }_i$ are pushed towards infinity [26], resulting in a large value of K. To address this issue, we incorporate a preconditioned matrix into the CG algorithm, which will be discussed in the following section.

3. Preconditioned Conjugate Gradient Method for SCRVM

3.1. Preconditioned Matrix

Here, we would rather solve the equivalent system ${\mathit{C}}^{\prime }{\mathit{Y}}^{\prime }={\mathit{B}}^{\prime }$ than solve $\mathit{C}\mathit{Y}=\mathit{B}$, where ${\mathit{C}}^{\prime }:={\mathit{P}}^{-1/2}{\mathit{CP}}^{-1/2}$, ${\mathit{B}}^{\prime }:={\mathit{P}}^{-1/2}\mathit{B}$, and ${\mathit{Y}}^{\prime }:={\mathit{P}}^{1/2}\mathit{Y}$. The matrix $\mathit{P}$ is called the preconditioned matrix. Before presenting the convergence proof of the preconditioned conjugate gradient method, the parallel conjugate gradient (PCG) algorithm is outlined, as shown in Algorithm 1.

Algorithm 1 $\mathbf{PCG}(\mathit{C},\mathit{B},\mathit{P},T,2M)$

1: $\mathbf{Initialize}{\alpha }_i^{\left(1\right)}\leftarrow 1\mathrm{for}i=1,\dots ,M.$ 2:$\mathbf{for}t=1,2,\dots ,T\mathbf{do}$ 3:$\mathrm{Define}\mathbf{C}\leftarrow c^{-2}{\mathbf{\Omega }}^H\mathbf{\Omega }+\mathit{E}.$ 4:$\mathrm{Draw}{\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_{2M}\sim \mathrm{Rademacher}\mathrm{distribution}.$ 5:$\mathrm{Define}\mathbf{B}\leftarrow \left[{\mathit{v}}_1,{\mathit{v}}_2,\dots ,{\mathit{v}}_{2M},c^{-2}{\mathbf{\Omega }}^H{\mathit{z}}_1,c^{-2}{\mathbf{\Omega }}^H{\mathit{z}}_2,\cdots ,c^{-2}{\mathbf{\Omega }}^H{\mathit{z}}_L\right].$ 6:${\mathbf{C}}^{\prime }\leftarrow {\mathbf{P}}^{-\frac{1}{2}}\mathbf{C}{\mathbf{P}}^{-\frac{1}{2}},{\mathbf{B}}^{\prime }\leftarrow {\mathbf{P}}^{-\frac{1}{2}}\mathbf{B}.$ 7:${\mathbf{Y}}^{\prime }\leftarrow \mathrm{CG}({\mathbf{C}}^{\prime },{\mathbf{B}}^{\prime }).$ 8:$\mathbf{Y}\leftarrow {\mathbf{P}}^{-\frac{1}{2}}{\mathbf{Y}}^{\prime }.$ 9:$\mathrm{Compute}{\tau }_i\leftarrow {\displaystyle \frac{1}{2M}}{\displaystyle \sum _{k=1}^{2M}}{v}_{k,i}·r_{k,i}\mathrm{for}i=1,\dots ,M.$ 10:$\mathbf{if}{\alpha }_i^{(t+1)}=\infty \mathbf{and}t<T\mathbf{then}$ 11:$\mathrm{Delete}\mathrm{the}\mathrm{corresponding}\mathrm{columns}\mathrm{of}\mathbf{\Omega }.$ 12:$\mathbf{else}$ 13:$\mathrm{Compute}{\gamma }_i\leftarrow 2-{\alpha }_i^{\left(t\right)}{\tau }_i.$ 14:$\mathrm{Update}{\alpha }_i^{(t+1)}\leftarrow {\displaystyle \frac{{\gamma }_i}{\frac{1}{L}{\displaystyle \sum _{l=1}^L}{\left|{\mu }_{il}\right|}^2}},\mathrm{for}i=1,\dots ,M.$ 15:$\mathbf{end}\mathbf{if}$ 16:$\mathbf{end}\mathbf{for}$ 17:$\mathbf{return}{\mathit{\alpha }}^{\left(T\right)},{\mathit{\mu }}_1,{\mathit{\mu }}_2,\cdots ,{\mathit{\mu }}_L,\mathit{\tau }.$

However, in practice, it is impossible to obtain ${\alpha }^{(t+1)}=\infty$ in Algorithm 1. So, we usually use the condition ${\alpha }_i^{(t+1)}>{10}^{12}$ to replace ${\alpha }^{(t+1)}=\infty$. Finally, we call the final Algorithm 1 CoFML. After running Algorithm 1, we can obtain the approximate model (2). In particular, when $n=2$ and the basis consists of real numbers, the l-th system of the second approximating model is given by the following:

$$\begin{array}{cc}\hfill {\widehat{F}}_{2,l}\left(s\right)& ={\mu }_{1,l}{\displaystyle \frac{\sqrt{2\mathfrak{R}\left(a_1\right)}}{s+a_1}}+{\mu }_{2,l}{\displaystyle \frac{\sqrt{2\mathfrak{R}\left(a_2\right)}}{s+a_2}},\hfill \\ \hfill & ={\displaystyle \frac{({\mu }_{1,l}\sqrt{2\mathfrak{R}\left(a_1\right)}+{\mu }_{2,l}\sqrt{2\mathfrak{R}\left(a_2\right)})s+({\mu }_{1,l}{a}_2\sqrt{2\mathfrak{R}\left(a_1\right)}+{\mu }_{2,l}{a}_1\sqrt{2\mathfrak{R}\left(a_2\right)})}{s^2+(a_1+a_2)s+a_1{a}_2}},\hfill \\ \hfill & ={\displaystyle \frac{({\mu }_{1,l}\sqrt{2a_1}+{\mu }_{2,l}\sqrt{2a_2})s+({\mu }_{1,l}{a}_2\sqrt{2a_1}+{\mu }_{2,l}{a}_1\sqrt{2a_2})}{s^2+(a_1+a_2)s+a_1{a}_2}}.\hfill \end{array}$$

To ensure the l-th steady-state values of the reduced model are equal to the original systems (1), we use the following constraint:

$$\begin{array}{c}\hfill {\displaystyle \frac{{\mu }_{1,l}{a}_2\sqrt{2a_1}+{\mu }_{2,l}{a}_1\sqrt{2a_2}}{a_1{a}_2}}\approx {\displaystyle \frac{B_{0,l}}{b_0}}.\end{array}$$

(24)

3.2. Convergence of CoFML

When the values of $a,b,c,d$ are small and $t\to \infty$, it is common for specific $\alpha$ values to approach infinity and effectively “prune” the associated “nuisance” parameters while maintaining finite values for $\alpha$ [26]. We designate these retained parameters as the “true” parameters.

Definition 1 

(CoFML Convergence). In Algorithm 1, let $\widehat{\mathit{\alpha }}:=\underset{t\to \infty }{lim}{\mathit{\alpha }}^{\left(t\right)}$, and the fulfillment of $(\mathcal{N},\mathcal{T},\widehat{\mathit{\alpha }})$-convergence for CoFML is established. Partition the indices ${\mathbb{N}}_M:=\{1,2,\dots ,M\}$ into a “nuisance” set denoted as $\mathcal{N}\subseteq {\mathbb{N}}_M$ and a “true” set denoted as $\mathcal{T}:={\mathbb{N}}_M\setminus \mathcal{N}$, respectively. In this partition, ${\widehat{\alpha }}_i$ is finite when $i\in \mathcal{T}$, while ${\widehat{\alpha }}_i=\infty$ when $i\in \mathcal{N}$.

Here, we denote the resulting matrices with the appropriate “true” and “nuisance” columns removed as ${\mathbf{\Omega }}_{\mathcal{T}}$ and ${\mathbf{\Omega }}_{\mathcal{N}}$, respectively. By leveraging the expression for the inverse of a partitioned matrix, we can demonstrate the following relationship:

$$\begin{array}{cc}\hfill \mathbf{\Lambda }& ={\left[c^{-2}{\mathbf{\Omega }}^H\mathbf{\Omega }+\mathbf{E}\right]}^{-1}\hfill \\ \hfill & ={\left[c^{-2}{\left[\begin{array}{cc}{\mathbf{\Omega }}_{\mathcal{T}}& {\mathbf{\Omega }}_{\mathcal{N}}\end{array}\right]}^H\left[\begin{array}{cc}{\mathbf{\Omega }}_{\mathcal{T}}& {\mathbf{\Omega }}_{\mathcal{N}}\end{array}\right]+\left[\begin{array}{cc}{\mathbf{E}}_{\mathcal{T}}\hfill & \mathbf{O}\hfill \\ \mathbf{O}\hfill & {\mathbf{E}}_{\mathcal{N}}\hfill \end{array}\right]\right]}^{-1}\hfill \\ \hfill & ={\left[c^{-2}\left[\begin{array}{c}{\mathbf{\Omega }}_{\mathcal{T}}^H\\ {\mathbf{\Omega }}_{\mathcal{N}}^H\end{array}\right]\left[\begin{array}{cc}{\mathbf{\Omega }}_{\mathcal{T}}& {\mathbf{\Omega }}_{\mathcal{N}}\end{array}\right]+\left[\begin{array}{cc}{\mathbf{E}}_{\mathcal{T}}& \mathbf{O}\\ \mathbf{O}& {\mathbf{E}}_{\mathcal{N}}\end{array}\right]\right]}^{-1}\hfill \\ \hfill & ={\left[\begin{array}{cc}{c}^{-2}{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}+{\mathbf{E}}_{\mathcal{T}}& c^{-2}{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{N}}\\ c^{-2}{\mathbf{\Omega }}_{\mathcal{N}}^H{\mathbf{\Omega }}_{\mathcal{T}}& c^{-2}{\mathbf{\Omega }}_{\mathcal{N}}^H{\mathbf{\Omega }}_{\mathcal{N}}+{\mathbf{E}}_{\mathcal{N}}\end{array}\right]}^{-1}\hfill \\ \hfill & \stackrel{t\to \infty }{\to }\left[\begin{array}{cc}{\mathbf{\Lambda }}_{\mathcal{T}}& \mathbf{O}\\ \mathbf{O}& \mathbf{O}\end{array}\right].\hfill \end{array}$$

Building upon the derivation above, we can establish the convergence theorem for the preconditioned matrix.

Theorem 2 

(PCG Convergence). Let ${\mathbf{C}}^{\left(t\right)}:=c^{-2}{\mathbf{\Omega }}^H\mathbf{\Omega }+{\mathbf{E}}^{\left(\mathbf{t}\right)}$ and ${\mathbf{P}}^{\left(t\right)}:=c^{-2}\mathit{I}+{\mathbf{E}}^{\left(\mathbf{t}\right)}$, respectively, represent the inverse-covariance matrix and preconditioned matrix at the t-th iteration of Algorithm 1. We let ${\mathbf{C}}^{\prime \left(t\right)}:={\left({\mathbf{P}}^{\left(t\right)}\right)}^{-1/2}{\mathbf{C}}^{\left(t\right)}{\left({\mathbf{P}}^{\left(t\right)}\right)}^{-1/2}$, ${\mathit{b}}^{\prime \left(t\right)}:={\left({\mathbf{P}}^{\left(t\right)}\right)}^{-1/2}{\mathit{b}}^{\left(t\right)}$, and ${\mathit{y}}^0\in {\mathbb{C}}^{M\times 1}$ be the initial solution, ${\mathit{y}}^{*}$ is the exact solution, and ${\mathit{y}}^{k,\left(t\right)}$ represents the solution obtained by the algorithm at the k-th step and t-th iteration. Then, given $(\mathcal{N},\mathcal{T},\widehat{\mathit{\alpha }})$-convergence, it follows that

$$\underset{t\to \infty }{lim}\parallel {\mathit{y}}^{k,\left(t\right)}-{\mathit{y}}^{*}{\parallel }_{{\widehat{\mathbf{C}}}^{\prime }}\le 2exp\{-k\sqrt{{\displaystyle \frac{1-\eta }{1+\eta }}}\}{\parallel {\mathit{y}}^0-{\mathit{y}}^{*}\parallel }_{{\widehat{\mathbf{C}}}^{\prime }},$$

(25)

where $\eta =\parallel {\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}{-\mathit{I}\parallel }_2$.

Proof. 

From Lemma 1, we have the following residual:

$$\begin{array}{cc}\hfill \parallel {\mathit{y}}^{k,\left(t\right)}-{\mathit{y}}^{*}{\parallel }_{{\mathbf{C}}^{\prime \left(t\right)}}& \le 2{\left({\displaystyle \frac{\sqrt{K^{\prime \left(t\right)}}-1}{\sqrt{K^{\prime \left(t\right)}}+1}}\right)}^k\parallel {\mathit{y}}^0-{\mathit{y}}^{*}{\parallel }_{{\mathbf{C}}^{\prime \left(t\right)}}\hfill \\ \hfill & \le 2{(1-{\displaystyle \frac{1}{\sqrt{K^{\prime \left(t\right)}}}})}^k\parallel {\mathit{y}}^0-{\mathit{y}}^{*}{\parallel }_{{\mathbf{C}}^{\prime \left(t\right)}}\hfill \\ \hfill & \le 2exp\{-{\displaystyle \frac{k}{\sqrt{K^{\prime \left(t\right)}}}}\}\parallel {\mathit{y}}^0-{\mathit{y}}^{*}{\parallel }_{{\mathbf{C}}^{\prime \left(t\right)}},\hfill \end{array}$$

(26)

where $K^{\prime \left(t\right)}={\lambda }_{max}\left({\mathbf{C}}^{\prime \left(t\right)}\right)/{\lambda }_{min}\left({\mathbf{C}}^{\prime \left(t\right)}\right)$ and $K^{\prime }=\underset{t\to \infty }{lim}{K}^{\prime \left(t\right)}.$ In (26), we notice that our goal is to bound $K^{\prime }={\lambda }_{max}\left({\mathbf{C}}^{\prime }\right)/{\lambda }_{min}\left({\mathbf{C}}^{\prime }\right)$.

Let $\mathbf{\Psi }:=c^{-2}{\mathbf{\Omega }}^H\mathbf{\Omega }-c^{-2}\mathit{I}$. So, we obtain the following:

$$\begin{array}{cc}\hfill {\mathbf{C}}^{\left(t\right)}& ={\mathbf{P}}^{\left(t\right)}+\mathbf{\Psi }⟹\hfill \\ \hfill {\mathbf{C}}^{\prime \left(t\right)}& =\mathbf{I}+{\left({\mathbf{P}}^{\left(t\right)}\right)}^{-\frac{1}{2}}\mathbf{\Psi }{\left({\mathbf{P}}^{\left(t\right)}\right)}^{-\frac{1}{2}}\hfill \\ \hfill & ⟹{\widehat{\mathbf{C}}}^{\prime }:=\underset{t\to \infty }{lim}{\mathbf{C}}^{\prime \left(t\right)}=\mathbf{I}+\left[\begin{array}{cc}{\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-\frac{1}{2}}{\mathbf{\Psi }}_{\mathcal{T},\mathcal{T}}{\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-\frac{1}{2}}& \mathit{O}\\ \mathit{O}& \mathit{O}\end{array}\right],\hfill \end{array}$$

(27)

where ${\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}:=c^{-2}{\mathit{I}}_{\mathcal{T}}+{\mathbf{E}}_{\mathcal{T}}$.

Ref. (27) demonstrates that if $\lambda$ is an eigenvalue of ${\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1/2}{\mathbf{\Psi }}_{\mathcal{T},\mathcal{T}}{\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1/2}$, then the eigenvalue of ${\widehat{\mathbf{C}}}^{\prime }$ is $1+\lambda$. $\mathbf{M}$ is similar to $\mathbf{N}$ if there exists an invertible matrix $\mathbf{T}$, such that $\mathbf{N}={\mathbf{T}}^{-1}\mathbf{M}\mathbf{T}$. Similar matrices have the same eigenvalues [49]. Consider $\mathbf{M}:={\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1/2}{\mathbf{\Psi }}_{\mathcal{T},\mathcal{T}}{\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1/2}$, $\mathbf{N}:={\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1}{\mathbf{\Psi }}_{\mathcal{T},\mathcal{T}}$ and $\mathbf{T}:={\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{1/2}$, so the eigenvalue of ${\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}{\mathbf{\Psi }}_{\mathcal{T},\mathcal{T}}$ is $\lambda$. Since the absolute value of the eigenvalue of a matrix does not exceed its spectral norm, we have

$$\begin{array}{cc}& \left|\lambda \right|\le {∥{\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1}{\mathbf{\Psi }}_{\mathcal{T},\mathcal{T}}∥}_2={∥{\widehat{\mathbf{P}}}_{\mathcal{T},\mathcal{T}}^{-1}\left(c^{-2}{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}-c^{-2}\mathit{I}\right)∥}_2\hfill \\ & \le {∥{\left(c^{-2}\mathit{I}\right)}^{-1}\left(c^{-2}{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}-c^{-2}\mathit{I}\right)∥}_2={∥{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}-\mathbf{I}∥}_2.\hfill \end{array}$$

It follows that ${\lambda }_{max}\left({\widehat{\mathbf{C}}}^{\prime }\right)\le 1+{∥{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}-\mathbf{I}∥}_2$ and ${\lambda }_{min}\left({\widehat{\mathbf{C}}}^{\prime }\right)\ge 1-{∥{\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}-\mathbf{I}∥}_2$, so we have

$$\begin{array}{c}\hfill K^{\prime }={\displaystyle \frac{{\lambda }_{max}}{{\lambda }_{min}}}\le {\displaystyle \frac{1+\parallel {\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}{-\mathbf{I}\parallel }_2}{1-\parallel {\mathbf{\Omega }}_{\mathcal{T}}^H{\mathbf{\Omega }}_{\mathcal{T}}{-\mathbf{I}\parallel }_2}},\end{array}$$

(28)

where we first let $t\to \infty$ on both sides of (26), and then substitute (28) into it to obtain (25). The Proof is completed. □

3.3. Computational Complexities of CoFML

After satisfying the convergence conditions mentioned above, we analyze the algorithm’s overall complexity. During each of the T iterations of CoFML, a maximum of $H(H<M)$ steps of conjugate gradient (CG) is required, resulting in an overall time complexity of $O\left(TH\right)$ and space complexity of $O\left(H\right)$ for each of the L systems.

4. Examples

This section provides three examples to illustrate the algorithm described above. The l-th system impulse response is defined as follows:

$$\begin{array}{c}\hfill h_l\left(t\right)={\displaystyle \frac{1}{2\pi j}}{\int }_{Q-j\infty }^{Q+j\infty }{e}^{\zeta t}{F}_l\left(\zeta \right)d\zeta ,\end{array}$$

(29)

where $F_l\left(\zeta \right)\in H_2(\mathrm{\Pi })$ is a high-order transfer function, and Q is chosen to be greater than the real part of the largest pole of $F_l\left(\zeta \right)$. The l-th system impulse response energy (IRE) is calculated as follows:

$$\begin{array}{c}\hfill IR{E}_l={\int }_0^{\infty }{h}_l^2\left(t\right)dt={\displaystyle \frac{1}{2\pi }}{\int }_{-\infty }^{\infty }{\left|F_l\left(j\omega \right)\right|}^{2}d\omega .\end{array}$$

(30)

Then, we consider three well-known examples when $L=1,2,4$, respectively.

Example 1. 

We first consider a SISO LTI system [16,50,51] as follows:

$$\begin{array}{c}\hfill F_{10}\left(s\right)={\displaystyle \frac{540.70748\times {10}^{17}}{{\displaystyle \prod _{l=1}^{10}}(s+b_l)}},\end{array}$$

(31)

where $b_1=2.04,b_2=18.3,b_3=50.13,b_4=95.15,b_5=148.85,b_6=205.16,b_7=257.21,b_8=298.03,b_9=320.97$, and $b_{10}=404.16$.

In this example, we use $N=251$ frequency-domain measurements within the interval $[-5j,5j]$ and $M=14$ basis functions within the interval $[0.1,10]$. By applying the CoFML method, we obtain $a_1=2.3846$, $a_2=6.9538$ and take the real part of $\mathit{\mu }$. Then, the 2nd partial sum is given by the following:

$$\begin{array}{c}\hfill {\widehat{F}}_2\left(s\right)={\displaystyle \frac{0.01053s+16.26}{s^2+9.338s+16.58}}.\end{array}$$

Figure 1 shows the step response comparison of the original system and the simplified model obtained by other methods, while Table 1 compares the IRE with different techniques. Figure 1 and Table 1 show that the CoFML method is effective.

Figure 1. Step responses of the original and reduced models.

Table 1. IRE of the reduced models.

Method	Reduced Model	IRE
Original system	$F_{10}\left(s\right)$	0.1503
CoFML	${\widehat{F}}_2\left(s\right)$	0.1422
AFD [14]	${\displaystyle \frac{-0.5367s+20.96}{s^2+11.86s+20.97}}$	0.1498
G. Parmar et al. [16]	${\displaystyle \frac{-28.367s+647.60193}{s^2+359.999s+647.60193}}$	8.2056
Edgar [52]	${\displaystyle \frac{-0.93s+26.28}{s^2+14.92s+26.4961}}$	0.1580
Therapos and Diamessis [51]	${\displaystyle \frac{-1.999638s+37.32915}{s^2+20.34s+37.332}}$	0.2027

Example 2. 

The next example is studied in [53] which is a 4th-order system, where

$$F_4\left(s\right)={\displaystyle \frac{\left[\begin{array}{c}28\hfill \\ 12\hfill \end{array}\right]s^3+\left[\begin{array}{c}496\hfill \\ 528\hfill \end{array}\right]s^2+\left[\begin{array}{c}1800\hfill \\ 1440\hfill \end{array}\right]s+\left[\begin{array}{c}2400\hfill \\ 4320\hfill \end{array}\right]}{2s^4+36s^3+204s^2+360s+240}}.$$

(32)

Here we use $N=334$ frequency-domain measurements in the interval $[-5j,5j]$ and $M=14$ basis numbers in the interval $[0.1,10]$. Using the CoFML method, we obtain $a_1=0.8615$, $a_2=1.6231$ and take the real part of the $\mathit{\mu }$. Then, the 2nd partial sum is given by the following:

$${\widehat{F}}_2\left(s\right)={\displaystyle \frac{\left[\begin{array}{c}12.3043\hfill \\ 9.5249\hfill \end{array}\right]s+\left[\begin{array}{c}14.4694\hfill \\ 26.2957\hfill \end{array}\right]}{s^2+2.4846s+1.3983}}.$$

(33)

Figure 2 shows the original and other reduced model step responses, while Table 2 compares the IRE with different methods. Figure 2 and Table 2 show the adequate CoFML model.

Figure 2. Step responses of the original and reduced models.

Table 2. IRE of the reduced models.

Method	Reduced Model	${\mathit{IRE}}_{\mathbf{1}}$	${\mathit{IRE}}_{\mathbf{2}}$
Original system	$F_4\left(s\right)$	11.7242	20.5297
CoFML	${\widehat{F}}_2\left(s\right)$	10.8311	19.4217
Routh [53]	${\displaystyle \frac{\left[\begin{array}{c}30\hfill \\ 24\hfill \end{array}\right]s+\left[\begin{array}{c}40\hfill \\ 72\hfill \end{array}\right]}{3s^2+6s+4}}$	10.2140	20.6014
S. Gugercin et al. [10,11]	${\displaystyle \frac{\left[\begin{array}{c}0.7108\hfill \\ 0.9026\hfill \end{array}\right]s^2+\left[\begin{array}{c}10.4003\hfill \\ 6.4600\hfill \end{array}\right]s+\left[\begin{array}{c}15.9178\hfill \\ 28.6521\hfill \end{array}\right]}{s^2+2.2218s+1.5918}}$	8.4488	17.9249

Example 3. 

We finally consider the transfer function study in [54]:

$$F_4\left(s\right)={\displaystyle \frac{\left[\begin{array}{cc}\hfill 14.96(s+1.7)(s+100)& \hfill 95150(s+1.898)(s+10)\\ \hfill 85.20(s+1.44)(s+100)& \hfill 124000(s+2.077)(s+10)\end{array}\right]}{(s+1.338354)(s+1.886647)(s+10)(s+100)}}.$$

(34)

In this example, we use $N=334$ frequency-domain measurements within the interval $[-5j,5j]$ and $M=8$ basis functions within the interval $[0.1,12]$. By applying the CoFML method, we obtain $a_1=1.8$, $a_2=3.5$ and take the real part of the $\mu$. With these values, the 2nd partial sum ${\widehat{F}}_2\left(s\right)$ is given by:

$${\widehat{F}}_2\left(s\right)={\displaystyle \frac{\left[\begin{array}{cc}\hfill 0.4517s+9.0010& \hfill 724.0193s+6141.1035\\ \hfill 3.6572s+46.0935& \hfill 856.7064s+8562.8471\end{array}\right]}{s^2+7s+9.36}}.$$

(35)

Figure 3 shows the step responses of the original system and the reduced models obtained using different methods. In contrast, Table 3 and Table 4 compare the IRE using other methods. From Figure 3 and Table 3 and Table 4, we can observe that the CoFML method is effective.

Figure 3. Step responses of the original and reduced models.

Table 3. IRE of the reduced models.

Model Reduction Method	Reduced Model
Original system	$F_4\left(s\right)$
CoFML	${\widehat{F}}_2\left(s\right)$
Padé [54]	${\displaystyle \frac{\left[\begin{array}{cc}\hfill 2.5095+1.2406s& \hfill 1782.9822+931.9822s\\ \hfill 12.1064+7.2706s& \hfill 2492.4439+1213.9036s\end{array}\right]}{s^2+3.1976s+2.4916}}$
S. Gugercin et al. [10,11]	${\displaystyle \frac{\left[\begin{array}{cc}\hfill -0.241s^2+0.7760s+2.8787& \hfill -8.9938s^2+932.5588s+2044.2005\\ \hfill -0.6891s^2+7.7785s+13.8874& \hfill -10.4496s^2+1202.2714s+2915.2559\end{array}\right]}{s^2+3.4801s+2.8581}}$

Table 4. IRE of the reduced models.

Method	${\mathit{IRE}}_{\mathbf{1}}$	${\mathit{IRE}}_{\mathbf{2}}$	${\mathit{IRE}}_{\mathbf{3}}$	${\mathit{IRE}}_{\mathbf{4}}$
Original system	0.1293	3.4819	$6.5224\times {10}^4$	$1.2412\times {10}^5$
CoFML	0.1280	3.5463	$6.9290\times {10}^4$	$1.2854\times {10}^5$
Padé [54]	0.1348	3.7616	$7.1415\times {10}^4$	$1.3136\times {10}^5$
S. Gugercin et al. [10,11]	0.2164	6.0371	$7.4435\times {10}^4$	$1.3888\times {10}^5$

5. Conclusions

In this paper, we developed the CoFML method to accelerate SCRVM. By solving the SCRVM inversion problem with unbiased estimates of the matrix diagonal elements, CoFML showed superior time and space efficiency compared to existing SCRVM techniques, especially when the number of unknowns M was large. Next, we theoretically analyzed the convergence of the CoFML algorithm, including convergence under preconditioned matrices. In addition, three well-known examples of results demonstrated that CoFML outperforms existing model reduction methods for MIMO systems, even with a small data set. Moreover, the applicability of CoFML extends to any scenario involving complex sparse Bayesian methods where covariance computation is required. This versatility opens up opportunities for its use in various fields such as compressed sensing, direction of arrival (DOA) estimation, multi-target tracking, and beyond.