Sparse DOD/DOA Estimation in a Bistatic MIMO Radar With Mutual Coupling Effect

Abstract

The unknown mutual coupling effect between antennas significantly degrades the target localization performance in the bistatic multiple-input multiple-output (MIMO) radar. In this paper, the joint estimation problem for the direction of departure (DOD) and direction of arrival (DOA) is addressed. By exploiting the target sparsity in the spatial domain and formulating a dictionary matrix with discretizing the DOD/DOA into grids, compressed sensing (CS)-based system model is given. However, in the practical MIMO radar systems, the target cannot be precisely on the grids, and the unknown mutual coupling effect degrades the estimation performance. Therefore, a novel CS-based DOD/DOA estimation model with both the off-grid and mutual coupling effect is proposed, and a novel sparse reconstruction method is proposed to estimate DOD/DOA with updating both the off-grid and mutual coupling parameters iteratively. Moreover, to describe the estimation performance, the corresponding Cramér–Rao lower bounds (CRLBs) with all the unknown parameters are theoretically derived. Simulation results show that the proposed method can improve the DOD/DOA estimation in the scenario with unknown mutual coupling effect, and outperform state-of-the-art methods.

1. Introduction

In multiple-input multiple-output (MIMO) radar systems [1,2], the independent waveforms are adopted in different transmitting antennas, so compared with the traditional array radars, the better performance of target estimation and detection can be achieved by using more spatial and waveform diversities [3,4,5]. Usually, the MIMO radar systems can be categorized into the following two types with different antenna distances: (1) Colocated MIMO radar system: The antennas in receiver and transmitter are close to each other, so the waveform diversity can be exploited to improve the radar performance [1,6,7]; (2) Distributed MIMO radar system: The antennas in receiver and transmitter are widely separated, so the radar performance can be improved by exploiting the diversity of target’s radar cross-section (RCS) [2,8].

Moreover, with different positions of transmitter and receiver, the colocated MIMO radar can also be categorized into monostatic and bistatic MIMO radar systems. The transmitter and receiver are close in the monostatic MIMO radar system [9], so more reliable beam-pattern design and target detection can be achieved. However, in the bistatic MIMO radar [9,10], the transmitter and receiver are widely separated, so the better performance of target localization can be achieved with the different view angles from transmitter and receiver. Therefore, in this paper, we consider the problem of the direction of departure (DOD) and the direction of arrival (DOA) estimation, and the bistatic MIMO radar system is adopted.

The DOD/DOA estimation problem in the MIMO radar system has been widely studied. For example, in the scenario with a non-uniform array, a novel method is proposed to construct a virtual MIMO array and estimate DOD/DOA in [11]. In [12,13], the DOD/DOA estimation method for the scenario with unknown correlated noise has been proposed, where the estimation method is based on the canonical correlation decomposition and the shift-invariance properties of Kronecker product. Additionally, some studies  [14,15,16,17] have proposed algorithms based on the multiple signal classification (MUSIC) and the estimation of signal parameters via rotational invariance techniques (ESPRIT) to estimate DOD/DOA in MIMO radar systems. However, these studies have not considered the mutual coupling between antennas in transmitter and receiver. In [18], the mutual coupling has been studied in the problem of direction finding. Additionally, the DOD/DOA estimation method with unknown mutual coupling is proposed in [19]. Different from these present papers, we propose a novel method to estimate the DOD/DOA in the bistatic MIMO radar system, where the sparsity of targets has been exploited to improve the estimation performance.

In this paper, we consider the problem of estimating the DOD/DOA in the bistatic MIMO radar system with mutual coupling between antennas. A novel iterative method based on compressed sensing (CS) is proposed to estimate the parameters including DOD/DOA, mutual coupling matrices, and target scattering coefficients, by exploiting the sparsity of targets in the spatial domain. Additionally, to further improve the estimation performance, an off-grid problem is formulated, and the parameters are polished iteratively by solving the off-grid problem. Furthermore, the corresponding Cramér–Rao lower bounds (CRLBs) for the estimated target parameters are derived theoretically. To summarize, we make the contributions as follows:

• Sparse DOD/DOA estimation model with mutual coupling effect: In the bistatic MIMO radar system, the DOD/DOA estimation model is proposed based on the sparse reconstruction model, and the unknown mutual coupling effect between antennas is also considered.
• Sparse DOD/DOA estimation method with off-grid effect: In the sparse reconstruction methods, the detection area is discretized into grids to formulated the dictionary matrix, so the off-grid effect limits the reconstruction performance. Therefore, combining both off-grid effect and mutual coupling effect, the sparse DOD/DOA estimation method is proposed.
• Theoretical CRLB expression for DOD/DOA estimation with mutual coupling effect: The corresponding CRLB with the unknown mutual coupling effect is theoretically derived to describe the estimation performance.

The remainder of this paper is organized as follows. The system model of bistatic MIMO radar is given in Section 2. The estimation method for DOD/DOA and mutual coupling matrices is proposed in Section 3. Section 4 derives the Cramér–Rao lower bound (CRLB). Then, Section 5 gives the computational complexity. Simulation results are given in Section 6. Finally, Section 7 concludes the paper.

Notations:${\mathit{I}}_N$ denotes an $N\times N$ identity matrix. $\mathcal{E}\left\{·\right\}$ denotes the expectation operation. $\mathcal{CN}\left(\mathit{a},\mathit{B}\right)$ denotes the complex Gaussian distribution with the mean being $\mathit{a}$ and the variance matrix being $\mathit{B}$. ${\parallel ·\parallel }_1$, ${\parallel ·\parallel }_2$, ⊗, $Tr\left\{·\right\}$, $vec\left\{·\right\}$, ${(·)}^{*}$, ${(·)}^T$ and ${(·)}^H$ denote the ${\ell }_1$ norm, the ${\ell }_2$ norm, the Kronecker product, the trace of a matrix, the vectorization of a matrix, the conjugate, the matrix transpose and the Hermitian transpose, respectively. For a matrix $\mathit{A}$, ${\left[\mathit{A}\right]}_n$ denotes the n-th column of $\mathit{A}$, and for a vector $\mathit{a}$, ${\left[\mathit{a}\right]}_n$ denotes the n-th entry of $\mathit{a}$.

2. The System Model of Bistatic MIMO Radar

2.1. Bistatic MIMO Radar System without Mutual Coupling

In this paper, the bistatic MIMO radar system [12,20,21] is considered and the radar system is shown in Figure 1, where M transmitting antennas and N receiving antennas are adopted. In each transmitting antenna, the orthogonal signal is transmitted. The transmitted waveform for the m-th transmitting antenna is denoted as $s_m(t,p)$ ($m=0,1,\dots ,M-1$) in the time domain, where p denotes the pulse index ($p=0,1,\dots ,P-1$) and the number of pulses is P. Therefore, for the transmitted waveforms, we have

$$\begin{array}{c}\hfill {\int }_{t\in T_p}{s}_m(t,p)s_{m^{\prime }}^H(t,p)dt=\left\{\begin{array}{cc}0,\hfill & m\ne m^{\prime }\hfill \\ 1,\hfill & m=m^{\prime }\hfill \end{array}\right.,\end{array}$$

(1)

where $T_p$ denotes the pulse duration.

Assuming that there are K far-field targets, the direction of departure (DOD) and the direction of arrival (DOA) for the k-th target ($k=0,1,\dots ,K-1$) are denoted as ${\varphi }_k$ and ${\psi }_k$, respectively. In each target, we assume that the the scattering coefficient is a type of Swerling II RCS [22] and follows the independent and identically distribution (i.i.d.) between pulses. Therefore, during the p-th pulse, the scattering coefficient of the k-th target can be denoted as ${\alpha }_k\left(p\right)$.

Without considering the mutual coupling between antennas, the received signals in the n-th antenna ($n=0,1,\dots ,N-1$) can be expressed as

$$\begin{array}{cc}\hfill r_n(t,p)=& \sum _{k=0}^{K-1}\sum _{m=0}^{M-1}{\alpha }_k\left(p\right)s_m(t,p)e^{-j2\pi \frac{d}{\lambda }\left(n sin {\psi }_k+m sin {\varphi }_k\right)}+w_n(t,p),\hfill \end{array}$$

(2)

where d denotes the fundamental antenna spacing, $\lambda$ denotes the wavelength, and $w_n(t,p)$ denotes the additive white Gaussian noise (AWGN) in the n-th receiving antenna during the p-th pulse, and $w_n(t,p)\sim \mathcal{CN}(0,{\sigma }_n^2)$.

After the matched filter $h_m(t,p)\triangleq s_m^{*}(t_0-t,p)$ for the m-th transmitted waveform and sampling at time $t_0$, we can obtain the result of pulse compression

$$\begin{array}{cc}{r}_{n,m}\left(p\right)\hfill & \triangleq {\int }_{t\in T_p}{r}_n(t,p)s_m^{*}(t,p)dt\hfill \\ & =\sum _{k=0}^{K-1}{\alpha }_k\left(p\right)e^{-j2\pi \frac{d}{\lambda }\left(n sin {\psi }_k+m sin {\varphi }_k\right)}+w_{n,m}\left(p\right),\hfill \end{array}$$

(3)

where we define $w_{n,m}\left(p\right)\triangleq {\int }_{t\in T_p}{w}_n(t,p)s^{*}(t,p)dt$, and $w_{n,m}\left(p\right)\sim \mathcal{CN}(0,{\sigma }_n^2)$. By collecting $r_{n,m}\left(p\right)$ into a vector ${\mathit{r}}_n\left(p\right)\triangleq {\left[r_{n,0}\left(p\right),r_{n,1}\left(p\right),\dots ,r_{n,M-1}\left(p\right)\right]}^T$, the vector form of received signal can be obtained as

$$\begin{array}{cc}\hfill {\mathit{r}}_n\left(p\right)& =\sum _{k=0}^{K-1}{e}^{-j2\pi \frac{nd}{\lambda }sin {\psi }_k}{\alpha }_k\left(p\right)\mathit{a}\left({\varphi }_k\right)+{\mathit{w}}_n\left(p\right),\hfill \end{array}$$

(4)

where the noise vector is defined as ${\mathit{w}}_n\left(p\right)\triangleq {\left[w_{n,0}\left(p\right),r_{n,1}\left(p\right),\dots ,w_{n,M-1}\left(p\right)\right]}^T$, and the steering vector in the transmitter is defined as

$$\begin{array}{c}\hfill \mathit{a}\left({\varphi }_k\right)\triangleq {\left[\begin{array}{c}1,e^{-j2\pi \frac{d}{\lambda }sin {\varphi }_k},\dots ,e^{-j2\pi \frac{(M-1)d}{\lambda }sin {\varphi }_k}\end{array}\right]}^T.\end{array}$$

(5)

Collect all the received signals into a matrix, and we can obtain

$$\begin{array}{cc}\hfill \mathit{R}\left(p\right)& \triangleq \left[\begin{array}{c}{\mathit{r}}_0\left(p\right),{\mathit{r}}_1\left(p\right),\dots ,{\mathit{r}}_{N-1}\left(p\right)\end{array}\right]=\sum _{k=0}^{K-1}{\alpha }_k\left(p\right)\mathit{a}\left({\varphi }_k\right){\mathit{b}}^T\left({\psi }_k\right)+\mathit{W}\left(p\right),\hfill \end{array}$$

(6)

where the steering vector in the receiver is defined as

$$\begin{array}{c}\hfill \mathit{b}\left({\psi }_k\right)\triangleq {\left[\begin{array}{c}1,e^{-j2\pi \frac{d}{\lambda }sin {\psi }_k},\dots ,e^{-j2\pi \frac{(N-1)d}{\lambda }sin {\varphi }_k}\end{array}\right]}^T,\end{array}$$

(7)

and the noise matrix is defined as $\mathit{W}\left(p\right)\triangleq \left[\begin{array}{c}{\mathit{w}}_0\left(p\right),{\mathit{w}}_1\left(p\right),\dots ,{\mathit{w}}_{N-1}\left(p\right)\end{array}\right]$.

Vectorizing the matrix of received signals into a vector $\mathit{r}\left(p\right)\triangleq vec\left\{\mathit{R}\left(p\right)\right\}$, the received signals can be expressed as the following vector form

$$\begin{array}{cc}\hfill \mathit{r}\left(p\right)& =\sum _{k=0}^{K-1}{\alpha }_k\left(p\right)vec\left\{\mathit{a}\left({\varphi }_k\right){\mathit{b}}^T\left({\psi }_k\right)\right\}+\mathit{w}\left(p\right)=\sum _{k=0}^{K-1}{\alpha }_k\left(p\right)\mathit{b}\left({\psi }_k\right)\otimes \mathit{a}\left({\varphi }_k\right)+\mathit{w}\left(p\right)\hfill \end{array}$$

(8)

where $\mathit{w}\left(p\right)\triangleq vec\left\{\mathit{W}\left(p\right)\right\}$, and $\mathit{w}\left(p\right)\sim \mathcal{CN}\left(\mathbf{0},{\sigma }_n^2{\mathit{I}}_{MN}\right)$. Therefore, without the mutual coupling effect between antennas, the problem of DOD/DOA estimation is formulated in (8), where both ${\varphi }_k$ and ${\psi }_k$ will be estimated from the received signal $\mathit{r}\left(p\right)$ without the knowledge of target scattering coefficient ${\alpha }_k\left(p\right)$.

2.2. Bistatic MIMO Radar System With Mutual Coupling

However, in the practical radar system, when the mutual coupling between the antennas in both transmitter and receiver is considered [23], the system model developed in (8) cannot be used. Therefore, this subsection will discuss the system model with mutual coupling. Usually, the mutual coupling matrices in the transmitter and receiver are respectively defined as [18]

$$\begin{array}{c}{\mathbf{C}}_T\triangleq \left(Z_{TA}+Z_{TL}\right){\left({\mathit{Z}}_T+Z_{TL}\mathit{I}\right)}^{-1},\hfill \end{array}$$

(9)

$$\begin{array}{c}{\mathbf{C}}_R\triangleq \left(Z_{RA}+Z_{RL}\right){\left({\mathit{Z}}_R+Z_{RL}\mathit{I}\right)}^{-1},\hfill \end{array}$$

(10)

where $Z_{TA}$ and $Z_{TL}$ denote the antenna impedance and terminating load in transmitter, and $Z_{RA}$ and $Z_{RL}$ denote the antenna impedance and terminating load in receiver. ${\mathit{Z}}_T$ and ${\mathit{Z}}_R$ denote the mutual impedance matrix in transmitter and receiver, respectively.

The $m_1$-th row and $m_2$-th column of mutual impedance matrix ${\mathit{Z}}_T$ can be expressed as [19,24,25]

$$\begin{array}{c}\hfill Z_{T,m_1,m_2}=\left\{\begin{array}{cc}30(0.5772+ln\left(2\gamma L\right)-g_C\left(2\gamma L\right)+j{g}_S\left(2\gamma L\right)),\hfill & m_1=m_2\hfill \\ 30(g_R(m_1,m_2)+j{g}_X(m_1,m_2)),\hfill & m_1\ne m_2\hfill \end{array}\right.\end{array}$$

(11)

where $\gamma \triangleq 2\pi /\lambda$, and L denotes the length of dipole antennas. $g_R(m_1,m_2)$ and $g_X(m_1,m_2)$ are defined respectively as

$$\begin{array}{cc}\hfill g_R(m_1,m_2)& \triangleq sin\left(\gamma L\right)[g_S\left({\nu }_0\right)-g_S\left({\mu }_0\right)+2g_S\left({\mu }_1\right)\hfill \\ & -2g_S\left({\nu }_1\right)]+cos\left(\gamma L\right)[g_C\left({\mu }_0\right)+g_C\left({\nu }_0\right)-2g_C\left({\mu }_1\right)\hfill \\ & -2g_C\left({\nu }_1\right)+2g_C\left(\gamma d(m_1,m_2)\right)]-[2g_C\left({\mu }_1\right)+2g_C\left({\nu }_1\right)\hfill \\ & -4g_C\left(\gamma d(m_1,m_2)\right)],\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill g_X(m_1,m_2)& \triangleq sin\left(\gamma L\right)[g_C\left({\nu }_0\right)-g_C\left({\mu }_0\right)+2g_C\left({\mu }_1\right)\hfill \\ & -2g_C\left({\nu }_1\right)]+cos\left(\gamma L\right)[-g_S\left({\mu }_0\right)-g_S\left({\nu }_0\right)+2g_S\left({\mu }_1\right)\hfill \\ & +2g_S\left({\nu }_1\right)-2g_S\left(\gamma d(m_1,m_2)\right)]+[2g_S\left({\mu }_1\right)+2g_S\left({\nu }_1\right)\hfill \\ & -4g_S\left(\gamma d(m_1,m_2)\right)],\hfill \end{array}$$

(13)

where $d(m_1,m_2)$ denotes the distance between the $m_1$-th antenna and the $m_2$-th antenna. ${\mu }_0$, ${\nu }_0$, ${\mu }_1$ and ${\nu }_1$ are defined respectively as

$$\begin{array}{c}{\mu }_0=\gamma \left(\sqrt{d^2(m_1,m_2)+L^2}-L\right),\hfill \end{array}$$

(14)

$$\begin{array}{c}{\nu }_0=\gamma \left(\sqrt{d^2(m_1,m_2)+L^2}+L\right),\hfill \end{array}$$

(15)

$$\begin{array}{c}{\mu }_1=\gamma \left(\sqrt{d^2(m_1,m_2)+0.25L^2}-0.5L\right),\hfill \end{array}$$

(16)

$$\begin{array}{c}{\nu }_1=\gamma \left(\sqrt{d^2(m_1,m_2)+0.25L^2}+0.5L\right).\hfill \end{array}$$

(17)

$g_C\left(x\right)$ and $g_S\left(x\right)$ are defined respectively as

$$\begin{array}{c}\hfill g_C\left(x\right)\triangleq {\int }_{-\infty }^x\frac{cos\left(t\right)}{t}dt,g_S\left(x\right)\triangleq {\int }_0^x\frac{sin\left(t\right)}{t}dt.\end{array}$$

(18)

Similarly, the mutual impedance matrix ${\mathit{Z}}_R$ can be also obtained from the expression of ${\mathit{Z}}_T$.

However, the expresses for ${\mathit{Z}}_T$ and ${\mathit{Z}}_R$ in (11) are too complex to analysis. Since ${\mathit{Z}}_T$ and ${\mathit{Z}}_R$ depend on the length of dipole antennas and the distances between antennas, the mutual coupling matrices ${\mathit{C}}_T$ and ${\mathit{C}}_R$ can be approximated, respectively, by two symmetric Toeplitz matrices

$$\begin{array}{c}\hfill {\mathit{C}}_T\approx T\left({\mathit{c}}_T\right),{\mathit{C}}_R\approx T\left({\mathit{c}}_R\right),\end{array}$$

(19)

where $T\left({\mathit{c}}_T\right)\in {\mathbb{C}}^{M\times M}$ is defined as

$$\begin{array}{c}\hfill T\left({\mathit{c}}_M\right)\triangleq \left[\begin{array}{ccccc}{c}_{T,0}& c_{T,1}& c_{T,2}& \dots & c_{T,M-1}\\ c_{T,1}& c_{T,0}& c_{T,1}& \dots & c_{T,M-2}\\ c_{T,2}& c_{T,1}& c_{T,0}& \dots & c_{T,M-3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ c_{T,M-1}& c_{T,M-2}& c_{T,M-3}& \dots & c_{T,0}\end{array}\right],\end{array}$$

(20)

and $T\left({\mathit{c}}_R\right)\in {\mathbb{C}}^{N\times N}$ is defined as

$$\begin{array}{c}\hfill T\left({\mathit{c}}_R\right)\triangleq \left[\begin{array}{ccccc}{c}_{R,0}& c_{R,1}& c_{R,2}& \dots & c_{R,N-1}\\ c_{R,1}& c_{R,0}& c_{R,1}& \dots & c_{R,N-2}\\ c_{R,2}& c_{R,1}& c_{R,0}& \dots & c_{R,N-3}\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ c_{R,N-1}& c_{R,N-2}& c_{R,N-3}& \dots & c_{R,0}\end{array}\right].\end{array}$$

(21)

Additionally, for the mutual coupling matrices, we also have

$$\begin{array}{c}1=|c_{T,0}|\ge |c_{T,1}|\ge \dots \ge |c_{T,M-1}|,\hfill \end{array}$$

(22)

$$\begin{array}{c}1=|c_{R,0}|\ge |c_{R,1}|\ge \dots \ge |c_{R,N-1}|.\hfill \end{array}$$

(23)

Therefore, in the scenario with mutual coupling between antennas, the received signal in (8) can be rewritten as

$$\begin{array}{cc}\mathit{r}\left(p\right)\hfill & =\sum _{k=0}^{K-1}{\alpha }_k\left(p\right)\left[{\mathit{C}}_R\mathit{b}\left({\psi }_k\right)\right]\otimes \left[{\mathit{C}}_T\mathit{a}\left({\varphi }_k\right)\right]+\mathit{w}\left(p\right)\hfill \\ & =\mathit{C}\mathit{A}\alpha \left(p\right)+\mathit{w}\left(p\right),\hfill \end{array}$$

(24)

where

$$\begin{array}{cc}\hfill \mathit{C}& \triangleq {\mathit{C}}_R\otimes {\mathit{C}}_T,\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill \alpha \left(p\right)& \triangleq {\left[\begin{array}{c}{\alpha }_0\left(p\right),{\alpha }_1\left(p\right),\dots ,{\alpha }_{K-1}\left(p\right)\end{array}\right]}^T,\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill \mathit{A}& \triangleq \left[\begin{array}{c}\mathit{b}\left({\psi }_0\right)\otimes \mathit{a}\left({\varphi }_0\right),\dots ,\mathit{b}\left({\psi }_{K-1}\right)\otimes \mathit{a}\left({\varphi }_{K-1}\right)\end{array}\right].\hfill \end{array}$$

(27)

The orthogonal signals are affected by the mutual coupling effect, but we describe the corresponding effect by a matrix, and the non-orthogonality is transferred into the steering vectors by the mutual coupling matrix.

Finally, collect the received signals from all pulses, and the matrix form of all received signals can be obtained as

$$\begin{array}{cc}\hfill \mathit{R}& \triangleq \left[\begin{array}{c}\mathit{r}\left(0\right),\mathit{r}\left(1\right),\dots ,\mathit{r}(P-1)\end{array}\right]=\mathit{CA}\mathbf{\Gamma }+\mathit{W},\hfill \end{array}$$

(28)

where $\mathit{W}\triangleq \left[\begin{array}{c}\mathit{w}\left(0\right),\mathit{w}\left(1\right),\dots ,\mathit{w}(P-1)\end{array}\right]$, $\mathbf{\Gamma }\triangleq \left[\begin{array}{c}\mathit{\alpha }\left(0\right),\mathit{\alpha }\left(1\right),\dots ,\mathit{\alpha }(P-1)\end{array}\right]$. Then, the vector form of all received signals can be expressed as

$$\begin{array}{cc}\hfill \mathit{r}& \triangleq {\left[\begin{array}{c}{\mathit{r}}^T\left(0\right),{\mathit{r}}^T\left(1\right),\dots ,{\mathit{r}}^T(P-1)\end{array}\right]}^T=\left({\mathit{I}}_P\otimes \mathit{C}\right)\left({\mathit{I}}_P\otimes \mathit{A}\right)\mathit{\alpha }+\mathit{w},\hfill \end{array}$$

(29)

where $\mathit{w}\triangleq {\left[\begin{array}{c}{\mathit{w}}^T\left(0\right),{\mathit{w}}^T\left(1\right),\dots ,{\mathit{w}}^T(P-1)\end{array}\right]}^T$, $\mathit{\alpha }\triangleq vec\left\{\mathbf{\Gamma }\right\}$.

Therefore, considering the mutual coupling between antennas in both receiver and transmitter, we will develop an algorithm to estimate the DOD/DOA in $\mathit{A}$ from the received signal $\mathit{r}$ in (29) without the knowledge of mutual coupling matrix $\mathit{C}$ and the scattering coefficient $\mathit{\alpha }$.

3. DOA/DOD and Mutual Coupling Matrix Estimation

With the received signal $\mathit{R}$, we propose a novel sparse-based method to estimate the DOD/DOA in the scenario with unknown mutual coupling matrix. The possible DOD and DOA are, respectively, from the following two discretized sets

$$\begin{array}{cc}\hfill {\mathbb{S}}_{\varphi }& \triangleq \left\{{\varphi }_{D,z_1}|z_1=0,1,\dots ,Z_1-1\right\},\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\mathbb{S}}_{\psi }& \triangleq \left\{{\psi }_{D,z_2}|z_2=0,1,\dots ,Z_2-1\right\},\hfill \end{array}$$

(31)

where ${\varphi }_{D,z_1}\le {\varphi }_{D,z_1+1}$ and ${\psi }_{D,z_2}\le {\psi }_{D,z_2+1}$.

Therefore, assume that the DOD and DOA of a target are respectively the $z_1$-th entry of ${\mathbb{S}}_{\varphi }$, i.e., ${\varphi }_{D,z_1}$, and the $z_2$-th entry of ${\mathbb{S}}_{\psi }$, i.e., ${\psi }_{D,z_2}$, so the steering vector for this target can be expressed as

$$\begin{array}{c}\hfill {\mathit{d}}_{z_1,z_2}=\mathit{b}\left({\psi }_{D,z_2}\right)\otimes \mathit{a}\left({\varphi }_{D,z_1}\right).\end{array}$$

(32)

Then, collecting the steering vectors for all the possible targets, a dictionary matrix can be formulated as

$$\begin{array}{c}\hfill \mathit{D}\triangleq \left[\begin{array}{c}{\mathit{d}}_{0,0},{\mathit{d}}_{0,1},\dots ,{\mathit{d}}_{0,Z_2-1},{\mathit{d}}_{1,0},\dots ,{\mathit{d}}_{Z_1-1,Z_2-1}\end{array}\right].\end{array}$$

(33)

Consequently, we can formulate the following compressed sensing (CS)-based problem [26,27] for the DOD/DOA estimation

$$\begin{array}{cc}\hfill \underset{\mathit{X}}{min}& {\parallel \mathit{X}\parallel }_{2,0}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {∥\mathit{R}-\mathit{CDX}∥}_F^2\le ϵ,\hfill \end{array}$$

(34)

where the norm ${\parallel \mathit{X}\parallel }_{2,0}$ denotes the number of rows in $\mathit{X}$ with the nonzero entries, and the parameter $ϵ$ is adopted to control the accuracy of sparse reconstruction. As shown in Figure 2, $\mathit{X}\in {\mathbb{C}}^{Q\times P}$ denotes a sparse matrix and the nonzero entries are the scattering coefficients from $\mathbf{\Gamma }$. The indexes of nonzero rows in $\mathit{X}$ indicate the DOD/DOA of targets.

In (34), both the sparse matrix $\mathit{X}$ and the mutual coupling matrix $\mathit{C}$ are unknown, so this paper proposes a novel method to estimate DOD/DOA with the unknown mutual coupling matrix. Additionally, the off-grid problem in DOD/DOA estimation is also considered, where the off-grid problem means that the actual values of DOD/DOA can be not exactly contained by the discretized DOD and DOA sets, i.e., ${\varphi }_k\notin {\mathbb{S}}_{\varphi }$ and ${\psi }_k\notin {\mathbb{S}}_{\psi }$, but ${\varphi }_{D,0}\le {\varphi }_k\le {\varphi }_{D,Z_1-1}$ and ${\psi }_{D,0}\le {\psi }_k\le {\psi }_{D,Z_2-1}$, for $k=0,1,\dots ,K-1$.

Unlike the traditional multiple measurement vectors (MMV) problem [28,29,30] in the CS theory, the mutual coupling matrix in the DOA/DOD estimation problem (34) is unknown, so the traditional MMV methods cannot be used directly. Therefore, a novel method is proposed to estimate DOD/DOA with the following objective function

$$\begin{array}{c}\hfill \left\{\widehat{\mathit{\varphi }},\widehat{\mathit{\psi }},\widehat{\mathit{X}},\widehat{\mathit{C}}\right\}=\underset{\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C}}{min}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C}),\end{array}$$

(35)

where ${\mathit{X}}^q$ denotes the q-th row of $\mathit{X}$, and

$$\begin{array}{c}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})\triangleq \mu {∥\mathit{X}∥}_{2,1}+{\parallel \mathit{R}-\mathit{CDX}\parallel }_F^2,\hfill \end{array}$$

(36)

$$\begin{array}{c}{\parallel \mathit{X}\parallel }_F\triangleq \sqrt{\sum _{p=0}^{P-1}\sum _{q=0}^{Q-1}{X}_{q,p}^2},\hfill \end{array}$$

(37)

$$\begin{array}{c}{\parallel \mathit{X}\parallel }_{2,1}\triangleq \sum _{q=0}^{Q-1}\sqrt{\sum _{p=0}^{P-1}{X}_{q,p}^2}=\sum _{q=0}^{Q-1}{\parallel {\mathit{X}}^q\parallel }_2.\hfill \end{array}$$

(38)

In (35), the ${\ell }_1$ norm is adopted as a relaxation form of ${\ell }_0$ norm [31].

A novel iterative method is proposed to solve the problem (35), and the flow chart of the proposed method is shown in Figure 3. First, ignoring the effect of mutual coupling, the CS-based method is adopted to estimate the sparse matrix $\widehat{\mathit{X}}$ with assuming $\mathit{C}=\mathit{I}$. Second, based on the estimated $\widehat{\mathit{X}}$, the mutual coupling matrix $\mathit{C}$ can be estimated as $\widehat{\mathit{C}}$ with the gradient descent method. Then, with the roughly estimated results, another gradient descent method is proposed to further polish the estimated results and solve the off-grid problem. Finally, Estimate DOD/DOA and mutual coupling matrix iteratively, and the estimated results are obtained when the estimation method is that of convergence. Details about the proposed method are given in the following subsections.

3.1. CS-Based DOD/DOA Estimation

The CS-based method is adopted to estimate DOD/DOA. Since the multiple pulses are adopted in the bistatic MIMO radar system, the simultaneous orthogonal matching pursuit (SOMP) method [32] can be adopted with ignoring the mutual coupling between antennas. In Algorithm 1, the details of the SOMP method for DOD/DOA estimation is given. At Step 4 of Algorithm 1, the ${\ell }_1$ norm is used to find the discretized DOD/DOA with the maximum correlation coefficient for all pulses.

Algorithm 1 Simultaneous orthogonal matching pursuit for DOD/DOA estimation

1: Input: received signal $\mathit{R}$, maximum number of targets K, dictionary matrix $\mathit{D}$. 2: Initialization: iteration $i=0$, support ${\mathbb{S}}_x=\varnothing$, residual matrix ${\mathit{Z}}_i=\mathit{R}$, assuming $\mathit{C}=\mathit{I}$. 3: for$i=0$ to $K-1$ do 4:  $\{{\widehat{z}}_1,{\widehat{z}}_2\}=arg{max}_{z_1,z_2}.{\parallel {\mathit{Z}}_i^H{\mathit{d}}_{z_1,z_2}\parallel }_1$. 5:  $j=z_1{Z}_1+z_2$. 6:  ${\mathbb{S}}_x\leftarrow {\mathbb{S}}_x\cup \left\{j\right\}$. 7:  ${\mathit{R}}_{i+1}={\mathit{D}}_{{\mathbb{S}}_x}{\mathit{D}}_{{\mathbb{S}}_x}^{†}\mathit{R}$, where ${\mathit{D}}_{{\mathbb{S}}_x}$ is formed by the columns from $\mathit{D}$ with the column indexes from the support ${\mathbb{S}}_x$. 8:  ${\mathit{Z}}_{i+1}=\mathit{R}-{\mathit{R}}_{i+1}$. 9:  $i=i+1$. 10: end for 11: Output: the estimated DOD/DOA from the support ${\mathbb{S}}_x$ and the estimated sparse matrix $\widehat{\mathit{X}}$.

3.2. Gradient Decent-Based Mutual Coupling Matrix Estimation

With the estimated DOD/DOA and the sparse matrix $\widehat{\mathit{X}}$, considering the symmetry characteristic of mutual coupling matrix, the mutual coupling vectors ${\mathit{c}}_T$ and ${\mathit{c}}_R$ can be estimated by the following objective function

$$\begin{array}{c}\hfill \left\{{\widehat{\mathit{c}}}_T,{\widehat{\mathit{c}}}_R\right\}=arg\underset{{\mathit{c}}_T,{\mathit{c}}_R}{min}g({\mathit{c}}_T,{\mathit{c}}_R),\end{array}$$

(39)

where the objective function is defined as

$$\begin{array}{c}\hfill g({\mathit{c}}_T,{\mathit{c}}_R)\triangleq {∥\mathit{R}-\left[T\left({\mathit{c}}_R\right)\otimes T\left({\mathit{c}}_T\right)\right]\mathit{D}\widehat{\mathit{X}}∥}_F^2.\end{array}$$

(40)

Therefore, a gradient decent method is proposed in this paper to estimate the mutual coupling vectors ${\mathit{c}}_T$ and ${\mathit{c}}_R$, and the details are given in Algorithm 2.

Here, the subgradients of objective function $g({\mathit{c}}_T,{\mathit{c}}_R)$ can be obtained as

$$\begin{array}{ccc}\hfill {\nabla }_{{\mathit{c}}_T^{*}}g({\mathit{c}}_T,{\mathit{c}}_R)& =\frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial {\mathit{c}}_T^{*}}\hfill & =\left[\begin{array}{c}\frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{T,0}^{*}},\dots ,\frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{T,m}^{*}},\dots \end{array}\right],\end{array}$$

(41)

$$\begin{array}{ccc}\hfill {\nabla }_{{\mathit{c}}_R^{*}}g({\mathit{c}}_T,{\mathit{c}}_R)& =\frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial {\mathit{c}}_R^{*}}\hfill & =\left[\begin{array}{c}\frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{R,0}^{*}},\dots ,\frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{R,n}^{*}},\dots \end{array}\right],\end{array}$$

(42)

where the subgradients of ${\displaystyle \frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{T,m}^{*}}}$ and ${\displaystyle \frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{R,n}^{*}}}$ are given respectively as

$$\begin{array}{cc}\hfill \frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{T,m}^{*}}& =2\mathcal{R}\left\{{vec}^T\left\{\mathit{R}-\mathit{CD}\widehat{\mathit{X}}\right\}{vec}^{*}\left\{-\left[T\left({\mathit{c}}_R\right)\otimes \frac{\partial T\left({\mathit{c}}_T\right)}{\partial c_{T,m}^{*}}\right]\mathit{D}\widehat{\mathit{X}}\right\}\right\},\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill \frac{\partial g({\mathit{c}}_T,{\mathit{c}}_R)}{\partial c_{R,n}^{*}}& =2\mathcal{R}\left\{{vec}^T\left\{\mathit{R}-\mathit{CD}\widehat{\mathit{X}}\right\}{vec}^{*}\left\{-\left[\frac{\partial T\left({\mathit{c}}_R\right)}{\partial c_{R,n}^{*}}\otimes T\left({\mathit{c}}_T\right)\right]\mathit{D}\widehat{\mathit{X}}\right\}\right\}.\hfill \end{array}$$

(44)

Algorithm 2 Mutual coupling matrix estimation

1: Input: received signal $\mathit{R}$, estimated sparse matrix $\widehat{\mathit{X}}$, dictionary matrix $\mathit{D}$, step size $\delta$, stop threshold ${ϵ}_S$. 2: Initialization:$$\begin{gathered} {\widehat{\mathit{c}}}_T={\left[1,{\mathbf{0}}_{1\times (M-1)}\right]}^T, \\ {\widehat{\mathit{c}}}_R={\left[1,{\mathbf{0}}_{1\times (N-1)}\right]}^T, \\ \widehat{\mathit{C}}=T\left({\widehat{\mathit{c}}}_R\right)\otimes T\left({\widehat{\mathit{c}}}_T\right), \\ e=\parallel \mathit{R}-\widehat{\mathit{C}}\mathit{D}\widehat{\mathit{X}}{\parallel }_F^2. \end{gathered}$$ 3: while$e\le {ϵ}_S$do 4:  Obtain ${\nabla }_{{\mathit{c}}_R^{*}}g({\mathit{c}}_T,{\mathit{c}}_R)$ and ${\nabla }_{{\mathit{c}}_R^{*}}g({\mathit{c}}_T,{\mathit{c}}_R)$ from (41) and (42), respectively. 5:  ${\widehat{\mathit{c}}}_T\leftarrow {\widehat{\mathit{c}}}_T-\delta {\nabla }_{{\mathit{c}}_T^{*}}g({\mathit{c}}_T,{\mathit{c}}_R)$. 6:  ${\widehat{\mathit{c}}}_R\leftarrow {\widehat{\mathit{c}}}_R-\delta {\nabla }_{{\mathit{c}}_R^{*}}g({\mathit{c}}_T,{\mathit{c}}_R)$. 7:  $\widehat{\mathit{C}}=T\left({\widehat{\mathit{c}}}_R\right)\otimes T\left({\widehat{\mathit{c}}}_T\right)$. 8:  $e^{\prime }={\parallel \mathit{R}-\widehat{\mathit{C}}\mathit{D}\widehat{\mathit{X}}\parallel }_F^2$. 9:  if $e^{\prime }>e$ then 10:   $\delta \leftarrow \frac{\delta }{2}$. 11:  end if 12:  $e=e^{\prime }$. 13: end while 14: Output: the estimated mutual coupling matrix $\widehat{\mathit{C}}$.

3.3. Polish the Estimated DOD/DOA and Mutual Coupling Matrix

The DOD/DOA are discretized and the dictionary matrix is formulated in Algorithm 1, so the estimated DOD/DOA must be in set ${\mathbb{S}}_{\varphi }$ and ${\mathbb{S}}_{\psi }$. However, in the practical scenarios, the DOD/DOA of targets are continuous and can be not exact in the sets with discretized angles. Therefore, with the roughly estimated DOD/DOA and mutual coupling matrix from Algorithms 1 and 2, this subsection proposes a gradient descent method to further polish the estimated results and solve the off-grid problem. The details to polish the estimation results is given in Algorithm 3. The mutual coupling effect is compensated in Algorithm 3, where we estimate the mutual coupling coefficients. Then, the estimated coefficients can be used to improve the performance of DOD/DOA estimation.

Algorithm 3 Polish the estimated DOD/DOA and mutual coupling matrix

1: Input: received signal $\mathit{R}$, estimated sparse matrix $\widehat{\mathit{X}}$, estimated mutual coupling matrxi $\widehat{\mathit{C}}$, dictionary matrix $\mathit{D}$, step size $\delta$, stop threshold ${ϵ}_S$. 2: Initialization: obtain ${\widehat{\mathit{c}}}_T$ and ${\widehat{\mathit{c}}}_R$ from $\widehat{\mathit{C}}$; obtain $\widehat{\mathit{\varphi }}$ and $\widehat{\mathit{\psi }}$ from $\widehat{\mathit{X}}$; $\widehat{\mathit{x}}=vec\left\{\widehat{\mathit{X}}\right\}$; $e=\parallel \mathit{R}-\widehat{\mathit{C}}\mathit{D}\widehat{\mathit{X}}{\parallel }_F^2$. 3: while$e\le {ϵ}_S$do 4:  Obtain ${\nabla }_{\mathit{\varphi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$, ${\nabla }_{\mathit{\psi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$, ${\nabla }_{{\mathit{x}}^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$, ${\nabla }_{{\mathit{c}}_T^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$, and ${\nabla }_{{\mathit{c}}_R^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$. 5:  ${\widehat{\mathit{c}}}_T\leftarrow {\widehat{\mathit{c}}}_T-\delta {\nabla }_{{\mathit{c}}_T^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$. 6:  ${\widehat{\mathit{c}}}_R\leftarrow {\widehat{\mathit{c}}}_R-\delta {\nabla }_{{\mathit{c}}_R^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$. 7:  $\widehat{\mathit{C}}=T\left({\widehat{\mathit{c}}}_R\right)\otimes T\left({\widehat{\mathit{c}}}_T\right)$. 8:  $\widehat{\mathit{\varphi }}\leftarrow \widehat{\mathit{\varphi }}-\delta {\nabla }_{\mathit{\varphi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$. 9:  $\widehat{\mathit{\psi }}\leftarrow \widehat{\mathit{\varphi }}-\delta {\nabla }_{\mathit{\psi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$. 10:  $\widehat{\mathit{x}}\leftarrow \widehat{\mathit{x}}-\delta {\nabla }_{{\mathit{x}}^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$. 11:  $e^{\prime }={\parallel \mathit{R}-\widehat{\mathit{C}}\mathit{D}\widehat{\mathit{X}}\parallel }_F^2$. 12:  if $e^{\prime }>e$ then 13:   $\delta \leftarrow \frac{\delta }{2}$. 14:  end if 15:  $e=e^{\prime }$. 16: end while 17: Output: the polished mutual coupling matrix $\widehat{\mathit{C}}$, the polished DOD/DOA $\widehat{\mathit{\varphi }}$ and $\widehat{\mathit{\psi }}$, and the polished sparse matrix $\widehat{\mathit{X}}$.

The gradient descent method based on the subgradients of objective function $f(\mathit{\varphi },\mathit{\psi },\mathit{X}\mathit{C})$, which are given in Proposition 1.

Theorem 1.

The subgradients of$f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$are

$$\begin{array}{c}{\nabla }_{\mathit{\varphi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=2\mathcal{R}\left\{{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^H({\mathit{I}}_P\otimes \mathit{C})\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \mathit{\varphi }}\right\},\hfill \end{array}$$

(45)

$$\begin{array}{c}{\nabla }_{\mathit{\psi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=2\mathcal{R}\left\{{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^H({\mathit{I}}_P\otimes \mathit{C})\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \mathit{\psi }}\right\},\hfill \end{array}$$

(46)

$$\begin{array}{c}{\nabla }_{{\mathit{x}}^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=\frac{\mu }{2}{\mathit{x}}^T\sum _{q=0}^{K-1}{\left(\sum _{p=0}^{P-1}{x}_{q,p}^2\right)}^{-\frac{1}{2}}+{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^T{({\mathit{I}}_P\otimes \mathit{CD})}^{*},\hfill \end{array}$$

(47)

$$\begin{array}{c}{\nabla }_{{\mathit{c}}_T^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})={\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^T\frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_T^{*}},\hfill \end{array}$$

(48)

$$\begin{array}{c}{\nabla }_{{\mathit{c}}_R^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})={\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^T\frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_R^{*}},\hfill \end{array}$$

(49)

where

1. 

The k-th column of${\displaystyle \frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \varphi }}$is

$$\begin{array}{c}{\left[\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \varphi }\right]}_k=\left({\mathit{I}}_P\otimes \left[\begin{array}{c}{\mathbf{0}}_{MN\times k},\mathit{b}\left({\psi }_k\right)\otimes \frac{\partial \mathbf{a}\left({\varphi }_k\right)}{\partial {\varphi }_k},{\mathbf{0}}_{MN\times (K-1-k)}\end{array}\right]\right)\mathit{x};\hfill \end{array}$$

(50)

2. 

The k-th column of${\displaystyle \frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \psi }}$is

$$\begin{array}{c}\hfill {\left[\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \psi }\right]}_k=\left({\mathit{I}}_P\otimes \left[\begin{array}{c}{\mathbf{0}}_{MN\times k},\frac{\partial \mathit{b}\left({\psi }_k\right)}{\partial {\psi }_k}\otimes \mathit{a}\left({\varphi }_k\right),{\mathbf{0}}_{MN\times (K-1-k)}\end{array}\right]\right)\mathit{x};\end{array}$$

(51)

3. 

The m-th column of${\displaystyle \frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_T^{*}}}$is

$$\begin{array}{c}\hfill {\left[\frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_T^{*}}\right]}_m=\left({\mathit{I}}_P\otimes {\mathit{C}}_R^{*}\otimes \frac{\partial {\mathit{C}}_T^{*}}{\partial c_{T,m}^{*}}\right)\left({\mathit{I}}_P\otimes {\mathit{D}}^{*}\right){\mathit{x}}^{*};\end{array}$$

(52)

4. 

The n-th column of${\displaystyle \frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_R^{*}}}$is

$$\begin{array}{c}\hfill {\left[\frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_R^{*}}\right]}_n=\left({\mathit{I}}_P\otimes \frac{\partial {\mathit{C}}_R^{*}}{\partial c_{R,n}^{*}}\otimes {\mathit{C}}_T^{*}\right)\left({\mathit{I}}_P\otimes {\mathit{D}}^{*}\right){\mathit{x}}^{*};\end{array}$$

(53)

5. 

The m-th entry of${\displaystyle \frac{\partial \mathit{a}\left({\varphi }_k\right)}{\partial {\varphi }_k}}$is

$$\begin{array}{c}\hfill {\left[\frac{\partial \mathit{a}\left({\varphi }_k\right)}{\partial {\varphi }_k}\right]}_m=-j2\pi \frac{md}{\lambda }cos{\varphi }_k{e}^{-j2\pi {\displaystyle \frac{md}{\lambda }}sin {\varphi }_k};\end{array}$$

(54)

6. 

The n-th entry of${\displaystyle \frac{\partial \mathit{b}\left({\psi }_k\right)}{\partial {\psi }_k}}$is

$$\begin{array}{c}\hfill {\left[\frac{\partial \mathit{b}\left({\psi }_k\right)}{\partial {\psi }_k}\right]}_n=-j2\pi \frac{nd}{\lambda }cos{\psi }_k{e}^{-j2\pi {\displaystyle \frac{nd}{\lambda }}sin {\psi }_k}.\end{array}$$

(55)

Here, we will proof this proposition.

Proof. 

The derivations for vectors or matrices are given in Appendix A. By defining $\mathit{x}\triangleq vec\left\{\mathit{X}\right\}$, we can obtain

$$\begin{array}{c}{\nabla }_{\mathit{\varphi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=\frac{\partial \parallel \mathit{r}-({\mathit{I}}_P\otimes \mathit{CD}){\mathit{x}\parallel }_2^2}{\partial \mathit{\varphi }}\hfill \\ =-\frac{\partial {\mathit{r}}^H({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial \mathit{\varphi }}-\frac{\partial {\mathit{x}}^H({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)\mathit{r}}{\partial \mathit{\varphi }}+\frac{\partial {\mathit{x}}^H({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial \mathit{\varphi }}\hfill \\ =-{\mathit{r}}^H\frac{\partial ({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial \mathit{\varphi }}-{\mathit{r}}^T\frac{\partial {\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^{*}}{\partial \mathit{\varphi }}+{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^T\frac{\partial \left({\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^{*}\right)}{\partial \mathit{\varphi }}\hfill \\ +{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^H\frac{\partial \left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}{\partial \mathit{\varphi }}\hfill \\ =2\mathcal{R}\left\{{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^H({\mathit{I}}_P\otimes \mathit{C})\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \mathit{\varphi }}\right\}.\hfill \end{array}$$

(56)

The k-th column of $\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \mathit{\varphi }}$ can be obtained as

$$\begin{array}{c}{\left[\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})\mathit{x}}{\partial \mathit{\varphi }}\right]}_k=\frac{\partial ({\mathit{I}}_P\otimes \mathit{D})}{\partial {\varphi }_k}\mathit{x}=\left({\mathit{I}}_P\otimes \frac{\partial \mathit{D}}{\partial {\varphi }_k}\right)\mathit{x}\hfill \\ =\left({\mathit{I}}_P\otimes \left[\begin{array}{c}{\mathbf{0}}_{MN\times k},\mathit{b}\left({\psi }_k\right)\otimes \frac{\partial \mathit{a}\left({\varphi }_k\right)}{\partial {\varphi }_k},{\mathbf{0}}_{MN\times (K-1-k)}\end{array}\right]\right)\mathit{x},\hfill \end{array}$$

(57)

where the m-th entry of ${\displaystyle \frac{\partial \mathit{a}\left({\varphi }_k\right)}{\partial {\varphi }_k}}$ is

$$\begin{array}{c}\hfill {\left[\frac{\partial \mathit{a}\left({\varphi }_k\right)}{\partial {\varphi }_k}\right]}_m=-j2\pi \frac{md}{\lambda }cos{\varphi }_k{e}^{-j2\pi \frac{md}{\lambda }sin {\varphi }_k}.\end{array}$$

(58)

Using the same method, ${\nabla }_{\mathit{\psi }}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$ can be also obtained. Additionally, we also have

$$\begin{array}{c}{\nabla }_{{\mathit{x}}^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=\frac{{\partial \mu \parallel \mathit{X}\parallel }_{2,1}+{\parallel \mathit{r}-({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\parallel }_2^2}{\partial {\mathit{x}}^{*}},\hfill \end{array}$$

(59)

and

$$\begin{array}{c}\frac{\partial \parallel \mathit{r}-({\mathit{I}}_P\otimes \mathit{CD}){\mathit{x}}^{*}{\parallel }_2^2}{\partial {\mathit{x}}^{*}}\hfill \\ =-\frac{\partial {\mathit{r}}^H({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial {\mathit{x}}^{*}}-\frac{\partial {\mathit{x}}^H({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)\mathit{r}}{\partial {\mathit{x}}^{*}}+\frac{\partial {\mathit{x}}^H({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial {\mathit{x}}^{*}}\hfill \\ =-{\left[({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)\mathit{r}\right]}^T\frac{\partial {\mathit{x}}^{*}}{\partial {\mathit{x}}^{*}}+{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^T\frac{\partial \left({\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^{*}\right)}{\partial {\mathit{x}}^{*}}\hfill \\ ={\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^T{\left[({\mathit{I}}_P\otimes \mathit{CD})\right]}^{*},\hfill \end{array}$$

(60)

where the k-th entry of ${\displaystyle \frac{{\partial \parallel \mathit{X}\parallel }_{2,1}}{\partial {\mathit{x}}^{*}}}$ is

$$\begin{array}{cc}\hfill {\left[\frac{{\partial \parallel \mathit{X}\parallel }_{2,1}}{\partial {\mathit{x}}^{*}}\right]}_k& =\frac{{\partial \parallel \mathit{X}\parallel }_{2,1}}{\partial x_k^{*}}=\frac{\partial {\sum }_{q=0}^{K-1}\sqrt{{\sum }_{p=0}^{P-1}{x}_{q,p}^2}}{\partial x_k^{*}}=\sum _{q=0}^{K-1}\frac{x_k}{2\sqrt{{\sum }_{p=0}^{P-1}{x}_{q,p}^2}},\hfill \end{array}$$

(61)

so we have

$$\begin{array}{c}\hfill \frac{{\partial \parallel \mathit{X}\parallel }_{2,1}}{\partial {\mathit{x}}^{*}}=\frac{1}{2}{\mathit{x}}^T\sum _{q=0}^{K-1}{\left(\sum _{p=0}^{P-1}{x}_{q,p}^2\right)}^{-\frac{1}{2}}.\end{array}$$

(62)

Therefore, we can obtain

$$\begin{array}{c}{\nabla }_{{\mathit{x}}^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=\frac{\mu }{2}{\mathit{x}}^T\sum _{q=0}^{K-1}{\left(\sum _{p=0}^{P-1}{x}_{q,p}^2\right)}^{-\frac{1}{2}}+{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^T{({\mathit{I}}_P\otimes \mathit{CD})}^{*}.\hfill \end{array}$$

we can also obtain ${\nabla }_{{\mathit{c}}_T^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$ as

$$\begin{array}{c}{\nabla }_{{\mathit{c}}_T^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})=\frac{{\partial \mu \parallel \mathit{X}\parallel }_{2,1}+{\parallel \mathit{r}-({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\parallel }_2^2}{\partial {\mathit{c}}_T^{*}}\hfill \\ =-\frac{\partial {\mathit{r}}^H({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial {\mathit{c}}_T^{*}}-\frac{\partial {\mathit{x}}^H({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)\mathit{r}}{\partial {\mathit{c}}_T^{*}}+\frac{\partial {\mathit{x}}^H({\mathit{I}}_P\otimes {\mathit{D}}^H{\mathit{C}}^H)({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial {\mathit{c}}_T^{*}}\hfill \\ =-{\mathit{r}}^H\frac{\partial ({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}}{\partial {\mathit{c}}_T^{*}}-{\mathit{r}}^T\frac{\partial {\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^{*}}{\partial {\mathit{c}}_T^{*}}+{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^T\frac{\partial \left({\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^{*}\right)}{\partial {\mathit{c}}_T^{*}}\hfill \\ +{\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}^H\frac{\partial \left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}\right]}{\partial {\mathit{c}}_T^{*}}\hfill \\ ={\left[({\mathit{I}}_P\otimes \mathit{CD})\mathit{x}-\mathit{r}\right]}^T\frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_T^{*}},\hfill \end{array}$$

(63)

and the m-th column of ${\displaystyle \frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_T^{*}}}$ is

$$\begin{array}{cc}\hfill {\left[\frac{\partial {({\mathit{I}}_P\otimes \mathit{CD})}^{*}{\mathit{x}}^{*}}{\partial {\mathit{c}}_T^{*}}\right]}_m=& \left({\mathit{I}}_P\otimes {\mathit{C}}_R^{*}\otimes \frac{\partial {\mathit{C}}_T^{*}}{\partial c_{T,m}^{*}}\right)\left({\mathit{I}}_P\otimes {\mathit{D}}^{*}\right){\mathit{x}}^{*}.\hfill \end{array}$$

(64)

Using the same method, ${\nabla }_{{\mathit{c}}_R^{*}}f(\mathit{\varphi },\mathit{\psi },\mathit{X},\mathit{C})$ can be also obtained. ☐

4. Cramér–Rao Lower Bound

The CRLB is adopted to show the lower bound on the variance of the estimated parameters including DOD/DOA ($\mathit{\varphi }$ and $\mathit{\psi }$), scattering coefficients ($\mathit{\alpha }$), and mutual coupling coefficients $\mathit{c}$. CRLB can be obtained from the Fisher information matrix (FIM)

$$\begin{array}{c}\hfill e_{\mathrm{CRLB}}={\parallel \mathit{d}\parallel }_2^2,\end{array}$$

(65)

where $\mathit{d}$ is a vector with the diagonal entries of ${\mathit{I}}^{-1}\left(\mathit{\theta }\right)$. $\mathit{I}\left(\mathit{\theta }\right)$ can be calculated as

$$\begin{array}{c}\hfill \mathit{I}\left(\mathit{\theta }\right)\triangleq \mathcal{E}\left\{{\left(\frac{\partial lnp}{\partial \mathit{\theta }}\right)}^H\left(\frac{\partial lnp}{\partial \mathit{\theta }}\right)\right\},\end{array}$$

(66)

where

$$\begin{array}{cc}\hfill \frac{\partial lnp}{\partial \mathit{\theta }}& =\left[\begin{array}{cccccc}{\displaystyle \frac{\partial lnp}{\partial \mathit{\varphi }}},& {\displaystyle \frac{\partial lnp}{\partial \mathit{\psi }}},& {\displaystyle \frac{\partial lnp}{\partial \mathit{\alpha }}},& {\displaystyle \frac{\partial lnp}{\partial {\mathit{\alpha }}^{*}}},& {\displaystyle \frac{\partial lnp}{\partial \mathit{c}}},& {\displaystyle \frac{\partial lnp}{\partial {\mathit{c}}^{*}}}\end{array}\right],\hfill \\ \hfill p& \triangleq f\left(\mathit{r}\right|\mathit{\varphi },\mathit{\psi },\mathit{\alpha },\mathit{c})=\frac{1}{{\pi }^{MNP}det\left({\sigma }_n^2\mathit{I}\right)}{e}^{-{\sigma }_n^{-2}{(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}^H(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })},\hfill \\ \hfill \mathbf{c}& \triangleq {\left[{\mathit{c}}_T^T,{\mathit{c}}_R^T\right]}^T.\hfill \end{array}$$

The subgradients of $lnp$ are calculated as follows:

• ${\displaystyle \frac{\partial lnp}{\partial \mathit{\varphi }}}$ is obtained as

  $$\begin{array}{c}\frac{\partial lnp}{\partial \mathit{\varphi }}=-{\sigma }_n^{-2}\frac{\partial {(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}^H(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}{\partial \mathit{\varphi }}\hfill \\ ={\sigma }_n^{-2}\left[{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^T\frac{\partial {(\mathit{I}\otimes \mathit{CA})}^{*}{\mathit{\alpha }}^{*}}{\partial \mathit{\varphi }}+{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^H\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{\varphi }}\right]\hfill \\ =2{\sigma }_n^{-2}\mathcal{R}\left\{{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^H\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{\varphi }}\right\},\hfill \end{array}$$

  (67)

  where the k-th column of ${\displaystyle \frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{\varphi }}}$ is

  $$\begin{array}{c}\hfill {\left[\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{\varphi }}\right]}_k=\left(\mathit{I}\otimes \mathit{C}\frac{\partial \mathit{A}}{\partial {\mathit{\varphi }}_k}\right)\mathit{\alpha },\end{array}$$

  (68)

  and

  $$\begin{array}{c}\hfill \frac{\partial \mathit{A}}{\partial {\varphi }_k}=\left[\begin{array}{c}{\mathbf{0}}_{MN\times k},\mathit{b}\left({\psi }_k\right)\otimes {\displaystyle \frac{\partial \mathit{a}\left({\varphi }_k\right)}{\partial {\varphi }_k}},{\mathbf{0}}_{MN\times (K-1-k)}\end{array}\right].\end{array}$$

  (69)
  With the same method, we can obtain ${\displaystyle \frac{\partial lnp}{\partial \mathit{\psi }}}$.
• ${\displaystyle \frac{\partial lnp}{\partial \mathit{\alpha }}}$ is obtained as

  $$\begin{array}{cc}\frac{\partial lnp}{\partial \mathit{\alpha }}\hfill & =-{\sigma }_n^{-2}\frac{\partial {(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}^H(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}{\partial \mathit{\alpha }}\hfill \\ & ={\sigma }_n^{-2}{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^H(\mathit{I}\otimes \mathit{CA}),\hfill \end{array}$$

  (70)

  and we have ${\displaystyle \frac{\partial lnp}{\partial {\mathit{\alpha }}^{*}}}={\left({\displaystyle \frac{\partial lnp}{\partial \mathit{\alpha }}}\right)}^{*}$.
• ${\displaystyle \frac{\partial lnp}{\partial \mathit{c}}}$ is obtained as

  $$\begin{array}{cc}\frac{\partial lnp}{\partial \mathit{c}}\hfill & =-{\sigma }_n^{-2}\frac{\partial {(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}^H(\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha })}{\partial \mathit{c}}\hfill \\ & ={\sigma }_n^{-2}\left[{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^T\frac{\partial \left({\left[(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^{*}\right)}{\partial \mathit{c}}+{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^H\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{c}}\right]\hfill \\ & ={\sigma }_n^{-2}{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^H\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{c}},\hfill \end{array}$$

  (71)

  where the n-th column of ${\displaystyle \frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{c}}}$ is

  $$\begin{array}{c}\hfill {\left[\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial \mathit{c}}\right]}_n=\left({\mathit{I}}_P\otimes \frac{\partial \mathit{C}}{\partial c_n}\mathit{A}\right)\mathit{\alpha }.\end{array}$$

  (72)
• ${\displaystyle \frac{\partial lnp}{\partial {\mathit{c}}^{*}}}$ is obtained as

  $$\begin{array}{cc}\hfill \frac{\partial lnp}{\partial {\mathit{c}}^{*}}& ={\sigma }_n^{-2}\left[{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^T\frac{\partial \left({\left[(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^{*}\right)}{\partial {\mathit{c}}^{*}}+{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^H\frac{\partial (\mathit{I}\otimes \mathit{CA})\mathit{\alpha }}{\partial {\mathit{c}}^{*}}\right]\hfill \\ & ={\sigma }_n^{-2}{\left[\mathit{r}-(\mathit{I}\otimes \mathit{CA})\mathit{\alpha }\right]}^T\frac{\partial (\mathit{I}\otimes {\mathit{C}}^{*}{\mathit{A}}^{*}){\mathit{\alpha }}^{*}}{\partial {\mathit{c}}^{*}},\hfill \end{array}$$

  (73)

  where the n-th column of $\frac{\partial (\mathit{I}\otimes {\mathit{C}}^{*}{\mathit{A}}^{*}){\mathit{\alpha }}^{*}}{\partial {\mathit{c}}^{*}}$ is

  $$\begin{array}{c}\hfill {\left[\frac{\partial (\mathit{I}\otimes {\mathit{C}}^{*}{\mathit{A}}^{*}){\mathit{\alpha }}^{*}}{\partial {\mathit{c}}^{*}}\right]}_n=\left(\mathit{I}\otimes \frac{\partial {\mathit{C}}^{*}}{\partial c_n^{*}}{\mathit{A}}^{*}\right){\mathit{\alpha }}^{*}.\end{array}$$

  (74)

Finally, the FIM is obtained as

$$\begin{array}{c}\mathcal{E}\left\{{\left(\frac{\partial lnp}{\partial \mathit{\theta }}\right)}^H\left(\frac{\partial lnp}{\partial \mathit{\theta }}\right)\right\}\hfill \\ ={\sigma }_n^{-2}\left[\begin{array}{cccccc}2\mathcal{R}\left\{{\Omega }_{\varphi }^H{\Omega }_{\varphi }\right\},& 2\mathcal{R}\left\{{\Omega }_{\varphi }^H{\Omega }_{\psi }\right\},& {\Omega }_{\varphi }^H\mathit{G},& {\Omega }_{\varphi }^T{\mathit{G}}^{*},& {\Omega }_{\varphi }^H{\Omega }_c,& {\Omega }_{\varphi }^T{\Omega }_c^{*}\\ 2\mathcal{R}\left\{{\Omega }_{\psi }^H{\Omega }_{\varphi }\right\},& 2\mathcal{R}\left\{{\Omega }_{\psi }^H{\Omega }_{\psi }\right\},& {\Omega }_{\psi }^H\mathit{G},& {\Omega }_{\psi }^T{\mathit{G}}^{*},& {\Omega }_{\psi }^H{\Omega }_c,& {\Omega }_{\psi }^T{\Omega }_c^{*}\\ {\mathit{G}}^H{\Omega }_{\varphi },& {\mathit{G}}^H{\Omega }_{\psi },& {\mathit{G}}^H\mathit{G},& \mathbf{0},& {\mathit{G}}^H{\Omega }_c,& \mathbf{0}\\ {\mathit{G}}^T{\Omega }_{\varphi }^{*},& {\mathit{G}}^T{\Omega }_{\psi }^{*},& \mathbf{0},& {\mathit{G}}^T{\mathit{G}}^{*},& \mathbf{0},& {\mathit{G}}^T{\Omega }_c^{*}\\ {\Omega }_c^H{\Omega }_{\varphi },& {\Omega }_c^H{\Omega }_{\psi },& {\Omega }_c^H\mathit{G},& \mathbf{0},& {\Omega }_c^H{\Omega }_c,& \mathbf{0}\\ {\Omega }_c^T{\Omega }_{\varphi }^{*},& {\Omega }_c^T{\Omega }_{\psi }^{*},& \mathbf{0},& {\Omega }_c^T{\mathit{G}}^{*},& \mathbf{0},& {\Omega }_c^T{\Omega }_c^{*}\end{array}\right],\hfill \end{array}$$

(75)

where $\mathit{G}\triangleq {\mathit{I}}_P\otimes \mathit{CA}$, ${\Omega }_{\varphi }\triangleq \frac{\partial \mathit{G}\alpha }{\partial \varphi }$, ${\Omega }_{\psi }\triangleq \frac{\partial \mathit{G}\alpha }{\partial \psi }$, ${\Omega }_c\triangleq \frac{\partial \mathit{G}\alpha }{\partial \mathit{c}}$. Then, with FIM, the corresponding CRLB can be obtained.

5. Computational Complexity

In Algorithm 1, to estimate the DOD/DOA using the SOMP method, the computational complexity is $\mathcal{O}(PMN+Q^3+Q^{2}MN+Q{\left(MN\right)}^2)$. In Algorithm 2 to estimate the mutual coupling matrix, the computational complexity is $\mathcal{O}(MNPQ+{\left(MN\right)}^{2}Q)$. Additionally, in Algorithm 3, the estimation results are polished, and the computational complexity is $\mathcal{O}(MNPQ+{\left(MN\right)}^{2}Q)$. Therefore, the computational complexity of the proposed method to estimate DOD/DOA and the mutual coupling matrix can be roughly expressed as $\mathcal{O}(Q^3+MN{Q}^2+{\left(MN\right)}^{2}Q+MNPQ)$. Usually, we have $MN\le Q$, so the roughly computational complexity can be simplified as $\mathcal{O}(Q^3+P{Q}^2)$.

6. Simulation Results

In this section, the simulation results are given to show the performance of the proposed algorithm. The simulation parameters are given in

Table 1. Simulation parameters.

Parameter	Value
Carrier frequency $f_c$	1 GHz
Speed of waveform c	$3\times {10}^8$ m/s
Wavelength $\lambda$	$0.3$ m
Pulse number P	100
Antenna space d	$0.15$ m
Antenna number in transmitter M	20
Antenna number in receiver N	20
Dictionary resolution $|{\varphi }_{D,z_1}-{\varphi }_{D,z_1+1}|$	$0.035$
Detection angle range	30°∼60°
Iteration number	$8\times {10}^3$
Target number K	2

. First, the reconstruction performance for the received signal is shown in Figure 4. The reconstruction error is defined as

$$\begin{array}{c}\hfill e_r=\frac{{∥\mathit{r}-\widehat{\mathit{r}}∥}_2^2}{{∥\mathit{r}∥}_2^2},\end{array}$$

(76)

where $\mathit{r}$ is the received signal defined in (29), and $\widehat{\mathit{r}}$ is the reconstruction signal with the estimated parameters including DOD/DOA, scattering coefficients and mutual coupling matrices. As shown in Figure 4, the proposed method polishes the estimated DOD/DOA and mutual coupling matrices iteratively, where Algorithm 2 is adopted to estimate mutual coupling matrices and Algorithm 3 is used to polished the estimated DOD/DOA and mutual coupling matrices. The relative reconstruction error is decreasing with increasing the number of iterations. Additionally, as shown in this figure, Algorithm 3 is more significant in improving the estimation performance than Algorithm 2. Therefore, it is efficient to polish the estimated results in the off-grid problem after the rough on-grid estimation.

Figure 5 shows the estimated DOD/DOA using different methods, where ◯ denotes the DOD/DOA of the target, × denotes the estimated DOD/DOA with the proposed method, and △ denotes the estimated DOD/DOA with the on-grid SOMP method [32]. As shown in this figure, when only the on-grid SOMP method is used to estimate the target DOD/DOA, the estimation error is larger than that using the proposed method. In the proposed method, we adopt the proposed off-grid method to further improve the on-grid result, so the proposed method can outperform the traditional on-grid method.

Figure 6 shows the reconstruction performance with the proposed method where the signal-to-noise ratios (SNRs) are 5 dB, 10 dB and 20 dB. With different SNRs, the same waveforms are adopted, so the correlation between waveforms are the same. As shown in this figure, with increasing the SNR of the received signal, better reconstruction performance can be achieved. Additional, when SNR = 20 dB, the reconstruction performance is almost the same as the one without noise. After about $8\times {10}^3$ iterations, the reconstruction performance is convergence, so we can adopt $8\times {10}^3$ as the maximum number of iterations in the following simulations. In Figure 7, we also compare the estimated results with the CRLB derived in this paper. As shown in this figure, the proposed estimation method can approach the CRLB, so the estimation method is efficient.

In Figure 8, we show the effect of mutual coupling on the estimation performance. As shown in this figure, better estimation performance can be achieved by improving the SNR of the received signal. The curves “with perfect information” are the simulation results with the perfect information of mutual coupling effect. The best estimation performance can be achieved by the methods with perfect information. Moreover, the mutual coupling has great effect on the estimation performance, so better reconstruction performance can be achieved by estimating the mutual coupling matrices during the DOD/DOA estimation. With different target numbers, Figure 9 shows the reconstruction estimation performance. The targets are uniformly distributed in the angle range from 30 to 60. When the target number is increasing, the reconstruction performance will be worse with the high correlation between the echoed waveforms from different targets. However, with better SNR, more targets can be estimated with the same reconstruction performance.

7. Conclusions

In the bistatic MIMO radar, the DOD/DOA estimation problem with mutual coupling effect between antennas has been addressed. After formulating the system model, the iterative method based on CS has been proposed to exploit the sparsity of targets in the detection area, where the estimation for DOD/DOA and mutual coupling has been polished by solving the off-grid problem. Then, the corresponding CRLBs for the parameters including DOD/DOA, mutual coupling matrices, and scattering coefficients, have been derived. Simulation results show that the proposed estimation method can approach the CRLB and achieve the better estimation performance than the traditional methods. Further work will focus on the estimation of moving targets in the MIMO radar system with mutual coupling.