Noncircular Sources-Based Sparse Representation Algorithm for Direction of Arrival Estimation in MIMO Radar with Mutual Coupling

Abstract

In this paper, a reweighted sparse representation algorithm based on noncircular sources is proposed, and the problem of the direction of arrival (DOA) estimation for multiple-input multiple-output (MIMO) radar with mutual coupling is addressed. Making full use of the special structure of banded symmetric Toeplitz mutual coupling matrices (MCM), the proposed algorithm firstly eliminates the effect of mutual coupling by linear transformation. Then, a reduced dimensional transformation is exploited to reduce the computational complexity of the proposed algorithm. Furthermore, by utilizing the noncircular feature of signals, the new extended received data matrix is formulated to enlarge the array aperture. Finally, based on the new received data, a reweighted matrix is constructed, and the proposed method further designs the joint reweighted sparse representation scheme to achieve the DOA estimation by solving the $l_1$-norm constraint minimization problem. The proposed method enlarges the array aperture due to the application of signal noncircularity, and in the presence of mutual coupling, the proposed algorithm provides higher resolution and better angle estimation performance than ESPRIT-like, $l_1$-SVD and $l_1$-SRDML (sparse representation deterministic maximum likelihood) algorithms. Numerical experiment results verify the effectiveness and advantages of the proposed method.

1. Introduction

With orthogonal transmitted waveforms, multiple-input multiple-output (MIMO) radar has drawn increasing attention in the field of wireless communications. Compared with the conventional phased-array radar, MIMO radar owns a number of advantages, such as higher resolution and better parameter identifiability [1]. In addition, MIMO radar can be classified into the following two types: statistical MIMO radar and colocated MIMO radar. Colocated MIMO radar can achieve more degrees of freedom and higher spatial resolution, because its closely-spaced antennas form a virtual array with a large aperture. Colocated MIMO radar includes the monostatic one and the bistatic one. In bistatic MIMO radar, the transmit array and the receive array are separated away, while they are close to each other in monostatic MIMO radar. In this paper, the angle estimation problem is investigated in monostatic MIMO radar.

Parameter estimation is an important aspect in wireless communications and sensor array signal processing [2]. In order to estimate the direction of arrival (DOA), a large number of subspace-based algorithms have been proposed [3,4,5]. Recently, the emerging sparse representation (SR) and compressed sensing (CS) have attracted more and more attention in various fields [6,7,8], such as sensor array signal processing, imaging and image processing, and so forth. Furthermore, it has been verified that SR-based DOA estimation algorithms have remarkable advantages over other methods [9], for instance they adapt better to challenging circumstances and provide higher angle resolution. A few SR-based algorithms, such as $l_1$-SVD (singular value decomposition) [9] and $l_1$-SRACV (array covariance vectors) [10], have been proposed via sparse signal recovery. In [11], based on the array covariance vector, the revised real-valued (RV) $l_1$-SVD approach has better DOA estimation performance and lower computation burden than $l_1$-SVD [9] and RV $l_1$-SVD [12].

However, in practical situations, the sensor array manifold is often affected by the errors of unknown mutual coupling. When they are taken into account, the DOA estimation performance of the above-mentioned methods is seriously degraded. With mutual coupling, the subspace-based MUSIC-like (multiple signal classification-like) [13] algorithm is proposed. Besides, a ESPRIT-like (estimation of signal parameters via rotational invariance techniques-like) algorithm is presented in [14], which owns a low calculation burden. In [15], the SR-based revised $l_1$-SVD algorithm eliminates the errors of mutual coupling, and it achieves angle estimation by solving the problem of the $l_1$-norm constraint minimization. In [16], in the presence of mutual coupling, by introducing the refined deterministic maximum likelihood (DML) procedure, a covariance vector-based sparse representation algorithm is proposed with outstanding angle estimation performance. We call the method in [16] $l_1$-SRDML (sparse representation deterministic maximum likelihood).

In practical communication systems and radar applications, the complex noncircular sources are widely used, such as binary phase shift keying (BPSK), multiple amplitude shift keying (MASK) and unbalanced quadrature phase shift keying (UQPSK) modulated signals [17]. Noncircular signals can be used to enlarge the array aperture without extra antennas, which contributes to the improvement of angle estimation performance [18]. Exploiting the signal noncircularity, some subspace-based methods have been developed for angle estimation in MIMO radar [17,19,20]. They have verified that compared with traditional subspace-based algorithms, the methods that are based on noncircular signals can achieve better DOA estimation performance. However, these subspace-based approaches do not take the errors of unknown mutual coupling into account. Moreover, they perform poorly when experiencing challenging circumstances, such as fewer snapshots, which can be solved by sparse representation-based methods. For DOA estimation in MIMO radar with mutual coupling, to the best of our knowledge, up to now, there are no references about SR-based methods that exploit the signal noncircularity to extend the virtual array. Consequently, utilizing noncircular sources and the sparse representation framework, we aim to eliminate the effect of mutual coupling and then achieve a better DOA estimation.

In this paper, by using the noncircular signals, we propose a sparse representation-based DOA estimation algorithm in MIMO radar with mutual coupling. The new method firstly eliminates the effect of mutual coupling by extracting the information of mutual coupling coefficients into a diagonal matrix. Secondly, based on the reduced dimensional transformation and the characteristics of signal noncircularity, the new extended received data matrix is formulated to enlarge the array aperture. Finally, a reweighted matrix is constructed, based on which a joint reweighted $l_1$-norm minimization sparse representation framework is designed to obtain the DOAs. In the presence of mutual coupling, the proposed algorithm performs better than the conventional algorithms on account of the error elimination and the application of noncircular signals.

The rest of this paper is organized as follows. In Section 2, the MIMO radar system model with mutual coupling and noncircular signals is described. In Section 3, the implementation process of the proposed method is described in detail, including mutual coupling elimination, noncircular signal-based extended matrix construction and the joint reweighted sparse representation-based DOA estimation scheme. In Section 4, we give some related remarks and discussions regarding the parameter setting and the computational complexity of the proposed method. Then, in Section 5, simulations are implemented to verify the efficiency and the advantages of the proposed method. Finally, our conclusions are summarized in Section 6.

Notation: ${(·)}^{\mathrm{H}}$, ${(·)}^{\mathrm{T}}$, ${(·)}^{*}$, ${(·)}^{-1}$, $\mathrm{E}(·)$ and $\det (·)$ denote conjugate-transpose, transpose, conjugate, inverse, expectation and determinant operators, respectively. ⊗ and ⊙ denote the Kronecker product and the Khatri–Rao product, respectively. ${\mathbf{I}}_K$ denotes a $K\times K$ dimensional unit matrix. $\parallel ·{\parallel }_1$, $\parallel ·{\parallel }_2$ and $\parallel ·{\parallel }_F$ denote the $l_1$-norm, the $l_2$-norm and the Frobenius-norm, respectively. $\mathrm{diag}(·)$ denotes the diagonalization operation.

2. Problem Formulation

2.1. MIMO Radar System Model with Mutual Coupling

Consider a narrowband monostatic MIMO radar system, shown in Figure 1, whose transmit and receive arrays are both half-wavelength d spaced uniform linear arrays (ULAs). In the transmit array, M antennas transmit M orthogonal noncircular waveforms, such as BPSK modulated signals. In the receive array, N antennas are impinged by P uncorrelated targets that can be deemed as point scatterers in the far-field. Different from the mutually-separated arrays in bistatic MIMO radar, the transmitter and the receiver are closely located in monostatic radar, as shown in Figure 2. Since the transmit array and the receive array are collocated at the same site in monostatic MIMO radar systems [17], the echo signal travels the same path as the transmitted signal. Therefore, for the p-th target, the directions of departure of the transmitted signal and arrival of the echo from the target are the same [21], i.e., ${\theta }_{DOD}={\theta }_{DOA}$, both of which are denoted as DOA ${\theta }_p,p=1,2,\dots P$. When mutual coupling is taken into account in both the transmit array and the receive array, at the receiver, the output of the matched filters at a certain snapshot is given by [16]:

$$\mathbf{x}(t)=\tilde{\mathbf{A}}\mathbf{s}(t)+\mathbf{n}(t)$$

(1)

where $\mathbf{x}(t)\in {\mathbb{C}}^{MN\times 1}$ is the received data vector, $\mathbf{s}(t)={[s_1(t),s_2(t),\dots ,s_P(t)]}^{\mathtt{T}}\in {\mathbb{C}}^{P\times 1}$ is the complex-valued noncircular signal vector and $\mathbf{n}(t)\in {\mathbb{C}}^{MN\times 1}$ is the additional Gaussian noise vector with zero-mean and covariance matrix ${\sigma }^2{\mathbf{I}}_{MN}$, in which ${\sigma }^2$ is the noise power. Meanwhile, the detailed expression of $\tilde{\mathbf{A}}$ is [14]:

$$\begin{array}{cc}\hfill \tilde{\mathbf{A}}& =[{\mathbf{C}}_t{\mathbf{a}}_t({\theta }_1)\otimes {\mathbf{C}}_r{\mathbf{a}}_r({\theta }_1),\dots ,{\mathbf{C}}_t{\mathbf{a}}_t({\theta }_P)\otimes {\mathbf{C}}_r{\mathbf{a}}_r({\theta }_P)]\hfill \\ & =({\mathbf{C}}_t\otimes {\mathbf{C}}_r)[{\mathbf{a}}_t({\theta }_1)\otimes {\mathbf{a}}_r({\theta }_1),\dots ,{\mathbf{a}}_t({\theta }_P)\otimes {\mathbf{a}}_r({\theta }_P)]\hfill \end{array}$$

(2)

where $\tilde{\mathbf{A}}\in {\mathbb{C}}^{MN\times P}$, ${\mathbf{a}}_t({\theta }_p)={[1,e^{j\pi sin({\theta }_p)},e^{j\pi 2sin({\theta }_p)},\dots ,e^{j\pi (M-1)sin({\theta }_p)}]}^{\mathtt{T}}$ is the transmit steering vector and ${\mathbf{a}}_r({\theta }_p)={[1,e^{j\pi sin({\theta }_p)},e^{j\pi 2sin({\theta }_p)},\dots ,e^{j\pi (N-1)sin({\theta }_p)}]}^{\mathtt{T}}$ is the receive steering vector, $p=1,2,\dots P$. ${\mathbf{C}}_t$ and ${\mathbf{C}}_r$ are mutual coupling matrices of the transmit array and the receive array, respectively. ${\mathbf{C}}_t$ and ${\mathbf{C}}_r$ can be modeled as banded symmetric Toeplitz matrices, which are expressed as [22,23,24]:

$$\begin{array}{c}\hfill {\mathbf{C}}_i=\left[\begin{array}{cccccc}{c}_{i0}& c_{i1}& \cdots & c_{ik}& \cdots & 0\\ c_{i1}& c_{i0}& c_{i1}& \cdots & \ddots & ⋮\\ ⋮& c_{i1}& c_{i0}& \ddots & \ddots & c_{ik}\\ c_{ik}& \cdots & \ddots & \ddots & c_{i1}& ⋮\\ ⋮& \ddots & \cdots & c_{i1}& c_{i0}& c_{i1}\\ 0& \cdots & c_{ik}& \cdots & c_{i1}& c_{i0}\end{array}\right]\end{array}$$

(3)

where $i=t,r$, $c_{ij}$ is the non-zero mutual coupling coefficient, $j=0,1,...,k$. In Figure 1, the coupling effects of the m-th and the n-th elements in the transmit and the receive arrays are described, respectively. ${\beta }_{(m,q)}$ is the coupling contribution from the q-th to the m-th elements in transmit array, and ${\overline{\beta }}_{(n,q^{\prime })}$ is the contribution from the $q^{\prime }$-th to the n-th elements in the receive array, $1\le m\le M$, $1\le n\le N$, and $|q-m|\le k$, $|q^{\prime }-n|\le k$. ${\beta }_{(m,q)}$ and ${\overline{\beta }}_{(n,q^{\prime })}$ contain the information of $c_{t|q-m|}$ and $c_{r|q^{\prime }-n|}$, respectively. Mutual coupling coefficients are factors related to the distance between the two antennas [23], and for the $k+1$ non-zero coefficients, they satisfy $0<\left|c_{ik}\right|<...<\left|c_{i1}\right|<\left|c_{i0}\right|=1$. Besides, for the ULA coupling model, just a few mutual coupling coefficients are non-zero ones [22,23,24], and without loss of generality, $2k<\min \{M,N\}$ is assumed. Then, by collecting J snapshots, the received data in MIMO radar with mutual coupling is represented as follows:

$$\mathbf{X}=\mathbf{C}\mathbf{A}\mathbf{S}+\mathbf{N}$$

(4)

where $\mathbf{X}\in {\mathbb{C}}^{MN\times J}$ is the data matrix, $\mathbf{C}=({\mathbf{C}}_t\otimes {\mathbf{C}}_r)\in {\mathbb{C}}^{MN\times MN}$ is the transmit-receive mutual coupling matrix. $\mathbf{A}=[\mathbf{a}({\theta }_1),\dots ,\mathbf{a}({\theta }_P)]\in {\mathbb{C}}^{MN\times P}$ is the steering matrix, with the steering vector $\mathbf{a}({\theta }_p)={\mathbf{a}}_t({\theta }_p)\otimes {\mathbf{a}}_r({\theta }_p),p=1,2,\dots P$. $\mathbf{S}=[\mathbf{s}(t_1),\dots ,\mathbf{s}(t_J)]\in {\mathbb{C}}^{P\times J}$ and $\mathbf{N}=[\mathbf{n}(t_1),\dots ,\mathbf{n}(t_J)]\in {\mathbb{C}}^{MN\times J}$ are the noncircular signal matrix and the complex Gaussian white noise matrix, respectively.

2.2. Noncircular Signals

For complex random variables and vectors, circularity is an important property [18]. Making full use of the properties of noncircular signals, the array aperture can be enlarged. The invariance specialityof rotation can be used to judge if a complex random sequence is noncircular [18]. More specifically, if stationary complex signal sequence $\overline{\mathbf{q}}$ is circular, it accords with $\mathrm{E}[\overline{\mathbf{q}}{\overline{\mathbf{q}}}^{\mathtt{H}}]\ne 0$ and $\mathrm{E}[\overline{\mathbf{q}}{\overline{\mathbf{q}}}^{\mathtt{T}}]=0$. By comparison, if the sequence is noncircular, then $\mathrm{E}[\overline{\mathbf{q}}{\overline{\mathbf{q}}}^{\mathtt{H}}]\ne 0$ and $\mathrm{E}[\overline{\mathbf{q}}{\overline{\mathbf{q}}}^{\mathtt{T}}]\ne 0$, which can be utilized to increase the number of effective array elements. In addition, for zero-mean stationary complex signal sequence $\overline{\mathbf{q}}$, it holds the property that:

$$\mathrm{E}[\overline{\mathbf{q}}{\overline{\mathbf{q}}}^{\mathtt{T}}]=\rho e^{j\psi }\mathrm{E}[\overline{\mathbf{q}}{\overline{\mathbf{q}}}^{\mathtt{H}}]$$

(5)

where $0\le \rho \le 1$ and ψ are noncircular rate and noncircular phase angle, respectively, and ρ depends on the signal modulation type. In this paper, we consider the signals with maximum noncircular rate, namely, $\rho =1$ in $\mathbf{s}(t)$, such as BPSK and MASK modulated signals that are widely used in MIMO radar systems. It has been pointed out that the noncircular signal vector $\mathbf{s}(t)$ in Equation (1) can be rewritten as [17,25]:

$$\mathbf{s}(t)={\mathbf{\Lambda }}_{\phi }{\mathbf{s}}_r(t)$$

(6)

where ${\mathbf{s}}_r(t)={[s_{r1}(t),s_{r2}(t),\dots ,s_{rP}(t)]}^{\mathtt{T}}$ is the real-valued part of noncircular signal $\mathbf{s}(t)$, and ${\mathbf{\Lambda }}_{\phi }=\mathrm{diag}([e^{j{\phi }_1},e^{j{\phi }_2},\dots ,e^{j{\phi }_P}])$ is the noncircular phase matrix. Based on the composition and structure of $\mathbf{s}(t)$, for the p-th target with $t=1,2,\dots ,J$, $\mathrm{E}[{\mathbf{s}}_p{\mathbf{s}}_p^{\mathtt{T}}]=e^{j2{\phi }_p}\mathrm{E}[{\mathbf{s}}_{rp}{\mathbf{s}}_{rp}^{\mathtt{T}}]=e^{j2{\phi }_p}\mathrm{E}[{\mathbf{s}}_p{\mathbf{s}}_p^{\mathtt{H}}]$ can be obtained. In addition, each element in $\mathbf{s}(t)$ accords with Equation (5), i.e., $\mathrm{E}[{\mathbf{s}}_p{\mathbf{s}}_p^{\mathtt{T}}]=e^{j\psi }\mathrm{E}[{\mathbf{s}}_p{\mathbf{s}}_p^{\mathtt{H}}]$ with $\rho =1$ [26]. This indicates that the relationship between ${\phi }_p$ and ${\psi }_p$ can be written as ${\phi }_p={\psi }_p/2$. With the application of signal noncircularity, $\mathbf{X}$ in Equation (4) is expressed as:

$$\mathbf{X}=\mathbf{C}\mathbf{A}{\mathbf{\Lambda }}_{\phi }{\mathbf{S}}_r+\mathbf{N}$$

(7)

where ${\mathbf{S}}_r=[{\mathbf{s}}_r(t_1),\dots ,{\mathbf{s}}_r(t_J)]$ is a real-valued matrix.

3. The Proposed Algorithm

3.1. Mutual Coupling Elimination

To successfully implement the noncircular source-based sparse representation method, we firstly eliminate the effect of mutual coupling. Exploiting the structure of mutual coupling matrices ${\mathbf{C}}_t$ and ${\mathbf{C}}_r$, we define two selection matrices as follows:

$$\begin{array}{cc}\hfill {\mathbf{\Gamma }}_1& =[{\mathbf{0}}_{(M-2k)\times k}{\mathbf{I}}_{(M-2k)\times (M-2k)}{\mathbf{0}}_{(M-2k)\times k}]\hfill \\ \hfill {\mathbf{\Gamma }}_2& =[{\mathbf{0}}_{(N-2k)\times k}{\mathbf{I}}_{(N-2k)\times (N-2k)}{\mathbf{0}}_{(N-2k)\times k}]\hfill \end{array}$$

(8)

where ${\mathbf{\Gamma }}_1\in {\mathbb{C}}^{(M-2k)\times M}$, and ${\mathbf{\Gamma }}_2\in {\mathbb{C}}^{(N-2k)\times N}$. ${\mathbf{\Gamma }}_1$ and ${\mathbf{\Gamma }}_2$ can be used to choose the central rows of the mutual coupling matrices and then transform the transmit-receive mutual coupling matrix $\mathbf{C}$ into a diagonal one. Thus, let $z_p=e^{j\pi sin({\theta }_p)}$, by multiplying the selection matrix ${\mathbf{\Gamma }}_1$ on the left side of the transmit steering vector with mutual coupling; the transformation of the i-th element in ${\mathbf{\Gamma }}_1{\mathbf{C}}_t{\mathbf{a}}_t({\theta }_p)\in {\mathbb{C}}^{(M-2k)\times 1}$ can be derived as:

$$\begin{array}{cc}\hfill [{\mathbf{\Gamma }}_1{\mathbf{C}}_t{\mathbf{a}}_t({\theta }_p)](i)& =c_{tk}{z}_p^{i-1}+c_{t(k-1)}{z}_p^i+\dots +z_p^{i+k-1}+\dots +c_{tk}{z}_p^{i+k-1+k}\hfill \\ & =({\sum }_{j=-k}^k{c}_{t\left|j\right|}{z}_p^{j+k})z_p^{i-1}\hfill \\ & ={\omega }_{tp}{z}_p^{i-1}\hfill \end{array}$$

(9)

where ${\omega }_{tp}$ is a scalar that just contains the information of ${\theta }_p$ and mutual coupling in the transmit array. Let $\overline{M}=M-2k$; thus, for $i=1,2,\dots ,\overline{M}$, we have ${\mathbf{\Gamma }}_1{\mathbf{C}}_t{\mathbf{a}}_t({\theta }_p)={\omega }_{tp}{\tilde{\mathbf{a}}}_t$, in which ${\tilde{\mathbf{a}}}_t$ is the new transmit steering vector and can be expressed as ${\tilde{\mathbf{a}}}_t=[1,z_p,z_p^2\dots ,z_p^{\overline{M}-1}]$, $p=1,2,\dots ,P$. Similar to Equation (9), we use the other selection matrix ${\mathbf{\Gamma }}_2$ to extract the mutual coupling coefficients in ${\mathbf{C}}_r{\mathbf{a}}_r({\theta }_p)$ into a scalar, that is:

$$\begin{array}{cc}\hfill [{\mathbf{\Gamma }}_1{\mathbf{C}}_r{\mathbf{a}}_r({\theta }_p)](i)& =c_{rk}{z}_p^{i-1}+c_{r(k-1)}{z}_p^i+\dots +z_p^{i+k-1}+\dots +c_{rk}{z}_p^{i+k-1+k}\hfill \\ & =({\sum }_{j=-k}^k{c}_{r\left|j\right|}{z}_p^{j+k})z_p^{i-1}\hfill \\ & ={\omega }_{rp}{z}_p^{i-1}\hfill \end{array}$$

(10)

where $p=1,2,\dots ,P$. The value of ${\omega }_{rp}$ depends on the mutual coupling coefficients and ${\theta }_p$. Then, for $i=1,2,\dots ,\overline{N}$, we further have ${\mathbf{\Gamma }}_2{\mathbf{C}}_r{\mathbf{a}}_r({\theta }_p)={\omega }_{rp}{\tilde{\mathbf{a}}}_r$ with ${\tilde{\mathbf{a}}}_r=[1,z_p,z_p^2\dots ,z_p^{\overline{N}-1}]$ being the new receive steering vector. As a result, exploiting the property of the Kronecker product operator, let $\mathbf{\Gamma }={\mathbf{\Gamma }}_1\otimes {\mathbf{\Gamma }}_2\in {\mathbb{C}}^{\overline{M}\overline{N}\times MN}$ be the selection matrix of transmit-receive array; we deduce that:

$$\begin{array}{cc}\hfill \mathbf{\Gamma }\mathbf{C}\mathbf{a}({\theta }_p)& =[{\mathbf{\Gamma }}_1{\mathbf{C}}_t{\mathbf{a}}_t({\theta }_p)]\otimes [{\mathbf{\Gamma }}_2{\mathbf{C}}_r{\mathbf{a}}_r({\theta }_p)]\hfill \\ & ={\omega }_{tp}{\omega }_{rp}\tilde{\mathbf{a}}({\theta }_p)\hfill \end{array}$$

(11)

where $\tilde{\mathbf{a}}({\theta }_p)={\tilde{\mathbf{a}}}_t({\theta }_p)\otimes {\tilde{\mathbf{a}}}_r({\theta }_p)\in {\mathbb{C}}^{\overline{M}\overline{N}\times 1}$ is the new transmit-receive steering vector. Thus, for J snapshots, the data matrix $\mathbf{X}$ in Equation (7) transfers into:

$$\overline{\mathbf{X}}=\overline{\mathbf{A}}\mathbf{D}{\mathbf{\Lambda }}_{\phi }{\mathbf{S}}_r+\overline{\mathbf{N}}$$

(12)

where $\overline{\mathbf{A}}=[\tilde{\mathbf{a}}({\theta }_1),\tilde{\mathbf{a}}({\theta }_2),\dots ,\tilde{\mathbf{a}}({\theta }_P)]\in {\mathbb{C}}^{\overline{M}\overline{N}\times P}$ and $\overline{\mathbf{N}}=\mathbf{\Gamma }\mathbf{N}\in {\mathbb{C}}^{\overline{M}\overline{N}\times J}$ are the new transmit-receive steering matrix and noise matrix, respectively; in addition, $\mathbf{D}=\mathrm{diag}[{\omega }_{t1}{\omega }_{r1},{\omega }_{t2}{\omega }_{r2},\dots ,{\omega }_{tP}{\omega }_{rP}]\in {\mathbb{C}}^{P\times P}$. It can be observed in Equation (12) that non-zero mutual coupling coefficients are extracted into a diagonal matrix $\mathbf{D}$. Hence, for the new steering matrix $\overline{\mathbf{A}}$, the errors of mutual coupling are eliminated.

3.2. Noncircular Signal-Based Extended Matrix Construction

Based on the new received data matrix $\overline{\mathbf{X}}$, a reduced dimensional transformation is introduced in the following. For the p-th target, the transmit-receive steering vector $\tilde{\mathbf{a}}({\theta }_p)$ can be expressed as:

$$\begin{array}{cc}\hfill \tilde{\mathbf{a}}({\theta }_p)& ={(1,\dots ,z_p^{\overline{N}-1};z_p,\dots ,z_p^{\overline{N}};\dots ;z_p^{\overline{M}-1},\dots ,z_p^{\overline{M}+\overline{N}-2})}^{\mathtt{T}}\hfill \end{array}$$

(13)

where $p=1,2,\dots ,P$. Many repeated terms are contained in $\tilde{\mathbf{a}}({\theta }_p)$, and the non-repeated elements can make up another vector $\mathbf{b}({\theta }_p)={[1,\exp (j\pi \sin {\theta }_p),\cdots ,\exp (j\pi (\overline{M}+\overline{N}-2)\sin {\theta }_p)]}^{\mathrm{T}}\in {\mathbb{C}}^{(\overline{M}+\overline{N}-1)\times 1}$. The relationship between $\tilde{\mathbf{a}}({\theta }_p)$ and $\mathbf{b}({\theta }_p)$ can be derived as $\tilde{\mathbf{a}}({\theta }_p)=\mathbf{Gb}({\theta }_p)$, in which $\mathbf{G}$ is represented as:

$$\begin{array}{c}\hfill \mathbf{G}={[{\mathbf{L}}_0^{\mathrm{T}},{\mathbf{L}}_1^{\mathrm{T}},\cdots ,{\mathbf{L}}_{\overline{M}-1}^{\mathrm{T}}]}^{\mathrm{T}}\end{array}$$

(14)

where $\mathbf{G}\in {\mathbb{C}}^{\overline{M}\overline{N}\times (\overline{M}+\overline{N}-1)}$, and ${\mathbf{L}}_m=[{\mathbf{0}}_{\overline{N}\times m},{\mathbf{I}}_{\overline{N}},{\mathbf{0}}_{\overline{N}\times (\overline{M}-m-1)}]\in {\mathbb{C}}^{\overline{N}\times (\overline{M}+\overline{N}-1)}$ for $m=0,1,...,\overline{M}-1$. Thus, based on $\mathbf{G}$, a reduced dimensional transformation matrix can be constructed as $\mathbf{R}={({\mathbf{G}}^{\mathrm{H}}\mathbf{G})}^{(-\frac{1}{2})}{\mathbf{G}}^{\mathrm{H}}$, then $\overline{\mathbf{X}}$ in Equation (12) turns into:

$$\begin{array}{cc}\hfill \mathbf{Y}& ={({\mathbf{G}}^{\mathrm{H}}\mathbf{G})}^{(-\frac{1}{2})}{\mathbf{G}}^{\mathrm{H}}\mathbf{G}\mathbf{B}\mathbf{D}{\mathbf{\Lambda }}_{\phi }{\mathbf{S}}_r+\mathbf{R}\overline{\mathbf{N}}\hfill \\ & =\mathbf{F}\mathbf{B}\mathbf{D}{\mathbf{\Lambda }}_{\phi }{\mathbf{S}}_r+{\overline{\mathbf{N}}}_d\hfill \end{array}$$

(15)

where $\mathbf{Y}$ and ${\overline{\mathbf{N}}}_d=\mathbf{R}\overline{\mathbf{N}}$ are the new reduced dimensional data matrix and the new noise matrix, respectively. $\mathbf{F}={({\mathbf{G}}^{\mathrm{H}}\mathbf{G})}^{(\frac{1}{2})}\in {\mathbb{C}}^{(\overline{M}+\overline{N}-1)\times (\overline{M}+\overline{N}-1)}$ can be directly calculated as:

$$\begin{array}{c}\hfill \mathbf{F}=\mathrm{diag}[1,\sqrt{2},...,\underset{|\overline{M}-\overline{N}|+1}{\underbrace{\min (\sqrt{\overline{M}},\sqrt{\overline{N}}),...,\min (\sqrt{\overline{M}},\sqrt{\overline{N}})}},....,\sqrt{2},1]\end{array}$$

(16)

By the transformation, the dimension of the data matrix is reduced from $\overline{M}\overline{N}\times J$ to $(\overline{M}+\overline{N}-1)\times J$. In addition, as $\mathbf{R}{\mathbf{R}}^{\mathtt{H}}={\mathbf{I}}_{\overline{M}+\overline{N}-1}$, the transformation does not bring in the additional spatial colored noises. In order to enlarge the array aperture by exploiting the noncircular signals, the new received data matrix $\mathbf{Y}\in {\mathbb{C}}^{(\overline{M}+\overline{N}-1)\times J}$ can be extended as $\mathbf{Z}={[{\mathbf{Y}}^{\mathtt{T}},{\mathbf{Y}}^{\mathtt{H}}]}^{\mathtt{T}}\in {\mathbb{C}}^{2(\overline{M}+\overline{N}-1)\times J}$. Since $\mathbf{F}$, $\mathbf{D}$ and ${\mathbf{S}}_r$ are real-valued matrices, $\mathbf{Z}$ is expressed as:

$$\begin{array}{c}\hfill \mathbf{Z}=\left[\begin{array}{c}\mathbf{F}\mathbf{B}\mathbf{D}{\mathbf{\Lambda }}_{\phi }\\ \mathbf{F}{\mathbf{B}}^{*}\mathbf{D}{\mathbf{\Lambda }}_{\phi }^{*}\end{array}\right]{\mathbf{S}}_r+\left[\begin{array}{c}{\overline{\mathbf{N}}}_d\\ {\overline{\mathbf{N}}}_d^{*}\end{array}\right]\end{array}$$

(17)

3.3. Joint Reweighted Sparse Representation-Based DOA Estimation Scheme

In order to successfully apply the sparse representation theory to estimate the DOAs, we bring in the SVD technique of $\mathbf{Z}$, i.e., $\mathbf{Z}=\mathbf{U}\mathbf{\Lambda }{\mathbf{V}}^{\mathtt{H}}$, where $\mathbf{\Lambda }=\mathrm{diag}({\omega }_1,{\omega }_2,\dots ,{\omega }_{\overline{M}+\overline{N}-1})$, and ${\omega }_1\ge {\omega }_2\ge \cdots \ge {\omega }_{\overline{M}+\overline{N}-1}$ are singular values. Let ${\mathbf{V}}_s$ be composed of the P vectors in $\mathbf{V}$ that correspond to $({\omega }_1,{\omega }_2,\dots ,{\omega }_P)$. With ${\mathbf{V}}_s$, the dimension of the observation matrix can be further reduced [9], which is conducive to the signal reconstruction, that is:

$$\begin{array}{c}\hfill {\mathbf{Z}}_v=\left[\begin{array}{c}\mathbf{F}\mathbf{B}\mathbf{D}{\mathbf{\Lambda }}_{\phi }{\mathbf{S}}_r{\mathbf{V}}_s\\ \mathbf{F}{\mathbf{B}}^{*}\mathbf{D}{\mathbf{\Lambda }}_{\phi }^{*}{\mathbf{S}}_r{\mathbf{V}}_s\end{array}\right]+\left[\begin{array}{c}{\overline{\mathbf{N}}}_d{\mathbf{V}}_s\\ {\overline{\mathbf{N}}}_d^{*}{\mathbf{V}}_s\end{array}\right]=\left[\begin{array}{c}\overline{\mathbf{B}}\overline{\mathbf{T}}\\ \tilde{\mathbf{B}}\tilde{\mathbf{T}}\end{array}\right]+\left[\begin{array}{c}{\overline{\mathbf{N}}}_{ds}\\ {\tilde{\mathbf{N}}}_{ds}\end{array}\right]\end{array}$$

(18)

where ${\mathbf{Z}}_v=\mathbf{Z}{\mathbf{V}}_s\in {\mathbb{C}}^{2(\overline{M}+\overline{N}-1)\times P}$, $\overline{\mathbf{B}}=\mathbf{F}\mathbf{B}$, $\overline{\mathbf{T}}=\mathbf{D}{\mathbf{\Lambda }}_{\phi }{\mathbf{S}}_r{\mathbf{V}}_s$, $\tilde{\mathbf{B}}=\mathbf{F}{\mathbf{B}}^{*}$, $\tilde{\mathbf{T}}=\mathbf{D}{\mathbf{\Lambda }}_{\phi }^{*}{\mathbf{S}}_r{\mathbf{V}}_s$, ${\overline{\mathbf{N}}}_{ds}$ and ${\tilde{\mathbf{N}}}_{ds}$ are the new noise matrices. ${\mathbf{Z}}_v$ can be divided into the following two parts; the first $(\overline{M}+\overline{N}-1)$ rows in ${\mathbf{Z}}_v$ make up ${\mathbf{Z}}_{v1}\in {\mathbb{C}}^{(\overline{M}+\overline{N}-1)\times P}$, and the remaining rows compose ${\mathbf{Z}}_{v2}\in {\mathbb{C}}^{(\overline{M}+\overline{N}-1)\times P}$, namely ${\mathbf{Z}}_v={[{\mathbf{Z}}_{v1}^{\mathtt{T}},{\mathbf{Z}}_{v2}^{\mathtt{T}}]}^{\mathtt{T}}$. By constructing two complete dictionaries ${\overline{\mathbf{B}}}_{\widehat{\theta }}=\mathbf{F}[\mathbf{b}({\widehat{\theta }}_1),\mathbf{b}({\widehat{\theta }}_2),\dots ,\mathbf{b}({\widehat{\theta }}_L)]$ and ${\tilde{\mathbf{B}}}_{\widehat{\theta }}=\mathbf{F}[{\mathbf{b}}^{*}({\widehat{\theta }}_1),{\mathbf{b}}^{*}({\widehat{\theta }}_2),\dots ,{\mathbf{b}}^{*}({\widehat{\theta }}_L)]$ with ${\left\{{\widehat{\theta }}_l\right\}}_{l=1}^L(L\gg P)$ being the discretized sampling grids of all potential DOAs, the corresponding sparse representation models can be separately formulated as:

$$\begin{array}{cc}\hfill {\mathbf{Z}}_{v1}& ={\overline{\mathbf{B}}}_{\widehat{\theta }}{\overline{\mathbf{T}}}_{\widehat{\theta }}+{\overline{\mathbf{N}}}_{ds}\hfill \\ \hfill {\mathbf{Z}}_{v2}& ={\tilde{\mathbf{B}}}_{\widehat{\theta }}{\tilde{\mathbf{T}}}_{\widehat{\theta }}+{\tilde{\mathbf{N}}}_{ds}\hfill \end{array}$$

(19)

where ${\overline{\mathbf{T}}}_{\widehat{\theta }}\in {\mathbb{C}}^{L\times P}$ and ${\tilde{\mathbf{T}}}_{\widehat{\theta }}\in {\mathbb{C}}^{L\times P}$ hold the same row support with $\overline{\mathbf{T}}$ and $\tilde{\mathbf{T}}$, respectively. For the two models in Equation (19), if they are solved independently by conventional sparse representation methods that are based on the $l_1$-norm penalty [9], the signal noncircularity cannot be utilized to extend the array aperture. In order to make the best of signal noncircularity, we design a joint sparse vector $\mathbf{r}\in {\mathbb{C}}^{L\times 1}$ whose k-th element is $\mathbf{r}(k)=\sqrt{{({\overline{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k))}^2+{({\tilde{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k))}^2}$, then the sparsities of ${\overline{\mathbf{T}}}_{\widehat{\theta }}$ and ${\tilde{\mathbf{T}}}_{\widehat{\theta }}$ are combined. In $\mathbf{r}(k)$, ${\overline{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k)$ and ${\tilde{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k)$ are defined as ${\overline{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k)=\parallel {\overline{\mathbf{T}}}_{\widehat{\theta }}(k,:){\parallel }_2$ and ${\tilde{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k)=\parallel {\tilde{\mathbf{T}}}_{\widehat{\theta }}(k,:){\parallel }_2$, respectively. More specifically, ${\overline{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k)$ is equal to the $l_2$-norm of the k-th row in ${\overline{\mathbf{T}}}_{\widehat{\theta }}$; similarly, ${\tilde{\mathbf{T}}}_{\widehat{\theta }}^{(l_2)}(k)$ is equal to the $l_2$-norm of the k-th row in ${\tilde{\mathbf{T}}}_{\widehat{\theta }}$. Let $\widehat{\mathbf{r}}$ represent the solution vector of the sparse signal reconstruction, $\widehat{\mathbf{r}}(k)\ge \mathbf{r}(k)$, $k=1,2,\dots ,L$. As a result, based on the $l_1$-norm constrained minimization, a joint sparse representation scheme is designed as follows:

$$\min \parallel \widehat{\mathbf{r}}{\parallel }_1,\mathrm{s}.\mathrm{t}.\parallel {\mathbf{Z}}_{v1}-{\overline{\mathbf{B}}}_{\widehat{\theta }}{\overline{\mathbf{T}}}_{\widehat{\theta }}{\parallel }_F^2\le {\eta }_1\parallel {\mathbf{Z}}_{v2}-{\tilde{\mathbf{B}}}_{\widehat{\theta }}{\tilde{\mathbf{T}}}_{\widehat{\theta }}{\parallel }_F^2\le {\eta }_2$$

(20)

where ${\eta }_1$ and ${\eta }_2$ are regularization parameters that set the error amount. To make the $l_1$-norm penalty better approximate the $l_0$-norm penalty, we introduce the reweighted matrix in the following. Let ${\mathbf{U}}_n$ be the noise subspace and composed of the $(\overline{M}+\overline{N}-1-P)$ vectors in ${\mathbf{U}}_n$ that correspond to $({\omega }_{P+1},{\omega }_{P+2},\dots ,{\omega }_{\overline{M}+\overline{N}-1})$. According to the subspace principle [13], for the true target ${\theta }_p$, ${[{({\omega }_{tp}{\omega }_{rp}{e}^{j{\phi }_p}\mathbf{F}\mathbf{b}({\theta }_p))}^{\mathtt{T}},{({\omega }_{tp}{\omega }_{rp}{e}^{-j{\phi }_p}\mathbf{F}\mathbf{b}{({\theta }_p)}^{*})}^{\mathtt{T}}]}^{\mathtt{T}}$ in $\mathbf{Z}$ is orthogonal to the noise subspace ${\mathbf{U}}_n$, that is:

$$\begin{array}{cc}\hfill Q({\theta }_p,{\phi }_p,{\omega }_{tp},{\omega }_{rp})& =[{\omega }_{tp}{\omega }_{rp}{e}^{j{\phi }_p}{\omega }_{tp}{\omega }_{rp}{e}^{-j{\phi }_p}]\overline{\mathbf{Q}}({\theta }_p)\left[\begin{array}{c}{\omega }_{tp}{\omega }_{rp}{e}^{j{\phi }_p}\\ {\omega }_{tp}{\omega }_{rp}{e}^{-j{\phi }_p}\end{array}\right]\to 0\hfill \end{array}$$

(21)

where:

$$\begin{array}{c}\hfill \overline{\mathbf{Q}}({\theta }_p)={\left[\begin{array}{c}\mathbf{F}\mathbf{b}({\theta }_p){\mathbf{0}}_{(\overline{M}+\overline{N}-1)\times 1}\\ {\mathbf{0}}_{(\overline{M}+\overline{N}-1)\times 1}\mathbf{F}{\mathbf{b}}^{*}({\theta }_p)\end{array}\right]}^{\mathtt{H}}{\mathbf{U}}_n{\mathbf{U}}_n^{\mathtt{H}}\left[\begin{array}{c}\mathbf{F}\mathbf{b}({\theta }_p){\mathbf{0}}_{(\overline{M}+\overline{N}-1)\times 1}\\ {\mathbf{0}}_{(\overline{M}+\overline{N}-1)\times 1}\mathbf{F}{\mathbf{b}}^{*}({\theta }_p)\end{array}\right]\end{array}$$

(22)

Since $Q({\theta }_p,{\phi }_p,{\omega }_{tp},{\omega }_{rp})\to 0$, ${\omega }_{tp}$, ${\omega }_{rp}$ and $e^{j{\phi }_p}$ are all non-zero scalars and the rank of ${\mathbf{U}}_n{\mathbf{U}}_n^{\mathtt{H}}$ satisfies $\mathrm{rank}({\mathbf{U}}_n{\mathbf{U}}_n^{\mathtt{H}})>2$, for the true DOA ${\theta }_p$, the rank reduction of $\overline{\mathbf{Q}}({\theta }_p)$ will take place, that is $\det [\overline{\mathbf{Q}}({\theta }_p)]\to 0$. Consequently, a reweighted vector is designed as:

$$\begin{array}{c}\hfill \mathbf{w}(i)=\det [\overline{\mathbf{Q}}({\widehat{\theta }}_i)]\end{array}$$

(23)

for $i=1,2,\dots ,L$. Thus, the reweighted matrix is:

$$\begin{array}{c}\hfill {\mathbf{W}}_r=\mathrm{diag}(\mathbf{w})/\max (\mathbf{w})\end{array}$$

(24)

where ${\mathbf{W}}_r\in {\mathbb{C}}^{L\times L}$ is a diagonal matrix. Large weights in ${\mathbf{W}}_r$ can punish the entries that are more likely to be zeros in $\mathbf{w}$, while small weights reserve the larger entries. Hence, ${\mathbf{W}}_r$ can enhance the solution of the sparse recovery problem in Equation (20). Then, for DOA estimation in the presence of mutual coupling, the noncircular source-based joint reweighted sparse representation framework transforms into:

$$\min \parallel {\mathbf{W}}_r\widehat{\mathbf{r}}{\parallel }_1,\mathrm{s}.\mathrm{t}.\parallel {\mathbf{Z}}_{v1}-{\overline{\mathbf{B}}}_{\widehat{\theta }}{\overline{\mathbf{T}}}_{\widehat{\theta }}{\parallel }_F\le \sqrt{{\eta }_1}\parallel {\mathbf{Z}}_{v2}-{\tilde{\mathbf{B}}}_{\widehat{\theta }}{\tilde{\mathbf{T}}}_{\widehat{\theta }}{\parallel }_F\le \sqrt{{\eta }_2}$$

(25)

Effective SOC (second order cone) programming software packages, such as SeDuMi [27] and CVX [28], can be used to solve Equation (25). As a result, by plotting $\widehat{\mathbf{r}}$, the DOA estimation is achieved.

4. Related Remarks

Remark 1. 

In the problem of the sparse signal reconstruction that is based on the $l_1$-norm minimization, the chosen values of the regularization parameters ${\eta }_1$ and ${\eta }_2$ are important for the estimation accuracy. In the proposed algorithm, noise matrix $\mathit{N}$ in Equation (4) is complex white Gaussian with zero mean. By mutual coupling elimination and reduced dimensional transformation, the corresponding selection matrix Γ and transformation matrix $\mathit{R}$ transform $\mathit{N}$ into ${\overline{\mathit{N}}}_d$. According to the designed matrix structure of Γ and $\mathit{R}$, $\mathbf{\Gamma }{\mathbf{\Gamma }}^{\mathit{H}}={\mathbf{I}}_{\overline{M}\overline{N}}$ and $\mathit{R}{\mathit{R}}^{\mathit{H}}={\mathbf{I}}_{\overline{M}+\overline{N}-1}$ can be obtained.

Based on the invariance speciality of linear transformation in an asymptotic normal distribution [29], if n is asymptotic normal with zero-mean and covariance matrix ${\sigma }^2{\mathbf{I}}_{MN}$, $\mathit{R}\mathbf{\Gamma }\mathit{n}$ complies with a zero-mean asymptotic normal distribution with covariance matrix $\mathit{R}\mathbf{\Gamma }({\sigma }^2{\mathbf{I}}_{MN}){(\mathit{R}\mathbf{\Gamma })}^{\mathit{H}}={\sigma }^2{\mathbf{I}}_{\overline{M}+\overline{N}-1}$. Therefore, ${\overline{\mathit{N}}}_d=[{\overline{\mathit{n}}}_{d1},{\overline{\mathit{n}}}_{d2},\dots ,{\overline{\mathit{n}}}_{dJ}]$ in Equation (15) is a zero-mean complex Gaussian white noise matrix.

In [9], it has been verified that if the noise ${\overline{\mathit{n}}}_d$ is independent and identically distributed (i.i.d.) Gaussian, then $\parallel {\overline{\mathit{N}}}_{ds}{\parallel }_F^2$ has approximately a chi-square distribution with $(\overline{M}+\overline{N}-1)P$ degrees of freedom upon normalization by the variance of ${\overline{\mathit{n}}}_d$, where ${\overline{\mathit{N}}}_{ds}={\overline{\mathit{N}}}_d{\mathit{V}}_s$ with ${\mathit{V}}_s$ being the signal subspace of $\mathit{Z}$ [9]. As a result, ${\eta }_1$ and ${\eta }_2$ can be chosen as the upper limit value of $\parallel {\overline{\mathit{N}}}_{ds}{\parallel }_F^2$ and $\parallel {\tilde{\mathit{N}}}_{ds}{\parallel }_F^2$ upon ${\sigma }^2$, with a high probability $1-\epsilon$ confidence interval, and $\epsilon =0.001$ is enough. In addition, ${\sigma }^2$ can be estimated by the average of the squares of $2(\overline{M}+\overline{N}-1)P$ smallest singular values of $\mathit{Z}$. By using MATLAB software, the function $chi2inv(1-\epsilon ,(\overline{M}+\overline{N}-1)P){\sigma }^2$ can be used to compute the regularization parameters.

Remark 2. 

The computational complexity of the proposed algorithm mainly focuses on eliminating the effect of mutual coupling, constructing the reweighted matrix and obtaining the sparse solution from Equation (25), which require $O\left\{\overline{M}\overline{N}MNJ\right\}$, $O\left\{[4(\overline{M}+\overline{N}-1)(2(\overline{M}+\overline{N}-1)-P)+4(\overline{M}+\overline{N}-1)(2(\overline{M}+\overline{N}-1)-P)+8(\overline{M}+\overline{N}-1)]\overline{L}\right\}$ and $O\left\{{\overline{L}}^{3}P\right\}$ calculation burden, respectively, where $\overline{L}$ is the number of the discretized sampling grids. Therefore, the total computational complexity of the proposed algorithm is $O\{\overline{M}\overline{N}MNJ+[4(\overline{M}+\overline{N}-1)(2(\overline{M}+\overline{N}-1)-P+1)]\overline{L}+{\overline{L}}^{3}P\}$. Although the constructions of the extended data matrix and the reweighted matrix increase the computational complexity, they enlarge the virtual array aperture and enhance the solution of the sparse reconstruction. As a result, the angle estimation performance of the proposed method is improved.

Remark 3. 

The prior knowledge of the number of emitting sources is important. This is because in the absence of a perfect knowledge of the target number P, either adding spurious sources or missing actual signals may lead to the deviations of the signal and the noise subspaces, which further cause the deviations of extended observation matrix ${\mathit{Z}}_v$ in Equation (18) and reweighted matrix ${\mathit{W}}_r$ in Equation (24). Therefore, the selected value of P affects the performance of the proposed algorithm. Fortunately, when P is unknown in practical applications, some effective methods can be used to estimate it, such as the Akaike information criterion (AIC) and the minimum description length (MDL) [30]. In the proposed algorithm, the number of emitting sources P is assumed to be known.

Remark 4. 

After performing the mutual coupling elimination and the reduced dimensional transformation, the received data matrix $\mathit{X}$ in Equation (4) turns into $\mathit{Y}$ in Equation (15), and the number of effective antenna elements reduces from $(M+N-1)$ to $(\overline{M}+\overline{N}-1)$. Then, by using the real-valued signals that satisfy ${\mathit{S}}_r={\mathit{S}}_r^{*}$ extracted from the noncircular signals, the extended matrix $\mathit{Z}$ with the dimension $2(\overline{M}+\overline{N}-1)\times J$ is constructed in Equation (17). It implies that the extended data matrix $\mathit{Z}$ corresponds to a virtual array, whose virtual antenna number is twice that of the array corresponding to $\mathit{Y}$. As the exploitation of the noncircularity of emitted signals enlarges the effective array aperture, the more diversity gain can be obtained in the proposed algorithm. In addition, the reweighted matrix enhances the sparse solution. Thus, the proposed algorithm considerably improves DOA estimation performance in the presence of mutual coupling.

Remark 5. 

Mutual coupling coefficients are related to the distance between two antennas. For uniform linear arrays (ULAs) of the considered MIMO radar system in the proposed algorithm, the mutual coupling coefficients between two antenna elements satisfy $c_{t(q,m)}=c_{t(m,q)}=c_{t|q-m|}$ for the transmit array and $c_{r(q^{\prime },n)}=c_{r(n,q^{\prime })}=c_{r|q^{\prime }-n|}$ for the receive array [22], in which $1\le q,m\le M$ and $1\le q^{\prime },n\le N$. Consequently, all of the effects of mutual coupling in ULA can be modeled as a banded symmetric Toeplitz matrix in Equation (3) [23], based on which selection matrices are constructed in Equation (8) to eliminate the mutual coupling. Mutual coupling matrix in Equation (3) is modeled under the circumstance of symmetric coupling in ULA. Hence, the proposed algorithm is not suitable for asymmetric coupling.

5. Simulation Results

In this section, using the ESPRIT-like [14] algorithm, the $l_1$-SVD [15] algorithm and the $l_1$-SRDML [16] algorithm for comparison, some simulation experiments are implemented to demonstrate the efficiency and the advantages of the proposed algorithm. The signal-to-noise ratio (SNR) is defined as SNR = $10{\log }_{10}{(|\left|\mathbf{C}\mathbf{A}\mathbf{S}\right||}_F^2/{\left|\right|\mathbf{N}\left|\right|}_F^2$). Let ${\widehat{\theta }}_p^i$ be the estimation of the true DOA ${\theta }_p$ for the i-th Monte Carlo trial; the root mean square error (RMSE) of angle estimation is defined as:

$$(1/P){\sum }_{p=1}^P\sqrt{(1/Q){\sum }_{i=1}^Q{({\widehat{\theta }}_p^i-{\theta }_p)}^2}$$

(26)

where Q is the total number of the Monte Carlo trials, which is $Q=200$ in the simulations. Consider a narrowband monostatic MIMO radar system, whose arrays are both half-wavelength spaced ULAs with the effects of mutual coupling, and the numbers of transmit and receive antennas are M and N, respectively. Let $K=k+1$ be the nonzero mutual coupling coefficient number; two different mutual coupling cases of the transmitter and the receiver are considered in the following simulations: (1) $K=2$ with the nonzero mutual coupling coefficients being $[c_{t0},c_{t1}]=[1,0.1844+j0.0427]$ and $[c_{r0},c_{r1}]=[1,0.0522-j0.1049]$; (2) $K=3$ with $[c_{t0},c_{t1},c_{t2}]=[1,0.6+j0.2,0.02+j0.13]$ and $[c_{r0},c_{r1},c_{r2}]=[1,0.5+j0.1,0.01+j0.05]$. The number of uncorrelated targets P is assumed to be known. Besides, the confidence interval is set to $0.99$, and the discretized grids are uniform with ${0.1}^{\circ }$ sampling from $-{90}^{\circ }$ to ${90}^{\circ }$, for the proposed method, as well as $l_1$-SVD and $l_1$-SRDML algorithms.

When performing the experiments, we solve the signal reconstruction problem of the noncircular source-based joint reweighted sparse representation framework in Equation (25). The sparse solution vector $\widehat{\mathbf{r}}$ is obtained by using the SOC programming software packages. When there are true DOAs of targets located at certain points of the complete dictionary, the corresponding elements in $\widehat{\mathbf{r}}$ are non-zero, and the rest are zero. Therefore, the true DOA locations in dictionary correspond to the largest P values of $|\widehat{\mathbf{r}}|$. In the simulations, the functional form $10{\log }_{10}[|\widehat{\mathbf{r}}|/\max (|\widehat{\mathbf{r}}|)]$ is used to plot the spatial spectrum of angle estimation. Then, by finding P peaks in the spectrum, DOA estimation is achieved.

Figure 3 depicts the spatial spectrum of the proposed method for the number of targets being $P=3$ and $P=5$, respectively, where $M=N=6$, $J=200$, SNR is fixed at SNR = 0 dB and mutual coupling Case (1) is considered. For $P=3$, the DOAs of the uncorrelated targets are ${\theta }_1=-{10}^{\circ }$, ${\theta }_2=0^{\circ }$, ${\theta }_3={10}^{\circ }$. For $P=5$, they are ${\theta }_1=-{20}^{\circ }$, ${\theta }_1=-{10}^{\circ }$, ${\theta }_1=0^{\circ }$, ${\theta }_2={10}^{\circ }$ and ${\theta }_3={20}^{\circ }$, respectively. From Figure 3, it can be observed that the spatial spectrum peaks of the proposed method are sharp, and the sidelobe suppression is low. In addition, for different target numbers P, the spectrum maintains the accuracy of DOA estimation. This indicates that the proposed method is effective to perform DOA estimation accurately.

Figure 4 shows the RMSE of DOA estimation versus SNR in different methods with mutual coupling Case (1), where $M=N=6$, $J=200$ and three targets with ${\theta }_1=-{10}^{\circ }$, ${\theta }_2=0^{\circ }$, ${\theta }_3={10}^{\circ }$ are considered. Additionally, the values of RMSE are computed by Equation (26) with Q trials. As can be seen in Figure 4, $l_1$-SRDML owns a lower RMSE than the ESPRIT-like and $l_1$-SVD algorithms, because the deterministic maximum likelihood procedure improves the estimation accuracy. In addition, the proposed method provides the best DOA estimation performance in all SNR regions, for the reason that the noncircularity-based joint reweighted sparse representation scheme enlarges the virtual array aperture and enhances the sparse solution.

Figure 5 illustrates the RMSE of DOA estimation versus SNR in different methods with mutual coupling Case (2), where $M=N=7$, $J=200$, and there are three uncorrelated targets located at ${\theta }_1=-{15}^{\circ }$, ${\theta }_2=0^{\circ }$, ${\theta }_3={15}^{\circ }$. From Figure 5, it is clear that the performance of $l_1$-SVD is inferior to the other algorithms, and the proposed method has the lowest RMSE in all analyzed methods. This means that when the number of nonzero mutual coupling coefficients increases, the proposed method still has superior DOA estimation performance.

Figure 6 shows the RMSE of DOA estimation versus snapshots of different methods with mutual coupling Case (1), where $M=N=6$, SNR = 0 dB, there are three uncorrelated targets with DOAs being ${\theta }_1=-{11.5}^{\circ }$, ${\theta }_2=0^{\circ }$, ${\theta }_3={11.5}^{\circ }$ and the number of snapshots varies from $J=50$ to $J=550$. It can be seen from Figure 6 that the proposed method performs well in the case of fewer snapshots, and with obvious superiority over the ESPRIT-like, $l_1$-SVD and $l_1$-SRDML algorithms, the proposed method performs the best in all pf the ranges of snapshots on account of the application of the reweighted matrix and the noncircularity of emitted signals.

Figure 7 demonstrates the target resolution probability of different methods versus SNR with mutual coupling Case (1), where $M=N=6$, $J=100$ and the DOAs of three targets are ${\theta }_1=-{20}^{\circ }$, ${\theta }_2=0^{\circ }$ and ${\theta }_3={20}^{\circ }$, respectively. They can be regarded as successfully detected when all of their absolute DOA estimation errors are within ${0.1}^{\circ }$. It can be observed that when SNR is high enough, all methods provide $100\%$ target resolution probability. Owing to the increased number of virtual array elements, more diversity gain is obtained in the proposed algorithm. Consequently, the proposed method provides higher resolution probability than the ESPRIT-like, $l_1$-SVD and $l_1$-SRDML algorithms in all SNR regions, as shown in the simulation results of Figure 7.

6. Conclusions

In this paper, we have proposed the noncircular source-based sparse representation algorithm for DOA estimation in MIMO radar with mutual coupling. On the basis of the error matrix structure and the properties of noncircular signals, the proposed method eliminates the effect of mutual coupling and obtains the extended observation matrix. Then, the joint reweighted sparse representation scheme is constructed to achieve the DOA estimation. The computational complexity of the proposed method has been analyzed, and the simulation results have verified that in the presence of mutual coupling, the proposed algorithm provides higher resolution and better angle estimation performance than the ESPRIT-like, $l_1$-SVD and $l_1$-SRDML algorithms.