Computationally Efficient Direction Finding for a Mixture of Circular and Strictly Noncircular Sources with Uniform Rectangular Arrays

Abstract

In this paper, a novel two-dimensional (2D) direction-of-arrival (DOA) estimation algorithm for the mixed circular and strictly noncircular sources is proposed. A general array model with a mixture of signals is firstly built based on uniform rectangular arrays (URAs), and then, the approach, which uses the rank-reduction-based ROOT-MUSIC, can solve 2D DOA estimation problem. Besides, the theoretical error of the proposed algorithm, a criterion of the performance for evaluation, is analyzed by the first-order Taylor expression using second-order statistics. As verified by the simulation results, a better DOA estimation performance and a lower computational complexity are achieved by the proposed algorithm than the existing methods resorting to the noncircularity of the incoming signals.

1. Introduction

Array signal processing has been widely used in the fields of sonar, radar, wireless communication, etc, and many excellent algorithms have been developed in the past few years. Especially the well-known multiple signal classification (MUSIC) algorithm has not only been applied in its time-reversal (TR) form to active location [1,2,3,4], but has also been widely used in direction-of-arrival (DOA) estimation (DOA is the research of passive location). In the last few years, DOA estimation became an important research branch of array signal processing [5,6,7]. Especially two-dimensional (2D) direction-of-arrival (DOA) estimation with different structured arrays, such as L-shaped uniform linear arrays (ULAs) [8,9,10], two-parallel ULAs [11,12,13], and uniform rectangular arrays (URAs) [14,15,16,17,18], has received much attention in past years. For URAs, the well-known multiple signal classification (MUSIC) algorithm can be used for 2D DOA estimation directly [16]; however, its computational complexity is very high. To overcome this problem, two efficient 2D DOA estimation methods have been proposed [17,18]. In [17], the reduced-dimension MUSIC algorithm was proposed, which reduced the computational complexity, and the 2D DOA estimation performance was very close to the 2D-MUSIC method. A preprocessing transformation matrix was introduced in [18], which transformed both the complex-valued covariance matrix and the complex-valued search vector into real-valued ones, then the 2D DOA estimation problem was decoupled into two successive real-valued one-dimensional (1D) DOA estimation problems with real-valued computations only. However, only the covariance matrix was considered, which characterizes the circular Gaussian distribution in the above 2D DOA methods.

In recent years, the issue of utilizing noncircular information has been attracting increasing attention. Abeida et al. presented a theoretical analysis of the resolution of the conventional and noncircular MUSIC algorithms and proved that the noncircular MUSIC algorithm for the threshold array signal-to-noise ratios are very sensitive to the noncircularity phase separation of the sources [19]. Pascal et al. introduced the optimal widely linear (WL) minimum variance distortionless response (MVDR) beamformer for the reception of an unknown signal of interest (SOI) corrupted by potentially second-order (SO) noncircular background noise and interference [20]. Wan et al. [21] proposed an algorithm that utilized the noncircular characteristic to solve the DOA estimation of coherently distributed sources. Because the covariance matrix and the elliptic covariance matrix (which is also named the complementary covariance or pseudo-covariance) are used in non-circular signals direction-finding method simultaneously, the DOA estimation performance can be improved greatly by exploiting the noncircularity information. Gan et al. studied the non-circular characteristics of the signals and proposed an automatically-paired 2D DOAs estimation method based on non-circular signals [22]. Based on strictly non-circular signals, Steinwandt et al. proposed the high-resolution $R-D$ NCstandard ESPRIT and the $R-D$ NC unitary ESPRIT algorithms, which are applicable to arbitrary shift-invariant $R-D$ antenna arrays and do not require a centro-symmetric array structure [23]. In [24], an extended rank reduction (ERARE) method was introduced for noncircular sources based on two-parallel ULAs, which made the estimation more accurate than that in [12].

However, in modern wireless communications, a usual situation is that some users send circular signals such as quadrature phase shift keying (QPSK) signals, but others send non-circular ones such as binary phase shift keying (BPSK) signals. Therefore, the signals impinging to the array may be mixed ones. As for the mixed signals situation, a key issue is how to estimate and distinguish the circular and non-circular signals. In [25,26], the mixed signals situation issue has been studied, and the corresponding methods based on 1D arrays have been proposed. Gao et al. [25] combined the observed data and their conjugate counterparts to construct two 1D DOA estimators for detecting the circular and non-circular signals, but the DOA estimation performance of the method degraded seriously in the condition of a small separating angle. Liu et al. proposed an improved algorithm in [26], which detected the circular and non-circular signals by using the difference between the circularity of the mixed sources. However, to the best of our knowledge, few research works have reported the 2D DOA estimation problem for mixed circular and non-circular signals. In [27], based on two-parallel uniform linear arrays (ULAs), Chen proposed an effective algorithm that combined the rank reduction method and MUSIC algorithm to form four 1D DOA estimators for solving the 2D mixed signals situation. Nevertheless, the computational complexity of the algorithm in [27] is still high, and the theoretical performance analysis is never mentioned in this research.

In this paper, we study the 2D DOA estimation problem based on uniform rectangular arrays (URAs) using the rank-reduction-based ROOT-MUSIC method and the theoretical performance analysis of the proposed algorithm. Firstly, we establish an array model with mixed circular and noncircular sources with URAs; secondly, to avoid seeking the peak of the spectrum and reduce the computation load, a novel algorithm based on ROOT-MUSIC and the rank-reduction method is proposed to solve the 2D DOA estimation issue; finally, the theoretical error of the proposed algorithm is derived as a benchmark. Particularly, the paper mainly discusses the uncorrelated signals impinging upon the array. If we utilize some decorrelation methods such as the spatial smoothing technologies [28,29] or four-order cumulants-based Toeplitz matrices reconstruction (FOC-TMR) method [13] to preprocess the correlated signals and obtain the de-correlated matrices, the proposed method can also be generalized to the case of correlated sources.

The rest of this paper is organized as follows. Section 2 presents the array signal model. The description of the proposed algorithm is introduced in Section 3. The theoretical error analysis of the proposed algorithm is derived in Section 4. Finally, the simulation results are given in Section 5, and conclusions are drawn in Section 6.

Notations: ${(·)}^{*}$, ${(·)}^T$ and ${(·)}^H$ represent conjugation, transpose and conjugate transpose. $E[·]$ is the expectation operation; $diag(·)$ and $blkdiag(·)$ stands for the diagonalization and block diagonalization operation, respectively; ${\mathbf{I}}_{\omega }$ denotes the $\omega \times \omega$ dimensional identity matrix; det$[·]$ indicates the determinant of a matrix; $arg(·)$ is the phase angle operator. Expect ${\mathbf{E}}_n$, ${\widehat{\mathbf{E}}}_n$, ${\mathbf{E}}_{n1}$, ${\widehat{\mathbf{E}}}_{n1}$, ${\mathbf{E}}_{n2}$ and ${\widehat{\mathbf{E}}}_{n2}$ representing the corresponding noise subspace; other variables have index n such as ${(·)}_n$, which denote ones related to the non-circular signal. ${(·)}_c$ indicates a variable associated with the circular signal.

2. Problem Formulation

In this paper, we suppose that the number of signals is known or is estimated by the existing number detection technique in advance [30]. As illustrated in Figure 1, consider that K uncorrelated far-field narrowband signals $s_k\left(t\right)$ (k = 1, 2, ..., K) impinging upon the array with $K_n$ noncircular signals $s_{n,k_n}\left(t\right)(k_n=1,2,...,K_n)$ and $K_c$ circular signals $s_{c,k_c}\left(t\right)(k_c=1,2,...,K_c)$, from directions $({\theta }_k,{\beta }_k)$, $k=1,2,...,K$, where $K=K_n+K_c$. The array is composed of uniform rectangular arrays (URAs) with $N\times M$ omnidirectional sensors spaced by $d_x$ in the x-axis direction and $d_y$ in the y-axis direction. $\lambda$ is the wavelength of the incident waves, and $d_x$ = $d_y$ = $\lambda /2$. The additive noises of the URAs are circular Gaussian with zero mean and variance ${\sigma }^2$, which are uncorrelated with the impinging signals. The received data vectors of the URAs at sample t can be expressed as:

$$\begin{array}{}\mathrm{(1)}& [H]{\mathbf{x}}_1\left(t\right)=\mathbf{A}{\mathbf{G}}_1\tilde{\mathbf{s}}\left(t\right)+{\mathbf{n}}_1\left(t\right)\mathrm{(2)}& {\mathbf{x}}_2\left(t\right)=\mathbf{A}{\mathbf{G}}_2\tilde{\mathbf{s}}\left(t\right)+{\mathbf{n}}_2\left(t\right)& ⋮\mathrm{(3)}& {\mathbf{x}}_N\left(t\right)=\mathbf{A}{\mathbf{G}}_N\tilde{\mathbf{s}}\left(t\right)+{\mathbf{n}}_N\left(t\right)\end{array}$$

where ${\mathbf{x}}_1\left(t\right)=[x_{11}\left(t\right),x_{12}\left(t\right),\dots ,x_{1M}\left(t\right)]$, ${\mathbf{x}}_2\left(t\right)=[x_{21}\left(t\right),x_{22}\left(t\right),\dots ,x_{2M}\left(t\right)]$, ... , ${\mathbf{x}}_N\left(t\right)=[x_{N1}\left(t\right),x_{N2}\left(t\right),\dots ,x_{NM}\left(t\right)]$. $\mathbf{A}=[\mathbf{a}\left({\theta }_1\right),\mathbf{a}\left({\theta }_2\right),\dots ,\mathbf{a}\left({\theta }_K\right)]$ is the steering matrix with each column denoted by $\mathbf{a}\left({\theta }_k\right)={[a_1\left({\theta }_k\right),\dots ,a_M\left({\theta }_k\right)]}^T$, $k=1,2,\dots ,K$, with $a_i\left({\theta }_k\right)=e^{-j\frac{2\pi }{\lambda }{d}_x(i-1)cos{\theta }_k}$, $i=1,2,\dots ,M$. ${\mathbf{G}}_{\zeta }$, $\zeta =1,2,\dots ,N$, is termed the steering element matrix given by ${\mathbf{G}}_{\zeta }=diag[{\upsilon }_{\zeta }\left({\beta }_1\right),{\upsilon }_{\zeta }\left({\beta }_2\right),\dots ,{\upsilon }_{\zeta }\left({\beta }_K\right)]$ with ${\upsilon }_{\zeta }\left({\beta }_k\right)=e^{j\frac{2\pi }{\lambda }(\zeta -1)d_{y}cos{\beta }_k}$. ${\mathbf{n}}_1\left(t\right)={[n_{1,1}\left(t\right),n_{1,2}\left(t\right),\dots ,n_{1,M}\left(t\right)]}^T$, ${\mathbf{n}}_2\left(t\right)={[n_{2,1}\left(t\right),n_{2,2}\left(t\right),\dots ,n_{2,M}\left(t\right)]}^T$, ... , and ${\mathbf{n}}_N\left(t\right)={[n_{N,1}\left(t\right),n_{N,2}\left(t\right),\dots ,n_{N,M}\left(t\right)]}^T$ indicate the circular Gaussian noise vectors of the URAs, respectively. $\tilde{\mathbf{s}}\left(t\right)$ is the mixed signal vector, which can be denoted as $\tilde{\mathbf{s}}\left(t\right)={[s_{n,1}\left(t\right),\dots ,s_{n,K_n}\left(t\right),s_{c,1}\left(t\right),\dots ,s_{c,K_c}\left(t\right)]}^T$, and we can see that there are $K_n$ noncircular and $K_c$ circular signals in it.

In practice, non-circularity and circularity are important properties of random variables; their concept directly comes from the geometrical interpretation of complex random variable. The signal would be called the circular source if its statistical characteristic has the rotational invariance characteristic, otherwise, it would be called the noncircular source. The work in [31] introduces the circularity and noncircularity in detail. Based on this, we only consider the rotational invariance characteristic of the first- and second-order statistical properties of the sources. For a complex random signal $s_k$, $k=1,2,...,K$, we defined $E[s_k]$, $E[s_k{s_k}^{*}]$ and $E[{s_k}^2]$ as the mean, the covariance and the elliptic covariance of the signal $s_k$, respectively. If the source’s first- and second-order statistical properties are rotational invariant for an arbitrary phase ${\phi }_k$ as follows:

$$E[s_k{e}^{j\phi }]=E[s_k]$$

(4)

$$E[s_k{e}^{j{\phi }_k}{\left(s_k{e}^{j{\phi }_k}\right)}^{*}]=E[s_k{s_k}^{*}]$$

(5)

$$E[s_k{e}^{j{\phi }_k}·s_k{e}^{j{\phi }_k}]=E[{s_k}^2]$$

(6)

The signal $s_k$ will be called the circular source. Conversely, the signal $s_k$ will be noncircular if the first- and second-order statistical properties are not rotational invariant.

Beyond that, reference [32] proposes a model to describe signal sources with arbitrary second-order non-circularity. The source $s_k$ is defined as follows:

$$s_k=e^{j{\phi }_k}\left(\sqrt{\frac{1+{\eta }_k}{2}}{s}_{I_k}+j\sqrt{\frac{1-{\eta }_k}{2}}{s}_{Q_k}\right),k=1,2,...K$$

(7)

where ${\phi }_k$ is the rotation phase, ${\eta }_k(0\le {\eta }_k\le 1)$ denotes the non-circularity coefficient and $s_{I_k}$ and $s_{Q_k}$ represent the in-phase and quadrature components of the complex signal $s_k$, respectively. Therefore, $s_k$ will represent a circular source if ${\eta }_k=0$, where the rotation phase ${\phi }_k$ is irrelevant and undetermined. If ${\eta }_k=1$, $s_k$ will represent a strictly non-circular signal.

Based on the above research, we defined the strictly non-circular signal as $s_{n,k_n}\left(t\right)=b_{n,k_n}{\overline{s}}_{n,k_n}\left(t\right)$, where ${\overline{s}}_{n,k_n}\left(t\right)$ is a real signal and $b_{n,k_n}=e^{j{\phi }_{k_n}}(k_n=1,2,\dots ,K_n)$ is an arbitrary phase shift for the signal. Due to the phase information of circular source being irrelevant, it can be represented as $s_{c,k_c}\left(t\right),k_c=1,2,\dots ,K_c$. Therefore, the mixed source signal vector can be modeled as $\tilde{\mathbf{s}}\left(t\right)=\mathbf{Bs}\left(t\right)$. $\mathbf{B}$ can be expressed as:

$$\mathbf{B}=diag\left[b_{n,1},\dots ,b_{n,K_n},\underset{K_c}{\underbrace{1,\dots ,1}}\right]=\left[\begin{array}{cc}{\mathbf{B}}_1& \mathbf{0}\\ \mathbf{0}& {\mathbf{B}}_2\end{array}\right]$$

(8)

where ${\mathbf{B}}_1=diag[b_{n,1},\dots ,b_{n,K_n}]$, ${\mathbf{B}}_2={\mathbf{I}}_{K_c}$, and $\mathbf{s}\left(t\right)$ is denoted as:

$$\mathbf{s}\left(t\right)={[{\overline{s}}_{n,1}\left(t\right),\dots ,{\overline{s}}_{n,K_n}\left(t\right),s_{c,1}\left(t\right),\dots ,s_{c,K_c}\left(t\right)]}^T$$

(9)

A new data vector $\mathit{f}\left(t\right)$ is defined by concatenating the received data vectors ${\mathbf{x}}_1\left(t\right)$, ${\mathbf{x}}_2\left(t\right)$, ... , and ${\mathbf{x}}_N\left(t\right)$ as follows:

$$\begin{array}{cc}\mathit{f}\left(t\right)\hfill & =\left[\begin{array}{c}{\mathbf{x}}_1\left(t\right)\\ {\mathbf{x}}_2\left(t\right)\\ ⋮\\ {\mathbf{x}}_N\left(t\right)\end{array}\right]=\left[\begin{array}{c}\mathbf{A}\left(\theta \right){\mathbf{G}}_1\left(\beta \right)\\ \mathbf{A}\left(\theta \right){\mathbf{G}}_2\left(\beta \right)\\ ⋮\\ \mathbf{A}\left(\theta \right){\mathbf{G}}_N\left(\beta \right)\end{array}\right]\mathbf{Bs}\left(t\right)+\left[\begin{array}{c}{\mathbf{n}}_1\left(t\right)\\ {\mathbf{n}}_2\left(t\right)\\ ⋮\\ {\mathbf{n}}_N\left(t\right)\end{array}\right]\hfill \\ & =\mathbf{C}(\theta ,\beta )\mathbf{Bs}\left(t\right)+\mathbf{n}\left(t\right)\hfill \end{array}$$

(10)

where $\mathbf{C}(\theta ,\beta )$ is the extend steering vector, and:

$$\mathbf{C}=[{\mathbf{C}}_1({\theta }_n,{\beta }_n){\mathbf{C}}_2({\theta }_c,{\beta }_c)]$$

(11)

In order to simplify the notation, the pair of angles $(\theta ,\beta )$ and t is omitted. In Equation (11),

$$\begin{array}{cc}{\mathbf{C}}_1\hfill & =\left[\begin{array}{ccc}\begin{array}{c}\mathbf{a}\left({\theta }_{n,1}\right){\upsilon }_1\left({\beta }_{n,1}\right)\\ \mathbf{a}\left({\theta }_{n,1}\right){\upsilon }_2\left({\beta }_{n,1}\right)\\ ⋮\\ \mathbf{a}\left({\theta }_{n,1}\right){\upsilon }_N\left({\beta }_{n,1}\right)\end{array}& \cdots & \begin{array}{c}\mathbf{a}\left({\theta }_{n,K_n}\right){\upsilon }_1\left({\beta }_{n,K_n}\right)\\ \mathbf{a}\left({\theta }_{n,K_n}\right){\upsilon }_2\left({\beta }_{n,K_n}\right)\\ ⋮\\ \mathbf{a}\left({\theta }_{n,K_n}\right){\upsilon }_N\left({\beta }_{n,K_n}\right)\end{array}\end{array}\right]\hfill \\ & =\left[\begin{array}{ccc}\begin{array}{c}{\mathbf{a}}_{n,1}{\upsilon }_{1_{n,1}}\\ {\mathbf{a}}_{n,1}{\upsilon }_{2_{n,1}}\\ ⋮\\ {\mathbf{a}}_{n,1}{\upsilon }_{N_{n,1}}\end{array}& \cdots & \begin{array}{c}{\mathbf{a}}_{n,K_n}{\upsilon }_{1_{n,K_n}}\\ {\mathbf{a}}_{n,K_n}{\upsilon }_{2_{n,K_n}}\\ ⋮\\ {\mathbf{a}}_{n,K_n}{\upsilon }_{N_{n,K_n}}\end{array}\end{array}\right]\hfill \end{array}$$

(12)

$$\begin{array}{cc}{\mathbf{C}}_2\hfill & =\left[\begin{array}{ccc}\begin{array}{c}\mathbf{a}\left({\theta }_{c,1}\right){\upsilon }_1\left({\beta }_{c,1}\right)\\ \mathbf{a}\left({\theta }_{c,1}\right){\upsilon }_2\left({\beta }_{c,1}\right)\\ ⋮\\ \mathbf{a}\left({\theta }_{c,1}\right){\upsilon }_N\left({\beta }_{c,1}\right)\end{array}& \cdots & \begin{array}{c}\mathbf{a}\left({\theta }_{c,K_c}\right){\upsilon }_1\left({\beta }_{c,K_c}\right)\\ \mathbf{a}\left({\theta }_{c,K_c}\right){\upsilon }_2\left({\beta }_{c,K_c}\right)\\ ⋮\\ \mathbf{a}\left({\theta }_{c,K_c}\right){\upsilon }_N\left({\beta }_{c,K_c}\right)\end{array}\end{array}\right]\hfill \\ & =\left[\begin{array}{ccc}\begin{array}{c}{\mathbf{a}}_{c,1}{\upsilon }_{1_{c,1}}\\ {\mathbf{a}}_{c,1}{\upsilon }_{2_{c,1}}\\ ⋮\\ {\mathbf{a}}_{c,1}{\upsilon }_{N_{c,1}}\end{array}& \cdots & \begin{array}{c}{\mathbf{a}}_{c,K_c}{\upsilon }_{1_{c,K_c}}\\ {\mathbf{a}}_{c,K_c}{\upsilon }_{2_{c,K_c}}\\ ⋮\\ {\mathbf{a}}_{c,K_c}{\upsilon }_{N_{c,K_c}}\end{array}\end{array}\right]\hfill \end{array}$$

(13)

${\mathbf{C}}_1$ is a $NM\times K_n$ matrix of noncircular signals and ${\mathbf{C}}_2$ is a $NM\times K_c$ matrix of circular signals.

As a classical procedure for existing noncircular DOA estimation algorithms [27,33,34], we can construct a new augmented data matrix $\stackrel{⌣}{\mathit{f}}$ by combining the vector $\mathit{f}$ and its conjugate counterpart ${\mathit{f}}^{*}$ as follows:

$$\stackrel{⌣}{\mathit{f}}=\left[\begin{array}{c}\mathit{f}\\ {\mathit{f}}^{*}\end{array}\right]=\left[\begin{array}{c}\mathbf{CBs}\\ {\mathbf{C}}^{*}{\mathbf{B}}^{*}{\mathbf{s}}^{*}\end{array}\right]+\left[\begin{array}{c}\mathbf{n}\\ {\mathbf{n}}^{*}\end{array}\right]=\stackrel{⌣}{\mathbf{C}}\stackrel{⌣}{\mathbf{s}}+\stackrel{⌣}{\mathbf{n}}$$

(14)

The procedure can extend the array virtually and enlarge the aperture of the array antenna, and the estimating precision of DOA can be improved by utilizing the noncircularity of signals. In Equation (14), $\stackrel{⌣}{\mathbf{C}}$ is a $2NM\times \left(K_n+2K_c\right)$ matrix, which contains the new steering vectors of the impinging sources. $\stackrel{⌣}{\mathbf{s}}$ is a $\left(K_n+2K_c\right)\times 1$ matrix of signals.

$$\stackrel{⌣}{\mathbf{C}}=[{\stackrel{⌣}{\mathbf{c}}}_{n,1},\dots ,{\stackrel{⌣}{\mathbf{c}}}_{n,K_n},{\stackrel{⌣}{\mathbf{c}}}_{c,1},\dots ,{\stackrel{⌣}{\mathbf{c}}}_{c,K_c}]$$

(15)

where ${\stackrel{⌣}{\mathbf{c}}}_{n,k_n}$ and ${\stackrel{⌣}{\mathbf{c}}}_{c,k_c}$ are the new steering vectors of non-circular signals and circular signals, respectively.

$${\stackrel{⌣}{\mathbf{c}}}_{n,k_n}=\left[\begin{array}{c}{b}_{n,k_n}\left(\begin{array}{c}{\mathbf{a}}_{n,k_n}{\upsilon }_{1_{n,k_n}}\\ {\mathbf{a}}_{n,k_n}{\upsilon }_{2_{n,k_n}}\\ ⋮\\ {\mathbf{a}}_{n,k_n}{\upsilon }_{N_{n,k_n}}\end{array}\right)\\ b_{n,k_n}^{*}\left(\begin{array}{c}{\mathbf{a}}_{n,k_n}^{*}{\upsilon }_{1_{n,k_n}}^{*}\\ {\mathbf{a}}_{n,k_n}^{*}{\upsilon }_{2_{n,k_n}}^{*}\\ ⋮\\ {\mathbf{a}}_{n,k_n}^{*}{\upsilon }_{N_{n,k_n}}^{*}\end{array}\right)\end{array}\right]$$

(16)

$k_n=1,2,\dots ,K_n$, which is a $2NM\times 1$ vector,

$${\stackrel{⌣}{\mathbf{c}}}_{c,k_c}=\left[\begin{array}{cc}\begin{array}{c}{\mathbf{a}}_{c,k_c}{\upsilon }_{1_{c,k_c}}\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{N_{c,k_c}}\end{array}& \begin{array}{c}{\mathbf{0}}_{M\times 1}\\ {\mathbf{0}}_{M\times 1}\\ ⋮\\ {\mathbf{0}}_{M\times 1}\end{array}\\ \begin{array}{c}{\mathbf{0}}_{M\times 1}\\ {\mathbf{0}}_{M\times 1}\\ ⋮\\ {\mathbf{0}}_{M\times 1}\end{array}& \begin{array}{c}{\mathbf{a}}_{c,k_c}^{*}{\upsilon }_{1_{c,k_c}}^{*}\\ {\mathbf{a}}_{c,k_c}^{*}{\upsilon }_{2_{c,k_c}}^{*}\\ ⋮\\ {\mathbf{a}}_{c,k_c}^{*}{\upsilon }_{N_{c,k_c}}^{*}\end{array}\end{array}\right]$$

(17)

$k_c=1,2,\dots ,K_c$, which is a $2NM\times 2$ matrix,

$$\begin{array}{c}\stackrel{⌣}{\mathbf{s}}\left(t\right)=[s_{n,1}\left(t\right),\cdots ,s_{n,K_n}\left(t\right),s_{c,1}\left(t\right),s_{c,1}^{*}\left(t\right),\cdots ,s_{c,K_c}\left(t\right),s_{c,K_c}^{*}\left(t\right){]}^T\hfill \end{array}$$

(18)

which is a $\left(K_n+2K_c\right)\times 1$ matrix of signals, and:

$$\stackrel{⌣}{\mathbf{n}}=\left[\begin{array}{c}\mathbf{n}\\ {\mathbf{n}}^{*}\end{array}\right]$$

(19)

which is a $2NM\times 1$ vector of noise.

The covariance matrix of $\stackrel{⌣}{\mathit{f}}$ is calculated by:

$$\stackrel{⌣}{\mathbf{R}}=E[\stackrel{⌣}{\mathit{f}}{\stackrel{⌣}{\mathit{f}}}^H]=\stackrel{⌣}{\mathbf{C}}{\stackrel{⌣}{\mathbf{R}}}_{\mathrm{s}}{\stackrel{⌣}{\mathbf{C}}}^H+{\sigma }^2{\mathbf{I}}_{2NM}$$

(20)

where ${\stackrel{⌣}{\mathbf{R}}}_{\mathrm{s}}=E[\stackrel{⌣}{\mathbf{s}}{\stackrel{⌣}{\mathbf{s}}}^H]$ is the covariance matrix of $\stackrel{⌣}{\mathbf{s}}$. ${\stackrel{⌣}{\mathbf{R}}}_{\mathrm{s}}$ is a full-rank matrix, since the incident signals are uncorrelated with each other. Then, the eigenvalue decomposition of $\stackrel{⌣}{\mathbf{R}}$ is:

$$\stackrel{⌣}{\mathbf{R}}={\mathbf{E}}_s{\Sigma }_s{\mathbf{E}}_s^H+{\mathbf{E}}_n{\Sigma }_n{\mathbf{E}}_n^H$$

(21)

where the $2NM\times \left(K_n+2K_c\right)$ matrix ${\mathbf{E}}_s$ and the $2NM\times (2NM-K_n-2K_c)$ matrix ${\mathbf{E}}_n$ are the signal subspace and noise subspace, respectively. The $(K_n+2K_c)\times (K_n+2K_c)$ matrix ${\Sigma }_s=diag({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_K)$ and the $(2NM-K_n-2K_c)\times (2NM-K_n-2K_c)$ matrix ${\Sigma }_n=diag({\lambda }_{K+1},{\lambda }_{K+2},\dots ,{\lambda }_{2NM})$ are diagonal matrices, where ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_K>{\lambda }_{K+1}=\cdots ={\lambda }_{2NM}={\sigma }^2$ are the eigenvalues of $\stackrel{⌣}{\mathbf{R}}$.

Remark 1.

In practice, the available observed data are finite. Thus, $\stackrel{⌣}{\mathbf{R}}$ can be approximated by:

$$\widehat{\stackrel{⌣}{\mathbf{R}}}=\frac{1}{L}\sum _{t=1}^L\stackrel{⌣}{f}(t){\stackrel{⌣}{f}}^H(t)$$

(22)

where L is the number of available data snapshots. Then, the eigenvalue decomposition of $\widehat{\stackrel{⌣}{\mathbf{R}}}$ is:

$$\widehat{\stackrel{⌣}{\mathbf{R}}}={\widehat{\mathbf{E}}}_s{\widehat{\Sigma }}_s{\widehat{\mathbf{E}}}_s^H+{\widehat{\mathbf{E}}}_n{\widehat{\Sigma }}_n{\widehat{\mathbf{E}}}_n^H$$

(23)

therefore, the noise subspace ${\mathit{E}}_n$ and signal subspace ${\mathit{E}}_s$ can be approximated by ${\widehat{\mathit{E}}}_n$ and ${\widehat{\mathit{E}}}_s$, respectively.

3. The Proposed Algorithm

In this section, we propose a 2D DOA estimation algorithm to solve the problem of estimating and distinguishing the mixed signals that are circular and non-circular in detail. Firstly, the method for estimating the mixed sources that are circular and noncircular is proposed; secondly, the algorithm, which only detects the circular signals, is proposed; and finally, we study how to distinguish these two kinds of sources.

Because both $\stackrel{⌣}{\mathbf{C}}$ and ${\mathbf{E}}_s$ have the same signal subspace, orthogonal to the noise subspace spanned by the matrix ${\mathbf{E}}_n$, we devise estimators to obtain the 2D DOAs of noncircular and circular signals using the rank-reduction-based Root-MUSIC method.

3.1. 2D DOA Estimation for Non-Circular Sources

Because the noise subspace ${\mathbf{E}}_n$ is orthogonal to ${\stackrel{⌣}{\mathbf{c}}}_{n,k_n}$ (the steering vectors of non-circular signals), the following equation can be obtained directly:

$${\mathbf{E}}_n^H{\stackrel{⌣}{\mathbf{c}}}_{n,k_n}=0$$

(24)

Then, together with Equations (16) and (24), we can get the following equation:

$$\begin{array}{cc}{\mathbf{E}}_n^H{\stackrel{⌣}{\mathbf{c}}}_{n,k_n}\hfill & ={\mathbf{E}}_n^H\left[\begin{array}{c}{b}_{n,k_n}\left(\begin{array}{c}{\mathbf{a}}_{n,k_n}{\upsilon }_{1_{{}_{n,k_n}}}\\ {\mathbf{a}}_{n,k_n}{\upsilon }_{2_{{}_{n,k_n}}}\\ ⋮\\ {\mathbf{a}}_{n,k_n}{\upsilon }_{N_{{}_{n,k_n}}}\end{array}\right)\\ b_{n,k_n}^{*}\left(\begin{array}{c}{{\mathbf{a}}^{*}}_{n,k_n}{\upsilon }_{1_{n,k_n}}^{*}\\ {{\mathbf{a}}^{*}}_{n,k_n}{\upsilon }_{2_{n,k_n}}^{*}\\ ⋮\\ {{\mathbf{a}}^{*}}_{n,k_n}{\upsilon }_{N_{n,k_n}}^{*}\end{array}\right)\end{array}\right]\hfill \\ & ={\mathbf{E}}_n^{H}blkdiag[{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n},\dots ,{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n}^{*},{\mathbf{a}}_{n,k_n}^{*},\dots ,{\mathbf{a}}_{n,k_n}^{*}].\hfill \\ & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left[\begin{array}{cc}{\upsilon }_{1_{{}_{n,k_n}}}& 0\\ \begin{array}{c}{\upsilon }_{2_{{}_{n,k_n}}}\\ ⋮\\ {\upsilon }_{N_{{}_{n,k_n}}}\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& {\upsilon }_{1_{n,k_n}}^{*}\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{\upsilon }_{2_{n,k_n}}^{*}\\ ⋮\\ {\upsilon }_{N_{n,k_n}}^{*}\end{array}\end{array}\right]\left[\begin{array}{c}{b}_{n,k_n}\\ b_{n,k_n}^{*}\end{array}\right]=\mathbf{0}\hfill \end{array}$$

(25)

Defining a vector ${\mathbf{p}}_n\left(l\right)={[1,l,...,l^{M-1}]}^T$, which is only related to l, where $l=e^{-j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{x}cos{\theta }_{n,k_n}}$. Therefore, we can obtain the formula ${\mathbf{p}}_n\left(l\right)={\mathbf{a}}_{n,k_n}$ and ${\mathbf{p}}_n\left(l^{-1}\right)={{\mathbf{a}}^{*}}_{n,k_n}$, then define a $2NM\times 2N$ matrix $\mathsf{\Omega }$(l), which is only related to l,

$$\begin{array}{c}\mathsf{\Omega }\left(l\right)=blkdiag[{\mathbf{p}}_n\left(l\right),{\mathbf{p}}_n\left(l\right),\cdots ,{\mathbf{p}}_n\left(l\right),{\mathbf{p}}_n\left(l^{-1}\right),{\mathbf{p}}_n\left(l^{-1}\right),\cdots ,{\mathbf{p}}_n\left(l^{-1}\right)]\hfill \\ \begin{array}{cc}\hfill & \end{array}=blkdiag[{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n},\cdots ,{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n}^{*},{\mathbf{a}}_{n,k_n}^{*},\cdots ,{\mathbf{a}}_{n,k_n}^{*}]\hfill \end{array}$$

(26)

and define a $2N\times 2N$ matrix:

$${\mathbf{Q}}_n\left(l\right)=l^{M-1}{\mathsf{\Omega }}^T\left(l^{-1}\right){\mathbf{E}}_n{\mathbf{E}}_n^H{l}^{M-1}\mathsf{\Omega }\left(l\right)$$

(27)

Note that if $(2NM-K_n-2K_c)\ge 2N$ and $\theta$ is not the true angle of the non-circular signal, ${\mathbf{Q}}_n\left(l\right)$ is of full rank because in this case, the column rank of ${\mathbf{E}}_n$ is not less than $2N$. Then, Equation (25) holds true only when $\theta$ equals the true angle of the signal (${\mathbf{Q}}_n\left(l\right)$ drops rank). Since the covariance matrix of $\stackrel{⌣}{\mathit{f}}$ is obtained from a finite number of samples, the reduction of the rank of ${\mathbf{Q}}_n\left(l\right)$ can roughly be replaced by the minimum of the determinant of ${\mathbf{Q}}_n\left(l\right)$. Therefore, we get the estimator of $\theta$ about non-circular signals as follows:

$$f_n\left(l\right)=det\left({\mathbf{Q}}_n\left(l\right)\right)=det\left(l^{M-1}{\mathsf{\Omega }}^T\left(l^{-1}\right){\mathbf{E}}_n{\mathbf{E}}_n^H{l}^{M-1}\mathsf{\Omega }\left(l\right)\right)$$

(28)

Notice that $f_n\left(l\right)$ is a $2\times 3N(M-1)$ order polynomial, showing that there are $3N(M-1)$ pairs of conjugated roots. The estimates $K_n$ signal DOAs of $\theta$ can be obtained by finding the closest roots of the unit circle, which are given by:

$${\widehat{\theta }}_{k_n}=arccos\left[-\frac{\lambda }{2\pi d_x}arg\left(l_{k_n}\right)\right],k_n=1,...,K_n$$

(29)

where $l_{k_n},k_n=1,...,K_n$ are the roots closest to the unit circle. Then, we take the estimated ${\widehat{\theta }}_{k_n}$ of non-circular source into Equation (25) to get the estimator of $\beta$:

$$f_n^{\prime }\left(u\right)=det\left({\mathbf{Q}}_n^{\prime }\left(u\right)\right)$$

(30)

where:

$${\mathbf{Q}}_n^{\prime }\left(u\right)=u^{(N-1)}{\mathsf{\Theta }}^T\left(u^{-1}\right){\mathsf{\Omega }}^T\left(l^{-1}\right){\mathbf{E}}_n{\mathbf{E}}_n^H\mathsf{\Omega }\left(l\right)u^{(N-1)}\mathsf{\Theta }\left(u\right)$$

(31)

$u=e^{j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{y}cos{\beta }_{n,k_n}}$; it means that ${\upsilon }_{m_{n,k_n}}=u^{m-1},m=1,\cdots N$ and:

$$\mathsf{\Theta }\left(u\right)=\left[\begin{array}{cc}1& 0\\ \begin{array}{c}u\\ ⋮\\ u^{N-1}\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& 1\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{u}^{*}\\ ⋮\\ {u^{*}}^{N-1}\end{array}\end{array}\right]=\left[\begin{array}{cc}{\upsilon }_{1_{n,k_n}}\left(\beta \right)& 0\\ \begin{array}{c}{\upsilon }_{2_{n,k_n}}\left(\beta \right)\\ ⋮\\ {\upsilon }_{N_{n,k_n}}\left(\beta \right)\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& {\upsilon }_{1_{n,k_n}}^{*}\left(\beta \right)\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{\upsilon }_{2_{n,k_n}}^{*}\left(\beta \right)\\ ⋮\\ {\upsilon }_{N_{n,k_n}}^{*}\left(\beta \right)\end{array}\end{array}\right]$$

(32)

To achieve the estimate of $\beta$, we need to solve the roots of $f_n^{\prime }\left(u\right)$. Due to $f_n^{\prime }\left(u\right)$ being a $2\times 3(N-1)$-order polynomial, it means there are $3(N-1)$ pairs of conjugated roots. The estimate of $\beta$ can be obtained by finding the closest roots of the unit circle; in this way, we can automatically pair the closest roots $u_1,u_2,...,u_{K_n}$ corresponding to ${\widehat{\theta }}_1,{\widehat{\theta }}_2,\dots ,{\widehat{\theta }}_{K_n}$, respectively, and the DOAs of $\beta$ are given by:

$${\widehat{\beta }}_{k_n}=arccos\left[\frac{\lambda }{2\pi d_y}arg\left(u_{k_n}\right)\right],k_n=1,...,K_n$$

(33)

where $u_{k_n},k_n=1,...,K_n$ are the roots closest to the unit circle, respectively.

3.2. 2D DOA Estimation for Circular Sources

Because the noise subspace ${\mathbf{E}}_n$ is also orthogonal to ${\stackrel{⌣}{\mathbf{c}}}_{c,k_c}$ (the steering vectors of circular signals), we can get the equation as follows:

$${\mathbf{E}}_n^H{\stackrel{⌣}{\mathbf{c}}}_{c,k_c}={\mathbf{E}}_n^H\left[\begin{array}{cc}{\mathbf{a}}_{c,k_c}{\upsilon }_{1_{{}_{c,k_c}}}& {\mathbf{0}}_{M\times 1}\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{2_{c,k_c}}& {\mathbf{0}}_{M\times 1}\\ ⋮& ⋮\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{N_{{}_{c,k_c}}}& {\mathbf{0}}_{M\times 1}\\ {\mathbf{0}}_{M\times 1}& {\mathbf{a}}_{c,k_c}^{*}{{\upsilon }^{*}}_{1_{{}_{c,k_c}}}\\ {\mathbf{0}}_{M\times 1}& {\mathbf{a}}_{c,k_c}^{*}{{\upsilon }^{*}}_{2_{{}_{c,k_c}}}\\ ⋮& ⋮\\ {\mathbf{0}}_{M\times 1}& {\mathbf{a}}_{c,k_c}^{*}{{\upsilon }^{*}}_{N_{{}_{c,k_c}}}\end{array}\right]=\mathbf{0}$$

(34)

from Equation (34), we can get:

$${\mathbf{E}}_n^H\left[\begin{array}{c}{\mathbf{a}}_{c,k_c}{\upsilon }_{1_{{}_{c,k_c}}}\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{N_{{}_{c,k_c}}}\\ {\mathbf{0}}_{M\times 1}\\ {\mathbf{0}}_{M\times 1}\\ ⋮\\ {\mathbf{0}}_{M\times 1}\end{array}\right]=\mathbf{0}$$

(35)

$${\mathbf{E}}_n^H\left[\begin{array}{c}{\mathbf{0}}_{M\times 1}\\ {\mathbf{0}}_{M\times 1}\\ ⋮\\ {\mathbf{0}}_{M\times 1}\\ {{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{1_{{}_{c,k_c}}}\\ {{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{2_{c,k_c}}\\ ⋮\\ {{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{N_{{}_{c,k_c}}}\end{array}\right]=\mathbf{0}$$

(36)

Partitioning the noise subspace ${\mathbf{E}}_n$ into ${\mathbf{E}}_n=\left[\begin{array}{c}{\mathbf{E}}_{n1}\\ {\mathbf{E}}_{n2}\end{array}\right]$, where ${\mathbf{E}}_{n1}$ and ${\mathbf{E}}_{n2}$ are two submatrices of the same size $NM\times (2NM-K_n-2K_c)$, Equations (35) and (36) can be changed to:

$$\begin{array}{c}\begin{array}{c}\hfill \end{array}{\left[\begin{array}{c}{\mathbf{a}}_{c,k_c}{\upsilon }_{1_{c,k_c}}\\ \begin{array}{c}{\mathbf{a}}_{c,k_c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{N_{c,k_c}}\end{array}\end{array}\right]}^H{\mathbf{E}}_{n1}{\mathbf{E}}_{n1}^H\left[\begin{array}{c}{\mathbf{a}}_{c,k_c}{\upsilon }_{1_{c,k_c}}\\ \begin{array}{c}{\mathbf{a}}_{c,k_c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{N_{c,k_c}}\end{array}\end{array}\right]\hfill \\ ={\left[\begin{array}{c}{\upsilon }_{1_{c,k_c}}\\ \begin{array}{c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\upsilon }_{N_{c,k_c}}\end{array}\end{array}\right]}^{H}blkdiag{[{\mathbf{a}}_{c,k_c},{\mathbf{a}}_{c,k_c},\cdots ,{\mathbf{a}}_{c,k_c}]}^H.\hfill \\ \begin{array}{c}\hfill \end{array}{\mathbf{E}}_{n1}{\mathbf{E}}_{n1}^{H}blkdiag[{\mathbf{a}}_{c,k_c},{\mathbf{a}}_{c,k_c},\cdots ,{\mathbf{a}}_{c,k_c}]\left[\begin{array}{c}{\upsilon }_{1_{c,k_c}}\\ \begin{array}{c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\upsilon }_{N_{c,k_c}}\end{array}\end{array}\right]=\mathbf{0}\hfill \end{array}$$

(37)

$$\begin{array}{c}\begin{array}{c}\hfill \end{array}{\left[\begin{array}{c}{{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{1_{c,k_c}}\\ \begin{array}{c}{{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{2_{c,k_c}}\\ ⋮\\ {{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{N_{c,k_c}}\end{array}\end{array}\right]}^H{\mathbf{E}}_{n2}{\mathbf{E}}_{n2}^H\left[\begin{array}{c}{{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{1_{c,k_c}}\\ \begin{array}{c}{{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{2_{c,k_c}}\\ ⋮\\ {{\mathbf{a}}^{*}}_{c,k_c}{{\upsilon }^{*}}_{N_{c,k_c}}\end{array}\end{array}\right]\hfill \\ ={\left[\begin{array}{c}{{\upsilon }^{*}}_{1_{c,k_c}}\\ \begin{array}{c}{{\upsilon }^{*}}_{2_{c,k_c}}\\ ⋮\\ {{\upsilon }^{*}}_{N_{c,k_c}}\end{array}\end{array}\right]}^{H}blkdiag{[{{\mathbf{a}}^{*}}_{c,k_c},{{\mathbf{a}}^{*}}_{c,k_c},\cdots ,{{\mathbf{a}}^{*}}_{c,k_c}]}^H.\hfill \\ \begin{array}{c}\hfill \end{array}{\mathbf{E}}_{n2}{\mathbf{E}}_{n2}^{H}blkdiag[{{\mathbf{a}}^{*}}_{c,k_c},{{\mathbf{a}}^{*}}_{c,k_c},\cdots ,{{\mathbf{a}}^{*}}_{c,k_c}]\left[\begin{array}{c}{{\upsilon }^{*}}_{1_{c,k_c}}\\ \begin{array}{c}{{\upsilon }^{*}}_{2_{c,k_c}}\\ ⋮\\ {{\upsilon }^{*}}_{N_{c,k_c}}\end{array}\end{array}\right]=\mathbf{0}\hfill \end{array}$$

(38)

As proven in [27], Equations (37) and (38) are equivalent to each other; therefore, the estimator over $\theta$, which corresponds to circular signals, can obtained based on Equation (37). Since:

$$\left[\begin{array}{c}{\upsilon }_{1_{c,k_c}}\\ \begin{array}{c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\upsilon }_{N_{c,k_c}}\end{array}\end{array}\right]\ne \mathbf{0}$$

(39)

Defining a vector ${\mathbf{p}}_c(\tilde{l})={[1,\tilde{l},...,{\tilde{l}}^{M-1}]}^T$ only related to $\tilde{l}$, where $\tilde{l}=e^{-j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{x}cos{\theta }_{c,k_c}}$. Therefore, we can get formula ${\mathbf{p}}_c(\tilde{l})={\mathbf{a}}_{c,k_c}$, then defining a $NM\times N$ matrix $\mathsf{\Lambda }$($\tilde{l}$) only related to $\tilde{l}$,

$$\begin{array}{c}\mathsf{\Lambda }(\tilde{l})=blkdiag[{\mathbf{p}}_c(\tilde{l}),{\mathbf{p}}_c(\tilde{l}),\cdots ,{\mathbf{p}}_c(\tilde{l})]=blkdiag[{\mathbf{a}}_{c,k_c},{\mathbf{a}}_{c,k_c},\cdots ,{\mathbf{a}}_{c,k_c}]\hfill \end{array}$$

(40)

and:

$${\mathbf{Q}}_c(\tilde{l})={\tilde{l}}^{M-1}{\mathsf{\Lambda }}^T({\tilde{l}}^{-1}){\mathbf{E}}_{n1}{\mathbf{E}}_{n1}^H\mathsf{\Lambda }(\tilde{l})$$

(41)

Note that if $(2NM-K_n-2K_c)\ge N$ and when $\theta$ is not the true angle of circular signal, ${\mathbf{Q}}_c(\tilde{l})$ is of full rank because in this case, the column rank of ${\mathbf{E}}_{n1}$ is not less than N. Then, Equation (37) holds true only when $\theta$ equals the true angle of circular signal (${\mathbf{Q}}_c(\tilde{l})$ drops rank). Since the covariance matrix of $\stackrel{⌣}{\mathit{f}}$ is obtained from a finite number of samples, the reduction of the rank of ${\mathbf{Q}}_c(\tilde{l})$ can roughly be replaced by the minimum of the determinant of ${\mathbf{Q}}_c(\tilde{l})$. Therefore, we get the estimator of $\theta$ about circular signals as follows:

$$f_c(\tilde{l})=det({\mathbf{Q}}_c(\tilde{l}))=det({\tilde{l}}^{M-1}{\mathsf{\Lambda }}^T({\tilde{l}}^{-1}){\mathbf{E}}_{n1}{\mathbf{E}}_{n1}^H\mathsf{\Lambda }(\tilde{l}))$$

(42)

Notice that $f_c(\tilde{l})$ is a $2\times N(M-1)$ order polynomial, which means that there are $N(M-1)$ pairs of conjugated roots, and the estimated $K_c$ signal DOAs of $\theta$ can be obtained by finding the closest roots of the unit circle, which are given by:

$${\widehat{\theta }}_{k_c}=arccos\left[-\frac{\lambda }{2\pi d_x}arg\left({\tilde{l}}_{k_c}\right)\right],k_c=1,...,K_c$$

(43)

where ${\tilde{l}}_{k_c},k_c=1,...,K_c$ are the roots closest to the unit circle. Then, we take the estimated ${\widehat{\theta }}_{k_c}$ of circular signals into Equation (37) to get the estimator of $\beta$:

$$f_c^{\prime }\left(\tilde{u}\right)=det\left({\mathbf{Q}}_c^{\prime }\left(\tilde{u}\right)\right)$$

(44)

where:

$${\mathbf{Q}}_c^{\prime }\left(\tilde{u}\right)={\tilde{u}}^{(N-1)}{\mathsf{\Psi }}^T\left({\tilde{u}}^{-1}\right){\tilde{l}}^{M-1}{\mathsf{\Lambda }}^T\left({\tilde{l}}^{-1}\right){\mathbf{E}}_{n1}{\mathbf{E}}_{n1}^H\mathsf{\Lambda }\left(\tilde{l}\right)\mathsf{\Psi }\left(\tilde{u}\right)$$

(45)

$\tilde{u}=e^{j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{y}cos{{\beta }_{c,}}_{k_c}}$; this means that ${\upsilon }_{m_{c,k_c}}={\tilde{u}}^{m-1},m=1,\cdots N$ and:

$$\mathsf{\Psi }\left(\tilde{u}\right)=\left[\begin{array}{c}1\\ \begin{array}{c}\tilde{u}\\ ⋮\\ {\tilde{u}}^{N-1}\end{array}\end{array}\right]=\left[\begin{array}{c}{\upsilon }_{1_{c,k_c}}\left(\beta \right)\\ \begin{array}{c}{\upsilon }_{2_{c,k_c}}\left(\beta \right)\\ ⋮\\ {\upsilon }_{N_{c,k_c}}\left(\beta \right)\end{array}\end{array}\right]$$

(46)

To obtain the estimate of $\beta$, we need to solve the roots of $f_c^{\prime }\left(\tilde{u}\right)$. According to $f_c^{\prime }\left(\tilde{u}\right)$ being a $2\times (N-1)$-order polynomial, this means there are $N-1$ pairs of conjugated roots. The estimate of $\beta$ can be obtained by finding the closest roots of the unit circle. Therefore, we can automatically pair the closest roots ${\tilde{u}}_1,{\tilde{u}}_2,...,{\tilde{u}}_{K_c}$ corresponding to ${\widehat{\theta }}_1,{\widehat{\theta }}_2,\dots ,{\widehat{\theta }}_{K_c}$, respectively, and the DOAs of $\beta$ are given by:

$${\widehat{\beta }}_{k_c}=arccos\left[\frac{\lambda }{2\pi d_y}arg\left({\tilde{u}}_{k_c}\right)\right],k_c=1,...,K_c$$

(47)

where ${\tilde{u}}_{k_c},k_c=1,...,K_c$ are the roots closest to the unit circle, respectively.

3.3. Identification of Circular and Noncircular Signals

In order to distinguish the 2D DOAs of circular and noncircular signals from the mixed signals, Equation (34) can be changed into:

$$\begin{array}{cc}{\mathbf{E}}_n^H{\stackrel{⌣}{\mathbf{c}}}_{c,k_c}\hfill & ={\mathbf{E}}_n^H\left[\begin{array}{cc}{\mathbf{a}}_{c,k_c}{\upsilon }_{1_{{}_{c,k_c}}}& {\mathbf{0}}_{M\times 1}\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{2_{c,k_c}}& {\mathbf{0}}_{M\times 1}\\ ⋮& ⋮\\ {\mathbf{a}}_{c,k_c}{\upsilon }_{N_{{}_{c,k_c}}}& {\mathbf{0}}_{M\times 1}\\ {\mathbf{0}}_{M\times 1}& {\mathbf{a}}_{c,k_c}^{*}{{\upsilon }^{*}}_{1_{{}_{c,k_c}}}\\ {\mathbf{0}}_{M\times 1}& {\mathbf{a}}_{c,k_c}^{*}{{\upsilon }^{*}}_{2_{{}_{c,k_c}}}\\ ⋮& ⋮\\ {\mathbf{0}}_{M\times 1}& {\mathbf{a}}_{c,k_c}^{*}{{\upsilon }^{*}}_{N_{{}_{c,k_c}}}\end{array}\right]\hfill \\ & ={\mathbf{E}}_n^{H}blkdiag[{\mathbf{a}}_{c,k_c},{\mathbf{a}}_{c,k_c},\cdots ,{\mathbf{a}}_{c,k_c},{\mathbf{a}}_{c,k_c}^{*},{\mathbf{a}}_{c,k_c}^{*},\cdots ,{\mathbf{a}}_{c,k_c}^{*}].\hfill \\ & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\left[\begin{array}{cc}{\upsilon }_{1_{{}_{c,k_c}}}& 0\\ \begin{array}{c}{\upsilon }_{2_{{}_{c,k_c}}}\\ ⋮\\ {\upsilon }_{N_{{}_{c,k_c}}}\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& {\upsilon }_{1_{c,k_c}}^{*}\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{\upsilon }_{2_{c,k_c}}^{*}\\ ⋮\\ {\upsilon }_{N_{c,k_c}}^{*}\end{array}\end{array}\right]=\mathbf{0}\hfill \end{array}$$

(48)

due to $\left[\begin{array}{cc}{\upsilon }_{1_{c,k_c}}& 0\\ \begin{array}{c}{\upsilon }_{2_{c,k_c}}\\ ⋮\\ {\upsilon }_{N_{c,k_c}}\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& {\upsilon }_{1_{c,k_c}}^{*}\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{\upsilon }_{2_{c,k_c}}^{*}\\ ⋮\\ {\upsilon }_{N_{c,k_c}}^{*}\end{array}\end{array}\right]\ne \mathbf{0}$, we can utilize method of noncircular to establish the estimator over $\theta$ and $\beta$ of circular signals as follows:

$$det\left({\tilde{l}}^{M-1}{\mathsf{\Omega }}^T\left({\tilde{l}}^{-1}\right){\mathbf{E}}_n{\mathbf{E}}_n^H{\tilde{l}}^{M-1}\mathsf{\Omega }\left(\tilde{l}\right)\right)=0$$

(49)

and:

$$det\left({\tilde{u}}^{(N-1)}{\mathsf{\Theta }}^T\left({\tilde{u}}^{-1}\right){\mathsf{\Omega }}^T\left({\tilde{l}}^{-1}\right){\mathbf{E}}_n{\mathbf{E}}_n^H\mathsf{\Omega }\left(\tilde{l}\right){\tilde{u}}^{(N-1)}\mathsf{\Theta }\left(\tilde{u}\right)\right)=0$$

(50)

This means the noncircular method can also be applied to solve circular signals. Therefore, we can achieve the 2D DOAs of both non-circular and circular sources from Equations (28)–(30) and (33) and only obtain the 2D DOAs of circular sources from Equations (42)–(44) and (47). Then, the purpose of distinguishing the circular and noncircular signals from the mixtures can be accomplished.

The proposed algorithm can be outlined as:

Step 1:

Construct the new data vector $\stackrel{⌣}{\mathit{f}}$ and calculate its approximate covariance $\widehat{\stackrel{⌣}{\mathbf{R}}}$ from Equations (14) and (22).

Step 2:

Perform EVDto $\widehat{\stackrel{⌣}{\mathbf{R}}}$ and achieve the noise subspace ${\widehat{\mathbf{E}}}_n$ from Equation (23).

Step 3:

Estimate the K 2D DOAs of the mixed signals from Equations (28)–(30) and (33).

Step 4:

Obtain ${\widehat{\mathbf{E}}}_{n1}$ by partitioning the matrix ${\widehat{\mathbf{E}}}_n$.

Step 5:

Estimate the $K_c$ 2D DOAs of the circular signals from Equations (42)–(44) and (47).

Step 6:

Compare the estimate achieved by Step 3 and Step 5 to distinguish the $K_n$ 2D DOAs of the noncircular signals and the $K_c$ 2D DOAs of the circular ones.

Remark 2.

To calculate $\widehat{\stackrel{⌣}{\mathbf{R}}}$, a computational complexity of $O({(2NM)}^{2}L)$ is needed. The computational complexity of eigendecomposition operation is $O({(2NM)}^3)$. The proposed method employs four 1D estimators of polynomial-rooting; therefore, the complexity for the proposed method is $O\left({(2NM)}^{2}L+{(2NM)}^3+4NM\times K+2NM\times K_c\right)$. While the algorithm in [27] employs several 1D spatial spectrum search procedures to obtain the 2D DOAs of signals, by defining the scanning interval of $\theta \in [0,\pi ]$ with an interval of $\mathsf{\Delta }\theta$ and $\beta \in [0,\pi ]$ with an interval of $\mathsf{\Delta }\beta$, respectively, the complexity for the algorithm in [27] is $O({\left(2NM\right)}^{2}L+{\left(2NM\right)}^3+\frac{\pi }{\mathsf{\Delta }\theta }{\left(2NM\right)}^2+K\frac{\pi }{\mathsf{\Delta }\beta }{\left(2NM\right)}^2+\frac{\pi }{\mathsf{\Delta }\theta }{\left(NM\right)}^2+K_c\frac{\pi }{\mathsf{\Delta }\beta }{\left(NM\right)}^2)$. The algorithm in [16] employs two direct 2D spatial spectrum search procedures, whose complexity for the method is $O\left({\left(NM\right)}^{2}L+{\left(NM\right)}^3+\frac{\pi }{\mathsf{\Delta }\theta }\frac{\pi }{\mathsf{\Delta }\beta }{\left(2NM\right)}^2+\frac{\pi }{\mathsf{\Delta }\theta }\frac{\pi }{\mathsf{\Delta }\beta }{\left(NM\right)}^2\right)$. Therefore, the computational complexity of the proposed method has been reduced greatly.

4. Theoretical Performance Analysis

The theoretical DOA estimation error is caused by the finite data effect, the sensor errors and the unknown noise structure [31,35], and in this section, we will study the theoretical perturbation of the proposed algorithm as a criterion for evaluation.

4.1. Theoretical Perturbation for Non-Circular Sources

According to Equation (25), defining ${\mathbf{a}}_{n,k_n}={[1,l,...,l^{M-1}]}^T$, ${\upsilon }_{m_{n,k_n}}=u^{m-1},m=1,\cdots ,N$ and $b_{n,k_n}=z$, where $l=e^{-j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{x}cos{\theta }_{n,k_n}}$, $u=e^{j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{y}cos{\beta }_{n,k_n}}$, $z=e^{j{\phi }_{n,k_n}}$, ${\phi }_{n,k_n}$ is the non-circular phase of non-circular sources. Therefore, Equation (25) can be changed into:

$$\begin{array}{c}{\mathbf{E}}_n^{H}blkdiag[{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n},\cdots ,{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n}^{*},{\mathbf{a}}_{n,k_n}^{*},\cdots ,{\mathbf{a}}_{n,k_n}^{*}]\left[\begin{array}{cc}1& 0\\ \begin{array}{c}u\\ ⋮\\ u^{N-1}\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& 1\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{u}^{*}\\ ⋮\\ {u^{*}}^{N-1}\end{array}\end{array}\right]\left[\begin{array}{c}z\\ z^{-1}\end{array}\right]=\mathbf{0}\hfill \end{array}$$

(51)

Defining:

$$\begin{array}{cc}\mathbf{M}(l,u,z)=\hfill & blkdiag[{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n},\cdots ,{\mathbf{a}}_{n,k_n},{\mathbf{a}}_{n,k_n}^{*},{\mathbf{a}}_{n,k_n}^{*},\cdots ,{\mathbf{a}}_{n,k_n}^{*}].\hfill \\ & \left[\begin{array}{cc}1& 0\\ \begin{array}{c}u\\ ⋮\\ u^{N-1}\end{array}& \begin{array}{c}0\\ ⋮\\ 0\end{array}\\ 0& 1\\ \begin{array}{c}0\\ ⋮\\ 0\end{array}& \begin{array}{c}{u}^{*}\\ ⋮\\ {u^{*}}^{N-1}\end{array}\end{array}\right]\left[\begin{array}{c}z\\ z^{-1}\end{array}\right]\hfill \end{array}$$

(52)

Then, we can define the equation as follows:

$$f(l,u,z)={\mathbf{M}}^H(l,u,z){\mathbf{E}}_n{\mathbf{E}}_n^H\mathbf{M}(l,u,z)$$

(53)

In practical situations, due to the influence of noise, the sensor errors and the finite data effect, we can only achieve ${\widehat{\mathbf{E}}}_n$, the approximation of ${\mathbf{E}}_n$, according to Equation (53), we can get the following formula:

$$\widehat{f}(l,u,z)={\mathbf{M}}^H(l,u,z){\widehat{\mathbf{E}}}_n{\widehat{\mathbf{E}}}_n^H\mathbf{M}(l,u,z)$$

(54)

The locations of the local minima of either the noise-free or perturbed spectral polynomials are obtained when the first derivative equal to zero. In other words, the roots of the derivative of $\widehat{f}(l,u,z)$ give the locations of the relative minima of $\widehat{f}(l,u,z)$. Thus, to find the perturbations of the DOAs (which correspond to minima of $\widehat{f}(l,u,z)$), we must calculate the perturbations of the roots of the derivative of $\widehat{f}(l,u,z)$. Define $({\widehat{l}}_{k_n},{\widehat{u}}_{k_n},{\widehat{z}}_{k_n})$ as the estimate of $(l,u,z)$ when the true angle is $\theta ={\theta }_{n,k_n}$, $\beta ={\beta }_{n,k_n}$, and $\phi ={\phi }_{n,k_n}$. Therefore, the first-order partial derivatives of $\widehat{f}(l,u,z)$ at $({\widehat{l}}_{k_n},{\widehat{u}}_{k_n},{\widehat{z}}_{k_n})$ are zeros. Approximating the perturbation of DOAs uses the first two terms in the Taylor series expansion of the first-order partial derivative of $\widehat{f}(l,u,z)$ about the true angles of arrival.

$${\widehat{f}}_l^{\prime }\left(k_n\right)+\mathsf{\Delta }{l}_{k_n}{\widehat{f}}_{l,l}^{″}\left(k_n\right)+\mathsf{\Delta }{u}_{k_n}{\widehat{f}}_{l,u}^{″}\left(k_n\right)+\mathsf{\Delta }{z}_{k_n}{\widehat{f}}_{l,z}^{″}\left(k_n\right)=0$$

(55)

$${\widehat{f}}_u^{\prime }\left(k_n\right)+\mathsf{\Delta }{l}_{k_n}{\widehat{f}}_{u,l}^{″}\left(k_n\right)+\mathsf{\Delta }{u}_{k_n}{\widehat{f}}_{u,u}^{″}\left(k_n\right)+\mathsf{\Delta }{z}_{k_n}{\widehat{f}}_{u,z}^{″}\left(k_n\right)=0$$

(56)

$${\widehat{f}}_z^{\prime }\left(k_n\right)+\mathsf{\Delta }{l}_{k_n}{\widehat{f}}_{z,l}^{″}\left(k_n\right)+\mathsf{\Delta }{u}_{k_n}{\widehat{f}}_{z,u}^{″}\left(k_n\right)+\mathsf{\Delta }{z}_{k_n}{\widehat{f}}_{z,z}^{″}\left(k_n\right)=0$$

(57)

where $\mathsf{\Delta }{l}_{k_n}={\widehat{l}}_{k_n}-l_{k_n}$, $\mathsf{\Delta }{u}_{k_n}={\widehat{u}}_{k_n}-u_{k_n}$ and $\mathsf{\Delta }{z}_{k_n}={\widehat{z}}_{k_n}-z_{k_n}$ are the perturbations of $l_{k_n}$, $u_{k_n}$ and $z_{k_n}$, respectively. Together with Equations (55)–(57), solve the set of equations as follows:

$$\begin{array}{cc}\mathsf{\Delta }{l}_{k_n}\hfill & =\frac{1}{\delta }\left|\begin{array}{ccc}-{{\widehat{f}}^{\prime }}_l\left(k_n\right)& {{\widehat{f}}^{″}}_{l,u}\left(k_n\right)& {{\widehat{f}}^{″}}_{l,z}\left(k_n\right)\\ -{{\widehat{f}}^{\prime }}_u\left(k_n\right)& {{\widehat{f}}^{″}}_{u,u}\left(k_n\right)& {{\widehat{f}}^{″}}_{u,z}\left(k_n\right)\\ -{{\widehat{f}}^{\prime }}_z\left(k_n\right)& {{\widehat{f}}^{″}}_{z,u}\left(k_n\right)& {{\widehat{f}}^{″}}_{z,z}\left(k_n\right)\end{array}\right|\hfill \\ & =\frac{1}{\delta }·\{{{\widehat{f}}^{\prime }}_l\left(k_n\right)[{\left({{\widehat{f}}^{″}}_{u,z}\left(k_n\right)\right)}^2-{{\widehat{f}}^{″}}_{u,u}\left(k_n\right){{\widehat{f}}^{″}}_{z,z}\left(k_n\right)]+{{\widehat{f}}^{\prime }}_u\left(k_n\right)[{{\widehat{f}}^{″}}_{l,u}\left(k_n\right){{\widehat{f}}^{″}}_{z,z}\left(k_n\right)-{{\widehat{f}}^{″}}_{l,z}\left(k_n\right){{\widehat{f}}^{″}}_{u,z}\left(k_n\right)]\hfill \\ & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{{\widehat{f}}^{\prime }}_z\left(k_n\right)[{{\widehat{f}}^{″}}_{l,z}\left(k_n\right){{\widehat{f}}^{″}}_{u,u}\left(k_n\right)-{{\widehat{f}}^{″}}_{l,u}\left(k_n\right){{\widehat{f}}^{″}}_{u,z}\left(k_n\right)]\}\hfill \end{array}$$

(58)

$$\begin{array}{cc}\mathsf{\Delta }{u}_{k_n}\hfill & =\frac{1}{\delta }\left|\begin{array}{ccc}{{\widehat{f}}^{″}}_{l,l}\left(k_n\right)& -{{\widehat{f}}^{\prime }}_l\left(k_n\right)& {{\widehat{f}}^{″}}_{l,z}\left(k_n\right)\\ {{\widehat{f}}^{″}}_{u,l}\left(k_n\right)& -{{\widehat{f}}^{\prime }}_u\left(k_n\right)& {{\widehat{f}}^{″}}_{u,z}\left(k_n\right)\\ {{\widehat{f}}^{″}}_{z,l}\left(k_n\right)& -{{\widehat{f}}^{\prime }}_z\left(k_n\right)& {{\widehat{f}}^{″}}_{z,z}\left(k_n\right)\end{array}\right|\hfill \\ & =\frac{1}{\delta }·\{{{\widehat{f}}^{\prime }}_l\left(k_n\right)[{{\widehat{f}}^{″}}_{u,l}\left(k_n\right){{\widehat{f}}^{″}}_{z,z}\left(k_n\right)-{{\widehat{f}}^{″}}_{u,z}\left(k_n\right){{\widehat{f}}^{″}}_{z,l}\left(k_n\right)]+{{\widehat{f}}^{\prime }}_u\left(k_n\right)[{\left({{\widehat{f}}^{″}}_{l,z}\left(k_n\right)\right)}^2-{{\widehat{f}}^{″}}_{l,l}\left(k_n\right){{\widehat{f}}^{″}}_{z,z}\left(k_n\right)]\hfill \\ & \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+{{\widehat{f}}^{\prime }}_z\left(k_n\right)[{{\widehat{f}}^{″}}_{l,l}\left(k_n\right){{\widehat{f}}^{″}}_{u,z}\left(k_n\right)-{{\widehat{f}}^{″}}_{l,z}\left(k_n\right){{\widehat{f}}^{″}}_{u,l}\left(k_n\right)]\}\hfill \end{array}$$

(59)

where:

$$\delta =\left|\begin{array}{ccc}{{\widehat{f}}^{″}}_{l,l}\left(k_n\right)& {{\widehat{f}}^{″}}_{l,u}\left(k_n\right)& {{\widehat{f}}^{″}}_{l,z}\left(k_n\right)\\ {{\widehat{f}}^{″}}_{u,l}\left(k_n\right)& {{\widehat{f}}^{″}}_{u,u}\left(k_n\right)& {{\widehat{f}}^{″}}_{u,z}\left(k_n\right)\\ {{\widehat{f}}^{″}}_{z,l}\left(k_n\right)& {{\widehat{f}}^{″}}_{z,u}\left(k_n\right)& {{\widehat{f}}^{″}}_{z,z}\left(k_n\right)\end{array}\right|$$

(60)

According to the expressions for Equation (54), the first derivative, ${\widehat{f}}_{\eta }^{\prime }\left(k_n\right)(\eta =l,u,z)$, and the second derivative, ${\widehat{f}}_{\tau ,\kappa }^{″}\left(k_n\right)(\tau ,\kappa =l,u,z)$, can be achieved, respectively. Using the value of $(l_{k_n},u_{k_n},z_{k_n})$ and Equations (58) and (59), we can obtain the perturbation $\mathsf{\Delta }l$ and $\mathsf{\Delta }u$. Then, the perturbation of $\theta$ and $\beta$ can be achieved from $\mathsf{\Delta }l$ and $\mathsf{\Delta }u$, respectively. The relation between an arrival angle and a signal root is given in [36]:

$$\mathsf{\Delta }{\theta }_{n,k_n}=C_{\theta ,k_n}\mathrm{Im}\left[\frac{\mathsf{\Delta }{l}_{k_n}}{l_{k_n}}\right]$$

(61)

$$\mathsf{\Delta }{\beta }_{n,k_n}=C_{\beta ,k_n}\mathrm{Im}\left[\frac{\mathsf{\Delta }{u}_{k_n}}{u_{k_n}}\right]$$

(62)

where $C_{\theta ,k_n}=\frac{\lambda }{2\pi d_{x}sin{\theta }_{n,k_n}}$ and $C_{\beta ,k_n}=-\frac{\lambda }{2\pi d_{y}sin{\beta }_{n,k_n}}$. Therefore, the theoretical perturbation of ${\theta }_{n,k_n}$ and ${\beta }_{n,k_n}$ can be obtained from Equations (61) and (62).

4.2. Theoretical Perturbation for Circular Sources

Similar to the noncircular method, defining ${\mathbf{a}}_{c,k_c}={[1,\tilde{l},...,{\tilde{l}}^{M-1}]}^T$ and ${\upsilon }_{m_{c,k_c}}={\tilde{u}}^{m-1},m=1,\cdots N$, where $\tilde{l}=e^{-j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{x}cos{\theta }_{c,k_c}}$ and $\tilde{u}=e^{j(2\pi /\phantom{(2\pi \lambda }\lambda )d_{y}cos{\beta }_{c,k_c}}$. Equation (35) can be changed to:

$${\mathbf{E}}_{n1}^H\left[\begin{array}{cccc}{\mathbf{a}}_{c,k_c}& & & \\ & {\mathbf{a}}_{c,k_c}& & \\ & & \ddots & \\ & & & {\mathbf{a}}_{c,k_c}\end{array}\right]\left[\begin{array}{c}1\\ \begin{array}{c}\tilde{u}\\ ⋮\\ {\tilde{u}}^{N-1}\end{array}\end{array}\right]=\mathbf{0}$$

(63)

defining:

$$\mathbf{M}(\tilde{l},\tilde{u})=\left[\begin{array}{cccc}{\mathbf{a}}_{c,k_c}& & & \\ & {\mathbf{a}}_{c,k_c}& & \\ & & \ddots & \\ & & & {\mathbf{a}}_{c,k_c}\end{array}\right]\left[\begin{array}{c}1\\ \begin{array}{c}\tilde{u}\\ ⋮\\ {\tilde{u}}^{N-1}\end{array}\end{array}\right]$$

(64)

therefore, the equation can be defined as follows:

$$f(\tilde{l},\tilde{u})={\mathbf{M}}^H(\tilde{l},\tilde{u}){\mathbf{E}}_{n1}{\mathbf{E}}_{n1}^H\mathbf{M}(\tilde{l},\tilde{u})$$

(65)

In practical application, we can only achieve ${\widehat{\mathbf{E}}}_{n1}$, the approximation of ${\mathbf{E}}_{n1}$; according to Equation (65), we can get the following equation:

$$\widehat{f}(\tilde{l},\tilde{u})={\mathbf{M}}^H(\tilde{l},\tilde{u}){\widehat{\mathbf{E}}}_{n1}{\widehat{\mathbf{E}}}_{n1}^H\mathbf{M}(\tilde{l},\tilde{u})$$

(66)

the roots of the derivative of $\widehat{f}(\tilde{l},\tilde{u})$ give the locations of the relative minima of $\widehat{f}(\tilde{l},\tilde{u})$. Define $({\widehat{\tilde{l}}}_{k_c},{\widehat{\tilde{u}}}_{k_c})$ as the estimate of $(\tilde{l},\tilde{u})$ when the true angle is $\theta ={\theta }_{c,k_c}$ and $\beta ={\beta }_{c,k_c}$. Therefore, the first-order partial derivatives of $\widehat{f}(\tilde{l},\tilde{u})$ at $({\widehat{\tilde{l}}}_{k_c},{\widehat{\tilde{u}}}_{k_c})$ are zeros. Approximating the perturbation of DOAs uses the first two terms in the Taylor series expansion of the first-order partial derivative of $\widehat{f}(\tilde{l},\tilde{u})$ at the true angles of arrival.

$${\widehat{f}}_{\tilde{l}}^{\prime }\left(k_c\right)+\mathsf{\Delta }{\tilde{l}}_{k_c}{\widehat{f}}_{\tilde{l},\tilde{l}}^{″}\left(k_c\right)+\mathsf{\Delta }{\tilde{u}}_{k_c}{\widehat{f}}_{\tilde{l},\tilde{u}}^{″}\left(k_c\right)=0$$

(67)

$${\widehat{f}}_{\tilde{u}}^{\prime }\left(k_c\right)+\mathsf{\Delta }{\tilde{l}}_{k_c}{\widehat{f}}_{\tilde{u},\tilde{l}}^{″}\left(k_c\right)+\mathsf{\Delta }{\tilde{u}}_{k_c}{\widehat{f}}_{\tilde{u},\tilde{u}}^{″}\left(k_c\right)=0$$

(68)

where $\mathsf{\Delta }{\tilde{l}}_{k_c}={\widehat{\tilde{l}}}_{k_c}-{\tilde{l}}_{k_c}$ and $\mathsf{\Delta }{\tilde{u}}_{k_c}={\widehat{\tilde{u}}}_{k_c}-{\tilde{u}}_{k_c}$ are the perturbations of ${\tilde{l}}_{k_c}$ and ${\tilde{u}}_{k_c}$, respectively. Together with Equations (67) and (68) to solve the set of equations as follows:

$$\begin{array}{c}\mathsf{\Delta }{\tilde{l}}_{k_c}=\frac{1}{\tilde{\delta }}\left|\begin{array}{cc}-{{\widehat{f}}^{\prime }}_{\tilde{l}}\left(k_c\right)& {{\widehat{f}}^{″}}_{\tilde{l},\tilde{u}}\left(k_c\right)\\ -{{\widehat{f}}^{\prime }}_{\tilde{u}}\left(k_c\right)& {{\widehat{f}}^{″}}_{\tilde{u},\tilde{u}}\left(k_c\right)\end{array}\right|=\frac{1}{\tilde{\delta }}·[{{\widehat{f}}^{\prime }}_{\tilde{u}}\left(k_c\right){{\widehat{f}}^{″}}_{\tilde{l},\tilde{u}}\left(k_c\right)-{{\widehat{f}}^{\prime }}_{\tilde{l}}\left(k_c\right){{\widehat{f}}^{″}}_{\tilde{u},\tilde{u}}\left(k_c\right)]\hfill \end{array}$$

(69)

$$\begin{array}{c}\mathsf{\Delta }{\tilde{u}}_{k_c}=\frac{1}{\tilde{\delta }}\left|\begin{array}{cc}{{\widehat{f}}^{″}}_{\tilde{l},\tilde{l}}\left(k_c\right)& -{{\widehat{f}}^{\prime }}_{\tilde{l}}\left(k_c\right)\\ {{\widehat{f}}^{″}}_{\tilde{u},\tilde{l}}\left(k_c\right)& -{{\widehat{f}}^{\prime }}_{\tilde{u}}\left(k_c\right)\end{array}\right|=\frac{1}{\tilde{\delta }}·[{{\widehat{f}}^{\prime }}_{\tilde{l}}\left(k_c\right){{\widehat{f}}^{″}}_{\tilde{u},\tilde{l}}\left(k_c\right)-{{\widehat{f}}^{\prime }}_{\tilde{u}}\left(k_c\right){{\widehat{f}}^{″}}_{\tilde{l},\tilde{l}}\left(k_c\right)]\hfill \end{array}$$

(70)

where:

$$\tilde{\delta }=\left|\begin{array}{cc}{{\widehat{f}}^{″}}_{\tilde{l},\tilde{l}}\left(k_c\right)& {{\widehat{f}}^{″}}_{\tilde{l},\tilde{u}}\left(k_c\right)\\ {{\widehat{f}}^{″}}_{\tilde{u},\tilde{l}}\left(k_c\right)& {{\widehat{f}}^{″}}_{\tilde{u},\tilde{u}}\left(k_c\right)\end{array}\right|$$

(71)

According to the expressions for Equation (66), the first derivative, ${\widehat{f}}_{\tilde{\eta }}^{\prime }\left(k_c\right)(\tilde{\eta }=\tilde{l},\tilde{u})$ and the second derivative, ${\widehat{f}}_{\tilde{\tau },\tilde{\kappa }}^{″}\left(k_c\right),(\tilde{\tau },\tilde{\kappa }=\tilde{l},\tilde{u})$ can be achieved, respectively. Using the value of $({\tilde{l}}_{k_c},{\tilde{u}}_{k_c})$ and Equations (69) and (70), we can obtain the perturbation $\mathsf{\Delta }\tilde{l}$ and $\mathsf{\Delta }\tilde{u}$. Then, the relation between an arrival angle and a signal root is as follows:

$$\mathsf{\Delta }{\theta }_{c,k_c}=C_{\theta ,k_c}\mathrm{Im}\left[\frac{\mathsf{\Delta }{\tilde{l}}_{k_c}}{{\tilde{l}}_{k_c}}\right]$$

(72)

$$\mathsf{\Delta }{\beta }_{c,k_c}=C_{\beta ,k_c}\mathrm{Im}\left[\frac{\mathsf{\Delta }{\tilde{u}}_{k_c}}{{\tilde{u}}_{k_c}}\right]$$

(73)

where $C_{\theta ,k_c}=\frac{\lambda }{2\pi d_{x}sin{\theta }_{c,k_c}}$ and $C_{\beta ,k_c}=-\frac{\lambda }{2\pi d_{y}sin{\beta }_{c,k_c}}$; therefore, the theoretical perturbation of ${\theta }_{c,k_c}$ and ${\beta }_{c,k_c}$ can be obtained.

5. Simulation Results

In this section, simulation results are provided to demonstrate the performance of the proposed algorithm. For Simulations (1)–(3), the URAs have $N=3$ rows and $M=8$ columns; both $d_x$ and $d_y$ are half wavelength. For all simulations, we utilize BPSK and QPSK to represent the strictly non-circular signal and the circular signal, respectively, and the sources can be realized base on (7). For instance, if we take ${\eta }_k=0$ and ${\phi }_k=0$, the QPSK signal $s_k\in \left(\frac{\sqrt{2}}{2}+\frac{\sqrt{2}}{2}j,\frac{\sqrt{2}}{2}-\frac{\sqrt{2}}{2}j,-\frac{\sqrt{2}}{2}+\frac{\sqrt{2}}{2}j,-\frac{\sqrt{2}}{2}-\frac{\sqrt{2}}{2}j\right)$ can be obtained. If we take ${\eta }_k=1$ and ${\phi }_k=0$, the BPSK signal $s_k\in \left(-1,+1\right)$ can be acquired. The power of additive white Gaussian noise is ${\sigma }_n^2$, and the signal-to-noise (SNR) is defined as $SNR=10{log}_{10}({\sigma }_s^2/{\sigma }_n^2)$. We use the root mean square error (RMSE) to evaluate the estimation performance, which is defined as:

$$RMSE=\sqrt{\frac{1}{K{M}_c}\sum _{k=1}^K\sum _{m=1}^{M_c}[{({\widehat{\epsilon }}_{m,k}-{\epsilon }_k)}^2]}$$

(74)

where $M_c$ is the number of Monte Carlo simulations, K is the number of signals, ${\widehat{\epsilon }}_{m,k}$ is the estimated ${\theta }_k$ or ${\beta }_k$ in the m-th Monte Carlo simulation and ${\epsilon }_k$ is the true value for either ${\theta }_k$ or ${\beta }_k$ of the k-th signal.

5.1. The 2D DOAs’ Scattergram of the Estimators

To demonstrate the performance of the proposed algorithm, we examine the scattergram of 2D $\theta$ and $\beta$ of the method. Five BPSK signals and three QPSK signals are considered here. The BPSK signals are from the directions $({\theta }_1={65}^{\circ },{\beta }_1={50}^{\circ })$, $({\theta }_2={90}^{\circ },{\beta }_2={105}^{\circ })$, $({\theta }_3={50}^{\circ },{\beta }_3={60}^{\circ })$, $({\theta }_4={125}^{\circ },{\beta }_4={85}^{\circ })$ and $({\theta }_5={30}^{\circ },{\beta }_5={115}^{\circ })$ and the QPSK signals from $({\theta }_6={105}^{\circ },{\beta }_6={95}^{\circ })$, $({\theta }_7={75}^{\circ },{\beta }_7={40}^{\circ })$ and $({\theta }_8={115}^{\circ },{\beta }_8={70}^{\circ })$. The SNR is 10 dB, and the number of snapshots is 500. Figure 2a,b indicates that the method can estimate and distinguish the 2D DOAs that are strictly noncircular and circular successfully.

5.2. Performance versus SNR

In this part, the performance of the proposed algorithm is studied with a varying SNR from −5 dB–15 dB. The number of snapshots is 500, and the number of Monte Carlo simulations is 500. Five BPSK signals and one QPSK signal are considered. The BPSK signals are from the directions $({\theta }_1={65}^{\circ },{\beta }_1={50}^{\circ })$, $({\theta }_2={90}^{\circ },{\beta }_2={105}^{\circ })$, $({\theta }_3={50}^{\circ },{\beta }_3={60}^{\circ })$, $({\theta }_4={125}^{\circ },{\beta }_4={85}^{\circ })$ and $({\theta }_5={30}^{\circ },{\beta }_5={115}^{\circ })$ and the QPSK signals from $({\theta }_6={105}^{\circ },{\beta }_6={95}^{\circ })$.

The proposed algorithm in theoretical analysis and experimental results, the algorithm of 2D-MUSIC, the algorithm in [18], the algorithm in [27] and the deterministic CRB (Cramer–Rao bound) in [37], are compared in terms of RMSE. Figure 3a,b shows that the proposed method is steadily better than the other three algorithms, and the experimental values of the proposed algorithm overlap together with the theoretical error ones.

5.3. Performance versus Snapshots

We study the performance of the proposed algorithm with a varying snapshot number from 30–850. The SNR is fixed at 10 dB; the number of Monte Carlo simulations and incident signals are the same as in the second experiment. The RMSE results for angle estimation are shown in Figure 4a,b. The proposed method is steadily better than the other three algorithms, and the experimental values of the proposed algorithm overlap together with the theoretical error ones.

5.4. Performance versus the Number of Noncircular Mixed Signals

Finally, we consider the performance of the number of noncircular signals based on the proposed algorithm. The URAs have $N=3$ rows and $M=6$ columns; the SNRs vary form 0 dB–30 dB; the number of snapshots is 1200; and the number of Monte Carlo simulations is 1000. There are four uncorrelated signals from directions $({\theta }_1={65}^{\circ },{\beta }_1={50}^{\circ })$, $({\theta }_2={105}^{\circ },{\beta }_2={95}^{\circ })$, $({\theta }_3={75}^{\circ },{\beta }_3={40}^{\circ })$ and $({\theta }_4={115}^{\circ },{\beta }_4={70}^{\circ })$, and the total number of sources remains unchanged. We consider cases about one, two, three and four BPSK signals, respectively. The results are shown in Figure 5a,b.

We can see that the RMSE of the experimental values of the proposed algorithm will overlap together with the theoretical error ones in a relatively high SNR, and the 2D DOA estimation performance of the proposed method improves from Case 1 to Case 4 because the dimension of the noise subspace has been extended by the increasing number of BPSK signals.

6. Conclusions

In this paper, a novel low-complexity 2D DOA estimation algorithm for mixed circular and non-circular signals has been proposed based on the URAs; besides, the theoretical error of the proposed algorithm is analyzed. As verified by the simulation results, the proposed method has a lower computational complexity and a better DOA estimation performance than the existing methods due to the usage of the noncircularity of the incoming signals.