Efficient Two-Dimensional Direction Finding Algorithm for Rectilinear Sources Under Unknown Mutual Coupling

Abstract

Digital communication signals in wireless systems may possess noncircularity, which can be used to enhance the degrees of freedom for direction-of-arrival (DOA) estimation in sensor array signal processing. On the other hand, the electromagnetic characteristics between sensors in uniform rectangular arrays (URAs), such as mutual coupling, may significantly deteriorate the estimation performance. To deal with this problem, a robust real-valued estimator for rectilinear sources was developed to alleviate unknown mutual coupling in URAs. An augmented covariance matrix was built up by extracting the real and imaginary parts of observations containing the circularity and noncircularity of signals. Then, the actual steering vector considering mutual coupling was reparameterized to make the rank reduction (RARE) property available. To reduce the computational complexity of two-dimensional (2D) spectral search, we individually estimated y-axis and x-axis direction-cosines in two stages following the principle of RARE. Finally, azimuth and elevation angle estimates were determined from the corresponding direction-cosines respectively. Compared with existing solutions, the proposed method is more computationally efficient, involving real-valued operations and decoupled 2D spectral searches into twice those of one-dimensional searches. Simulation results verified that the proposed method provides satisfactory estimation performance that is robust to unknown mutual coupling and close to the counterparts based on 2D spectral searches, but at the cost of much fewer calculations.

1. Introduction

Uniform rectangular array (URA), due to its versatility in providing both azimuthal and elevational coverage simultaneously, has attracted much attention in many applications, such as radar, sonar, and wireless communication [1]. Two-dimensional (2D) direction finding with URA is thus of practical use in many situations. Based on the subspace principle, the classical 1D super resolution algorithms such as multiple signal classification (MUSIC) [2] and compressive sensing [3] can be readily generalized to URA. Moreover, to further reduce the computational complexity or increase the accuracy of the direction finding results, some effective schemes have been proposed [4,5,6,7,8]. In [4], the authors proposed a 2D unitary ESPRIT algorithm for URA, which can be realized in both element space and beamspace. Also, its main procedure is formulated in terms of real-valued operation and is therefore computationally efficient. In [5], a dual-size spatial invariance array structure was proposed. Based on the ESPRIT theory, it utilizes unambiguous but low-accuracy estimates to disambiguate high-accuracy but periodically ambiguous direction cosine estimates. In [6], a 2D matrix pencil method is developed to convert the complex data matrix to a real one, which can reduce the complexity of the computation significantly. In [7], to relieve the computational burden involved in 2D spectral searches, the authors decoupled the 2D MUSIC problem into two successive real-valued one-dimensional (1D) DOA estimation problems. In [8], a 2D unitary ESPRIT-like DOA estimation algorithm was developed for coherent signals based on a URA. It can provide better decorrelation performance and closed-form solutions due to the properties of the 2D ESPRIT-like method.

Recently, in modern wireless communication systems, rectilinear or strictly noncircular signals such as amplitude modulated (AM), binary phase shift keying (BPSK), and pulse-amplitude modulated (PAM) signals are commonly applied. Notice that both conjugated and unconjugated covariance matrices of rectilinear signals include the second-order statistical information [9]. Hence, the rectilinearity can be utilized to promote the estimation accuracy and increase the maximum resolvable sources. Accordingly, various algorithms have been developed to fully exploit the rectilinearity for DOA estimation performance enhancement. In [10], Abeida and Delmas proposed a tailored MUSIC algorithm for noncircular sources called NCMUSIC, which combines the statistics of conjugated and unconjugated covariance matrices and thus outperforms the standard MUSIC algorithm. In [11], the sparse representation technique was introduced for representing conjugated and unconjugated covariance jointly with a sparsity constraint, which surpasses the performance of NCMUSIC. However, it is computationally demanding since it requires complicate convex optimization procedures. In [12], the difference and sum coarray was utilized for the DOA estimation of noncircular quasi-stationary sources, which can extend the degrees of freedom (DOF) of the array effectively. In [13,14], the scenario of coexisting circular and noncircular sources is considered, and various algorithms have been further developed to distinguish the mixed sources and estimate the DOA parameters. Based on an URA, the authors proposed a root-MUSIC method to obtain the 2D DOA parameters and classification of the mixed sources [15]. In [16], the authors proposed a localization algorithm for rectilinear sources based on the rank reduction (RARE) principle, which can cope with the mixed near-field and far-field sources.

Generally, the above mentioned DOA estimation algorithms are based on the assumption that the array manifold is ideal without any perturbation. However, in practical direction finding systems, systematic errors such as mutual coupling, gain and phase errors, and sensor position errors are inevitable [17]. These model mismatches would substantially degrade the DOA estimation performance of these algorithms. Mutual coupling effect, as an electromagnetic interaction between the array elements, causes the current distribution of each sensor to depend on the excitations from both their own and other adjacent element. It deviates the signal subspace of the observed array covariance matrix from the ideal array steering vector, thus causing a decrease in direction finding performance. Therefore, the mutual coupling effect cannot be ignored in the development of these algorithms.

Fortunately, many effective schemes have been proposed by various researchers to compensate for the mutual coupling effect or to obtain the DOA estimates blindly. One class of compensation methods are based on calibration sources. Using a set of predefined sources in known locations, the mutual coupling coefficients can be calculated by the least square method [18]. However, the calibration sources are often inaccessible in real-time direction finding applications. Moreover, the mutual coupling characteristics of the array may vary as the environmental conditions change, and therefore the calibration source-based methods are only effective under certain circumstances. Consequently, another class of auto-calibration algorithms is advanced to compensate for the mutual coupling effect without any calibration sources [19,20,21,22,23,24,25]. In [19], an eigenstructure-based algorithm was developed for the DOA estimation and mutual coupling auto-calibration at the same time. However, it requires multidimensional alternating minimization which is computationally prohibitive. To avoid exhaustive multidimensional searches over the unknown DOA and mutual coupling parameters, the authors proposed a modified MUSIC algorithm [20]. By taking the sensors at either side of the array as auxiliary ones, the authors found that the MUSIC algorithm is resilient against unknown mutual coupling effects. However, the array aperture loss problem of these methods is severe particularly under the strong mutual coupling scenario, because only the observations from middle subarray array elements can be utilized for DOA estimation. Aiming at resolving this problem, Liao et al. utilized the banded symmetric Toeplitz property of the mutual coupling matrix and transformed this matrix into gain and phase perturbations of the array [21], where the perturbations of the middle subarray elements are identical. Consequently, the observation data from all the array elements can be fully utilized for DOA estimation via simple 1D RARE method. Furthermore, a fourth order cumulant (FOC) method was also proposed [22], which performs better especially for strong mutual coupling effects. In [23,24,25], the authors utilized sparse reconstruction methods to improve the DOA estimation performance under unknown mutual coupling. However, these methods are computationally demanding due to the l₁-norm minimization problem. In [26], the authors presented a method for the localization of mixed sources using the noncircular information of the impinging signals under unknown mutual coupling. In [27], an ESPRIT-based algorithm is proposed for the ODA estimation of strictly noncircular sources with unknown mutual coupling, which has large virtual array aperture and low computational load. According to the latest work by B. Friedlander [28], the mutual coupling matrix is a direction dependent matrix, and not a constant matrix as is commonly assumed, which may impact some of the work presented in the DOA estimation literature. However, few of the works on DOA estimation have taken into account the direction-dependent mutual coupling effect. In [29], a decoupled 2D DOA estimation algorithm is proposed for uniform circular arrays. Yet, it assumes that the mutual coupling effect is only elevation-dependent. Also, it is an offline technique, for which is costly to build a calibration setup. In [30], a unified transformation method is proposed for the DOA estimation under direction-dependent mutual coupling. This method can be used in many array structures, but it requires convex optimization and iteration processes, which is computationally inefficient. In [31], the authors proposed a RARE-based low complexity DOA estimation method to solve the direction-dependent mutual coupling problem. Moreover, coprime sparse arrays are proposed to reduce the mutual coupling effect between the elements [32,33,34,35].

For the 2D array calibration problem, several effective methods have been developed [36,37,38,39,40]. In [36], the authors proposed a blind calibration scheme for uniform circular array based on azimuthal symmetric structures in the mutual coupling matrix. However, it only provides the azimuth angle estimates of the radiating sources, while the elevation angles are assumed to be zeros. In [37], a robust method was proposed to calibrate the mutual coupling and channel gain/phase inconsistency of uniform circular array. Yet, it requires calibration sources and 2D spectral searches. In [38], the auxiliary sensor-based algorithm is generalized for URA and uniform hexagon arrays, where the elements at the boundary of the array are set as the auxiliary ones. In order to mitigate the array aperture loss problem in the auxiliary sensor-based method, Wu et al. proposes a RARE-based auto-calibration method for URA [40], which can fully utilize the array observations. However, all these planar array calibration methods require exhaustive 2D spectral searches for the DOA estimation, which is computationally demanding.

To the extent of our knowledge, few works have been devoted to the 2D direction finding algorithms for rectilinear sources in the presence of unknown mutual coupling effects. Despite the various advantages that were manifested in the approaches cited above, there are some fundamentally inherent problems to be overcome in view of the previous analyses: (1) all of the above-mentioned 2D DOA estimation works only use the covariance matrix for direction finding and mutual coupling calibration, the rectilinearity of the sources are not fully utilized. (2) These works require 2D spectral searches for the azimuth and elevation angle estimations, which is computationally demanding. (3) Generally, subspace-based direction finding algorithms require complex-valued operations for the calculation of covariance matrix, eigenvalue decomposition (EVD), and spectral searching. Many works have been devoted to converting the complex-valued covariance into a real-valued one. The computational complexity of EVD and spectral searches for this kind of transformation can be decreased by a factor of four. Yet, the transformation itself and the computation of the covariance still require additional complex-valued operations.

Therefore, aiming at addressing these problems, we developed a computationally efficient 2D direction finding algorithm for rectilinear signals under unknown mutual coupling. Based on the rectilinearity of the signal, the received data is transformed into the real-valued domain. Then the structure of the virtual steering vector is utilized to decouple the DOA parameter from other nuisance parameters. According to the RARE principle, the y-axis direction-cosine estimates are first obtained via a 1D spectral search. Consequently, the x-axis direction-cosines are then generated through another RARE estimator, which also only requires a 1D spectral search with the y-axis direction-cosine estimates.

The main contributions of this paper are as follows:

(1)

The proposed DOA estimation method avoids the two-dimensional spectral search and effectively reduces the computational complexity;

(2)

Only real-valued operations are needed, and its computation load is reduced to 1/4 of the complex value operations;

(3)

The estimation accuracy is improved by the efficient use of the source rectilinearity.

The rest of this paper is organized as follows. In Section 2, we present the signal model for the rectilinear sources of a URA under unknown mutual coupling. In Section 3, derivation of the proposed real-valued two stage 2D DOA estimation algorithm is detailed. Then we discuss the computational complexity and the maximum number of resolvable sources in Section 4. In Section 5, several numerical simulations are carried out to validate the performance of the proposed algorithm. In Section 6, the paper ends with conclusions. Finally, in the Appendix A, we have derived the Cramér-Rao Bound (CRB) for the problem.

Throughout the paper, we make use of the following notations as listed in

Table 1. List of notations.

Notations	Explanation
${(\cdot )}^{*}$	conjugate operator
${(\cdot )}^T$	transpose operator
${(\cdot )}^H$	conjugate transpose operator
Re{·}	real part of the operand
Im{·}	imaginary part of the operand
$E\{\cdot \}$	expectation operator
vec(·)	vectorization operator
e	Hadamard product operator
$\otimes$	Kronecker product operator
Tr{·}	trace of the operand
det(·)	determinant of the operand
In	identity matrix of order n

.

2. Signal Model

As shown in Figure 1, consider a URA composed of M = M_x × M_y omnidirectional elements, with the inter element spacing along the x-axis and the y-axis being d_x and d_y, respectively. Suppose that there are K far-field narrowband sources radiating onto the URA from directions $({\theta }_1,{\varphi }_1)$, …,$({\theta }_K,{\varphi }_K)$, where ${\theta }_k$ and ${\varphi }_k$ are the azimuth and elevation angles of the k-th source.

Under the ideal scenario, the received data of each element only depends on the incident wavefield. Accordingly, the ideal array received data vector without mutual coupling can be modeled as

$$x(t)={\displaystyle \sum _{k=1}^{K}a({\theta }_k,{\varphi }_k)}{s}_k(t)+n(t)=As(t)+n(t)$$

(1)

where $x(t)={[x_1(t),\cdots ,x_{M_x}(t),x_{M_x+1}(t),\cdots ,x_{2M_x}(t),\cdots ,x_{M_y{M}_x}(t)]}^T$, $s(t)={[s_1(t),s_2(t),\cdots ,s_K(t)]}^T$ is the source signal vector, $n(t)={[n_1(t),\cdots ,n_{M_x}(t),n_{M_x+1}(t),\cdots ,n_{2M_x}(t),\cdots ,n_{M_y{M}_x}(t)]}^T$ denotes the array noise vector, and $A=[a({\theta }_1,{\varphi }_1),\cdots ,a({\theta }_K,{\varphi }_K)]$ represents the ideal steering matrix, with its steering vectors being

$$a({\theta }_k,{\varphi }_k)=a_y({\psi }_{y,k})\otimes a_x({\psi }_{x,k}),k=1,\cdots ,K$$

(2)

where

$$a_x({\psi }_{x,k})={\left[1,e^{j{\psi }_{x,k}},\cdots ,e^{j(M_x-1){\psi }_{x,k}}\right]}^T$$

(3)

$$a_y({\psi }_{y,k})={\left[1,e^{j{\psi }_{y,k}},\cdots ,e^{j(M_y-1){\psi }_{y,k}}\right]}^T$$

(4)

Herein, ${\psi }_{x,k}$ and ${\psi }_{y,k}$ are the electrical angles of the kth source with respect to the x-axis and the y-axis, with ${\psi }_{x,k}=2\pi d_x\cos {\theta }_k\sin {\varphi }_k/\lambda$ and ${\psi }_{y,k}=2\pi d_y\sin {\theta }_k\sin {\varphi }_k/\lambda$ where λ is the carrier wavelength.

However, in practical direction finding systems, a mismatch may be introduced by the electromagnetic interaction between the array elements. Therefore, taking into account the mutual coupling effect between the array elements, we can modify the received data model as

$$x(t)=CAs(t)+n(t)$$

(5)

where C is the $M_x{M}_y\times M_x{M}_y$ mutual coupling matrix, with its elements being mutual coupling coefficients (MCCs). As illustrated in [19], the mutual coupling effect between the elements is inversely proportional to their distance. Thus, the MCCs between the neighboring sensors are approximately equal. When two elements are far apart, the mutual coupling effect is negligible, and the coefficients approach zero. As in [30], the general mutual coupling model for URA is developed by the following block matrix

$$C=\left[\begin{array}{ccccccc}{C}_1^{}& C_2^{}& \cdots & C_P^{}& & & \\ C_2^{}& C_1^{}& C_2^{}& & \ddots & & \\ ⋮& \ddots & & \ddots & & \ddots & \\ C_P^{}& & C_2^{}& C_1^{}& C_2^{}& & C_P^{}\\ & \ddots & & \ddots & \ddots & \ddots & ⋮\\ & & \ddots & & C_2^{}& C_1^{}& C_2^{}\\ & & & C_P^{}& \cdots & C_2^{}& C_1^{}\end{array}\right]$$

(6)

and the $M_x\times M_x$ submatrix C_p (p = 1,…,P) is expressed as [27]

$$C_p=\left[\begin{array}{cccccc}{c}_{p-1,0}& c_{p-1,1}& \cdots & c_{p-1,P-1}& & \\ c_{p-1,1}& c_{p-1,0}& c_{p-1,1}& & \ddots & \\ ⋮& c_{p-1,1}& c_{p-1,0}& \ddots & & c_{p-1,P-1}\\ c_{p-1,P-1}& & \ddots & \ddots & c_{p-1,1}& ⋮\\ & \ddots & & c_{p-1,1}& c_{p-1,0}& c_{p-1,1}\\ & & c_{p-1,P-1}& \cdots & c_{p-1,P-1}& c_{p-1,0}\end{array}\right]$$

(7)

The complex mutual coupling coefficient $c_{m,n}={\rho }_{m,n}{e}^{j{\xi }_{m,n}}$ symbolizes the electromagnetic interaction between a reference sensor defined as the origin of a Cartesian coordinate system and the sensor located at (±md_x, ±nd_y) under the corresponding coordinate [30]. Note that $c_{0,0}$ is especially defined as 1. Therefore, the mutual coupling matrix C can be further expressed as

$$C={\displaystyle \sum _{p=1}^P{J}_p\otimes C_p}$$

(8)

and $J_p$ is a $M_y\times M_y$ selection matrix, whose elements are

$${\left[J_p\right]}_{m,n}=\{\begin{array}{l}1,\left|m-n\right|=p-1\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}$$

(9)

Moreover, for rectilinear sources, the signal vector is of the following form [10]

$$s(t)=\Phi s_0^{}(t)$$

(10)

where $s_0^{}(t)={[s_{0,1}^{}(t),\cdots ,s_{0,K}^{}(t)]}^T$, with $s_{0,k}(t)$ being the real-valued zero-phase version of the k-th source signal, $\Phi =\mathrm{diag}\text{{}{e}_{}^{j{\phi }_1},\cdots ,e_{}^{j{\phi }_K}\text{}}$ contains the noncircular phase shifts of the sources [9].

Owing to the above analysis, the received data vector of the planar array can be formulated as

$$x(t)=CA\Phi s_0(t)+n(t)$$

(11)

With the received data vector x(t), the main problem to be addressed in this paper is to obtain the 2D DOA estimates of rectilinear signals under unknown mutual coupling effects. Unlike the traditional methods which require multidimensional spectral searches and omit the rectilinearity of the signals, a novel algorithm is investigated in the next section for reducing the computational complexity and promoting the estimation performance.

3. Proposed Real-Valued 2D DOA Estimation Algorithm

In order to generate accurate 2D DOA estimates with low computational complexity, we creatively utilize the rectilinearity of signals and factorize the steering vector with respect to the DOA and other nuisance parameters. Our technique proceeds through the following stages: (1) based on the rectilinearity of the signal, the received data is transformed into the real-valued domain; (2) the structure of the virtual steering vector is utilized to decouple the DOA parameter from other nuisance parameters; (3) according to the RARE principle, the y-axis direction-cosine estimates can be obtained via a 1D spectral search; (4) with the y-axis direction-cosine estimates, the x-axis direction-cosine can be found through another RARE estimator, which also only requires a 1D spectral search. The following subsections give the details of the proposed algorithm.

3.1. Extended Real-Valued Signal Model

It is well known that by transforming complex-valued data into the real-valued domain, the computational cost can be effectively reduced by a factor of four. Therefore, it inspired us to look for a more efficient algorithm to reduce the complex-valued computational loads.

Based on the sources’ rectilinearity, the transformation can be realized by extracting the real and imaginary parts of x(t), respectively.

$$x_R(t)=\mathrm{Re}[x(t)],x_I(t)=\mathrm{Im}[x(t)]$$

(12)

From Equation (11), $x_R(t)$ and $x_I(t)$ can be further expressed as

$$x_R(t)=A_R{s}_0(t)+n_R(t)$$

(13)

$$x_I(t)=A_I{s}_0(t)+n_I(t)$$

(14)

where $A_R=[v_R({\theta }_1,{\varphi }_1,{\phi }_1,C),\cdots ,v_R({\theta }_K,{\varphi }_K,{\phi }_K,C)]$ and $A_I=[v_I({\theta }_1,{\varphi }_1,{\phi }_1,C),\cdots ,v_I({\theta }_K,{\varphi }_K,{\phi }_K,C)]$ are the real-valued manifold matrices of $x_R(t)$ and $x_I(t)$. $n_R(t)$ and $n_I(t)$ symbolize the real and imaginary components of sensor noise vector $n(t)$, respectively. Accordingly, the real-valued virtual steering vectors in $A_R$ is of the following form

$$v_R({\theta }_k,{\varphi }_k,{\phi }_k,C)=\left[Ca({\theta }_k,{\varphi }_k)\cdot e^{j{\phi }_k}+C^{*}{a}^{*}({\theta }_k,{\varphi }_k)\cdot e^{-j{\phi }_k}\right]/2$$

(15)

Based on Equations (2) and (8), we can obtain the following equation from the Kronecker product property $(A\otimes B)(C\otimes D)=(AC)\otimes (BD)$ [41]

$$Ca({\theta }_k,{\varphi }_k)=\left({\displaystyle \sum _{p=1}^P{J}_p\otimes C_p}\right)\left(a_y({\psi }_{y,k})\otimes a_x({\psi }_{x,k})\right)={\displaystyle \sum _{p=1}^P\left(J_p{a}_y({\psi }_{y,k})\right)\otimes \left(C_p{a}_x({\psi }_{x,k})\right)}$$

(16)

Therefore, the real-valued virtual steering vector $v_R({\theta }_k,{\varphi }_k,{\phi }_k,C)$ can be further written as

$$\begin{array}{c}{v}_R({\theta }_k,{\varphi }_k,{\phi }_k,C)={\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,c}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{c,p}({\psi }_{x,k})a_{x,c}({\psi }_{x,k})\right)}-{\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,s}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{c,p}({\psi }_{x,k})a_{x,s}({\psi }_{x,k})\right)}\\ -{\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,c}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{s,p}({\psi }_{x,k})a_{x,s}({\psi }_{x,k})\right)}-{\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,s}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{s,p}({\psi }_{x,k})a_{x,c}({\psi }_{x,k})\right)}\end{array}$$

(17)

where

$$a_{x,c}({\psi }_{x,k})={\left[1,\cos ({\psi }_{x,k}),\cdots ,\cos \left((M_x-1){\psi }_{x,k}\right)\right]}^T$$

(18)

$$a_{x,s}({\psi }_{x,k})={\left[0,\sin ({\psi }_{x,k}),\cdots ,\sin \left((M_x-1){\psi }_{x,k}\right)\right]}^T$$

(19)

$$a_{y,c}({\psi }_{y,k})={\left[1,\cos ({\psi }_{y,k}),\cdots ,\cos \left((M_y-1){\psi }_{y,k}\right)\right]}^T$$

(20)

$$a_{y,s}({\psi }_{y,k})={\left[0,\sin ({\psi }_{y,k}),\cdots ,\sin \left((M_y-1){\psi }_{y,k}\right)\right]}^T$$

(21)

$${\Gamma }_{c,p}({\psi }_{x,k})=h_c^{(p)}\cdot \mathrm{diag}\left[{\alpha }_{c,1}^{(p)},\cdots ,{\alpha }_{c,P-1}^{(p)},1,\cdots ,1,{\beta }_{c,1}^{(p)},\cdots ,{\beta }_{c,P-1}^{(p)}\right]$$

(22)

$${\Gamma }_{s,p}({\psi }_{x,k})=h_s^{(p)}\cdot \mathrm{diag}\left[{\alpha }_{s,1}^{(p)},\cdots ,{\alpha }_{s,P-1}^{(p)},1,\cdots ,1,{\beta }_{s,1}^{(p)},\cdots ,{\beta }_{s,P-1}^{(p)}\right]$$

(23)

$$h_c^{(p)}={\displaystyle \sum _{i=1-P}^{P-1}{\rho }_{p-1,\left|i\right|}\cos \left({\xi }_{p-1,\left|i\right|}+{\phi }_k+i\cdot 2\pi {\lambda }^{-1}{\psi }_{x,k}\right)}$$

(24)

$$h_s^{(p)}={\displaystyle \sum _{i=1-P}^{P-1}{\rho }_{p-1,\left|i\right|}\sin \left({\xi }_{p-1,\left|i\right|}+{\phi }_k+i\cdot 2\pi {\lambda }^{-1}{\psi }_{x,k}\right)}$$

(25)

Note that ${\Gamma }_{c,p}({\psi }_{x,k})$ and ${\Gamma }_{s,p}({\psi }_{x,k})$ are $M_x\times M_x$ diagonal matrices comprising $M_x-2P+2$ ones between ${\alpha }_{c,P-1}^{(p)}$ and ${\beta }_{c,1}^{(p)}$ (also between ${\alpha }_{s,P-1}^{(p)}$ and ${\beta }_{s,1}^{(p)}$). $h_c^{(p)}$ and $h_s^{(p)}$ are two scalar parameters determined by the x-axis direction-cosine, rectilinearity phases, and MCCs. For $p=1,2,\cdots ,P-1$, we have

$${\alpha }_{c,q}^{(p)}=\frac{h_c^{(p)}-{\displaystyle \sum _{i=q}^{P-1}{\rho }_{p-1,i}\cos \left({\xi }_{p-1,i}+{\phi }_k-i\cdot 2\pi {\lambda }^{-1}{\psi }_{x,k}\right)}}{h_c^{(p)}},q=1,\dots ,P-1$$

(26)

$${\alpha }_{s,q}^{(p)}=\frac{h_s^{(p)}-{\displaystyle \sum _{i=q}^{P-1}{\rho }_{p-1,i}\sin \left({\xi }_{p-1,i}+{\phi }_k-i\cdot 2\pi {\lambda }^{-1}{\psi }_{x,k}\right)}}{h_s^{(p)}},q=1,\dots ,P-1$$

(27)

$${\beta }_{c,q}^{(p)}=\frac{h_c^{(p)}-{\displaystyle \sum _{i=P-q}^{P-1}{\rho }_{p-1,i}\cos \left({\xi }_{p-1,i}+{\phi }_k+i\cdot 2\pi {\lambda }^{-1}{\psi }_{x,k}\right)}}{h_c^{(p)}},q=1,\dots ,P-1$$

(28)

$${\beta }_{s,q}^{(p)}=\frac{h_s^{(p)}-{\displaystyle \sum _{i=P-q}^{P-1}{\rho }_{p-1,i}\sin \left({\xi }_{p-1,i}+{\phi }_k+i\cdot 2\pi {\lambda }^{-1}{\psi }_{x,k}\right)}}{h_s^{(p)}},q=1,\dots ,P-1$$

(29)

Similarly, the real-valued virtual steering vectors of $A_I$ can be defined as

$$\begin{array}{c}{v}_I({\theta }_k,{\varphi }_k,{\phi }_k,C)={\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,c}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{s,p}({\psi }_{x,k})a_{x,c}({\psi }_{x,k})\right)}-{\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,s}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{s,p}({\psi }_{x,k})a_{x,s}({\psi }_{x,k})\right)}\\ +{\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,c}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{c,p}({\psi }_{x,k})a_{x,s}({\psi }_{x,k})\right)}+{\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,s}({\psi }_{y,k})\right)\otimes \left({\Gamma }_{c,p}({\psi }_{x,k})a_{x,c}({\psi }_{x,k})\right)}\end{array}$$

(30)

Based on the above real-valued signal model, we can concatenate the real part and the imaginary part to generate an augmented data vector in the real-valued domain.

$$y\left(t\right)=\left[\begin{array}{c}{x}_R^{}(t)\\ x_I^{}(t)\end{array}\right]=\left[\begin{array}{c}{A}_R^{}\\ A_I^{}\end{array}\right]s_0^{}(t)+\left[\begin{array}{c}{n}_R^{}(t)\\ n_I^{}(t)\end{array}\right]=A_E^{}{s}_0^{}(t)+n_E^{}(t)$$

(31)

where $A_E^{}=[a_E^{}({\theta }_1^{},{\varphi }_1^{},{\phi }_1^{},C),\cdots ,a_E^{}({\theta }_K^{},{\varphi }_K^{},{\phi }_K^{},C)]$ is the augmented manifold matrix, and the corresponding real-valued steering vector of the k-th source is defined as

$$a_E^{}({\theta }_k^{},{\varphi }_k^{},{\phi }_k^{},C)=\left[\begin{array}{c}{v}_R^{}({\theta }_k^{},{\varphi }_k^{},{\phi }_k^{},C)\\ v_I^{}({\theta }_k^{},{\varphi }_k^{},{\phi }_k^{},C)\end{array}\right]$$

(32)

Therefore, the covariance matrix of the augmented real-valued data matrix y(t) is of the following form

$$R_y=E\left\{y(t)y^T(t)\right\}=A_E{R}_{S_0}{A}_E^T+{\sigma }_n^2{I}_{2M}$$

(33)

where $R_{S_0}^{}=E\text{{}{s}_0^{}(t)s_0^T(t)\text{}}$, $I_{2M}$ denotes a 2M × 2M identity matrix. In practical applications, $R_y$ can be approximated from the snapshots generated at L distinct time instants

$${\widehat{R}}_y=\frac{1}{L}{\displaystyle \sum _{t=1}^L\left\{y(t)y^T(t)\right\}}$$

(34)

Through the eigenvalue decomposition of $R_y^{}$, we can obtain that

$$R_y^{}=E_s^{}{\Lambda }_s^{}{E}_s^H+{\sigma }_n^2{E}_n^{}{E}_n^H$$

(35)

Herein, ${\Lambda }_s^{}$ is a diagonal matrix comprised of the largest K eigenvalues of $R_y^{}$. $E_s^{}$ is the signal subspace matrix which contains the eigenvectors corresponding to the largest K eigenvalues. $E_n^{}$ is the noise subspace matrix, which is composed of the remaining 2M-K eigenvectors of $R_y^{}$.

Based on the subspace principle that the noise subspace is orthogonal to the signal subspace, we can construct the following MUSIC spectrum function for estimating the azimuth and elevation angles of the sources.

$$P(\theta ,\varphi ,\phi ,C)=a_E^T(\theta ,\varphi ,\phi ,C)E_n{E}_n^T{a}_E(\theta ,\varphi ,\phi ,C)$$

(36)

Unfortunately, this multidimensional spectrum function presented above requires a (P + 2)-dimension joint spectral search for 2D DOAs, MCCs, and noncircular phases, which is computationally prohibitive. Therefore, to decrease the complexity of the algorithm, we first factorize the augmented steering vector $a_E^{}({\theta }_k^{},{\varphi }_k^{},{\phi }_k^{},C)$ with respect to the y-axis direction-cosine and present a novel real-valued RARE estimator in the next subsection, which only requires a simple 1D spectral search procedure.

3.2. First Stage Rank Reduction DOA Estimation

In this stage, the real-valued steering vector $a_E^{}$ in Equation (32) is first factorized with respect to the y-axis direction-cosine and other nuisance parameters. From Equations (17) and (30), the mutual coupling effect in the two virtual steering vectors can be deemed as angle-dependent gain and phase perturbations. However, it is clear that the central elements on the main diagonal of both ${\Gamma }_{c,p}({\psi }_x)$ and ${\Gamma }_{s,p}({\psi }_x)$ comprise a vector of all ones, which implies that the middle subarray can be handled as an unperturbed one. Therefore, we can obtain the following equations

$${\Gamma }_{c,p}({\psi }_x)a_{x,c}({\psi }_x)=T_{x,c}({\psi }_x){\alpha }_{x,c}^{(p)}$$

(37)

$${\Gamma }_{c,p}({\psi }_x)a_{x,s}({\psi }_x)=T_{x,s}({\psi }_x){\alpha }_{x,c}^{(p)}$$

(38)

$${\Gamma }_{s,p}({\psi }_x)a_{x,s}({\psi }_x)=T_{x,s}({\psi }_x){\alpha }_{x,s}^{(p)}$$

(39)

$${\Gamma }_{s,p}({\psi }_x)a_{x,c}({\psi }_x)=T_{x,c}({\psi }_x){\alpha }_{x,s}^{(p)}$$

(40)

where $T_{x,c}({\psi }_x)$ and $T_{x,s}({\psi }_x)$ are $M_x\times (2P-1)$ real matrices, and ${\alpha }_{x,c}^{(p)}$ and ${\alpha }_{x,s}^{(p)}$ are $(2P-1)\times 1$ real vectors, which can be expressed as

$$T_{x,c}({\psi }_x)=\left[\begin{array}{cccccc}1& & & & & \\ & \cos ({\psi }_x)& & & & 0\\ & & \ddots & & & \\ & & & \cos ((P-1){\psi }_x)& & \\ & & & ⋮& & \\ & & & \cos ((M_x-P){\psi }_x)& & \\ & 0& & & \ddots & \\ & & & & & \cos ((M_x-1){\psi }_x)\end{array}\right]$$

(41)

$$T_{x,s}({\psi }_x)=\left[\begin{array}{cccccc}0& & & & & \\ & \sin ({\psi }_x)& & & & 0\\ & & \ddots & & & \\ & & & \sin ((P-1){\psi }_x)& & \\ & & & ⋮& & \\ & & & \sin ((M_x-P){\psi }_x)& & \\ & 0& & & \ddots & \\ & & & & & \sin ((M_x-1){\psi }_x)\end{array}\right]$$

(42)

$${\alpha }_{x,c}^{(p)}=h_c^{(p)}{\left[{\alpha }_{c,1}^{(p)},\cdots ,{\alpha }_{c,P-1}^{(p)},1,{\beta }_{c,1}^{(p)},\cdots ,{\beta }_{c,P-1}^{(p)}\right]}^T$$

(43)

$${\alpha }_{x,s}^{(p)}=h_s^{(p)}{\left[{\alpha }_{s,1}^{(p)},\cdots ,{\alpha }_{s,P-1}^{(p)},1,{\beta }_{s,1}^{(p)},\cdots ,{\beta }_{s,P-1}^{(p)}\right]}^T$$

(44)

Similarly, for the y-axis direction-cosine related steering vector, we have that

$$J_p{a}_{y,c}({\psi }_y)=T_{y,c}({\psi }_y){\alpha }_{y,c}^{(p)}-T_{y,s}({\psi }_y){\alpha }_{y,s}^{(p)}$$

(45)

$$J_p{a}_{y,s}({\psi }_y)=T_{y,s}({\psi }_y){\alpha }_{y,c}^{(p)}+T_{y,c}({\psi }_y){\alpha }_{y,s}^{(p)}$$

(46)

where $T_{y,c}({\psi }_y)$ and $T_{y,s}({\psi }_y)$ are $M_y\times (2P-1)$ real matrices, and ${\alpha }_{y,c}^{(p)}$ and ${\alpha }_{y,s}^{(p)}$ are $(2P-1)\times 1$ real vectors, which can be expressed as

$$T_{y,c}({\psi }_y)=\left[\begin{array}{cccccc}1& & & & & \\ & \cos ({\psi }_y)& & & & 0\\ & & \ddots & & & \\ & & & \cos ((P-1){\psi }_y)& & \\ & & & ⋮& & \\ & & & \cos ((M_y-P){\psi }_y)& & \\ & 0& & & \ddots & \\ & & & & & \cos ((M_y-1){\psi }_y)\end{array}\right]$$

(47)

$$T_{y,s}({\psi }_y)=\left[\begin{array}{cccccc}0& & & & & \\ & \sin ({\psi }_y)& & & & 0\\ & & \ddots & & & \\ & & & \sin ((P-1){\psi }_y)& & \\ & & & ⋮& & \\ & & & \sin ((M_y-P){\psi }_y)& & \\ & 0& & & \ddots & \\ & & & & & \sin ((M_y-1){\psi }_y)\end{array}\right]$$

(48)

For simplicity, we also denote $T_{y,c}({\psi }_y)$ as $T_{y,c}$ in the following derivation. Based on the relationships derived above, we can factorize the virtual steering vectors $v_R$ and $v_I$ in Equations (17) and (30) as

$$\begin{array}{l}{v}_R(\theta ,\varphi ,\phi ,C)=\\ (T_{y,c}{\alpha }_{y,c})\otimes (T_{x,c}{\alpha }_{x,c})-(T_{y,c}{\alpha }_{y,c})\otimes (T_{x,s}{\alpha }_{x,s})-(T_{y,s}{\alpha }_{y,s})\otimes (T_{x,c}{\alpha }_{x,c})+(T_{y,s}{\alpha }_{y,s})\otimes (T_{x,s}{\alpha }_{x,s})\\ -(T_{y,s}{\alpha }_{y,c})\otimes (T_{x,s}{\alpha }_{x,c})-(T_{y,s}{\alpha }_{y,c})\otimes (T_{x,c}{\alpha }_{x,s})-(T_{y,c}{\alpha }_{y,s})\otimes (T_{x,s}{\alpha }_{x,c})-(T_{y,c}{\alpha }_{y,s})\otimes (T_{x,c}{\alpha }_{x,s})\end{array}$$

(49)

$$\begin{array}{l}{v}_I(\theta ,\varphi ,\phi ,C)=\\ (T_{y,c}{\alpha }_{y,c})\otimes (T_{x,c}{\alpha }_{x,s})+(T_{y,c}{\alpha }_{y,c})\otimes (T_{x,s}{\alpha }_{x,c})-(T_{y,s}{\alpha }_{y,s})\otimes (T_{x,c}{\alpha }_{x,s})-(T_{y,s}{\alpha }_{y,s})\otimes (T_{x,s}{\alpha }_{x,c})\\ -(T_{y,s}{\alpha }_{y,c})\otimes (T_{x,s}{\alpha }_{x,s})+(T_{y,s}{\alpha }_{y,c})\otimes (T_{x,c}{\alpha }_{x,c})-(T_{y,c}{\alpha }_{y,s})\otimes (T_{x,s}{\alpha }_{x,s})+(T_{y,c}{\alpha }_{y,s})\otimes (T_{x,c}{\alpha }_{x,c})\end{array}$$

(50)

where ${\alpha }_{x,c}$, ${\alpha }_{x,s}$, ${\alpha }_{y,c}$, and ${\alpha }_{y,s}$ are defined as

$${\alpha }_{x,c}={\displaystyle \sum _{p=1}^P{\alpha }_{x,c}^{(p)}},{\alpha }_{x,s}={\displaystyle \sum _{p=1}^P{\alpha }_{x,s}^{(p)}}$$

(51)

$${\alpha }_{y,c}={\displaystyle \sum _{p=1}^P{\alpha }_{y,c}^{(p)}},{\alpha }_{y,s}={\displaystyle \sum _{p=1}^P{\alpha }_{y,s}^{(p)}}$$

(52)

Therefore, utilizing the Kronecker product property $(AB)\otimes (CD)=(A\otimes C)(B\otimes D)$, we can write the virtual steering vector $a_E^{}$ as the following expression

$$\begin{array}{ll}{a}_E({\psi }_x,{\psi }_y,\phi ,C)& =\left[\begin{array}{c}{v}_R\\ v_I\end{array}\right]\\ & =\left[\begin{array}{cc}\left(T_{y,c}\otimes I_x\right)\left(I_y\otimes T_{x,c}\right)-\left(T_{y,s}\otimes I_x\right)\left(I_y\otimes T_{x,s}\right)& -\left(T_{y,c}\otimes I_x\right)\left(I_y\otimes T_{x,s}\right)-\left(T_{y,s}\otimes I_x\right)\left(I_y\otimes T_{x,c}\right)\\ \left(T_{y,c}\otimes I_x\right)\left(I_y\otimes T_{x,s}\right)+\left(T_{y,s}\otimes I_x\right)\left(I_y\otimes T_{x,c}\right)& \left(T_{y,c}\otimes I_x\right)\left(I_y\otimes T_{x,c}\right)-\left(T_{y,s}\otimes I_x\right)\left(I_y\otimes T_{x,s}\right)\end{array}\right]\left[\begin{array}{c}{\alpha }_{y,c}\otimes {\alpha }_{x,c}-{\alpha }_{y,s}\otimes {\alpha }_{x,c}\\ {\alpha }_{y,c}\otimes {\alpha }_{x,s}-{\alpha }_{y,s}\otimes {\alpha }_{x,c}\end{array}\right]\end{array}$$

(53)

where I_x is a M_x × M_x identity matrix and I_y is a (2P − 1) × (2P − 1) identity matrix. By factorizing the first matrix, we can reorganize $a_E^{}$ in the following form

$$\begin{array}{ll}{a}_E({\psi }_x,{\psi }_y,\phi ,C)& =\left[\begin{array}{c}{v}_R\\ v_I\end{array}\right]\\ & =\underset{T_1({\psi }_y)}{\underbrace{\left[\begin{array}{cc}{T}_{y,c}\otimes I_x& -T_{y,s}\otimes I_x\\ T_{y,s}\otimes I_x& T_{y,c}\otimes I_x\end{array}\right]}}\underset{{\alpha }_1({\psi }_x,{\psi }_y,\phi ,C)}{\underbrace{\left[\begin{array}{cc}{I}_y\otimes T_{x,c}& -I_y\otimes T_{x,s}\\ I_y\otimes T_{x,s}& I_y\otimes T_{x,c}\end{array}\right]\left[\begin{array}{c}{\alpha }_{y,c}\otimes {\alpha }_{x,c}-{\alpha }_{y,s}\otimes {\alpha }_{x,c}\\ {\alpha }_{y,c}\otimes {\alpha }_{x,s}-{\alpha }_{y,s}\otimes {\alpha }_{x,c}\end{array}\right]}}\end{array}$$

(54)

where $T_1({\psi }_y)$ is a $2M_x{M}_y\times 2M_x(2P-1)$ matrix which only depends on the y-axis direction-cosine, and ${\alpha }_1({\psi }_x,{\psi }_y,\phi ,C)$ is a $2M_x(2P-1)\times 1$ vector which depends on the x-axis and y-axis direction-cosines, MCCs, and noncircular phases.

Substituting the above equation into Equation (36), we have

$${\alpha }_1^T({\psi }_x,{\psi }_y,\phi ,C)Q_1({\psi }_y){\alpha }_1({\psi }_x,{\psi }_y,\phi ,C)=0$$

(55)

where $Q_1({\psi }_y)$ is defined as

$$Q_1({\psi }_y)=T_1^T({\psi }_y)E_n{E}_n^T{T}_1({\psi }_y)$$

(56)

It is obvious that $Q_1({\psi }_y)$ is a $2M_x(2P-1)\times 2M_x(2P-1)$ real matrix, $T_1({\psi }_y)$ is a $2M_x{M}_y\times 2M_x(2P-1)$ real matrix, and $E_n$ is the $2M_x{M}_y\times (2M_x{M}_y-K)$ real-valued noise subspace matrix. Accordingly, if $2M_x{M}_y-K\ge 2M_x(2P-1)$, $Q_1({\psi }_y)$ will generally be of full column rank. Note that ${\alpha }_1$ is presumed to be a non-zero vector, thus Equation (55) holds only when $Q_1({\psi }_y)$ rank drops, which implies that $Q_1({\psi }_y)$ becomes a rank-deficient matrix when ${\psi }_y={\psi }_{y,k},k=1,\cdots ,K$. Therefore, based on the principle of RARE, the y-axis direction-cosines can be estimated from the highest peaks of the following 1D spectral search function:

$$P_1({\psi }_y)=\frac{1}{\det \left[T_1^T({\psi }_y)E_n{E}_n^T{T}_1({\psi }_y)\right]}$$

(57)

Since the y-axis direction-cosine ${\psi }_y$ has been decoupled with other nuisance parameters in this stage, we can simplify the previous multidimensional optimization problem in Equation (36) into a 1D spectral search problem, which is computationally more efficient. Additionally, the operations involved in this stage are all real-valued, which means the computational load can be reduced by a factor of four compared with its complex-valued counterparts.

Our next objective is to estimate the values of x-axis direction-cosines and correctly pair them with the corresponding y-axis direction-cosines. Therefore, in the following subsection, we will present another RARE function which can generate the estimation of x-axis direction-cosines ${\psi }_{x,k},k=1,\cdots ,K$ without any additional matching procedures.

3.3. Second Stage Rank Reduction DOA Estimation

In the second stage, the real-valued steering vector $a_E^{}$ in Equation (32) is factorized in another form with respect to the x-axis direction-cosine and other nuisance parameters. Since the y-axis direction-cosine ${\psi }_y$ has already been obtained in the previous stage, we can calculate the values of ${\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,c}({\psi }_{y,k})\right)}$ and ${\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,s}({\psi }_{y,k})\right)}$. Also we denote that

$${\beta }_{y,c}={\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,c}({\psi }_{y,k})\right)},{\beta }_{y,s}={\displaystyle \sum _{p=1}^P\left(J_p{a}_{y,s}({\psi }_{y,k})\right)}$$

(58)

Therefore, the virtual steering vectors $v_R$ and $v_I$ in Equations (17) and (30) can be also factorized as

$$v_R({\theta }_k,{\varphi }_k,{\phi }_k,C)={\beta }_{y,c}\otimes \left[T_{x,c}{\alpha }_{x,c}^{}-T_{x,s}{\alpha }_{x,s}^{}\right]-{\beta }_{y,s}\otimes \left[T_{x,s}{\alpha }_{x,c}^{}+T_{x,c}{\alpha }_{x,s}^{}\right]$$

(59)

$$v_I({\theta }_k,{\varphi }_k,{\phi }_k,C)={\beta }_{y,c}\otimes \left[T_{x,c}{\alpha }_{x,s}^{}+T_{x,s}{\alpha }_{x,c}^{}\right]-{\beta }_{y,s}\otimes \left[T_{x,s}{\alpha }_{x,s}^{}-T_{x,c}{\alpha }_{x,c}^{}\right]$$

(60)

Utilizing the Kronecker product property $\beta \otimes (T\cdot \alpha )=(\beta \otimes T)\cdot \alpha$, we have the following expression

$$v_R({\theta }_k,{\varphi }_k,{\phi }_k,C)=\left({\beta }_{y,c}\otimes T_{x,c}\right){\alpha }_{x,c}^{}-\left({\beta }_{y,c}\otimes T_{x,s}\right){\alpha }_{x,s}^{}-\left({\beta }_{y,s}\otimes T_{x,s}\right){\alpha }_{x,c}^{}-\left({\beta }_{y,s}\otimes T_{x,c}\right){\alpha }_{x,s}^{}$$

(61)

$$v_I({\theta }_k,{\varphi }_k,{\phi }_k,C)=\left({\beta }_{y,c}\otimes T_{x,c}\right){\alpha }_{x,s}^{}+\left({\beta }_{y,c}\otimes T_{x,s}\right){\alpha }_{x,c}^{}-\left({\beta }_{y,s}\otimes T_{x,s}\right){\alpha }_{x,s}^{}+\left({\beta }_{y,s}\otimes T_{x,c}\right){\alpha }_{x,c}^{}$$

(62)

where $I_{\beta }$ is a M_y × M_y identity matrix. Consequently, the extended real-valued steering vector $a_E$ can be written in the following form

$$a_E({\psi }_x,{\psi }_y,\phi ,C)=\left[\begin{array}{c}{v}_R\\ v_I\end{array}\right]=\underset{T_2({\psi }_x)}{\underbrace{\left[\begin{array}{cc}{\beta }_{y,c}\otimes T_{x,c}-{\beta }_{y,s}\otimes T_{x,s}& -{\beta }_{y,c}\otimes T_{x,s}-{\beta }_{y,s}\otimes T_{x,c}\\ {\beta }_{y,c}\otimes T_{x,s}+{\beta }_{y,s}\otimes T_{x,c}& {\beta }_{y,c}\otimes T_{x,c}-{\beta }_{y,s}\otimes T_{x,s}\end{array}\right]}}\underset{{\alpha }_2({\psi }_x,\phi ,C)}{\underbrace{\left[\begin{array}{c}{\alpha }_{x,c}\\ {\alpha }_{x,s}\end{array}\right]}}$$

(63)

where $T_2({\psi }_x)$ is a $2M_x{M}_y\times 2(2P-1)$ matrix which only depends on the x-axis direction-cosine, and ${\alpha }_2({\psi }_x,\phi ,C)$ is a $2(2P-1)$ vector which depends on the x-axis direction-cosines, MCCs, and noncircular phases.

Again, according to the RARE principle, we substitute the above equation into Equation (36)

$${\alpha }_2^T({\psi }_x,\phi ,C)Q_2({\psi }_x){\alpha }_2({\psi }_x,\phi ,C)=0$$

(64)

where $Q_2({\psi }_x)$ is defined as

$$Q_2({\psi }_x)=T_2^T({\psi }_x)E_n{E}_n^T{T}_2({\psi }_x).$$

(65)

Note that $Q_2({\psi }_x)\in {ℝ}^{2(2P-1)\times 2(2P-1)}$ and $T_2({\psi }_x)\in {ℝ}^{2M_x{M}_y\times 2(2P-1)}$. If $2M_x{M}_y-K\ge 2(2P-1)$, then $Q_2({\psi }_x)$ is of full column rank in general. As both ${\alpha }_{x,c}$ and ${\alpha }_{x,s}$ are supposed to be non-zero vectors, ${\alpha }_2$ must be non-zero as well. One can deduce that (64) holds only if the rank of $Q_2({\psi }_x)$ drops, that is, rank deficiency occurs in $Q_2({\psi }_x)$ provided that ${\psi }_x={\psi }_{x,k},k=1,\cdots ,K$. By the principle of RARE, the x-axis direction-cosines can be obtained by searching for the highest peaks of the following spatial spectrum function:

$$P_2({\psi }_x)=\frac{1}{\det \left[T_2^T({\psi }_x)E_n{E}_n^T{T}_2({\psi }_x)\right]}.$$

(66)

Since the x-axis direction-cosine ${\psi }_x$ has been decoupled with other nuisance parameters in the second stage, the multidimensional optimization problem can be reduced to a computationally efficient 1D spectral searching problem. Moreover, the computations involved are all real-valued, which means the computational load can be reduced by a factor of four compared with its complex-valued counterparts. Also, due to the orthogonality between the signal subspace and noise subspace, the 2D DOA parameters are automatically paired without any additional matching procedures.

Therefore, the 2D DOA estimates can be obtained from the following equations:

$${\widehat{\theta }}_k=\mathrm{arc}\cot \left(\left(\frac{\lambda {\widehat{\psi }}_x}{2\pi d_x}\right)/\left(\frac{\lambda {\widehat{\psi }}_y}{2\pi d_y}\right)\right)$$

(67)

$${\widehat{\varphi }}_k=\arcsin \left(\sqrt{{\left(\frac{\lambda {\widehat{\psi }}_x}{2\pi d_x}\right)}^2+{\left(\frac{\lambda {\widehat{\psi }}_y}{2\pi d_y}\right)}^2}\right)$$

(68)

3.4. Procedures of the Proposed Algorithm

The steps of the proposed algorithm can be summarized as follows.

(1) Collect the real and imaginary parts of x(t) based on Equation (12);

(2) construct the augmented data vector via Equation (31);

(3) calculate the covariance matrix of y(t) by Equation (34);

(4) perform the eigenvalue decomposition of ${\widehat{R}}_y^{}$, and extract the noise subspace matrix ${\widehat{E}}_n^{}$;

(5) formulate the matrix $T_1({\psi }_y)$ for each y-axis direction-cosine searching grid according to Equation (54);

(6) find the estimation of y-axis direction-cosines ${\psi }_{y,k},k=1,\cdots ,K$ through the 1D spectral search in Equation (57);

(7) compute the values of ${\beta }_{y,c}$, ${\beta }_{y,s}$ and $T_2({\psi }_x)$ for each x-axis direction-cosines estimates based on Equations (58) and (63);

(8) obtain the estimates of x-axis direction-cosines ${\psi }_{x,k},k=1,\cdots ,K$ according to Equation (66);

(9) calculate the DOA estimates of the azimuth and elevation angles according to Equations (67) and (68).

4. Discussion

In this section, we compare the performance of the proposed algorithm with respect to two state-of-art 2D DOA estimation algorithms considering mutual coupling effects: the auxiliary sensor-based method (AUX) in [39] and the 2D rank reduction-based method (2D-RARE) in [40].

4.1. Computational Complexity

In this subsection, the computational complexity of the proposed algorithm is evaluated with respect to AUX and 2D-RARE. Note that the computational complexity is investigated in terms of the major multiplications in constructing the statistical matrices, EVDs, and spectral searches. Suppose that an M_x × M_y URA is utilized and the grid size for spectral searching is $\Delta \theta$. From [29], it can be observed that AUX constructs a (M_x − 2P + 2)(M_y − 2P + 2) × (M_x − 2P + 2)(M_y − 2P + 2) covariance matrix, takes the EVD of this matrix, and performs a 2D spectral search for DOA estimation, which require a computational complexity of $O(4{(M_x-2P+2)}^2{(M_y-2P+2)}^{2}L)$, $O(4{(M_x-2P+2)}^3{(M_y-2P+2)}^3/3)$, and $O(4{(M_x-2P+2)}^2{(M_y-2P+2)}^2(360°/\Delta \theta )(90°/\Delta \theta ))$, respectively. 2D-RARE constructs a M_xM_y × M_xM_y covariance matrix, performs the EVD, calculates the rank reduction testing matrix, and performs a 2D RARE spectral search, which require a computational complexity of $O(4{(M_x{M}_y)}^{2}L)$, $O(4{(M_x{M}_y)}^3/3)$, and $O\left((360°/\Delta \theta )(90°/\Delta \theta ))(M_x{M}_y(M_x{M}_y-K){(2P-1)}^2+(M_x{M}_y-K){(2P-1)}^4+{(M_x{M}_y)}^3)\right)$, respectively [30]. For the proposed algorithm, the establishment of the 2M_xM_y × 2M_xM_y real-valued statistical matrix takes $O(4{(M_x{M}_y)}^{2}L)$ operations, the EVD of this matrix requires $O(8{(M_x{M}_y)}^3/3)$ multiplications, the first stage of the 1D RARE spectral search takes $O\left((360°/\Delta \theta ){(2M_x(2P-1))}^3\right)$, and the second stage of the 1D RARE spectral search has a complexity of $O\left(K\cdot (90°/\Delta \theta ){(4P-2)}^3\right)$. It is worth noting that the calculations involved in the proposed algorithm are in the real-valued domain. Thus, with the same matrix dimensions, the computational load can be effectively reduced by a factor of four.

4.2. Maximum Number of Resolvable Sources

In this subsection, we compare the maximum number of resolvable sources for the three algorithms. Note that AUX only utilizes the (M_x − 2P + 2)(M_y − 2P + 2) sensors in the middle of the URA. As a result of the subspace identification technique, at least one eigenvector is required for spanning the noise subspace. Therefore, AUX is capable of resolving no more than (M_x − 2P + 2)(M_y − 2P + 2) − 1 sources. Similarly, 2D-RARE requires that (2P − 1)² ≤ M_xM_y-K. Thus, it can also resolve M_xM_y − (2P − 1)² sources at most. For the proposed method, it requires $2M_x{M}_y-K\ge 2M_x(2P-1)$ and $2M_x{M}_y-K\ge 2(2P-1)$ in the two consequent RARE estimators, which means the maximum number of resolvable sources for the proposed method is $2M_x{M}_y-2M_x(2P-1)$. Therefore, it can be concluded that the alleviation of computational loads is at the cost of DOF reduction, since the signal subspace has been decoupled.

4.3. Dependence of Mutual Coupling on Directions

In this subsection, we discuss the dependence of antenna mutual coupling effects on the angles of signals. According to the latest work by B. Friedlander [28], the mutual coupling matrix is a direction dependent matrix, and not a constant matrix as is commonly assumed. In this work, the mutual coupling matrix of the URA is assumed to be direction-independent with a block banded Toeplitz structure, which has been simplified compared with the analytic manifold [28]. Therefore, the DOA estimation performance may deteriorate due to the mismatch between the assumed model and the physical characteristics of the array antennas.

5. Simulation Results

In this section, we present several numerical simulations to analyze the DOA estimation performance of the proposed algorithm with respect to 2D-MUSIC, AUX, and 2D-RARE. In the following simulations, a 10 × 10-elements URA is employed with the inter-element spacing being d_x = d_y =$\lambda$/2. The impinging sources are BPSK modulated with equal power and uncorrelated to each other, while the noise is amenable to the spatially white complex Gaussian distribution. The mutual coupling length is assumed to be $P=2$, and MCCs are $c_{0,1}=c_{1,0}=0.7527+j0.4854$, $c_{1,1}=0.2825+j0.2801$. The estimation performance is assessed in terms of spatial spectrum, root mean-square error (RMSE), and computational complexity. The RMSE is measured by 500 independent Monte Carlo runs:

$$RMSE=\sqrt{\frac{1}{1000K}{\displaystyle \sum _{n=1}^{500}{\displaystyle \sum _{k=1}^K\left({\left({\theta }_k-{\widehat{\theta }}_{n,k}\right)}^2+{\left({\varphi }_k-{\widehat{\varphi }}_{n,k}\right)}^2\right)}}}$$

(69)

where ${\theta }_k$ is the DOA of the k-th source, and ${\widehat{\theta }}_{n,k}$ stands for the estimate of  in the nth trial.

In the first case, we investigate the DOA spectra of the three algorithms. We consider that three incident rectilinear signals radiate from (${\theta }_1=-20°,{\varphi }_1=40°$), (${\theta }_2=40°,{\varphi }_2=30°$), and (${\theta }_3=80°,{\varphi }_3=50°$), respectively. The corresponding x-axis direction-cosines are (0.604, 0.383, 0.133) in radians, and y-axis direction-cosines are (–0.220, 0.321, 0.754) in radians, respectively. The signal-to-noise ratio (SNR) is 10 dB, and the number of snapshots is fixed at 1000. In Figure 2d, the proposed algorithm generates three sharp peaks corresponding to the y-axis direction-cosines of the three BPSK sources in the first DOA estimation stage. In Figure 2e, the three spectra produce three sharp peaks at the desired locations of the x-axis direction-cosines, respectively. Note that the DOA estimates of in the two stages are automatically paired without any postprocessing. From Figure 2a, it is can be seen that 2D-MUSIC fails to distinguish all three sources, and its two resolvable spectra peaks are biased due to the mismatch caused by the mutual coupling effects. Although AUX and 2D-RARE can generate three peaks corresponding to the sources in the 2D DOA spectrum plain, the former sacrifices eight sensors (i.e., the auxiliary sensors) to make the remaining part of the URA coupling-free while the latter requires exhaustive 2D spectral searches to obtain the joint azimuth and elevation DOA estimates.

In the second experiment, we studied the RMSE of the DOA estimates as a function of the SNR. The number of snapshots remained at 1000. Other simulation settings were the same as the first experiment. Figure 3 depicts the RMSE of the proposed method while the RMSEs of the other three solutions, 2D-MUSIC, AUX, 2D-RARE, and the corresponding CRB for noncircular source DOA estimation in the presence of mutual coupling is also calculated for comparison. It is clear that 2D-RARE performs the best among the four algorithms, overall AUX takes second place, our method provides better accuracies than it at 10 dB and above but is inferior to AUX at low SNRs, and 2D-MUSIC has the worst performance due to the low robustness to the unknown mutual coupling. Besides, the performance of the proposed method, 2D-RARE, and AUX bears down on the CRB asymptotically at high SNRs, while the RMSE of 2D-MUSIC is stabilized at approximately $4^{\circ }$ when the SNR is larger than 5dB.

In the third experiment, we examine the variation of RMSE with the increase of the number of snapshots from 10 to 1000, where the SNR is set as 10dB. Figure 4 demonstrates that RMSEs of the proposed method, 2D-RARE, and AUX decline steadily as the snapshot size raises. This is because that the covariance and elliptic covariance matrices can be estimated more accurately from a larger number of observations. Additionally, the performance of our developed approach is almost the same as 2D-RARE and strictly superior to AUX, asymptotically approaching the corresponding CRB, while the RMSE of 2D-MUSIC saturates at $4^{\circ }$ as similar to the second scenario, which is the worst performance.

In the last experiment, the computational burdens of the three algorithms versus the variation of the number of sensors were compared. The searching step size for DOA estimation was set as 0.1°, and the number of snapshots was fixed at 200. Suppose the URA is square, i.e., $M_x=M_y$, and the number of elements ranges from 10 to 40. It can be observed from Figure 5 that our method is the most efficient compared with other two as it is only reliant upon a 1D spectral search, followed by AUX, and 2D-RARE requires the highest complexity.

6. Conclusions

In this paper, a 2D real-valued DOA estimator that is robust to unknown mutual coupling was developed for rectilinear sources. Taking advantage of the noncircularity in signals and the structured mutual coupling matrix revealed as block Toeplitz, a RARE estimator in real-value field was proposed to extract the angular information from the coupling coefficients. Compared with state-of-the-art approaches, our solution has satisfactory estimation performance with more efficient computations. A series of numerical simulations showed that the proposed method has its own advantages on estimation accuracy and computational complexity over previous algorithms with a favorable robustness to unknown mutual coupling. To apply the proposed algorithm to further applications, we may extend the proposed algorithm to considering the physical characteristics of the array antennas, such as direction-dependent mutual coupling and unstructured mutual coupling matrix in further studies.