DOA Estimation under Unknown Mutual Coupling and Multipath with Improved Effective Array Aperture

Abstract

Subspace-based high-resolution direction of arrival (DOA) estimation significantly deteriorates under array manifold perturbation and rank deficiency of the covariance matrix due to mutual coupling and multipath propagation, respectively. In this correspondence, the unknown mutual coupling can be circumvented by the proposed method without any passive or active calibration process, and the DOA of the coherent signals can be accurately estimated accordingly. With a newly constructed matrix, the deficient rank can be restored, and the effective array aperture can be extended compared with conventional spatial smoothing. The proposed method achieves a good robustness and DOA estimation accuracy with unknown mutual coupling. The simulation results demonstrate the validity and efficiency of the proposed method.

1. Introduction

In array signal processing, direction of arrival (DOA) estimation of incident signals is an important research issue. Many high-resolution methods based on the orthogonality of subspaces have been studied extensively over the years, such as MUSIC [1] and ESPRIT [2]. Most of these algorithms assume that the array manifold is known and the signals are uncorrelated. However, these two prerequisites are not always guaranteed in practice as mutual coupling between the elements is an intrinsic characteristic of antenna arrays and multipath propagation caused by reflective surfaces are very common in urban areas, both resulting in a failure of subspace-based methods [3,4].

In electromagnetic engineering, the mutual coupling matrix can be calculated using techniques such as finite element or method of moment techniques [5,6,7]. However, these calculation depends heavily on the physical shape of the antennas and are specific for each antenna array. It is thus often preferable to carry out a joint estimation of the mutual coupling matrix and signal DOA from a signal processing perspective. Weiss and Friedlander [3] proposed an iterative algorithm that compensates for mutual coupling and gain and phase perturbations, in uniform linear and circular arrays. Subsequently, Sellone et al. proposed another iterative method, which alternatingly minimises a cost function with respect to two complex valued symmetric Toeplitz matrices and a complex Hermitian Toeplitz matrix [8]. Compared with the method proposed in [3], this method is less sensitive to the array perturbations and can get an effective DOA estimate without a preliminary estimate in many cases. However, the computational complexities of both are high because the non-linear multidimensional search process is random in nature.

More recently, several DOA estimation algorithms which are not sensitive to mutual coupling have been developed [9,10,11]. By setting a group of sensors as auxiliary sensors, the DOAs can be directly estimated without compensating for mutual coupling. Moreover, the result can be refined iteratively by estimating the mutual coupling coefficients for compensation. The main advantage of these methods is that no iteration is required but the aperture is reduced by making some sensors auxiliary.

All the algorithms reviewed above are only effective in a scenario where the incident signals are uncorrelated to each other and do not undergo multipath propagation. In a typical wireless communication system where multipath propagation is unavoidable, these algorithms may not be effective.

To tackle these two thorny issues, a class of auto-calibration methods, which are able to estimate the mutual coupling coefficients and DOAs of coherent signals jointly, has been addressed in [12,13]. In [12], the mutual coupling coefficients and coherent DOAs can be estimated jointly in an alternating manner, but this results in a large computational complexity for the nonlinear multidimensional search and is not applicable in some systems due to the requirement of pilot symbols. A noniterative optimisation method is given in [13] to estimate the DOAs of mixed signals which include coherent signals, but the mutual coupling coefficients have to be estimated from uncorrelated signals first. Hence, it may not be effective when there are only coherent signals. Recently, a spatial smoothing scheme studied in [14] estimates the DOAs of coherent signals and is insensitive to unknown mutual coupling, but only the forward smoothing was considered and the conjugate information contained within the signals was not utilised. More recently, Toeplitz matrix reconstruction directly in data domain [15] showed good performance for real-time applications in decorrelating coherent signals without mutual coupling compensation, but at the cost of halving the effective array aperture which is very limiting in practice.

In this correspondence, we propose a new matrix reconstruction method which is insensitive to mutual coupling and has an improved aperture after rank restoration. The main differences between our method and the methods in [14,15] lie in:

• Our method exploits the conjugate of the received data for the purpose of rank restoration, which gives rise to more degrees of freedom (DOF) as well as an extended aperture, while both the algorithms in [14,15] ignore such information, which significantly restricts the identifiability of DOA estimates.
• Our method makes use of all entries of the covariance matrix of the middle subarray, and the performance in terms of estimation resolution will thus be improved, whereas the spatial smoothing technique adopted in [14], only averages the covariance matrices of the subarrays but does not take advantage of the cross correlations between them, and thus the estimation performance will be compromised to some extent.
• Our approach offers a clear advantage in how it adapts to changes in the number of signals, whereas the method in [15] has a fixed number of DOF regardless of the number of signals.

These differences combine together so that our method can offer better estimation performance than existing techniques that incorporate forward-only averaging.

The rest of the paper is organised as follows. In Section 2, the signal model when coherent signals and mutual coupling coexist is presented. In Section 3, we develop a preprocessing strategy to estimate the DOAs of the coherent signals which is blind to the mutual coupling effects and does not require array calibration. In Section 4, simulation results show the validity and efficiency of our proposed method. Finally, some concluding remarks are given in Section 5.

Throughout this paper, the notations that will be used are listed as follows. The operators ${(·)}^T$, ${(·)}^{*}$, ${(·)}^H$, ${(·)}^{-1}$, $E[·]$, ${\parallel ·\parallel }_F$, $\lceil ·\rceil$, and $\lfloor ·\rfloor$ denote the operation of transpose, conjugate, conjugate transpose, inverse, expectation, the Frobenius norm, ceiling, and flooring of a decimal number, respectively. The symbol $\text{diag}\{z_1,z_2\}$ represents a diagonal matrix with diagonal entries $z_1,z_2$ and $\text{blkdiag}\{{\mathbf{Z}}_1,{\mathbf{Z}}_2\}$ represents a block diagonal matrix with diagonal entries ${\mathbf{Z}}_1,{\mathbf{Z}}_2$. The symbol $\text{rank}\left(\mathbf{Z}\right)$ denotes the rank of a matrix Z. The symbol $\mathbf{Z}(a:b,c:d)$ denotes a constructed submatrix by the entries from rows a to b and columns c to d of $\mathbf{Z}$, and the symbol $\mathbf{Z}(a,b)$ denotes the entry in the a-th row and b-th column of Z.

2. Problem Formulation

Consider a number of N narrowband far-field coherent signals impinging on a uniform linear array (ULA) with M identical omnidirectional sensors. Assume that these signals are classified into K groups, which result from K statistically uncorrelated far-field sources $s_k\left(t\right),k=1,2,\cdots ,K$ with $P_k$ multipath signals for each source. In the k-th coherent group, the signal coming from direction ${\theta }_{kp}$, $p=1,2,\cdots ,P_k$ corresponds to the p-th multipath propagation of the source $s_k\left(t\right)$, and the complex fading coefficient is ${\alpha }_{kp}$. It is readily seen that the total number of coherent signals satisfies $N={\sum }_{k=1}^K{P}_k$. Considering the effect of mutual coupling between the array elements, the corresponding $M\times 1$ array output vector is then given by

$$\begin{array}{cc}\hfill \mathbf{x}\left(t\right)& =\sum _{k=1}^K\sum _{p=1}^{P_k}\mathbf{C}\mathbf{a}\left({\theta }_{kp}\right){\alpha }_{kp}{s}_k\left(t\right)+\mathbf{n}\left(t\right)\hfill \\ & =\mathbf{C}\mathbf{A}\Gamma \mathbf{s}\left(t\right)+\mathbf{n}\left(t\right)\hfill \end{array}$$

(1)

where $\mathbf{a}\left(\theta \right)={\left[\begin{array}{c}1,e^{j\frac{2\pi d}{\lambda }sin \theta },\cdots ,e^{j\frac{2\pi (M-1)d}{\lambda }sin \theta }\end{array}\right]}^T\in {\mathbb{C}}^M$ is the steering vector with λ and d being the wavelength of carrier signal and the spacing between adjacent elements, respectively, C denotes the mutual coupling matrix (MCM), $\mathbf{A}=\left[\begin{array}{c}{\mathbf{A}}_1,\cdots ,{\mathbf{A}}_K\end{array}\right]$ with ${\mathbf{A}}_k=\left[\begin{array}{c}\mathbf{a}\left({\theta }_{k1}\right),\cdots ,\mathbf{a}\left({\theta }_{k{P}_k}\right)\end{array}\right]$, $\mathbf{\text{Γ}}=\text{blkdiag}\{{\alpha }_1,\cdots ,{\alpha }_K\}$ with ${\alpha }_k={\left[\begin{array}{c}{\alpha }_{k1},\cdots ,{\alpha }_{k{P}_k}\end{array}\right]}^T$ containing attenuation information of the k-th coherent group, $\mathbf{s}\left(t\right)={\left[\begin{array}{c}{s}_1\left(t\right),\cdots ,s_K\left(t\right)\end{array}\right]}^T$, and $\mathbf{n}\left(t\right)$ is white Gaussian noise with the power ${\sigma }_n^2$ for each entry. Besides, we assume that the array manifold A is unambiguous, i.e., the steering vectors ${\left\{\mathbf{a}\left({\theta }_i\right)\right\}}_{i=1}^N$ are linearly independent for any set of distinct ${\left\{{\theta }_i\right\}}_{i=1}^N$.

As described in [3,12,13,14,15], it is often sufficient to consider the ULA coupling model that has just finite non-zero coefficients, and a banded symmetric Toeplitz matrix can be used to model mutual coupling. To elaborate on the structure of the MCM, we consider the coupling effects to the i-th element shown in Figure 1 as a instance, where $S_{i,u}=c_{|u-i|}\mathbf{A}(u,:)\Gamma \mathbf{s}\left(t\right)$, $1\le i,u\le M$, is the coupling contribution from the u-th sensor to the i-th sensor with $c_{|u-i|}$ being the mutual coupling coefficient. It is known that the effect of mutual coupling between two sensor elements is inversely related to their distance and can be ignored when the separation is more than a few wavelengths. More precisely, we assume that when the distance between two sensors is more than $Pd$ where d is the inter-element spacings in a uniform linear array, the mutual coupling coefficient can be approximated as zero. We also assume that the mutual coupling coefficients only depend on the distance between two sensor elements. Based on these assumptions, the observed output at the i-th sensor can be expressed as

$$\begin{array}{cc}\hfill x_i\left(t\right)& =\sum _{\genfrac{}{}{0pt}{}{u=i-P+1}{1\le i,u\le M}}^{i+P-1}{S}_{i,u}+n_i\left(t\right)\hfill \\ & =\left[{\mathbf{0}}_{1\times (i-P)},c_{P-1},\cdots ,c_1,1,c_1,\cdots ,c_{P-1},{\mathbf{0}}_{1\times (M-i-P+1)}\right]\mathbf{A}\Gamma \mathbf{s}\left(t\right)+n_i\left(t\right)\hfill \end{array}$$

(2)

where $n_i\left(t\right)$ is the additive noise at the i-th sensor. Considering mutual coupling for all the M sensors and stacking $x_1\left(t\right),x_2\left(t\right),\cdots ,x_M\left(t\right)$ in a column, one obtains Equation (1) where

$$\begin{array}{cc}\hfill \mathbf{C}& =\left[\begin{array}{cccccc}1& c_1& \cdots & c_{P-1}& & \mathbf{0}\\ c_1& 1& c_1& \cdots & \ddots & \\ ⋮& c_1& 1& c_1& \cdots & c_{P-1}\\ c_{P-1}& \cdots & \ddots & \ddots & \ddots & ⋮\\ & \ddots & \cdots & c_1& 1& c_1\\ \mathbf{0}& & c_{P-1}& \cdots & c_1& 1\end{array}\right]\hfill \\ & =\text{Toeplitz}\left\{\left[1,c_1,\cdots ,c_{P-1},{\mathbf{0}}_{1\times (M-P)}\right]\right\}\hfill \end{array}$$

(3)

with $0<|c_1|,|c_2|,\cdots ,|c_{P-1}|<c_0=1$.

From Equation (1), the array covariance matrix is given by

$$\begin{array}{cc}\hfill {\mathbf{R}}_x& =E\left[\mathbf{x}\left(t\right){\mathbf{x}}^H\left(t\right)\right]=\mathbf{C}\mathbf{A}\Gamma {\mathbf{R}}_s{\Gamma }^H{\mathbf{A}}^H{\mathbf{C}}^H+{\sigma }_n^2{\mathbf{I}}_M\hfill \end{array}$$

(4)

where ${\mathbf{R}}_s=E\left[\mathbf{s}\left(t\right){\mathbf{s}}^H\left(t\right)\right]=\mathrm{diag}\{{\sigma }_1^2,\cdots ,{\sigma }_K^2\}$ is the signal covariance matrix, and ${\mathbf{I}}_M$ represents an $M\times M$ identity matrix. In the case of finite snapshots, the array covariance matrix can be calculated as ${\widehat{\mathbf{R}}}_x=\frac{1}{L}{\sum }_{t=1}^L\mathbf{x}\left(t\right){\mathbf{x}}^H\left(t\right)$, where L is the total number of snapshots.

Figure 1. Mutual coupling to the i-th element of a uniform linear array.

Figure 1. Mutual coupling to the i-th element of a uniform linear array.

3. DOA Estimation without Mutual Coupling Calibration

3.1. Mutual Coupling Circumvention

Referring to [9,10,11], we know that the middle subarray, defined as the middle $M-2P+2$ elements, is insensitive to mutual coupling, and its actual steering vector is equivalent to an ideal one (i.e., no coupling effect) scaled by a scalar. In order to combat signal coherency as well as effect of unknown mutual coupling in the ULA, we only make use of the output of the middle subarray by virtue of their steering vector. All other elements are defined to belong to auxiliary subarrays as shown in Figure 2. Inspired by [9], a selection matrix J is defined as

$$\begin{array}{cc}\hfill \mathbf{J}& =\left[\begin{array}{c}{\mathbf{0}}_{(M-2P+2)\times (P-1)},{\mathbf{I}}_{M-2P+2},{\mathbf{0}}_{(M-2P+2)\times (P-1)}\end{array}\right]\hfill \end{array}$$

(5)

and the covariance matrix of the the middle subarray is then given by

$$\begin{array}{cc}\hfill {\tilde{\mathbf{R}}}_x& =\mathbf{J}{\mathbf{R}}_x{\mathbf{J}}^H=\mathbf{J}\mathbf{C}\mathbf{A}\Gamma {\mathbf{R}}_s{\Gamma }^H{\mathbf{A}}^H{\mathbf{C}}^H{\mathbf{J}}^H+{\sigma }_n^2\mathbf{J}{\mathbf{J}}^H\hfill \\ & =\tilde{\mathbf{C}}\mathbf{A}\Gamma {\mathbf{R}}_s{\Gamma }^H{\mathbf{A}}^H{\tilde{\mathbf{C}}}^H+{\sigma }_n^2{\mathbf{I}}_{M-2P+2}\hfill \end{array}$$

(6)

where

$$\begin{array}{cc}\hfill \tilde{\mathbf{C}}& =\left[\begin{array}{cccccccc}{c}_{P-1}& \cdots & 1& \cdots & c_{P-1}& 0& \cdots & 0\\ 0& c_{P-1}& \cdots & 1& \cdots & c_{P-1}& \cdots & 0\\ ⋮& \ddots & \ddots & \cdots & \ddots & \cdots & \ddots & ⋮\\ 0& \cdots & 0& c_{P-1}& \cdots & 1& \cdots & c_{P-1}\end{array}\right].\hfill \end{array}$$

(7)

Figure 2. Allocation of the middle subarray and auxiliary subarray.

Figure 2. Allocation of the middle subarray and auxiliary subarray.

According to the selected MCM and signal models above, one has the following parameterisation for joint estimation of MCM and DOAs:

$$\begin{array}{cc}\hfill \tilde{\mathbf{C}}\mathbf{a}\left(\theta \right)& =\left[\begin{array}{c}{c}_{P-1}+\cdots +{\beta }^{P-1}+\cdots +c_{P-1}{\beta }^{2P-2}\\ c_{P-1}\beta +\cdots +{\beta }^P+\cdots +c_{P-1}{\beta }^{2P-1}\\ ⋮\\ c_{P-1}{\beta }^{M-2P+1}+\cdots +{\beta }^{M-P}+\cdots +c_{P-1}{\beta }^{M-1}\end{array}\right]\hfill \\ & =\left(c_{P-1}+\cdots +{\beta }^{P-1}+\cdots +c_{P-1}{\beta }^{2P-2}\right)\left[\begin{array}{c}1\\ \beta \\ ⋮\\ {\beta }^{M-2P+1}\end{array}\right]\hfill \\ & =\mu \left(\theta \right)\tilde{\mathbf{a}}\left(\theta \right)\hfill \end{array}$$

(8)

with $\mu \left(\theta \right)=c_{P-1}+\cdots +{\beta }^{P-1}+\cdots +c_{P-1}{\beta }^{2P-2}$, $\beta =e^{j\frac{2\pi d}{\lambda }sin \theta }$, and $\tilde{\mathbf{a}}\left(\theta \right)={\left[1,\beta ,\cdots ,{\beta }^{M-2P+1}\right]}^T$. It should be noted that generally $\mu \left(\theta \right)$ in Equation (8) is assumed to be nonzero. Hence, the covariance matrix of the middle array in Equation (6) can be rewritten as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{R}}}_x& =\left[\begin{array}{c}\tilde{\mathbf{a}}\left({\theta }_1\right),\tilde{\mathbf{a}}\left({\theta }_2\right),\cdots ,\tilde{\mathbf{a}}\left({\theta }_N\right)\end{array}\right]\mathrm{diag}\{\mu \left({\theta }_1\right),\mu \left({\theta }_2\right),\cdots ,\mu \left({\theta }_N\right)\}\Gamma {\mathbf{R}}_s\hfill \\ & \times {\Gamma }^H\mathrm{diag}{\{\mu \left({\theta }_1\right),\mu \left({\theta }_2\right),\cdots ,\mu \left({\theta }_N\right)\}}^H{\left[\begin{array}{c}\tilde{\mathbf{a}}\left({\theta }_1\right),\tilde{\mathbf{a}}\left({\theta }_2\right),\cdots ,\tilde{\mathbf{a}}\left({\theta }_N\right)\end{array}\right]}^H+{\sigma }_n^2{\mathbf{I}}_{M-2P+2}\hfill \\ & =\tilde{\mathbf{A}}\mathbf{D}\Gamma {\mathbf{R}}_s{\Gamma }^H{\mathbf{D}}^H{\tilde{\mathbf{A}}}^H+{\sigma }_n^2{\mathbf{I}}_{M-2P+2}\hfill \\ & =\tilde{\mathbf{A}}\tilde{\Gamma }{\mathbf{R}}_s{\tilde{\Gamma }}^H{\tilde{\mathbf{A}}}^H+{\sigma }_n^2{\mathbf{I}}_{M-2P+2}\hfill \end{array}$$

(9)

where $\tilde{\mathbf{A}}=\left[\begin{array}{c}\tilde{\mathbf{a}}\left({\theta }_1\right),\tilde{\mathbf{a}}\left({\theta }_2\right),\cdots ,\tilde{\mathbf{a}}\left({\theta }_N\right)\end{array}\right]$, $\mathbf{D}=\mathrm{diag}\{\mu \left({\theta }_1\right),\mu \left({\theta }_2\right),\cdots ,\mu \left({\theta }_N\right)\}$, and $\tilde{\Gamma }=\mathbf{D}\Gamma$ is still a $N\times K$ block diagonal matrix. This indicates that the middle subarray, which involves $M-2P+2$ sensors, is insensitive to mutual coupling but is still affected by signal coherency.

3.2. DOA Estimation of Coherent Signals

In this section, we will propose a rank restoration algorithm using an improved matrix reconstruction technique for DOA estimation of coherent signals in the presence of unknown mutual coupling.

Here a new matrix from the i-th row of ${\tilde{\mathbf{R}}}_x$ is constructed as

$$\begin{array}{cc}\hfill & \mathbf{B}\left(i\right)\hfill \\ & \triangleq \left[\begin{array}{cccccccc}{\tilde{\mathbf{R}}}_x(i,m)& {\tilde{\mathbf{R}}}_x(i,m+1)& \cdots & {\tilde{\mathbf{R}}}_x(i,m+q-1)& {\tilde{\mathbf{R}}}_x^{*}(i,1)& {\tilde{\mathbf{R}}}_x^{*}(i,2)& \cdots & {\tilde{\mathbf{R}}}_x^{*}(i,q)\\ {\tilde{\mathbf{R}}}_x(i,m-1)& {\tilde{\mathbf{R}}}_x(i,m)& \cdots & {\tilde{\mathbf{R}}}_x(i,m+q-2)& {\tilde{\mathbf{R}}}_x^{*}(i,2)& {\tilde{\mathbf{R}}}_x^{*}(i,3)& \cdots & {\tilde{\mathbf{R}}}_x^{*}(i,q+1)\\ ⋮& ⋮& \ddots & ⋮& ⋮& ⋮& \ddots & ⋮\\ {\tilde{\mathbf{R}}}_x(i,1)& {\tilde{\mathbf{R}}}_x(i,2)& \cdots & {\tilde{\mathbf{R}}}_x(i,q)& {\tilde{\mathbf{R}}}_x^{*}(i,m)& {\tilde{\mathbf{R}}}_x^{*}(i,m+1)& \cdots & {\tilde{\mathbf{R}}}_x^{*}(i,m+q-1)\end{array}\right]\hfill \end{array}$$

(10)

where $q=M-2P+3-m$. From Equations (1) and (9), the received signal ${\tilde{x}}_i\left(t\right)$ at the i-th element of the middle array is

$$\begin{array}{cc}\hfill {\tilde{x}}_i\left(t\right)& =\sum _{k=1}^K\sum _{p=1}^{P_k}\mu \left({\theta }_{kp}\right){\alpha }_{kp}{\beta }_{kp}^{i-1}{s}_k\left(t\right)+n_i\left(t\right),i=1,2,\cdots ,M-2P+2\hfill \end{array}$$

(11)

where ${\beta }_{kp}=e^{j\frac{2\pi d}{\lambda }sin {\theta }_{kp}}$ and $n_i\left(t\right)$ are the white Gaussian noise at the i-th sensor.

Then, the $(i,j)$-th entry of ${\tilde{\mathbf{R}}}_x$ can be expressed as

$$\begin{array}{cc}\hfill & {\tilde{\mathbf{R}}}_x(i,j)\hfill \\ & =E\left[{\tilde{x}}_i\left(t\right){\tilde{x}}_j^{*}\left(t\right)\right]\hfill \\ & =E[\left(\sum _{k=1}^K\sum _{p=1}^{P_k}\mu \left({\theta }_{kp}\right){\alpha }_{kp}{\beta }_{kp}^{i-1}{s}_k\left(t\right)+n_i\left(t\right)\right)\hfill \\ & \times \left(\sum _{k=1}^K\sum _{p=1}^{P_k}{\mu }^{*}\left({\theta }_{kp}\right){\alpha }_{kp}^{*}{\beta }_{kp}^{-(j-1)}{s}_k^{*}\left(t\right)+n_j^{*}\left(t\right)\right)]\hfill \\ & =\sum _{k=1}^K\sum _{l=1}^{P_k}{\beta }_{kl}^{-(j-1)}{\mu }^{*}\left({\theta }_{kl}\right){\alpha }_{kl}^{*}{\sigma }_k^2\sum _{p=1}^{P_k}\mu \left({\theta }_{kp}\right){\alpha }_{kp}{\beta }_{kp}^{i-1}+{\sigma }_n^2{\delta }_{i,j}\hfill \end{array}$$

(12)

where ${\delta }_{i,j}$ is the Kronecker delta. Substituting Equation (12) into the matrix in Equation (10) allows $\mathbf{B}\left(i\right)$ to be written as

$$\begin{array}{cc}\hfill \mathbf{B}\left(i\right)& =\left[\begin{array}{c}{\mathbf{A}}_c{\Phi }^{1-m}\mathbf{b}\left(i\right),{\mathbf{A}}_c{\Phi }^{-m}\mathbf{b}\left(i\right),\cdots ,{\mathbf{A}}_c{\Phi }^{2-m-q}\mathbf{b}\left(i\right),{\mathbf{A}}_c\mathbf{b}\left(i\right),{\mathbf{A}}_c\Phi \mathbf{b}\left(i\right),\cdots ,{\mathbf{A}}_c{\Phi }^{q-1}\mathbf{b}\left(i\right)\end{array}\right]\hfill \\ & +\left[\begin{array}{c}{\mathbf{K}}_i,{\mathbf{L}}_i\end{array}\right]\hfill \\ & ={\mathbf{A}}_c\left[\begin{array}{c}{\Phi }^{1-m}\mathbf{b}\left(i\right),{\Phi }^{-m}\mathbf{b}\left(i\right),\cdots ,{\Phi }^{2-m-q}\mathbf{b}\left(i\right),\mathbf{b}\left(i\right),\Phi \mathbf{b}\left(i\right),\cdots ,{\Phi }^{q-1}\mathbf{b}\left(i\right)\end{array}\right]+\left[\begin{array}{c}{\mathbf{K}}_i,{\mathbf{L}}_i\end{array}\right]\hfill \\ & ={\mathbf{A}}_c\mathbf{G}\left(i\right)+\left[\begin{array}{c}{\mathbf{K}}_i,{\mathbf{L}}_i\end{array}\right]\hfill \end{array}$$

(13)

where ${\mathbf{A}}_c=\mathbf{F}\left[\begin{array}{c}\tilde{\mathbf{a}}\left({\theta }_1\right),\tilde{\mathbf{a}}\left({\theta }_2\right),\cdots ,\tilde{\mathbf{a}}\left({\theta }_N\right)\end{array}\right]$, $\mathbf{F}=\left[\begin{array}{c}{\mathbf{I}}_{M-2P+3-q},{\mathbf{0}}_{(M-2P+3-q)\times (q-1)}\end{array}\right]$, $\mathbf{b}\left(i\right)={\left[b_{11}\left(i\right),\cdots ,b_{1P_1}\left(i\right),\cdots ,b_{K1}\left(i\right),\cdots ,b_{K{P}_K}\left(i\right)\right]}^T$, $b_{kp}\left(i\right)={\sigma }_k^2{\mu }^{*}\left({\theta }_{kp}\right){\alpha }_{kp}^{*}{\sum }_{l=1}^{P_k}\mu \left({\theta }_{kl}\right){\alpha }_{kp}{\beta }_{kl}^{i-1}$, $\mathbf{G}\left(i\right)=\left[{\Phi }^{1-m}\mathbf{b}\left(i\right),{\Phi }^{-m}\mathbf{b}\left(i\right),\cdots ,{\Phi }^{2-m-q}\mathbf{b}\left(i\right),\mathbf{b}\left(i\right),\Phi \mathbf{b}\left(i\right),\cdots ,{\Phi }^{q-1}\mathbf{b}\left(i\right)\right]$ with $\Phi =\text{diag}\left\{{\beta }_{11},\cdots ,{\beta }_{1P_1},\cdots ,{\beta }_{K1},\cdots ,{\beta }_{K{P}_K}\right\}$, ${\mathbf{K}}_i,{\mathbf{L}}_i\in {\mathbb{R}}^{m\times q}$, and their entries are defined as ${\mathbf{K}}_i(u,v)=\left\{\begin{array}{cc}\hfill {\sigma }_n^2,& u-v+i=m\hfill \\ \hfill 0,& \text{otherwise}\hfill \end{array}\right.$, ${\mathbf{L}}_i(u,v)=\left\{\begin{array}{cc}\hfill {\sigma }_n^2,& u+v=i+1\hfill \\ \hfill 0,& \text{otherwise}\hfill \end{array}\right.$, respectively.

It is known that the effect of having only a finite number of snapshots, especially when the number is small, may cause biased estimates of subspaces. To further restore the rank and make the matrix reconstruction method more robust against the effects of noise and finite snapshots, all the $M-2P+2$ rows of ${\tilde{\mathbf{R}}}_x$ are then exploited to construct the following square matrix

$$\begin{array}{cc}\hfill \overline{\mathbf{R}}& =\sum _{i=1}^{M-2P+2}\mathbf{B}\left(i\right){\mathbf{B}}^H\left(i\right)\hfill \\ & ={\mathbf{A}}_c\left(\sum _{i=1}^{M-2P+2}\mathbf{G}\left(i\right){\mathbf{G}}^H\left(i\right)\right){\mathbf{A}}_c^H+\sum _{i=1}^{M-2P+2}\left({\mathbf{K}}_i{\mathbf{K}}_i^H+{\mathbf{L}}_i{\mathbf{L}}_i^H\right)\hfill \end{array}$$

(14)

It can be readily verified that ${\sum }_{i=1}^{M-2P+2}\left({\mathbf{K}}_i{\mathbf{K}}_i^H+{\mathbf{L}}_i{\mathbf{L}}_i^H\right)=\eta {\mathbf{I}}_m$ where

$$\eta =\left\{\begin{array}{cc}\hfill 2(m-1){\sigma }_n^2,& m>q\hfill \\ \hfill 2q{\sigma }_n^2,& m\le q\hfill \end{array}\right.$$

(15)

Defining ${\mathbf{R}}_d\triangleq {\mathbf{A}}_c\left({\sum }_{i=1}^{M-2P+2}\mathbf{G}\left(i\right){\mathbf{G}}^H\left(i\right)\right){\mathbf{A}}_c^H$, then $\overline{\mathbf{R}}={\mathbf{R}}_d+\eta {\mathbf{I}}_m$.

Next we examine whether the rank of ${\mathbf{R}}_d$ has been restored sufficiently to resolve all N incident coherent signals.

Proposition 1. 

When $m\ge N+1$ and $2q\ge P_{max}$, $rank\left({\mathbf{R}}_d\right)=N$ for K groups of coherent signals, where $P_{max}=max\{P_1,P_2,\cdots ,P_K\}$.

Proof: 

Since ${\mathbf{A}}_c$ is unambiguous and $m>N$, one has $\text{rank} \left({\mathbf{A}}_c\right)=N$. It is easy to identify that $\text{rank} \left({\mathbf{R}}_d\right)=\text{rank} \left({\sum }_{i=1}^{M-2P+2}\mathbf{G}\left(i\right){\mathbf{G}}^H\left(i\right)\right)$. Referring to the Lemma 1 in [16], we know that given $\mathbf{H}\in {\mathbb{C}}^{I\times J}$, the diagonal matrix Q has no identical entries on the diagonal, and $\text{rank} \left(\mathbf{H}\right)=r<I$, then $\text{rank} \left(\left[\begin{array}{c}\mathbf{H},\mathbf{Q}\mathbf{H}\end{array}\right]\right)=r+1$. Substituting back into our problem, we find that the diagonal entries of Φ are not the same as each other, and $\text{rank} \left(\mathbf{b}\left(i\right)\right)=1$, and thus $\text{rank} \left(\left[\mathbf{b}\left(i\right),\Phi \mathbf{b}\left(i\right)\right]\right)=2$. If Lemma 1 is applying successively to the submatrices in $\mathbf{G}\left(i\right)$, one has a) if $2q\ge N$, then $\text{rank} \left(\mathbf{G}\left(i\right)\right)=N$; b) $\text{rank} \left(\mathbf{G}\left(i\right)\right)=2q$, otherwise. If the latter holds, according to the assumption that the K coherent groups are uncorrelated to each other, ${\tilde{\mathbf{R}}}_x$ has K linearly independent rows. Therefore, $\text{rank} \left({\mathbf{R}}_d\right)=\text{rank} \left({\sum }_{i=1}^{M-2P+2}\mathbf{G}\left(i\right){\mathbf{G}}^H\left(i\right)\right)=\text{min}\{2qK,N\}$. As $2q\ge P_{max}$, one has $2qK\ge K{P}_{max}\ge N$, and $\text{rank} \left({\mathbf{R}}_d\right)=N$ eventually.              ■

Remark 1. 

The parameter m plays a significant role in rank restoration since it determines the array’s DOF and effective aperture. We consider two extreme cases to discuss the choices of m. If $m=M-2P+3-\lceil \frac{P_{max}}{2}\rceil$, i.e., m achieves its upper bound, then one barely restores the rank deficiency with the minimum number of columns of $\mathbf{B}\left(i\right)$, but this may result in some signals being undetected, especially at low SNRs or for few snapshots. On the other hand, If $m=N+1$, i.e., m achieves its lower bound, then one restores the rank deficiency with redundant columns of $\mathbf{B}\left(i\right)$, but this may cause biased estimates due to the noise subspace being one-dimensional. Based on our simulation results, the proposed method performs well when m lies between these two bounds, away from the extremes. However, the optimum choice of m is still an open problem and is beyond the scope of this paper.

When $m>N$ and $2q\ge P_K$, the non-singularity of the smoothed covariance matrix $\overline{\mathbf{R}}$ can be recovered, then the MUSIC algorithm can be employed to estimate the DOAs of coherent signals. More specifically, one can perform the eigen-decomposition of $\overline{\mathbf{R}}$ as

$$\begin{array}{cc}\hfill \overline{\mathbf{R}}& ={\mathbf{U}}_s{\Sigma }_s{\mathbf{U}}_s^H+{\mathbf{U}}_n{\Sigma }_n{\mathbf{U}}_n^H\hfill \end{array}$$

(16)

where ${\Sigma }_s$ is a diagonal matrix consisting of the N largest eigenvalues, and ${\Sigma }_n$ is a diagonal matrix consisting of the $m-N$ smallest eigenvalues. The columns of ${\mathbf{U}}_s$ are the eigenvectors corresponding to the N largest eigenvalues, while the columns of ${\mathbf{U}}_n$ are the eigenvectors corresponding to the $m-N$ smallest eigenvalues. According to the subspace principle, the columns of ${\mathbf{U}}_s$ span the signal subspace, which coincides with the range space of A, and the signal subspace is orthogonal to the noise subspace spanned by the columns of ${\mathbf{U}}_n$. Thus,

$$\begin{array}{c}\hfill \parallel {\mathbf{a}}^H\left({\theta }_i\right){\mathbf{U}}_n{\parallel }_F^2=0,i=1,2,\cdots ,N\end{array}$$

(17)

This indicates that the DOAs of the coherent signals can be obtained by finding the N peaks from the spatial spectrum function

$$\begin{array}{cc}\hfill P\left(\theta \right)& =\frac{1}{\parallel {\mathbf{a}}^H\left(\theta \right){\mathbf{U}}_n{\parallel }_F^2}\hfill \end{array}$$

(18)

So far, we have described the proposed algorithm for coherent signal DOA estimation in the presence of unknown mutual coupling. The major steps of the proposed algorithm are summarised as follows:

Step 1. Obtain L snapshots of the received signal $\mathbf{x}\left(t\right)$ at $t=t_1,t_2,\cdots ,t_L$, and form the following matrix as:

$$\begin{array}{cc}\hfill \mathbf{x}& =\left[\mathbf{x}\left(t_1\right),\mathbf{x}\left(t_2\right),\cdots ,\mathbf{x}\left(t_L\right)\right]\hfill \end{array}$$

(19)

Step 2. Calculate the covariance matrix using the above data matrix by

$$\begin{array}{cc}\hfill {\mathbf{R}}_x& =\frac{1}{L}\mathbf{x}{\mathbf{x}}^H\hfill \end{array}$$

(20)

Step 3. Select the middle array and calculate corresponding covariance matrix ${\tilde{\mathbf{R}}}_x$ according to Equation (6);

Step 4. Utilise rows of ${\tilde{\mathbf{R}}}_x$ to construct the matrix $\mathbf{B}\left(i\right),i=1,2,\cdots ,M-2P+2$, according to Equation (10);

Step 5. Construct the overall rank-restored matrix $\overline{\mathbf{R}}$ as Equation (14);

Step 6. Perform the eigen-decomposition of $\overline{\mathbf{R}}$, and obtain the noise subspace ${\mathbf{U}}_n$;

Step 7. Scan the direction over [−90°, 90°] with a step size of 0.1°. Calculate the spacial spectrum using Equation (18), and obtain the DOA estimates ${\theta }_1,{\theta }_2,\cdots ,{\theta }_N$.

Although the mutual coupling information for a sensor array is a cumbersome calculation using electromagnetic simulators and measurement in practice, for some circumstances, such as high precision applications in satellite navigation and airborne early warning radar, it has to be obtained and compensated for a-priori. In the presence of known mutual coupling, we should first eliminate the effects of mutual coupling before applying the matrix reconstruction for rank restoration. To be specific, the following steps have to be carried out:

Step 1. Perform eigen-decomposition of ${\mathbf{R}}_x$, one has

$$\begin{array}{cc}\hfill {\mathbf{R}}_x& ={\tilde{\mathbf{U}}}_s{\tilde{\Sigma }}_s{\tilde{\mathbf{U}}}_s^H+{\tilde{\mathbf{U}}}_n{\tilde{\Sigma }}_n{\tilde{\mathbf{U}}}_n^H\hfill \end{array}$$

(21)

Since ${\tilde{\mathbf{U}}}_s$ and $\mathbf{C}\mathbf{A}\Gamma$ span the same signal subspace, there holds ${\tilde{\mathbf{U}}}_s=\mathbf{C}\mathbf{A}\Gamma \mathbf{T}$ where $\mathbf{T}$ is a nonsingular matrix.

Step 2. Reconstruct a covariance matrix by compensating for mutual coupling effects as

$$\begin{array}{cc}\hfill {\mathbf{R}}_o& ={\mathbf{C}}^{-1}{\tilde{\mathbf{U}}}_s{\tilde{\Sigma }}_s{\tilde{\mathbf{U}}}_s^H{\left({\mathbf{C}}^H\right)}^{-1}\hfill \end{array}$$

(22)

Then, following the similar Steps 4–7 in the unknown mutual coupling case one can resolve the coherent signals.

Remark 2. 

Once the mutual coupling information is known, one can compensate the coupling effects by Equation (22), and then the additional information of the auxiliary sensors, which had to be abandoned in the case of unknown mutual coupling, can now be exploited. Compared with Equation (14) using ${\left\{\mathbf{B}\left(i\right)\right\}}_{i=1}^{M-2P+2}$, there are more matrices ${\left\{\mathbf{B}\left(i\right)\right\}}_{i=1}^M$ available after mutual coupling compensation, which means that more DOF and a larger effective aperture can be achieved, and improved estimation performance can thus be expected. In fact, our method can handle up to $M-\lceil \frac{P_{max}}{2}\rceil$ coherent signals in the presence of known mutual coupling. The identifiability of DOA estimation for the proposed and comparative methods with unknown mutual coupling will be discussed in Section 3.3.

3.3. Separable Signal Number

The coherent signals are resolved with an increased effective array aperture compared with previous techniques. This makes better use of the DOF of the original ULA and allows more signals to be resolved. This motivates us to discuss the number of separable signals of the proposed method. Compared with the standard spatial smoothing (SS) technique, which estimates the coherent signals by averaging subarrays, under unknown mutual coupling [14], the maximum number of signals estimated by our method can be relaxed. If $m>N$ and $2q\ge P_{max}$, the proposed method can estimate a maximum number of $M-2P+2-\lceil \frac{P_{max}}{2}\rceil$ coherent signals since $M-2P+2\ge N+\lceil \frac{P_{max}}{2}\rceil$, while the standard spatial smoothing can estimate at most $M-2P+2-P_{max}$ coherent signals since $P_{max}$ subarrays are required and $M-2P+3-P_{max}>N$. As we can see, the key factor influencing the number of resolvable coherent signals is the maximum number of signals in one coherent group, in other words, the larger $P_{max}$ is, the better our approach performs. Our approach also has better DOA estimation accuracy than SS as will be demonstrated by the simulation results in the next section. The method by Mao et al. [15], referred to as matrix reconstruction in data (MRD), can also achieve better DOA estimation accuracy than SS, provided that the effective array aperture is the same, but this comes at a cost of the number of signals which can be resolved. It can resolve at most $\lfloor \frac{M-2P+1}{2}\rfloor$ coherent signals and the array aperture is fixed regardless of the number of coherent sources. The spatial smoothing techniques are more flexible in this aspect as the number of subarrays and effective array aperture can be varied. Both the algorithms in [14,15] also ignore the conjugate of the received data in rank restoration, which significantly restricts the number of resolvable signals in practice.

Table 1 lists the minimum number of array elements required to resolve a given number of signals by the three methods. For simplicity, we assume that each group has the same number of coherent signals. We can see that our method can use less array elements than the other two algorithms, to estimate the same number of signals.

Table 1. Minimum number of array elements required.

Coupling Length	Coherent Signals		Total Signals	Number of Array Elements
	Groups	Signals in Each Group		SS	MRD	Proposed Method
2	1	2	2	6	7	5
2	2	2	4	8	11	7
3	1	4	4	12	13	10
3	2	3	6	13	17	12
4	1	6	6	18	19	15
4	2	4	8	18	23	16
4	2	6	12	24	31	21
3	3	6	18	28	41	23

Table 1. Minimum number of array elements required.

Coupling Length	Coherent Signals		Total Signals	Number of Array Elements
	Groups	Signals in Each Group		SS	MRD	Proposed Method
2	1	2	2	6	7	5
2	2	2	4	8	11	7
3	1	4	4	12	13	10
3	2	3	6	13	17	12
4	1	6	6	18	19	15
4	2	4	8	18	23	16
4	2	6	12	24	31	21
3	3	6	18	28	41	23

4. Simulation Results and Discussion

In this section, a series of numerical experiments under different conditions are conducted to examine the performance of the proposed method. Simulations are carried out for a 14-element ULA with half-wavelength spacing between adjacent elements. For simplicity, we assume that all coherent signals have identical power ${\sigma }_s^2$, and the input SNR is defined as $10{log}_{10}({\sigma }_s^2/{\sigma }_n^2)$. SS, MRD, and the proposed method with known mutual coupling dealt with in Equation (22) are chosen for comparison. The mutual coupling is assumed can be negligible at a distance larger than $1.5\lambda$. Hence, $P=3$ and the mutual coupling coefficients are assumed to be $c_0=1,c_1=-0.1545+0.4755j,c_2=0.1618-0.1176j$. The accuracy of the DOA estimate is measured from 1000 Monte Carlo runs in terms of the root mean square error (RMSE) which is defined as

$$\begin{array}{cc}\hfill \mathrm{RMSE}& =\sqrt{\frac{1}{1000N}\sum _{n=1}^{1000}\sum _{i=1}^N{({\widehat{\theta }}_i^{\left(n\right)}-{\theta }_i)}^2}\hfill \end{array}$$

(23)

where ${\widehat{\theta }}_i^{\left(n\right)}$ is the estimate of ${\theta }_i$ for the n-th trial, and N is the number of coherent signals. Additionally, to assess the overall reliability of all the algorithms, the probability of resolution is defined as

$$\text{Probability} \text{of} \text{resolution}=\frac{F_r}{F}$$

(24)

where F is the number of trials, and $F_r$ is the number of successful estimations for which the absolute DOA estimation errors are within 1°.

4.1. DOA Estimation of Coherent Signals From One Group

Figure 3 depicts the RMSE of DOA estimates of a group of four coherent signals from [−26°, −8°, 9°, 30°]. The fading amplitudes and phases of the coherent signals are [1, 0.8, 0.7, 0.4] and [25.7°, 71.17°, 300.83°, 128.37°], respectively. Here, for fair comparison it is first assumed that the three algorithms have an identical aperture after rank restoration, i.e., m = 5. From this figure, we can see that as the SNR and the number of snapshots increase, the RMSE of DOA estimation decreases gradually for all of the methods. The proposed method is significantly superior to MRD and SS, especially at low SNRs and few snapshots, since more column vectors have been utilised for rank restoration with the same array aperture, and is only inferior to the one when mutual coupling is known. Although MRD slightly outperforms SS, both have poor DOA estimation accuracy at low SNRs and few snapshots, and there is a clear discrepancy between these two and the proposed method even for snapshot sizes up to 1000 and beyond at 0 dB as shown in Figure 3b.

Figure 4 depicts the probability of resolution of successful estimation versus input SNR and the number of snapshots for the same scenario as in Figure 3. It can be observed that all methods attain a 100% successful estimation above 11 dB. As the SNR decreases, the probability of successful estimation starts dropping for each method at a certain point, known as SNR threshold, until it eventually becomes zero. With unknown mutual coupling, the proposed method has the lowest SNR threshold followed by MRD and then SS which has the highest SNR threshold. The performance of the proposed method when the mutual coupling is known is also shown for comparison. Evidently the proposed method is superior to the counterparts. When the SNR is fixed at 0 dB, our approach can achieve 100% probability with unknown mutual coupling above 1800 snapshots. Even if the number of snapshots is greater than 2000, the MRD and SS algorithms can barely reach 50% success probability. Consistent with the illustration in Figure 3, MRD has a slightly better performance than SS in terms of probability of resolution versus SNR and the number of snapshots.

The simulations above demonstrate the performance of all algorithms with an identical aperture m = 5. As discussed in Section 3.3, the effective array aperture after rank restoration by MRD is always halved, in other words, the effective array aperture is fixed and cannot adapt to changes in the number of signals. In contrast, both the proposed and SS algorithm are adaptive to complex situations, such as multiple coherent groups, allowing a trade off between array aperture and rank restoration. To evaluate this advantage, we set m = 6 for the proposed and SS algorithm and compare them with the fixed array aperture MRD algorithm, i.e., m = 5, while keeping the other configurations the same as the first scenario.

Figure 3. Root mean square error (RMSE) of the direction of arrival (DOA) estimates of one group coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 5.

Figure 3. Root mean square error (RMSE) of the direction of arrival (DOA) estimates of one group coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 5.

Figure 4. Probability of resolution for one group of coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 5.

Figure 4. Probability of resolution for one group of coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 5.

Figure 5 and Figure 6 show that compared with Figure 3 and Figure 4 the performance of both our approach and SS meliorates, and our approach is still superior to SS and MRD and approaches to the case with known mutual coupling, while the SS method has the largest improvement in the accuracy and reliability. This is mainly because the effective array aperture is improved while keeping the rank completely restored. This group of simulations fully demonstrates that the proposed and SS algorithm have a clear advantage over the MRD algorithm, and the fixed aperture imposed by MRD is not optimal for rank restoration.

Figure 5. RMSE of the DOA estimates of one group coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6 for the proposed and SS method.

Figure 5. RMSE of the DOA estimates of one group coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6 for the proposed and SS method.

Figure 6. Probability of resolution for one group of coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6 for the proposed and SS method.

Figure 6. Probability of resolution for one group of coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6 for the proposed and SS method.

4.2. DOA Estimation of Coherent Signals From Multiple Groups

We consider two groups of five coherent signals from [−36°, −22°] and [−10°, 5°, 19°] impinge on the ULA. The fading amplitudes of the coherent signals are [1, 0.9] and [1, 0.8, 0.6], while the fading phases are [48.74°, 121.15°] and [189.35°, 35.66°, 283.56°], respectively. The total number of signals is equal to half the elements in the middle array, and thus MRD fails to work. To enable the proposed algorithm and SS work, we select m = 6, 7 for rank restoration.

Based on the simulation settings, the results of the RMSE versus SNR and the number of snapshots are depicted in Figure 7. As shown in this figure, the proposed approach performs the best over the whole range of SNR and the number of snapshots values for coherent signal estimation. When m = 6, the proposed algorithm with known or unknown mutual coupling is inferior to the first scenario. One possible explanation is that the fading coefficients in the two scenarios do not have same level of impact for rank restoration in general. SS has larger estimation errors than our method as it only exploits of 4 subarrays for rank restoration while our method effectively utilises twice as many. When m = 7, both the proposed and SS algorithm achieve a better performance since the limitation of effective array aperture is relaxed. Although the SS algorithm improves significantly for m = 7, it is still notably worse than our proposed method.

Next, in Figure 8 we plot the probability of resolution of the two methods by varying the SNR and total number of snapshots. It can be seen that the proposed approach still outperforms the SS algorithms for both configuration m = 6, 7. The performance of the SS method improves considerably when m = 7 but is strictly worse than our method.

Figure 7. RMSE of the DOA estimates of two groups coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6, 7.

Figure 7. RMSE of the DOA estimates of two groups coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6, 7.

Figure 8. Probability of resolution for two groups of coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6, 7.

Figure 8. Probability of resolution for two groups of coherent signals versus (a) SNR when the number of snapshots is 500; (b) the number of snapshots when SNR = 0 dB. Effective array aperture m = 6, 7.

5. Conclusions

This paper addresses the problem of DOA estimation when unknown mutual coupling and multipath coexist. An efficient approach has been presented that first circumvents the unknown mutual coupling effect using the output of the selected middle array, and then reconstructs the covariance matrix of the middle array for rank restoration. The proposed method has a better DOA estimation accuracy than the existing methods and can resolve more coherent signals. This is achieved due to two main factors. Firstly replacing standard spatial smoothing in the rank restoration step by a new matrix reconstruction algorithm takes advantage of the information in the off the main diagonal entries of the spatial covariance matrix. Secondly the new matrix reconstruction algorithm for restoring the rank of the coherent signal subspace exploits the conjugate information of the entries of the covariance matrix. These are ignored by standard spatial smoothing techniques and can effectively improve the array aperture or equivalently improve the rank restoration. Extensive simulation results demonstrate the validity and efficiency of the proposed method.