DOA Estimation Using Fourth-Order Cumulants in Nested Arrays with Structured Imperfections

Abstract

Recently developed super nested array families have drawn much attention owing to their merits on keeping the benefits of the standard nested arrays while further mitigating coupling in dense subarray portions. In this communication, a new mutual coupling model for nested arrays is constructed. Analyzing the structure of the newly formed mutual coupling matrix, a transformation of the distorted steering vector to separate angular information from the mutual coupling coefficients is revealed. By this property, direction of arrival (DOA) estimates can be determined via a grid search for the minimum of a determinant function of DOA, which is induced by the rank reduction property. We also extend the robust DOA estimation method to accommodate the unknown mutual coupling and gain-phase mismatches in the nested array. Compared with the schemes of super nested array families on reducing the mutual coupling effects, the solutions presented in this paper has two advantages: (a) It is applicable to the standard nested arrays without rearranging the configuration to increase the inter-element spacing, alleviating the cross talk in dense uniform linear arrays (ULAs) as well as gain-phase errors in sparse ULA parts; (b) Perturbations in nested arrays are estimated in colored noise, which is significant but rarely discussed before. Simulations results corroborate the superiority of the proposed methods using fourth-order cumulants.

1. Introduction

Sensor array signal processing is a significant research area owing to its wide applications to radar, sonar, navigation, wireless communications, et al. [1,2,3,4]. Virtues of array signal processing, such as direction of arrival (DOA) estimation and beamforming, lie in spatial diversity and capability to determine the angular information of electromagnetic waves and enhance the reception of signals of interest while mitigating the interferences [5,6,7,8,9]. The basic but widely employed array geometry is the uniform linear array (ULA), where the inter-element spacing is usually less than or equal to half a wavelength to avoid spatial aliasing. However, the performance of ULAs is subject to a limited degrees of freedom (DOFs) that is linearly dependent on to the number of sensors. For instance, an M-elements ULA can offer up to $M-1$ DOFs, i.e., only $M-1$ signals can be resolved at most [10,11]. To increase the DOFs, a simple way is to introduce extra sensors, which may result in to difficulties in practical implementations. Besides, ULAs with a large number of sensors may suffer from prominent mutual coupling effects between physical sensors.

Sparse linear arrays (SLAs), a promising countermeasure, can tackle the above issues to a large extent. The critical operation on the DOFs enhancement is to vectorize the covariance matrix of the observations, rendering $\mathcal{O}\left(M^2\right)$ virtual sensors with M physical elements in the difference coarrays, which means one can probably resolve more DOAs than the physical sensors provided that there is no correlation between signals [12]. Compared with the ULA, this is a big leap in raising DOFs. In addition, the electromagnetic cross talk between sensors is alleviates since the inter-element spacing becomes larger in SLAs [13].

Recently, a new type of sparse array, so-called the nested array [14], is put forward in signal processing society and causes tremendous repercussions because the positions of sensors can be arranged in a regular design, the synthetic apertures becomes larger, and the resultant enhanced DOFs have a closed form expression with respect to the given number of sensors [15,16,17,18,19,20,21,22]. The nested array is constructed by cascading ULAs that differ in systematically designed inter-element spacing. In particular, the difference coarray formed from a two-level nested array has consecutive virtual sensors without any holes, while the hole-free property in the difference coarray cannot be guaranteed by higher level nested arrays that possess higher DOFs. The DOFs of nested arrays can be further increased by making the outer ULA much sparser, i.e., enlarging its aperture, as well as putting up an auxiliary sensor [21]. In [23,24], the one-dimensional nested array is extended to the two-dimensional case.

However, the mutual coupling effects of the nested array cannot be ignored at least as physical sensors in the inner ULA are deployed relatively close [25,26,27,28,29,30,31,32,33,34,35,36,37,38]. In [39], the super nested array in the context of the second-order statistics is designed to significantly mitigate the cross talk between sensors while preserving all advantages of the standard nested arrays, by increase the inter-element spacing of the inner ULA to maintain the coarray but alleviate the adverse electromagnetic effects. In [40], a high order super nested array is introduced to further reduce the mutual coupling between sensors. In [41], by splitting the inner ULA into two or four sections and rearranging them at the two sides of the remaining outer ULA, the so-called augmented nested array (ANA) is developed. Compared with the super nested array, the ANA can further reduce the unknown mutual coupling but at a cost of complicated design to guarantee the difference coarray hole-free. In [42], a new sparse array configuration, namely MISC array, is optimized to obtain maximum inter-element spacing, where a reasonable distribution of three sparse ULAs and two additional sensors is necessary. The MISC array is less susceptive to mutual coupling effects while achieving higher DOFs.

It should be noted that: (a) the aforementioned nested array configurations cannot fundamentally avoid the mutual coupling effects, i.e., all the DOA estimation methods performing on such arrays still work in the presence of perturbations, and (b) the existing nested array processing using the second-order statistics is vulnerable to colored noise. To obtain more robust estimates, we would deal with these two issues from another perspective—consider the array imperfections and jointly resolve DOAs and the parameterized perturbations—rather than reduce the errors by a new geometry design. This idea is reasonable in the sense that the super nested arrays and its successors occupying large areas are not applicable to some circumstances, such as airborne platforms, with a limited space for devices. In this correspondence, a new cumulant-based DOA estimation method for nested arrays with perturbations is proposed. The mutual coupling of the nested array is first analysed and parameterized in terms of the relationship of coupling range and inter-element spacing, giving a new structure of the mutual coupling matrix (MCM). Then we provide a proof that the distorted steering vector can be factorized into a new matrix, including angular information only, multiplied by the coupling coefficient vector. Leveraging this property, we develop a MUSIC-like estimator to determine the DOA estimates based on the rank reduction (RARE) technique in the context fourth-order cumulants (FOC). In addition, the proposed method can be extended to adapt to a more challenging scenario where mutual coupling and gain-phase errors coexist. Analytical specifications show that our solutions are more robust to array imperfections as compared to the standard nested array processing.

The remainder of this paper is organized as follows. In Section 2, an nested array model with unknown mutual coupling is introduced. In Section 3, the MUSIC-like estimators for nested arrays with perturbations using fourth-order cumulants are developed. Section 4 provides numerical examples for demonstrating the validity and efficiency of our proposed algorithms. Finally, some concluding remarks are given in Section 5.

2. Problem Formulation

2.1. Signal Model

Consider N narrowband non-Gaussian signals impinging on a linear M-elements nested array, composed of two different ULAs concatenating together. Assume that the inter-element spacing of the inner ULA with $M_1$ elements is $d_I$ and the inter-element spacing of the outer ULA with $M_2$ elements is $d_O$ satisfying $d_O=(M_1+1)d_I$. Without loss of generality, we set $d_I=\frac{\lambda }{2}$ with $\lambda$ being the wavelength. To be specific, the sensors of a 2-level nested array are deployed at the positions that can be formulated as a union of the position set of inner ULA, ${\mathbb{S}}_I$, and the position set of outer ULA, ${\mathbb{S}}_O$, which can be expressed as

$$\begin{array}{cc}\hfill \mathbb{P}& ={\mathbb{S}}_I\cup {\mathbb{S}}_O\hfill \\ \hfill & =\left\{m{d}_I\right|m=0,\cdots ,M_1-1\}\cup \left\{(n(M_1+1)+M_1)d_I\right|n=0,\cdots ,M_2-1\}\hfill \end{array}$$

(1)

If there exits mutual coupling between the array elements, the corresponding $M\times 1$ array observation vector is then given by

$$\begin{array}{cc}\hfill \mathbf{x}\left(t\right)& =\sum _{i=1}^N\mathbf{C}\mathbf{a}\left({\theta }_i\right)s_i\left(t\right)+\mathbf{n}\left(t\right)=\mathbf{C}\mathbf{A}\mathbf{s}\left(t\right)+\mathbf{n}\left(t\right)\hfill \end{array}$$

(2)

where $\mathbf{a}\left({\theta }_i\right)={\left[\begin{array}{c}1,e^{j\frac{2\pi p_2}{\lambda }sin{\theta }_i},\cdots ,e^{j\frac{2\pi p_M}{\lambda }sin{\theta }_i}\end{array}\right]}^T\in {\mathbb{C}}^M$ is the steering vector, $\mathbf{A}=\left[\begin{array}{c}\mathbf{a}\left({\theta }_1\right),\cdots ,\mathbf{a}\left({\theta }_N\right)\end{array}\right]$ is the array manifold, $\mathbf{C}$ denotes the MCM, and $\mathbf{s}\left(t\right)={\left[\begin{array}{c}{s}_1\left(t\right),\cdots ,s_N\left(t\right)\end{array}\right]}^T$. We assume that the incident signals ${\left\{s_i\left(t\right)\right\}}_{i=1}^N$ are independent of each other, the noise vector $\mathbf{n}\left(t\right)$ is amenable to the Gaussian distribution, and it is independent of the signals. Besides, we assume that $\mathbf{A}$ is unambiguous, i.e., the steering vectors ${\left\{\mathbf{a}\left({\theta }_i\right)\right\}}_{i=1}^N$ are linearly independent for any set of distinct ${\left\{{\theta }_i\right\}}_{i=1}^N$.

As described in [27,28,29,30,31,32,33,34,35,36,37,38,43], it may reasonably consider that there are remarkable negative correlation between the coupling strength and the inter-element spacing, and the cross talk effects can be neglected if the distance between two sensors is larger than several times the length of the minimum inter-element spacing $d_I$. By assuming that the minimum inter-element spacing of two sensors without mutual coupling to be $Q{d}_I$, the generalized MCM can be approximated by

$${\mathbf{C}}_{ij}=\left\{\begin{array}{cc}\hfill c_{|p_i-p_j|},& |p_i-p_j|\le Q{d}_I\hfill \\ \hfill 0,& |p_i-p_j|>Q{d}_I\hfill \end{array}\right..$$

(3)

For the case of 2-level nested arrays, if $Q<M_1+1$, the MCM can be specifically expressed as

$$\begin{array}{cc}\hfill \mathbf{C}& =\left[\begin{array}{cccccccc}1& c_1& \cdots & c_{P-1}& & & & \mathbf{0}\\ c_1& 1& c_1& \cdots & \ddots & & & \\ c_2& c_1& 1& c_1& \cdots & c_{P-1}& & \\ & \ddots & c_1& 1& \cdots & c_{P-2}& & \\ & & \ddots & \ddots & \ddots & ⋮& & \\ & & & \ddots & c_1& 1& & \\ & & & & & & \ddots & \\ \mathbf{0}& & & & & & & 1\end{array}\right]\hfill \\ \hfill & =\mathrm{blkdiag}\left\{\mathrm{Toeplitz}\left\{\left[1,c_1,\cdots ,c_{P-1},{\mathbf{0}}_{1\times (M_1+1-P)}\right]\right\},{\mathbf{I}}_{(M_2-1)}\right\}\hfill \end{array}$$

(4)

where $\mathrm{Toeplitz}\left\{\mathbf{r}\right\}$ denotes a symmetric Toeplitz matrix constructed by the vector $\mathbf{r}$, and $0<|c_1|,|c_2|,\cdots ,|c_{P-1}|<c_0=1$ are the mutual coupling coefficients.

2.2. Parameters Setting

To make the signal model (simulations) much easier to follow and replicate, it should be emphasized herein that throughout this communication, the following parameters are variables in the performance analysis and assessment: signal-to-noise ratio (SNR), the number of snapshots L, and the number of Monte Carlo runs; the fixed parameters adopted in the numerical calculus, deductions, and simulations include: the number of total sensors M, inter-element spacing of the inner ULA $d_I$, the number of sensors the inner ULA $M_1$, the number of sensors the outer ULA $M_2$, the number of signals N, DOAs ${\left\{{\theta }_i\right\}}_{i=1}^N$, the coupling range P, and the mutual coupling coefficients ${\left\{c_i\right\}}_{i=1}^{P-1}$.

3. Proposed FOC-Based DOA Estimator without Mutual Coupling Compensation

3.1. FOC Matrix Construction

Considering the received signals are assumed to be non-Gaussian, one can establish the array FOC matrix between the received data blocks as

$$\begin{array}{cc}\hfill \mathbf{B}& =\mathrm{cum}\left\{\mathbf{x}\left(t\right),{\mathbf{x}}^H\left(t\right),{\mathbf{x}}^{*}\left(t\right),{\mathbf{x}}^T\left(t\right)\right\}\hfill \end{array}$$

(5)

The entry of $\mathbf{B}$ in the $\left[\left(k_1-1\right)M+k_2\right]$-th row and the $\left[\left(l_1-1\right)M+l_2\right]$-th column is defined as

$$\begin{array}{cc}\hfill & \mathbf{B}\left(\left(k_1-1\right)M+k_2,\left(l_1-1\right)M+l_2\right)\hfill \\ \hfill & =\mathrm{cum}\{x_{k_1}\left(t\right),x_{l_1}^{*}\left(t\right),x_{k_2}^{*}\left(t\right),x_{l_2}\left(t\right)\}\hfill \\ \hfill & =E\left[x_{k_1}\left(t\right)x_{k_2}^{*}\left(t\right)x_{l_1}^{*}\left(t\right)x_{l_2}\left(t\right)\right]-E\left[x_{k_1}\left(t\right)x_{k_2}^{*}\left(t\right)\right]E\left[x_{l_1}^{*}\left(t\right)x_{l_2}\left(t\right)\right]\hfill \\ \hfill & -E\left[x_{k_1}\left(t\right)x_{l_1}^{*}\left(t\right)\right]E\left[x_{k_2}^{*}\left(t\right)x_{l_2}\left(t\right)\right]-E\left[x_{k_1}\left(t\right)x_{l_2}\left(t\right)\right]E\left[x_{k_2}^{*}\left(t\right)x_{l_1}^{*}\left(t\right)\right]\hfill \end{array}$$

(6)

Substituting (2) into (5), and exploiting a series of properties of FOC in [44], one can further get

$$\begin{array}{cc}\hfill \mathbf{B}& =\mathrm{cum}\left\{\mathbf{C}\mathbf{A}\mathbf{s}\left(t\right),{\left(\mathbf{C}\mathbf{A}\mathbf{s}\left(t\right)\right)}^H,{\left(\mathbf{C}\mathbf{A}\mathbf{s}\left(t\right)\right)}^{*},{\left(\mathbf{C}\mathbf{A}\mathbf{s}\left(t\right)\right)}^T\right\}\hfill \\ \hfill & =\mathrm{cum}\left\{\sum _{p=1}^N\mathbf{C}\mathbf{a}\left({\theta }_p\right)s_p\left(t\right),\sum _{m=1}^N{\left(\mathbf{C}\mathbf{a}\left({\theta }_m\right)\right)}^H{s}_m^{*}\left(t\right),\sum _{q=1}^N\mathbf{C}{\mathbf{a}}^{*}\left({\theta }_q\right)s_q^{*}\left(t\right),\sum _{n=1}^N{\left(\mathbf{C}\mathbf{a}\left({\theta }_n\right)\right)}^T{s}_n\left(t\right)\right\}\hfill \\ \hfill & =\sum _{p=1}^N\sum _{m=1}^N\sum _{q=1}^N\sum _{n=1}^N\left(\mathbf{C}\mathbf{a}\left({\theta }_p\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\theta }_q\right)\right)}^{*}\right){\left(\mathbf{C}\mathbf{a}\left({\theta }_m\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\theta }_m\right)\right)}^{*}\right)}^H\mathrm{cum}\left\{s_p\left(t\right),s_m^{*}\left(t\right),s_q^{*}\left(t\right),s_n\left(t\right)\right\}\hfill \\ \hfill & =\sum _{i=1}^N\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\right)}^{*}\right){\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\right)}^{*}\right)}^H\mathrm{cum}\left\{s_i\left(t\right),s_i^{*}\left(t\right),s_i^{*}\left(t\right),s_i\left(t\right)\right\}\hfill \\ \hfill & =\sum _{i=1}^N{\gamma }_i\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\right)}^{*}\right){\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\theta }_i\right)\right)}^{*}\right)}^H\hfill \\ \hfill & =\left(\mathbf{C}\mathbf{A}\circ {\left(\mathbf{C}\mathbf{A}\right)}^{*}\right){\mathbf{C}}_s{\left(\mathbf{C}\mathbf{A}\circ {\left(\mathbf{C}\mathbf{A}\right)}^{*}\right)}^H\hfill \end{array}$$

(7)

where ${\gamma }_i\triangleq \mathrm{cum}\left\{s_i\left(t\right),s_i^{*}\left(t\right),s_i^{*}\left(t\right),s_i\left(t\right)\right\}$, and ${\mathbf{C}}_s\triangleq \mathrm{diag}\left\{{\gamma }_1,\cdots ,{\gamma }_N\right\}\in {\mathbb{R}}^{N\times N}$.

To resolve the DOA estimates from the FOC built above, some approaches [45,46,47,48] have been proposed. However, these methods are developed for ULAs in the presence of unknown mutual coupling, and cannot be directed applied to the case of nested arrays as the mutual coupling effects of the two array geometries are distinct from each other. By this reason, a new estimator using FOC should be investigated for combating the unknown mutual coupling.

3.2. Robust DOA Estimation Against Unknown Mutual Coupling

By examining the special structure of the MCM of the nested arrays, one finds that the actual steering vector can be factorized in another way by

$$\begin{array}{cc}\hfill \mathbf{C}\mathbf{a}\left(\theta \right)& =\mathbf{T}\left(\theta \right)\mathbf{c}\hfill \end{array}$$

(8)

where $\mathbf{c}=\left[1,c_1,\cdots ,c_{P-1}\right]\in {\mathbb{C}}^P$, and $\mathbf{T}\left(\theta \right)=\left[\begin{array}{c}{\mathbf{T}}_1+{\mathbf{T}}_2\\ {\mathbf{T}}_3\end{array}\right]\in {\mathbb{C}}^{M\times P}$ where ${\mathbf{T}}_1$, ${\mathbf{T}}_2\in {\mathbb{C}}^{\left(M_1+1\right)\times P}$, and ${\mathbf{T}}_3\in {\mathbb{C}}^{\left(M_2-1\right)\times P}$ are given by

$${\left[{\mathbf{T}}_1\right]}_{p,q}=\left\{\begin{array}{cc}\hfill {\left[{\mathbf{a}}_1\right]}_{p+q-1},& p+q\le M_1+2\hfill \\ \hfill 0,& \mathrm{otherwise}\hfill \end{array}\right.$$

(9)

$${\left[{\mathbf{T}}_2\right]}_{p,q}=\left\{\begin{array}{cc}\hfill {\left[{\mathbf{a}}_1\right]}_{p-q+1},& p\ge q\ge 2\hfill \\ \hfill 0,& \mathrm{otherwise}\hfill \end{array}\right.$$

(10)

$$\begin{array}{cc}\hfill {\mathbf{T}}_3& =\left[{\mathbf{a}}_2,{\mathbf{0}}_{\left(M_2-1\right)\times \left(P-1\right)}\right]\hfill \end{array}$$

(11)

with ${\mathbf{a}}_1={\mathbf{F}}_1\mathbf{a}\left(\theta \right)$, ${\mathbf{F}}_1=\left[{\mathbf{I}}_{M_1+1},{\mathbf{0}}_{(M_1+1)\times \left(M_2-1\right)}\right]$, ${\mathbf{a}}_2={\mathbf{F}}_2\mathbf{a}\left(\theta \right)$, and ${\mathbf{F}}_2=\left[{\mathbf{0}}_{(M_2-1)\times \left(M_1+1\right)},{\mathbf{I}}_{M_2-1}\right]$.

Proof of (8).

One can first rewrite the left-hand side of (8) as

$$\begin{array}{cc}\hfill \mathbf{C}\mathbf{a}\left(\theta \right)& =\left[\begin{array}{cc}{\mathbf{C}}_1& \mathbf{0}\\ \mathbf{0}& {\mathbf{I}}_{M_2-1}\end{array}\right]\left[\begin{array}{c}{\mathbf{a}}_1\\ {\mathbf{a}}_2\end{array}\right]=\left[\begin{array}{c}{\mathbf{C}}_1{\mathbf{a}}_1\\ {\mathbf{a}}_2\end{array}\right]\hfill \end{array}$$

(12)

where ${\mathbf{C}}_1=\mathrm{Toeplitz}\left\{\left[1,c_1,\cdots ,c_{P-1},{\mathbf{0}}_{1\times (M_1+1-P)}\right]\right\}\in {\mathbb{C}}^{\left(M_1+1\right)\times \left(M_1+1\right)}$. By Lemma 3 in [25], ${\mathbf{C}}_1{\mathbf{a}}_1=\overline{\mathbf{T}}\left(\theta \right)\mathbf{c}$ where $\overline{\mathbf{T}}\left(\theta \right)={\mathbf{T}}_1+{\mathbf{T}}_2$. Considering the first entry of c is 1, one can make an identical transformation between ${\mathbf{a}}_2$ and c, that is, ${\mathbf{a}}_2=\left[{\mathbf{a}}_2,{\mathbf{0}}_{\left(M_2-1\right)\times \left(P-1\right)}\right]\mathbf{c}={\mathbf{T}}_3\mathbf{c}$. As a result, one has $\mathbf{C}\mathbf{a}\left(\theta \right)=\left[\begin{array}{c}\left({\mathbf{T}}_1+{\mathbf{T}}_2\right)\mathbf{c}\\ {\mathbf{T}}_3\mathbf{c}\end{array}\right]=\left[\begin{array}{c}{\mathbf{T}}_1+{\mathbf{T}}_2\\ {\mathbf{T}}_3\end{array}\right]\mathbf{c}=\mathbf{T}\left(\theta \right)\mathbf{c}$. This completes the proof of (8). □

Now performing the singular value decomposition (SVD) on $\mathbf{B}$, one has

$$\begin{array}{cc}\hfill \mathbf{B}& =\mathbf{U}\mathsf{\Sigma }{\mathbf{V}}^H\hfill \end{array}$$

(13)

where $\mathsf{\Sigma }=\mathrm{diag}\left\{{\lambda }_1,\cdots ,{\lambda }_{M^2}\right\}$ consists of $M^2$ singular values satisfying ${\lambda }_1\ge \cdots \ge {\lambda }_N>{\lambda }_{N+1}=\cdots ={\lambda }_{M^2}=0$. The columns of ${\mathbf{U}}_s\triangleq \mathbf{U}(:,1:N)$ are the singular vectors corresponding to the N largest eigenvalues, while the columns of ${\mathbf{U}}_n\triangleq \mathbf{U}\left(:,N+1:M^2\right)$ are the singular vectors corresponding to the $M^2$ singular values which are all zeros. The signal subspace is spanned by the columns of ${\mathbf{U}}_s$, whereas the noise subspace is spanned by the columns of ${\mathbf{U}}_n$. One can construct the following function

$$\begin{array}{cc}\hfill p\left(\theta ,\mathbf{c}\right)& ={∥{\left(\left(\mathbf{C}\mathbf{a}\left(\theta \right)\right)\otimes {\left(\mathbf{C}\mathbf{a}\left(\theta \right)\right)}^{*}\right)}^H{\mathbf{U}}_n∥}_2^2\hfill \\ \hfill & ={∥{\left(\left(\mathbf{T}\left(\theta \right)\mathbf{c}\right)\otimes {\left(\mathbf{T}\left(\theta \right)\mathbf{c}\right)}^{*}\right)}^H{\mathbf{U}}_n∥}_2^2\hfill \\ \hfill & ={\left(\mathbf{c}\otimes {\mathbf{c}}^{*}\right)}^H{\left(\mathbf{T}\left(\theta \right)\otimes {\mathbf{T}}^{*}\left(\theta \right)\right)}^H{\mathbf{U}}_n{\mathbf{U}}_n^H\left(\mathbf{T}\left(\theta \right)\otimes {\mathbf{T}}^{*}\left(\theta \right)\right)\left(\mathbf{c}\otimes {\mathbf{c}}^{*}\right)\hfill \\ \hfill & ={\mathbf{g}}^H\left(\mathbf{c}\right)\mathbf{M}\left(\theta \right)\mathbf{g}\left(\mathbf{c}\right)\hfill \end{array}$$

(14)

where $\mathbf{g}\left(\mathbf{c}\right)=\mathbf{c}\otimes {\mathbf{c}}^{*}\in {\mathbb{C}}^{P^2}$, and $\mathbf{M}\left(\theta \right)={\left(\mathbf{T}\left(\theta \right)\otimes {\mathbf{T}}^{*}\left(\theta \right)\right)}^H{\mathbf{U}}_n{\mathbf{U}}_n^H\left(\mathbf{T}\left(\theta \right)\otimes {\mathbf{T}}^{*}\left(\theta \right)\right)\in {\mathbb{C}}^{P^2\times P^2}$. Next one can estimate DOAs from the determinant $\mathbf{M}\left(\theta \right)$. It can be found that the size of ${\left(\mathbf{T}\left(\theta \right)\otimes {\mathbf{T}}^{*}\left(\theta \right)\right)}^H{\mathbf{U}}_n$ is $P^2\times (M^2-N)$ and if $P^2\le M^2-N$, the matrix ${\left(\mathbf{T}\left(\theta \right)\otimes {\mathbf{T}}^{*}\left(\theta \right)\right)}^H{\mathbf{U}}_n$ is normally of full row rank and $\mathbf{M}\left(\theta \right)$ is of full rank. However, when $\theta$ is equal to any DOA of interest, i.e., $\theta ={\theta }_i,i=1,\cdots ,N$, Equation (14) is zero according to the orthogonality between the signal and noise subspaces. Since $\mathbf{g}\left(\mathbf{c}\right)\ne \mathbf{0}$, (14) is available only if the matrix $\mathbf{M}\left(\theta \right)$ is rank deficient, i.e., the determinant of $\mathbf{M}\left(\theta \right)$ is zero. Hence, one can determine DOA estimates by searching the minimum of $det\left\{\mathbf{M}\left(\theta \right)\right\}$ that is close to zero as

$$\begin{array}{cc}\hfill \widehat{\theta }& =arg\underset{\theta }{min}det\left\{\mathbf{M}\left(\theta \right)\right\}.\hfill \end{array}$$

(15)

3.3. Mutual Coupling Coefficient Estimation

With the DOA estimates, the orthogonality between the signal and noise subspaces can be exploited to estimate the mutual coupling coefficients. To this end, one can stack a series of equations as

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{\mathbf{U}}_n^H\left(\left(\mathbf{C}\mathbf{a}\left({\widehat{\theta }}_1\right)\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\widehat{\theta }}_1\right)\right)}^{*}\right)\\ ⋮\\ {\mathbf{U}}_n^H\left(\left(\mathbf{C}\mathbf{a}\left({\widehat{\theta }}_N\right)\right)\otimes {\left(\mathbf{C}\mathbf{a}\left({\widehat{\theta }}_N\right)\right)}^{*}\right)\end{array}\right]& =\left[\begin{array}{c}{\mathbf{0}}_{\left(M^2-N\right)\times 1}\\ ⋮\\ {\mathbf{0}}_{\left(M^2-N\right)\times 1}\end{array}\right].\hfill \end{array}$$

(16)

Substituting (8) into (16), one gets

$$\begin{array}{cc}\hfill \underset{\mathbf{H}}{\underbrace{\left[\begin{array}{c}{\mathbf{U}}_n^H\left(\mathbf{T}\left({\widehat{\theta }}_1\right)\otimes {\mathbf{T}}^{*}\left({\widehat{\theta }}_1\right)\right)\\ ⋮\\ {\mathbf{U}}_n^H\left(\mathbf{T}\left({\widehat{\theta }}_N\right)\otimes {\mathbf{T}}^{*}\left({\widehat{\theta }}_N\right)\right)\end{array}\right]}}\left(\mathbf{c}\otimes {\mathbf{c}}^{*}\right)& =\left[\mathbf{h},\overline{\mathbf{H}}\right]\left[\begin{array}{c}1\\ \overline{\mathbf{c}}\end{array}\right]={\mathbf{0}}_{N\left(M^2-N\right)\times 1}.\hfill \end{array}$$

(17)

where $\mathbf{h}$ is the first column of $\mathbf{H}$ while $\overline{\mathbf{H}}$ contains the remaining columns of $\mathbf{H}$, and $\overline{\mathbf{c}}=\left[{\mathbf{0}}_{\left(P^2-1\right)\times 1},{\mathbf{I}}_{P^2-1}\right]\left(\mathbf{c}\otimes {\mathbf{c}}^{*}\right)\in {\mathbb{C}}^{P^2-1}$.

One can carry out the estimation of $\overline{\mathbf{c}}$ in the least squares sense, that is

$$\begin{array}{cc}\hfill \overline{\mathbf{c}}& =-{\overline{\mathbf{H}}}^{+}\mathbf{h}.\hfill \end{array}$$

(18)

Then the mutual coupling coefficients ${\mathbf{c}}_1=\left[c_1,c_2\cdots ,c_{P-1}\right]$ can be extracted from $\overline{\mathbf{c}}$ by

$$\begin{array}{cc}\hfill {\mathbf{c}}_1& ={\overline{\mathbf{F}}}_1{\overline{\mathbf{c}}}^{*}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathbf{c}}_1& ={\overline{\mathbf{F}}}_2\overline{\mathbf{c}}\hfill \end{array}$$

(20)

where ${\overline{\mathbf{F}}}_1=\left[{\mathbf{I}}_{P-1},{\mathbf{0}}_{\left(P-1\right)\times \left(P^2-P\right)}\right]$ and ${\overline{\mathbf{F}}}_2={\left[{\mathbf{e}}_P,{\mathbf{e}}_{2P},\cdots ,{\mathbf{e}}_{P(P-1)}\right]}^T$ with ${\mathbf{e}}_i\in {\mathbb{R}}^{P^2-1}$, $i=P,2P,\cdots ,P(P-1)$, is a column vector with 1 at the i-th entry and 0 elsewhere. Finally, an improved estimate of ${\mathbf{c}}_1$ can be obtained by averaging the above results as

$$\begin{array}{cc}\hfill {\mathbf{c}}_1& =\frac{1}{2}\left({\overline{\mathbf{F}}}_1{\overline{\mathbf{c}}}^{*}+{\overline{\mathbf{F}}}_2\overline{\mathbf{c}}\right).\hfill \end{array}$$

(21)

3.4. Extension to Partly Calibrated Nested Array with Unknown Mutual Coupling

In this section, the nested array is partly calibrated, i.e., the inner ULA suffers from the mutual coupling effect while the outer ULA has unknown gain and phase errors.

In practice, one may encounter an even more challenging scenario where the mutual coupling and gain-phase mismatch coexist in the nested array. However, it is difficult or even not able to get the issue solved as there are too many unknowns to be handled by the limited DOFs. To make DOA estimation feasible, one has to relieve the nuisance imperfections on the array to some extend, by calibrating a portion of sensors well. This class of arrays, referred to as partly calibrated nested arrays, is common in practice as the aperture of the inner ULA is typically much smaller than that of the whole array and, hence, the inner ULA can be assumed to be well calibrated for gain-phase error while the calibration of the outer ULA may be poor due to unknown channel imbalances between sensors that are located far away from each other. Through above analysis, the MCM is rectified as

$$\begin{array}{cc}\hfill \overline{\mathbf{C}}& =\mathrm{blkdiag}\left\{\mathrm{Toeplitz}\left\{\left[1,c_1,\cdots ,c_{P-1},{\mathbf{0}}_{1\times (M_1+1-P)}\right]\right\},{\mathsf{{\rm Y}}}_{(M_2-1)\times (M_2-1)}\right\}\hfill \end{array}$$

(22)

where $\mathsf{{\rm Y}}=\mathrm{diag}\left\{\mathit{\beta }\right\}$ with $\mathit{\beta }={\left[{\mathit{\beta }}_1,{\mathit{\beta }}_2,\cdots ,{\mathit{\beta }}_{M_2-1}\right]}^T\in {\mathbb{C}}^{M_2-1}$ containing the element gains ${\left|{\mathit{\beta }}_i\right|}_{i=1}^{M_2-1}$ and phases ${\left\{\arg \left({\mathit{\beta }}_i\right)\right\}}_{i=1}^{M_2-1}$. Following a similar idea, one can derive

$$\begin{array}{cc}\hfill \overline{\mathbf{C}}\mathbf{a}\left(\theta \right)& =\overline{\mathbf{T}}\left(\theta \right)\mathit{\alpha }\hfill \end{array}$$

(23)

where $\mathit{\alpha }={\left[{\mathbf{c}}^T,{\mathit{\beta }}^T\right]}^T$ and $\overline{\mathbf{T}}\left(\theta \right)=\mathrm{blkdiag}\left\{{\mathbf{T}}_1+{\mathbf{T}}_2,{\overline{\mathbf{T}}}_3\right\}\in {\mathbb{C}}^{M\times \left(P+M_2-1\right)}$ with ${\overline{\mathbf{T}}}_3=\mathrm{diag}\left\{\left[e^{j\frac{2\pi p_{M_1+2}}{\lambda }sin\theta },\cdots ,e^{j\frac{2\pi p_M}{\lambda }sin\theta }\right]\right\}$. To prove (23), one needs to prove $\mathsf{{\rm Y}}{\mathbf{a}}_2={\overline{\mathbf{T}}}_3\mathit{\beta }$ that can be easily verified by the fact that $\mathrm{diag}\left\{\mathbf{b}\right\}\mathbf{d}=\mathrm{diag}\left\{\mathbf{d}\right\}\mathbf{b}$ holds for arbitrary vectors $\mathbf{b}$ and $\mathbf{d}$ with the same size. Then, by the same principle behind (13)–(15), one can determine the DOA estimates by

$$\begin{array}{cc}\hfill \widehat{\theta }& =arg\underset{\theta }{min}det\left\{\overline{\mathbf{M}}\left(\theta \right)\right\},\hfill \end{array}$$

(24)

where $\overline{\mathbf{M}}={\left(\overline{\mathbf{T}}\left(\theta \right)\otimes {\overline{\mathbf{T}}}^{*}\left(\theta \right)\right)}^H{\mathbf{U}}_n{\mathbf{U}}_n^H\left(\overline{\mathbf{T}}\left(\theta \right)\otimes {\overline{\mathbf{T}}}^{*}\left(\theta \right)\right)\in {\mathbb{C}}^{{\left(P+M_2-1\right)}^2\times {\left(P+M_2-1\right)}^2}$.

Following the same procedures (16)–(21) and denoting $\overline{\mathit{\alpha }}=\left[{\mathbf{0}}_{\left({\left(P+M_2-1\right)}^2-1\right)\times 1},{\mathbf{I}}_{{\left(P+M_2-1\right)}^2-1}\right]\left(\mathit{\alpha }\otimes {\mathit{\alpha }}^{*}\right)\in {\mathbb{C}}^{{\left(P+M_2-1\right)}^2-1}$, one can get the estimate of $\overline{\mathit{\alpha }}$ by

$$\begin{array}{cc}\hfill \overline{\mathit{\alpha }}& =-{\tilde{\mathbf{H}}}^{+}\overline{\mathbf{h}}.\hfill \end{array}$$

(25)

where $\overline{\mathbf{h}}$ is the first column of $\breve{\mathbf{H}}$ that is defined as

$$\begin{array}{cc}\hfill \breve{\mathbf{H}}& =\left[\begin{array}{c}{\mathbf{U}}_n^H\left(\overline{\mathbf{T}}\left({\widehat{\theta }}_1\right)\otimes {\overline{\mathbf{T}}}^{*}\left({\widehat{\theta }}_1\right)\right)\\ ⋮\\ {\mathbf{U}}_n^H\left(\overline{\mathbf{T}}\left({\widehat{\theta }}_N\right)\otimes {\overline{\mathbf{T}}}^{*}\left({\widehat{\theta }}_N\right)\right)\end{array}\right],\hfill \end{array}$$

(26)

and $\tilde{\mathbf{H}}$ includes the rest columns of $\breve{\mathbf{H}}$. The final mutual coupling coefficient and gain-phase error estimates are determined by $\overline{\mathit{\alpha }}$ as

$$\begin{array}{cc}\hfill {\mathbf{c}}_1& =\frac{1}{2}\left({\tilde{\mathbf{F}}}_1{\overline{\mathit{\alpha }}}^{*}+{\tilde{\mathbf{F}}}_2\overline{\mathit{\alpha }}\right)\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill \mathit{\beta }& =\frac{1}{2}\left({\tilde{\mathbf{F}}}_3{\overline{\mathit{\alpha }}}^{*}+{\tilde{\mathbf{F}}}_4\overline{\mathit{\alpha }}\right)\hfill \end{array}$$

(28)

where ${\tilde{\mathbf{F}}}_1=\left[{\mathbf{I}}_{P-1},{\mathbf{0}}_{\left(P-1\right)\times \left({\left(P+M_2-1\right)}^2-P\right)}\right]$, ${\overline{\mathbf{F}}}_2={\left[{\overline{\mathbf{e}}}_{P+M_2-1},{\overline{\mathbf{e}}}_{2\left(P+M_2-1\right)},\cdots ,{\overline{\mathbf{e}}}_{(P-1)(P+M_2-1)}\right]}^T$, ${\tilde{\mathbf{F}}}_3=\left[{\mathbf{0}}_{\left(M_2-1\right)\times \left(P-1\right)},{\mathbf{I}}_{M_2-1},{\mathbf{0}}_{\left(M_2-1\right)\times \left({\left(P+M_2-1\right)}^2-P-M_2+1\right)}\right]$, and ${\overline{\mathbf{F}}}_4=\left[{\overline{\mathbf{e}}}_{P\left(P+M_2-1\right)},{\overline{\mathbf{e}}}_{\left(P+1\right)\left(P+M_2-1\right)},\cdots ,\right.{\left.{\overline{\mathbf{e}}}_{(P+M_2-2)(P+M_2-1)}\right]}^T$ with ${\overline{\mathbf{e}}}_i\in {\mathbb{R}}^{{(P+M_2-1)}^2-1}$, $i=P+M_2-1,2(P+M_2-1),\cdots ,(P+M_2-2)(P+M_2-1)$, is a column vector with 1 at the i-th entry and 0 elsewhere.

The major steps of the proposed algorithm are summarised in

Table 1. Summary of the proposed method against the perturbations of nested arrays.

Step 1.	Compute the cumulant matrix B from observations $\mathbf{x}\left(t\right)$ according to (5).
Step 2.	Extract the noise subspace ${\mathbf{U}}_n$ by performing SVD on B.
Step 3.	Construct $\mathbf{T}\left(\theta \right)$ (or $\overline{\mathbf{T}}\left(\theta \right)$) composed by ${\mathbf{T}}_1$ in (9), ${\mathbf{T}}_2$ in (22),
	and ${\mathbf{T}}_3$ in (11) (or ${\overline{\mathbf{T}}}_3$ in (23)).
Step 4.	Determine the DOA estimates ${\left\{{\widehat{\theta }}_i\right\}}_{i=1}^N$ by searching for N
	minima of $det\left\{\mathbf{M}\left(\theta \right)\right\}$ defined in (14) (or $\overline{\mathbf{M}}\left(\theta \right)$ in (24)).
Step 5.	Reconstruct $\mathbf{H}$ with the DOA estimates ${\widehat{\theta }}_i$ in (17) (or $\breve{\mathbf{H}}$ in (26)).
Step 6.	Estimate $\overline{\mathbf{c}}$ in (18) (or $\overline{\mathit{\alpha }}$ in (25)).
Step 7.	Obtain estimates of the mutual coupling coefficients ${\widehat{\mathbf{c}}}_1$ in (21)
	(or ${\widehat{\mathbf{c}}}_1$ in (27) and estimates of gain-phase errors $\widehat{\mathit{\beta }}$ in (28)).

.

Remark 1.

The methods presented in this paper are valid on condition that the MCM is isolated from any angular information of the incident signals and are not applicable to arbitrary geometries of antenna arrays. However, they still work for some commonly used antenna arrays like dipole, monopole, and slot arrays [49]. On the other hand, the MCM could be direction variant for directional antenna arrays. In such a case, the proposed methods preform well as long as the MCM possesses a similar structure to (4) or (22), even if the mutual coupling coefficients ${\mathit{c}}_1$ are direction-dependent. This is because the transformation of product of the nominal steering vector and the direction-independent MCM still holds for the direction-dependent MCM by (8) or (23). To be specific, the array observation vector, of size $M\times 1$, can be expressed as $\mathit{x}\left(t\right)=\left[{\mathit{C}}_1\mathit{a}\left({\theta }_1\right),{\mathit{C}}_2\mathit{a}({\theta }_2,\cdots ,{\mathit{C}}_N\mathit{a}\left({\theta }_N\right))\right]\mathit{s}\left(t\right)+\mathit{n}\left(t\right)$ where ${\mathit{C}}_i$ is reliant upon ${\theta }_i$, different from each other in general. It should be noted that if ${\left\{{\mathit{C}}_i\right\}}_{i=1}^N$ have the same banded symmetric Toeplitz structure as shown in (4), then one still has ${\mathit{C}}_i\mathit{a}\left({\theta }_i\right)=\mathit{T}\left({\theta }_i\right){\mathit{c}}_i$. As a result, one can readily deduce that $det\left\{\mathit{M}\left(\theta \right)\right\}=0$ holds for $\theta ={\theta }_i,i=1,2,\cdots ,N$, and the DOA estimates can be obtained by searching the minimum of $det\left\{\mathit{M}\left(\theta \right)\right\}$. It should be noted that the structure of MCM of directional antenna arrays may not stay the same for all angles in practice, causing an arbitrariness on the composition of $\mathit{M}\left(\theta \right)$ and, hence our solution is inapplicable to arbitrary direction-dependent mutual coupling via searching the spatial spectrum of a unified function $det\left\{\mathit{M}\left(\theta \right)\right\}$. Only the direction-independent model is discussed herein while its direction-dependent counterpart will be investigated in the future.

Remark 2.

As with the previous work [49], the coupling range P in (4) is assumed to be known a priori, otherwise it has to be determined properly. If P is selected to be smaller than the true value, the remaining mutual coupling may still deteriorate the estimation performance, whereas if the chosen P is larger than the true one, the accuracy and resolution of estimates are confined due to the loss of array aperture. A pragmatic approach to identify P is to measure the minimal distance between two antennas where the coupling effect is negligible. In practice, the coupling range depends on various factors of antenna design, such as array geometry, ground structure, material characteristic, etc, and the effect of mutual coupling is weak enough to be neglected in many applications when the inter-element spacing of an antenna array is larger than half a wavelength, which means that P is relatively small.

Remark 3.

It should be noted that one may get some pseudo estimates in addition to the true DOAs, satisfying (15) or (24). Without loss of generality, we assume that there are $N_0\ge N$ DOA estimates, pseudo and true, one obtains by a spectral search of (15) or (24). Substituting the DOA estimates into (21) or (27), on can determine $N_0$ mutual coupling coefficients vectors and reconstruct the corresponding MCM ${\left\{{\widehat{\mathit{C}}}_i\right\}}_{i=1}^{N_0}$. Defining a spatial spectrum equation as follows

$$\begin{array}{cc}\hfill f& ={∥{\left(\left({\widehat{\mathit{C}}}_i\mathit{a}\left({\widehat{\theta }}_i\right)\right)\otimes {\left({\widehat{\mathit{C}}}_i\mathit{a}\left({\widehat{\theta }}_i\right)\right)}^{*}\right)}^H{\mathit{U}}_n∥}_2^2,i=1,2,\cdots ,N_0,\hfill \end{array}$$

(29)

one can sift the estimates of true DOAs and mutual coupling coefficients from the pseudo ones by identifying whether $f=0$ or not as the orthogonality between the signal subspace $\left({\widehat{\mathit{C}}}_i\mathit{a}\left({\widehat{\theta }}_i\right)\right)\otimes {\left({\widehat{\mathit{C}}}_i\mathit{a}\left({\widehat{\theta }}_i\right)\right)}^{*}$ and the noise subspace ${\mathit{U}}_n$ does not hold for the false DOAs and mutual coupling coefficients. In practice, only finite snapshots are available, so the true DOA and mutual coupling coefficient estimates correspond to the minima of (29) that returns large values for the rest $N_0-N$ pseudo estimates. By this means, one can tell the true estimates from the false ones.

Remark 4.

It is known that ${\left(\mathit{T}\left(\theta \right)\otimes {\mathit{T}}^{*}\left(\theta \right)\right)}^H{\mathit{U}}_n\in {\mathbb{C}}^{P^2\times (M^2-N)}$ and ${\left(\overline{\mathit{T}}\left(\theta \right)\otimes {\overline{\mathit{T}}}^{*}\left(\theta \right)\right)}^H{\mathit{U}}_n\in {\mathbb{C}}^{{(P+M_2-1)}^2\times (M^2-N)}$, so the proposed method works on condition that $P^2\le M^2-N$ for the case of unknown mutual coupling only and ${(P+M_2-1)}^2\le M^2-N$ for the scenario where unknown mutual coupling and gain-phase error occur on dense ULAs and sparse ULA parts, respectively. This provides upper bounds on the number of resolvable signals for these two circumstances: one is $N\le M^2-P^2$, and the other is $N\le M^2-{(P+M_2-1)}^2$, which means that the developed algorithm can handle more signals in the former than in the latter, or equivalently, more DOFs and effective aperture are available when mutual coupling takes place only. Therefore, the proposed Method 1 has better performance than Method 2 in that intuitively, the more the unknowns are, the worse the estimation arises, while a reverse phenomenon can be observed since Method 1 does not take the gain-phase error into consideration, causing a considerable model mismatch.

4. Simulation Results and Discussion

In this section, various numerical experiments under different conditions are carried out to examine the performance of the proposed algorithms. Simulations are conducted for 2-level nested array with four elements in each level. For simplicity, we assume that all signals have an identical power. Similar to the settings in [50,51], the signals are modeled as $\mathbf{s}\left(t\right)=\mathbf{F}\left(t\right)\mathbf{r}\left(t\right)$, where $\mathbf{F}\left(t\right)=\mathrm{diag}\left\{f_1\left(t\right),\cdots ,f_K\left(t\right)\right\}$, $\mathbf{r}\left(t\right)={\left[r_1\left(t\right),\cdots ,r_K\left(t\right)\right]}^T$ with $f_i\left(t\right)$ and $r_i\left(t\right)$ being zero-mean Gaussian processes with unit-variance and ${\sigma }_s^2$-variance, respectively. The noise is assumed to be a first order spatial autoregressive process, and the $(a,b)$-th entry of the noise covariance matrix is given by $\mathbf{R}(a,b)={\sigma }_n^2{0.8}^{|\frac{p_a-p_b}{\lambda }|}$ [52,53]. The SNR is defined as $\mathrm{SNR}=10{log}_{10}({\sigma }_s^2/{\sigma }_n^2)$. The accuracy of the DOA estimate, the statistical performance of the algorithms, is measured from 800 Monte Carlo runs in terms of the root mean square error (RMSE) which is defined as

$$\begin{array}{cc}\hfill {\mathrm{RMSE}}_{\theta }& =\sqrt{\frac{1}{800N}\sum _{n=1}^{800}\sum _{i=1}^N{({\widehat{\theta }}_i^{\left(n\right)}-{\theta }_i)}^2}\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill {\mathrm{RMSE}}_c& =\sqrt{\frac{1}{800{∥{\mathbf{c}}_1∥}_2^2}\sum _{n=1}^{800}{∥{\widehat{\mathbf{c}}}_1^{\left(n\right)}-{\mathbf{c}}_1∥}_2^2}\times 100\hfill \end{array}$$

(31)

where ${\widehat{\theta }}_i^{\left(n\right)}$ and ${\widehat{\mathbf{c}}}_1^{\left(n\right)}$ are the estimates of ${\theta }_i$ and ${\mathbf{c}}_1$, respectively, for the n-th trial, and N is the number of signals.

In the first scenario, we consider that two independent sources from $\left[-{17}^{\circ },{43}^{\circ }\right]$ impinge on the nested array. The resultant RMSEs of DOA estimates are illustrated in Figure 1. It can be seen that the proposed Method 1 approaches the FOC-based estimator with known coupling asymptotically and significantly outperforms the Method 2 with the increase of SNR. The main reason is that Method 1 deals with the outer ULA as calibrated and, hence more DOFs are obtained for DOA estimation, whereas Method 2 considers a even worse case, where the outer ULA has gain-phase errors, and the corresponding DOFs are occupied by the nuisance parameters. Besides, the estimation error of Method 2 decreases clearly as the number of snapshots increases and stabilize at approximately ${0.35}^{\circ }$ with the observation size larger than 500 snapshots, while both Method 1 and 4-MUSIC provide much better accuracy, saturating at RMSEs of ${0.05}^{\circ }$ and ${0.03}^{\circ }$, respectively, than Method 1 through all snapshot sizes when SNR is fixed at 5 dB.

Table 2. Root mean square error (RMSE) of ${\mathbf{c}}_1$ versus SNR when $L=500$.

	Method 1	Method 2
SNR	RMSE${}_{\mathit{c}}$	RMSE${}_{\mathit{c}}$
5 dB	8.534%	35.94%
7 dB	5.621%	23.413%
9 dB	3.7062%	13.881%
11 dB	2.5199%	8.2154%
13 dB	1.7612%	4.9845%
15 dB	1.2811%	3.2019%

and

Table 3. RMSE of ${\mathbf{c}}_1$ versus the number of snapshots when $\mathrm{SNR}=5$ dB.

	Method 1	Method 2
The Number	RMSE${}_{\mathit{c}}$	RMSE${}_{\mathit{c}}$
of Snapshots
1000	8.1907%	35.118%
1200	8.0853%	34.504%
1400	8.0204%	34.884%
1600	8.0126%	34.465%
1800	7.9647%	34.305%
2000	7.9368%	34.043%

list the RMSEs of the mutual coupling coefficient estimates under different SNRs and snapshot sizes. Similar to the performance of DOA estimation in the first scenario, the above tabular results show that the proposed Method 1 is able to offer satisfactory estimation of mutual coupling, especially for moderate SNRs and the numbers of snapshots. A plausible explanation is that compared with Method 2, using the calibrated subarray has implicit superiorities, such as a lower Cramer-Rao lower bound for mutual coupling estimation, since its unknowns are less than those with the partly calibrated array. As a result, Method 1 can achieve better performance on mutual coupling calibration.

5. Conclusions

In this paper, an FOC-based DOA estimation method for nested arrays is proposed to simultaneously improve the robustness of direction finding of non-Gaussian signals to array imperfections and mitigate the spatially colored noise. Under the assumptions that the mutual coupling effects are direction-independent and its range is known a priori, the mutual coupling effects of nested arrays is analytically specified from the aspect of the relationship between coupling strength and inter-element spacing, and a new factorization of the distorted steering vector into a matrix, containing bearing information only multiplied by the coupling coefficient vector. Combining this property with RARE technique, a MUSIC-like estimator is developed to obtain the DOA estimates in FOC. Additionally, the proposed estimator can be extended to handle a much harsher issue that mutual coupling and gain-phase mismatches separately occur on dense and sparse sunbarrays, respectively. Analytical specifications show that our solutions are more robust to array imperfections as compared to the standard nested array processing. Compared with the schemes of super nested array families on reducing the mutual coupling effects, our solutions enjoy two merits: (a) It is applicable to the standard nested arrays without rearranging the configuration to increase the inter-element spacing, isolating the cross talk in dense subarrays as well as gain-phase errors in sparse ULA parts; (b) Perturbations in nested arrays are estimated in colored noise, which has not been fully addressed. Simulations results corroborate that the proposed methods are advantageous to self array calibration as well as immunity to colored noise.