Compressed Sensing-Based DOA Estimation with Unknown Mutual Coupling Effect

Abstract

The performance of a direction-finding system is significantly degraded by the imperfection of an array. In this paper, the direction-of-arrival (DOA) estimation problem is investigated in the uniform linear array (ULA) system with the unknown mutual coupling (MC) effect. The system model with MC effect is formulated. Then, by exploiting the signal sparsity in the spatial domain, a compressed-sensing (CS)-based system model is proposed with the MC coefficients, and the problem of DOA estimation is converted into that of a sparse reconstruction. To solve the reconstruction problem efficiently, a novel DOA estimation method, named sparse-based DOA estimation with unknown MC effect (SDMC), is proposed, where both the sparse signal and the MC coefficients are estimated iteratively. Simulation results show that the proposed method can achieve better performance of DOA estimation in the scenario with MC effect than the state-of-the-art methods, and improve the DOA estimation performance about $31.64\%$ by reducing the MC effect by about 4 dB.

1. Introduction

In array systems, the direction-of-arrival (DOA) estimation problems have been widely investigated. A basic DOA estimation method is based on the discrete Fourier transform (DFT) of the received signal in the spatial domain [1,2], but the resolution of such technique is limited, and cannot estimate multiple signals in one beam-width. Usually, if one can distinguish two signals within a beam-width (as defined by half the distance between the nulls of the central maximum), this is defined as super-resolution [3]. To improve the performance of DOA estimation, methods based on the maximum likelihood and the subspace have been proposed, such as the multiple signal classification (MUSIC) method [4,5,6], the Root-MUSIC method [7], and the estimating signal parameters via rotational invariance techniques (ESPRIT) method [8]. To estimate the DOA, the beamspace-based methods have also been proposed [9]. The beamspace-based methods can design the beamspace matrix to achieve a spatial beampattern that is as close as possible to a certain desired one. For example, tensor modeling is adopted for the mono-static MIMO radar in [10] to design the transmit array interpolation and the beamspace jointly. In the colocated multiple-input and multiple-output (MIMO) radar system, Ref. [11] proposed a beamspace design method to estimate the DOA. A direction finding problem is invested in [12], and an energy-focusing method is given. However, the subspace-based methods for DOA estimation only consider the subspaces of signals and noise, and the signal sparsity has not been considered.

The robust beamformer algorithms dealing with possible mismatches have also been proposed. For example, the worst-case performance is optimized in [13] with an arbitrary unknown signal-steering vector mismatch. The extension of the Capon beamformer with uncertain steering vectors and diagonal loading approaches is proposed in [14]. The doubly constrained robust Capon beamformer with ellipsoid uncertainty set is proposed in [15]. The robust filters for radar pulse-Doppler processing is proposed in [16], where the constraints of Doppler filter sidelobes and the uncertainties are considered. The robustness of Capon beamformer against array steering vector errors and noise is improved in [17] by the covariance matrix fitting. A robust beamforming via worst-case signal-to-interference-plus-noise ratio (SINR) maximization with uncertainty model is proposed in [18], where the maximization problem is solved by convex optimization.

The compressed-sensing (CS)-based methods have been proposed to exploit the sparsity of signals in the spatial domain [19,20,21,22,23,24,25,26,27]. Ref. [9] proposes the iterative adaptive approach (IAA), multi-snapshot sparse Bayesian learning (M-SBL) and maximum likelihood-based IAA (IAA-ML) to exploit the sparsity in beamforming design. The Bayesian compressive sensing (BCS) is proposed in [28,29,30] and used to reconstruct the sparse signals developed for the sparse signal reconstruction with the CS measurements. an off-grid sparse Bayesian inference (OGSBI) method is proposed in [31] to improve the sparse-based DOA estimation.

However, in a practical array system, we cannot ignore the mutual coupling (MC) effect between antennas and the performance of DOA estimation is degraded significantly [32,33,34]. The methods considering the unknown MC effect are proposed in [35,36,37]. Traditionally, the MC effects among the antennas can be described by a symmetric Toeplitz matrix [38,39,40]. However, in the present papers, the signal sparsity and the MC effect have not been considered simultaneously.

In this paper, the problem of DOA estimation is investigated in the scenario with an unknown MC effect. By exploiting the signal sparsity in the spatial domain, compressed-sensing (CS)-based system model with the additional MC coefficients is formulated. Then, the problem of DOA estimation is converted into that of sparse reconstruction. A novel reconstruction method named unknown MC effect (SDMC) is proposed and estimating the MC coefficients and the sparse signals iteratively. The DOA estimation performance of the proposed method is compared with the state-of-the-art methods in the uniform linear array (ULA) system. To summarize, we make the contributions as follows:

• The CS-based system model with unknown MC effect: A system model considering both the MC effect and the signal sparsity is proposed and converts the DOA estimation problem into a sparse reconstruction problem.
• The CS-based DOA estimation method with unknown MC coefficients: With the CS-based system model, a novel CS-based method (SDMC) is proposed and estimates both the DOA and the MC coefficient iteratively.
• The theoretical expressions of the gradient descent method: In the proposed SDMC method, the MC coefficients are estimated by the gradient descent method, and the corresponding expressions for the unknown parameters are derived theoretically.

The remainder of this paper is organized as follows. The ULA system model with MC effect is elaborated in Section 2. The proposed method (SDMC) with unknown MC s presented in Section 3. Simulation results are given in Section 4. Finally, Section 5 concludes the paper.

Notations: Matrices are denoted by capital letters in boldface (e.g., $\mathit{A}$), and vectors are denoted by lowercase letters in boldface (e.g., $\mathit{a}$). $\mathcal{CN}\left(a,b\right)$ denotes the complex Gaussian distribution with the mean being a and the variance being b. ${\parallel ·\parallel }_F$, ${\parallel ·\parallel }_1$, ⊗, $Tr\left\{·\right\}$, ${(·)}^{*}$, ${(·)}^{\mathrm{T}}$ and ${(·)}^{\mathrm{H}}$ denote the Frobenius norm, the ${\ell }_1$ norm, the Kronecker product, the trace of a matrix, the conjugate, the matrix transpose and the Hermitian transpose, respectively. ${\mathbb{C}}^{M\times N}$ denotes the set of $M\times N$ matrices with the entries being complex numbers. For a vector $\mathit{a}$, ${\left[\mathit{a}\right]}_n$ denotes the n-th entry of $\mathit{a}$.

2. ULA System for DOA Estimation with MC Effect

2.1. System Model without MC Effect

In this paper, a uniform linear array (ULA) system is adopted to estimate the direction-of-arrival (DOA), and the number of antennas is N with the inter-antenna element spacing being d. As shown in Figure 1, the K unknown signals are received by the ULA system, and we denote the direction of the k-th signal ($k=0,1,\cdots ,K-1$) as ${\theta }_k$. Then, the received signals are sampled with the sampling frequency being $f_{\mathrm{s}}$, and the sampled signal during the $m{T}_{\mathrm{s}}$ sampling time ($T_{\mathrm{s}}=1/f_{\mathrm{s}}$ denotes the sampling interval) at the n-th antenna ($n=0,1,\cdots ,N-1$) can be expressed as

$$\begin{array}{c}\hfill y_{n,m}=\sum _{k=0}^{K-1}{s}_{k,m}{e}^{j2\pi \frac{nd}{\lambda }sin{\theta }_k}+w_{n,m},\end{array}$$

(1)

where $s_{k,m}$ denotes the k-th unknown signal during the $m{T}_{\mathrm{s}}$ sampling time, $\lambda$ denotes the wavelength, and $w_{n,m}\sim \mathcal{CN}\left(0,{\sigma }_{\mathrm{n}}^2\right)$ denotes the additive white Gaussian noise (AWGN) with the variance being ${\sigma }_{\mathrm{n}}^2$ during the $m{T}_{\mathrm{s}}$ sampling time.

To simplify the expression of received signals, with $a_n(\theta )\triangleq e^{j2\pi \frac{nd}{\lambda }sin\theta }$, we can define the following steering vector

$$\begin{array}{c}\hfill \mathit{a}(\theta )\triangleq {\left[\begin{array}{c}1,e^{j2\pi \frac{d}{\lambda }sin\theta },\cdots ,e^{j2\pi \frac{(N-1)d}{\lambda }sin\theta }\end{array}\right]}^{\mathrm{T}},\end{array}$$

(2)

and collect the received signals from all antennas as

$$\begin{array}{c}\hfill {\mathit{y}}_m\triangleq {\left[\begin{array}{c}{y}_{0,m},y_{1,m},\cdots ,y_{N-1,m}\end{array}\right]}^{\mathrm{T}}.\end{array}$$

(3)

Then, we can have the following vector form of received signals during the $m{T}_{\mathrm{s}}$ sampling time

$$\begin{array}{cc}\hfill {\mathit{y}}_m& =\sum _{k=0}^{K-1}{s}_{k,m}\mathit{a}({\theta }_k)+{\mathit{w}}_m\hfill \\ & =\mathit{A}(\mathbf{\theta }){\mathit{s}}_m+{\mathit{w}}_m\hfill \end{array}$$

(4)

where we define $\mathit{A}(\mathbf{\theta })\triangleq \left[\begin{array}{c}\mathit{a}({\theta }_0),\mathit{a}({\theta }_1),\cdots ,{\mathit{a}}_{{\theta }_{K-1}}\end{array}\right]$, $\mathbf{\theta }\triangleq {\left[\begin{array}{c}{\theta }_0,{\theta }_1,\cdots ,{\theta }_{K-1}\end{array}\right]}^{\mathrm{T}}$, the noise vector is defined as ${\mathit{w}}_m\triangleq {\left[\begin{array}{c}{w}_{0,m},w_{1,m},\cdots ,w_{N-1,m}\end{array}\right]}^{\mathrm{T}}$, and the signal vector is ${\mathit{s}}_m\triangleq {\left[\begin{array}{c}{s}_{0,m},s_{1,m},\cdots ,s_{K-1,m}\end{array}\right]}^{\mathrm{T}}$.

When M samples are used to estimate the DOA, the received signals can be expressed as

$$\begin{array}{c}\hfill \mathit{Y}=\mathit{A}(\mathbf{\theta })\mathit{S}+\mathit{W}\end{array}$$

(5)

where $\mathit{Y}\triangleq \left[\begin{array}{c}{\mathit{y}}_0,{\mathit{y}}_1,\cdots ,{\mathit{y}}_{M-1}\end{array}\right]$, $\mathit{S}\triangleq \left[\begin{array}{c}{\mathit{s}}_0,{\mathit{s}}_1,\cdots ,{\mathit{s}}_{M-1}\end{array}\right]$, and $\mathit{W}\triangleq \left[\begin{array}{c}{\mathit{w}}_0,{\mathit{w}}_1,\cdots ,{\mathit{w}}_{M-1}\end{array}\right]$.

2.2. System Model with MC Effect

The system model in (5) has not considered the unknown MC effect between the antennas. When we use the parameter $c_n$ to describe the MC coefficient between the $n_1$-th ($n_1\in \{0,1,\cdots ,N-1\}$) and $n_2$-th ($n_2\in \{0,1,\cdots ,N-1\}$) antennas ($|n_1-n_2|=n$, $n\in \{0,1,\cdots ,N-1\}$), the received signal in (5) can be rewritten as

$$\begin{array}{c}\hfill \mathit{Y}=\mathit{CA}(\mathbf{\theta })\mathit{S}+\mathit{W},\end{array}$$

(6)

where we define $\mathit{c}\triangleq {\left[\begin{array}{c}{c}_0,c_1,\cdots ,c_{N-1}\end{array}\right]}^{\mathrm{T}}$ ($c_0=1$). Usually, the MC effect can be described by a symmetric Toeplitz matrix [32,38,39,40], so $\mathit{C}$ denotes the Toeplitz matrix of MC effect

$$\begin{array}{c}\hfill \mathit{C}\triangleq Toep\left\{\mathit{c}\right\}=\left[\begin{array}{cccc}{c}_0& c_1& \cdots & c_{N-1}\\ c_1& c_0& \cdots & c_{N-2}\\ ⋮& ⋮& \ddots & ⋮\\ c_{N-1}& c_{N-2}& \cdots & c_0\end{array}\right].\end{array}$$

(7)

Finally, we obtain the system model of DOA estimation with unknown MC effect in (6). The DOA vector $\mathbf{\theta }$ will be estimated with the unknown parameters $\mathit{C}$ and signals $\mathit{S}$ in the following sections.

3. Sparse-Based DOA Estimation Method

3.1. Sparse System Model

By exploiting the signal sparsity in the spatial domain, a sparse-based DOA estimation method is proposed to estimate the DOA with unknown parameters $\mathit{C}$ and signals $\mathit{S}$. First, we discretize the spatial angle into Q grids $\mathbf{\zeta }\triangleq {\left[\begin{array}{c}{\zeta }_0,{\zeta }_1,\cdots ,{\zeta }_{Q-1}\end{array}\right]}^{\mathrm{T}}$, and the corresponding dictionary matrix can be formulated as

$$\begin{array}{c}\hfill \mathit{D}\triangleq \left[\begin{array}{c}{\mathit{d}}_0,{\mathit{d}}_1,\cdots ,{\mathit{d}}_{Q-1}\end{array}\right],\end{array}$$

(8)

where ${\mathit{d}}_q\triangleq \mathit{a}({\zeta }_q)$. Therefore, the received signals in (6) can be expressed as

$$\begin{array}{c}\hfill \mathit{Y}=\mathit{CDX}+\mathit{W}(\mathit{Mutual}\mathit{Coupling}\mathit{Matrix}),\end{array}$$

(9)

where $\mathit{X}$ is a sparse matrix. During the DOA estimation, we assume that the DOAs stay the same, so the non-zero positions among the columns of $\mathit{X}$ are the same. Therefore, the columns of $\mathit{X}$ have the same support set. The positions of non-zero entries in $\mathit{X}$ indicate the corresponding DOA.

Then, the DOA estimation problem is converted into the sparse reconstruction problem (9). In the traditionally sparse reconstruction problems, we usually have the following form

$$\begin{array}{c}\hfill \mathit{Y}={\mathsf{\Phi }}^{\prime }{\mathsf{\Psi }}^{\prime }\mathit{X}+\mathit{W}\end{array}$$

(10)

where ${\mathsf{\Phi }}^{\prime }$ and ${\mathsf{\Psi }}^{\prime }$ denote the measurement matrix and the dictionary matrix, respectively. ${\mathsf{\Phi }}^{\prime }$ and ${\mathsf{\Psi }}^{\prime }$ are both known in the traditional CS problems. However, compare (9) and (10), and we can see that these two problems are different with the unknown MC matrix $\mathit{C}$. Therefore, the traditional methods for the sparse reconstruction cannot be adopted directly.

With the matrix operations, the matrix $\mathit{D}$ in (9) can be rewritten as $\mathit{G}\triangleq \left[\begin{array}{c}\mathit{G}({\zeta }_0),\mathit{G}({\zeta }_1),\cdots ,\mathit{G}({\zeta }_{Q-1})\end{array}\right]$, where $\mathit{G}(\zeta )$ is defined as $\mathit{G}(\zeta )\triangleq {\mathit{G}}_0(\zeta )+{\mathit{G}}_1(\zeta )$. ${\mathit{G}}_0(\zeta )$ and ${\mathit{G}}_1(\zeta )$ are respectively defined as

$$\begin{array}{cc}\hfill {\mathit{G}}_1(\zeta )& \triangleq \left[\begin{array}{cccc}{a}_0(\zeta )& a_1(\zeta )& \cdots & a_{N-1}(\zeta )\\ a_1(\zeta )& a_2(\zeta )& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ a_{N-2}(\zeta )& a_{N-1}(\zeta )& \cdots & 0\\ a_{N-1}(\zeta )& 0& \cdots & 0\end{array}\right],\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\mathit{G}}_2(\zeta )& \triangleq \left[\begin{array}{ccccc}0& 0& \cdots & 0& 0\\ 0& a_0(\zeta )& \cdots & 0& 0\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& a_{N-3}(\zeta )& \cdots & a_0(\zeta )& 0\\ 0& a_{N-2}(\zeta )& \cdots & a_1(\zeta )& a_0(\zeta )\end{array}\right].\hfill \end{array}$$

(12)

Therefore, we can use the vector $\mathit{c}$ to describe the mutual coupling effect, and the sparse-based system model in (9) can be also rewritten by the following equation with a vector form of mutual coupling effect

$$\begin{array}{c}\hfill \mathit{Y}=\mathit{G}(\mathit{X}\otimes \mathit{c})+\mathit{W}(\mathit{Mutual}\mathit{Coupling}\mathit{Vector}).\end{array}$$

(13)

Finally, we can formulate two sparse-based system models for DOA estimation in (9) and (13), where the DOA will be estimated with the unknown MC vector $\mathit{c}$ and the unknown received signals $\mathit{X}$. We will propose a novel DOA estimation based on the sparse reconstruction with unknown MC effect.

3.2. DOA Estimation with Unknown MC Effect

In this subsection, the sparse-based DOA estimation method will be proposed in the scenario with unknown MC effect. Usually, the MC effect between antennas is much smaller than 1, i.e., $|c_n|\ll 1$ ($n\ge 1$). Therefore, the initial value of $\mathit{c}$ can be chosen as ${\left[\begin{array}{c}1,{\mathbf{0}}_{1\times N-1}\end{array}\right]}^{\mathrm{T}}$. Then, with the initial value of MC vector, the sparse-based system model (9) can be used to reconstruct the sparse matrix $\mathit{X}$. With the reconstructed $\mathit{X}$, we use the system model (13) to update the MC vector $\mathit{c}$, where the gradient descent-based method is proposed. The details about the proposed DOA estimation method with unknown MC effect is given in Algorithm 1, and the proposed algorithm is named as SDMC (Sparse-based DOA estimation with unknown Mutual Coupling effect). In Algorithm 1, the sparse reconstruction for DOA estimation in the scenario with unknown mutual coupling. The proposed algorithm is based on the simultaneous orthogonal matching pursuit (SOMP) algorithm with multiple measurement vectors (MMVs). The SOMP algorithm has been developed in [41,42,43] to show the performance of sparse reconstruction. Therefore, with estimating the mutual coupling coefficients, the proposed algorithm can achieve better estimation performance than SOMP algorithm.

At Step 14 of Algorithm 1, the gradient descent method is adopted to update the MC vector iteratively. The gradient descent method is proposed in Algorithm 2. We use the following Lemma to derive the gradient descent method.

Algorithm 1 DOA estimation with unknown MC effect (SDMC algorithm).

1: Input: received signal $\mathit{Y}$, the dictionary matrix $\mathit{D}$, the number of signals K, the matrix $\mathit{G}$, and the number of iterations $N_{\mathrm{ite}}$.   2: Initialization: the iteration indices $i_1=1$ and $i_2=1$, support set $\mathbb{S}=\varnothing$, and the MC vector $\widehat{\mathit{c}}={\left[\begin{array}{c}1,{\mathbf{0}}_{1\times N-1}\end{array}\right]}^{\mathrm{T}}$.   3: while$i_1\le N_{\mathrm{ite}}$do   4:  Obtain the MC matrix $\widehat{\mathit{C}}=Toep\left\{\widehat{\mathit{c}}\right\}$.   5:  Set $i_2=1$, the support set $\mathbb{S}=\varnothing$, and the residual matrix ${\mathit{Z}}_{i_2}=\mathit{Y}$.   6:  while $i_2\le K$ do   7:   $\mathcal{I}=arg{max}_q{\parallel {\mathit{Z}}_{i_2}^{\mathrm{H}}\widehat{\mathit{C}}{\mathit{d}}_q\parallel }_1$.   8:   $\mathbb{S}\leftarrow \mathbb{S}\cup \mathcal{I}$.   9:   $\widehat{\mathit{X}}={\mathbf{0}}_{Q\times M}$. 10:   ${\widehat{\mathit{X}}}_{\mathbb{S},:}={(\widehat{\mathit{C}}{\mathit{D}}_{\mathbb{S}})}^{†}\mathit{Y}$ 11:   ${\mathit{Z}}_{i+1}=\mathit{Y}-\widehat{\mathit{C}}{\mathit{D}}_{\mathbb{S}}{\widehat{\mathit{X}}}_{\mathbb{S},:}$ 12:   $i_2\leftarrow i_2+1$. 13:  end while 14:  The gradient descent method is adopted to estimate the MC vector $\widehat{\mathit{c}}$. 15: end while 16: Output: the estimated support set $\mathbb{S}$, and the esitmated sparse matrix $\widehat{\mathit{X}}$.

Algorithm 2 MC effect estimation.

1: Input: received signal $\mathit{Y}$, the dictionary matrix $\mathit{D}$, the first order derivative of dictionary matrix $\mathsf{\Xi }$, the off-grid vector $\nu$, iteration number $N_{\mathrm{ite}}$.   2: Initialization:$i=1$, MC vector $\widehat{\mathbb{C}}=[1,{\mathbf{0}}_{1\times (N-1)}^{\mathrm{T}}]$, step $\iota =0.01\delta$.   3: while$i\le N_{\mathrm{ite}}$do   4:  $e_i=g(\widehat{\mathit{c}})$.   5:  if $i>=2$ and $e_i>e_{i-1}$ then   6:   $\iota \leftarrow \frac{\iota }{2}$.   7:  end if   8:  Obtain $\frac{\partial g(\mathit{c})}{\partial \mathit{c}}$ from (18).   9:  $\widehat{\mathit{c}}\leftarrow \widehat{\mathit{c}}-\iota \frac{\partial g(\mathit{c})}{\partial \mathit{c}}$ 10:  $i\leftarrow i+1$. 11: end while 12: Output: the estimated MC vector $\widehat{\mathit{c}}$.

Lemma 1.

The complex vectors $\mathit{u}\in {\mathbb{C}}^{P\times 1}$ and $\mathit{v}\in {\mathbb{C}}^{P\times 1}$ are the functions of complex vector $\mathit{x}\in {\mathbb{C}}^{N\times 1}$, and the derivation of ${\mathit{u}}^{\mathrm{H}}\mathit{v}$ can be expressed as

$$\begin{array}{c}\frac{\partial {\mathit{u}}^H\mathit{v}}{\partial \mathit{x}}={\mathit{v}}^T\frac{\partial ({\mathit{u}}^{*})}{\partial \mathit{x}}+{\mathit{u}}^H\frac{\partial \mathit{v}}{\partial \mathit{x}}.\hfill \end{array}$$

(14)

Additionally, when $\mathit{A}\in {\mathbb{C}}^{M\times P}$ is also a function of $\mathit{x}$, the derivation of $\mathit{Au}$ is

$$\begin{array}{c}\frac{\partial \mathit{Au}}{\partial \mathit{x}}=\left[\begin{array}{c}\frac{\partial \mathit{A}}{\partial x_0}\mathit{u}+\mathit{A}\frac{\partial \mathit{u}}{\partial x_0},\dots ,\frac{\partial \mathit{A}}{\partial x_n}\mathit{u}+\mathit{A}\frac{\partial \mathit{u}}{\partial x_n},\dots \end{array}\right].\hfill \end{array}$$

(15)

Proof. 

See Appendix A. □

With the MC vector $\mathit{c}$, we define the objective function of gradient descent method from (13) as

$$\begin{array}{c}\hfill g(\mathit{c})={∥\mathit{Y}-\mathit{G}(\mathit{X}\otimes \mathit{c})∥}_F^2,\end{array}$$

(16)

where $g(\mathit{c})$ is a real function. Then, the derivation of objective function subject to the MC vector $\mathit{c}$ can be obtained as

$$\begin{array}{cc}\hfill \frac{\partial g(\mathit{c})}{\partial {\mathit{c}}^{*}}& =\frac{\partial Tr\left\{\left[\mathit{Y}-\mathit{G}(\mathit{X}\otimes \mathit{c})\right]{\left[\mathit{Y}-\mathit{G}(\mathit{X}\otimes \mathit{c})\right]}^{\mathrm{H}}\right\}}{\partial {\mathit{c}}^{*}},\hfill \end{array}$$

(17)

where $\frac{\partial g(\mathit{c})}{\partial {\mathit{c}}^{*}}$ can be written as a vector. Therefore, the n-th entry of $\frac{\partial g(\mathit{c})}{\partial {\mathit{c}}^{*}}$ is

$$\begin{array}{c}{\left[\frac{\partial g(\mathit{c})}{\partial {\mathit{c}}^{*}}\right]}_n=\frac{\partial g(\mathit{c})}{\partial c_n^{*}}\hfill \\ =Tr\left\{\frac{\partial \left[\mathit{Y}-\mathit{G}(\mathit{X}\otimes \mathit{c})\right]{\left[\mathit{Y}-\mathit{G}(\mathit{X}\otimes \mathit{c})\right]}^{\mathrm{H}}}{\partial c_n^{*}}\right\}\hfill \\ =Tr\left\{-\mathit{Y}\frac{\partial {(\mathit{X}\otimes \mathit{c})}^{\mathrm{H}}}{\partial c_n^{*}}{\mathit{G}}^{\mathrm{H}}+\mathit{G}(\mathit{X}\otimes \mathit{c})\frac{\partial {(\mathit{X}\otimes \mathit{c})}^{\mathrm{H}}}{\partial c_n^{*}}{\mathit{G}}^{\mathrm{H}}\right\}\hfill \\ =Tr\left\{-\mathit{Y}{\left(\mathit{X}\otimes {\mathit{e}}_N^n\right)}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}+\mathit{G}(\mathit{X}\otimes \mathit{c}){\left(\mathit{X}\otimes {\mathit{e}}_N^n\right)}^{\mathrm{H}}{\mathit{G}}^{\mathrm{H}}\right\}\hfill \\ =Tr\left\{{\mathit{G}}^{\mathrm{H}}\left[\mathit{G}(\mathit{X}\otimes \mathit{c})-\mathit{Y}\right]{\left(\mathit{X}\otimes {\mathit{e}}_N^n\right)}^{\mathrm{H}}\right\}.\hfill \end{array}$$

(18)

Therefore, we can obtain $\frac{\partial g(\mathit{c})}{\partial {\mathit{c}}^{*}}$, and realize the gradient descent method. We give an example to estimate the MC coefficient. In the scenario with 4 antennas and one signal, the MC vector is $\mathit{c}=\left[\begin{array}{c}1,-0.29074+0.090316i,0.074697+0.067692i,0.014648-0.028532i\end{array}\right]$. Figure 2 shows the mean square error (MSE) in estimating the MC coefficients with the proposed iterative algorithm. After the 20 iterations, the estimated MC is estimated as $\widehat{\mathit{c}}=\left[\begin{array}{c}1,-0.22928+0.097898i,0.00077922+0.016609i,0.0078645-0.0023455i\end{array}\right]$, and the corresponding MSE is $0.0126$. Therefore, the proposed method can be used to estimate MC coefficients, and improve the DOA estimation performance.

4. Simulation Results

In this section, the simulation results about the DOA estimation in the uniform linear array (ULA) system with unknown MC effect are given, and the simulation parameters are given in

Table 1. Simulation Parameters.

Parameter	Value
The signal-to-noise ratio (SNR) of received signal	20 dB
The number of samples M	100
The number of antennas N	25
The number of antennas with MC effect N	8
The number of signals K	3
The space between antennas d	$0.5$ wavelength
The grid space	$0.05$°
The direction range	$\left[-20°,20°\right]$
The minimum DOA space between signals	10°

. All experiments are carried out in Matlab R2017b on a PC with a 2.9 GHz Intel Core i5 and 8 GB of RAM. The Matlab codes of SDMC algorithm have been available online at https://drive.google.com/drive/folders/1SaWqd6TaHVCPxsD9gjXHgRjR2Pq7wHYH?usp=sharing. The DOA estimation performance of the proposed method is compared the following methods

• MUSIC-like method proposed in [40] is a MUSIC-based method considering the MC effect.
• OGSBI method proposed in [31] is the method for DOA estimation based on the sparse Bayesian inference.
• SOMP method proposed in [44] is a sparse reconstruction method and can be used in the DOA estimation with multiple measurements.

For three signals, we show the histogram bar chart of DOA estimation results in Figure 3, where the ground-truth DOAs of three signals are $-15.35$°, $-1.9$° and $12.95$°, respectively. 5000 Monte Carlo experiments are carried out to show the estimated DOAs. The SNR of the received signal is 10 dB, and the MC effect between adjacent antennas is $-10$ dB. In the simulations, the MC coefficient is generated by the following expression

$$\begin{array}{c}\hfill c_n=\left\{\begin{array}{cc}1,\hfill & n=0\hfill \\ (1+{\alpha }_{\mathrm{c}}){10}^{\frac{n{\beta }_{\mathrm{c}}}{20}}{e}^{j{\varphi }_{\mathrm{c}}},\hfill & n=1,2,\cdots \hfill \end{array}\right.\end{array}$$

(19)

where ${\alpha }_{\mathrm{c}}$ and ${\varphi }_{\mathrm{c}}$ follow the uniform distribution in $[-0.05,0.05]$ and $[0,2\pi ]$, respectively. ${\beta }_{\mathrm{c}}$ is the MC effect in dB between adjacent antennas. As shown in

Table 2. DOA estimation results.

Ground-Truth DOA	SOMP Method			Proposed Method
	Mean	Variance	Successful Ratio	Mean	Variance	Successful Ratio
$-15.35$°	$-15.362$°	$7.2464\times {10}^{-3}$	$\mathbf{87.74}\%$	$-15.363$°	$2.3801\times {10}^{-3}$	$\mathbf{98.76}\%$
$-1.9$°	$-1.906$°	$3.0415\times {10}^{-3}$	$\mathbf{97.62}\%$	$-1.9178$°	$3.2941\times {10}^{-3}$	$\mathbf{96.44}\%$
$12.95$°	$12.938$°	$8.3775\times {10}^{-3}$	$\mathbf{83.68}\%$	$12.942$°	$2.0618\times {10}^{-3}$	$\mathbf{99.36}\%$

, the proposed method (SDMC) can achieve much better performance of DOA estimation than the traditional method based on SOMP. The successful ratio is defined as the probability of estimation error that is less than $0.1$°, i.e., $p(|\widehat{\theta }-\theta |\le 0.1°)$, where $\theta$ denotes the ground-truth DOA and $\widehat{\theta }$ is the estimated DOA. Therefore, by estimating the MC coefficients, the better DOA estimation performance can be improved using the proposed method (SDMC).

For different MC effect, Figure 4 shows the corresponding performance of DOA estimation using the proposed method and the SOMP method. The DOA estimation performance is measured by the root-mean-square error (RMSE)

$$\begin{array}{c}\hfill \mathrm{RMSE}\triangleq \sqrt{\frac{{\sum }_{i=0}^{N_{\mathrm{i}}-1}{\parallel {\widehat{\mathbf{\theta }}}_i-{\mathbf{\theta }}_i\parallel }_2^2}{K{N}_{\mathrm{i}}}},\end{array}$$

(20)

where with the i-th experiment, ${\mathbf{\theta }}_i$ is vector with the ground-truth DOA and ${\widehat{\mathbf{\theta }}}_i$ is that with estimated DOA, and $N_{\mathrm{i}}$ denotes the number of Monte Carlo experiments. As shown in this figure, the better estimation performance can be achieved by the proposed method, and the effect of MC is reduced by about 4 dB using the additional estimation of MC coefficients. Moreover, the proposed method is more efficient when the MC effect is from $-8$ dB to $-12$ dB.

With the MC effect being $-10$ dB, the DOA estimation performance with different SNRs is given in Figure 5. The curves “with perfect MC” are the DOA estimation performance with the perfect MC information and are the best performance that can be achieved using the corresponding methods. When the MC coefficients are unknown, and SNR is 20 dB, the RMSE of OGSBI method is $0.760$°, that of the MUSIC-like method is $0.0629$°, that of SOMP method is $0.0761$°, and that of the proposed method is $0.043$°. Therefore, compared with MUSIC-like and SOMP methods, the DOA estimation performance is improved about $31.64\%$ and $43.50\%$ by the proposed method. Moreover, with performance MC information, the RMSE of SOMP method is $0.0352$°, so the gap between the proposed method and the one with perfect MC information is about $18.13\%$.

We show the DOA estimation performance with different numbers of antennas in Figure 6. As shown in this figure, by increasing the number of antennas, the DOA estimation performance is improved. When we have more than 20 antennas, the proposed method achieves the best performance of DOA estimation. Since the more antennas can have better resolution in DOA estimation, the proposed method is more efficient in the scenario with better resolution. Additionally, with more antennas, the performance improvement by the proposed method is also more significant.

In Figure 7, the DOA estimation performance with different numbers of samples is shown, and the number of samples is from 10 to 100. As shown in this figure, increasing the number of samples can improve the DOA estimation performance. For the proposed method, the estimation error is decreased from $0.0766$° to $0.0435$° ($43.21\%$) with increasing the number of samples from 10 to 100. When the number of samples is greater than 20, the proposed method can achieve the best estimation performance than other methods including the OGSBI method, the MUSIC-like method and the SOMP method. Therefore, the proposed method is efficient in the scenario with an unknown mutual coupling effect.

5. Conclusions

In this paper, the DOA estimation problem in the ULA system has been investigated with an unknown MC effect. The system model with MC effect has been formulated. To exploit the signal sparsity in the spatial domain, a CS-based method has been proposed to convert the problem of DOA estimation into that of sparse reconstruction. Then, a novel method named SDMC is proposed to estimate DOA by updating the sparse signals and the MC coefficients iteratively. Simulation results show that the proposed method can improve the performance of DOA estimation significantly, outperform the state-of-the-art methods, and reduce the MC effect about 4 dB. Future work will probably focus on the DOA estimation problem in the scenario with moving platforms.