Deep Unfolding Sparse Bayesian Learning Network for Off-Grid DOA Estimation with Nested Array

Abstract

Recently, deep unfolding networks have been widely used in direction of arrival (DOA) estimation because of their improved estimation accuracy and reduced computational cost. However, few have considered the existence of a nested array (NA) with off-grid DOA estimation. In this study, we present a deep sparse Bayesian learning (DSBL) network to solve this problem. We first establish the signal model for off-grid DOA with NA. Then, we transform the array output into a real domain for neural networks. Finally, we construct and train the DSBL network to determine the on-grid spatial spectrum and off-grid value, where the loss function is calculated using reconstruction error and the sparsity of network output, and the layers correspond to the steps of the sparse Bayesian learning algorithm. We demonstrate that the DSBL network can achieve better generalization ability without training labels and large-scale training data. The simulation results validate the effectiveness of the DSBL network when compared with those of existing methods.

1. Introduction

The direction of arrival (DOA) estimation for UAV emitters has been an important application in the field of array signal processing [1,2,3]. A non-uniform array is an array structure with non-uniform spacing between elements [4,5]. In the case of the same number of elements, the non-uniform array has a larger array aperture than the uniform array, which can improve the resolution of parameter estimation [6,7,8]. In addition, when the array aperture is the same, a non-uniform array requires fewer elements, which can reduce the hardware cost of the signal processing system and suppress the impact of mutual coupling between elements [9].

The sparse representation method divides the spatial domain into discrete grids, and grid mismatch (GM) occurs when the DOA does not fall on the grid [10], which can reduce the estimation performance of signal parameters. In addition, if the estimation accuracy is improved by reducing the spacing of the grid, the dimensionality of the overcomplete dictionary will lead to an increase in computational complexity in the process of sparse reconstruction. According to sparse reconstruction conditions such as the mutual incoherence property (MIP) and the restricted isometric property (RIP) [11,12], the high correlation between different columns in the overcomplete dictionary with small grid spacing will lead to the failure of sparse reconstruction algorithms. In response to grid mismatch, quantization errors are introduced into the signal model, which does not strictly limit the signal to fall on the grid [13,14,15,16,17]. Yang et al., proposed a mathematical model using basis pursuit denoising (BPDN) to jointly solve the nearest grid and corresponding quantization errors [18]. Compared to the sparse global least squares method [19], the regularization parameters in this method can be set through off-grid mathematical models and noise. In addition, using the off-grid mathematical model of the first-order Taylor expansion, Yang et al., proposed off-grid sparse Bayesian inference (OGSBI) for off-grid DOA estimation [19], which is suitable for both single and multiple snapshot situations and can reduce computational complexity through singular value decomposition. Moreover, Jagannath et al., analyzed the performance of the off-grid mathematical model and quantization error estimation with the first-order Taylor expansion [20]. Yang et al., developed iterative algorithm with the off-grid model from a Bayesian perspective [21]. Tan proposed a joint sparse recovery method to solve the problem of overcomplete dictionary mismatching, which can improve the accuracy of off-grid parameter estimation [22]. In addition, Wu et al., utilized the perturbation covariance matrix to improve the convergence of sparse Bayesian learning methods for off-grid parameter estimation [23].

The above-mentioned methods can be summarized as model-driven methods [24]. In recent years, deep learning has been gradually applied to DOA estimation [25,26,27]. However, deep neural networks and convolutional neural networks belong to the black box, and their generalization ability for untrained data is relatively poor. In addition, overfitting may occur during the training process. Noticeably, deep unfolding networks (DUNs) construct the iterative process of the sparse reconstruction method into the hidden layers of networks [28]. Since the hidden layer corresponds to an iterative process of the sparse reconstruction method, a DUN requires fewer layers for convergence than sparse reconstruction methods, which can accelerate DOA estimation. Compared to the traditional deep neural networks and convolutional neural networks, the parameters of the hidden layer in the DUNs have certain mathematical meanings, which correspond to the calculation process of iterative solutions [29,30]. During the training process, the deep unfolding network can learn the regular pattern of data and have generalization ability for untrained samples [31].

Accordingly, the contribution of this work is to construct a deep unfolding network for off-grid DOA estimation with a nested array, which can reduce computational complexity and improve the estimation accuracy. Utilizing the quantization error, we establish a mathematical model for off-grid DOA estimation with the first-order derivative of an overcomplete dictionary. In order to reduce the computational complexity, we transform the complex domain covariance vector of nested arrays into a real domain covariance vector. Considering the dual advantages of model-driven and data-driven methods in deep unfolding networks, we transform the iterative steps of the SBL algorithm into a cascaded form of neural networks and construct a deep SBL network. By alternating the on-grid spatial spectrum and the off-grid quantization error, off-grid angle estimation is achieved. The experimental results show that the computational complexity of the DSBL network is lower than that of the model-driven SBL algorithm. Moreover, the proposed DSBL network can improve the estimation accuracy under a low signal-to-noise ratio.

Notations: Throughout this paper, the italic letters (e.g., a), lowercase boldface letters (e.g., a), and the capital boldface letters (e.g., A) denote variables, vectors, and matrices, respectively. $\parallel ·{\parallel }_{\hspace{1em}1}$ and $\parallel ·{\parallel }_2$ denote ${\mathcal{l}}_1$ norm and ${\mathcal{l}}_2$ norm, respectively. $\otimes$ and $\odot$ denote Kronecker and Khatri–Rao products, respectively. $E(·)$, $\mathrm{vec}(·)$, and $\mathrm{diag}(·)$ denote mathematical expectation, vectorization operator, and diagonal operator, respectively. ${(·)}^{*}$, ${(·)}^{\mathrm{T}}$, and ${(·)}^{\mathrm{H}}$ denote complex conjugate, transpose, and Hermitian transpose, respectively. $\Re (·)$ and $\Im (·)$ denote the real part and imaginary part of a complex value, respectively.

2. Signal Model for Off-Grid DOA with NA

In practice, the geometry of a nested array (NA) contains M elements, where the internal spacing of the first subarray is d, which is located at the following position:

$$\left\{{\xi }_m\left|{\xi }_m=md,m=1,2,\mathrm{\dots },M/2\right.\right\},$$

(1)

and the internal spacing of the second subarray is (M/2 + 1)d, which is located at the following position:

$$\left\{{\xi }_m\left|{\xi }_m=(m-M/2)(M/2+1)d,m=M/2+1,M/2+2,\mathrm{\dots },M\right.\right\}.$$

(2)

Considering that the configuration of NAs is impinged by K narrowband signals from different DOAs, the array output at the nth snapshot can be expressed as follows:

$$\begin{array}{cc}\hfill \mathit{x}(n)& ={\left[x_1(n)x_1(n)\cdots x_M(n)\right]}^{\mathrm{T}}\hfill \\ \hfill & ={\displaystyle \sum _{k=1}^K\mathit{a}({\theta }_k)s_k(n)}+\mathit{w}(n)\hfill \\ \hfill & =\mathit{A}\mathit{s}(n)+\mathit{w}(n)\hspace{1em},\hfill \end{array}$$

(3)

where $s(n)=[s_1(n)\hspace{1em}{s}_2(n)\cdots s_K(n){]}^{\mathrm{T}}$ denotes the vector of K sources at the nth snapshot, A = $[a({\theta }_1)\hspace{1em}a({\theta }_2)\hspace{1em}\cdots a({\theta }_K)]$ stands for the steering matrix, $a({\theta }_k)={[a_1({\theta }_k)\hspace{1em}{a}_2({\theta }_k)\cdots a_M({\theta }_k)]}^{\mathrm{T}}$ with $a_M({\theta }_k)=\exp (-\mathrm{j}(2\pi {\xi }_m\sin {\theta }_k/\lambda \left)\right)$ stands for the steering vector of the kth signals, $\lambda$ denotes the wavelength of signals, and w(n) = ${[w_1(n)\hspace{1em}{w}_1(n)\cdots w_M(n)]}^{\mathrm{T}}$ denotes the vector of Gaussian white noise at the nth snapshot.

We consider the covariance matrix of array output for NAs, which are calculated as follows:

$$\begin{array}{cc}\hfill \mathit{R}& =E\left(x(n)x^{\mathrm{H}}(n)\right)=\mathit{A}\mathrm{diag}\left({\left[\begin{array}{cccc}{\sigma }_1^2& {\sigma }_2^2& \cdots & {\sigma }_K^2\end{array}\right]}^{\mathrm{T}}\right){\mathit{A}}^{\mathrm{H}}+{\sigma }_{\mathrm{w}}^2{I}_M\hfill \\ \hfill & \approx \frac{1}{N}{\displaystyle \sum _{n=1}^{N}x(n)x^{\mathrm{H}}(n)}\hspace{1em},\hfill \end{array}$$

(4)

where ${\sigma }_k^2$ denotes the power of the kth signal and I_M denotes the M × M dimensional identity matrix.

Therefore, the vector form of the covariance matrix can be expressed as follows:

$$\begin{array}{cc}\hfill \mathit{y}& =\mathrm{vec}\left(\mathit{R}\right)\hfill \\ \hfill & =\left({\mathit{A}}^{\ast }\odot \mathit{A}\right){[{\sigma }_1^2\hspace{1em}{\sigma }_2^2\cdots {\sigma }_K^2]}^{\mathrm{T}}+{\sigma }_{\mathrm{W}}^2{[{\mathit{\eta }}_1^{\mathrm{T}}\hspace{1em}{\mathit{\eta }}_2^{\mathrm{T}}\cdots {\mathit{\eta }}_M^{\mathrm{T}}]}^{\mathrm{T}}\hspace{1em},\hfill \end{array}$$

(5)

where ${\mathit{\eta }}_m^{\mathrm{T}}$ denotes the M dimensional vector, the mth element of ${\mathit{\eta }}_m^{\mathrm{T}}$ is 1, and the remaining elements are 0; the equivalent steering matrix ${\mathit{A}}^{\ast }\odot \mathit{A}$ can be calculated as follows:

$${\mathit{A}}^{\ast }\odot \mathit{A}=[{\mathit{a}}^{*}({\theta }_1)\otimes \mathit{a}({\theta }_1){\mathit{a}}^{*}({\theta }_2)\otimes \mathit{a}({\theta }_2)\cdots {\mathit{a}}^{*}({\theta }_K)\otimes \mathit{a}({\theta }_K)] ,$$

(6)

The scenario of off-grid DOA estimation is shown in Figure 1. In the off-grid case, using the first-order derivative of the overcomplete dictionary on the grid, the sparse representation of the covariance vector in (5) can be constructed as follows:

$$\mathit{y}=(\mathit{\Phi }+\mathit{F}\mathit{B})\mathit{z}+{\sigma }_{\mathrm{w}}^2{[{\mathit{\eta }}_1^{\mathrm{T}}\hspace{1em}{\mathit{\eta }}_2^{\mathrm{T}}\cdots {\mathit{\eta }}_M^{\mathrm{T}}]}^{\mathrm{T}},$$

(7)

where ${\sigma }_{\mathrm{w}}^2$ denotes the noise power, $\mathit{z}={[z_1\hspace{1em}{z}_2\hspace{1em}\cdots \hspace{1em}{z}_Q]}^{\mathrm{T}}$ denotes the spatial spectrum on the grid, and the values of $\{{\theta }_{q_1},{\theta }_{q_2},\dots ,{\theta }_{q_K}\}$ in the spatial spectrum are $\{{\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_K^2\}$. In addition, $\mathit{\Phi }+\mathit{F}\mathit{B}$ denotes the off-grid overcomplete dictionary, and $\mathit{\Phi }$ denotes the on-grid overcomplete dictionary, which can be expressed as follows:

$$\begin{array}{cc}\hfill \mathit{\Phi }& =[\mathit{\phi }({\theta }_1)\hspace{1em}\mathit{\phi }({\theta }_2)\cdots \mathit{\phi }({\theta }_Q)]\hfill \\ \hfill & =[{\mathit{a}}^{*}({\theta }_1)\otimes \mathit{a}({\theta }_1) {\mathit{a}}^{*}({\theta }_2)\otimes \mathit{a}({\theta }_2)\cdots {\mathit{a}}^{*}({\theta }_Q)\otimes \mathit{a}({\theta }_Q)]\hfill \end{array}$$

(8)

where $\mathit{a}({\theta }_q)={[a_1({\theta }_q)\hspace{1em}{a}_2({\theta }_q)\cdots a_M({\theta }_q)]}^{\mathrm{T}}$ denotes the steering vector corresponding to the qth angle in the angle set $\{{\theta }_1,{\theta }_2,\dots ,{\theta }_Q\}$ and F denotes the first derivative of the overcomplete dictionary on the grid, which can be expressed as follows:

$$\begin{array}{cc}\hfill \mathit{F}& =\left[\mathit{f}({\theta }_1)\hspace{1em}\mathit{f}({\theta }_2)\cdots \mathit{f}({\theta }_Q)\right]\hfill \\ \hfill & =\left[{\mathit{\phi }}^{\mathit{\prime }}({\theta }_1)\hspace{1em}{\mathit{\phi }}^{\mathit{\prime }}({\theta }_2)\cdots {\mathit{\phi }}^{\mathit{\prime }}({\theta }_Q)\right]\hfill \\ \hfill & =\left[{\left({\mathit{a}}^{*}({\theta }_1)\otimes \mathit{a}({\theta }_1)\right)}^{\prime }\hspace{1em}{\left({\mathit{a}}^{*}({\theta }_2)\otimes \mathit{a}({\theta }_2)\right)}^{\prime }\cdots {\left({\mathit{a}}^{*}({\theta }_Q)\otimes \mathit{a}({\theta }_Q)\right)}^{\prime }\right]\hfill \end{array}$$

(9)

In addition, B denotes the quantization error matrix, which can be expressed as the following:

$$\mathit{B}=\mathrm{diag}(\mathit{b})$$

(10)

where $\mathit{b}={[b_1\hspace{1em}{b}_2\cdots b_Q]}^{\mathrm{T}}$ represents the quantization error vector, the values of $\{{\theta }_{q_1},{\theta }_{q_2},\dots ,{\theta }_{q_K}\}$ in the quantization error vector are $\{{\Delta }_1,{\Delta }_2,\cdots ,{\Delta }_K\}$, and the values of the remaining elements in the quantization error vector are 0; ${\Delta }_{q_k}={\theta }_k-{\theta }_{q_k}$ denotes the quantization error between the DOA of the kth signal and the closest angle on the grid.

3. Proposed Algorithm

In this section, the DSBL network is used to determine the on-grid spatial spectrum and off-grid quantization error via NA, where the layers of the DSBL network correspond to the steps of the model-driven SBL method. Since the neural networks are suitable for dealing with the real-valued data, we transformed the array output into the real domain in advance.

3.1. Transformation of the Array Output to the Real Domain

Considering that the neural networks are suitable for dealing with real-valued data, we rewrote Equation (7) to the following form:

$$\left[\begin{array}{c}\Re \left(y\right)\\ \Im \left(y\right)\end{array}\right]=\left[\begin{array}{cc}\Re (\mathit{\Phi }+\mathit{F}\mathit{B})& -\Im (\mathit{\Phi }+\mathit{F}\mathit{B})\\ \Im (\mathit{\Phi }+\mathit{F}\mathit{B})& \Re (\mathit{\Phi }+\mathit{F}\mathit{B})\end{array}\right]\left[\begin{array}{c}\Re \left(z\right)\\ \Im \left(z\right)\end{array}\right]+{\left[\begin{array}{c}\Re \left({\sigma }_{\mathrm{w}}^2{[\begin{array}{cccc}{\eta }_1^{\mathrm{T}}& {\eta }_2^{\mathrm{T}}& \cdots & {\eta }_M^{\mathrm{T}}\end{array}]}^{\mathrm{T}}\right)\\ \Im \left({\sigma }_{\mathrm{w}}^2{[\begin{array}{cccc}{\eta }_1^{\mathrm{T}}& {\eta }_2^{\mathrm{T}}& \cdots & {\eta }_M^{\mathrm{T}}\end{array}]}^{\mathrm{T}}\right)\end{array}\right]}_{·}$$

(11)

Since z is considered as the DOA spatial spectrum with source power, and its imaginary part is zero, Equation (11) can be equivalently rewritten as follows:

$$\begin{array}{cc}\hfill \left[\begin{array}{c}\Re \left(\mathit{y}\right)\\ \Im \left(\mathit{y}\right)\end{array}\right]& =\left[\begin{array}{c}\Re (\mathit{\Phi }+\mathit{F}\mathit{B})\\ \Im (\mathit{\Phi }+\mathit{F}\mathit{B})\end{array}\right]\mathit{z}+\left[\begin{array}{c}{\sigma }_{\mathrm{w}}^2{[{\mathit{\eta }}_1^{\mathrm{T}}\hspace{1em}{\mathit{\eta }}_2^{\mathrm{T}}\cdots {\mathit{\eta }}_M^{\mathrm{T}}]}^{\mathrm{T}}\\ 0_{M^2\times 1}\end{array}\right]\hfill \\ \hfill & =\left(\left[\begin{array}{c}\Re \left(\mathit{\Phi }\right)\\ \Im \left(\mathit{\Phi }\right)\end{array}\right]+\left[\begin{array}{c}\Re \left(\mathit{F}\right)\\ \Im \left(\mathit{F}\right)\end{array}\right]\mathit{B}\right)\mathit{z}+\left[\begin{array}{c}{\sigma }_{\mathrm{w}}^2{[{\mathit{\eta }}_1^{\mathrm{T}}\hspace{1em}{\mathit{\eta }}_2^{\mathrm{T}}\cdots {\mathit{\eta }}_M^{\mathrm{T}}]}^{\mathrm{T}}\\ 0_{M^2\times 1}\end{array}\right]\hfill \end{array}$$

(12)

where

$$\Re (\mathit{\Phi })=\left[\Re (\mathit{\phi }({\theta }_1))\hspace{1em}\Re (\mathit{\phi }({\theta }_2))\cdots \Re (\mathit{\phi }({\theta }_q))\cdots \Re (\mathit{\phi }({\theta }_Q))\right]$$

(13)

$$\Im (\mathit{\Phi })=\left[\Im (\mathit{\phi }({\theta }_1))\hspace{1em}\Im (\mathit{\phi }({\theta }_2))\cdots \Im (\mathit{\phi }({\theta }_q))\cdots \Im (\mathit{\phi }({\theta }_Q))\right]$$

(14)

and $\mathit{\phi }({\theta }_q)$ denotes the qth column of $\mathit{\Phi }$. Furthermore, $\Re (\mathit{F})$ and $\Im (\mathit{F})$ can be expressed as follows:

$$\Re (\mathit{F})=\left[\Re \left(\mathit{f}({\theta }_1)\right)\hspace{1em}\Re \left(\mathit{f}({\theta }_2)\right)\cdots \Re \left(\mathit{f}({\theta }_q)\right)\cdots \Re \left(\mathit{f}({\theta }_Q)\right)\right]$$

(15)

$$\Im (\mathit{F})=\left[\Im \left(\mathit{f}({\theta }_1)\right)\hspace{1em}\Im \left(\mathit{f}({\theta }_2)\right)\cdots \Im \left(\mathit{f}({\theta }_q)\right)\cdots \Im \left(\mathit{f}({\theta }_Q)\right)\right]$$

(16)

where $\mathit{f}({\theta }_q)$ denotes the qth column of $\mathit{F}$.

3.2. Deep Unfolding Sparse Bayesian Learning Network

In order to accelerate the convergence speed of the SBL algorithm, we expanded the iterative steps of the SBL method into the network of cascade form, where the estimation of off-grid DOA is calculated by the peaks of the on-grid spatial spectrum and the corresponding off-grid quantization errors. As shown in Figure 2, the DSBL network contains L layers for on-grid spatial spectrum estimation and off-grid quantization error estimation, where the previous layer of off-grid quantization error matrix B is used to estimate the current layer of on-grid spatial spectrum Z, and the current layer of on-grid spatial spectrum Z is used to estimate the previous layer of off-grid quantization error matrix B.

In practice, the covariance vector in the real domain can be expressed as follows:

$$\tilde{\mathit{y}}=\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)\mathit{z}+\left[\begin{array}{c}{\sigma }_{\mathrm{w}}^2{[\begin{array}{cccc}{\mathit{\eta }}_1^{\mathrm{T}}& {\mathit{\eta }}_2^{\mathrm{T}}& \cdots & {\mathit{\eta }}_M^{\mathrm{T}}\end{array}]}^{\mathrm{T}}\\ 0_{M^2\times 1}\end{array}\right]$$

(17)

where $\tilde{\mathit{y}}={[\Re \left({\mathit{y}}^{\mathrm{T}}\right)\hspace{1em}\Im \left({\mathit{y}}^{\mathrm{T}}\right)]}^{\mathrm{T}}$, $\tilde{\mathit{\Phi }}={[\Re \left({\mathit{\Phi }}^{\mathrm{T}}\right)\Im \left({\mathit{\Phi }}^{\mathrm{T}}\right)]}^{\mathrm{T}}$, and $\tilde{\mathit{F}}={[\mathit{\Re }{\left(\mathit{F}\right)}^{\mathrm{T}}\hspace{1em}\Im {\left(\mathit{F}\right)}^{\mathrm{T}}]}^{\mathrm{T}}$.

Considering that the noise part in Equation (17) makes the optimization problem more nebulous [32], we constructed the following convex relaxation:

$$\min \left({‖\tilde{\mathit{y}}-\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)\mathit{z}‖}_2^2+\zeta {‖\mathit{z}‖}_1\right)\hspace{1em}.$$

(18)

By integrating the amplitudes of spatial power in the SBL framework [33], the probability of $\tilde{y}$ with respect to the hyperparameters z and ${\sigma }_{\mathrm{w}}^2$ can be expressed as follows:

$$\begin{array}{cc}p\left(\tilde{\mathit{y}}\left|\mathit{z},{\sigma }_{\mathrm{w}}^2\right.\right)\hfill & \hfill ={\displaystyle \int p\left(\tilde{\mathit{y}}\left|\mathit{z},{\sigma }_{\mathrm{w}}^2\right.\right)}p\left(\mathit{z}\left|\mathit{\gamma }\right.\right)d\mathit{z}\\ \hfill & =\frac{\exp \left(-\mathrm{tr}\left({\tilde{\mathit{y}}}^{\mathrm{H}}{\Sigma }_{\tilde{\mathit{y}}}^{-1}\tilde{\mathit{y}}\right)\right)}{\det (\pi {\Sigma }_{\tilde{y}})}\hspace{1em},\hfill \end{array}$$

(19)

where $\mathit{z}=[z_1\hspace{1em}{z}_2\cdots z_Q{]}^{\mathrm{T}}$ denotes the on-grid spatial power spectrum and ${\Sigma }_{\tilde{y}}=\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)\mathit{Z}{\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)}^{\mathrm{H}}+{\sigma }_{\mathrm{w}}^2{I}_{2M^2}$ with Z = diag(z) denotes the covariance matrix of the array output.

Therefore, the hyperparameters z and ${\sigma }_{\mathrm{w}}^2$ can be estimated by maximizing $p(\tilde{\mathit{y}}\left|\mathit{z},{\sigma }_{\mathrm{w}}^2\right.)$, which can be considered as the type-II maximum likelihood (ML) problem and is derived from the EM method in [34]. In this study, z and ${\sigma }_{\mathrm{w}}^2$ are updated by exploiting the iterative E-steps and M-steps of the EM method.

The estimation of the on-grid spatial spectrum in the DSBL network consists of L layers, where $\tilde{y}$, $\tilde{\mathit{\Phi }}$, and $\tilde{\mathit{F}}$ are considered as the input of each layer, and the initial hyperparameters are set as ${\mathit{{\rm Z}}}^{(0)}=\mathrm{eye}(Q)$ and ${({\sigma }_{\mathrm{w}}^2)}^{(0)}=1$. Based on the EM method, the E-step in the lth layer for on-grid spatial spectrum estimation is performed to calculate the posteriori mean and posteriori covariance:

$${\mu }_z^{(l)}={\mathit{Z}}^{(l-1)}{\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)}^{\mathrm{H}}{({\Sigma }_{\tilde{y}}^{(l-1)})}^{-1}\tilde{\mathit{y}}\hspace{1em},$$

(20)

$${\Sigma }_z^{(l)}={\mathit{Z}}^{(l-1)}-{\mathit{Z}}^{(l-1)}{\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)}^{\mathrm{H}}{({\Sigma }_{\tilde{y}}^{(l-1)})}^{-1}\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right){\mathit{Z}}^{(l-1)}\hspace{1em},$$

(21)

for l = 1, 2, …, L, where ${\Sigma }_{\tilde{y}}^{(l-1)}=\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right){\mathit{Z}}^{(l-1)}{\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)}^{\mathrm{H}}+{({\sigma }_{\mathrm{w}}^2)}^{(l-1)}{\mathit{I}}_{2M^2}$ denotes the array covariance.

Moreover, the M-step in the lth layer for on-grid spatial spectrum estimation is performed to calculate the following:

$${\mathit{Z}}^{(l)}={\mathit{\mu }}_z^{(l)}{({\mathit{\mu }}_z^{(l)})}^{\mathrm{T}}+{\Sigma }_z^{(l)}\hspace{1em},$$

(22)

and the corresponding noise variance for on-grid spatial spectrum estimation is derived from the following:

$${({\sigma }_{\mathrm{w}}^2)}^{(l)}=\frac{1}{2M^2}{‖\tilde{\mathit{y}}-\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right){\mathit{\mu }}_z^{(l)}‖}_2^2+\frac{{({\sigma }_{\mathrm{w}}^2)}^{(l-1)}}{2M^2}(Q-{\displaystyle \sum _{q=1}^Q\frac{{({\Sigma }_z^{(l)})}_{q,q}}{{({\mathit{Z}}^{(l)})}_{q,q}}})\begin{array}{c}\\ ,\end{array}$$

(23)

where ${({\Sigma }_z^{(l)})}_{q,q}$ and ${({\mathit{Z}}^{(l)})}_{q,q}$ denote the (q,q)th element of ${\Sigma }_z^{(l)}$ and ${\mathit{Z}}^{(l)}$, respectively.

As for the estimation of the off-grid quantization error matrix B, the sparse representation of the covariance vector in the real domain can be constructed as follows:

$$\tilde{\mathit{y}}-\tilde{\mathit{\Phi }}\mathit{z}=\tilde{\mathit{F}}(\mathit{B}\mathit{z})+\left[\begin{array}{c}{\sigma }_{\mathrm{w}}^2{[{\mathit{\eta }}_1^{\mathrm{T}}\hspace{1em}{\mathit{\eta }}_2^{\mathrm{T}}\cdots {\mathit{\eta }}_M^{\mathrm{T}}]}^{\mathrm{T}}\\ 0_{M^2\times 1}\end{array}\right]\begin{array}{c}\\ ,\end{array}$$

(24)

Similarly, the optimization problem of off-grid quantization error estimation can be expressed as follows:

$$\min \hspace{1em}\left({‖\tilde{\mathit{g}}-\tilde{\mathit{F}}(\mathit{B}\mathit{z})‖}_2^2+{\zeta }_2{‖\mathit{B}\mathit{z}‖}_1\right),$$

(25)

where $\tilde{\mathit{g}}=\tilde{\mathit{y}}-\tilde{\mathit{\Phi }}\mathit{z}$. In this study, the estimation of the off-grid quantization error in the DSBL network consists of L layers; the initial hyperparameters are set as ${\mathit{{\rm B}}}^{(0)}=\mathrm{eye}(Q)$ and ${({\sigma }_B^2)}^{(0)}=1$.

Based on the EM method, the E-step in the lth layer for off-grid quantization error estimation is performed to calculate the posteriori mean and posteriori covariance:

$${\mathit{\mu }}_B^{(l)}={\mathit{\Gamma }}^{(l-1)}{\tilde{\mathit{F}}}^{\mathrm{H}}{({\Sigma }_{\tilde{\mathit{g}}}^{(l-1)})}^{-1}{\tilde{\mathit{g}}}^{(l-1)}\hspace{1em},$$

(26)

$${\Sigma }_{\mathit{B}}^{(l)}={\mathit{\Gamma }}^{(l-1)}-{\mathit{\Gamma }}^{(l-1)}{\tilde{\mathit{F}}}^{\mathrm{H}}{({\Sigma }_{\tilde{\mathit{g}}}^{(l-1)})}^{-1}\tilde{\mathit{F}}{\mathit{\Gamma }}^{(l-1)}\hspace{1em},$$

(27)

for l = 1, 2, …, L, where ${\tilde{\mathit{g}}}^{(l-1)}=\tilde{\mathit{y}}-\tilde{\mathit{\Phi }}\mathrm{diag}({\mathit{Z}}^{(l-1)})$, and ${\Sigma }_{\tilde{\mathit{g}}}^{(l-1)}=\tilde{\mathit{F}}{\mathit{\Gamma }}^{(l-1)}{\tilde{\mathit{F}}}^{\mathrm{H}}+{({\sigma }_B^2)}^{(l-1)}{\mathit{I}}_{2M^2}$ denotes the array covariance.

Moreover, the M-step in the lth layer of the off-grid quantization error estimation is performed to calculate the following:

$${\mathit{\Gamma }}^{(l)}={\mathit{\mu }}_{\mathit{B}}^{(l)}{({\mathit{\mu }}_{\mathit{B}}^{(l)})}^{\mathrm{T}}+{\Sigma }_{\mathit{B}}^{(l)}\hspace{1em},$$

(28)

and the corresponding noise variance of the off-grid quantization error estimation is derived from the following:

$${({\sigma }_{\mathit{{\rm B}}}^2)}^{(l)}=\frac{1}{2M^2}{‖\tilde{\mathit{g}}-\tilde{\mathit{F}}{\mathit{\mu }}_{\mathit{{\rm B}}}^{(l)}‖}_2^2+\frac{{({\sigma }_{\mathit{{\rm B}}}^2)}^{(l-1)}}{2M^2}(Q-{\displaystyle \sum _{q=1}^Q\frac{{({\Sigma }_{\mathit{{\rm B}}}^{(l)})}_{q,q}}{{({\mathit{\Gamma }}^{(l)})}_{q,q}}})\begin{array}{c}\\ ,\end{array}$$

(29)

where ${({\Sigma }_{\mathit{B}}^{(l)})}_{q,q}$ and ${({\mathit{\Gamma }}^{(l)})}_{q,q}$ denote the (q,q)th element of ${\Sigma }_{\mathit{{\rm B}}}^{(l)}$ and ${\mathit{\Gamma }}^{(l)}$, respectively.

Therefore, by employing the output of the lth layer for the on-grid spatial spectrum ${\mathit{Z}}^{(l)}$ and off-grid quantization error ${\Gamma }^{(l)}$, the qth element on the diagonal of the off-grid quantization error matrix ${\mathit{B}}^{(l)}$ can be calculated as follows:

$$b_q^{(l)}={\Gamma }_q^{(l)}/z_q^{(l)},$$

(30)

Generally, in the training progress of the proposed DSBL network, a stochastic gradient descent (SGD) is exploited to renew the trainable parameters. Referring to the convex relaxation in Equations (18) and (25), the loss function is defined as follows:

$$\min \left({\displaystyle \sum _{t=1}^T\left({‖{\tilde{y}}_t-\left(\tilde{\mathit{\Phi }}+\tilde{\mathit{F}}\mathit{B}\right)\mathrm{diag}({\mathit{Z}}_t^{(L)})‖}_2^2+{\zeta }_1{‖\mathrm{diag}({\mathit{Z}}_t^{(L)})‖}_1+{\zeta }_2{‖{\mathit{B}}_t^{(L)}\mathrm{diag}({\mathit{Z}}_t^{(L)})‖}_1\right)}\right)\begin{array}{c}\\ .\end{array}$$

(31)

for t = 1, 2, …, T, where T stands for the total number of samples in the dataset, ${\tilde{y}}_t$ stands for the network input, and ${\mathit{Z}}_t^{(L)}$ and ${\mathit{B}}_t^{(L)}$ stand for the estimated DOA spectrum from the network output. Moreover, the proposed DSBL network can determine the off-grid DOA without training labels and large-scale training data, which have generalization abilities with interpretable parameters and layers for off-grid DOA estimation.

Based on the output of the lth layer of the on-grid spatial spectrum estimation and the output of the lth layer of the off-grid quantization error estimation, the off-grid DOA of the kth signal can be calculated as follows:

$${\widehat{\theta }}_k={\theta }_k^{(L)}+b_k^{(L)}$$

(32)

where ${\theta }_k^{(L)}$ stands for the angle corresponding to the kth spectral peak in ${\mathit{z}}^{(L)}$ and $b_k^{(L)}$ stands for the value of the kth spectral peak in ${\mathit{b}}^{(L)}$, where ${\mathit{b}}^{(L)}={[{\mathit{b}}_1^{(L)}\hspace{1em}{\mathit{b}}_2^{(L)}\cdots {\mathit{b}}_Q^{(L)}]}^{\mathrm{T}}$.

3.3. Network Implementation of Proposed Method

Overall, the main steps of the trained DSBL network for DOA estimation are summarized in Algorithm 1 (The process of implementing a proposed DSBL algorithm).

Algorithm 1 DSBL Network for Off-Grid DOA Estimation with NA
1:	Calculate the covariance matrix using Equation (4).
2:	Apply the vector form of covariance matrix in Equation (5).
3:	Combine real and imaginary parts in Equation (12) as the input of the DSBL network.
4:	Perform the trained DSBL network to acquire spatial spectrum and off-grid quantization error.
5:	Obtain off-grid DOA from the peaks of the spatial spectrum and the corresponding off-grid quantization error in Equation (3).

As for the computational complexity of the proposed DSBL network, the covariance matrix in Equation (4) requires M²N multiplications and M²(N − 1) additions; the vectorization in Equation (5) and combination in Equation (12) do not require additional calculation. When employing the trained deep unfolded SBL network to obtain the on-grid spatial spectrum and off-grid quantization error, the calculation of ${\Sigma }_{\tilde{y}}^{(l)}$ and ${\Sigma }_{\tilde{\mathit{g}}}^{(l)}$ requires 4M⁴Q + 2M²Q multiplications and 4M⁴(Q − 1) additions; the calculation of ${\mu }_z^{(l)}$ and ${\mu }_B^{(l)}$ requires 4QM²(M² + 1) multiplications, Q(4M⁴ − 1) additions, and O(2M²) for the inverse operator; the calculation of ${\Sigma }_{\mathrm{z}}^{(l)}$ and ${\Sigma }_{\mathit{B}}^{(l)}$ requires Q(4M⁴ + 2M² + 2M²Q + Q²) multiplications, Q(4M⁴ − 2M² + 2M²Q + Q² − 2Q) additions, 2M² subtractions, and O(2M²) for the inverse operator; the calculation of ${\mathit{Z}}^{(l)}$ and ${\mathit{B}}^{(l)}$ requires Q² multiplications and 2Q² additions; and the calculation of ${({\sigma }_{\mathrm{w}}^2)}^{(l)}$ in Equation (23) requires 2M² (Q + 1) multiplications, 2QM² + Q − 2 additions, 2M² subtractions, and Q + 2 divisions.

4. Computer Simulation Experiments

In this section, a two-level NA with six sensors was exploited to investigate the performance of the DSBL network for off-grid DOA estimation. Specifically, the locations were set as [1,2,3,4,8,12]d, and the angle interval of the overcomplete dictionary was set as 1°. Of the samples, 80% of those in the dataset were used for network training, and 20% were used for network validation. Each sample was generated by two signals, with an off-grid angle between −60° and 60°. The signal-to-noise ratio (SNR) was selected from 0 dB to 20 dB, and the number of snapshots was selected from 100 to 500. During the training process of the network, the batch size, epoch, and learning rate were set to 16, 16, and 0.01, respectively.

4.1. Layer Number Determination

In this subsection, we employed simulation experiments to determine the layer number. During the training process of the DSBL network, the appropriate layer number was determined by the mean square error (MSE). In this research, the MSE is defined as follows:

$$\mathrm{MSE}=\frac{1}{T}{\displaystyle \sum _{t=1}^T{\left(({\mathit{Z}}_t^{(L)}-{\mathit{Z}}_t^{\mathrm{label}})+({\mathit{B}}_t^{(L)}-{\mathit{B}}_t^{\mathrm{label}})\right)}^2}\begin{array}{c}\\ .\end{array}$$

(33)

where ${\mathit{Z}}_t^{(L)}$ and ${\mathit{Z}}_t^{\mathrm{label}}$ denote the output and label of the on-grid spatial spectrum for the tth data, respectively; ${\mathit{B}}_t^{(L)}$ and ${\mathit{B}}_t^{\mathrm{label}}$ denote the output and label of the off-grid quantization error matrix for the tth data, respectively.

During the training and validation process of the DSBL network, the variation in the RMSE with the epoch is shown in Figure 3. From the figure, it can be seen that the RMSEs of layers 10, 20, 30, and 40 gradually decrease with the increase in epoch during the training process, and the RMSE of the 40-layer network is smaller than that of the other layer networks. This indicates that the estimation accuracy of the 40-layer network is better than that of the other layer networks. Due to insufficient training in the initial stage, when the epoch is less than five, the RMSEs of the networks with more layers are greater than those of networks with fewer layers. As the epoch increases, the RMSEs of networks with more layers gradually decrease compared to networks with fewer layers. After the training process of the DSBL network is completed, the RMSE of the 40-layer network is slightly smaller than that of the 30-layer network. In order to balance the accuracy of the off-grid angle estimation and computational complexity, the DSBL network is set to 30 layers. In addition, as shown in Figure 3b, during the validation process, the RMSEs of the 10-, 20-, 30-, and 40-layer networks gradually decreased with the increase in epoch, indicating that there was no overfitting during the training process of the DSBL network.

4.2. Comparison of Convergence Performance

When the off-grid DOA of the test samples are set to −10.95° and 2.98°, Figure 4 shows the relative error of the off-grid DOA estimation. The red solid line represents the relative error of the DSBL network, and the blue dashed line represents the relative error of the model-driven algorithm. As shown in Figure 4, compared to the 8-layer DSBL network, the model-driven algorithm converges after 16 iterations. Due to the computational complexity of each layer in the DSBL network being the same as in the model-driven algorithm, the DSBL network can converge in a shorter time.

4.3. Generalization Ability for Off-Grid DOA Estimation

In order to verify the generalization ability of the deep unfolding network for off-grid DOAs under different numerical conditions, a total of 120 test samples were generated, with each containing a signal. The angle of the spatial spectrum on the grid was set to −60° to 59° with an interval of 1°; the off-grid quantization error was set to a random value of 0° to 1°; and the signal-to-noise ratio was set to 5 dB. The off-grid DOA estimates obtained through the DSBL network are shown in Figure 5a, and the off-grid estimation error is shown in Figure 5b. It can be seen that the DSBL network has the ability to generalize the off-grid angle under different numerical conditions. Due to the fact that the DSBL network models the iterative steps of the corresponding sparse reconstruction algorithm as hidden layers of the neural network, the network parameters have certain mathematical meanings. During the training process, the deep unfolding network can learn the rules hidden behind the data. Therefore, for untrained data, the DSBL network can also estimate the off-grid angle.

In this study, the DOA estimates and errors are shown in Figure 5a,b, respectively, where the abscissa denotes the testing index and the red dots and blue dots denote the first sources and the second sources, respectively. Therefore, we can conclude that the trained DSBL network has a generalization ability for the DOA estimation of off-grid sources.

4.4. RMSE Comparison of DOA Estimation

In this subsection, root mean square error (RMSE) analysis is performed to investigate the performance of the proposed DSBL network. In this study, the RMSE of DOA estimation is defined as follows:

$$\mathrm{RMSE}\left(\theta \right)=\sqrt{\frac{1}{VK}{\displaystyle \sum _{v=1}^V{\displaystyle \sum _{k=1}^K{\left({\widehat{\theta }}_k^{(v)}-{\theta }_k\right)}^2}}},$$

(34)

where ${\theta }_k$ denotes the real DOA of the kth source and ${\widehat{\theta }}_k^{(v)}$ denotes the estimated DOA of the kth source in the vth Monte Carlo simulation experiment.

The RMSE of the proposed DSBL network was compared with the FOCUSS network in [35], the RVSBL algorithm in [36], the JSR algorithm in [13], and the Cramér–Rao lower bound (CRLB) in [37]. In total, 500 simulation experiments were performed to calculate the RMSEs of two sources, where the DOAs were set as −10.1° and 20.8°, respectively. The RMSE versus the SNR and snapshot number are shown in Figure 6a,b, respectively, where the RMSE gradually decreases with the increase in the SNR and snapshot number. Since the noise power is not updated in the iterative process of the JSR algorithm and RVSBL algorithm, the accuracy of the DSBL network outperforms the existing methods.

5. Conclusions

In this research, the DSBL network was constructed for off-grid DOA estimation using the geometry of NAs with mutual coupling. Firstly, the array covariance of the NA was transformed into an equivalent single snapshot, which can form a continuous array with virtual sensors and increase the degrees of freedom. Then, the vectorization of the array covariance was transformed into the real domain and considered as the input to the DSBL network. Next, the DSBL network was constructed and trained to determine the MCC, where the iterative steps of the EM algorithm were transformed into the layers of the DSBL network, and the loss function was only related to the reconstruction error and the sparsity of the network output. Therefore, the training labels and large-scale training data were not required during the training process of the DSBL network. Finally, the off-grid DOA can be obtained from the peaks of the spatial spectrum and the corresponding off-grid quantization error. The simulation results demonstrate that the proposed DSBL network has better generalization ability with interpretable parameters and layers for off-grid DOA estimation with different source numbers. Compared with the joint sparse recovery method, the SBL method, and the RARE method, the proposed DSBL network achieved a more accurate DOA estimation in the cases of limited snapshot numbers and low SNRs.