DOA Estimation with Coprime Arrays Using Toeplitz and Hankel-Based Structured Covariance Reconstruction

Abstract

Coprime arrays are attractive for direction-of-arrival (DOA) estimation because they can generate a large virtual aperture from a limited number of physical sensors. Their performance, however, deteriorates markedly when coherent sources coexist with unknown nonuniform sensor noise. To cope with this difficulty, this paper develops a structured DOA estimation scheme that integrates difference-coarray lag averaging, Toeplitz positive semidefinite covariance reconstruction, Hankel-based low-rank refinement, and forward–backward spatial smoothing. The sample covariance of the physical coprime array is first mapped into the coarray domain, where repeated lags are averaged, and missing lags are treated by a mask, rather than by zero padding. A Hermitian Toeplitz positive semidefinite virtual covariance matrix is then recovered in the lag domain with redundancy-aware weighting. To further enhance robustness under source coherence, the reconstructed covariance sequence is refined through a Hankel-structured low-rank restoration step. The recovered virtual covariance is finally processed by forward–backward spatial smoothing, and DOAs are obtained from the MUSIC spectrum. Simulation results under coherent-source and unknown nonuniform-noise scenarios show that the proposed method yields a lower estimation error than representative baselines, preserves clear spectral separation in multi-source cases, and maintains reliable two-source resolution under different angular separations. Additional experiments further examine RMSE trends with respect to SNR, snapshots, source number, and computational costs.

1. Introduction

Direction-of-arrival (DOA) estimation is a central problem in array signal processing, and it plays an important role in radar, sonar, wireless communications, electronic surveillance, and intelligent sensing systems [1,2,3]. Classical high-resolution methods such as MUSIC and ESPRIT have established the foundation of modern subspace-based DOA estimation. Their performance, however, is usually derived under ideal assumptions, including spatially white noise and mutually uncorrelated sources. In practical sensing environments, these assumptions are often violated. In particular, coherent sources may destroy the rank structure required by subspace separation, while sensor-dependent nonuniform noise may distort the covariance matrix and degrade the reliability of conventional estimators.

In addition to conventional MUSIC, Root-MUSIC has been widely used as a stronger subspace-based DOA estimator. Root-MUSIC exploits the polynomial-rooting property of the uniform linear array steering vector and avoids exhaustive angular spectral search. Therefore, it can reduce the influence of finite grid resolution and usually provides a more efficient classical subspace baseline. However, when the covariance matrix is incomplete, when coherent sources cause rank deficiency, or when unknown nonuniform noise distorts the covariance structure, Root-MUSIC may still suffer from degraded estimation accuracy.

Sparse arrays have attracted extensive interest because they can generate enlarged virtual apertures and increased degrees of freedom with relatively few physical sensors. Among the existing sparse configurations, coprime arrays are especially appealing due to their structured construction, low mutual coupling, and flexible difference-coarray geometry [4,5,6]. By exploiting the difference coarray, a coprime array can be mapped to a virtual uniform linear array (ULA) with an aperture significantly larger than that of the physical array, which makes it suitable for high-resolution DOA estimation and underdetermined scenarios.

Beyond coprime and nested arrays, Zhao et al. proposed a k-level extended sparse array based on the sum-difference coarray, which increases the uniform degrees of freedom and reduces coarray redundancy for DOA estimation [7]. This study further shows that sparse-array geometry design and coarray-structure exploitation are important routes for improving angular resolution and underdetermined DOA estimation performance.

A considerable body of work has focused on covariance reconstruction and virtual-array interpolation for coprime arrays. Early studies investigated matrix completion, nuclear-norm-based interpolation, and related reconstruction methods to recover missing lags in the coarray domain [8,9,10,11,12]. Subsequent research further improved the exploitation of the full virtual aperture through enhanced interpolation and covariance fitting strategies [13,14,15]. Although these approaches improve the effective aperture and often yield better resolution than direct processing on the physical array, many of them are primarily designed for incoherent sources or ideal noise conditions.

Another major obstacle arises from unknown nonuniform noise. When the sensor noise powers are unequal, the diagonal entries of the covariance matrix are contaminated by nonuniform noise terms, and the standard white-noise model assumed by conventional subspace methods no longer holds. For uniform linear arrays, several methods have been developed to estimate source and noise parameters under nonuniform noise, including maximum-likelihood formulations, iterative covariance fitting, and nonuniform-noise-aware subspace estimation [16,17,18]. Spatial smoothing strategies have also been studied for coherent sources in the presence of nonuniform noise [19]. Nevertheless, these methods are mainly designed for dense ULAs and do not fully exploit the virtual-domain advantages provided by coprime arrays.

To extend nonuniform-noise suppression to sparse arrays, several coprime-array-based methods have been reported. Representative approaches combine diagonal removal, covariance reconstruction, matrix completion, or prediction-based interpolation to improve robustness under sensor-dependent noise [20,21,22]. In parallel, coherent-source DOA estimation with coprime arrays has also received increasing attention. Atomic norm minimization, reweighted atomic norm minimization, and low-rank reconstruction techniques have shown that Toeplitz-structured covariance recovery is effective for coherent-source processing in sparse-array settings [23,24,25,26].

Recently, deep-learning-based DOA estimation has received increasing attention because neural networks can learn nonlinear mappings between covariance features and DOA-related representations. For sparse arrays and coprime arrays, learning-based covariance reconstruction and interpolation are particularly relevant. A recent DL-based coprime array interpolation method uses a Deep Matrix Iterative Network to reconstruct an interpolated virtual covariance matrix under a self-supervised loss based on the Hermitian positive semidefinite Toeplitz condition [27].

Despite these advances, jointly handling coherent sources, incomplete coarray observations, and unknown nonuniform noise remains difficult. On the one hand, the virtual covariance samples obtained from the difference coarray may be incomplete and statistically uneven, and some of them may be affected by unknown diagonal noise terms. On the other hand, source coherence weakens the effective rank condition required by conventional subspace decomposition. The existing methods usually emphasize either nonuniform-noise suppression or coherent-source recovery, whereas fewer approaches integrate structured virtual covariance recovery, coherence-aware spectral refinement, and subspace-based estimation within one consistent framework.

Motivated by these observations, this paper develops a structured DOA estimation method for coprime arrays in the presence of coherent sources and unknown nonuniform noise. The physical-array sample covariance is first mapped into the difference-coarray domain through lag averaging. A weighted Toeplitz positive semidefinite reconstruction model is then formulated in the lag domain to recover a valid virtual covariance matrix while avoiding artificial zero filling of missing lags. To further improve robustness against coherence-induced degradation, a Hankel-structured low-rank restoration stage is applied to the reconstructed covariance sequence. Finally, forward–backward spatial smoothing (FBSS) and MUSIC are employed to obtain high-resolution DOA estimates from the recovered virtual ULA covariance.

The main contributions and findings of this work are summarized as follows:

• A structured DOA estimation framework is developed for coprime arrays in the presence of coherent sources and unknown nonuniform noise. The framework integrates difference-coarray lag averaging, lag-domain Toeplitz positive semidefinite reconstruction, Hankel-based low-rank restoration, and forward–backward spatial smoothing into a unified processing chain.
• A nonzero-lag weighted Toeplitz reconstruction model is formulated to reduce the bias caused by unknown nonuniform diagonal noise while avoiding artificial zero filling of missing coarray lags. The redundancy-aware weighting further improves the reliability of the fitted lag observations.
• A Hankel-enhanced covariance-sequence restoration stage is introduced after Toeplitz reconstruction. This step strengthens the spectral structure of the recovered covariance sequence and improves robustness for coherent-source scenarios.
• Extensive simulations, including RMSE curves, confidence intervals, CRLB references, source-number analysis, two-source resolution experiments, recovered MUSIC spectra, and runtime comparison, verify the effectiveness and limitations of the proposed method compared with Root-MUSIC, DL-DOA, RNM, and LowRank baselines.

The rest of this paper is organized as follows. Section 2 introduces the signal model and the problem formulation. Section 3 presents the proposed method. Section 4 reports simulation results and discussion. Section 5 concludes the paper.

2. Signal Model and Problem Formulation

Consider a coprime array composed of two sparse subarrays characterized by a coprime integer pair, $(M,N)$. Let the set of physical sensor positions be

$$\mathcal{P}=\{p_1,p_2,\dots ,p_L\},$$

(1)

where L denotes the number of distinct physical sensors, and each $p_{\mathcal{l}}$ is measured in units of the basic inter-element spacing d.

Assume that K narrowband far-field sources impinge on the array from directions

$$\mathit{\theta }={[{\theta }_1,{\theta }_2,\dots ,{\theta }_K]}^T.$$

(2)

At snapshot t, the received signal can be written as

$$\mathbf{x}\left(t\right)=\mathbf{A}\left(\mathit{\theta }\right)\mathbf{s}\left(t\right)+\mathbf{n}\left(t\right),$$

(3)

where $\mathbf{x}\left(t\right)\in {\mathbb{C}}^{L\times 1}$ is the array observation vector, $\mathbf{s}\left(t\right)\in {\mathbb{C}}^{K\times 1}$ is the source vector, $\mathbf{n}\left(t\right)\in {\mathbb{C}}^{L\times 1}$ is the additive noise vector, and

$$\mathbf{A}\left(\mathit{\theta }\right)=[\mathbf{a}\left({\theta }_1\right),\mathbf{a}\left({\theta }_2\right),\dots ,\mathbf{a}\left({\theta }_K\right)]\in {\mathbb{C}}^{L\times K}$$

(4)

is the array manifold matrix.

For a source arriving from direction $\theta$, the steering vector is

$$\mathbf{a}\left(\theta \right)={\left[e^{j2\pi (d/\lambda )p_1\sin \theta },e^{j2\pi (d/\lambda )p_2\sin \theta },\dots ,e^{j2\pi (d/\lambda )p_L\sin \theta }\right]}^T,$$

(5)

where $\lambda$ is the wavelength. In this paper, the inter-element spacing is set to $d=\lambda /2$ unless otherwise stated.

The ensemble covariance matrix of the received data is

$${\mathbf{R}}_{xx}=\mathbb{E}\left\{\mathbf{x}\left(t\right){\mathbf{x}}^H\left(t\right)\right\}=\mathbf{A}{\mathbf{R}}_s{\mathbf{A}}^H+{\mathbf{R}}_n,$$

(6)

where ${\mathbf{R}}_s=\mathbb{E}\left\{\mathbf{s}\left(t\right){\mathbf{s}}^H\left(t\right)\right\}$ is the source covariance matrix, and ${\mathbf{R}}_n=\mathbb{E}\left\{\mathbf{n}\left(t\right){\mathbf{n}}^H\left(t\right)\right\}$ is the noise covariance matrix.

The noise is assumed to be spatially uncorrelated but nonuniform across sensors, namely

$${\mathbf{R}}_n=\mathrm{diag}({\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_L^2),$$

(7)

where the sensor noise powers ${\sigma }_{\mathcal{l}}^2$ are generally different. This model contaminates the diagonal entries of the covariance matrix and breaks the equal-noise assumption required by many classical subspace estimators.

Since the ensemble covariance is unavailable in practice, it is approximated from T snapshots as

$${\widehat{\mathbf{R}}}_{xx}={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}\mathbf{x}\left(t\right){\mathbf{x}}^H\left(t\right).$$

(8)

For sparse arrays, the enlarged virtual aperture is obtained from the difference coarray. Each covariance entry, ${\left[{\widehat{\mathbf{R}}}_{xx}\right]}_{i,j}$, corresponds to a spatial lag

$${\mathcal{l}}_{ij}=p_i-p_j.$$

(9)

Hence, the physical covariance matrix may be reinterpreted as a collection of coarray observations defined on the difference set

$$\mathcal{D}=\{p_i-p_j|1\le i,j\le L\}.$$

(10)

For a given lag, $\mathcal{l}\in \mathcal{D}$, define the index set

$${\mathcal{S}}_{\mathcal{l}}=\left\{(i,j)\right|p_i-p_j=\mathcal{l}\},$$

(11)

and let

$$c\left(\mathcal{l}\right)=|{\mathcal{S}}_{\mathcal{l}}|$$

(12)

denote the redundancy count of lag ℓ. The averaged coarray observation at lag ℓ is then

$$r_{\mathrm{obs}}\left(\mathcal{l}\right)={\displaystyle \frac{1}{c\left(\mathcal{l}\right)}}\sum _{(i,j)\in {\mathcal{S}}_{\mathcal{l}}}{\left[{\widehat{\mathbf{R}}}_{xx}\right]}_{i,j}.$$

(13)

Because ${\widehat{\mathbf{R}}}_{xx}$ is Hermitian, the lag observations satisfy

$$r_{\mathrm{obs}}(-\mathcal{l})=r_{\mathrm{obs}}^{*}\left(\mathcal{l}\right).$$

(14)

Therefore, it is sufficient to retain only the nonnegative lags and form the lag observation vector

$${\mathbf{r}}_{\mathrm{obs}}={[r_{\mathrm{obs}}\left(0\right),r_{\mathrm{obs}}\left(1\right),\dots ,r_{\mathrm{obs}}(L_v-1)]}^T,$$

(15)

where $L_v$ is the length of the target virtual ULA.

At this point, two difficulties appear naturally. First, the difference coarray is generally incomplete, which means that not every lag between 0 and $L_v-1$ is observed. Let

$$\Omega \subseteq \{0,1,\dots ,L_v-1\}$$

(16)

be the set of observed nonnegative lags.

Second, the zero-lag observation behaves differently from the off-zero lags under unknown nonuniform noise. When $i=j$, the covariance entry contains both signal power and sensor-noise power. After lag averaging, the zero-lag statistic, therefore, includes an aggregate contribution of the unknown diagonal noise covariance in (7). In contrast, for $i\ne j$, the noise term vanishes because ${\mathbf{R}}_n$ is diagonal. As a result, the nonzero-lag observations mainly reflect source-induced spatial correlation, whereas the zero-lag observation is additionally affected by unknown nonuniform noise.

The problem considered in this work is, therefore, to recover a structured virtual covariance matrix from incomplete lag observations under coherent sources and unknown nonuniform noise and then use the recovered covariance for high-resolution DOA estimation.

3. Proposed Method

The proposed framework addresses three coupled difficulties in coprime-array DOA estimation: incomplete coarray observations, unknown nonuniform noise, and source coherence. Instead of handling them independently, the method forms a sequential reconstruction pipeline in which each stage improves the quality of the input to the next stage. Lag averaging first converts the physical sparse-array covariance into virtual-domain observations. A lag-domain Toeplitz positive semidefinite reconstruction then recovers a physically valid virtual covariance matrix while avoiding direct fitting of the noise-contaminated zero lag. A Hankel-structured low-rank refinement is subsequently introduced to enhance the spectral consistency of the recovered covariance sequence under coherent-source conditions. Finally, FBSS and MUSIC are performed on the refined virtual covariance.

3.1. Lag Averaging and Virtual Coarray Observation Model

Starting from the sample covariance matrix ${\widehat{\mathbf{R}}}_{xx}$ in (8), covariance entries associated with the same spatial lag are grouped and averaged according to (13). This step does more than reindex the measurements. It also establishes the bridge between the physical sparse array and its virtual ULA representation. Once covariance terms with identical sensor-position differences are mapped to the same lag, the coprime array can be processed through its difference coarray, whose virtual aperture is typically much larger than that of the original physical array.

To indicate which lags are available, define the binary observation mask

$$m\left(\mathcal{l}\right)=\left\{\begin{array}{cc}1,\hfill & \mathcal{l}\in \Omega ,\hfill \\ 0,\hfill & \mathcal{l}\notin \Omega .\hfill \end{array}\right.$$

(17)

This mask-based representation is preferable to direct zero filling. Missing lags are not assigned arbitrary numerical values; instead, they are left outside the data-fidelity term and inferred through structural constraints during the reconstruction stage. This is important because hard zero filling may introduce artificial bias into the virtual covariance sequence and distort the Toeplitz structure expected from a true virtual ULA covariance. In addition, the redundancy count $c\left(\mathcal{l}\right)$ is retained for each observed lag. Since lags with larger redundancy are obtained by averaging more covariance entries, they are generally more reliable and may, therefore, be assigned higher weights during reconstruction.

Under unknown nonuniform noise, the zero-lag observation differs fundamentally from all nonzero lags. Since the noise covariance matrix is diagonal, the off-diagonal covariance entries contain no direct noise contribution. Hence, the nonzero-lag observations mainly encode source-induced correlation. The zero-lag observation, however, contains both signal power and aggregated sensor-noise power. If this term is fitted together with the off-zero lags, the recovery model is forced to explain statistics that include unknown and sensor-dependent noise contributions, which may bias the reconstructed covariance. For this reason, the proposed method excludes the zero lag from the data-fitting term and relies on the observed off-zero lags, together with Toeplitz and positive semidefinite constraints to infer a consistent virtual covariance matrix.

3.2. Lag-Domain Toeplitz Positive Semidefinite Reconstruction

Let $\mathbf{T}\in {\mathbb{C}}^{L_v\times L_v}$ denote the reconstructed covariance matrix of the target virtual ULA. Since the virtual array is uniform, the ideal covariance matrix should have a Hermitian Toeplitz structure. Therefore, its entries satisfy

$${\left[\mathbf{T}\right]}_{i,j}=t_{i-j},$$

(18)

where $t_{i-j}$ is the covariance value associated with spatial lag $i-j$. The covariance sequence is represented by the first column

$$\mathbf{t}={[t_0,t_1,\dots ,t_{L_v-1}]}^T.$$

(19)

For each observed nonzero lag, ℓ, the redundancy-aware weight is explicitly defined as

$$w_{\mathcal{l}}={\displaystyle \frac{c\left(\mathcal{l}\right)}{{max}_{k\in \Omega ,k\ge 1}c\left(k\right)}},\mathcal{l}\in \Omega ,\mathcal{l}\ge 1,$$

(20)

where $c\left(\mathcal{l}\right)$ is the redundancy count of lag ℓ. This definition assigns a larger weight to lags observed by more covariance-entry pairs because such lags are statistically more reliable after averaging. The normalization in (20) keeps the weights within a comparable numerical range and avoids changing the overall scale of the optimization problem.

The virtual covariance matrix is reconstructed by solving the following explicit lag-domain Toeplitz positive semidefinite optimization problem:

$$\widehat{\mathbf{T}}=\arg \underset{\mathbf{T},\mathbf{t}}{\min }\sum _{\mathcal{l}\in \Omega ,\mathcal{l}\ge 1}{w}_{\mathcal{l}}{\left|t_{\mathcal{l}}-r_{\mathrm{obs}}\left(\mathcal{l}\right)\right|}^2+\lambda \mathrm{tr}\left(\mathbf{W}\mathbf{T}\right),$$

(21)

subject to

$$\mathbf{T}⪰0,$$

(22)

$$\mathbf{T}=\mathrm{Toeplitz}\left(\mathbf{t}\right),$$

(23)

and

$$t_0\in \mathbb{R}.$$

(24)

In (21), the scalar $t_{\mathcal{l}}$ represents the reconstructed covariance value at spatial lag ℓ, while $r_{\mathrm{obs}}\left(\mathcal{l}\right)$ denotes the averaged coarray observation obtained from the physical-array sample covariance at the same lag. The set $\Omega$ contains the observed nonnegative lags, and the condition $\mathcal{l}\ge 1$ indicates that only the observed nonzero lags are used in the data-fitting term. This is because the zero-lag component contains both source power and unknown nonuniform sensor-noise power, whereas the nonzero lags mainly preserve source-induced spatial correlation.

Equation (20) defines the redundancy-aware lag weight. Lags with larger redundancy counts are assigned larger weights because they are obtained by averaging more covariance entries and are, therefore, statistically more reliable. Equation (21) gives the proposed lag-domain Toeplitz covariance reconstruction objective. The first term, ${\sum }_{\mathcal{l}\in \Omega ,\mathcal{l}\ge 1}{w}_{\mathcal{l}}{|t_{\mathcal{l}}-r_{\mathrm{obs}}\left(\mathcal{l}\right)|}^2$, measures the weighted mismatch between the reconstructed covariance lags and the observed coarray lags. The second term, $\lambda \mathrm{tr}\left(\mathbf{W}\mathbf{T}\right)$, is a structured regularization term, where $\lambda$ is the regularization parameter, $\mathbf{W}$ is the reweighting matrix, and $\mathrm{tr}(·)$ denotes the matrix trace. This term suppresses finite-snapshot perturbations and promotes a compact signal-subspace representation in the reconstructed covariance matrix.

Equation (22) imposes the positive semidefinite constraint on the recovered covariance matrix. From a signal-processing perspective, this constraint ensures that the received power associated with any spatial weighting vector is nonnegative, which is a necessary property of a physically valid covariance matrix. Equation (23) enforces the Hermitian Toeplitz structure of the virtual ULA covariance matrix, meaning that the covariance between two virtual sensors depends only on their spatial separation, rather than on their absolute sensor indices. Equation (24) guarantees that the zero-lag covariance component is real-valued. With these constraints, the incomplete coarray observations are converted into a structured virtual ULA covariance matrix for the subsequent Hankel refinement, forward–backward spatial smoothing, and MUSIC estimation.

3.3. Hankel-Enhanced Low-Rank Restoration

To improve robustness under coherent-source conditions, a Hankel-based low-rank restoration stage is introduced after Toeplitz reconstruction. The rationale is that a superposition of a small number of complex exponentials admits a low-rank Hankel representation. Since the covariance sequence generated by a few sources shares this spectral structure, imposing low rank on a Hankel matrix formed from the reconstructed sequence helps suppress residual perturbations and enhance spectral consistency.

From the Toeplitz-reconstructed sequence $\widehat{\mathbf{t}}$, construct the Hankel matrix

$$\mathcal{H}\left(\widehat{\mathbf{t}}\right)=\left[\begin{array}{cccc}{\widehat{t}}_0& {\widehat{t}}_1& \cdots & {\widehat{t}}_{L_2-1}\\ {\widehat{t}}_1& {\widehat{t}}_2& \cdots & {\widehat{t}}_{L_2}\\ ⋮& ⋮& \ddots & ⋮\\ {\widehat{t}}_{L_1-1}& {\widehat{t}}_{L_1}& \cdots & {\widehat{t}}_{L_v-1}\end{array}\right],$$

(25)

where $L_0=L_v$ denotes the length of the covariance sequence, and the Hankel dimensions are chosen as

$$L_1=⌊{\displaystyle \frac{L_0+1}{2}}⌋,L_2=L_0-L_1+1.$$

(26)

This choice satisfies $L_1+L_2-1=L_0$ and makes the Hankel matrix square or near-square. A square or near-square Hankel matrix is preferred because it avoids an excessively unbalanced matrix shape and improves the numerical behavior of nuclear-norm minimization.

The refined covariance sequence is obtained by solving the following Hankel nuclear-norm regularized problem:

$$\widehat{\mathbf{z}}=\arg \underset{\mathbf{z}}{\min }{\parallel \mathcal{H}\left(\mathbf{z}\right)\parallel }_{*}+{\displaystyle \frac{\mu }{2}}{\parallel \mathbf{z}-\widehat{\mathbf{t}}\parallel }_2^2,$$

(27)

where ${\parallel ·\parallel }_{*}$ is the nuclear norm, $\mathbf{z}$ is the restored covariance sequence, and $\mathcal{H}\left(\mathbf{z}\right)$ denotes the Hankel matrix generated from $\mathbf{z}$. Compared with the previous formulation using both $\mathbf{H}$ and $\mathbf{z}$ as optimization variables, (27) avoids the redundant equality constraint $\mathbf{H}=\mathcal{H}\left(\mathbf{z}\right)$ and directly optimizes the restored covariance sequence.

The parameters $\lambda$ and $\mu$ control the two regularization stages and can be chosen in practice by a small validation experiment or sensitivity analysis under representative SNR and snapshot settings. Specifically, $\lambda$ balances lag-domain data fitting and trace-based covariance regularization in (21); a very small value may overfit finite-snapshot perturbations, whereas an excessively large value may over-smooth the reconstructed covariance and weaken weak-source components. The Hankel parameter $\mu$ in (27) balances low-rank smoothing and fidelity to the Toeplitz-reconstructed sequence. A smaller $\mu$ gives stronger smoothing, while a larger $\mu$ preserves more of the Toeplitz-reconstructed sequence. In the simulations, $\lambda$ is selected from a moderate range that stabilizes the lag-fitting residual while preserving clear MUSIC peaks, and $\mu =30$ is adopted as a practical compromise. The results are not highly sensitive to $\mu$ within a moderate interval around 30, but overly small or overly large values may lead to over-smoothing or insufficient Hankel refinement, respectively.

After solving (27), the refined virtual covariance matrix is reconstructed as

$${\tilde{\mathbf{R}}}_v=\mathrm{Toeplitz}\left(\widehat{\mathbf{z}}\right).$$

(28)

3.4. Forward–Backward Spatial Smoothing and MUSIC Estimation

Let the virtual ULA contain $L_v$ sensors, and let the chosen subarray length be $L_s$. Then, the number of overlapping forward subarrays is

$$Q=L_v-L_s+1.$$

(29)

Let ${\mathbf{J}}_q$ denote the selection matrix that extracts the qth forward subarray. The forward smoothed covariance matrix is

$${\mathbf{R}}_{\mathrm{F}}={\displaystyle \frac{1}{Q}}{\displaystyle \sum _{q=1}^Q}{\mathbf{J}}_q{\tilde{\mathbf{R}}}_v{\mathbf{J}}_q^H.$$

(30)

Let $\mathbf{\Pi }$ be the exchange matrix of suitable dimension. The backward smoothed covariance matrix is

$${\mathbf{R}}_{\mathrm{B}}=\mathbf{\Pi }{\mathbf{R}}_{\mathrm{F}}^{*}\mathbf{\Pi },$$

(31)

and the final FBSS covariance matrix is

$${\mathbf{R}}_{\mathrm{FB}}={\displaystyle \frac{1}{2}}\left({\mathbf{R}}_{\mathrm{F}}+{\mathbf{R}}_{\mathrm{B}}\right).$$

(32)

It should be noted that applying FBSS to the reconstructed virtual ULA covariance matrix does not destroy the Toeplitz covariance property. Since the forward subarrays are selected as contiguous virtual ULA segments, forward smoothing preserves the lag-dependent covariance structure. The backward smoothing operation uses conjugate reversal through the exchange matrix, which also preserves the Hermitian Toeplitz structure. Therefore, FBSS enhances coherent-source decorrelation while keeping the recovered covariance matrix consistent with the virtual ULA model. Placing FBSS after Toeplitz reconstruction and Hankel refinement is deliberate. Spatial smoothing is most effective when the input covariance matrix is already complete and structurally reliable. If FBSS were applied directly to incomplete or severely contaminated coarray samples, the resulting subarray covariances would inherit reconstruction errors and noise distortions. By first recovering a physically consistent virtual covariance and then refining its spectral sequence, the proposed method provides a more reliable input to FBSS and improves the quality of the subsequent subspace decomposition.

Applying eigendecomposition to ${\mathbf{R}}_{\mathrm{FB}}$ yields

$${\mathbf{R}}_{\mathrm{FB}}={\mathbf{E}}_s{\mathbf{\Lambda }}_s{\mathbf{E}}_s^H+{\mathbf{E}}_n{\mathbf{\Lambda }}_n{\mathbf{E}}_n^H,$$

(33)

where ${\mathbf{E}}_s$ and ${\mathbf{E}}_n$ denote the signal and noise subspaces, respectively.

For a candidate angle, $\theta$, let ${\mathbf{a}}_v\left(\theta \right)$ be the steering vector of the smoothed virtual ULA. The MUSIC pseudospectrum is then

$$P_{\mathrm{MUSIC}}\left(\theta \right)={\displaystyle \frac{1}{{\mathbf{a}}_v^H\left(\theta \right){\mathbf{E}}_n{\mathbf{E}}_n^H{\mathbf{a}}_v\left(\theta \right)}}.$$

(34)

The DOA estimates are obtained from the locations of the K dominant peaks.

3.5. Advantages, Disadvantages, and Comparison with Other Methods

Compared with the Root-MUSIC applied after basic difference-coarray lag averaging, the proposed method further introduces Toeplitz-PSD covariance reconstruction, Hankel low-rank refinement, and FBSS before MUSIC estimation. Therefore, it can better suppress missing-lag errors, nonuniform-noise-induced covariance distortion, and coherence-induced rank deficiency. Root-MUSIC is computationally efficient, but its performance may be limited when the virtual covariance matrix is incomplete or strongly perturbed by coherent sources and unknown nonuniform noise.

Compared with direct coarray interpolation or zero-filling methods, the proposed method does not assign artificial values to missing lags. Instead, missing lags are inferred through Toeplitz and positive semidefinite structural constraints. This improves the physical consistency of the recovered virtual covariance matrix and avoids the bias introduced by hard zero padding. Compared with general nuclear-norm or low-rank reconstruction methods, the proposed method additionally uses the Hermitian Toeplitz covariance structure of a virtual ULA and a Hankel-based spectral refinement stage. The Toeplitz stage reconstructs a valid covariance matrix, while the Hankel stage enhances the low-rank spectral sequence associated with a small number of sources. These two structures are complementary.

The proposed scheme also involves disadvantages. Its computational complexity is higher than that of conventional MUSIC because convex optimization and iterative structured reconstruction are required. The performance depends on the selected regularization parameters, the quality of the sample covariance matrix, the available coarray aperture, and the assumed or estimated number of sources. When the SNR is extremely low, the number of snapshots is very small, or severe model mismatch exists, the reconstructed covariance may still be affected by residual errors. Therefore, the proposed method is more suitable for scenarios where estimation accuracy and robustness are more important than strict real-time computation.

3.6. Algorithm Summary

The overall procedure of the proposed method is summarized in Algorithm 1.

Algorithm 1 Proposed DOA Estimation Method for Coprime Arrays under Unknown Nonuniform Noise

1: Compute the sample covariance matrix from the physical coprime-array measurements. 2: Map covariance entries into the difference-coarray domain and perform lag averaging. 3: Construct the nonnegative-lag observation vector, redundancy counts, and observation mask. 4: Reconstruct a Hermitian Toeplitz positive semidefinite virtual covariance matrix by solving the lag-domain weighted optimization problem using only observed off-zero lags. 5: Extract the reconstructed covariance sequence and perform Hankel low-rank restoration. 6: Rebuild the refined virtual ULA covariance matrix from the restored sequence. 7: Apply forward–backward spatial smoothing to the refined virtual covariance. 8: Compute the MUSIC spectrum and estimate the DOAs from the dominant spectral peaks.

4. Simulation Results and Discussion

4.1. Simulation Settings

Unless otherwise stated, the coprime array parameters are set to $M=4$ and $N=5$, which yield the physical sensor positions

$$\{0,4,5,8,10,12,15,16\}.$$

(35)

The number of physical sensors is, therefore, $L=8$. The wavelength is normalized to unity, and the basic inter-element spacing is set to $d=\lambda /2$. The number of Monte Carlo trials is set to $N_{\mathrm{MC}}=100$ for RMSE evaluation.

The SNR in this paper is defined at the physical sensor-element level before difference-coarray processing, virtual covariance reconstruction, Hankel refinement, FBSS, or DOA estimation.

The source directions used in the simulations are listed as follows. For the two-source case, $\mathit{\theta }=[-{13}^{\circ },{13}^{\circ }]$. For the three-source case, $\mathit{\theta }=[-{30}^{\circ },0^{\circ },{21}^{\circ }]$. For the four-source case, $\mathit{\theta }=[-{36}^{\circ },-{12}^{\circ },{12}^{\circ },{36}^{\circ }]$. These settings are used to test the proposed method under different source numbers and angular separations.

The sources are modeled as coherent signals. The source covariance matrix is generated as

$${\mathbf{R}}_s=(1-\rho ){\mathbf{I}}_K+\rho {\mathbf{1}}_K{\mathbf{1}}_K^T,$$

(36)

where $\rho =0.999$ is the coherence coefficient. This setting creates a challenging coherent-source scenario for subspace-based DOA estimation.

The noise is spatially uncorrelated but nonuniform across sensors. Specifically, the noise covariance matrix is

$${\mathbf{R}}_n={\sigma }_0^2\mathrm{diag}({\eta }_1,{\eta }_2,\dots ,{\eta }_L),$$

(37)

where ${\sigma }_0^2$ is determined by the required SNR, and ${\eta }_{\mathcal{l}}$ denotes the relative noise-power factor of the ℓth sensor. In the simulations, ${\eta }_{\mathcal{l}}$ is generated from a log-normal distribution and normalized to have unit mean, i.e.,

$${\eta }_{\mathcal{l}}={\displaystyle \frac{exp\left(0.6u_{\mathcal{l}}\right)}{\frac{1}{L}{\sum }_{i=1}^{L}exp\left(0.6u_i\right)}},u_{\mathcal{l}}\sim \mathcal{N}(0,1).$$

(38)

This model produces unequal sensor noise powers while keeping the average noise level controlled by the specified SNR.

The compared methods are defined as follows. Root-MUSIC denotes the strengthened classical subspace baseline applied after difference-coarray lag averaging and basic virtual ULA covariance construction. This baseline does not use the proposed Toeplitz-PSD reconstruction, Hankel low-rank refinement, or FBSS. RNM denotes the reweighted atomic norm minimization method for coherent DOA estimation with coprime arrays. LowRank denotes the low-rank matrix recovery method for coherent DOA estimation with coprime arrays under nonuniform noise. DL-DOA denotes the learning-based coprime array interpolation baseline, where the incomplete virtual covariance matrix is reconstructed by a deep matrix iterative network before DOA estimation. The proposed method denotes the full structured pipeline consisting of difference-coarray lag averaging, Toeplitz-PSD reconstruction, Hankel low-rank refinement, FBSS, and MUSIC estimation.

The RMSE-versus-SNR experiment uses

$$\mathrm{SNR}\in \{-20,-15,-10,-5,0,5,10,15,20\}\mathrm{dB}.$$

The RMSE-versus-snapshots experiment uses

$$T\in \{50,500,1000,1500,2000,2500\}.$$

The experiments include RMSE versus SNR for different methods, RMSE versus snapshots from 50 to 2500 for different methods, RMSE versus SNR under different source numbers for the proposed method, RMSE versus snapshots from 50 to 2500 under different source numbers for the proposed method, two-source resolution capability, recovered MUSIC spectra for multi-target cases, and runtime comparison under varying Monte Carlo trial numbers.

4.2. RMSE Versus SNR for Different Methods

Figure 1 compares the RMSE performance of different methods when the number of snapshots is fixed at 500 and the SNR varies from $-20$ dB to 20 dB. The compared methods include Root-MUSIC, the proposed method, RNM, LowRank and DL-DOA.

Figure 1 evaluates the noise robustness of the compared methods under coherent-source and unknown nonuniform-noise conditions. As the SNR increases, the RMSE values of most methods decrease rapidly in the low-SNR region and then gradually approach a stable error floor in the medium- and high-SNR regions. This trend is consistent with the fact that a higher SNR provides a more reliable sample covariance matrix and improves the stability of subspace estimation.

Root-MUSIC exhibits a relatively large RMSE over most SNR values because it relies on the basic coarray-domain virtual covariance and does not include the proposed Toeplitz-PSD reconstruction, Hankel low-rank refinement, or FBSS. Under coherent sources, the signal covariance matrix becomes rank-deficient, and unknown nonuniform noise further distorts the diagonal covariance entries. As a result, the separation between the signal subspace and the noise subspace is weakened, leading to larger DOA estimation errors.

RNM and LowRank achieve better performance than Root-MUSIC because they introduce structured covariance recovery or low-rank constraints. However, their RMSE curves flatten in the medium- and high-SNR regions, indicating that residual reconstruction errors and model mismatch still limit their final accuracy. DL-DOA performs well in the low-SNR region and remains superior to the traditional reconstruction baselines in most cases, showing the benefit of learning-aided covariance interpolation. Nevertheless, after the SNR becomes higher, the proposed method achieves a slightly lower RMSE than DL-DOA.

The proposed method obtains the lowest RMSE over the medium- and high-SNR regions. This improvement is mainly due to the combined effect of nonzero-lag Toeplitz-PSD reconstruction, Hankel low-rank refinement, and forward–backward spatial smoothing. The Toeplitz reconstruction reduces the effect of unknown nonuniform noise by avoiding direct fitting of the zero-lag component, the Hankel refinement enhances the spectral consistency of the recovered covariance sequence, and FBSS improves coherent-source decorrelation. Therefore, the proposed method shows stronger robustness and a lower estimation error floor. The CRLB decreases monotonically with increasing SNR and provides the theoretical lower-bound reference for the tested scenario.

4.3. RMSE Versus Snapshots for Different Methods

Figure 2 shows the RMSE performance of different methods when the SNR is fixed at 0 dB and the number of snapshots varies from 50 to 2500.

As shown in Figure 2, the RMSE of all methods generally decreases as the number of snapshots increases. This is because more snapshots lead to a more accurate sample covariance estimate, which improves the reliability of the subsequent covariance reconstruction and subspace decomposition. The most obvious improvement appears when the number of snapshots increases from 50 to 500. After that, the curves become flatter, which means that the performance is gradually dominated by the intrinsic reconstruction capability of each method rather than by sample variance alone.

Root-MUSIC has the largest RMSE and improves slowly with increasing snapshots. This result indicates that simply increasing the number of snapshots cannot fully overcome the rank deficiency caused by coherent sources or the covariance distortion caused by unknown nonuniform noise. RNM and LowRank show better performance than Root-MUSIC because they exploit structured recovery, but they still remain clearly above the proposed method.

DL-DOA achieves a competitive RMSE and remains close to the proposed method. However, it is still slightly inferior to the proposed method over the tested snapshot range. This indicates that the learning-based interpolation strategy can improve covariance recovery, but its online performance may still depend on training conditions, network generalization, and the consistency between training and testing scenarios.

The proposed method achieves the lowest RMSE among all compared methods. The advantage becomes clearer when sufficient snapshots are available, because the redundancy-aware lag averaging and Toeplitz-PSD reconstruction can make better use of the improved covariance statistics. Meanwhile, the Hankel low-rank refinement and FBSS further stabilize the recovered virtual covariance matrix. Therefore, the proposed method maintains better estimation accuracy under different sample-support conditions. The CRLB decreases as the number of snapshots increases, which agrees with the theoretical expectation that more independent observations reduce the lower bound of estimation error.

4.4. RMSE Versus SNR Under Different Source Numbers

Figure 3 presents the RMSE versus SNR of the proposed method for different source numbers, where $K=2$, $K=3$, and $K=4$. The number of snapshots is fixed at 500.

Figure 3 shows that the RMSE decreases as the SNR increases for different source numbers. In the very low-SNR region, especially at $-20$ dB and $-15$ dB, the estimation error is relatively large because the covariance estimate is strongly affected by noise. When the SNR increases to $-10$ dB and above, the RMSE drops rapidly and then gradually reaches a stable level.

The results also show that the estimation difficulty increases as the number of sources grows. In general, the $K=2$ case achieves the lowest RMSE, while the $K=3$ and $K=4$ cases have larger errors. This is reasonable because more sources lead to a denser signal subspace, stronger inter-source interaction, and more difficult spectral peak separation. For $K=2$, the spatial structure is simpler and the recovered virtual covariance can support clearer MUSIC peak separation.

For $K=3$ and $K=4$, the RMSE curves become flatter in the medium- and high-SNR regions. This indicates that when the number of coherent sources increases, the residual reconstruction error and source-separation difficulty become more dominant. Nevertheless, the proposed method still maintains stable performance for all tested source numbers, which verifies the effectiveness of the Toeplitz–Hankel–FBSS processing chain in multi-source coherent scenarios.

4.5. RMSE Versus Snapshots Under Different Source Numbers

Figure 4 shows the RMSE versus snapshots of the proposed method for different source numbers with the SNR fixed at 0 dB.

Figure 4 further evaluates the influence of sample support on the proposed method under different source numbers. For all tested cases, increasing the number of snapshots improves the RMSE performance. The improvement is especially significant when the number of snapshots increases from 50 to 500, because the sample covariance matrix becomes much more stable in this range.

The $K=2$ case shows the lowest RMSE over the whole snapshot range. Its RMSE decreases rapidly at the beginning and then gradually approaches a low error floor. This demonstrates that the proposed method can effectively exploit additional snapshots when the number of sources is small and the spectral peaks are relatively easy to separate.

For $K=3$ and $K=4$, the RMSE values are higher than those of $K=2$. The $K=4$ case has the largest RMSE because the covariance sequence contains more spectral components and the corresponding MUSIC spectrum is more crowded. In this situation, Toeplitz reconstruction and Hankel refinement need to preserve more source components simultaneously, which increases the reconstruction difficulty.

Although the RMSE curves for $K=3$ and $K=4$ become relatively flat after 500 snapshots, they do not show severe instability. This indicates that the proposed method remains robust under increased source multiplicity. The confidence intervals also become narrow in the medium- and large-snapshot regions, showing that the Monte Carlo results are stable and repeatable.

4.6. Two-Source Resolution Capability Under Different Angular Separations

To further evaluate the angular resolution capability of the proposed method, a two-source experiment with different target separations is conducted. Two coherent sources are symmetrically located around broadside with separations $\Delta \theta \in \{8^{\circ },{14}^{\circ },{20}^{\circ },{26}^{\circ }\}$, corresponding to the DOA pairs $[-4^{\circ },4^{\circ }]$, $[-7^{\circ },7^{\circ }]$, $[-{10}^{\circ },{10}^{\circ }]$, and $[-{13}^{\circ },{13}^{\circ }]$, respectively. This experiment verifies whether the proposed Toeplitz–Hankel–FBSS reconstruction framework can maintain clear spectral separation when two coherent sources are located at different angular distances.

Figure 5 shows the normalized MUSIC spectra for the four tested source separations. It can be observed that the proposed method produces two distinguishable dominant peaks in all cases. When the source separation is relatively small, such as $\Delta \theta =8^{\circ }$, the two peaks are closer to each other, but they are still clearly separated around the true DOA positions. As the separation increases from $8^{\circ }$ to ${26}^{\circ }$, the spectral peaks become more isolated, and the valley between the two peaks becomes wider. This result demonstrates that the proposed method has good two-source resolution capability under coherent-source and unknown nonuniform-noise conditions.

Figure 6, Figure 7, Figure 8 and Figure 9 provide detailed views of the MUSIC spectra and the estimated DOAs for each source separation. For $\Delta \theta =8^{\circ }$, the estimated DOAs are approximately $[-4.{29}^{\circ },4.{32}^{\circ }]$, which are close to the true directions $[-4^{\circ },4^{\circ }]$. This indicates that the proposed method can still resolve two relatively close coherent sources. For $\Delta \theta ={14}^{\circ }$, $\Delta \theta ={20}^{\circ }$, and $\Delta \theta ={26}^{\circ }$, the estimated DOAs remain very close to the true DOAs, and the spectral peaks become progressively more separated.

The reason for this behavior is that larger angular separation reduces the overlap between the spatial signatures of the two sources, making the signal subspace easier to distinguish from the noise subspace. More importantly, the proposed method first reconstructs a physically consistent virtual covariance matrix through Toeplitz-positive semidefinite recovery, then enhances the covariance sequence using Hankel low-rank restoration, and finally applies FBSS to improve coherent-source decorrelation. Therefore, the resulting MUSIC spectrum preserves clear peak locations and stable resolution performance for different two-source separations.

4.7. Recovered MUSIC Spectra Under Different Source Numbers

Different from the two-source angular-separation experiment in Section 4.6, this subsection focuses on the spectral behavior of the proposed method when the number of coherent sources increases. Figure 10, Figure 11 and Figure 12 present the recovered MUSIC spectra of the proposed method for $K=2$, $K=3$, and $K=4$, respectively. These figures are used to examine whether the proposed structured covariance recovery can still produce clear and identifiable spectral peaks as the multi-source scenario becomes more crowded.

For $K=2$, the spectrum exhibits sharp dominant peaks and relatively low sidelobes, which indicates that the signal and noise subspaces are well separated after the proposed reconstruction stages. The case of $K=3$ is emphasized because it is more challenging and more representative than the two-source case. When three coherent sources are present, the signal subspace dimension increases, the spectral peaks are closer to mutual interference, and the covariance reconstruction must preserve more source components simultaneously. Therefore, $K=3$ is an appropriate intermediate case for verifying the resolution capability and robustness of the proposed method.

Figure 11 shows that the recovered MUSIC spectrum still provides three distinguishable peaks around the true DOA positions. This finding indicates that the proposed Toeplitz–Hankel–FBSS processing chain can recover a high-quality virtual covariance matrix and maintain reliable subspace separation in a multi-source coherent scenario. For $K=4$, the spectral scene becomes even denser and the peak-separation task is more difficult, but the main peaks remain identifiable. These spectrum plots provide visual evidence that the proposed method improves not only numerical RMSE performance but also practical spectral separability.

4.8. Complexity and Runtime Platform

To complement the numerical performance comparison, the computational burden of the compared methods is analyzed according to their dominant operations. Let $L_v$ denote the length of the virtual ULA, G denote the number of angular grid points, $I_{\mathrm{R}}$ denote the number of RNM reweighting iterations, $I_{\mathrm{L}}$ denote the number of LowRank recovery iterations, and $I_{\mathrm{P}}$ denote the number of iterations used in the proposed structured reconstruction stage.

For Root-MUSIC, the main computational operations are eigendecomposition of the virtual covariance matrix and polynomial rooting. Therefore, its dominant complexity can be approximately written as

$${\mathcal{C}}_{\mathrm{Root}-\mathrm{MUSIC}}=\mathcal{O}\left(L_v^3\right)+\mathcal{O}\left(L_v^3\right)=\mathcal{O}\left(L_v^3\right).$$

(39)

This method requires the lowest computational cost among the compared approaches because it does not involve convex optimization or iterative covariance reconstruction.

For RNM, the dominant cost comes from repeatedly solving the reweighted atomic norm minimization problem. In each iteration, a semidefinite programming problem with a block positive semidefinite constraint is solved, and the weighting matrix is updated for the next iteration. If an interior-point-type solver is used, the complexity can be approximately expressed as

$${\mathcal{C}}_{\mathrm{RNM}}\approx \mathcal{O}\left(I_{\mathrm{R}}{\left(2L_v\right)}^6\right)+\mathcal{O}(L_v^3+G{L}_v^2).$$

(40)

The first term corresponds to the iterative semidefinite reconstruction process, while the second term corresponds to eigendecomposition and MUSIC spectral search after covariance recovery. Since RNM repeatedly solves a weighted atomic norm minimization problem, its computational burden is significantly higher than that of Root-MUSIC.

For the LowRank method, the rank minimization problem is relaxed into a nuclear-norm or trace-minimization problem under positive semidefinite and data-fitting constraints. The dominant cost is, therefore, associated with solving a low-rank matrix recovery problem. Its approximate computational complexity can be written as

$${\mathcal{C}}_{\mathrm{LowRank}}\approx \mathcal{O}\left(I_{\mathrm{L}}{L}_v^6\right)+\mathcal{O}(L_v^3+G{L}_v^2).$$

(41)

Compared with RNM, LowRank usually has a lower computational burden because it does not require repeated reweighting of the atomic norm. However, it still involves convex low-rank matrix recovery, so its complexity remains much higher than that of Root-MUSIC.

For DL-DOA, the offline training stage is not included in the online runtime comparison. During online testing, the main computational cost comes from covariance preprocessing, neural-network inference, and the final subspace-based DOA estimation. Therefore, its complexity can be approximately represented as

$${\mathcal{C}}_{\mathrm{DL}-\mathrm{DOA}}=\mathcal{O}\left({\mathcal{C}}_{\mathrm{net}}\right)+\mathcal{O}(L_v^3+G{L}_v^2),$$

(42)

where ${\mathcal{C}}_{\mathrm{net}}$ denotes the inference complexity of the trained neural network. Since ${\mathcal{C}}_{\mathrm{net}}$ depends on the network architecture, number of layers, and hardware acceleration platform, the online runtime of DL-DOA is not directly comparable unless the same trained model and computing environment are used.

For the proposed method, the main computational burden comes from two structured reconstruction stages, namely Toeplitz positive semidefinite covariance reconstruction and Hankel low-rank refinement. After these reconstruction stages, FBSS and MUSIC estimation are performed. Therefore, the approximate complexity can be expressed as

$${\mathcal{C}}_{\mathrm{Proposed}}\approx \mathcal{O}\left(I_{\mathrm{P}}{L}_v^6\right)+\mathcal{O}\left(I_{\mathrm{H}}{L}_{\mathrm{H}}^6\right)+\mathcal{O}(L_v^3+G{L}_v^2),$$

(43)

where $I_{\mathrm{H}}$ denotes the iteration number of the Hankel low-rank refinement stage, and $L_{\mathrm{H}}$ denotes the effective dimension of the Hankel matrix. The first term represents Toeplitz-PSD reconstruction, the second term represents Hankel low-rank refinement, and the last term represents eigendecomposition and MUSIC spectral search.

From the above analysis, Root-MUSIC has the lowest complexity because it only requires eigendecomposition and polynomial rooting. LowRank and RNM have higher complexity because both require convex matrix recovery, and RNM is usually more computationally demanding than LowRank because of its reweighted iterative atomic norm minimization. DL-DOA can be efficient in the online stage once the network has been trained, but its inference time depends strongly on the network scale and GPU implementation. The proposed method introduces both Toeplitz-PSD reconstruction and Hankel low-rank refinement; therefore, its computational burden is higher than that of Root-MUSIC and LowRank and is comparable to that of RNM in the tested implementation.

All runtime experiments were conducted on the same computing platform equipped with an Intel Core i7-13700KF processor (Intel Corporation, Santa Clara, CA, USA), an NVIDIA GeForce RTX 4070 SUPER GPU (NVIDIA Corporation, Santa Clara, CA, USA), and 16 GB of RAM.

4.9. Runtime Comparison

DL-DOA is not listed in the runtime table because its online inference time depends on the trained network architecture and GPU acceleration platform, while its offline training time is not directly comparable with the iterative reconstruction time of the other methods.

As shown in

Table 1. Runtime comparison of different methods under varying Monte Carlo trial numbers.

Monte Carlo	Root-MUSIC	Proposed	RNM	Low Rank
10	0.4836 s	37.1541 s	33.6921 s	5.2067 s
20	0.9653 s	74.1906 s	67.1532 s	10.8214 s
50	2.4098 s	185.9744 s	170.3804 s	26.4985 s
100	4.8365 s	371.6014 s	338.0092 s	54.2669 s

, Root-MUSIC requires the least runtime. LowRank is slower than Root-MUSIC because of its low-rank recovery stage, but it is much faster than RNM and the proposed method. RNM and the proposed method have comparable runtime, since both rely on iterative structured covariance reconstruction. The proposed method is slightly slower than RNM in terms of the reported mean runtime, but the difference is small compared with the gap between these two methods and LowRank. Thus, the results should be interpreted as showing comparable runtimes for RNM and the proposed method, while LowRank is faster.

The proposed method trades this additional computational cost for improved robustness and estimation accuracy under coherent sources and unknown nonuniform noise. Therefore, it is more suitable for applications where high estimation reliability is more important than strict real-time processing.

4.10. Discussion

The simulation results consistently show that the proposed method can exploit the enlarged virtual aperture of the coprime array while reducing the adverse effects of unknown nonuniform noise and source coherence. The lag-domain Toeplitz reconstruction improves covariance consistency because it relies on reliable observed lags and enforces a valid covariance structure. The subsequent Hankel restoration further enhances robustness by refining the recovered covariance sequence under coherent-source conditions. After FBSS, the final virtual covariance exhibits improved rank behavior, which leads to more stable MUSIC spectra and better peak separation.

Taken together, these results indicate that the main strength of the proposed method lies not in a single isolated operation but in the cooperation among several structured stages. Lag averaging, off-zero-lag covariance fitting, Toeplitz-PSD reconstruction, Hankel refinement, and FBSS each address a different source of degradation. Their combined effect yields a more reliable virtual covariance matrix for DOA estimation under challenging sensing conditions.

5. Conclusions

This paper presented a structured DOA estimation framework for coprime arrays in the presence of coherent sources and unknown nonuniform noise. The proposed method first maps the physical-array covariance into the difference-coarray domain through lag averaging. A lag-domain Hermitian Toeplitz positive semidefinite reconstruction model is then used to recover a structured virtual covariance matrix from observed nonzero lags, thereby reducing the influence of unknown nonuniform diagonal noise and avoiding artificial zero filling of missing lags. A Hankel low-rank restoration stage is further introduced to enhance the spectral consistency of the recovered covariance sequence, followed by FBSS and MUSIC estimation.

The simulation results show that the proposed method achieves lower RMSE and clearer spectral separation than Root-MUSIC, DL-DOA, RNM, and LowRank baselines under coherent-source and unknown nonuniform-noise conditions. The RMSE curves with respect to SNR and snapshots, the source-number experiments, the two-source angular-separation test, and the recovered MUSIC spectra jointly verify the robustness and resolution capability of the proposed Toeplitz–Hankel–FBSS processing chain.

The proposed method is suitable for sparse-array DOA estimation scenarios where high angular resolution is required under coherent-source interference and sensor-dependent noise, such as radar, sonar, passive localization, wireless sensing, and acoustic source localization. Its main limitation is the increased computational cost caused by the structured reconstruction stages and the need for appropriate parameter selection. Future work will focus on reducing the computational burden, extending the method to two-dimensional arrays, incorporating source-number estimation, and developing adaptive or learning-aided weighting strategies for time-varying noise environments.