Deep Learning-Driven Atomic Norm Optimization for Accurate Downlink Channel Estimation in FDD Systems

Abstract

In this paper, we propose a downlink (DL) channel estimation scheme for frequency-division duplex (FDD) multi-antenna orthogonal frequency-division multiplexing (OFDM) systems, leveraging atomic norm minimization (ANM) and deep neural networks (DNN). Unlike time-division duplex (TDD) systems, where uplink (UL) and DL channels are reciprocal, FDD systems do not share this reciprocity, leading to increased channel training overhead. However, both theoretical analyses and empirical evidence reveal that key channel characteristics—such as angles of arrival and departure, path delays, and the number of propagation paths—exhibit partial reciprocity between UL and DL. Building on this insight, we design a DL channel estimation scheme that exploits frequency-independent UL parameters along with estimated DL channel gains. Our method integrates ANM with DNN to enhance estimation accuracy and efficiency. Specifically, ANM formulates the estimation problem while avoiding the off-grid errors inherent in traditional grid-based methods. To further mitigate performance degradation in clustered-path channels and reduce computational complexity, we introduce a DNN-based architecture that predicts channel parameters. The DNN captures hidden relationships between received pilot signals and frequency-independent channel parameters, enabling accurate estimation with linear time complexity. During training, ANM assists in serving users, ensuring reliable performance. Once the DNN is fully trained, it takes over to balance quality of service (QoS) and latency, providing an efficient and accurate solution for DL channel estimation in FDD-OFDM systems.

1. Introduction

One of the fundamental challenges in modern wireless communication systems is channel estimation, as most beamforming and user scheduling strategies in massive multiple-input multiple-output (MIMO) systems rely on accurate channel state information (CSI) to achieve optimal performance. In time-division duplexing (TDD) mode, this issue is mitigated since uplink (UL) CSI can be directly used as downlink (DL) CSI due to channel reciprocity. However, in frequency-division duplexing (FDD) mode, channel reciprocity generally does not hold because UL and DL CSI differ significantly. A straightforward approach to address this problem is to introduce a feedback mechanism: the transmitter sends pilot signals, the receiver estimates DL CSI, and the estimated CSI is fed back to the transmitter. In most wireless systems, an orthogonal set of pilot signals is required for channel estimation. While this additional training overhead is manageable for systems with a small number of antennas, it becomes prohibitive in massive MIMO systems employing large-scale antenna arrays [1], as the vast number of orthogonal pilot signals required would consume excessive resources within the limited channel coherence time and frequency [2].

To tackle the challenges associated with FDD systems, various strategies have been explored. One common approach is to extract DL CSI from UL CSI by assuming partial reciprocity, such as similarity between the UL angle-of-arrival (AoA) and DL angle-of-departure (AoD), or assuming they are nearly identical [3]. However, fully eliminating feedback for DL CSI estimation in FDD mode typically requires the assumption that UL and DL channel gains are highly correlated or nearly identical [4]. These assumptions, however, lack rigorous theoretical justification and experimental validation [5]. In contrast, studies such as [6] have shown that UL and DL cluster gains generally differ.

Leveraging angular reciprocity, CSI estimation techniques can further exploit the sparse nature of massive MIMO channels to reduce, though not entirely eliminate, DL feedback overhead due to frequency-dependent UL/DL channel gains. The assumption that propagation between the base station (BS) and user equipment (UE) follows a limited number of dominant paths is widely accepted, as recent works such as [7] have proved that the reciprocity holds in both sparse and rich scatter channels. Frequency-independent parameters can be estimated from UL pilots and used for DL CSI estimation, requiring only minimal feedback. A widely adopted method in this context involve quantizing the spatial domain using predefined grids or codebooks, reducing feedback to grid point indices for AoA and AoD estimation. Examples include modular quantization [8] and beam pattern training [9]. On the other hand, sensing-aided channel path estimation, such as [10,11], is of more favor as it recovers the channel paths by extracting the sensing parameters, which avoids tedious beam training within the quantized angular domain.

Despite their effectiveness, these methods have inherent limitations. Grid-based quantization methods suffer from off-grid errors. Increasing the grid density improves accuracy but significantly raises computational complexity, leading to a trade-off between precision and efficiency. On the other hand, sensing-based approaches require DL channel sparsity, which is often inferred from prior knowledge or UL feedback. The former can be unreliable in dynamic environments, while the latter introduces additional training overhead.

To overcome these challenges, we propose a low-complexity, high-accuracy channel estimation framework for FDD massive MIMO systems based on atomic norm minimization (ANM). Our approach focuses on the dominant propagation path and defines an atomic set encompassing all potential paths. The channel parameters of the most significant path are extracted by solving an ANM problem. Unlike existing ANM-based channel estimation methods, e.g., [12,13], that rely on a single semidefinite optimization, our approach explicitly estimates and extracts real multi-path channels, making it well-suited for spatial diversity.

Furthermore, we observe that ANM can be interpreted as a mapping function between received signals and channel paths, which allows us to replace it with a deep neural network (DNN) to further enhance efficiency and accuracy. During DNN training, we mitigate potential gradient descent stalls by introducing a simplified weight compression algorithm. Conventional weight compression methods require retraining for each compression parameter and iterative loss function evaluation, resulting in high complexity. In contrast, our approach selects the compression parameter that maximizes the number of activated neurons, avoiding iterative loss function searches while maintaining comparable performance.

By integrating ANM with a DNN, we propose a comprehensive DL channel estimation strategy for FDD massive MIMO systems. The system initially employs ANM during DNN training and subsequently switches to the trained DNN for real-time CSI estimation, significantly reducing computational complexity and improving estimation accuracy.

The major contributions of this paper are summarized as follows:

• We propose an ANM-based channel estimation framework for FDD massive MIMO OFDM systems. The method estimates frequency-independent channel parameters along with path gains and formulates an ANM problem solvable via off-the-shelf solvers. Unlike grid-based methods, our approach avoids off-grid errors and reduces complexity without exhaustive grid searches.
• We introduce a DNN-assisted approach that enhances ANM-based estimation by reducing computational overhead and improving accuracy. Additionally, we propose a simplified weight compression scheme to prevent training stagnation. The DNN, trained using parameters obtained from exhaustive search, enables low-latency channel prediction post-training.

The rest of this paper is organized as follows: Section 2 formulates the channel estimation problem. Section 3 derives the ANM problem. Section 4 uses DNN to solve the ANM problem. Section 5 presents numerical results. Section 6 concludes the paper.

Notations: Throughout this paper, we use the following notations: bold uppercase letters (e.g., $\mathbf{A}$) denote matrices, bold lowercase letters (e.g., $\mathbf{a}$) denote vectors, $\mathcal{A}$ represents a set, and $\alpha$ denotes a scalar. The pseudo-inverse, conjugate-transpose (Hermitian), conjugate, and transpose of matrix $\mathbf{A}$ are denoted as ${\mathbf{A}}^{†}$, ${\mathbf{A}}^H$, ${\mathbf{A}}^{*}$, and ${\mathbf{A}}^T$, respectively. The Kronecker product and Hadamard product of matrices $\mathbf{A}$ and $\mathbf{B}$ are represented as $\mathbf{A}\otimes \mathbf{B}$ and $\mathbf{A}\odot \mathbf{B}$, respectively.

2. System Model and Problem Formulation

2.1. Channel Model

In this section, we consider an OFDM-based massive MIMO system operating in FDD mode. The system consists of a base station (BS) equipped with M antennas and S subcarriers, serving multiple pieces of single-antenna user equipment (UE), which is a typical configuration for lower hardware complexity, power consumption and cost [14]. It is well known that the channel in massive MIMO systems is typically characterized by a limited number of propagation paths [4]. Let $h_m^U\left(s\right)$ denote the uplink (UL) channel between UE and the mth BS antenna on subcarrier s, given by

$$h_m^U\left(s\right)=\sum _{l=1}^{L^U}{g}_l^U{e}^{j2\pi \left({\mathsf{\Phi }}_{l,s}^U+{\mathsf{\Theta }}_{l,s}^U\right)},$$

(1)

where $L^U$ represents the total number of propagation paths, $g_l^U$ is the gain of the lth path, ${\mathsf{\Phi }}_{l,s}^U$ accounts for the phase difference due to the path delay, and ${\mathsf{\Theta }}_{l,m}^U$ represents the phase difference caused by the relative arrival time variation of the path. For simplicity, we assume that both M and S are even integers, such that $m=-M/2,\dots ,-1,0,1,\dots ,M/2$ and $s=-S/2,\dots ,-1,0,1,\dots ,S/2$.

The phase delay ${\mathsf{\Phi }}_{l,s}^U$ is related to the difference between the UL and DL central frequencies, denoted as $\Delta F$, with an inter-subcarrier spacing of $\Delta f$ (i.e., $\Delta F=S\Delta f$). Defining a delay ${\tau }_l^U$ for path l, which satisfies $0\le {\tau }_l^U<1/\Delta f$, we express ${\mathsf{\Phi }}_{l,s}^U$ as

$${\mathsf{\Phi }}_{l,s}^U=s\Delta f{\tau }_l^U,$$

(2)

Similarly, the antenna element delay ${\mathsf{\Theta }}_{l,s}^U$ is determined by the inter-element spacing of the antenna array. Assuming a uniform linear array (ULA) at the BS, ${\mathsf{\Theta }}_{l,s}^U$ can be written as

$${\mathsf{\Theta }}_{l,s}^U=m{\displaystyle \frac{d}{{\lambda }^U}}sin{\theta }_l^U,$$

(3)

where d is the inter-element spacing, ${\lambda }^U$ is the wavelength, and $sin{\theta }_l^U$ represents the signal arrival angle. For other array configurations, such as co-prime [15] and nested arrays [16], similar calculations apply, and their distinct properties warrant further investigation in future work.

Substituting (2) and (3) into (1), we obtain

$$h_m^U\left(s\right)=\sum _{l=1}^{L^U}{g}_l^U{e}^{j2\pi (s\Delta f{\tau }_l^U+m{\displaystyle \frac{d}{{\lambda }^U}}sin{\theta }_l^U)},$$

(4)

To represent the UL channel across all antennas, we stack $h_m^U\left(s\right)$ for $s=1,\dots ,S$ and $m=1,\dots ,M$ into the vector ${\mathit{h}}^U$:

$${\mathit{h}}^U=\sum _{l=1}^{L^U}{g}_l^U\mathit{p}\left({\tau }_l^U\right)\otimes \mathit{q}\left({\theta }_l^U\right),$$

(5)

where

$$\mathit{p}\left({\tau }_l^U\right)={\left[e^{-j2\pi \Delta f{\tau }_l^U},\dots ,e^{-j2\pi S\Delta f{\tau }_l^U}\right]}^H,$$

(6)

$$\mathit{q}\left({\theta }_l^U\right)={\left[e^{-j2\pi {\displaystyle \frac{d}{{\lambda }^U}}sin{\theta }_l^U},\dots ,e^{-j2\pi {\displaystyle \frac{d}{{\lambda }^U}}Msin{\theta }_l^U}\right]}^H,$$

(7)

Since FDD systems do not exhibit full reciprocity between DL and UL channels, additional estimation and feedback steps are required. However, prior work [17] has shown that sparsity reciprocity and angular reciprocity hold, meaning that the number of propagation paths and the angles of departure (AoD) in the DL closely match those of the angles of arrival (AoA) in the UL. Letting $L^D$ denote the number of DL propagation paths, with corresponding delays ${\tau }^D$ and angles ${\theta }^D$, we approximate $L^D\approx L^U$, ${\tau }^D\approx {\tau }^U$, and ${\theta }^D\approx {\theta }^U$.

From (4), we see that the parameters $L^D,{\tau }^D,{\theta }^D$—excluding the DL channel gain $g^D$—can be inferred from the UL channel. Therefore, estimating $g^D$ requires an additional step. For notation simplicity, we omit frequency-dependent superscripts and use L, $\tau$, and $\theta$ to represent their UL and DL counterparts in the remainder of this paper.

2.2. Frequency-Independent Parameter Extraction

Let $t\left(s\right)$ denote the pilot signal transmitted on subcarrier s. Given the UL channel model (4), the received signal $y_m^U\left(s\right)$ at the BS antenna m is given by

$$y_m^U\left(s\right)=h_m^U\left(s\right)t\left(s\right)+n_m^U\left(s\right),$$

(8)

where $n_m^U\left(s\right)\sim \mathcal{CN}(0,{\sigma }^2)$ represents Gaussian noise. Normalizing the pilot power, we set $\left|t\left(s\right)\right|=1/\sqrt{M}$.

For DL channel estimation, both $g_l^D$ and frequency-independent parameters ${\tau }_l,{\theta }_l$ must be determined. Stacking the received pilot signals across subcarriers and antennas, we obtain

$${\mathit{y}}^U=\sum _{l=1}^{L^U}{g}_l^U\left(\mathit{p}\left({\tau }_l^U\right)\otimes \mathit{q}\left({\theta }_l^U\right)\right)\mathit{t}+\mathit{n},$$

(9)

where $\mathit{t}=\left[t\left(1\right),\dots ,t\left(S\right)\right]$. Defining the joint function

$$\mathit{u}(\tau ,\theta )=\mathit{p}\left(\tau \right)\otimes \mathit{q}\left(\theta \right),$$

(10)

we rewrite (8) as

$${\mathit{y}}^U=\sum _{l=1}^{L^U}{g}_l^U\mathit{u}({\tau }_l^U,{\theta }_l^U)\mathit{t}+\mathit{n},$$

(11)

This indicates that for any received $\mathit{y}$, we need to recover $\left\{{\tau }_l^U,{\theta }_l^U\right\}$ (namely $\left\{{\tau }_l,{\theta }_l\right\}$) for $l=1,\dots ,L$, without any prior knowledge of the number of propagation paths L. In this paper, this is achieved in two steps: first, estimating $\left\{L,\tau ,\theta \right\}$, followed by determining the DL gain $g_l^D$ with an extra feedback mechanism.

2.3. Estimation of DL Channel

Given the same channel model and leveraging angular reciprocity, the DL channel can be expressed as

$$h_m^D\left(s\right)=\sum _{l=1}^{L^U}{g}_l^D{e}^{j2\pi \left(s\Delta f{\tau }_l^U+m{\displaystyle \frac{d}{{\lambda }^D}}sin{\theta }_l^U\right)},$$

(12)

The frequency-independent parameters ${\tau }_l$ and ${\theta }_l$ are derived from the UL channel, whereas the DL path gain $g_l^D$ in (12) requires separate estimation and feedback. The DL channel gain $g_l^D$ in (12) is obtained from the received signal at the UE side. In a multi-subcarrier system, pilot signals are inserted at regular subcarrier intervals. We assume that the channel gains for adjacent subcarriers remain nearly identical, introducing minimal error [18].

To estimate $g_l^D$, the BS transmits pilots and informs the UE of the estimated $\left\{{\tau }_l,{\theta }_l\right\}$. These pilots are beamformed towards ${\theta }_l$ to maximize the signal-to-noise ratio (SNR), requiring only a single OFDM symbol frequency division multiplexed over all ${\theta }_l$. Let us assume that a pilot signal is inserted at every K subcarrier, and the total number of subcarriers S is divisible by K. The subcarriers containing pilot signals are indexed as $s_1,\dots ,s_I,I=S/K$.

To achieve maximum SNR, each pilot signal ${\mathit{t}}_i$ on subcarrier $s_i$ is beamformed along one of the L propagation paths. In a sparse massive MIMO channel, the number of paths L is typically much smaller than the number of subcarriers (i.e., $L\ll S$), ensuring that the I subcarriers sufficiently cover all L propagation directions when the gap K is small enough. The beamformed pilot signal ${\mathit{t}}_i$ is given by

$${\mathit{t}}_i=\mathit{q}\left({\theta }_{\overline{i}}\right),$$

(13)

where $\overline{i}=i$mod$L$. The received signal at the UE on subcarrier $s_i$ is expressed as

$$y^D\left(s_i\right)=\sum _{l=1}^L{g}_l^D{e}^{j2\pi \left(\Delta F+s_i\Delta f\right){\tau }_l}{\mathit{q}}^H\left({\theta }_l\right){\mathit{t}}_i+n^D,$$

(14)

for $i=1,\dots ,I$. Given the pilot signal structure ${\mathit{s}}_l$, the UE is informed of the estimated ${\tau }_l$ and ${\theta }_l$ and stacks all received signals into

$${\mathit{y}}^D=\mathit{Q}{\mathit{g}}^D+\mathit{n},$$

(15)

where ${\mathit{y}}^D\in {\mathbb{C}}^{K\times 1}$, ${\mathit{g}}^D\in {\mathbb{C}}^{L\times 1}$ and $\mathit{Q}\in {\mathbb{C}}^{K\times L}$ are defined as

$${\mathit{y}}^D=\left[\begin{array}{c}{y}^D\left(n_1\right)\\ ⋮\\ y^D\left(n_K\right)\end{array}\right],$$

(16)

$${\mathit{g}}^D=\left[\begin{array}{c}{g}^D\left(n_1\right)\\ ⋮\\ g^D\left(n_L\right)\end{array}\right],$$

(17)

$$\mathit{Q}=\left[{\mathit{q}}^D\left({\theta }_1\right)\dots {\mathit{q}}^D\left({\theta }_L\right)\right],$$

(18)

where ${\mathit{q}}^D\left({\theta }_l\right)$ is given by

$${\mathit{q}}^D\left({\theta }_l\right)=e^{j2\pi \left(\Delta F+n_i\Delta f\right){\tau }_l}{\left[\mathit{q}\left({\theta }_1\right)\dots \mathit{q}\left({\theta }_K\right)\right]}^H{\mathit{t}}_l.$$

(19)

To estimate ${\mathit{g}}^D$ from (15), we apply the pseudo-inverse estimator to obtain a least squares (LS) estimate:

$${\mathit{g}}^D={\mathit{Q}}^{†}{\mathit{y}}^D.$$

(20)

So far, with (17) and (20), the DL path gains $g^D\left(n_l\right),l=1,\dots ,L$ can be obtained.

2.4. Necessity of Angular Reciprocity

As discussed in Section 1, in an FDD system, leveraging angular reciprocity significantly reduces channel training overhead. Beyond reducing overhead, we also analyze the accuracy implications of using angular reciprocity.

Our proposed approach first estimates the directional angles ${\theta }_l$, the number of paths L, and the path delays ${\tau }_l$, before separately estimating the DL channel gains $g_l^D$ independently via extra feedback. An alternative method would first estimate ${\theta }_l$ and L using traditional DOA estimation methods, then determine ${\tau }_l$ and $g_l^D$ through feedback. We compare these two approaches in terms of complexity and accuracy.

Complexity: The computational complexity of the proposed grid-free scheme will be analyzed in Section 3.3. In contrast, we evaluate the complexity of conventional DOA estimation methods as a benchmark. Specifically, we consider the MUSIC [19] and ESPRIT [20] algorithms. For MUSIC, the complexity per propagation path l to estimate ${\theta }_l$ is $\mathcal{O}\left(M^2{S}^2+M^3{S}^3+U_m{V}_m\left(M^2{S}^2-1\right)\right)$, leading to an overall complexity of $\mathcal{O}\left(L{M}^2{S}^2+L{M}^3{S}^3+L{U}_m{V}_m\left(M^2{S}^2-1\right)\right)$, where $U_m$ represents the step size. Similarly, the complexity for ESPRIT is $\mathcal{O}\left(M^2{S}^2+M^3{S}^3+2\left(M-1\right)S+2\left(S-1\right)M+6\right)$ per path and $\mathcal{O}\left(L{M}^2{S}^2+L{M}^3{S}^3+2L\left(M-1\right)S+2L\left(S-1\right)M+6L\right)$ in total. As will be demonstrated later, despite only estimating the DOA ${\theta }_l$, both algorithms exhibit higher complexity than the proposed grid-free method, which jointly estimates ${\theta }_l$ and path delay ${\tau }_l$.

Accuracy: In mmWave channels, it is widely recognized that propagation paths exhibit a very narrow angular spread, a fact confirmed by extensive practical measurements [21]. For example, measurements at 28 GHz and 73 GHz report a −10 dB average root-mean-square (RMS) lobe angular spread of $6.8^{\circ }$ in azimuth and $6.7^{\circ }$ in elevation for 28 GHz non-line-of-sight (NLOS) scenarios [22]. At 73 GHz in NLOS conditions, these values further reduce to $3.7^{\circ }$ and $2.2^{\circ }$, respectively. In a multipath channel model, such a small angular spread demands an extremely high-resolution algorithm to distinguish paths, despite their limited number. Moreover, practical measurements indicate that path delays are generally minimal. For instance, tests in both 28 GHz and 73 GHz environments reveal that, in line-of-sight (LOS) cases, most path delays are below 2 ns, while in NLOS conditions, they extend to approximately 60 ns and 20 ns, respectively [23]. These observations highlight a critical issue: in an alternative approach where the downlink (DL) channel is obtained by first estimating the direction of arrival (DOA), followed by path delay and gain estimation, errors in DOA estimation propagate to subsequent parameters. This occurs because the phase of the channel vector depends on both $sin\theta$ and $\tau$, as shown in (1). In some cases, DOA estimation inaccuracies may completely obscure the true path delay. To mitigate such error accumulation, the proposed scheme jointly estimates $\theta$ and $\tau$. The resulting atomic norm minimization (ANM) formulation generates a 2D power spectrum, where each parameter occupies a separate dimension, preventing mutual interference.

3. Parameter Estimation with Atomic Norm Minimization

In FDD mode, where partial channel reciprocity holds, the DL channel can be inferred from the feedback DL path gains $g^D\left(n_l\right)$ and the frequency-independent parameters $\left\{L^D,{\tau }^D,{\theta }^D\right\}$. Thus, accurately estimating $\left\{L^D,{\tau }^D,{\theta }^D\right\}$ is essential. Conventional methods rely on exhaustive searches over a densely sampled grid, assuming that these parameters lie precisely on grid points. However, this either introduces off-grid errors or significantly increases computational complexity due to the trade-off between grid resolution and search efficiency. To overcome this limitation, in this section we propose an ANM-based approach that achieves high-resolution estimation of $\left\{L^D,{\tau }^D,{\theta }^D\right\}$ with reduced complexity.

3.1. The Atomic Norm Minimization Problem

For simplicity, we assume the BS employs a half-wavelength ULA, i.e., $d={\displaystyle \frac{{\lambda }^U}{2}}$. Given ${\theta }_l\in \left[0,\pi \right)$, we define $\nu \in \left[0,0.5\right)$ as ${\displaystyle \frac{d}{{\lambda }^U}}sin{\theta }_l$. Similarly, since $0\le {\tau }_l^U<1/\Delta f$, we introduce $\mu \in \left[0,1\right)$ as $\Delta f\tau$. Applying these transformations to (9), we obtain

$$\begin{array}{cc}\hfill {\mathit{y}}^U& =\sum _{l=1}^L{g}_l^U\mathit{p}\left({\mu }_l\right)\otimes \mathit{q}\left({\nu }_l\right)+\mathit{n}\hfill \\ & =\sum _{l=1}^L{g}_l^U{\mathit{u}}_l\left({\mu }_l,{\nu }_l\right)+\mathit{n}\hfill \\ & =\mathit{U}\mathit{g}+\mathit{n},\hfill \end{array}$$

(21)

where ${\mathit{u}}_l(\mu ,\nu )=\mathit{p}\left(\mu \right)\otimes \mathit{q}\left(\nu \right)$. The matrix $\mathit{U}\in {\mathbb{C}}^{MN\times L}$ is formed by stacking all $\mathit{u}{(\mu ,\nu )}_l$ for $l=1,\dots ,L$:

$$\mathit{U}=\left[\mathit{u}\left({\mu }_1,{\nu }_1\right),\dots ,\mathit{u}\left({\mu }_L,{\nu }_L\right)\right],$$

(22)

and $\mathit{g}\in {\mathbb{C}}^{L\times 1}$ is constructed by stacking all $g_l^U$:

$$\mathit{g}={\left[g_1^U,\dots ,g_L^U\right]}^T.$$

(23)

Since the number of paths L is unknown, simultaneously estimating ${\mu }_l,{\nu }_l$, and ${\mathit{g}}_l$ is challenging. Paths with small gains $g_l$ may be buried in noise $\mathit{n}$, making them indistinguishable in ${\mathit{y}}^U$. Given the channel’s sparsity, we focus on identifying dominant paths by minimizing L. This problem, which seeks to recover the minimal set of contributing components, is naturally formulated as an ANM problem [24]. Compared to most existing works which directly extract channel paths from CSI, channel paths recovered from ANM are estimated in the continuous domain, thus avoiding off-grid error and enhancing accuracy [25].

To cast (21) as an ANM problem, we define the atomic set $\mathcal{A}$ for the reconstructed signal $\mathit{y}$ as

$$\mathcal{A}\triangleq \left\{{\mathit{u}}_l\left(\mu ,\nu \right):\nu \in \left[0,0.5\right),\mu \in \left[0,1\right)\right\},$$

(24)

and the atomic norm $\left|\mathit{y}\right|\mathcal{A},0$, corresponding to the number of contributing atoms as 

$${∥\mathit{y}∥}_{\mathcal{A},0}\triangleq \inf \left\{L:\sum _{i=1}^L{g}_l^U\mathit{u}\left({\mu }_l,{\nu }_l\right),\mathit{u}\left({\mu }_l,{\nu }_l\right)\in \mathcal{A}\right\}.$$

(25)

Since solving (25) is NP-hard, we relax it by minimizing the sum of absolute path gains, leading to

$${∥\mathit{y}∥}_{\mathcal{A}}\triangleq \inf \left\{\sum _{l=1}^L\left|g_l^U\right|:\sum _{l=1}^L{g}_l^U\mathit{u}\left({\mu }_l,{\nu }_l\right),\left(\mu ,\nu \right)\in \mathcal{A}\right\}.$$

(26)

To exclude paths dominated by noise, we define an error margin $ϵ$ based on the noise power:

$$ϵ={\sigma }^{2}N\Delta f,$$

(27)

and an error vector $\mathit{z}$ satisfying $\left|\mathit{z}\right|2⩽ϵ$. Thus, our final optimization problem is

$$\begin{array}{cccc}\hfill & \underset{\mathit{g},\mathit{z}}{\mathrm{minimize}}\hfill & \hfill & {∥\mathit{y}∥}_{\mathcal{A}}\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \mathit{y}=\mathit{U}\mathit{g}+\mathit{z}\hfill \\ \hfill & \hfill & & {∥\mathit{z}∥}_2⩽ϵ,\hfill \end{array}$$

(28)

where the first constraint is derived from Equation (21) and the second constraint from the noise power threshold Equation (27).

Problem (28) shares similarities with the direction-of-departure and direction-of-arrival (DOD-AOA) estimation problem in radar systems, for which grid-free ANM approaches have been extensively studied. For instance, ref. [13] reformulates ANM as a positive semidefinite relaxation (SDR) problem by constructing a block Toeplitz matrix, which can be efficiently solved using standard techniques. However, ref. [26] shows that such a Toeplitz construction may degrade estimation accuracy. In this work, we propose a grid-free approach by solving the dual problem of (28) and constructing a tailored Toeplitz matrix leveraging the properties of $g_l^U$.

Notably, since the number of paths L—and thus the dimensions of $\mathit{U}$ ($MN\times L$) and $\mathit{g}$ ($L\times 1$)—is unknown in advance, problem (28) cannot be solved directly. To address this, we first extract the most dominant path and subtract its contribution from the received signal $\mathit{y}$. The ANM process is then iteratively executed until a predefined stopping criterion is met. Through this iterative approach, dominant paths are sequentially identified and removed, allowing the DL channel to be estimated based on these dominant components due to its inherent sparsity. For each iteration, extracting the most dominant path $\left\{{\mu }_1,{\nu }_1,g_1\right\}$ is feasible, as the problem has a bounded infimum for any practical $\mathit{y}$, given that at least one propagation path is always present. Therefore, we solve the problem by analyzing the values of $\mathit{U}$ and $\mathit{g}$ under this bounded infimum condition. It will be demonstrated that, through this process, the dimensions of $\mathit{U}$ and $\mathit{g}$ can be effectively reduced.

3.2. Construction of the Dual Problem

To derive the dual problem of (28), we introduce the dual variables $\mathit{\alpha }\in {\mathbb{C}}^{MN\times 1}$ and $\beta \in {\mathcal{R}}^{1+}$. The Lagrangian function associated with (28) is formulated as

$$\begin{array}{cc}\hfill & L\left(\mathit{y},\mathit{\alpha },\beta \right)=\hfill \\ & {∥\mathit{y}∥}_{\mathcal{A}}+\mathrm{Re}\left[{\mathit{\alpha }}^H\left(\mathit{y}-\mathit{U}\mathit{g}-\mathit{z}\right)\right]+\beta \left({\mathit{z}}^H\mathit{z}-{ϵ}^2\right).\hfill \end{array}$$

(29)

The dual function $f\left(\mathit{\alpha },\beta \right)$ is obtained by minimizing the Lagrangian (29) over $\mathit{g}$ and $\mathit{z}$:

$$\begin{array}{cc}\hfill f\left(\mathit{\alpha },\beta \right)=& \underset{\mathit{g},\mathit{z}}{\inf }L\left(\mathit{y},\mathit{\alpha },\beta \right)\hfill \\ \hfill =& \underset{\mathit{g},\mathit{z}}{\inf }\{\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{y}-{\mathit{\alpha }}^H\mathit{z}\right]+\beta \left({\mathit{z}}^H\mathit{z}-{ϵ}^2\right)\hfill \\ & +{∥\mathit{y}∥}_{\mathcal{A}}-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]\}.\hfill \end{array}$$

(30)

To solve for $\mathit{z}$, we compute the partial derivative of $f\left(\mathit{\alpha },\beta \right)$ with respect to $\mathit{z}$:

$${\displaystyle \frac{\partial \left(\mathit{\alpha },\beta \right)}{\partial \mathit{n}}}=-\mathit{\alpha }+2\beta \mathit{z}.$$

(31)

Setting (31) to zero yields the optimal $\mathit{z}$:

$${\mathit{z}}_0={\displaystyle \frac{\mathit{\alpha }}{2\beta }}.$$

(32)

Substituting (32) into (30), we obtain

$$\begin{array}{cc}\hfill f\left(\mathit{\alpha },\beta \right){|}_{{\mathit{z}}_0}=& \mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{y}\right]-{\displaystyle \frac{{\mathit{\alpha }}^H\mathit{\alpha }}{2\beta }}+\beta \left({\displaystyle \frac{{\mathit{\alpha }}^H\mathit{\alpha }}{4{\beta }^2}}-{ϵ}^2\right)\hfill \\ \hfill +& \underset{\mathit{g}}{\inf }\left({∥\mathit{y}∥}_{\mathcal{A}}-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]\right).\hfill \end{array}$$

(33)

Next, differentiating with respect to $\beta$ gives

$${\displaystyle \frac{\partial \left(\mathit{\alpha },\beta \right)}{\partial \beta }}={\displaystyle \frac{{\mathit{\alpha }}^H\mathit{\alpha }}{4{\beta }^2}}-{ϵ}^2.$$

(34)

Setting (34) to zero yields the optimal $\beta$:

$${\beta }_0={\displaystyle \frac{{∥\mathit{\alpha }∥}_2}{2ϵ}}.$$

(35)

Substituting (32) and (35) into (30) gives

$$\begin{array}{cc}\hfill f\left(\mathit{\alpha }\right){|}_{{\mathit{z}}_0,{\beta }_0}=& \mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{y}\right]-ϵ{∥\mathit{\alpha }∥}_2\hfill \\ \hfill +& \underset{\mathit{g}}{\inf }\left({∥\mathit{y}∥}_{\mathcal{A}}-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]\right).\hfill \end{array}$$

(36)

To maximize (36) over $\mathit{\alpha }$, we examine $\underset{\mathit{g}}{\inf }\left({∥\mathit{y}∥}_{\mathcal{A}}-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]\right)$. By (26), we have

$$\begin{array}{cc}\hfill & {∥\mathit{y}∥}_{\mathcal{A}}-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]\hfill \\ & =\sum _{l=1}^L{g}_l^U-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]\hfill \\ & =\sum _{l=1}^L\left\{g_l^U-\mathrm{Re}\left[{\left({\mathit{\alpha }}^H\mathit{U}\right)}_l{g}_l^U\right]\right\}\hfill \\ & =\sum _{l=1}^L\left|g_l^U\right|\left\{1-\left|{\left({\mathit{U}}^H\mathit{\alpha }\right)}_l\right|cos{\varphi }_l\right\},\hfill \end{array}$$

(37)

where ${\left({\mathit{\alpha }}^H\mathit{U}\right)}_l$ is the lth entry of ${\mathit{\alpha }}^H\mathit{U}$ and ${\varphi }_l$ is the angle between ${\left({\mathit{U}}^H\mathit{\alpha }\right)}_l$ and $g_l^U$.

Proposition 1. 

${\left|\mathit{y}\right|}_{\mathcal{A}}-\mathrm{Re}\left[{\mathit{\alpha }}^H\mathit{U}\mathit{g}\right]$ has a finite infimum only if either $\left|{\left({\mathit{\alpha }}^H\mathit{U}\right)}_l\right|=1$ or the path gain $g_l^U\ne 0$.

Proof. 

Since we have

$$\begin{array}{c}\hfill \left|g_l^U\right|\left\{1-\left|{\left({\mathit{U}}^{\mathrm{H}}\mathit{\alpha }\right)}_l\right|cos{\varphi }_l\right\}\ge \left|g_l^U\right|\left\{1-\left|{\left({\mathit{U}}^{\mathrm{H}}\mathit{\alpha }\right)}_l\right|\right\},\end{array}$$

(38)

when $\left|{\left({\mathit{\alpha }}^{\mathrm{H}}\mathit{U}\right)}_l\right|>1$, the lower bound of (37) becomes $-\infty$, implying that the infimum does not exist. Similarly, when $\left|{\left({\mathit{\alpha }}^{\mathrm{H}}\mathit{U}\right)}_l\right|<1$ and $\left|g_l^U\right|\to \infty$, the lower bound of (37) approaches ∞, meaning the infimum is also undefined in this case.

Thus, to ensure a bounded infimum, we must have $\left|{\left({\mathit{\alpha }}^{\mathrm{H}}\mathit{U}\right)}_l\right|=1$. Under this condition, maximizing (36) over $\mathit{\alpha }$ leads to the following optimization problem:

$$\begin{array}{cccc}\hfill & \underset{\mathit{\alpha }}{\mathrm{maximize}}\hfill & \hfill & \mathrm{Re}\left[{\mathit{\alpha }}^{\mathrm{H}}\mathit{y}\right]-ϵ{∥\mathit{\alpha }∥}_2\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \left|{\left({\mathit{U}}^{\mathrm{H}}\mathit{\alpha }\right)}_l\right|=1,\forall l=1,\dots ,L.\hfill \end{array}$$

(39)

This problem can be reformulated into a structured optimization problem by imposing a Toeplitz matrix constraint.    □

Proposition 2. 

There exists a Hermitian matrix $\mathit{D}\in {\mathbb{C}}^{MN\times MN}$ such that for any ${\mathit{U}}_i\in \mathcal{A}$, we have ${\mathit{U}}_i^{\mathrm{H}}\mathit{D}{\mathit{U}}_i=1$, where ${\mathit{U}}_i$ represents the ith column of ${\mathit{U}}^{\mathrm{H}}$ and the corresponding row of $\mathit{U}$.

Proof. 

See Appendix A.    □

Using the Schur complement, we obtain

$$\left[\begin{array}{cc}\mathit{D}\hfill & \mathit{\alpha }\hfill \\ {\mathit{\alpha }}^H\hfill & 1\hfill \end{array}\right]=0.$$

(40)

Thus, the unitary amplitude constraint in problem (39) is reformulated as

$$\begin{array}{cccc}\hfill & \underset{\mathit{\alpha },\mathit{D}\in \mathcal{H}}{\mathrm{maximize}}\hfill & \hfill & \mathrm{Re}\left[{\mathit{\alpha }}^{\mathit{H}}\mathit{y}\right]-ϵ{∥\mathit{\alpha }∥}_2\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \left[\begin{array}{cc}\mathit{D}\hfill & \mathit{\alpha }\hfill \\ {\mathit{\alpha }}^H\hfill & 1\hfill \end{array}\right]=0,\hfill \end{array}$$

(41)

where $\mathcal{H}$ is the set of all Hermitian matrices $\mathit{D}\in {\mathbb{C}}^{MN\times MN}$ satisfying the constraints in (A4). The optimization problem (41) can be solved using off-the-shelf solvers, such as the SDPT3 toolbox for Matlab [27]. After obtaining the optimal $\mathit{\alpha }$ from (41), we define $P(\mu ,\nu )$ as the power spectrum of ${\mathit{u}}_l(\mu ,\nu )=\mathit{p}\left(\mu \right)\otimes \mathit{q}\left(\nu \right)$ over the domain $\nu \in [0,0.5)$ and $\mu \in [0,1)$:

$$P\left(\mu ,\nu \right)={\mathit{u}}_l{\left(\mu ,\nu \right)}^H\mathit{\alpha }{\mathit{\alpha }}^H{\mathit{u}}_l\left(\mu ,\nu \right).$$

(42)

By locating the highest peak of $P(\mu ,\nu )$ within $\nu \in [0,0.5)$ and $\mu \in [0,1)$, we estimate ${\nu }_1$ and ${\mu }_1$ for the most dominant UL channel path. The corresponding path gain $g_1^U$ is then computed as

$$g_1^U={\displaystyle \frac{{\mathit{u}}_l^H\left({\mu }_1,{\nu }_1\right){\mathit{y}}^U}{{∥{\mathit{u}}_l^H\left({\mu }_1,{\nu }_1\right)∥}^2}}.$$

(43)

Once the first UL channel parameters $g_1^U,{\mu }_1,{\nu }_1$ are determined, we define the residual signal as

$${\mathit{y}}_r^U={\mathit{y}}^U-g_1^u{\mathit{u}}_l\left({\mu }_1,{\nu }_1\right).$$

(44)

By replacing ${\mathit{y}}^U$ with ${\mathit{y}}_r^U$, we iteratively apply the same ANM approach to extract subsequent UL channel parameters $g_2^U,{\mu }_2,{\nu }_2$. Repeating this process for all paths, we obtain $g_l^U,{\mu }_l,{\nu }_l$ for $l=1,\dots ,L$. Since the exact number of paths L is unknown, we monitor the residual signal power ${\left|{\mathit{y}}_r^U\right|}^2$ after each iteration and compare it with the noise power ${ϵ}^2$. The process terminates when

$${∥{\mathit{y}}_r^U∥}^2<ϵ,$$

(45)

indicating that the residual signal is dominated by noise, making further extraction unnecessary.

3.3. Complexity Analysis

The proposed ANM approach consists of two main stages: solving the optimization problem (41) and identifying peaks in the power spectrum (42). In the first stage, constructing the matrix $\mathit{D}$ requires $\mathcal{O}\left(MS\right)$ operations, followed by solving (41) using SDPT3, which has a computational complexity of $\mathcal{O}\left({\displaystyle \frac{1}{2}}{M}^2{S}^2+2M{S}^3\right)$, as outlined in [28]. For the second stage, searching for peaks in the power spectrum demands $\mathcal{O}\left(p_m{p}_{n}MS\right)$ operations, where $p_m$ and $p_n$ represent the search step sizes for $\mu$ and $\nu$, respectively. Additionally, evaluating the stopping criterion requires $\mathcal{O}\left(MS\right)$. Given that there are L paths, the iterative procedure is expected to run L times. A summary of the complexity analysis is presented in

Table 1. Complexity of ANM.

Stage	Complexity (FLOPs)
Check Stopping Criterion	$\mathcal{O}\left(LMS\right)$
Build Matrix $\mathit{D}$	$\mathcal{O}\left(LMS\right)$
Solve (41) with SDPT3	$\mathcal{O}\left({\displaystyle \frac{1}{2}}{M}^2{S}^2+2M{S}^3\right)$
Find the peaks from spectrum	$\mathcal{O}\left(L{p}_m{p}_{n}MS\right)$
Overall complexity	$\mathcal{O}\left(M^2{S}^2+M{S}^3\right)$

, while a comparison of complexity with existing message passing techniques as well as conventional methods (i.e., MUSIC and ESPRIT) is presented in

Table 2. Comparison of Complexity with Message Passing Techniques.

Stage	Complexity	FLOPs
Proposed ANM	$\mathcal{O}\left(M^2{S}^2+M{S}^3\right)$	5.2 × 1010
Joint Data Detection [29]	$\mathcal{O}\left(N_{it}{M}^2{S}^3\right)$	4.6 × 1013
Unitary Approximation [30]	$\mathcal{O}\left(N_{it}\left(M^2{S}^3+M^3{S}^2\right)\right)$	6.6 × 1013
MUSIC	$\mathcal{O}\left(M^2\left(L+N_{it}\right)S^3+M^3{S}^2\right)$	7.5 × 1013
ESPRIT	$\mathcal{O}\left(M^{2}L+M^3+M{S}^2+S^3\right)$	1.6 × 1013

, where $N_{it}$ denotes the number of iterations, where $M=256$, $S=512$, $L=3$, $N_{it}=5$ are considered.

4. Parameter Estimation with DNN

Table 1 indicates that the most computationally intensive stage is solving problem (41). This step can be further optimized by adopting a lower-complexity approach. Additionally, when multiple propagation paths are closely spaced, the ANM approach struggles to distinguish them, as observed in numerical results. Therefore, replacing this procedure with a method that offers both lower complexity and higher resolution would be highly beneficial. This motivates us to develop a deep neural network (DNN)-based architecture for fast and accurate channel parameter prediction from the received signal ${\mathit{y}}^U$ in this section.

In recent years, DNNs have gained significant attention and have demonstrated remarkable success in various domains, including image recognition, natural language processing, and autonomous driving [31]. One of the key advantages of DNN architectures is their ability to provide predictions with nearly linear time complexity once trained, making them well-suited for modern telecommunication systems that require low-latency processing. The DNN framework consists of multiple layers, each acting as a nonlinear mapping function. Given the input ${\mathit{y}}^U$ and $ϵ$, the expected output $\left\{{\mu }_1,{\nu }_1\right\}$ can be expressed as

$$\begin{array}{c}\hfill \left\{{\mu }_1,{\nu }_1\right\}=f^n\left(f^{n-1}\left(\dots f^1\left({\mathit{y}}^U,ϵ\right)\right)\right),\end{array}$$

(46)

where $f^n$ represents the mapping function of the nth layer. Once ${\mu }_1$ and ${\nu }_1$ are determined, $g_1^U$ is computed using (43), thereby identifying the first propagation path. Subsequently, the same procedure in (44) is applied iteratively for the remaining paths until the stopping criterion in (45) is met.

4.1. Exhaustive Search for Training Data Set

Before deploying the DNN for channel parameter prediction, it must first be trained in a supervised manner. This requires obtaining the desired output $\left\{{\tilde{\mu }}_1,{\tilde{\nu }}_1\right\}$ for any given ${\mathit{y}}^U$ and noise power $ϵ$. To achieve this, we adopt the conventional 2D-grid-based approach, where the 2D spatial range $\mu \in \left[0,1\right)$ and $\nu \in \left[0,0.5\right)$ is discretized into ${\gamma }_1\times {\gamma }_2$ grid points. Let $\mathcal{B}$ denote the set of all grid points, given by

$$\mathcal{B}=\left\{{\displaystyle \frac{m}{{\gamma }_1}},{\displaystyle \frac{n}{2{\gamma }_2}}:m=1,\dots ,{\gamma }_1;n=1,\dots ,{\gamma }_2\right\}.$$

(47)

The estimated parameters ${\tilde{\mu }}_1,{\tilde{\nu }}_1$ are obtained via an exhaustive search:

$${\tilde{\mu }}_1,{\tilde{\nu }}_1=arg\underset{\left({\mu }_1,{\nu }_1\right)\in \mathcal{B}}{max}{\displaystyle \frac{{\left|{\mathit{u}}^H\left(\mu ,\nu \right){\mathit{y}}^U\right|}^2}{{∥{\mathit{u}}^H\left(\mu ,\nu \right)∥}^2}},$$

(48)

and the corresponding channel gain ${\tilde{g}}_1^U$ is computed using (43).

The input–output pairs $\left\{{\mathit{y}}^U,ϵ\right\}$ and $\left\{{\tilde{\mu }}_1,{\tilde{\nu }}_1\right\}$ form the training dataset for supervised DNN training. Since training occurs offline and computational complexity is not a primary concern, ${\gamma }_1$ and ${\gamma }_2$ can be set sufficiently large to enhance the precision of the estimated parameters $\left\{{\tilde{g}}_1^U,{\tilde{\mu }}_1,{\tilde{\nu }}_1\right\}$.

4.2. Architecture of the Proposed DNN

The proposed DNN architecture, illustrated in Figure 1, consists of three types of layers: input, hidden, and output layers.

The input layer comprises three neurons, corresponding to $\mathrm{Re}\left({\mathit{y}}^U\right)$, $\mathrm{Im}\left({\mathit{y}}^U\right)$ and $ϵ$. The received signal ${\mathit{y}}^U$ is decomposed into its real and imaginary components, $\mathrm{Re}\left({\mathit{y}}^U\right)$ and $\mathrm{Im}\left({\mathit{y}}^U\right)$, respectively. As noted in [32], normalizing the inputs often improves learning efficiency and convergence speed. Following the approach in [33], normalization is applied in the pre-processing phase to preserve critical hidden features. The normalization factor ${\Delta }_n$ is defined as the $\mathit{l}$2-norm of the real input values:

$${\Delta }_n=\underset{i=1,\dots ,MN}{max}{∥\mathrm{Re}\left({\mathit{y}}^U\right)∥}_2^2,$$

(49)

where it is worth noting that the Frobenius norm of a vector is essentially a special case of the 2-norm, but the 2-norm is more consistent with vector operation specifications. All real and imaginary inputs are then divided by ${\Delta }_n$.

For the hidden layers, the rectified linear unit (ReLU) function, defined as $\mathrm{ReLU}\left(a\right)=max\left(0,a\right)$, is employed as the activation function. ReLU ensures nonnegative data flow and has been demonstrated to enhance training efficiency [34].

The output layer contains two neurons corresponding to ${\mu }_1$ and ${\nu }_1$. Only predictions satisfying $\mu \in \left[0,1\right)$ and $\nu \in \left[0,0.5\right)$ are considered valid. The channel gain $g_1^U$ is computed using (43), while predictions outside the valid range or with gains below $ϵ$ are marked as invalid. The output regulation is summarized as follows:

$$g_1^U=\left\{\begin{array}{c}0,\mathrm{if}\left\{\begin{array}{c}\mu \notin \left[0,1\right),\hfill \\ \mathrm{or}\nu \notin \left[0,0.5\right)\hfill \\ \mathrm{or}{g}_1^U<ϵ\hfill \end{array}\right.\hfill \\ g_1^U\mathrm{otherwise}\hfill \end{array}\right.$$

(50)

If $g_1^U=0$, the DNN has failed to provide a valid prediction, necessitating further training. Invalid outputs are discarded until a valid prediction is obtained.

The loss function, $\mathrm{Loss}\left({\mu }_1,{\nu }_1\right)$, is defined as the sum of the squared $\mathit{l}$2-norm distances between the actual output $\left({\mu }_1,{\nu }_1\right)$ and predicted values $\left({\tilde{\mu }}_1,{\tilde{\nu }}_1\right)$ obtained via exhaustive search:

$$\begin{array}{cc}\hfill \mathrm{Loss}\left({\mu }_1,{\nu }_1\right)& ={∥{\mu }_1^U-{\tilde{\mu }}_1^U∥}_F^2+{∥{\nu }_1^U-{\tilde{\nu }}_1^U∥}_F^2,\mathrm{if}{g}_1^U\ne 0.\hfill \end{array}$$

(51)

The proposed model is a multilayer perceptron (MLP) with multiple hidden layers in a fully connected feedforward network. Each neuron in one layer is connected to every neuron in the subsequent layer, with data flowing in a forward direction. The input vector ${\mathit{x}}_0$ is defined as

$${\mathit{x}}_0={\left[\mathrm{Re}\left({\mathit{y}}^U\right)\mathrm{Im}\left({\mathit{y}}^U\right)ϵ\right]}^T\in {\mathbb{C}}^{\left(2MN+1\right)\times 1}.$$

(52)

Denoting the output of the ith hidden layer as $\mathit{x}i$, the transformation performed at layer i is given by

$${\mathit{x}}_{i+1}=f\left({\mathit{w}}_i^T{\mathit{x}}_i+{\mathit{b}}_i\right),$$

(53)

where ${\mathit{w}}_i$ and ${\mathit{b}}_i$ are the weight and bias vectors for the ith hidden layer, updated during training.

4.3. Simplified LM-WC for Training

During the training stage, the weight and bias vectors for each hidden layer are updated by minimizing the loss function (51). In general, training algorithms fall into two categories: first-order methods, such as conventional gradient descent and error back-propagation [35], and second-order methods, including Newton’s method and the well-known Levenberg–Marquardt (LM) algorithm [36]. It is widely recognized that second-order algorithms typically achieve faster convergence, making them preferable for complex scenarios. In this work, we adopt the LM algorithm for training.

However, a common challenge in DNN training is the occurrence of diminishing gradients in certain hidden layer neurons, causing the learning process to stall—an issue known as the flat-spot problem [37]. This phenomenon has been frequently observed in our numerical experiments. First-order algorithms can mitigate this issue by learning additional parameters to normalize neuron activations. However, this approach is not practical for the LM algorithm, as it involves computing a large Jacobian matrix [38]. To address this issue, ref. [38] proposed the weight compression (LM-WC) algorithm, which adjusts neuron outputs to ensure they remain within the activation function’s active region. This compression effectively modifies the inputs to the network’s activation function, guiding them toward an activated state, as illustrated in Figure 2. By applying weight compression, the likelihood of successful training significantly increases.

The LM-WC algorithm effectively mitigates the flat-spot problem with high probability but introduces an additional compression parameter that must be determined. Consequently, it can only function properly after this parameter has been appropriately identified. In [38], the parameter is chosen to minimize the loss function, which requires re-training the network. Turning to our proposed DNN with the ReLU activation function, we observe that the compression process can be further simplified. If certain weight vectors remain unchanged for an extended period, the corresponding neurons are considered ‘stuck’. When the number of stuck neurons becomes sufficiently large, we compress the weight vector so that the maximum possible number of stuck neurons get activated (i.e., their ReLU inputs become positive). Given the original weight vector $\mathit{w}$, the compressed weight vector ${\mathit{w}}_c$ is defined as

$${\mathit{w}}_c=\mathit{c}\mathit{I}\mathit{w},$$

(54)

where $\mathit{c}$ is the weight compression vector, and $\mathit{I}$ is an identity matrix of appropriate size. For a hidden layer with M outputs and N neurons, $\mathit{w}$ and $\mathit{c}$ can be further expressed as

$$\begin{array}{cc}\hfill & \mathit{w}=\left[{\mathit{w}}_1,\dots ,{\mathit{w}}_N\right]\hfill \\ \hfill & \mathit{c}=\left[{\mathit{c}}_1,\dots ,{\mathit{c}}_N\right].\hfill \end{array}$$

(55)

The value of ${\mathit{c}}_n$ is computed as

$${\mathit{c}}_n={\displaystyle \frac{\mathsf{\Omega }M{\mathit{w}}_n}{{\sum }_{m=1}^M{a}_m}},$$

(56)

where $\mathsf{\Omega }>0$ is the compression parameter, and $a_m$ represents the output of the ReLU activation function. Instead of tuning $\mathsf{\Omega }$ during training to minimize error, as done in [38], our simplified LM-WC algorithm selects $\mathsf{\Omega }$ such that the number of positive $a_m$ values is maximized. That is, we determine $\mathsf{\Omega }$ by solving

$$\mathsf{\Omega }=arg\underset{\mathsf{\Omega }\in \mathrm{R}}{max}\mathrm{M}\left(\mathsf{\Omega }\right),$$

(57)

where $\mathrm{M}\left(\mathsf{\Omega }\right)\in [0,M]$ denotes the number of positive $a_m$ values after applying the compression parameter $\mathsf{\Omega }$ in (56).

Since the exact range of $\mathsf{\Omega }$ is unknown, an exhaustive search is impractical. Instead, we adopt an alternative approach to obtain a near-optimal solution. Beginning with an initial value of $\mathsf{\Omega }$, we iteratively multiply and divide $\mathsf{\Omega }$ by a constant factor $\rho$ and compare the values of $\mathrm{M}\left(\mathsf{\Omega }\rho \right)$ and $\mathrm{M}\left(\mathsf{\Omega }/\rho \right)$, recording the larger one. This process continues by evaluating $\mathrm{M}\left(\mathsf{\Omega }{\rho }^2\right)$ and $\mathrm{M}\left(\mathsf{\Omega }/{\rho }^2\right)$, updating the best value accordingly. The procedure stops at stage s when both $\mathrm{M}\left(\mathsf{\Omega }{\rho }^s\right)$ and $\mathrm{M}\left(\mathsf{\Omega }/{\rho }^s\right)$ are smaller than the current maximum $\mathrm{M}\left({\mathsf{\Omega }}_{opt}\right)$. At this point, weight compression is performed using $\mathsf{\Omega }={\mathsf{\Omega }}_{opt}$. A summary of the proposed simplified LM-WC algorithm is presented in Algorithm 1. The value of compression parameter $\mathsf{\Omega }$ ought to be a value adjacent to the activation function, while a successful step size $\rho$ should start from a marginal threshold. Recommended values of $\mathsf{\Omega }=1$ and $\rho =1.1$, tested in [38], demonstrate strong performance in our simulations.

Algorithm 1 Simplified LM-WC algorithm

Require:  Compression step size $\rho$, initial $\mathsf{\Omega }$ Ensure:  Near-optimal ${\mathsf{\Omega }}_{opt}$ 1: Index $s=1$, current optimal $O=0$ 2: while $\mathrm{M}\left(\mathsf{\Omega }/{\rho }^s\right)>O$ or $\mathrm{M}\left(\mathsf{\Omega }{\rho }^s\right)>O$ do 3:     if $\mathrm{M}\left(\mathsf{\Omega }/{\rho }^s\right)>\mathrm{M}\left(\mathsf{\Omega }{\rho }^s\right)$ then 4:         $O=\mathrm{M}\left(\mathsf{\Omega }/{\rho }^s\right)$, ${\mathsf{\Omega }}_{opt}=\mathsf{\Omega }/{\rho }^s$ 5:     else 6:         $O=\mathrm{M}\left(\mathsf{\Omega }{\rho }^s\right)$, ${\mathsf{\Omega }}_{opt}=\mathsf{\Omega }{\rho }^s$ 7:     end if 8:     $s=s+1$ 9: end while

4.4. The Prediction Stage

The proposed DNN architecture requires a supervised learning dataset, consisting of input samples $\left\{{\mathit{y}}^U,ϵ\right\}$ and their corresponding desired outputs $\left\{{\tilde{\mu }}_1,{\tilde{\nu }}_1\right\}$, obtained via an exhaustive search approach. Given an input $\left\{{\mathit{y}}^U,ϵ\right\}$, we perform an exhaustive search to determine $\left\{{\tilde{g}}_1^U,{\tilde{\mu }}_1,{\tilde{\nu }}_1\right\}$. Using these values in (44), we compute the residual signal ${\mathit{y}}_r^U$. Setting ${\mathit{y}}_r^U$ as the new input $\left\{{\mathit{y}}^U\right\}$, another round of exhaustive search is conducted to extract the corresponding $\left\{{\tilde{g}}_1^U,{\tilde{\mu }}_1,{\tilde{\nu }}_1\right\}$. By repeating this process iteratively, we can generate up to L training datasets from a single received signal ${\mathit{y}}^U$.

In a real-world system, these two approaches can be integrated as follows. Initially, when the DNN is untrained, the base station (BS) employs the atomic norm minimization (ANM) method to estimate the downlink (DL) channel and serve the user equipment (UE). Concurrently, the DNN undergoes offline training using received signals and channel parameters estimated via a high-resolution exhaustive search. After a certain training period, we evaluate the DNN’s readiness by monitoring the probability of invalid predictions (where $g_1^U=0$). If this probability falls below a predefined threshold, the system can switch to the DNN-based approach for serving UE. Once the DNN is deployed, it remains crucial to periodically validate its performance using the ANM approach. If the probability of invalid predictions exceeds the threshold, the system should revert to the ANM method while continuing to refine the DNN with new training data.

5. Numerical Results

We evaluate the performance of the proposed approach in a multi-user wideband mmWave MIMO-OFDM system in this section. The carrier frequency is set to $f_c=28$ GHz, with a total bandwidth of $f_s=8$ GHz and $S=512$ subcarriers. The BS is equipped with $M=256$ antennas, arranged with half-wavelength inter-element spacing. There are eight single-antenna pieces of UE uniformly distributed over the angular range $\left(-\pi /2,\pi /2\right)$. The sparse mmWave channel for each UE is generated following the configuration in [39]. Each UE has $L=3$ propagation paths, with channel gains $g_l\sim \mathcal{CN}\left(0,1\right)$. The angles of arrival ${\theta }_l$ are uniformly distributed as ${\theta }_l\sim \mathcal{U}\left(-\pi /6,\pi /6\right)$, while the propagation delays ${\tau }_l$ follow ${\tau }_l\sim \mathcal{U}\left(0,2\times {10}^{-8}s\right)$. All simulation results are averaged over 3000 independent trials.

To assess the accuracy of channel estimation, we adopt the normalized mean square error (NMSE) metric, computed as

$$\mathbb{E}\left({\displaystyle \frac{1}{N}}\sum _{n=1}^N{∥\mathit{h}-{\mathit{h}}^e∥}_2^2/{∥\mathit{h}∥}_2^2\right),$$

(58)

where $\mathit{h}$ represents the actual channel, and ${\mathit{h}}^e$ is the estimated channel.

5.1. Performance of the Atomic Norm Minimization Approach

We first evaluate the performance of the proposed ANM approach by measuring NMSE against SNR. Two scenarios are considered: (a) the propagation paths for a single UE are evenly distributed within the angular spreading range ${\theta }_l\sim \mathcal{U}\left(-\pi /6,\pi /6\right)$, and (b) the propagation paths for a single UE are clustered within a narrow angular spreading range of $4^{\circ }$, as considered in [40]. In the latter case, if the central angle of arrival for UE p is ${\theta }_p$, then the directional angles of its three paths are ${\theta }_p-4^{\circ }$, ${\theta }_p$, and ${\theta }_p+4^{\circ }$, respectively. For comparison, we use the unified scheme from [40] as a lower-bound benchmark, which reduces training overhead but sacrifices some performance due to the use of basis vectors for channel estimation. The linear minimum mean square error (LMMSE) scheme serves as the upper-bound benchmark, where each antenna is assigned an orthogonal pilot signal to ensure maximum accuracy. Additionally, we compare the proposed scheme with the Newtonized orthogonal matching pursuit (NOMP) scheme [41].

Figure 3 presents the NMSE performance of the proposed scheme alongside the benchmark methods under different channel conditions. The results show that our approach achieves competitive performance compared to both the unified scheme [40] and the NOMP scheme [41]. The main performance loss in the unified scheme stems from the reduced number of inserted pilot signals and the limited basis used for channel estimation. For the NOMP approach, two primary sources of performance loss exist: first, its initial grid-based search introduces off-grid errors, especially when using a low over-sampling rate (only two, as in [41]); second, its Newtonized step descending search may converge to a local rather than a global optimum. In contrast, our proposed approach effectively mitigates both off-grid errors and local optima, except when identifying peaks in the power spectrum (42). Fortunately, peak detection over a given spectrum is a much simpler task than the iterative search and refinement process used in the NOMP approach, leading to lower overall error. Numerical results further reveal that all methods tend to achieve better NMSE performance when propagation paths are evenly distributed. This is because, in the clustered channel model where paths are confined within a narrow angular range, accurately extracting their directions becomes more challenging. Although our approach outperforms NOMP in clustered channels, its performance degradation is slightly more pronounced. This is due to the ANM algorithm’s tendency to minimize the number of detected paths, making it more sensitive to scenarios where paths are closely spaced. This sensitivity is a key motivation for introducing the DNN-based approach in this work.

Additionally, we evaluate the bit error rate (BER) performance of the proposed approach and benchmark methods. Simulations employ quadrature phase shift keying (QPSK) modulation with maximum likelihood (ML) detection. The results, shown in Figure 4, indicate that both the NOMP and proposed approaches achieve performance comparable to the LMMSE method. Moreover, consistent with our NMSE observations, the proposed ANM approach performs better when propagation paths are evenly distributed than when they are closely clustered.

5.2. Performance of the DNN-Based Approach

To assess the effectiveness of the proposed DNN, we conduct supervised training using data obtained from an exhaustive search over a high-resolution grid. The DNN is designed as a medium-sized network comprising one input layer, three fully connected hidden layers, and one output layer. The hidden layers contain 128, 256, and 384 neurons, respectively. The training and evaluation are performed on a personal computer equipped with an Intel^® Core^TM i7-11700K processor (Intel Corporation, Santa Clara, CA, USA) and 32 GB of RAM, using Matlab R2020b (The MathWorks Inc., Natick, MA, USA). For the training process, we compare three algorithms: the standard Levenberg–Marquardt (LM) algorithm, our proposed simplified LW-WC method (introduced in Section 4.3), and the conventional scaled conjugate gradient algorithm.

For training, we generate a total of 25,600 samples of $\mathit{y}$, with the corresponding $\mu ,\nu ,g$ as inputs and desired outputs. The dataset consists of 12,800 samples from the sparse channel model (Scenario a) and 12,800 samples from the clustered channel model (Scenario b), following the channel configuration described in Section 6. The training process is conducted with the following hyperparameters: a batch size of 64,120 epochs, and a total of 48,000 iterations, with a learning rate of 0.1.

Figure 5 compares the training accuracy of three algorithms: LM, our modified LM-WC, and the conventional gradient descent algorithm, based on their training loss. It can be observed that the second-order algorithms, LM and LM-WC, converge significantly faster than the first-order gradient descent algorithm. Additionally, as indicated in the figure, a flat-spot problem arises. To address this, the LM-WC algorithm applies weight compression when the average gradient remains unchanged by at least $5\%$ for five consecutive epochs. By implementing our simplified LM-WC algorithm, the training process exits the flat spot approximately three epochs earlier. This demonstrates that the proposed LM-WC effectively mitigates training stalls while maintaining nearly the same performance. Figure 6 illustrates the average gradient during DNN training using different algorithms. When weight compression is applied, the gradient improves almost immediately, further confirming that LM-WC can boost the gradient when it becomes trapped in a flat spot. However, while LM-WC accelerates training convergence, its overall performance in terms of loss function and gradient remains nearly identical to that of conventional LM.

We then compare the training results of the proposed simplified LM-WC algorithm with two benchmarks: conventional LM and gradient descent. The training process allows for a maximum of 200 epochs to ensure all algorithms converge to their final error level while also passing validation and testing data. The number of epochs and the total training time required for all testing data to be successfully verified are recorded. To provide a comprehensive evaluation of LM-WC, we consider different values of the learning rate $\gamma$ and compression step size $\rho$ in Algorithm 1.

Table 3. Training Result.

Algorithm	Epochs	Time (s)	Final MSE
Gradient Descent	134.5	31.2	1.13 × 10−7
LM	97.7	929.6	1.43 × 10−11
LM-WC (0.1,1.1)	75.5	876.2	1.44 × 10−11
LM-WC (0.1,1.5)	78.3	852.0	1.77 × 10−11
LM-WC (0.01,1.1)	83.0	956.5	2.23 × 10−12
LM-WC (0.01,1.5)	86.7	953.7	3.14 × 10−12

summarizes the results, where $\mathrm{LM}-\mathrm{WC}(\gamma ,\rho )$ in the algorithm column represents the simplified LM-WC algorithm with a learning rate $\gamma$ and compression step size $\rho$. The results indicate that, in general, LM-WC requires fewer epochs to converge compared to conventional LM but spends more time per epoch due to the additional weight compression operations. The trade-off is that LM-WC achieves convergence with fewer epochs, making it more suitable for high-speed computational environments or scenarios where severe flat-spot issues occur. Additionally, a smaller learning rate $\gamma$ results in more epochs and longer convergence time but tends to yield better performance. Meanwhile, a smaller compression step size $\rho$ enhances precision in weight compression, which can further improve results, though at the cost of increased computation time for optimizing the compression parameter $\mathsf{\Omega }$.

Finally, we utilize the trained DNN to perform ANM-based channel parameter estimation and assess its effectiveness. Since offline training is used and training time is not a primary concern, the DNN can be trained for an extended period to enhance prediction accuracy. In this study, we adopt a DNN trained with the optimal WC parameters listed in Table 3, specifically LM-WC (0.01,1.1), for our DNN-aided ANM. The results, presented in Figure 7 and Figure 8, compare the proposed ANM with LMMSE and ANM-only benchmarks. As seen in the NMSE comparison in Figure 7, the DNN-ANM approach significantly outperforms the conventional ANM and achieves performance nearly on par with the LMMSE method. Notably, in scenarios with clustered paths—where many algorithms struggle—the DNN-ANM method excels. Since channel parameter extraction can be viewed as a pattern recognition problem, where distinct paths must be identified, the DNN-ANM effectively captures the subtle variations among clustered paths. This leads to superior performance compared to traditional mathematical approaches such as the benchmark NOMP.

6. Conclusions

In this work, we propose two approaches for channel parameter estimation in FDD systems: the Atomic Norm Minimization (ANM) approach and the Deep Neural Network (DNN)-based approach. The ANM method constructs an atomic set based on the phases of propagation paths and sequentially extracts each path by solving a dual minimization problem. This method effectively mitigates the off-grid mismatching error present in conventional grid-based techniques. To address potential path confusion in clustered channel environments and reduce computational complexity, we employ a DNN trained to perform parameter extraction. Additionally, we introduce a simplified weight compression algorithm to overcome training stalls. These two approaches can be used in a complementary manner: the ANM method remains operational while the DNN is training, and once the DNN is deployed, the ANM approach can serve as a validation tool for its predictions. Simulation results demonstrate that the ANM approach outperforms conventional methods, while the DNN-based approach further improves estimation accuracy, achieving performance close to that of the LMMSE method. The DNN training requires 25,600 samples (about 10 h of offline training), and its performance degrades significantly when the sample size is insufficient. In the future, few-shot learning (e.g., meta-learning) can be used to reduce the demand for training data.