Deep Learning for Joint Pilot, Channel Feedback and Sub-Array Hybrid Beamforming in FDD Massive MU-MIMO-OFDM Systems

Abstract

In frequency division duplex (FDD) massive multiple-input multiple-output orthogonal frequency division multiplexing (MIMO-OFDM) systems, the sub-array multi-user (MU) hybrid beamforming architecture is highly attractive because of its low hardware cost and high energy efficiency. However, downlink channel state information (CSI) acquisition and hybrid beamformer optimization remain challenging due to the large feedback overhead and the non-convexity of the beamforming design. To address these issues, we propose an end-to-end deep learning (DL) framework that jointly optimizes pilot training, CSI feedback, and hybrid beamforming, overcoming the limitations of conventional independently designed modules. At the core of the network, we introduce the star efficient location attention (StarELA) module, which combines the implicit high-dimensional representation capability of star operations (element-wise multiplication) with the fine-grained feature localization of efficient location attention (ELA). In addition, for wideband digital beamformer generation, we exploit inter-subcarrier correlation and design a frequency–domain seed generation and interpolation upsampling strategy, which significantly reduces network parameters. Experimental results show that the proposed method approaches the upper-bound performance of conventional hybrid beamforming with ideal CSI, while consistently outperforming existing benchmark methods.

1. Introduction

Following the global commercial deployment of 5G networks, research on the vision and enabling technologies of sixth-generation mobile communications (6G) has been fully initiated [1]. As cornerstones for enhancing spectral efficiency and system capacity, massive multiple-input multiple-output (MIMO) and the emerging extremely large-scale MIMO (XL-MIMO) [2] will play a central role in 6G evolution. By deploying large-scale antenna arrays at the base station (BS), MIMO systems can significantly boost throughput and suppress multi-user interference through spatial multiplexing and beamforming. However, realizing these benefits relies on the availability of accurate channel state information (CSI) at the BS. Specifically for downlink transmission, the BS requires precise CSI to compute optimal beamforming matrices to focus signal energy on target user equipment (UE), thereby achieving superior system performance.

In time division duplex (TDD) systems, the BS can leverage channel reciprocity to acquire downlink CSI from uplink pilots. However, in frequency division duplex (FDD) systems, such reciprocity is absent as uplink and downlink operate on different frequency bands. Consequently, UEs must estimate downlink CSI from pilots transmitted by the BS and feed it back for beamforming. With the massive scaling of antenna arrays, the associated feedback overhead grows prohibitively, posing a severe bottleneck for FDD massive MIMO systems [3].

Despite these overhead challenges, the dual-band nature of FDD architectures also opens up novel possibilities for multi-band fusion techniques. Recent studies have demonstrated that fusing uplink and downlink communication signals across different FDD bands can significantly enhance target sensing capabilities in integrated sensing and communication (ISAC) systems [4].

In addition, hardware cost and power consumption are also pivotal factors in practical deployment. Fully-connected digital beamforming, which requires one radio frequency (RF) chain per antenna, is often cost-prohibitive, especially in mmWave massive MIMO systems. To address this, the sub-array hybrid beamforming architecture [5,6] illustrated in Figure 1 offers a promising alternative by connecting each antenna to a single RF chain, which significantly reduces the required number of phase shifters (PSs). However, this architecture introduces rigorous design constraints [5]: the analog beamforming matrix must satisfy constant modulus and specific antenna connectivity requirements. Consequently, optimizing the digital beamformer becomes a non-convex problem that is difficult to solve using traditional methods.

1.1. Related Works

To address CSI feedback and beamforming challenges in FDD systems, research has evolved from conventional separated designs to deep learning-based approaches. In traditional communication frameworks, pilot design, channel estimation, feedback, and beamforming are optimized as independent modules. Early studies mainly relied on compressed sensing (CS) algorithms [7] to exploit channel sparsity in the angular or delay domain for CSI estimation; the resulting CSI was then fed back via codebook-based quantization or CS-based compression. Based on reconstructed CSI, the BS designed beamformers using zero-forcing (ZF), minimum mean-square error (MMSE) [8], and methods based on successive interference cancellation (SIC) [9] or alternating minimization (AltMin) [5].

Despite their effectiveness, these methods have apparent limitations. CS-based schemes depend strongly on sparsity assumptions and can degrade in rich-scattering environments. Codebook-based feedback faces a fundamental trade-off between overhead and accuracy. AltMin-based beamforming can approach fully digital performance, but its iterative optimization incurs high computational complexity, limiting real-time applicability in low-latency scenarios.

In recent years, deep learning (DL) has shown strong potential for wireless communications. CsiNet [10] modeled CSI as an image and introduced an autoencoder-based feedback network, significantly outperforming conventional CS baselines. Subsequent studies further improved feedback accuracy [11,12,13,14,15]. CRNet [11] introduced multi-resolution feature extraction, CsiNet+ [12] adopted deeper architectures, and TransNet [13] employed Transformer mechanisms to capture long-range dependencies.

However, most of these DL-based methods still optimize CSI reconstruction error, typically measured by the mean squared error (MSE), as the primary objective. The BS uses fed-back CSI for downlink beamforming, and a lower reconstruction error does not necessarily lead to a higher beamforming gain. In massive MIMO systems, beamforming performance is more sensitive to physically meaningful features (e.g., dominant-path angles and gains), while MSE treats all entries equally and may preserve less relevant details. As a result, methods with low MSE can still yield suboptimal system performance, especially under the separated design where pilot, feedback, and beamforming are optimized independently. Therefore, CSI feedback should be evaluated based on its impact on downstream beamforming and overall communication performance rather than solely on reconstruction error.

To resolve these limitations, recent studies [16,17,18,19,20,21,22] have explored end-to-end joint modeling of pilot training, channel feedback, and beamforming with direct sum-rate maximization. Ref. [16] proposed a joint feedback and multi-user (MU) fully digital precoding network that achieved performance close to perfect-CSI linear precoding. Ref. [17] considered FDD orthogonal frequency division multiplexing (OFDM) systems and proposed a joint CSI feedback and precoding method based on deep joint source-channel coding techniques. Ref. [22] developed a precoding-oriented feedback mechanism with a dedicated loss function design to balance feedback overhead and sum rate, further demonstrating the advantage of end-to-end joint optimization.

Nevertheless, most existing end-to-end methods assume fully digital or fully connected hybrid beamforming at the BS. Although fully connected architectures can achieve strong performance, they require many phase shifters and complex RF interconnections, resulting in high hardware costs and limited scalability for deployment [23].

Beyond these implementation costs, practical CSI acquisition is further affected by hardware non-idealities, including phase noise, nonlinear amplification, synchronization offsets, and finite-resolution phase shifters. Such impairments are especially detrimental to phase-sensitive applications, such as interferometric angle-of-arrival estimation [24] and beamforming [25,26], where even small phase errors can introduce artifacts and degrade CSI accuracy. Since traditional separated block designs are highly vulnerable to the propagation of such estimation errors, there is a strong motivation to develop end-to-end joint optimization frameworks. By directly maximizing the ultimate system performance rather than relying on intermediate explicit CSI estimation, end-to-end learning naturally mitigates the accumulation of these cascading errors.

Regarding the sub-array architecture, which is of greater practical value but involves more complex constraints, relevant research is still in its infancy. Ref. [21] pioneered the end-to-end joint design of sub-array multi-user hybrid beamforming architectures, proposing a lightweight attention network called EFBAttnNET. However, this work only considered narrowband flat fading channels. In practical wideband OFDM systems, channels exhibit frequency-selective fading. Directly extending narrowband architectures to wideband scenarios fails to leverage inter-subcarrier frequency correlations, resulting in linear scaling of model parameters with the number of subcarriers.

1.2. Contributions

The main contributions of this paper are summarized as follows:

• We introduce the sub-array hybrid beamforming architecture into FDD MU-MIMO-OFDM systems and propose a DL-based end-to-end framework for joint pilot training, CSI feedback, and hybrid beamforming. The proposed framework, comprising the pilot network, the feedback network, and the beamforming network, is trained unsupervisedly to maximize the system sum rate. It adaptively learns task-oriented pilot patterns and feedback features, avoiding explicit channel estimation and feedback in conventional separated designs.
• We design a star efficient location attention (StarELA) module that combines the implicit high-dimensional mapping capability of star operations (element-wise multiplication) with the position-aware local interaction modeling of efficient location attention (ELA) [27]. As the core block in both the feedback and beamforming networks, StarELA improves channel feature extraction efficiency and effectiveness.
• We propose a seed generation and interpolation upsampling strategy for the digital beamforming branch of the beamforming network at the BS. By exploiting strong inter-subcarrier correlation in wideband OFDM channels, this mechanism reduces model parameters while preserving frequency–domain smoothness and generalization of the generated digital beamformers.
• Simulation results demonstrate that the proposed method approaches the performance of conventional sub-array hybrid beamforming with ideal CSI and outperforms existing deep learning baselines under different feedback bit budgets, exhibiting superior robustness even with limited pilot overhead.

The rest of this paper is organized as follows. Section 2 introduces the system model and formulates the joint optimization problem. Section 3 details the proposed end-to-end deep learning framework, encompassing the pilot network, the feedback network, the hybrid beamforming network, and the core StarELA module. Section 4 presents the numerical results and comprehensive performance comparisons. Finally, Section 5 concludes the paper and discusses future research directions.

2. System Model and Problem Formulation

In this section, we first introduce the system model under the sub-array hybrid beamforming architecture; then we describe the processes of downlink pilot transmission, CSI feedback, and beamforming; and finally, we formulate the optimization problem for maximizing the system sum rate.

2.1. System Model

We consider an FDD massive MU-MIMO-OFDM system with sub-array hybrid beamforming, where a BS equipped with $N_t$ uniform linear array (ULA) antennas and K RF chains serves K single-antenna UEs over $N_c$ OFDM subcarriers. The received signal at the k-th UE on the n-th subcarrier is given by:

$$y_k\left[n\right]={\mathbf{h}}_k{\left[n\right]}^H{\mathbf{v}}_k\left[n\right]s_k\left[n\right]+\sum _{i\ne k}{\mathbf{h}}_k{\left[n\right]}^H{\mathbf{v}}_i\left[n\right]s_i\left[n\right]+z_k\left[n\right]$$

(1)

where ${\mathbf{h}}_k\left[n\right]\in {\mathbb{C}}^{N_t\times 1}$ and ${\mathbf{v}}_k\left[n\right]\in {\mathbb{C}}^{N_t\times 1}$ are the downlink channel vector and equivalent beamforming vector for the k-th UE on the n-th subcarrier, respectively. $s_k\left[n\right]$ denotes the data symbol transmitted to the k-th UE on the n-th subcarrier, and $z_k\left[n\right]\sim \mathcal{CN}(0,{\sigma }_n^2)$ is additive white Gaussian noise (AWGN).

For sub-array hybrid beamforming, the $N_t$ antennas are uniformly allocated to K RF chains, where the number of antennas connected to a single RF chain is given by $N_g=N_t/K$. The equivalent beamforming vector ${\mathbf{v}}_k\left[n\right]$ can be defined as follows:

$${\mathbf{v}}_k\left[n\right]=\mathbf{A}\mathbf{C}{\mathbf{d}}_k\left[n\right]$$

(2)

where $\mathbf{A}\triangleq \mathrm{diag}\left(e^{j{\mathit{\theta }}_{\mathbf{A}}}\right)\in {\mathbb{C}}^{N_t\times N_t}$ is the analog beamformer, which is a constant modulus diagonal matrix. ${\mathit{\theta }}_{\mathbf{A}}\in {\mathbb{R}}^{N_t\times 1}$ is the phase vector of $\mathbf{A}$. ${\mathbf{d}}_k\left[n\right]\in {\mathbb{C}}^{K\times 1}$ is the digital beamformer for UE k on subcarrier n. The antenna connection matrix $\mathbf{C}={\mathbf{I}}_K\otimes {\mathbf{1}}_{N_g}\in {\{0,1\}}^{N_t\times K}$ is a block-diagonal matrix representing the physical connection constraints between RF chains and antenna arrays in the sub-array architecture, where ${\mathbf{I}}_K\in {\mathbb{R}}^{K\times K}$ is the identity matrix and ${\mathbf{1}}_{N_g}\in {\mathbb{R}}^{N_g\times 1}$ is an all-ones vector.

Furthermore, let $\mathbf{D}\left[n\right]\triangleq [{\mathbf{d}}_1\left[n\right],\dots ,{\mathbf{d}}_K\left[n\right]]\in {\mathbb{C}}^{K\times K}$ denote the digital beamforming matrix on subcarrier n. The equivalent beamforming matrix $\mathbf{V}\left[n\right]\triangleq [{\mathbf{v}}_1\left[n\right],\dots ,{\mathbf{v}}_K\left[n\right]]\in {\mathbb{C}}^{N_t\times K}$ must satisfy the following power constraint:

$${\left|\mathbf{V}\left[n\right]\right|}_F^2={\left|\mathbf{A}\mathbf{C}\mathbf{D}\left[n\right]\right|}_F^2\le P/N_c,\forall n.$$

(3)

where P denotes the BS’s total transmit power budget per OFDM symbol, summed over all $N_t$ antennas and all $N_c$ subcarriers.

For the channel vector ${\mathbf{h}}_k\left[n\right]$, we adopt a typical sparse channel model [28], where each UE’s channel consists of $L_{path}$ dominant propagation paths, as shown below:

$${\mathbf{h}}_k\left[n\right]=\sqrt{{\displaystyle \frac{N_t}{L_{path}}}}\sum _{p=1}^{L_{path}}{\alpha }_{p,k}{e}^{-j2\pi {\displaystyle \frac{n}{N_c}}{\tau }_{p,k}{F}_s}{\mathbf{a}}_t\left({\varphi }_{p,k}\right)$$

(4)

where ${\alpha }_{p,k}$, ${\varphi }_{p,k}$, and ${\tau }_{p,k}$ denote the complex gain, angle of departure (AoD), and delay of the p-th path for the k-th UE, respectively. $F_s$ is the OFDM sampling rate. The adopted model is intended to characterize sparse directional wideband propagation conditions that are more representative of mmWave scenarios, in which hybrid beamforming is especially well suited. Accordingly, it does not explicitly target dense indoor rich-scattering channels dominated by strong diffuse multipath.

The transmitter antenna employs a ULA, so the transmit array response vector can be expressed as:

$${\mathbf{a}}_t\left(\varphi \right)={\displaystyle \frac{1}{\sqrt{N_t}}}{\left[1,e^{j{\displaystyle \frac{2\pi d}{\lambda }}\sin \left(\varphi \right)},e^{j2{\displaystyle \frac{2\pi d}{\lambda }}\sin \left(\varphi \right)},\dots ,e^{j(N_t-1){\displaystyle \frac{2\pi d}{\lambda }}\sin \left(\varphi \right)}\right]}^T$$

(5)

where $\lambda$ denotes the carrier wavelength, and the antenna spacing is chosen as $d=\lambda /2$. The phase shift between adjacent antenna elements is:

$$\Delta \psi ={\displaystyle \frac{2\pi d}{\lambda }}\sin \left(\theta \right).$$

(6)

For any visible angle $\theta \in [-{90}^{\circ },{90}^{\circ }]$, when $d\le \lambda /2$, we have $\Delta \psi \in [-\pi ,\pi ]$, which guarantees a unique mapping from the physical angle to the array response and, thus, avoids spatial aliasing. In contrast, when $d>\lambda /2$, phase wrapping causes different directions to share the same phase progression, resulting in spatial aliasing and grating lobes. Although any spacing satisfying $d\le \lambda /2$ is sufficient to prevent aliasing, the half-wavelength setting is commonly adopted because it is the largest alias-free spacing and therefore provides the highest angular resolution for a given number of antennas. Hence, $d=\lambda /2$ is particularly suitable for angle-domain channel modeling and hybrid beamforming design.

2.2. Problem Formulation

In FDD systems, UEs need to estimate downlink CSI from received downlink pilot signals and feed it back to the BS. We consider the pilot transmitted by the BS as $\tilde{\mathbf{X}}\in {\mathbb{C}}^{N_t\times L}$, where L is the pilot length. The pilot signal received by UE k on subcarrier n can be expressed as:

$${\tilde{\mathbf{y}}}_k\left[n\right]={\mathbf{h}}_k{\left[n\right]}^H\tilde{\mathbf{X}}+{\tilde{\mathbf{z}}}_k\left[n\right].$$

(7)

where ${\tilde{\mathbf{y}}}_k\left[n\right]\in {\mathbb{C}}^{1\times L}$, and ${\tilde{\mathbf{z}}}_k\left[n\right]\sim \mathcal{CN}(0,{\sigma }_n^2{I}_L)$ is AWGN.

We define ${\tilde{\mathbf{Y}}}_k\triangleq {[{\tilde{\mathbf{y}}}_k\left[1\right],\dots ,{\tilde{\mathbf{y}}}_k\left[N_c\right]]}^T\in {\mathbb{C}}^{N_c\times L}$, ${\mathbf{H}}_k\triangleq {[{\mathbf{h}}_k\left[1\right],\dots ,{\mathbf{h}}_k\left[N_c\right]]}^H\in {\mathbb{C}}^{N_c\times N_t}$, and ${\tilde{\mathbf{Z}}}_k\triangleq {[{\tilde{\mathbf{z}}}_k\left[1\right],\dots ,{\tilde{\mathbf{z}}}_k\left[N_c\right]]}^T\in {\mathbb{C}}^{N_c\times L}$. Then, the pilot signals received by UE k over all subcarriers can be written as:

$${\tilde{\mathbf{Y}}}_k={\mathbf{H}}_k\tilde{\mathbf{X}}+{\tilde{\mathbf{Z}}}_k.$$

(8)

In the considered wideband OFDM system, the BS transmits L consecutive OFDM pilot symbols over the $N_c$ subcarriers. After digital baseband processing and inverse fast Fourier transform (IFFT), the resulting RF signals are weighted by the analog phase-shifter network in the sub-array architecture. Because the employed analog phase shifters apply the same phase control across the entire operating band, the equivalent antenna–domain pilot matrix $\tilde{\mathbf{X}}$ is modeled as frequency-flat with respect to the subcarrier index. Consequently, the same antenna–domain pilot matrix is shared across all subcarriers, whereas the received pilot observations remain frequency-selective due to the subcarrier-dependent channel response. Moreover, under the phase-only analog implementation, $\tilde{\mathbf{X}}$ satisfies the following element-wise constant modulus constraint:

$$|{\left[\tilde{\mathbf{X}}\right]}_{i,j}|=\sqrt{{\displaystyle \frac{P}{N_c{N}_t}}},\forall i,j.$$

(9)

Specifically, the pilot transmission is subject to the same total power budget P as the downlink data transmission, so that each subcarrier has an average power of $P/N_c$, yielding the element-wise magnitude $\sqrt{P/\left(N_c{N}_t\right)}$.

Upon receiving the pilot signal, UE k extracts channel features and compresses them into a bitstream ${\mathbf{q}}_k$ via a feedback mapping function $\mathcal{F}(·)$. This process can be expressed as:

$${\mathbf{q}}_k=\mathcal{F}\left({\tilde{\mathbf{Y}}}_k\right),{\mathbf{q}}_k\in {\{\pm 1\}}^{B\times 1}$$

(10)

The BS collects feedback bitstreams from all UEs $\mathbf{Q}={[{\mathbf{q}}_1,{\mathbf{q}}_2,\dots ,{\mathbf{q}}_K]}^T\in {\{\pm 1\}}^{K\times B}$, and uses a beamforming design function $\mathcal{B}(·)$ to jointly recover channel features and generate beamformers. This process is formulated as:

$$({\mathit{\theta }}_{\mathbf{A}},\widehat{\mathbf{D}})=\mathcal{B}\left(\mathbf{Q}\right).$$

(11)

where $\widehat{\mathbf{D}}=[\widehat{\mathbf{D}}\left[1\right],\dots ,\widehat{\mathbf{D}}\left[N_c\right]]\in {\mathbb{C}}^{N_c\times K\times K}$ is the raw digital beamformer, and ${\mathit{\theta }}_{\mathbf{A}}$ is the phase vector of the analog beamformer.

Finally, to satisfy the transmit power constraint in Equation (3), the BS performs power normalization on $\widehat{\mathbf{D}}$:

$$\mathbf{D}\left[n\right]={\displaystyle \frac{\sqrt{P/N_c}}{\parallel \mathbf{A}\mathbf{C}\widehat{\mathbf{D}}{\left[n\right]\parallel }_F}}\widehat{\mathbf{D}}\left[n\right],\forall n.$$

(12)

We define the system sum rate as the sum of the achievable rates of all UEs and maximize it as the system objective. Based on the received signal model in Equation (1), the optimization problem is formulated as:

$$\begin{array}{cc}\hfill \underset{\tilde{\mathbf{X}},\mathcal{F}(·),\mathcal{B}(·)}{\max }& R={\displaystyle \frac{1}{N_c}}\sum _{n=1}^{N_c}\sum _{k=1}^K{\log }_2\left(1+{\displaystyle \frac{|{\mathbf{h}}_k{\left[n\right]}^H\mathbf{A}\mathbf{C}{\mathbf{d}}_k{\left[n\right]|}^2}{{\sum }_{i\ne k}{\left|{\mathbf{h}}_k{\left[n\right]}^H\mathbf{A}\mathbf{C}{\mathbf{d}}_i\left[n\right]\right|}^2+{\sigma }_n^2}}\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& |{\left[\tilde{\mathbf{X}}\right]}_{i,j}|=\sqrt{P/N_c{N}_t},\forall i,j,\hfill \\ \hfill & {|\left[\mathbf{A}\right]}_{m,m}|=1,\forall m,\hfill \\ \hfill & \parallel \mathbf{A}\mathbf{C}{\mathbf{d}}_k{\left[n\right]\parallel }_F^2\le P/N_c,\forall n.\hfill \end{array}$$

(13)

3. Proposed Method

In this section, based on the system model and optimization objective in Section 2, we propose a DL-based method for joint downlink pilot training, uplink channel feedback, and hybrid beamforming for sub-array architecture. As illustrated in Figure 2, the overall architecture comprises K parallel branches of pilot and feedback networks for distributed CSI acquisition, followed by a central hybrid beamforming network at the BS. Additionally, the system incorporates a sub-array antenna connection matrix and a power normalization module to enforce hardware constraints.

3.1. StarELA Module

To efficiently model received pilot signals and beamforming parameters under limited computational overhead, inspired by StarNet [29] and StarCANet [14], we propose a general feature processing unit named StarELA. StarELA fully exploits the advantage of star operations (element-wise multiplication) in constructing implicit high-dimensional spaces and integrates the ELA [27] mechanism, forming a compact module with strong nonlinear representation ability. The detailed structure of StarELA is shown in Figure 3.

StarELA first adopts depthwise convolution with a $7\times 7$ kernel for local contextual feature extraction, effectively expanding the receptive field while keeping parameter and computational costs controllable. Subsequently, features are mapped into an expanded feature space through two parallel $1\times 1$ pointwise convolution branches. One branch is nonlinearly activated and element-wise multiplied with the other branch. This parameter-free nonlinear interaction maps low-dimensional features into an implicit high-dimensional linear space, thereby improving feature representation.

Following the star operation, we introduce ELA for position-aware recalibration. Specifically, ELA decomposes the input into two orthogonal 1D feature vectors via global average pooling to capture long-range spatial dependencies. These vectors are processed with 1D convolutions to model local interactions, followed by group normalization (GN) to improve training stability. Subsequently, the generated attention weights are activated via the sigmoid function and aggregated to element-wise reweight the input features. This design achieves precise localization of salient regions with minimal computational overhead, effectively preventing information loss from channel reduction. Finally, StarELA restores the original channel dimension using a $1\times 1$ pointwise convolution, followed by a $7\times 7$ depthwise convolution for feature integration. Residual connections and DropPath are also incorporated to stabilize training and enhance generalization.

Overall, StarELA maintains a highly compact structure, avoids expensive global attention computation, and is suitable for deployment in resource-constrained sub-array hybrid beamforming systems. As the core feature interaction unit in our framework, its specific usage and role are further described in the following subnet designs.

3.2. Pilot Network

As the initiation of the proposed pipeline, we introduce the pilot network to perform downlink pilot training. It is designed to learn pilot sequences that maximize system performance subject to hardware constraints and is jointly optimized with the subsequent networks.

During the pilot transmission stage, the BS broadcasts a common downlink pilot matrix $\tilde{\mathbf{X}}\in {\mathbb{C}}^{N_t\times L}$ to all UEs, with the received signal model governed by Equations (7) and (8). Given the hybrid beamforming architecture, $\tilde{\mathbf{X}}$ is strictly bound by the element-wise constant modulus constraint, as detailed in Equation (9).

In [16], downlink pilot training is modeled as a bias-free fully connected layer with pilots learned as network weights; however, this formulation does not apply to our hybrid beamforming system, where pilots are implemented by analog phase shifters with fixed amplitudes.

To address this, we adopt a phase-parameterized modeling strategy. Instead of optimizing $\tilde{\mathbf{X}}$ directly, we introduce a phase matrix ${\mathit{\theta }}_{\tilde{\mathbf{X}}}\in {\mathbb{R}}^{N_t\times L}$ as the trainable parameter set and express the pilot matrix as:

$$\tilde{\mathbf{X}}=\sqrt{{\displaystyle \frac{P}{N_c{N}_t}}}{e}^{j{\mathit{\theta }}_{\tilde{\mathbf{X}}}}=\sqrt{{\displaystyle \frac{P}{N_c{N}_t}}}\left[\cos \left({\mathit{\theta }}_{\tilde{\mathbf{X}}}\right)+j\sin \left({\mathit{\theta }}_{\tilde{\mathbf{X}}}\right)\right].$$

(14)

This formulation intrinsically satisfies the constant modulus constraint, avoiding the need for complex non-convex projections or additional penalty terms. Furthermore, it facilitates seamless integration into the end-to-end backpropagation process of deep learning frameworks.

For network initialization, the pilot phase matrix ${\mathit{\theta }}_{\tilde{\mathbf{X}}}$ is initialized via uniform sampling of a discrete Fourier transform (DFT) matrix. This ensures favorable initial orthogonality and spectral coverage, promoting training stability and faster convergence. Subsequently, ${\mathit{\theta }}_{\tilde{\mathbf{X}}}$ is jointly optimized with feedback network parameters ${\mathsf{\Theta }}_{\mathcal{F}}$ and hybrid beamforming network parameters ${\mathsf{\Theta }}_{\mathcal{B}}$ to maximize the system sum-rate.

In practical deployments, the analog phase shifters execute both pilot transmission and analog beamforming via time-division multiplexing (TDM). They are initially configured with the learned phase matrix ${\mathit{\theta }}_{\tilde{\mathbf{X}}}$ to transmit downlink pilots, and dynamically reconfigured during data transmission to apply the generated analog beamformer phase vector ${\mathit{\theta }}_{\mathbf{A}}$. This TDM operation allows the system to fully reuse the sub-array hardware without incurring additional RF chain costs.

3.3. Feedback Network

The feedback network is deployed at the UE side to map the high-dimensional and noisy received pilot signals ${\tilde{\mathbf{Y}}}_k$ into compact binary bitstreams ${\mathbf{q}}_k$, as illustrated in Figure 4a. Its architecture is designed to efficiently extract and compress the coupled spatial frequency features embedded in the pilot observations under limited feedback overhead. Since conventional fully connected or standard convolutional networks either incur high parameter complexity at the UE side or are less effective in modeling long-range dependencies across antennas and subcarriers, we adopt a deep encoder architecture with StarELA as the core feature extraction block. Owing to its ability to capture both local structures and global contextual dependencies with moderate computational complexity, this design enables the UE to generate compact binary representations while preserving the essential channel semantics required for subsequent beamforming tasks at the BS.

To make the complex pilot observation ${\tilde{\mathbf{Y}}}_k$ compatible with real-valued neural network, we first split it into real and imaginary parts and stack them into a two channel tensor ${\tilde{\mathbf{Y}}}_k^{\mathrm{in}}$. A $1\times 1$ convolution is then used to expand the feature channel dimension to $C_{up}=8$, followed by two cascaded StarELA modules for feature refinement.

Within StarELA, the $7\times 7$ depthwise convolution mitigates local noise fluctuations in the pilot domain, while the star operation exploits the nonlinear coupling between pilot sequences and channel frequency responses to extract compact semantic features. The ELA mechanism factorizes 2D features into orthogonal 1D encodings and generates location-sensitive attention weights to recalibrate the features, enabling the preservation of salient information arising from multipath effects.

Subsequently, the refined features are projected back to two channels through a $1\times 1$ convolution. Following a flattening operation, the features are compressed into a real-valued vector of length B by a multi-layer perceptron (MLP) comprising cascaded fully connected (FC) layers with batch normalization (BN).

Finally, to generate the uplink feedback bitstream ${\mathbf{q}}_k$, the feature vector is binarized. However, applying the sign function directly results in zero gradients almost everywhere during backpropagation, thereby impeding training. To address this gradient vanishing problem, we adopt a sigmoid-adjusted straight-through estimator (STE) with slope annealing [16]. Specifically, during the forward propagation phase, the quantizer employs the $\mathrm{sign}(·)$; whereas in the backward propagation phase, the derivative of the sign function is approximated by the gradient of a scaled sigmoid function. The approximation function is formulated as:

$${\mathcal{Q}}_{\mathrm{soft}}\left(x\right)=2\sigma (\alpha ·x)-1={\displaystyle \frac{2}{1+e^{-\alpha ·x}}}-1,$$

(15)

where $\sigma (·)$ denotes the sigmoid function and $\alpha$ is the annealing factor.

In our implementation, the annealing factor follows the explicit schedule:

$${\alpha }_t=\min ({\alpha }_0{r}^t,10),{\alpha }_0=1,r=1.0007$$

(16)

where t denotes the training update step. Therefore, the quantizer starts from a smooth surrogate to ensure stable gradient propagation in the early stage and gradually approaches hard binary feedback as training proceeds. The upper bound of 10 prevents the surrogate gradient from becoming excessively sharp and numerically unstable in the late stage, while still ensuring that the final feedback symbols are effectively discrete.

3.4. Hybrid Beamforming Network

Deployed at the BS, the hybrid beamforming network is designed to generate the analog beamformer $\mathbf{A}$ and digital beamformer $\mathbf{D}$ from the collected feedback bitstreams $\mathbf{Q}$. As illustrated in Figure 4b, the proposed network consists of two stages: multi-user feature refinement and fusion, and hybrid beamformer generation.

The network architecture is designed according to the physical characteristics of the wideband MU-MIMO-OFDM system and the hardware constraints of hybrid beamforming. Since suppressing multi-user interference requires joint spatial processing, a shared StarELA-based backbone is adopted to refine per-user features and capture their correlations in a unified global representation. Based on this global context, two dedicated branches are used to generate the analog and digital beamformers, respectively. This branch decomposition is motivated by their different physical properties: the analog beamformer is frequency-flat due to the phase-shifter implementation, whereas the digital beamformer must adapt to frequency-selective channel variations. To avoid the redundancy and parameter growth caused by direct per-subcarrier regression, the digital branch adopts a frequency–domain seed generation and interpolation strategy to recover the full-band beamformer from a compact set of anchor features.

3.4.1. Multi-User Feature Refinement and Fusion

To efficiently recover per-user channel spatial structures, we adopt a parallel processing strategy. The BS first expands the received feedback bits $\mathbf{Q}\in {\{\pm 1\}}^{K\times B}$ via two FC layers into K independent feature tensors. These tensors are treated as a batch and fed into a shared backbone consisting of four cascaded StarELA modules. Leveraging the feature enhancement capabilities of StarELA, the backbone progressively reconstructs fine-grained channel details. The integrated ELA mechanism further enforces positional recalibration, encouraging the reconstructed features to preserve consistent physical patterns along subcarrier and antenna dimensions.

Following independent refinement, the feature tensors of the K users are flattened, concatenated, and aggregated by a FC fusion layer. This fusion layer learns the spatial correlations among users, yielding a global context vector ${\mathbf{f}}_{fusion}$ that serves as the high-quality basis for beamforming design.

3.4.2. Hybrid Beamformer Generation

Based on the global features ${\mathbf{f}}_{fusion}$, the network splits into two branches to generate analog and digital beamformers.

(1) Analog Beamforming Branch: Given that analog phase shifters are frequency-flat, this branch directly maps ${\mathbf{f}}_{fusion}$ to an $N_t$ dimensional phase vector ${\mathit{\theta }}_{\mathbf{A}}$ via a FC layer. The constant modulus analog beamformer is then constructed using Euler’s formula: $\mathbf{A}=\mathrm{diag}\left(e^{j{\mathit{\theta }}_{\mathbf{A}}}\right)$.

(2) Digital Beamforming Branch: In wideband OFDM systems, the digital beamformer should adapt to frequency-selective fading across subcarriers. Direct regression of all subcarrier parameters by FC layers often leads to overfitting and parameter explosion as $N_c$ increases. To address this, we propose an efficient Frequency–Domain Seed Generation and Interpolation Upsampling strategy, which effectively reduces the parameter complexity from $O(N_c·K^2)$ to $O(N_c/S·K^2)$, as follows:

• Seed Generation: Exploiting the frequency correlation of channels, we first map ${\mathbf{f}}_{fusion}$ to a low-resolution seed tensor ${\mathbf{F}}_{seed}\in {\mathbb{R}}^{C_d\times N_{seed}}$, where $C_d=64$ is hidden channel dimension, $N_{seed}=N_c/S$ denotes the number of seed nodes and $S=4$ is the downsampling factor.
• Interpolation and Refinement: To recover full-band resolution, ${\mathbf{F}}_{seed}$ is upsampled along the frequency dimension via linear interpolation to obtain coarse features ${\mathbf{F}}_{up}\in {\mathbb{R}}^{C_d\times N_c}$. According to the channel model in Equation (4), the frequency–domain evolution across subcarriers is governed by a complex exponential function. However, since the subcarrier interval covered by the downsampling factor S is typically much smaller than the channel coherence bandwidth, this exponential phase rotation can be effectively approximated as a locally linear transformation. Therefore, we adopt a simple and lightweight linear interpolation for upsampling. Subsequently, we introduce a convolution block composed of $1\times 3$ 1D convolution, BN, and ReLU to compensate for any linear approximation errors and recover the precise nonlinear channel dynamics.

Finally, a 1D convolution layer projects the refined features to the raw digital beamformer $\widehat{\mathbf{D}}\in {\mathbb{C}}^{N_c\times K\times K}$, which is subsequently normalized to obtain $\mathbf{D}$ satisfying the total transmit power constraint according to Equation (12).

4. Numerical Results

4.1. Experimental Settings and Benchmarks

4.1.1. System Parameters and Dataset Generation

We consider a single-cell FDD massive MIMO-OFDM, where a BS equipped with $N_t=32$ antennas serves $K=2$ single-antenna UEs over $N_c=32$ subcarriers. The case of $K=2$ is adopted as a representative multi-user setting, which allows us to evaluate the proposed framework under a controlled level of inter-user interference while keeping the simulation complexity manageable. Furthermore, since the proposed architecture is formulated with scalable parallel branches and shared network weights, it is inherently generalized to support a larger number of users in practical deployments.

Following the channel model in Equation (4), each UE’s channel consists of $L_{path}=2$ propagation paths, with the AoD following a uniform distribution of $[-{30}^{\circ },{30}^{\circ }]$. This setting represents a sparse directional propagation scenario with limited angular spread, which is more consistent with sectorized cellular coverage than with dense indoor rich-scattering environments. The total transmit power $P=1$, and the system signal-to-noise ratio (SNR) is set to 10 dB.

As for the channel dataset, we generate 100,000 training samples and 10,000 testing samples based on these settings. All performance evaluations are conducted on the test set to ensure a fair comparison.

4.1.2. Training Settings

The proposed method adopts an end-to-end unsupervised learning strategy, with the loss function defined as the negative of the system sum rate, i.e., $\mathcal{L}=-R$. The network is implemented based on the PyTorch 2.6.0 framework and trained on an NVIDIA RTX 4090 GPU. We use the AdamW optimizer with an initial learning rate of $1\times {10}^{-3}$. To mitigate local optima and stabilize convergence, we applied a cosine annealing learning rate scheduler with warm-up. Specifically, the learning rate warms up for the first 15 epochs and then decays to a minimum of $5\times {10}^{-5}$ over a total of 300 epochs, with a batch size of 500.

4.1.3. Benchmarks

To validate the proposed method, we compare it against the following baselines, encompassing both traditional and deep learning-based schemes:

• Full Digital ZF with Perfect CSI: The BS is assumed to have perfect CSI and performs fully digital ZF beamforming. This serves as a theoretical upper bound.
• SDR-AltMin with Perfect CSI and Infinite Feedback: With perfect CSI at the BS, hybrid beamforming is designed using semidefinite relaxation based AltMin (SDR-AltMin) [5], representing the upper bound of conventional hybrid beamforming without estimation or feedback distortion.
• SDR-AltMin with OMP-CE and Infinite Feedback: Each UE estimates downlink CSI via orthogonal matching pursuit (OMP) [30], and the estimated CSI is fed back losslessly. The BS then applies SDR-AltMin.
• SDR-AltMin with OMP-CE and Finite Feedback: Each UE first performs OMP-based channel estimation, then quantizes path gain $\alpha$ and AoD $\varphi$ using Lloyd-Max quantization [31] for limited feedback; the BS reconstructs CSI and applies SDR-AltMin.
• JEFB-ResNet: As a deep learning baseline, we re-implement the residual network-based joint channel estimation feedback beamforming model (JEFB-ResNet) [17] and adapt it to the sub-array architecture for fair comparison.

4.2. Performance Comparison

To comprehensively evaluate the proposed method, we compare the system sum rate against various benchmarks under two pilot-length settings, i.e., $L=8$ and $L=4$.

Figure 5 presents the sum rate performance with a pilot length of $L=8$. It can be observed that the proposed method consistently outperforms the deep learning-based benchmark JEFB-ResNet across the entire feedback bit range ($B=3$ to 60). Specifically, at $B=60$, our method achieves a sum rate of 12.34 bits/s/Hz, providing a significant gain of 0.7 bits/s/Hz over JEFB-ResNet. Moreover, when $B\ge 5$, the proposed method already surpasses the SDR-AltMin with OMP-CE and Infinite Feedback scheme. This result highlights the critical advantage of end-to-end joint optimization: unlike traditional separated pipelines where residual channel estimation errors propagate to the beamforming stage and create a performance bottleneck, our framework is trained directly with the sum rate objective. This allows the network to allocate limited feedback bits to the most beamforming-relevant features (e.g., dominant path angles), thereby maximizing effective beamforming gain even with quantized feedback.

To further investigate the robustness of the proposed method under severely limited pilot resources, we evaluate the system sum rate when $L=4$ ($L\ll N_t=32$), as shown in Figure 6. In this scenario, traditional CS-based methods exhibit sharp performance degradation because sparse support recovery becomes unreliable with insufficient observations. In contrast, the proposed method demonstrates remarkable robustness, maintaining a high sum rate of 12.03 bits/s/Hz at $B=60$, only a marginal decrease of 0.3 bits/s/Hz compared to the $L=8$ case. Furthermore, the proposed method continues to outperform JEFB-ResNet by a clear margin. This indicates that the StarELA module possesses superior feature extraction capabilities, enabling the system to effectively recover channel semantic information even when pilot overhead is halved.

Overall, the experimental results verify that the proposed method efficiently exploits bit resources, with the sum rate increasing steadily with B and gradually approaching the upper bound of SDR-AltMin with perfect CSI. The consistent superiority over JEFB-ResNet under both pilot settings confirms that the introduced StarELA and hybrid beamformer generation strategy significantly enhances the system’s adaptability and spectral efficiency in FDD massive MIMO-OFDM systems.

5. Conclusions

In this paper, we propose an end-to-end deep learning framework for sub-array hybrid beamforming in FDD massive MU-MIMO-OFDM systems, jointly optimizing pilots, feedback, and beamforming. This approach eliminates the performance bottleneck caused by module independence in traditional designs. The framework incorporates a novel StarELA module for robust feature extraction and employs a digital beamformer generation strategy based on frequency–domain interpolation to reduce parameter complexity in wideband systems. Simulation results validate that our method outperforms existing deep learning benchmarks and closely approaches the theoretical upper bound of traditional algorithms with perfect CSI, demonstrating exceptional robustness in scenarios with limited pilot resources.

Future work will further validate the proposed framework through real-world over-the-air measurement campaigns and investigate its scalability under larger numbers of users and more complex multi-user interference conditions. In practical deployments, the achievable performance may be affected by non-ideal hardware factors, such as antenna mutual coupling, phase shifter quantization, and mismatch between measured channels and the adopted sparse channel model. Such studies will provide further insight into the robustness and practical feasibility of the proposed method.