Deep Learning-Enhanced Hybrid Beamforming Design with Regularized SVD Under Imperfect Channel Information

Abstract

We propose a low-complexity hybrid beamforming method for massive Multiple-Input Multiple-Output (MIMO) systems that is robust to Channel State Information (CSI) estimation errors. These errors stem from hardware impairments, pilot contamination, limited training, and fast fading, causing spectral-efficiency loss. However, existing hybrid beamforming solutions typically either assume near-perfect CSI or rely on greedy/black-box designs without an explicit mechanism to regularize the error-distorted singular modes, leaving a gap in unified, low-complexity, and theoretically grounded robustness. We unfold the Alternating Direction Method of Multipliers (ADMM) into a trainable Deep Learning (DL) network, termed DL-ADMM, to jointly optimize Radio-Frequency (RF) and baseband precoders and combiners. In DL-ADMM, the ADMM update mappings are learned (layer-wise parameters and projections) to amortize the joint RF/baseband optimization, whereas Regularized Singular Value Decomposition (RSVD) acts as an analytical regularizer that reshapes the observed channel’s singular values to suppress noise amplification under imperfect CSI. RSVD is integrated to stabilize singular modes and curb noise amplification, yielding a unified and scalable design. For ${\sigma }_e^2=0.1$, the proposed DL-ADMM-Reg achieves approximately 8–11 bits/s/Hz higher spectral efficiency than Orthogonal Matching Pursuit (OMP) at Signal-to-Noise Ratio (SNR) $=20$–40 dB, while remaining within <1 bit/s/Hz of the digital-optimal benchmark across both $(N_t,N_r)=(32,32)$ and $(64,64)$ settings. Simulations confirm higher spectral efficiency and robustness than OMP and Adaptive Phase Shifters (APSs).

1. Introduction

Massive Multiple-Input Multiple-Output (MIMO) technology has emerged as a cornerstone of modern wireless communication systems, providing remarkable improvements in spectral efficiency, energy efficiency, and overall system capacity. By employing large antenna arrays, massive MIMO enables simultaneous multi-user transmission and robust interference suppression, which are critical for beyond-5G and 6G networks [1]. To exploit these benefits, efficient beamforming strategies are indispensable. Hybrid beamforming, which integrates low-dimensional digital baseband processing with high-dimensional analog phase shifters, offers a promising compromise between hardware cost and spectral performance, especially in millimeter-wave (mmWave) and terahertz (THz) communication systems [2,3]. Despite its advantages, hybrid beamforming strongly depends on accurate Channel State Information (CSI). In practice, CSI acquisition is limited by pilot overhead, channel non-reciprocity, hardware impairments, and fast-fading environments, which lead to channel estimation errors [4,5]. These errors misguide the beamforming design, causing mismatches in precoders and combiners and ultimately degrading spectral efficiency and channel capacity. In massive MIMO with hundreds of antennas, such mismatches are amplified due to the high dimensionality of the channel, making robustness to CSI errors a critical requirement [6,7].

Several methods have been proposed to address hybrid beamforming design. Conventional algorithms such as Greedy Hybrid Precoding (GHP) [8], which sequentially selects analog beamformers to maximize array gain, Phase Pursuit (PP) [9], which refines phase shifter values in an iterative manner to approximate the optimal precoder, and Orthogonal Matching Pursuit (OMP) [10], which exploits sparse representations of the channel to iteratively select the best matching beams, achieve reasonable performance under perfect CSI. However, these methods face several shortcomings. They are highly sensitive to CSI accuracy; even small estimation errors can result in incorrect selection of baseband and RF chain values, leading to significant spectral efficiency degradation [11]. In addition, many designs rely on predefined analog codebooks, which restrict the search space and fail to adapt to dynamic environments, especially under mobility or non-stationary channels [10,12]. Another drawback is their high computational complexity, as iterative schemes like OMP scale poorly with large antenna arrays and quickly become impractical for real-time systems [13]. More recently, the Adaptive Phase Shifters (APSs) hybrid design [14] has been proposed to reduce the number of required phase shifters and hardware cost while maintaining good spectral efficiency, but robustness under imperfect CSI remains a challenge. Finally, even when channel estimation improves, residual mismatches between the actual and estimated channels remain unavoidable, limiting the performance of these approaches.

In summary, existing hybrid beamforming methods still exhibit two fundamental limitations. First, they are often highly sensitive to noise and CSI estimation errors, where small perturbations can distort the effective singular subspaces and lead to beam misalignment and pronounced spectral-efficiency loss. Second, their robustness is limited in dynamic channels, since codebook/greedy selection and iterative refinement may fail to adapt reliably under mobility and non-stationary propagation, and can converge to suboptimal solutions in large arrays. These issues motivate a design that is both computationally efficient and explicitly regularizes the error-corrupted channel structure.

1.1. Deep Learning-Based Hybrid Beamforming

To overcome these limitations, data-driven strategies have recently gained traction. Machine Learning (ML) and Deep Learning (DL)-based methods have been applied to hybrid beamforming, CSI estimation, and channel prediction [15,16,17]. Although DL frameworks can approximate nonlinear mappings and significantly reduce computational burden, they also have weaknesses. Many such solutions operate as black boxes, lacking interpretability and theoretical guarantees, which makes their performance difficult to predict under varying CSI error statistics. Recent DL-based hybrid beamforming has been explored via supervised networks that learn CSI-to-hybrid-precoder/combiner mappings for mmWave MIMO [18] and via two-stage neural architectures for hybrid precoder design in very-large-scale massive MIMO systems [19]. These predictor-centric approaches motivate our model-driven unfolded design with explicit spectral stabilization to improve robustness under CSI uncertainty. Furthermore, DL models often suffer from limited generalization; those trained on specific datasets may fail to perform reliably in different propagation environments. In addition, the training process itself is demanding, requiring large amounts of labeled data and substantial computational resources, which may not always be practical in real-world deployments.

While standard DL can learn an end-to-end mapping from the observed channel to hybrid beamformers, it is often insufficient under imperfect CSI because it does not explicitly control the conditioning of the error-corrupted channel matrix. In particular, CSI errors can distort the singular spectrum, where small (and noise-sensitive) singular modes induce noise amplification and unstable beam alignment; a purely data-driven model may inadvertently fit these distortions, leading to limited generalization across changing error statistics and propagation environments. Parallel to DL-based research, mathematical regularization techniques have been proposed to address noise sensitivity in matrix factorization. Regularized Singular Value Decomposition (RSVD) has demonstrated effectiveness in suppressing noise amplification and stabilizing solutions to ill-posed problems [20,21]. However, RSVD alone does not address the joint optimization of digital and analog beamformers, and its direct application to hybrid beamforming remains underexplored.

The coexistence of these two streams—DL-based approximators and RSVD-based mathematical stabilizers—suggests a complementary integration. DL frameworks can provide efficient approximations for joint hybrid beamforming optimization, while RSVD offers theoretical robustness against channel estimation errors. Yet, to the best of our knowledge, limited work has explored a unified design that leverages both strategies to achieve low-complexity and error-resilient hybrid beamforming in massive MIMO systems.

In summary, recent studies have investigated DL for hybrid beamforming through (i) end-to-end (E2E) supervised networks that directly learn a mapping from (possibly imperfect) CSI to analog/digital precoders and combiners [22], (ii) convolutional neural network (CNN)-assisted designs that exploit channel covariance/features to predict codebook indices or analog phase profiles [23], and (iii) unsupervised/label-free training that maximizes a differentiable spectral-efficiency objective [24]. While these data-driven approaches can reduce online computation, they are often treated as black-box predictors with limited interpretability and may generalize poorly when the CSI error statistics, signal-to-noise ratio (SNR), or array dimensions deviate from the training distribution; moreover, robustness is typically induced implicitly via data augmentation rather than by an explicit conditioning/regularization mechanism [22,23,24]. In contrast, our method is model-driven: the proposed deep-unfolded Alternating Direction Method of Multipliers (DL-ADMM) preserves the algorithmic structure of the constrained hybrid beamforming problem, where each layer corresponds to a principled ADMM update with learnable step sizes/penalties and closed-form projection operators, yielding interpretable intermediate variables (primal/dual iterates) and predictable complexity scaling. Furthermore, RSVD is introduced as an analytical regularizer that reshapes the error-distorted singular modes of the observed channel to suppress noise amplification, thereby providing a theoretically grounded robustness component that complements the learned optimizer.

1.2. Motivation and Contribution

Motivated by these gaps, this paper introduces a novel Regularized Hybrid Beamforming framework that integrates a DL-enhanced Alternating Direction Method of Multipliers (ADMM) [25] with RSVD theory. By simultaneously optimizing the baseband and RF chains while regularizing the singular values of the observed channel, the proposed scheme mitigates the detrimental effects of estimation mismatches, reduces computational complexity, and enhances spectral efficiency across a wide range of system conditions.

The main contributions of this work are summarized as follows:

• DL-Enhanced ADMM for Hybrid Beamforming: We enhance the ADMM [25] by embedding it within a DL-based architecture, termed DL-ADMM, enabling low-complexity joint optimization of transmitter and receiver baseband and RF chain components.
• RSVD-Based Robustness: We incorporate RSVD theory [20] into the hybrid beamforming framework to suppress noise amplification and strengthen singular values of the observed channel, thereby mitigating the impact of CSI estimation errors.
• Theoretical Foundation: We establish new theoretical results (Theorem 1 and Corollary 1) that prove RSVD achieves lower noise sensitivity compared to standard SVD, providing rigorous justification for its integration into hybrid beamforming.
• Complexity and Efficiency Analysis: We analyze the computational complexity of the proposed scheme and compare it against conventional methods such as OMP and APS, demonstrating that DL-ADMM achieves reduced per-iteration complexity without sacrificing performance.
• Extensive Simulations: Through comprehensive experiments under diverse antenna configurations, SNR ranges, and channel estimation error variances, we validate that the proposed DL-ADMM-Reg outperforms baseline approaches in terms of spectral efficiency, robustness, and stability.

The remainder of this paper is structured as follows. Section 2 introduces the system model and problem formulation for spectral efficiency under imperfect CSI. Section 3 presents the proposed regularized hybrid beamforming framework, including the DL-ADMM scheme and RSVD-based regularization. Section 4 provides extensive simulation results to benchmark the proposed method against conventional algorithms. Finally, Section 5 concludes the paper and outlines potential directions for future research.

2. System Model and Spectral Efficiency

2.1. System Model

Consider a point-to-point massive MIMO system with $N_t$ transmit antennas and $N_r$ receive antennas, designed to support $N_s$ independent data streams as depicted in Figure 1. Hybrid beamforming is adopted to reduce the number of required RF chains, which are denoted by $N_{RFt}$ and $N_{RFr}$ for the transmitter and receiver, respectively. These values satisfy the practical constraint $N_s\le N_{RFt}\le N_t$ and $N_s\le N_{RFr}\le N_r$. The baseband equivalent input–output relation is expressed as

$$\tilde{\mathbf{y}}=\sqrt{\varrho }{\mathbf{W}}_{BB}^{*}{\mathbf{W}}_{RF}^{*}\mathbf{H}{\mathbf{F}}_{RF}{\mathbf{F}}_{BB}\mathbf{x}+{\mathbf{W}}_{BB}^{*}{\mathbf{W}}_{RF}^{*}\mathbf{n},$$

(1)

where $\tilde{\mathbf{y}}\in {\mathbb{C}}^{N_s\times 1}$ is the processed received signal, $\varrho$ is the average transmit power, and $\mathbf{x}\in {\mathbb{C}}^{N_s\times 1}$ represents the transmitted symbols with normalized power. The additive noise vector is $\mathbf{n}\sim \mathcal{CN}(0,{\sigma }^2{\mathbf{I}}_{N_r})$.

Hybrid beamforming employs both analog and digital components: the transmitter uses analog precoding ${\mathbf{F}}_{RF}\in {\mathbb{C}}^{N_t\times N_{RFt}}$ and digital precoding ${\mathbf{F}}_{BB}\in {\mathbb{C}}^{N_{RFt}\times N_s}$, while the receiver uses analog combining ${\mathbf{W}}_{RF}\in {\mathbb{C}}^{N_r\times N_{RFr}}$ and digital combining ${\mathbf{W}}_{BB}\in {\mathbb{C}}^{N_{RFr}\times N_s}$. Typically, the analog precoder/combiner entries are constrained to have unit modulus due to hardware phase shifters, i.e., $|{\left[{\mathbf{F}}_{RF}\right]}_{i,j}|=1$ and $|{\left[{\mathbf{W}}_{RF}\right]}_{i,j}|=1$ [10,13].

The physical channel $\mathbf{H}$ is modeled by a clustered geometric model that captures multipath propagation in mmWave and sub-THz systems:

$$\mathbf{H}=\sqrt{{\displaystyle \frac{N_r{N}_t}{N_{cl}{N}_{ray}}}}\sum _{i=1}^{N_{cl}}\sum _{j=1}^{N_{ray}}{\alpha }_{ij}{\mathbf{a}}_r({\varphi }_{ij}^r,{\theta }_{ij}^r){\mathbf{a}}_t^H({\varphi }_{ij}^t,{\theta }_{ij}^t),$$

(2)

where $N_{cl}$ and $N_{ray}$ denote the number of scattering clusters and rays per cluster, respectively. The path gain of the j-th ray in the i-th cluster is represented by ${\alpha }_{ij}\sim \mathcal{CN}(0,{\sigma }_{\alpha }^2)$. The vectors ${\mathbf{a}}_t(·)$ and ${\mathbf{a}}_r(·)$ are the array response vectors of the transmit and receive UPAs, parameterized by the azimuth and elevation angles of departure and arrival, respectively.

To explicitly reflect the channel conditions, we note that the clustered model in (2) implies an angular-domain sparsity typical for mmWave/sub-THz propagation, where only a small number of dominant paths contribute to the channel. Let ${\mathbf{A}}_t\in {\mathbb{C}}^{N_t\times G_t}$ and ${\mathbf{A}}_r\in {\mathbb{C}}^{N_r\times G_r}$ denote transmit/receive array-response dictionaries constructed on quantized AoD/AoA grids. The channel can be equivalently expressed in a sparse beamspace form as

$$\mathbf{H}={\mathbf{A}}_r{\mathbf{H}}_b{\mathbf{A}}_t^H,{\parallel {\mathbf{H}}_b\parallel }_0\approx N_{cl}{N}_{ray}\ll N_r{N}_t,$$

(3)

where ${\mathbf{H}}_b$ is sparse with non-zero entries corresponding to the dominant propagation directions.

Regarding noise statistics, the receiver noise is modeled as spatially white complex Gaussian, $\mathbf{n}\sim \mathcal{CN}(\mathbf{0},{\sigma }^2{\mathbf{I}}_{N_r})$, and the post-combining effective noise becomes

$$\tilde{\mathbf{n}}={\mathbf{W}}_{BB}^{*}{\mathbf{W}}_{RF}^{*}\mathbf{n}\sim \mathcal{CN}\left(\mathbf{0},{\sigma }^2{\mathbf{W}}_{BB}^{*}{\mathbf{W}}_{RF}^{*}{\mathbf{W}}_{RF}{\mathbf{W}}_{BB}\right),$$

(4)

which reduces to $\mathcal{CN}(\mathbf{0},{\sigma }^2{\mathbf{I}}_{N_s})$ under the common semi-unitary normalization

$${\mathbf{W}}_{BB}^{*}{\mathbf{W}}_{RF}^{*}{\mathbf{W}}_{RF}{\mathbf{W}}_{BB}={\mathbf{I}}_{N_s}.$$

(5)

Finally, to model imperfect channel knowledge, we assume that beamformer design is based on an estimated channel

$$\widehat{\mathbf{H}}=\mathbf{H}+\mathbf{E},\mathrm{vec}\left(\mathbf{E}\right)\sim \mathcal{CN}\left(\mathbf{0},{\sigma }_e^2{\mathbf{I}}_{N_r{N}_t}\right),$$

(6)

where ${\sigma }_e^2$ controls the CSI error level (i.e., the estimation NMSE). Consequently, all subsequent optimization/design steps use $\widehat{\mathbf{H}}$ rather than the ideal $\mathbf{H}$, motivating the proposed robustness mechanisms.

2.2. Achievable Spectral Efficiency

Based on (1) and (2), the effective precoder and combiner can be written as $\mathbf{F}={\mathbf{F}}_{RF}{\mathbf{F}}_{BB}$ and $\mathbf{W}={\mathbf{W}}_{RF}{\mathbf{W}}_{BB}$. Ideally, the optimal solution is obtained by aligning these matrices with the right and left singular vectors of the channel, respectively. By computing the singular value decomposition (SVD) of $\mathbf{H}={\mathbf{USV}}^{*}$, the optimal digital-only precoder and combiner are defined as ${\mathbf{F}}_{\mathrm{opt}}=\mathbf{V}(:,1:N_s)$ and ${\mathbf{W}}_{\mathrm{opt}}=\mathbf{U}(:,1:N_s)$. However, the hybrid constraints prohibit direct implementation, and the near-optimal hybrid design must solve

$$\left\{\begin{array}{c}{\min }_{({\mathbf{W}}_{BB},{\mathbf{W}}_{RF})}{\parallel {\mathbf{W}}_{\mathrm{opt}}-{\mathbf{W}}_{RF}{\mathbf{W}}_{BB}\parallel }_F,\hfill \\ s.t.\hfill \\ \parallel {\mathbf{W}}_{RF}{\mathbf{W}}_{BB}{\parallel }_F=\sqrt{N_s},|{\mathbf{W}}_{R{F}_{i,j}}|=1.\hfill \end{array}\right.\left\{\begin{array}{c}{\min }_{({\mathbf{F}}_{BB},{\mathbf{F}}_{RF})}{\parallel {\mathbf{F}}_{\mathrm{opt}}-{\mathbf{F}}_{RF}{\mathbf{F}}_{BB}\parallel }_F,\hfill \\ s.t.\hfill \\ \parallel {\mathbf{F}}_{RF}{\mathbf{F}}_{BB}{\parallel }_F=\sqrt{N_s},|{\mathbf{F}}_{R{F}_{i,j}}|=1.\hfill \end{array}\right.$$

(7)

where ${\parallel .\parallel }_F$ denotes the Frobenius norm. Such approximations are common in hybrid beamforming design, and various algorithms such as OMP [10] have been proposed to solve (7) with different complexity–performance trade-offs.

Once the hybrid precoders and combiners are obtained, the spectral efficiency is defined as

$$R={log}_2\left(\det \left({\mathbf{I}}_{N_s}+{\displaystyle \frac{\varrho }{N_s}}{\mathbf{R}}_n^{-1}\mathbf{G}{\mathbf{G}}^{*}\right)\right),$$

(8)

where $\mathbf{G}={\mathbf{W}}^{*}\mathbf{H}\mathbf{F}$ is the effective channel after hybrid processing, and ${\mathbf{R}}_n={\sigma }^2{\mathbf{W}}^{*}\mathbf{W}$ is the equivalent noise covariance.

When channel estimation errors occur, the observed channel is modeled as

$${\mathbf{H}}_{\mathrm{obs}}=\mathbf{H}+{\mathbf{H}}_e,$$

(9)

where ${\mathbf{H}}_e$ represents the estimation error matrix with variance ${\sigma }_e^2$ [4,5]. Performing SVD on ${\mathbf{H}}_{\mathrm{obs}}=\widehat{\mathbf{U}}\widehat{\mathbf{S}}\widehat{\mathbf{V}}*$ leads to inaccurate beamforming solutions $\widehat{\mathbf{F}}$ and $\widehat{\mathbf{W}}$. The corresponding achievable spectral efficiency is

$$\widehat{R}={log}_2\left(\det \left({\mathbf{I}}_{N_s}+{\displaystyle \frac{\varrho }{N_s}}{\widehat{\mathbf{R}}}_n^{-1}\widehat{\mathbf{G}}{\widehat{\mathbf{G}}}^{*}\right)\right),$$

(10)

with $\widehat{\mathbf{G}}=\widehat{\mathbf{W}}*\mathbf{H}\widehat{\mathbf{F}}$ and ${\widehat{\mathbf{R}}}_n={\sigma }^2\widehat{\mathbf{W}}*\widehat{\mathbf{W}}$.

In the case of perfect CSI, the spectral efficiency in (8) using ${\mathbf{F}}_{\mathrm{opt}},{\mathbf{W}}_{\mathrm{opt}}$ serves as the upper bound and is often referred to as the Optimal performance. Under imperfect CSI, the spectral efficiency is inevitably reduced to a lower bound that we term Mismatch, which no conventional algorithm can surpass [6,26,27]. This highlights the inefficiency of traditional approaches under estimation errors and motivates the development of robust and regularized beamforming strategies.

2.3. Conventional ADMM for Hybrid Beamforming

In this section, we detail the conventional ADMM [25] procedure used to approximate the optimal unconstrained singular-vector beams under hybrid (unit–modulus) constraints.

• Problem restatement

Given a target matrix $\mathbf{O}\in {\mathbb{C}}^{a\times c}$ (e.g., ${\mathbf{W}}_{\mathrm{opt}}$ or ${\mathbf{F}}_{\mathrm{opt}}$ from the SVD of $\mathbf{H}$), the hybrid factorization seeks

$$\underset{\mathbf{R},\mathbf{B}}{\min }\parallel \mathbf{O}-\mathbf{R}\mathbf{B}{\parallel }_F^2{\mathrm{s}.\mathrm{t}.\parallel \mathbf{R}\mathbf{B}\parallel }_F=\wp ,|{\left[\mathbf{R}\right]}_{i,j}|=1,$$

(11)

where $\wp$ is the normalization (power) constant, $\mathbf{R}\in {\mathbb{C}}^{a\times b}$ represents the analog (unit–modulus) matrix, and $\mathbf{B}\in {\mathbb{C}}^{b\times c}$ is the digital baseband matrix.

Our goal is to obtain a hardware-feasible hybrid realization that closely approximates the SVD-derived fully digital target matrix $\mathbf{O}\in {\mathbb{C}}^{a\times c}$ (e.g., ${\mathbf{F}}_{\mathrm{opt}}$ or ${\mathbf{W}}_{\mathrm{opt}}$). Specifically, we seek matrices $\mathbf{R}\in {\mathbb{C}}^{a\times b}$ and $\mathbf{B}\in {\mathbb{C}}^{b\times c}$ such that the product $\mathbf{R}\mathbf{B}$ minimizes the approximation error ${\parallel \mathbf{O}-\mathbf{R}\mathbf{B}\parallel }_F^2$, while satisfying the unit-modulus constraint ${|\left[\mathbf{R}\right]}_{i,j}|=1$ imposed by analog phase shifters and the power-normalization constraint ${\parallel \mathbf{R}\mathbf{B}\parallel }_F=\wp$ (where $\wp$ is a known constant). Therefore, (11) provides a clear restatement of the hybrid beamforming/combining design as a constrained matrix factorization problem whose solution yields the closest feasible hybrid approximation to $\mathbf{O}$.

• Augmented Lagrangian

Introduce an auxiliary variable $\mathbf{Z}\in {\mathbb{C}}^{a\times c}$ and multiplier $\mathbf{Y}\in {\mathbb{C}}^{a\times c}$ to decouple the bilinear term via the constraint $\mathbf{Z}=\mathbf{R}\mathbf{B}$. The (scaled) augmented Lagrangian reads

$${\mathcal{L}}_{\rho }(\mathbf{R},\mathbf{B},\mathbf{Z},\mathbf{Y})=\parallel \mathbf{Z}-\mathbf{O}{\parallel }_F^2+{\displaystyle \frac{\rho }{2}}\parallel \mathbf{Z}-\mathbf{R}\mathbf{B}+\frac{1}{\rho }\mathbf{Y}{\parallel }_F^2,$$

(12)

with penalty $\rho >0$.

• Z-update (quadratic projection)

Fix $(\mathbf{R},\mathbf{B},\mathbf{Y})$. Minimizing (12) with respect to $\mathbf{Z}$ yields a closed form:

$${\mathbf{Z}}^{(k+1)}=arg\underset{\mathbf{Z}}{\min }\parallel \mathbf{Z}-\mathbf{O}{\parallel }_F^2+{\displaystyle \frac{\rho }{2}}\parallel \mathbf{Z}-\left({\mathbf{R}}^{\left(k\right)}{\mathbf{B}}^{\left(k\right)}-\frac{1}{\rho }{\mathbf{Y}}^{\left(k\right)}\right){\parallel }_F^2={\displaystyle \frac{\mathbf{O}+\rho \left({\mathbf{R}}^{\left(k\right)}{\mathbf{B}}^{\left(k\right)}-\frac{1}{\rho }{\mathbf{Y}}^{\left(k\right)}\right)}{1+\rho }}.$$

(13)

• B-update (least squares with power normalization)

Fix $(\mathbf{R},\mathbf{Z},\mathbf{Y})$. The digital matrix solves a regularized least-squares problem:

$${\mathbf{B}}_{\mathrm{unc}}^{(k+1)}=arg\underset{\mathbf{B}}{\min }\parallel {\mathbf{Z}}^{(k+1)}-{\mathbf{R}}^{\left(k\right)}\mathbf{B}+\frac{1}{\rho }{\mathbf{Y}}^{\left(k\right)}{\parallel }_F^2={\left({\mathbf{R}}^{\left(k\right)\ast }{\mathbf{R}}^{\left(k\right)}\right)}^{-1}{\mathbf{R}}^{\left(k\right)\ast }\left({\mathbf{Z}}^{(k+1)}+\frac{1}{\rho }{\mathbf{Y}}^{\left(k\right)}\right).$$

(14)

Enforce the Frobenius-norm power constraint in (11) by scaling:

$${\mathbf{B}}^{(k+1)}={\mathbf{B}}_{\mathrm{unc}}^{(k+1)}{\displaystyle \frac{\wp }{\parallel {\mathbf{R}}^{\left(k\right)}{\mathbf{B}}_{\mathrm{unc}}^{(k+1)}{\parallel }_F}}.$$

(15)

• R-update (unit–modulus projection)

Fix $(\mathbf{B},\mathbf{Z},\mathbf{Y})$. Disregarding (temporarily) the modulus constraint, the analog update solves

$${\mathbf{R}}_{\mathrm{ls}}^{(k+1)}=arg\underset{\mathbf{R}}{\min }\parallel {\mathbf{Z}}^{(k+1)}-\mathbf{R}{\mathbf{B}}^{(k+1)}+\frac{1}{\rho }{\mathbf{Y}}^{\left(k\right)}{\parallel }_F^2=\left({\mathbf{Z}}^{(k+1)}+\frac{1}{\rho }{\mathbf{Y}}^{\left(k\right)}\right){\mathbf{B}}^{(k+1)\ast }{\left({\mathbf{B}}^{(k+1)}{\mathbf{B}}^{(k+1)\ast }\right)}^{-1}.$$

(16)

Project ${\mathbf{R}}_{\mathrm{ls}}^{(k+1)}$ element-wise onto the complex unit circle to enforce ${|\left[\mathbf{R}\right]}_{i,j}|=1$:

$${\mathbf{R}}^{(k+1)}=exp\left(j\angle \left({\mathbf{R}}_{\mathrm{ls}}^{(k+1)}\right)\right),\mathrm{i}.\mathrm{e}.,{\left[{\mathbf{R}}^{(k+1)}\right]}_{i,j}=e^{j\angle \left({\left[{\mathbf{R}}_{\mathrm{ls}}^{(k+1)}\right]}_{i,j}\right)}.$$

(17)

• Dual update (scaled multipliers)

Finally update the Lagrange multiplier:

$${\mathbf{Y}}^{(k+1)}={\mathbf{Y}}^{\left(k\right)}+\rho \left({\mathbf{Z}}^{(k+1)}-{\mathbf{R}}^{(k+1)}{\mathbf{B}}^{(k+1)}\right).$$

(18)

• Stopping rule and feasibility

The iterations (13)–(18) continue until primal and dual residuals fall below tolerances, e.g.,

$$\underset{\mathrm{primal} \mathrm{residual}}{\underbrace{\parallel {\mathbf{Z}}^{(k+1)}-{\mathbf{R}}^{(k+1)}{\mathbf{B}}^{(k+1)}{\parallel }_F}}\le {\epsilon }_{\mathrm{pri}},\underset{\mathrm{dual} \mathrm{residual}}{\underbrace{\rho \parallel {\mathbf{Z}}^{(k+1)}-{\mathbf{Z}}^{\left(k\right)}{\parallel }_F}}\le {\epsilon }_{\mathrm{dual}}.$$

(19)

At convergence, scale $\mathbf{B}$ once more if necessary to satisfy ${\parallel \mathbf{R}\mathbf{B}\parallel }_F=\wp$ exactly, cf. (15).

The conventional ADMM updates above require repeated large matrix multiplications and inversions in (14)–(16), careful penalty selection $\rho$, and full reruns for every new channel realization; under imperfect CSI, the iterates may converge to sub-optimal factorizations that reduce spectral efficiency. The DL-unfolded counterpart replaces hand-tuned quantities (e.g., $\rho$, preconditioners, projections) with trainable mappings, amortizes computation across channel instances, and incorporates robustness terms aligned with the SVD structure of the observed channel, thereby improving convergence speed and resilience to estimation errors while keeping per-layer operations inexpensive.

3. DL-ADMM Assistant Regularized Hybrid Beamforming

This section aims to address the issue of wasted computational complexity in hybrid beamforming design under imperfect CSI conditions, while simultaneously enhancing the achievable channel capacity. To achieve this, we integrate DL into the ADMM, yielding a DL-ADMM scheme that leverages both data-driven approximation and model-based optimization.

3.1. DL-ADMM Scheme

The hybrid beamforming design problem can be formulated as the constrained optimization task

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\min }_{(\mathbf{B},\mathbf{R})}{\parallel \mathbf{O}-\mathbf{RB}\parallel }_F,\hfill \\ {\mathrm{s}.\mathrm{t}.\parallel \mathbf{RB}\parallel }_F=\wp ,\left|{\mathbf{R}}_{i,j}\right|=1,\hfill \end{array}\right.\end{array}$$

(20)

where $\wp$ is the normalization (power) constant, $\mathbf{O}\in {\mathbb{C}}^{a\times c}$ represents the target matrix (either the optimal precoder or combiner), $\mathbf{R}\in {\mathbb{C}}^{a\times b}$ is the analog beamformer constrained to unit-modulus entries, and $\mathbf{B}\in {\mathbb{C}}^{b\times c}$ is the corresponding digital component. This problem is inherently non-convex due to the modulus constraints and coupled variables [10,12]. Traditional algorithms such as OMP and alternating minimization are computationally demanding, and they require multiple iterations that scale poorly with antenna array size [12,13].

To efficiently solve (20), we adopt a DL-enhanced ADMM framework. By integrating DL into ADMM, the iterative update rules are unfolded into a layer-wise neural network architecture, where trainable parameters are learned offline and directly applied at runtime, thus drastically reducing online computational complexity. Hence, reformulating (20) through the augmented Lagrangian leads to

$$\begin{array}{c}\hfill \underset{(\mathbf{R},\mathbf{B},\mathbf{Z},\mathbf{Y})}{\min }{\parallel \mathbf{Z}-\mathbf{O}\parallel }_F^2+\langle \mathbf{Y},\mathbf{Z}-\mathbf{R}\mathbf{B}\rangle +{\displaystyle \frac{\rho }{2}}{\parallel \mathbf{Z}-\mathbf{R}\mathbf{B}\parallel }_F^2,\end{array}$$

(21)

where $\rho$ is the penalty parameter, $\mathbf{Z}$ is an auxiliary variable, and $\mathbf{Y}$ is the Lagrange multiplier. The optimization proceeds by alternating updates for $\mathbf{Z}$, $\mathbf{R}$, and $\mathbf{B}$, followed by dual variable updates for $\mathbf{Y}$. Each update step is mapped into a neural network layer, with trainable weights ${\rho }^{\left(k\right)}$, ${\mathbf{\Omega }}^{\left(k\right)}$, and initializations learned from data.

Algorithm 1 summarizes the DL-ADMM procedure, where each iteration corresponds to one network layer. This unrolling strategy allows the network to inherit the interpretability and convergence properties of ADMM while learning problem-specific parameters that improve performance and robustness to imperfect CSI. Such “algorithm unfolding” methods have recently gained wide attention in wireless communications, as they achieve near-optimal accuracy with dramatically lower runtime compared to conventional iterative solvers.

Algorithm 1 DL-ADMM

Require: O, number of layers K, and dimensions (a, b, c)

Ensure: R, B

1:  Initialize $\mathbf{R}\in {\mathbb{C}}^{a\times b}$ and $\mathbf{B}\in {\mathbb{C}}^{b\times c}$ with random values.

2:  for  $k=1:K$ do

3:   ${\mathbf{Z}}^{\left(k\right)}\leftarrow \left(\mathbf{O}+{\rho }^{\left(k\right)}\odot (\mathbf{R}\mathbf{B}-{\mathbf{Y}}^{\left(k\right)}\oslash {\rho }^{\left(k\right)})\right)\oslash \left(1+{\rho }^{\left(k\right)}\right)$

4:   $\mathbf{R}\leftarrow exp(i\times \mathrm{angle}\left(({\mathbf{Z}}^{\left(k\right)}+{\mathbf{Y}}^{\left(k\right)}\oslash {\rho }^{\left(k\right)}){\mathbf{\Omega }}^{\left(k\right)}\right))$

5:   $\mathbf{B}\leftarrow \Psi (\frac{1}{a}{\mathbf{IR}}^{*}({\mathbf{Z}}^{\left(k\right)}+{\mathbf{Y}}^{\left(k\right)}\oslash {\rho }^{\left(k\right)}),\mathbf{R})$

6:   ${\mathbf{Y}}^{(k+1)}\leftarrow {\mathbf{Y}}^{\left(k\right)}+{\rho }^{\left(k\right)}\odot ({\mathbf{Z}}^{\left(k\right)}-\mathbf{R}\mathbf{B})$

7:  end for

Here, $\odot$ and $\oslash$ denote element-wise multiplication and division, respectively, and $\frac{1}{a}{\mathbf{IR}}^{*}$ is a low-complexity approximation for the pseudo-inverse of $\mathbf{R}$. The trainable variables include ${\rho }^{\left(k\right)}\in {\mathbb{R}}^{a\times c}$, ${\mathbf{Y}}^{\left(1\right)}\in {\mathbb{C}}^{a\times c}$, and ${\mathbf{\Omega }}^{\left(k\right)}\in {\mathbb{C}}^{c\times b}$. The activation function $\Psi (·)$ ensures the Frobenius norm constraint on the digital matrix:

$$\begin{array}{c}\hfill \Psi (\mathbf{B},\mathbf{R})=\mathbf{B}{\displaystyle \frac{\wp }{{\parallel \mathbf{R}\mathbf{B}\parallel }_F}}.\end{array}$$

(22)

The loss function is designed to jointly minimize the approximation error to the optimal unconstrained beamformers and to preserve the effective channel structure:

$$\begin{array}{ccc}\hfill loss& =& \parallel {\mathbf{W}}_{RF}{\mathbf{W}}_{BB}-{\mathbf{W}}_{\mathrm{opt}}{\parallel }_F+{\parallel {\mathbf{F}}_{RF}{\mathbf{F}}_{BB}-{\mathbf{F}}_{\mathrm{opt}}\parallel }_F\hfill \\ & +& \parallel {\mathbf{W}}_{BB}^{*}{\mathbf{W}}_{RF}^{*}{\mathbf{H}}_{obs}{\mathbf{F}}_{RF}{\mathbf{F}}_{BB}-\overline{\mathbf{S}}{\parallel }_F,\hfill \end{array}$$

(23)

wherein $\wp =\sqrt{N_s}$, $\overline{\mathbf{S}}=\mathbf{S}(1:N_s,1:N_s)$, and ${\mathbf{W}}_{\mathrm{opt}}=\mathbf{U}(:,1:N_s)$, ${\mathbf{F}}_{\mathrm{opt}}=\mathbf{V}(:,1:N_s)$ are derived from the SVD of $\mathbf{H}={\mathbf{USV}}^{*}$. Two parallel instances of Algorithm 1 are trained simultaneously: one with $\mathbf{O}={\mathbf{W}}_{\mathrm{opt}}$ and dimensions $(a,b,c)=(N_r,N_{RFr},N_s)$, and the other with $\mathbf{O}={\mathbf{F}}_{\mathrm{opt}}$ and $(a,b,c)=(N_t,N_{RFt},N_s)$. This DL-ADMM scheme provides a balance between conventional optimization and data-driven learning. It reduces per-iteration complexity, accelerates convergence, and remains robust to CSI imperfections, as confirmed in prior works on model-driven DL for communications [28].

To facilitate understanding of Algorithm 1, we added Figure 2 to visualize the computation carried out in a representative unfolded layer k. The figure shows that each layer corresponds to one ADMM iteration and introduces only two learnable quantities: the layer-dependent penalty parameter ${\rho }^{\left(k\right)}$ and the scaling matrix ${\Omega }^{\left(k\right)}$ employed within the analog/constraint-update module, while all other steps remain deterministic and follow closed-form ADMM updates and projection operators. Therefore, the number of trainable parameters per layer is determined solely by the degrees of freedom of ${\rho }^{\left(k\right)}$ and ${\Omega }^{\left(k\right)}$, using the same parameterization across layers. Regarding initialization, $\mathbf{R}$ and $\mathbf{B}$ are randomly initialized, the dual variable ${\mathbf{Y}}^{\left(0\right)}$ is initialized to zero, and the layer-wise trainable variables are initialized by setting ${\rho }^{\left(k\right)}=1$ and drawing ${\Omega }^{\left(k\right)}$ randomly for all $k=1,\dots ,K$, after which they are optimized jointly in an end-to-end manner via gradient descent.

3.2. Regularized System

This section demonstrates how RSVD theory [20] can mitigate noise and minimize the impact of channel estimation errors by redesigning the singular value. To clarify these concepts, the following theorem and corollary are presented.

Theorem 1. 

Let $\mathbf{H}\in {\mathbb{C}}^{M\times N}$ have singular value decomposition $\mathbf{H}=\mathbf{U}{\mathbf{S}\mathbf{V}}^{*}$, where $\mathbf{S}=diag({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_r)$. For the system $\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{n}$ with Gaussian noise $\mathbf{n}$, the Regularized SVD (RSVD) solution is:

$$\begin{array}{c}\hfill {\mathbf{x}}_{RSVD}=\mathbf{V}{\mathbf{S}}_{reg}^{-1}{\mathbf{U}}^{*}\mathbf{y},{\mathbf{S}}_{reg}^{-1}=\mathit{diag}\left({\displaystyle \frac{{\sigma }_i}{{\sigma }_i^{1+{\alpha }_i}+\lambda }}\right),\end{array}$$

(24)

where $0<\lambda \le {\min }_i\left\{{\sigma }_i^2\right\}$. If ${\alpha }_i\ge {log}_{{\sigma }_i}({\sigma }_i^2-\lambda )-1$, the RSVD solution reduces noise sensitivity, satisfying:

$$\parallel {\mathbf{x}}_{RSVD}{-\mathbf{x}\parallel }_2\le {\parallel {\mathbf{x}}_{SVD}-\mathbf{x}\parallel }_2,$$

where ${\mathbf{x}}_{SVD}=\mathbf{V}{\Sigma }^{-1}{\mathbf{U}}^{*}\mathbf{y}$ is the standard SVD solution.

Proof. 

The least-squares solution ${\mathbf{x}}_{SVD}$ using the Moore–Penrose pseudoinverse of $\mathbf{H}$ is ${\mathbf{x}}_{SVD}=\mathbf{V}{\mathbf{S}}^{-1}{\mathbf{U}}^{*}\mathbf{y},{\mathbf{S}}^{-1}=\mathrm{diag}\left({\displaystyle \frac{1}{{\sigma }_i}}\right),$ with error due to noise $\mathbf{n}$:

$$e_{SVD}={\mathbf{x}}_{SVD}-\mathbf{x}=\mathbf{V}\mathrm{diag}\left({\displaystyle \frac{1}{{\sigma }_i}}\right){\mathbf{U}}^{*}\mathbf{n}.$$

The RSVD solution is ${\mathbf{x}}_{RSVD}=\mathbf{V}{\mathbf{S}}_{\mathrm{reg}}^{-1}{\mathbf{U}}^{*}\mathbf{y},$ with error:

$$e_{RSVD}={\mathbf{x}}_{RSVD}-\mathbf{x}=\mathbf{V}\mathrm{diag}\left({\displaystyle \frac{{\sigma }_i}{{\sigma }_i^{1+{\alpha }_i}+\lambda }}\right)\mathbf{U}*\mathbf{n}.$$

Therefore, the error norms are $\parallel e_{SVD}{\parallel }_2=\sqrt{{\sum }_{i=1}^N{\left({\displaystyle \frac{u_i^{*}{n}_i}{{\sigma }_i}}\right)}^2}$, and $\parallel e_{RSVD}{\parallel }_2=\sqrt{{\sum }_{i=1}^N{\left({\displaystyle \frac{u_i^{*}{n}_i{\sigma }_i}{{\sigma }_i^{1+{\alpha }_i}+\lambda }}\right)}^2}.$ For ${\alpha }_i\ge {log}_{{\sigma }_i}({\sigma }_i^2-\lambda )-1$,

$${\sigma }_i^{1+{\alpha }_i}+\lambda \ge {\sigma }_i^2\mathrm{and}{\displaystyle \frac{{\sigma }_i}{{\sigma }_i^{1+{\alpha }_i}+\lambda }}\le {\displaystyle \frac{1}{{\sigma }_i}},$$

ensuring RSVD reduces noise amplification more effectively. □

Corollary 1. 

In massive MIMO systems, where channel estimation errors are modeled as ${\mathbf{H}}_{obs}=\widehat{\mathbf{U}}\widehat{\mathbf{S}}\widehat{\mathbf{V}}*$ ($\widehat{\mathbf{S}}=\mathit{diag}({\tilde{\sigma }}_1,\dots ,{\tilde{\sigma }}_r)$), the RSVD solution achieves an error norm given by:

$$\parallel {\mathbf{e}}_{RSVD}{\parallel }_2=\sqrt{\sum _{i=1}^N{\left({\displaystyle \frac{{\tilde{\sigma }}_i}{{\tilde{\sigma }}_i^{1+{\alpha }_i}+\lambda }}\right)}^2{\parallel {(\widehat{\mathbf{U}}*{\mathbf{H}}_e\mathbf{x})}_i+{(\widehat{\mathbf{U}}*\mathbf{n})}_i\parallel }^2}.$$

Under the condition ${\alpha }_i>{log}_{{\tilde{\sigma }}_i}({\tilde{\sigma }}_i^2-\lambda )-1$, RSVD effectively reduces noise amplification and channel estimation errors, ensuring:

$$\parallel {\mathbf{e}}_{RSVD}{\parallel }_2\le {\parallel {\mathbf{e}}_{SVD}\parallel }_2.$$

Theorem 1 and Corollary 1 show that RSVD improves standard SVD by adding a regularization term, enhancing stability and robustness. It limits noise amplification by modifying small singular values, balancing data fidelity, and noise suppression.

3.3. RSVD in Hybrid Beamforming

The previous subsection established a DL-ADMM framework for low-complexity hybrid beamforming design. However, when the observed channel ${\mathbf{H}}_{\mathrm{obs}}$ contains estimation errors, even well-designed hybrid beamformers ${\widehat{\mathbf{W}}}_{BB},{\widehat{\mathbf{W}}}_{RF},{\widehat{\mathbf{F}}}_{RF},{\widehat{\mathbf{F}}}_{BB}$ cannot fully mitigate the mismatch between the actual and observed singular structures of the channel. As a result, the achievable spectral efficiency degrades, especially at high SNR where error amplification in the singular values becomes dominant [5,6,20]. To address this issue, we propose a Regularized Hybrid Beamforming mechanism based on RSVD.

• Motivation

In standard SVD-based designs, the optimal precoders and combiners align with the largest singular modes of $\mathbf{H}$. However, under imperfect CSI, the estimated singular values ${\tilde{\sigma }}_i$ can deviate significantly from their true values. Small singular values in particular are highly sensitive to noise and estimation error, which can lead to severe mismatch in the effective channel $\mathbf{G}$ and hence reduced spectral efficiency. RSVD counteracts this problem by adding a bias (regularization term $\lambda$) that prevents small singular values from dominating the beamformer design. This approach is theoretically justified by Theorem 1 and Corollary 1, which establish improved conditioning and noise robustness.

• Proposed mechanism

As shown in Figure 3, the proposed design introduces a regularization filter $Reg$ between the data streams and the baseband precoder. The construction of $Reg$ proceeds in the following steps:

• Step 1: Initialization. Set $Reg={\mathbf{I}}_{N_s}$ as the initial filter. Compute the observed channel SVD, ${\mathbf{H}}_{\mathrm{obs}}=\widehat{\mathbf{U}}\widehat{\mathbf{S}}\widehat{\mathbf{V}}*$, and design ${\widehat{\mathbf{W}}}_{BB}$, ${\widehat{\mathbf{W}}}_{RF}$, ${\widehat{\mathbf{F}}}_{RF}$, and ${\widehat{\mathbf{F}}}_{BB}$ using DL-ADMM under imperfect CSI. At this stage, the hybrid beamformers approximate the unconstrained SVD solution, but are still vulnerable to error-induced spectral efficiency loss.
• Step 2: Regularization parameter design. Following (10), Theorem 1, and Corollary 1, we enforce a robustness inequality that guarantees improved performance:

  $$\begin{array}{c}\hfill \parallel {\overline{\mathbf{G}}}_{1}Re{g}^2{\overline{\mathbf{G}}}_2{\parallel }_{\infty }\ge \parallel {\overline{\mathbf{S}}}^2{\parallel }_{\infty }+P>{\parallel {\overline{\mathbf{S}}}^2\parallel }_{\infty },\end{array}$$

  (25)

  where $\overline{\mathbf{S}}=\widehat{\mathbf{S}}(1:N_s,1:N_s)$ is the truncated singular value matrix, ${\overline{\mathbf{G}}}_1={\mathbf{H}}_{\mathrm{obs}}{\widehat{\mathbf{F}}}_{RF}{\widehat{\mathbf{F}}}_{BB}$ and ${\overline{\mathbf{G}}}_2={\widehat{\mathbf{W}}}_{BB}*{\widehat{\mathbf{W}}}_{RF}*{\mathbf{H}}_{\mathrm{obs}}{\widehat{\mathbf{F}}}_{RF}{\widehat{\mathbf{F}}}_{BB}$ capture the effective channel components, and $P>0$ is a penalty term that can be chosen as a function of $f(\parallel {\overline{\mathbf{G}}}_1{\overline{\mathbf{G}}}_2-{\overline{\mathbf{S}}}^2{\parallel }_F)$. The use of the ∞-norm ensures that the dominant channel gains are reinforced, guaranteeing that the regularized design does not underperform compared to the unregularized case.

From (25), we derive the adjustment factor ${\alpha }_i$ for each singular mode as

$$\begin{array}{c}\hfill {\alpha }_i={log}_{{\tilde{\sigma }}_i}\left({\displaystyle \frac{\sqrt{{max}_i\left({\tilde{\sigma }}_i^2\right)+P}}{{\tilde{\sigma }}_i}}-\lambda /\sqrt{\parallel {\overline{\mathbf{G}}}_1{\parallel }_{\infty }{\parallel {\overline{\mathbf{G}}}_2\parallel }_{\infty }}\right)-1,\end{array}$$

(26)

where $\lambda$ is a tunable regularization constant. The parameter ${\alpha }_i$ ensures that the contribution of each singular mode is properly balanced under CSI uncertainty.

• Step 3: Construction of $Reg$. Using the ${\alpha }_i$ values and $\lambda$, define the diagonal regularization filter as

  $$\begin{array}{c}\hfill Reg=\mathrm{diag}\left({\displaystyle \frac{{\sigma }_1^{1+{\alpha }_1}+\lambda }{{\sigma }_1}},\dots ,{\displaystyle \frac{{\sigma }_{N_s}^{1+{\alpha }_{N_s}}+\lambda }{{\sigma }_{N_s}}}\right).\end{array}$$

  (27)
  This filter amplifies strong singular values while preventing weak and error-prone ones from dominating the effective channel.

• Resulting spectral efficiency

The new effective channel becomes

$$\breve{\mathbf{G}}=\widehat{\mathbf{W}}*\mathbf{H}\widehat{\mathbf{F}}Reg,$$

(28)

leading to the regularized spectral efficiency

$$\begin{array}{c}\hfill \breve{R}={log}_2\left(\det \left({\mathbf{I}}_{N_s}+{\displaystyle \frac{\varrho }{N_s}}{\widehat{\mathbf{R}}}_n^{-1}\breve{\mathbf{G}}\breve{\mathbf{G}}*\right)\right),\end{array}$$

(29)

with ${\widehat{\mathbf{R}}}_n={\sigma }^2\widehat{\mathbf{W}}*\widehat{\mathbf{W}}$. This value $\breve{R}$ represents an enhanced achievable rate that is robust to estimation errors and always outperforms (or at least matches) the unregularized mismatch rate $\widehat{R}$.

Based on extensive numerical experiments under varying SNR and CSI-error levels, we select the RSVD-related hyperparameters using a simple stability–complexity rule and keep them fixed across all tests unless otherwise stated. In practice, RSVD is applied to the estimated effective channel to obtain a compact low-rank spectral representation, and regularization is then introduced by softly damping the weakest (most noise-sensitive) spectral components before they are used in the subsequent beamformer updates. This “spectrum-aware” damping is implemented as a diagonal stabilizer that reduces the influence of ill-conditioned modes, thereby preventing error amplification caused by imperfect CSI while preserving the dominant signal subspace.

While ADMM unfolding and singular-value regularization are each well established in the literature, the novelty of our approach lies in their coupled co-design for hybrid beamforming under imperfect CSI. Specifically, the proposed DL–ADMM unrolling does not merely replace iterations with layers; it learns layer-wise ADMM quantities (e.g., penalty/step parameters and update mappings) to efficiently estimate the hybrid beamforming factors $({\mathbf{F}}_{\mathrm{RF}},{\mathbf{F}}_{\mathrm{BB}},{\mathbf{W}}_{\mathrm{RF}},{\mathbf{W}}_{\mathrm{BB}})$ in an amortized manner. These learned beamformers are then explicitly integrated with an RSVD-style stabilization stage that constructs a mode-adaptive diagonal filter, $\mathbf{Reg}$, to prevent noise and CSI mismatch from being amplified by ill-conditioned singular modes. Unlike prior works that apply SVD/RSVD as a standalone post-processing step, our stabilization is tied to the learned effective channel through the robustness constraint and the resulting mode-wise adjustment $\left\{{\epsilon }_i\right\}$, and it is further aligned with learning via a singular-structure preservation term in the training objective. Therefore, the two components do not form a simple sequential stack: the unfolded optimizer shapes the effective channel on which the RSVD stabilization operates, and the stabilization in turn suppresses weak, error-sensitive modes that would otherwise degrade the unfolded solution under estimation mismatch. This joint design yields a unified architecture that is simultaneously computationally efficient and robust by construction.

Note: The proposed DL-ADMM is trained and evaluated on channel realizations generated from the clustered geometric mmWave/sub-THz model in (2), which implies an angular-domain sparse propagation structure (few dominant paths). Imperfect CSI is incorporated during both training and testing by forming the observed channel as ${\mathbf{H}}_{\mathrm{obs}}=\mathbf{H}+{\mathbf{H}}_e$, where $\mathrm{vec}\left({\mathbf{H}}_e\right)\sim \mathcal{CN}(\mathbf{0},{\sigma }_e^2\mathbf{I})$ (cf. (6)). Since the network takes ${\mathbf{H}}_{\mathrm{obs}}$ as input and outputs hybrid precoders/combiners, the same trained model is applied at inference across the full operating SNR range considered in the following section; as SNR increases, the impact of CSI mismatch becomes more pronounced and the RSVD component yields larger robustness gains by suppressing error-sensitive singular modes, whereas at low SNR the performance gap naturally narrows in the noise-limited regime. Finally, the framework is not tied to a single channel model: if propagation deviates from sparse clustered conditions (e.g., richer scattering with larger $N_{cl}{N}_{ray}$, different angular spreads, or Rayleigh/Rician fading), the same unfolded architecture can be retrained/fine-tuned using samples drawn from the target channel distribution, enabling a controlled assessment of performance shifts under alternative channel statistics without modifying the proposed design.

4. Simulation Results

This section presents simulation results to evaluate the efficiency of the proposed DL-ADMM algorithm (Algorithm 1) and the regularized hybrid beamforming scheme called DL-ADMM-reg. We consider transmitter and receiver arrays of sizes $(N_t=32,N_r=32)$ and $(N_t=64,N_r=64)$. The channel follows a clustered model with $N_{\mathrm{cl}}=4$ clusters and $N_{\mathrm{ray}}=10$ rays per cluster. The angular spread is set to ${10}^{°}$, and the average transmit power is normalized to $\varrho =1$. A total of 2000 channels are generated, equally divided into training and testing sets. The DL-ADMM framework is implemented in TensorFlow, using a depth of five layers. Training is performed for 1000 epochs with a learning rate of 0.01 and default exponential decay. The activation $\Psi (·)$ enforces the Frobenius-norm constraint, while ${\rho }^{\left(k\right)}$ and ${\Omega }^{\left(k\right)}$ are initialized randomly and updated through gradient descent. Baseline methods include OMP [10], APS [14], and two benchmarks: Optimal (digital-only unconstrained precoding with perfect CSI) and Mismatch (digital-only unconstrained precoding under imperfect CSI).

To make the proposed RSVD-based regularization reproducible, we specify how the key parameters are set in practice. For the penalty parameter P in the RSVD filter, we select $P=2.6\parallel {\overline{\mathbf{G}}}_1{\overline{\mathbf{G}}}_2-{\overline{\mathbf{S}}}^2{\parallel }_F$ for the $32\times 32$ array and $P=2\parallel {\overline{\mathbf{G}}}_1{\overline{\mathbf{G}}}_2-{\overline{\mathbf{S}}}^2{\parallel }_F$ for the $64\times 64$ case; these values were determined empirically via numerical experiments to balance robustness (stability under CSI uncertainty) and efficiency (avoiding over-regularization). Moreover, we use $\lambda ={\min }_i\left\{{\sigma }_i^2\right\}$ for both settings to anchor the shrinkage to the weakest singular mode, which prevents ill-conditioned components from dominating the update, and we set ${\alpha }_i={log}_{{\sigma }_i}({\sigma }_i^2-\lambda )-1$ to adapt the mode-wise attenuation according to the relative strength of each singular component. In our experiments, keeping the same $\lambda$ and ${\alpha }_i$ rule across both array sizes yielded stable behavior without case-by-case tuning, while only P was mildly adjusted to reflect the change in problem scale.

To avoid attributing improvements to particular baseline configuration choices, we emphasize that the simulation settings above are adopted as representative mmWave/XL-MIMO test conditions and are not inherently tailored to favor the proposed method. The same channel model, dataset split, training protocol, and computational budget are applied consistently to all learning-based components, and all competing baselines are implemented under their standard assumptions and identical CSI conditions to ensure a fair comparison. Importantly, the proposed DL-ADMM and DL-ADMM-reg are not tied to the specific array sizes, clustered parameters, or training hyperparameters reported here; the same design and evaluation pipeline can be applied without modification to other antenna dimensions, propagation environments, or SNR/CSI-uncertainty regimes. Therefore, the reported gains should be interpreted as stemming from the proposed model-driven unfolding and explicit regularization mechanism rather than from any particular choice of the illustrative simulation configuration.

4.1. Convergence Analysis

To further assess the stability of the proposed DL-ADMM framework, we analyze the convergence behavior of the training and validation losses for both antenna configurations. Figure 4 illustrates the evolution of the loss function over five epochs for (a) $(32\times 32)$ and (b) $(64\times 64)$ antenna arrays. Each epoch consists of 200 iterations, meaning that the full training process involves only 1000 iterations in total. This setting demonstrates the efficiency of the proposed method, as convergence is achieved with a relatively small number of updates compared to conventional iterative optimization algorithms.

In both cases, the losses decrease sharply within the first two epochs (i.e., within the first 400 iterations) and gradually stabilize thereafter, indicating rapid convergence of the unfolded network. An important observation is that the training and validation loss curves remain consistently close throughout the training process. This demonstrates that the model generalizes well to unseen data and avoids the common problem of overfitting. Specifically, in the $(32\times 32)$ case, both losses converge to values around 7–8, while in the $(64\times 64)$ case they converge around 13–14. The slightly higher residual loss in the larger array is expected due to the increased number of optimization variables, but the overall convergence trend remains stable. These results confirm that the DL-ADMM architecture not only achieves low-complexity optimization but also provides reliable convergence under both small- and large-scale antenna settings, ensuring robustness in practical hybrid beamforming deployments.

4.2. Spectral Efficiency vs. SNR

Figure 5 illustrates the spectral efficiency performance of the proposed DL-ADMM-Reg scheme compared with APS, OMP, conventional ADMM, and benchmark curves (Optimal and Mismatch) under an estimation error variance of ${\sigma }_e^2=0.1$. Across the entire SNR range, DL-ADMM-Reg exhibits performance very close to the Optimal curve, consistently outperforming APS, OMP, and conventional ADMM. The improvement over the Mismatch baseline highlights the effectiveness of the RSVD filter in mitigating singular value distortion caused by channel estimation errors. Compared to conventional ADMM, the DL-ADMM-Reg approach achieves higher robustness and stability, benefiting from its data-driven initialization and learned parameter tuning, which accelerate convergence and reduce sensitivity to CSI imperfections. APS achieves competitive performance compared to OMP, benefiting from its adaptive clustering approach, but it still lags behind ADMM-based schemes due to its reliance on centroid-based precoding. OMP shows the lowest spectral efficiency, as its reliance on predefined codebooks reduces adaptability under imperfect CSI. Comparing subfigures (a) and (b), the larger $64\times 64$ array in Figure 5b delivers higher spectral efficiency than the $32\times 32$ case in Figure 5a. This is a direct result of the additional spatial degrees of freedom, which enhance multiplexing and beamforming gain. Importantly, DL-ADMM-Reg demonstrates strong scalability with antenna size, retaining stability and robustness across different configurations.

To complement the spectral-efficiency curves, we report the average relative error of each hybrid beamforming method with respect to the Optimal (digital-only) benchmark, where a smaller value indicates a closer match to the unconstrained optimum under the same channel realizations. As shown in

Table 1. Average relative error of hybrid beamforming methods with respect to the Optimal (digital-only) benchmark.

$({\mathit{N}}_{\mathit{t}},{\mathit{N}}_{\mathit{r}})$	ADMM	APS	OMP	DL-ADMM-Reg	[18]	[19]
$(32,32)$	0.10	0.05	0.30	0.02	0.01	0.02
$(64,64)$	0.08	0.05	0.31	0.004	0.005	0.01

, DL-ADMM-Reg yields the lowest error among the conventional baselines (ADMM, APS, and OMP), demonstrating that the proposed RSVD-stabilized unfolded design more reliably preserves the dominant eigenmodes of the effective channel and reduces mismatch-induced degradation. Compared with recent DL baselines [18,19], DL-ADMM-Reg achieves similarly low error while offering stronger consistency across antenna dimensions; notably, for the larger $(64,64)$ array the error remains very small, highlighting that the proposed model-driven unfolding coupled with explicit regularization scales favorably and maintains proximity to the optimal performance.

4.3. Spectral Efficiency vs. Channel Estimation Error

Figure 6 presents spectral efficiency as a function of the channel estimation error variance ${\sigma }_e^2$ at fixed $SNR=10$ dB. Both antenna configurations are considered. The Optimal curve provides the theoretical upper bound, while the Mismatch curve highlights the worst-case performance when imperfect CSI is directly used without compensation. The results show that DL-ADMM-Reg consistently delivers higher spectral efficiency across all error levels. In the $32\times 32$ configuration, spectral efficiency decreases rapidly as ${\sigma }_e^2$ increases, reflecting limited robustness of smaller arrays. In the $64\times 64$ configuration, however, the degradation is less severe: the larger number of antennas increases spatial diversity and reduces sensitivity to estimation errors. This demonstrates that the proposed method not only provides robust compensation for CSI errors but also leverages the inherent robustness of large-scale antenna systems.

In conclusion, Figure 5 and Figure 6 provides a clear quantitative validation of the proposed scheme. As the channel estimation error increases, the spectral efficiency of OMP and APS degrades rapidly because their beam selection/phase updates are highly sensitive to error-distorted singular subspaces, whereas the proposed DL-ADMM-RSVD maintains a substantially higher rate by jointly optimizing RF/baseband variables and explicitly regularizing vulnerable singular modes. For example, at ${\sigma }_e^2=0.1$, DL-ADMM-RSVD achieves an approximately 8–11 bits/s/Hz gain over OMP in the tested SNR range (20–40 dB), while remaining within <1 bit/s/Hz of the digital-optimal benchmark, demonstrating both robustness and near-optimality under imperfect CSI. This consistent gap across error levels confirms that the advantage is not anecdotal but stems from the proposed model-driven unfolding and RSVD-based conditioning control.

4.4. Error Analysis

An essential aspect of evaluating hybrid beamforming performance under imperfect CSI is understanding the statistical distribution of channel estimation errors. To this end, we investigate the error distribution for two antenna configurations, $(32\times 32)$ and $(64\times 64)$. Figure 7 illustrates both histograms and empirical probability density functions (PDFs) of the estimation errors. The histograms (left column) show the raw occurrence of error samples, while the smoothed PDFs (right column), obtained using kernel density estimation, reveal that the errors closely follow a Gaussian distribution centered at zero. This confirms the common assumption in the massive MIMO literature that channel estimation errors can be modeled as normally distributed random variables. For the $(32\times 32)$ configuration, the error variance is relatively small, resulting in a narrow Gaussian spread around the mean. In contrast, for the $(64\times 64)$ configuration, the distribution is wider, reflecting a higher variance due to the increased number of antennas and associated estimation complexity. Despite the difference in variance, both cases align well with the Gaussian assumption, validating its suitability for error modeling in the proposed framework. The Gaussian nature of estimation errors is particularly relevant for the proposed Regularized Hybrid Beamforming, as the RSVD-based filter is designed to stabilize the impact of normally distributed noise on the singular values of the observed channel matrix. By confirming normality in the error distribution, these results reinforce the effectiveness of the proposed design.

4.5. Analysis of Complexity

To better understand the computational efficiency of the proposed DL-ADMM algorithm, we now analyze its per-iteration complexity and compare it with conventional hybrid beamforming methods. In Algorithm 1, the main computational cost arises in lines 4–6, where the term $\mathbf{R}\mathbf{B}$ is repeatedly recalculated for the updates of $\mathbf{Z}$, $\mathbf{R}$, and $\mathbf{B}$. Assuming $\mathbf{R}\in {\mathbb{C}}^{a\times b}$ and $\mathbf{B}\in {\mathbb{C}}^{b\times c}$, the product $\mathbf{R}\mathbf{B}$ has dimension $a\times c$ and requires $\mathcal{O}\left(abc\right)$ operations. Since this computation appears three times per iteration, the total dominant cost is $\mathcal{O}\left(3abc\right)$. This makes the DL-ADMM scheme both structured and computationally efficient. To relate $(a,b,c)$ to the hybrid beamforming problem, note that a corresponds to the number of antennas ($N_t$ or $N_r$), b corresponds to the number of RF chains ($N_{RFt}$ or $N_{RFr}$), and c corresponds to the number of data streams ($N_s$). Thus, the complexity of the DL-ADMM algorithm per iteration can be expressed as $\mathcal{O}\left(3N_{t\left(r\right)}{N}_{RFt\left(r\right)}{N}_s\right)$, where $N_{t\left(r\right)}$ and $N_{RFt\left(r\right)}$ refer to the transmitter (or receiver) dimensions depending on whether precoders or combiners are being computed.

Figure 8 illustrates the practical computational burden of the compared algorithms under the parameter settings of

Table 2. Complexities of different algorithms.

Algorithm	Dominant Complexity (per Iteration)
ADMM	$\mathcal{O}(3N_{t\left(r\right)}{N}_{RFt\left(r\right)}{N}_s\ast \left(numberofiterations\right))$
OMP	$\mathcal{O}\left(N_{t\left(r\right)}^2{N}_{RFt\left(r\right)}{N}_s\right)$ [13]
APS	$\mathcal{O}\left(2N_{t\left(r\right)}{N}_{RFt\left(r\right)}{N}_{cl}M\right)$, ($M=10\left(numberofiterations\right)$ ) [14].
DL-ADMM	$\mathcal{O}(3N_{t\left(r\right)}{N}_{RFt\left(r\right)}{N}_s\ast \left(numberoflayers\right))$

. As seen, OMP incurs the highest cost, scaling rapidly with antenna size due to its quadratic dependence on $N_{t\left(r\right)}$. APS exhibits moderate complexity, but its reliance on clustering and iterative updates ($M=10$) makes it more expensive than ADMM-based schemes. Conventional ADMM and DL-ADMM have nearly identical theoretical complexity; however, DL-ADMM avoids iterative convergence checks by unfolding ADMM steps into a fixed-depth network, leading to reduced runtime overhead. Importantly, DL-ADMM achieves a balance between low computational complexity and improved robustness, making it well-suited for large-scale massive MIMO under imperfect CSI.

The results reveal a clear performance–complexity trade-off when comparing DL-ADMM-RSVD with the benchmark beamforming methods. Greedy/codebook-based schemes such as OMP typically incur lower implementation overhead but suffer a pronounced spectral-efficiency loss under imperfect CSI, whereas fully digital or high-precision iterative solutions can approach near-optimal performance at the cost of substantially higher per-iteration matrix operations and slower runtime scaling with $(N_t,N_r)$. In contrast, DL-ADMM-RSVD achieves near-digital performance while keeping complexity predictable and moderate: it runs a fixed number of unfolded layers (iterations) with structured ADMM updates and simple projection steps, and the RSVD regularization adds only a one-time SVD-type operation whose cost is amortized across the fixed-depth network. Quantitatively, for the same antenna/RF-chain settings, our method requires far fewer iterative search steps than OMP and avoids the repeated greedy atom selection, while delivering significantly higher spectral efficiency; compared with conventional iterative optimizers, it reduces the number of required iterations by replacing convergence-driven stopping with a fixed-depth forward pass. These comparisons show that the proposed approach offers a favorable operating point—substantial robustness and spectral-efficiency gains for a controlled and scalable computational budget.

5. Conclusions

In this paper, we proposed a novel hybrid beamforming mechanism for massive MIMO systems to explicitly address the challenges posed by channel estimation errors. Our approach integrates a low-complexity DL-based ADMM framework withn a RSVD strategy, enabling simultaneous optimization of both baseband and RF chain components. Unlike conventional OMP or APS schemes, which rely on predefined codebooks and are highly sensitive to CSI imperfections, the proposed DL-ADMM design learns adaptive mappings that significantly reduce computational complexity while maintaining near-optimal performance. To further enhance robustness, we introduced a regularization filter, $Reg$, that strengthens the singular values of the observed channel. This addition mitigates the mismatch effects caused by estimation errors, providing theoretical guarantees of improved stability and noise resilience. Simulation results demonstrated that the proposed DL-ADMM-Reg method consistently outperforms conventional baselines and the mismatch benchmark across a wide range of SNR values and channel estimation error variances. Notably, the method achieves spectral efficiency close to the optimal bound, even under moderate-to-severe CSI imperfections, and exhibits improved scalability with larger antenna arrays. Overall, the combination of DL-based unfolding for complexity reduction and RSVD-based regularization for robustness presents a promising direction for practical hybrid beamforming in beyond-5G and 6G systems. Future research could extend this framework to multi-user MIMO scenarios, explore its application to wideband mmWave and THz channels, and investigate hardware implementation aspects to further validate its real-time feasibility.