From Model-Driven to AI-Native Physical Layer Design: Deep Learning Architectures and Optimization Paradigms for Wireless Communications

Abstract

The increasing complexity of next-generation wireless systems challenges the scalability and generalization capabilities of traditional model-driven physical layer (PHY) design, which relies on analytically derived channel models and optimization frameworks. This paper presents a comprehensive survey and critical review of deep learning (DL) architectures enabling the transition toward AI-native PHY design. A unified optimization perspective is developed in which all PHY tasks—including channel estimation, channel state information (CSI) feedback, massive MIMO processing, signal detection, channel coding, beamforming, resource allocation, and semantic-aware transmission—are formulated under a common empirical risk minimization (ERM) framework. Neural architectures such as autoencoders, convolutional and recurrent networks, transformers, and reinforcement learning models are examined through their underlying optimization formulations, loss functions, training methodologies, and representation learning mechanisms. The review compares model-driven and AI-native approaches in terms of performance metrics, computational complexity, robustness, generalization capability, and practical deployment constraints, including hardware limitations, energy efficiency, and real-time feasibility. The analysis highlights the conditions under which AI-native architectures provide adaptability and performance improvements while identifying trade-offs in complexity, latency, and interpretability. The study concludes by outlining prioritized research directions toward fully adaptive and self-optimizing wireless communication systems.

1. Introduction

The evolution of wireless communication systems has been characterized by systematic improvements in capacity, reduced latency, spectral efficiency, and reliability. Earlier generations—from the Third Generation (3G) through the Fourth Generation (4G) and Fifth Generation (5G)—were largely built upon refinements of traditional signal processing techniques and classical information-theoretic principles. However, emerging next-generation wireless systems, particularly those envisioned under the Sixth Generation (6G) umbrella, are increasingly conceived as architectures in which Artificial Intelligence (AI) is tightly integrated across multiple layers of the communication protocol stack, with particular emphasis on the physical layer (PHY) [1,2]. This transition reflects a broader move from strictly model-driven frameworks toward what the research community now terms AI-native system design—a design philosophy in which learning-based components are not merely adjuncts to conventional processing blocks but serve as first-class architectural elements optimized jointly across the entire transmission–reception chain.

Traditionally, the wireless physical layer has been developed through analytical approaches grounded in explicit propagation channel models, statistical signal representations, and optimization techniques derived from convex or structured mathematical formulations. While these model-based methods have achieved remarkable success—enabling, for example, near-capacity-achieving turbo codes and Low-Density Parity-Check (LDPC) decoders—they face fundamental limitations when addressing highly complex communication environments. In particular, high-dimensional Massive Multiple-Input Multiple-Output (MIMO) configurations, Reconfigurable Intelligent Surfaces (RIS), nonlinear channel behaviours, and stringent latency constraints render analytically tractable solutions increasingly elusive [3]. Under such conditions, deriving accurate closed-form solutions or computationally feasible estimators becomes prohibitively difficult.

In this context, Deep Learning (DL) techniques have emerged as a compelling alternative, capable of learning high-dimensional representations and nonlinear mappings directly from data without relying on simplified or inaccurate channel assumptions. By enabling end-to-end optimization, implicit channel modelling, and adaptive signal processing, DL architectures underpin the development of AI-native physical layer designs that fundamentally reshape conventional block-based communication architectures.

Scope and nature of this work. This article constitutes a comprehensive survey and critical review of the state of the art in deep learning for the wireless physical layer. It does not propose a new model, framework, or algorithm; rather, it systematically synthesizes, categorizes, and critically evaluates the existing body of research, identifies open challenges, and delineates future research directions. In doing so, it provides a unified mathematical notation and optimization perspective that connects disparate works spanning channel estimation, signal detection, beamforming, resource allocation, coding, modulation, and semantic communications.

1.1. Contributions of This Article

This survey presents the following principal contributions:

• Unified optimization perspective across all PHY tasks. Unlike prior surveys that treat individual physical layer functions in isolation, this work develops a unified optimization paradigm in which channel estimation, signal detection, beamforming, resource allocation, coding, and modulation are all formulated under a common empirical risk minimization (ERM) framework. This unified viewpoint reveals structural commonalities among seemingly disparate tasks—such as the shared role of encoder–decoder architectures in both autoencoder-based modulation and learned channel estimation—and provides a principled basis for end-to-end system optimization.
• Structured analysis of deep neural network architectures applied to key PHY components, supported by rigorous mathematical formulations, clearly defined objective functions, and explicit optimization constraints.
• Systematic review of learning-based techniques for channel estimation, Channel State Information (CSI) feedback compression, Massive MIMO detection, channel coding, beamforming, and resource allocation, organized by both architectural paradigm and application domain.
• Formal examination of loss functions, training paradigms, and optimization frameworks underlying AI-driven communication systems, including supervised, unsupervised, self-supervised, and reinforcement learning approaches.
• Comparative discussion of model-driven and AI-native approaches in terms of achievable performance, computational complexity, generalization capability, and robustness to model mismatch.
• Critical analysis of implementation challenges, including computational latency, hardware acceleration on Field-Programmable Gate Arrays (FPGAs) and Application-Specific Integrated Circuits (ASICs), adversarial robustness, scalability, and interoperability within Third Generation Partnership Project (3GPP) Release 18/19 [4] and International Telecommunication Union Radiocommunication Sector (ITU-R) International Mobile Telecommunications 2030 (IMT-2030) [5] standardization frameworks.
• End-to-end mathematical consistency. All equations, loss functions, and optimization formulations are presented in a unified and internally consistent notation—with boldface for vectors and matrices—thereby advancing beyond the fragmented treatments found in Huang et al. [6] and Letaief et al. [7], which address network architecture and roadmap visions, respectively, but do not provide the level of mathematical detail and cross-task consistency offered here.

1.2. Novelty and Differentiation from Existing Surveys

The literature on deep learning for wireless communications has grown substantially since the seminal contribution of O’Shea and Hoydis [8], who introduced the autoencoder interpretation of end-to-end communication system design. Subsequent surveys by Qin et al. [9] and Zhang et al. [10] provided valuable overviews of DL applications across different PHY subsystems. While these works laid important groundwork, the present survey is distinguished from them along several critical dimensions:

Compared with O’Shea and Hoydis (2017) [8]: The pioneering work in [8] introduced the concept of modelling the communication system as an autoencoder and demonstrated the feasibility for simple channel models. However, it did not address the breadth of PHY tasks covered here (e.g., Massive MIMO detection, RIS-assisted beamforming, semantic communications) nor did it provide the unified mathematical formulation linking all subsystems under a common optimization framework. The present survey extends that foundational insight to a comprehensive taxonomy spanning seven major PHY domains.

Compared with Qin et al. (2019) [9]: The survey in [9] offered a concise tutorial on DL applications in physical layer communications, covering channel estimation, signal detection, and end-to-end learning. However, it did not include reinforcement-learning-based resource allocation, semantic and task-oriented communications, or hardware implementation analysis. Moreover, the mathematical treatment in [9] was primarily illustrative rather than rigorous. In contrast, the present work provides complete problem formulations with explicit objective functions, constraints, and solution methodologies for each PHY subsystem.

Compared with Zhang et al. (2020) [10]: Ref. [10] focused on AI for 5G and Beyond-5G (B5G) from a circuits and systems perspective, emphasizing implementation efficiency and hardware optimization. While that perspective is valuable, it did not systematically address the information-theoretic foundations of AI-native design, the connection between ERM and mutual information maximization, or the emerging domain of semantic communications. The present survey integrates these theoretical foundations with practical implementation considerations.

Compared with Huang et al. [6] and Letaief et al. [7]: These works presented visionary roadmaps for green 6G networks and AI-empowered wireless systems, respectively. While they provided high-level architectural perspectives, neither offered the detailed mathematical formulations, per-subsystem optimization analyses, or the unified notation framework that characterizes the present survey.

In summary, the key differentiators of this survey are: (i) completeness of PHY coverage spanning all major subsystems within a single, self-contained article; (ii) a unified mathematical notation and optimization framework that links all subsystems; (iii) explicit hardware implementation analysis including FPGA/ASIC deployment considerations; (iv) information-theoretic grounding connecting empirical risk minimization to mutual information maximization and the Information Bottleneck principle; and (v) positioning within current standardization efforts, including 3GPP Release 18/19 and ITU-R IMT-2030.

1.3. Motivation for AI in the Physical Layer

The integration of AI and DL into the wireless physical layer is motivated by several converging factors. Modern communication channels—particularly in millimetre-wave (mmWave) and terahertz (THz) frequency bands—exhibit highly complex propagation characteristics, including severe fading, dynamic blockage, nonlinear dispersion, and atmospheric absorption effects that are difficult to model analytically. Additionally, Mmassive MIMO systems with hundreds or thousands of antennas generate extremely high-dimensional signal spaces in which classical optimization methods become computationally intractable.

Next-generation wireless systems further require real-time adaptation to rapidly varying channel conditions, heterogeneous traffic patterns, and diverse Quality-of-Service (QoS) requirements. Traditional block-wise optimization—whereby the transmitter and receiver components are designed independently—may lead to globally suboptimal performance. In contrast, deep neural networks enable joint end-to-end optimization of the entire transmission–reception chain. Moreover, DL architectures are capable of discovering latent statistical regularities in channel data that are not explicitly captured by simplified analytical models, enabling improved adaptability and robustness.

The system model that underlies the AI-native PHY design philosophy can be stated formally as follows. Consider a discrete-time baseband communication system in which a transmitter encodes a message $s\in \mathcal{S}$ into a transmitted signal vector $\mathbf{x}\in {\mathbb{C}}^{N_t}$, which propagates through a stochastic channel characterized by the conditional distribution $p\left(\mathbf{y}\mid \mathbf{x},\mathbf{H}\right)$, where $\mathbf{H}\in {\mathbb{C}}^{N_r\times N_t}$ denotes the channel matrix with $N_t$ transmit and $N_r$ receive antennas, and $\mathbf{y}\in {\mathbb{C}}^{N_r}$ is the received signal vector. The system is subject to an average transmit power constraint $\mathbb{E}\left[\parallel \mathbf{x}{\parallel }^2\right]\le P$ and must operate within a prescribed latency budget $T_{\mathrm{m}\mathrm{a}\mathrm{x}}$ and a target error probability $P_e\le ϵ$. The fundamental objective is to design encoding and decoding functions that maximize the rate of reliable information transfer under these constraints.

1.3.1. Information-Theoretic Foundations of AI-Native PHY

From an information-theoretic perspective, the physical layer aims to reliably transmit a random message $S$ across a stochastic channel $p\left(\mathbf{y}\mid \mathbf{x},\mathbf{H}\right)$ subject to physical constraints. According to Shannon’s channel coding theorem [11], reliable communication is achievable whenever the transmission rate $R$ satisfies:

$$R\le I\left(\mathbf{x};\mathbf{y}\right)=H\left(\mathbf{y}\right)-H\left(\mathbf{y}\mid \mathbf{x}\right)$$

(1)

where $I\left(\mathbf{x};\mathbf{y}\right)$ denotes the mutual information between transmitted and received signals, $H\left(\mathbf{y}\right)$ is the entropy of the received signal, and $H\left(\mathbf{y}\mid \mathbf{x}\right)$ is the conditional entropy representing residual uncertainty due to channel noise.

Traditional model-driven designs attempt to approach this theoretical bound through analytically derived modulation schemes, channel codes, and optimal detection rules. However, in highly nonlinear, high-dimensional, or poorly modelled channels—such as those encountered in mmWave, THz, and RIS-assisted environments—explicitly deriving capacity-achieving input distributions and optimal decoders becomes analytically intractable.

Connection between Empirical Risk Minimization and Mutual Information Maximization. AI-native approaches reinterpret the communication chain as a parametric mapping optimized through Empirical Risk Minimization (ERM). To understand this connection intuitively, consider that the goal of a communication system is to maximize the mutual information $I\left(\mathbf{x};\mathbf{y}\right)$ between the transmitted and received signals—that is, to ensure that the received signal carries as much information as possible about what was sent. In an end-to-end autoencoder formulation [5,6], the transmitter (encoder) and receiver (decoder) are jointly parameterized by neural networks with parameters $\mathsf{\theta }$ and $\mathsf{\varphi }$, respectively. The system is trained by minimizing the empirical risk:

$$\mathcal{L}\left(\mathsf{\theta },\mathsf{\varphi }\right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathcal{l}\left(s_i,\hspace{0.17em}{\widehat{s}}_i\left(\mathsf{\theta },\mathsf{\varphi }\right)\right)$$

(2)

where $\mathcal{l}\left(\cdot ,\cdot \right)$ is a task-appropriate loss function (e.g., categorical cross-entropy for symbol detection), $s_i$ is the $i$-th transmitted message, and ${\widehat{s}}_i$ is the decoder output.

The key insight linking ERM to information theory is the following: when the loss function is chosen as the cross-entropy, minimizing Equation (2) is equivalent to minimizing the conditional entropy $H\left(S\mid \widehat{S}\right)$—that is, the uncertainty about the transmitted message given the decoder output. Since the mutual information can be decomposed as $I\left(S;\widehat{S}\right)=H\left(S\right)-H\left(S\mid \widehat{S}\right)$, and $H\left(S\right)$ is fixed by the source distribution, minimizing the cross-entropy loss directly maximizes the mutual information between the transmitted message and its reconstruction. Thus, the data-driven ERM objective implicitly pursues the same goal as the classical information-theoretic design criterion, but without requiring knowledge of the channel distribution in closed form.

The Information Bottleneck Principle in Wireless PHY. Beyond the direct ERM–mutual information connection, the Information Bottleneck (IB) framework [12] provides a powerful conceptual and mathematical tool for understanding intermediate neural representations in AI-native communication systems. The IB principle formalizes the fundamental trade-off between compression and relevance in learned representations. Specifically, for an intermediate representation $\mathbf{Z}$ of the input $\mathbf{X}$, the IB objective seeks to solve:

$$\underset{p\left(\mathbf{z}\mid \mathbf{x}\right)}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.33em}I\left(\mathbf{X};\mathbf{Z}\right)-\beta \hspace{0.17em}I\left(\mathbf{Z};S\right)$$

(3)

where $I\left(\mathbf{X};\mathbf{Z}\right)$ measures the complexity of the representation (how much information from the raw input is retained), $I\left(\mathbf{Z};S\right)$ measures the relevance of the representation (how much information about the task-relevant variable $S$ is preserved), and $\beta >0$ is a Lagrange multiplier controlling the trade-off.

In the context of the wireless physical layer, the IB principle has direct and practical implications. At the transmitter, the encoder must compress the source message into a transmitted signal $\mathbf{x}$ that fits within the available bandwidth and power budget—this corresponds to minimizing $I\left(\mathbf{X};\mathbf{Z}\right)$. Simultaneously, the encoded signal must preserve sufficient information for the receiver to reliably decode the intended message—this corresponds to maximizing $I\left(\mathbf{Z};S\right)$. Neural network layers in an autoencoder-based communication system naturally perform successive compression-and-refinement operations that approximate the IB optimal trade-off: early layers extract broad features while deeper layers distil task-relevant information and discard channel-induced redundancy.

This perspective is especially pertinent to semantic and task-oriented communications, where the goal is not to reconstruct every transmitted bit perfectly but rather to convey task-relevant meaning. The IB framework provides the theoretical justification for discarding source information that is irrelevant to the downstream task, thereby achieving higher effective throughput. Recent work by Cai et al. [13] has formalized an end-to-end information-theoretic framework for task-oriented semantic communications over MIMO channels, demonstrating that IB-based optimization can yield significant gains in task performance relative to conventional bit-level transmission strategies. In practical wireless applications, the IB perspective supports the design of adaptive modulation, neural decoding, and intelligent beamforming strategies capable of operating near information-theoretic limits without requiring explicit channel models.

1.3.2. Transition from Model-Driven to AI-Native Design

The conventional physical layer is constructed through functional decomposition into independent processing blocks: source coding, channel coding, modulation, equalization, demodulation, and decoding. Each block is optimized separately according to theoretical channel models and criteria such as minimizing the Bit Error Rate (BER) or maximizing Shannon capacity. While this modular approach simplifies design and analysis, it foregoes potential gains from joint optimization across blocks.

Mathematically, consider the standard discrete-time baseband MIMO system model:

$$\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{n}$$

(4)

where $\mathbf{x}\in {\mathbb{C}}^{N_t}$ is the transmitted symbol vector, $\mathbf{H}\in {\mathbb{C}}^{N_r\times N_t}$ is the MIMO channel matrix capturing path loss, fading, and spatial correlation, $\mathbf{n}\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_n^2{\mathbf{I}}_{N_r}\right)$ is complex Additive White Gaussian Noise (AWGN), and $\mathbf{y}\in {\mathbb{C}}^{N_r}$ is the received signal vector. The system operates under the constraints:

$$\mathbb{E}\left[\parallel \mathbf{x}{\parallel }^2\right]\le P, R\le {\mathrm{l}\mathrm{o}\mathrm{g}}_2\mathrm{d}\mathrm{e}\mathrm{t}\left({\mathbf{I}}_{N_r}+{\displaystyle \frac{1}{{\sigma }_n^2}}\mathbf{H}\mathbf{Q}{\mathbf{H}}^H\right)$$

(5)

where $P$ denotes the maximum average transmit power, $\mathbf{Q}=\mathbb{E}\left[\mathbf{x}{\mathbf{x}}^H\right]$ is the input covariance matrix, and ${\left(\cdot \right)}^H$ denotes the conjugate transpose.

In the traditional paradigm, the transmitter and receiver are designed as separate functions:

$$\mathbf{x}=f_{\mathrm{tx}}\left(s\right), \widehat{s}=f_{\mathrm{rx}}\left(\mathbf{y},\widehat{\mathbf{H}}\right)$$

(6)

where $s\in \mathcal{S}$ denotes the information bits, $\widehat{\mathbf{H}}$ is a channel estimate obtained through pilot-based training, and $\widehat{s}$ represents the decoded bits. The functions $f_{\mathrm{tx}}\left(\cdot \right)$ and $f_{\mathrm{rx}}\left(\cdot \right)$ are independently designed—for example, $f_{\mathrm{tx}}$ may implement a combination of LDPC encoding and Quadrature Amplitude Modulation (QAM) mapping, while $f_{\mathrm{rx}}$ may cascade a Minimum Mean-Square Error (MMSE) equalizer with a Viterbi decoder.

In contrast, the AI-native approach models the entire communication chain as parameterized neural network mappings $f_{\mathsf{\theta }}\left(\cdot \right)$ and $g_{\mathsf{\varphi }}\left(\cdot \right)$, jointly optimized end-to-end:

$$\underset{\mathsf{\theta },\mathsf{\varphi }}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.33em}{\mathbb{E}}_{s,\mathbf{H},\mathbf{n}}\left[\mathcal{L}\left(s,\hspace{0.17em}{g}_{\mathsf{\varphi }}\left(f_{\mathsf{\theta }}\left(s\right),\mathbf{H},\mathbf{n}\right)\right)\right]$$

(7)

subject to $\mathbb{E}\left[\parallel f_{\mathsf{\theta }}\left(s\right){\parallel }^2\right]\le P$, where $\mathcal{L}\left(\cdot ,\cdot \right)$ is a differentiable loss function (e.g., categorical cross-entropy for classification or mean squared error for regression), and the expectation is taken over the joint distribution of messages, channel realizations, and noise. This formulation reframes communication system design as an ERM problem, enabling representation learning directly from data rather than relying exclusively on analytically derived channel models. By jointly optimizing $\mathsf{\theta }$ and $\mathsf{\varphi }$, the AI-native approach can discover encoding and decoding strategies that exploit channel structure in ways that are inaccessible to conventional block-wise design.

1.4. Comparison with Related Surveys

The depth of coverage, mathematical rigor, and specific PHY subsystems addressed in this survey are carefully differentiated from prior review works in the field. Table 1 provides a structured comparison across the dimensions most relevant to AI-native PHY design.

Table 1. Comparison of this survey with related reviews on deep learning for the wireless physical layer. Dimensions include coverage of PHY subsystems (channel estimation, Massive MIMO detection, channel coding, beamforming, semantic communications), mathematical formulation depth, hardware implementation analysis, and alignment with standardization frameworks.

Dimension	O’Shea & Hoydis (2017) [8]	Qin et al. (2019) [9]	Zhang et al. (2020) [10]	Huang et al. (2019) [6]	Letaief et al. (2019) [7]	This Survey
Channel Estimation	Partial	✓	Partial	✗	Partial	✓
Massive MIMO Detection	✗	Partial	✓	✗	Partial	✓
Channel Coding	✗	Partial	✓	✗	✗	✓
Beamforming & Beam Mgmt.	✗	✗	Partial	✗	Partial	✓
Resource Allocation (DRL)	✗	✗	✗	Partial	Partial	✓
Adaptive Modulation & Waveforms	✗	✗	✗	✗	✗	✓
Semantic Communications	✗	✗	✗	✗	✗	✓
Unified ERM Math. Framework	✗	✗	✗	✗	✗	✓
Hardware Impl. (FPGA/ASIC)	✗	✗	✓	Partial	✗	✓
3GPP/ITU-R Standardization	✗	✗	✗	✗	Partial	✓
End-to-End Autoencoder	✓	Partial	✗	✗	✗	✓
Information-Theoretic Grounding (IB)	✗	✗	✗	✗	✗	✓

Legend: ✓ = comprehensively covered; Partial = partially covered; ✗ = not covered.

This survey is distinguished from prior works by: (i) complete coverage of all major PHY subsystems—channel estimation, Massive MIMO detection, channel coding, beamforming, resource allocation, modulation, and semantic communications—within a single article; (ii) a unified mathematical notation linking all subsystems under a common ERM-based optimization framework; (iii) explicit hardware implementation analysis including FPGA/ASIC deployment considerations and computational complexity profiling; (iv) corrected and complete equation sets presented in consistent notation for all methods; and (v) positioning within 3GPP Release 18/19 and ITU-R IMT-2030 standardization frameworks.

1.5. Organization of the Article

The remainder of this article is structured as follows. Section 2 presents the fundamental deep learning architectures employed in physical layer design, providing the necessary background on Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, autoencoders, Generative Adversarial Networks (GANs), transformers, and Deep Reinforcement Learning (DRL). Section 3 reviews AI-based channel estimation and prediction techniques, covering problem formulations, DL architectures, channel sparsity exploitation, and pilot-free methods. Section 4 discusses signal detection using neural networks in MIMO systems. Section 5 examines intelligent beamforming and beam management, including beam prediction, tracking, multi-user beamforming, and RIS-assisted configurations. Section 6 analyses spectral resource management using reinforcement learning, covering Markov Decision Process (MDP) formulations, Deep Q-Networks (DQN), policy gradient methods, and Multi-Agent Reinforcement Learning (MARL). Section 7 presents adaptive modulation and intelligent waveforms, including neural constellation design and Peak-to-Average Power Ratio (PAPR) reduction. Section 8 explores semantic and task-oriented communications, addressing semantic extraction, end-to-end semantic transceivers, and goal-oriented communication. Section 9 discusses implementation challenges and practical considerations, including computational complexity, hardware acceleration, adversarial robustness, and standardization. Section 10 outlines future research directions and emerging trends. Finally, Section 11 presents the conclusions of this survey.

2. Fundamental Deep Learning Architectures for the Physical Layer

This section provides a self-contained overview of the fundamental deep learning architectures that are employed throughout the physical layer applications surveyed in subsequent sections. While the reader is assumed to have a general familiarity with machine learning concepts, this background section establishes the notation, architectural details, and mathematical formulations that will be referenced consistently in Section 3, Section 4, Section 5, Section 6, Section 7 and Section 8. Each architecture is presented with its canonical mathematical description and its particular relevance to wireless PHY tasks is highlighted.

Figure 1 illustrates the principal neural network families—feedforward DNNs, CNNs, RNNs/LSTMs, autoencoders, GANs, transformers, and DRL agents—and their typical mapping to PHY tasks such as channel estimation, signal detection, beamforming, and resource allocation. Key insight: different architectures exploit different structural priors (spatial locality for CNNs, temporal dependencies for RNNs, compression–reconstruction for autoencoders), and the choice of architecture should be guided by the underlying signal processing task. Takeaway: no single architecture dominates across all PHY domains; rather, the optimal choice depends on the dimensionality, temporal dynamics, and optimization objective of the specific application.

Figure 1. Overview of deep learning architectures for the wireless physical layer.

2.1. Feedforward Deep Neural Networks (DNNs)

Feedforward DNNs constitute the most fundamental deep learning architecture and serve as universal function approximators. For symbol detection and other classification-oriented PHY tasks, a fully connected network maps received signals to symbol probabilities through a sequence of affine transformations and nonlinear activations.

Consider a DNN with $L$ hidden layers. The forward pass is defined recursively as:

$${\mathbf{a}}^{\left(0\right)}=\mathbf{y}$$

(8)

$${\mathbf{a}}^{\left(\mathcal{l}\right)}=\sigma \left({\mathbf{W}}^{\left(\mathcal{l}\right)}{\mathbf{a}}^{\left(\mathcal{l}-1\right)}+{\mathbf{b}}^{\left(\mathcal{l}\right)}\right), \mathcal{l}=1,\dots ,L$$

(9)

$$\widehat{\mathbf{s}}=\mathrm{softmax}\left({\mathbf{W}}^{\left(L+1\right)}{\mathbf{a}}^{\left(L\right)}+{\mathbf{b}}^{\left(L+1\right)}\right)$$

(10)

where $\mathbf{y}\in {\mathbb{C}}^{N_r}$ is the received signal vector, ${\mathbf{W}}^{\left(\mathcal{l}\right)}\in {\mathbb{R}}^{d_{\mathcal{l}}\times d_{\mathcal{l}-1}}$ and ${\mathbf{b}}^{\left(\mathcal{l}\right)}\in {\mathbb{R}}^{d_{\mathcal{l}}}$ are the weight matrix and bias vector of the $\mathcal{l}$-th layer, $\sigma \left(\cdot \right)$ denotes a nonlinear activation function such as the Rectified Linear Unit (ReLU) or hyperbolic tangent (tanh), and $\widehat{\mathbf{s}}$ is the output probability vector over the symbol alphabet. In PHY applications, DNNs are used for tasks including symbol detection [14], CSI feedback compression, and power control, where the mapping from input features to output decisions can be learned directly from training data.

2.2. Convolutional Neural Networks (CNNs)

CNNs exploit spatial or temporal structure in signals through convolution operations with shared, shift-invariant parameters. For a one-dimensional input signal ${\mathbf{a}}^{\left(\mathcal{l}-1\right)}$, the convolution operation at the $\mathcal{l}$-th layer is defined as:

$$a_j^{\left(\mathcal{l}\right)}=\sigma \left(\sum _k{w}_k^{\left(\mathcal{l}\right)}\hspace{0.17em}{a}_{j-k}^{\left(\mathcal{l}-1\right)}+b^{\left(\mathcal{l}\right)}\right)$$

(11)

where the index $k$ represents temporal or spatial displacement, and $w_k^{\left(\mathcal{l}\right)}$ are the learned filter coefficients. CNNs are particularly effective for processing temporal symbol sequences and for exploiting spatial correlation in antenna arrays. In the context of channel estimation, two-dimensional CNNs can process the time-frequency channel response as an image-like structure, enabling denoising and interpolation that leverage the inherent correlation structure of wireless channels [15].

2.3. Recurrent Neural Networks (RNNs) and LSTMs

For modelling temporal channel evolution—such as time-varying fading or Doppler-induced variations—Recurrent Neural Networks maintain a hidden state that captures temporal dependencies. The standard RNN update is:

$${\mathbf{h}}_t=\sigma \left({\mathbf{W}}_h{\mathbf{h}}_{t-1}+{\mathbf{W}}_x{\mathbf{x}}_t+{\mathbf{b}}_h\right)$$

(12)

where ${\mathbf{h}}_t\in {\mathbb{R}}^{d_h}$ is the hidden state at time step $t$, ${\mathbf{x}}_t$ is the input at time $t$, and ${\mathbf{W}}_h$, ${\mathbf{W}}_x$, ${\mathbf{b}}_h$ are learnable parameters.

Long Short-Term Memory (LSTM) architectures extend the basic RNN with gating mechanisms that mitigate the vanishing gradient problem and enable modelling of long-range temporal dependencies:

$${\mathbf{f}}_t={\sigma }_g\left({\mathbf{W}}_f\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_f\right)$$

(13)

$${\mathbf{i}}_t={\sigma }_g\left({\mathbf{W}}_i\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_i\right)$$

(14)

$${\tilde{\mathbf{c}}}_t=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\mathbf{W}}_c\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_c\right)$$

(15)

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot {\tilde{\mathbf{c}}}_t$$

(16)

$${\mathbf{o}}_t={\sigma }_g\left({\mathbf{W}}_o\left[{\mathbf{h}}_{t-1},{\mathbf{x}}_t\right]+{\mathbf{b}}_o\right)$$

(17)

$${\mathbf{h}}_t={\mathbf{o}}_t\odot \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\mathbf{c}}_t\right)$$

(18)

where ${\mathbf{f}}_t$, ${\mathbf{i}}_t$, and ${\mathbf{o}}_t$ denote the forget, input, and output gates, respectively, ${\mathbf{c}}_t$ represents the cell state, ${\sigma }_g\left(\cdot \right)$ is the sigmoid activation function, and $\odot$ denotes element-wise (Hadamard) multiplication. In wireless PHY applications, LSTM networks are employed for channel prediction in time-varying environments [16], enabling proactive resource allocation and predictive beamforming.

2.4. Autoencoders (AEs)

The communication system can be holistically modelled as a learned encoder–channel–decoder structure, following the autoencoder interpretation introduced by O’Shea and Hoydis [8]. The encoder maps a message $s$ to a transmitted signal, the signal passes through the physical channel, and the decoder reconstructs the message:

$$\mathbf{x}=f_{\mathsf{\theta }}\left(s\right), \widehat{s}=g_{\mathsf{\varphi }}\left(\mathbf{y}\right)$$

(19)

where $f_{\mathsf{\theta }}:\mathcal{S}\to {\mathbb{C}}^{N_t}$ is the neural encoder and $g_{\mathsf{\varphi }}:{\mathbb{C}}^{N_r}\to \mathcal{S}$ is the neural decoder. The objective is to minimize the reconstruction loss:

$$\underset{\mathsf{\theta },\mathsf{\varphi }}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.33em}{\mathbb{E}}_{s,\mathbf{H},\mathbf{n}}\left[\mathcal{l}\left(s,\hspace{0.17em}{g}_{\mathsf{\varphi }}\left(\mathbf{H}{f}_{\mathsf{\theta }}\left(s\right)+\mathbf{n}\right)\right)\right]$$

(20)

Variational Autoencoders (VAEs) incorporate probabilistic regularization by introducing a latent variable $\mathbf{z}$ and optimizing the Evidence Lower Bound (ELBO):

$${\mathcal{L}}_{\mathrm{VAE}}={\mathbb{E}}_{q_{\mathsf{\varphi }}\left(\mathbf{z}\mid \mathbf{y}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}{p}_{\mathsf{\theta }}\left(\mathbf{y}\mid \mathbf{z}\right)\right]-D_{\mathrm{KL}}\left(q_{\mathsf{\varphi }}\left(\mathbf{z}\mid \mathbf{y}\right)\hspace{0.17em}\parallel \hspace{0.17em}p\left(\mathbf{z}\right)\right)$$

(21)

where $D_{\mathrm{KL}}\left(\cdot \parallel \cdot \right)$ denotes the Kullback–Leibler divergence, serving as a regularization term that encourages the learned latent distribution to remain close to a tractable prior $p\left(\mathbf{z}\right)$. In PHY applications, autoencoders enable end-to-end learning of modulation, coding, and signal shaping strategies that are jointly optimized for the channel [8,17].

2.5. Generative Adversarial Networks (GANs)

GANs provide a framework for learning to generate realistic synthetic data and are employed in wireless communications primarily for channel modelling and data augmentation. A GAN consists of two competing networks—a generator $G_{{\mathsf{\theta }}_G}$ and a discriminator $D_{{\mathsf{\theta }}_D}$—trained via the minimax objective:

$$\underset{{\mathsf{\theta }}_G}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{{\mathsf{\theta }}_D}{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{\mathbb{E}}_{\mathbf{H}\sim p_{\mathrm{data}}}\left[\mathrm{l}\mathrm{o}\mathrm{g}{D}_{{\mathsf{\theta }}_D}\left(\mathbf{H}\right)\right]+{\mathbb{E}}_{\mathbf{z}\sim p\left(\mathbf{z}\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(1-D_{{\mathsf{\theta }}_D}\left(G_{{\mathsf{\theta }}_G}\left(\mathbf{z}\right)\right)\right)\right]$$

(22)

The generator $G_{{\mathsf{\theta }}_G}$ is trained to produce synthetic channel realizations that are statistically indistinguishable from real channel samples as evaluated by the discriminator $D_{{\mathsf{\theta }}_D}$. At convergence, the generator implicitly learns the underlying channel distribution $p_{\mathrm{data}}\left(\mathbf{H}\right)$, enabling applications such as channel simulation for training data augmentation, domain adaptation across deployment environments, and stochastic channel modelling without closed-form distribution assumptions.

2.6. Transformers and Attention Mechanisms

Originally developed for Natural Language Processing (NLP) tasks, transformer architectures have been extended to wireless signal processing applications through self-attention mechanisms that capture long-range dependencies without the sequential processing bottleneck of RNNs. The scaled dot-product attention is computed as:

$$\mathrm{Attention}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}}\right)\mathbf{V}$$

(23)

where $\mathbf{Q}\in {\mathbb{R}}^{n\times d_k}$, $\mathbf{K}\in {\mathbb{R}}^{n\times d_k}$, and $\mathbf{V}\in {\mathbb{R}}^{n\times d_v}$ denote the query, key, and value matrices obtained from the input signals via learned linear projections, and $d_k$ is the dimensionality of the key vectors. The scaling factor $1/\sqrt{d_k}$ prevents the dot products from growing excessively large in high-dimensional spaces.

Transformer architectures are particularly effective in modelling long-range temporal dependencies and spatial correlations in Massive MIMO systems [18]. Multi-head attention further enhances representational capacity by computing multiple attention functions in parallel and concatenating their outputs, enabling the model to attend to information from different representation subspaces simultaneously.

2.7. Deep Reinforcement Learning (DRL)

For sequential decision-making and resource allocation problems where the optimal action depends on the evolving system state, DRL models the system as a Markov Decision Process (MDP) defined by the tuple $\left(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma \right)$, where $\mathcal{S}$ denotes the state space, $\mathcal{A}$ the action space, $\mathcal{P}:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \left[0,1\right]$ the state transition probabilities, $\mathcal{R}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ the reward function, and $\gamma \in \left[0,1\right)$ the discount factor [19].

The objective is to learn an optimal policy ${\pi }^{\ast }$ that maximizes the expected cumulative discounted return:

$${\pi }^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\pi }{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\mathrm{\infty }}{\gamma }^t\hspace{0.17em}{r}_t\right]$$

(24)

where $r_t=\mathcal{R}\left(s_t,a_t\right)$ is the reward obtained at time step $t$ under state $s_t$ and action $a_t$. Algorithms such as Deep Q-Networks (DQN) [20], Policy Gradient methods, Actor–Critic architectures, and Proximal Policy Optimization (PPO) [21] are employed to approximate the optimal policy ${\pi }^{\ast }$ or the action-value function $Q^{\ast }\left(s,a\right)$ using neural networks. In wireless PHY applications, DRL is applied to dynamic spectrum access, power control, beam management, and resource scheduling, where the environment evolves stochastically and decisions must be made sequentially under uncertainty [22].

3. AI-Based Channel Estimation and Prediction

Accurate channel estimation constitutes a fundamental prerequisite for coherent wireless communication systems, as it directly governs the performance of subsequent receiver functions including equalization, detection, and Multiple-Input Multiple-Output (MIMO) processing [23]. This section provides a systematic review of Artificial Intelligence (AI)-based channel estimation and prediction techniques that address the limitations of classical model-driven estimators in emerging wireless scenarios. Traditional approaches—such as Least Squares (LS), Minimum Mean Square Error (MMSE), and pilot-assisted methods—are well understood under stationary channel conditions; however, they exhibit fundamental limitations in environments characterized by high user mobility, high-dimensional antenna configurations, and severely constrained pilot resources [24,25]. These limitations are particularly pronounced in systems operating at millimeter-wave (mmWave) and sub-terahertz (sub-THz) frequencies, where propagation channels exhibit sparse multipath structures, rapid temporal variations, and highly directional characteristics. Deep Learning (DL) architectures offer a promising alternative by learning complex, high-dimensional channel representations directly from data, without relying on simplified analytical channel models. This section reviews the problem formulation, representative neural architectures, sparse and pilot-free estimation strategies, transfer and meta-learning techniques, and comparative performance analyses.

Figure 2 conceptually illustrates the channel estimation problem in high-dimensional wireless systems. The estimator must reconstruct the full channel frequency response from limited and noisy pilot observations while exploiting spatial, frequency, and temporal correlations inherent in the propagation environment.

Figure 2. Channel estimation problem in next-generation wireless systems.

Figure 2 depicts a multi-antenna transmitter–receiver pair operating over a wideband channel with $N$ subcarriers and $N_t\times N_r$ antenna dimensions. Pilot symbols (shown as shaded resource elements) are inserted at sparse locations in the time–frequency grid, while the estimator must infer the full channel state information (CSI) across all resource elements. The key insight is that AI-native estimators exploit learned spatial, frequency, and temporal correlations to interpolate channel coefficients at non-pilot positions with substantially lower mean square error (MSE) than classical LS or MMSE methods, particularly under low signal-to-noise ratio (SNR) and limited pilot density conditions. The takeaway is that neural architectures can adaptively learn channel statistics from data, enabling robust estimation without requiring explicit knowledge of channel covariance matrices or stationarity assumptions.

3.1. Problem Formulation

Consider a multi-carrier Orthogonal Frequency Division Multiplexing (OFDM) or Orthogonal Frequency Division Multiple Access (OFDMA) system with $N$ subcarriers, $N_t$ transmit antennas, and $N_r$ receive antennas. The received signal at subcarrier $k$ and time index $t$ is expressed as:

$$\mathbf{y}\left[k,t\right]=\mathbf{H}\left[k,t\right]\hspace{0.17em}\mathbf{x}\left[k,t\right]+\mathbf{n}\left[k,t\right], k=1,\dots ,N$$

(25)

where $\mathbf{H}\left[k,t\right]\in {\mathbb{C}}^{N_r\times N_t}$ denotes the channel frequency response matrix, $\mathbf{x}\left[k,t\right]\in {\mathbb{C}}^{N_t\times 1}$ represents the transmitted symbol vector, and $\mathbf{n}\left[k,t\right]\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_n^2\mathbf{I}\right)$ is additive white Gaussian noise (AWGN). The channel estimation problem consists of recovering $\mathbf{H}\left[k,t\right]$ from the observations $\mathbf{y}\left[k,t\right]$ at known pilot positions and, potentially, from previously detected data symbols.

• System Model Assumptions and Constraints. The system model in (25) assumes a block-fading channel that remains approximately constant over each OFDM symbol but may vary across symbols. The pilot overhead is constrained by a fraction ${\rho }_p=N_p/N$ of total subcarriers allocated to pilots, where $N_p$ denotes the number of pilot subcarriers. The fundamental objective is to minimize the normalized mean square error (NMSE):

  $$\mathrm{NMSE}={\displaystyle \frac{\mathbb{E}\left[\parallel \widehat{\mathbf{H}}-\mathbf{H}{\parallel }_F^2\right]}{\mathbb{E}\left[\parallel \mathbf{H}{\parallel }_F^2\right]}}$$

  (26)

  subject to constraints on pilot density, computational complexity, and latency.
• Traditional LS Estimation. At pilot positions where $\mathbf{x}\left[k,t\right]$ is known, the LS estimate is given by ${\widehat{\mathbf{H}}}_{\mathrm{LS}}\left[k,t\right]=\mathbf{y}\left[k,t\right]\hspace{0.17em}{\mathbf{x}}^{†}\left[k,t\right]$, where ${\left(\cdot \right)}^{†}$ denotes the Moore–Penrose pseudo-inverse. The LS estimator does not require statistical knowledge of the channel; however, it exhibits high MSE under low SNR conditions due to noise amplification, and it does not exploit inter-subcarrier or inter-antenna correlations.
• Traditional MMSE Estimation. The MMSE estimator minimizes the expected MSE:

  $${\widehat{\mathbf{H}}}_{\mathrm{MMSE}}={\mathbf{R}}_{\mathbf{H}\mathbf{H}}{\left({\mathbf{R}}_{\mathbf{H}\mathbf{H}}+{\sigma }_n^2{\left({\mathbf{X}}^H\mathbf{X}\right)}^{-1}\right)}^{-1}{\widehat{\mathbf{H}}}_{\mathrm{LS}}$$

  (27)

  where ${\mathbf{R}}_{\mathbf{H}\mathbf{H}}=\mathbb{E}\left[\mathbf{H}{\mathbf{H}}^H\right]$ is the channel covariance matrix. Although MMSE yields the optimal linear estimate under Gaussian assumptions, it requires knowledge of second-order channel statistics ${\mathbf{R}}_{\mathbf{H}\mathbf{H}}$, which may be non-stationary in dynamic environments, and involves matrix inversion with cubic complexity $\mathcal{O}\left(N_r^3\right)$, rendering it computationally prohibitive for massive MIMO configurations.

3.2. Deep Learning Architectures for Channel Estimation

DL-based approaches reinterpret channel estimation as a supervised regression problem, where a neural network $f_{\mathsf{\theta }}$ learns the mapping from noisy observations to channel coefficients:

$$\widehat{\mathbf{H}}=f_{\mathsf{\theta }}\left({\mathbf{y}}_p,{\mathbf{x}}_p\right)$$

(28)

with parameters $\mathsf{\theta }$ optimized by minimizing the empirical risk:

$${\mathsf{\theta }}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathsf{\theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.33em}{\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{\left({\mathbf{y}}_p,\mathbf{H}\right)\in \mathcal{D}}{∥f_{\mathsf{\theta }}\left({\mathbf{y}}_p\right)-\mathbf{H}∥}_F^2$$

(29)

where $\mathcal{D}$ denotes the training dataset comprising input–output pairs of pilot observations and ground-truth channel realizations. Unlike LS and MMSE estimators, neural estimators implicitly learn channel statistics from data without requiring explicit covariance matrices. We review the principal architectures below. Fully Connected Networks (FCNs). Deep Neural Network (DNN)-based estimators directly map pilot observations to channel coefficients through successive fully connected layers. While effective in moderate dimensions, the parameter count scales as $\mathcal{O}\left(N_r{N}_{t}N\right)$, which limits their applicability to large-scale MIMO systems. FCN estimators typically serve as baseline architectures for benchmarking more structured designs.

• Convolutional Neural Network (CNN)-Based Estimators. Wireless channels exhibit spatial correlations across antennas and frequency correlations across subcarriers. CNNs exploit this structure through shared convolutional kernels:

  $${\mathbf{Z}}^{\left(l+1\right)}=\sigma \left({\mathbf{W}}^{\left(l\right)}\ast {\mathbf{Z}}^{\left(l\right)}+{\mathbf{b}}^{\left(l\right)}\right)$$

  (30)

  where $\ast$ denotes the convolution operation, ${\mathbf{W}}^{\left(l\right)}$ and ${\mathbf{b}}^{\left(l\right)}$ are the learnable kernel weights and biases of layer $l$, and $\sigma \left(\cdot \right)$ is a nonlinear activation function. CNN-based architectures—including residual networks (ResNets) and densely connected networks—have demonstrated improved NMSE under spatially correlated channel conditions by leveraging parameter sharing and translation equivariance. The weight-sharing property reduces the total parameter count by orders of magnitude compared to FCNs, yielding complexity $\mathcal{O}\left(K^2{C}_{\mathrm{in}}{C}_{\mathrm{out}}L\right)$ where $K$ is the kernel size, $C_{\mathrm{in}},C_{\mathrm{out}}$ are input/output channels, and $L$ is the number of layers.

Transformer-Based Estimators. Transformers capture long-range dependencies across subcarriers and antenna elements through self-attention mechanisms, without imposing fixed convolutional receptive fields:

$$\mathrm{Attention}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}}\right)\mathbf{V}$$

(31)

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ denote the query, key, and value matrices derived from the input via learned linear projections, and $d_k$ is the key dimension. Self-attention dynamically weights contributions from different subcarriers or antenna elements, enabling adaptive estimation across frequency bands and spatial dimensions. Transformer-based estimators are particularly effective in wideband massive MIMO systems where channel correlations span large frequency–spatial regions [18]. The computational complexity per attention layer is $\mathcal{O}\left(N^2{d}_k\right)$, which may be reduced through sparse or linear attention approximations.

• Generative Adversarial Network (GAN)-Assisted Refinement. GAN-based approaches refine coarse initial estimates (e.g., from LS) by training a generator network $G_{\mathsf{\varphi }}$ to produce realistic channel estimates that are indistinguishable from true channel realizations as judged by a discriminator network $D_{\mathsf{\psi }}$:

  $$\underset{\mathsf{\varphi }}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{\mathsf{\psi }}{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{\mathbb{E}}_{\mathbf{H}}\left[\mathrm{l}\mathrm{o}\mathrm{g}{D}_{\mathsf{\psi }}\left(\mathbf{H}\right)\right]+{\mathbb{E}}_{{\widehat{\mathbf{H}}}_{\mathrm{LS}}}\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(1-D_{\mathsf{\psi }}\left(G_{\mathsf{\varphi }}\left({\widehat{\mathbf{H}}}_{\mathrm{LS}}\right)\right)\right)\right]$$

  (32)

A reconstruction term ensures estimation fidelity:

$${\mathcal{L}}_{\mathrm{recon}}={∥G_{\mathsf{\varphi }}\left({\widehat{\mathbf{H}}}_{\mathrm{LS}}\right)-\mathbf{H}∥}_F^2.$$

(33)

These approaches are particularly relevant in pilot-scarce regimes, where the generator implicitly learns the channel distribution and uses this prior to denoise coarse estimates. However, GAN training introduces additional complexity due to the minimax optimization and potential mode collapse, requiring careful hyperparameter tuning.

• Trade-offs in Architecture Selection. The choice among FCN, CNN, Transformer, and GAN architectures involves trade-offs in estimation accuracy, computational complexity, latency, and interpretability. FCNs offer simplicity but poor scalability; CNNs provide an effective balance of accuracy and efficiency through parameter sharing; Transformers achieve the highest accuracy for large-scale systems at the cost of quadratic attention complexity; and GANs excel in low-pilot scenarios but introduce training instability. Practical deployment must consider hardware constraints, inference latency budgets (e.g., sub-millisecond for Ultra-Reliable Low-Latency Communication (URLLC) applications), and energy consumption targets.

3.3. Channel Prediction in Time-Varying Environments

In high-mobility wireless systems and latency-critical applications, channel prediction enables proactive compensation mechanisms and predictive resource scheduling. The temporal evolution of the channel can be modelled as a stochastic process:

$$\mathbf{H}\left[t+1\right]=g\left(\mathbf{H}\left[t\right],\mathbf{H}\left[t-1\right],\dots ,\mathbf{H}\left[t-\tau \right]\right)+\mathsf{ϵ}\left[t\right]$$

(34)

where $g\left(\cdot \right)$ is a nonlinear state-transition function, $\tau$ is the memory depth, and $\mathsf{ϵ}\left[t\right]$ represents stochastic innovation noise. Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) architectures approximate the prediction function:

$$\widehat{\mathbf{H}}\left[t+\mathsf{\Delta }\right]=f_{\mathsf{\theta }}^{\mathrm{pred}}\left(\mathbf{H}\left[t\right],\mathbf{H}\left[t-1\right],\dots ,\mathbf{H}\left[t-\tau \right]\right)$$

(35)

with training objective:

$${\mathcal{L}}_{\mathrm{pred}}={\displaystyle \frac{1}{T}}\sum _{t=1}^T\parallel \widehat{\mathbf{H}}\left[t+\mathsf{\Delta }\right]-\mathbf{H}\left[t+\mathsf{\Delta }\right]{\parallel }_F^2$$

(36)

Transformer-based sequence models further improve multi-step prediction by capturing long-range temporal dependencies through self-attention across the temporal dimension, enabling more stable predictions over longer horizons [16]. In vehicular scenarios at speeds exceeding 120 km/h, LSTM-based predictors have been shown to reduce multi-step prediction NMSE by 4–6 dB relative to classical autoregressive (AR) models, while attention-augmented architectures provide additional gains of 1–2 dB.

• Real-World Deployment Considerations. Channel prediction requires inference latency shorter than the channel coherence time; for vehicular channels at 28 GHz with Doppler spreads exceeding 1 kHz, the coherence time may be below 1 ms, imposing stringent computational constraints. Model compression techniques—including pruning, quantization, and knowledge distillation—are essential for real-time deployment on edge hardware [26,27].

3.4. Sparse and Pilot-Free Channel Estimation

This subsection consolidates two complementary strategies for reducing pilot overhead: exploiting channel sparsity through compressed sensing and learning-based methods, and blind estimation techniques that eliminate pilot symbols entirely.

3.4.1. Compressed Sensing with Deep Learning

• Millimeter-wave and terahertz channels: Typically exhibit sparse structures in the angular domain due to limited scattering paths. This sparsity can be exploited to reduce estimation overhead and computational complexity. The theory of compressed sensing (CS) establishes that sparse signals can be recovered from sub-Nyquist measurements through ${\mathcal{l}}_1$-norm optimization:

  $${\widehat{\mathbf{h}}}_s=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathbf{h}}_s}{\mathrm{m}\mathrm{i}\mathrm{n}}\parallel {\mathbf{h}}_s{\parallel }_1 \mathrm{subject} \mathrm{to} \parallel \mathsf{\Phi }{\mathbf{h}}_s-{\mathbf{y}}_p{\parallel }_2\le ϵ,$$

  (37)

  where ${\mathbf{h}}_s$ denotes the sparse angular-domain representation of the channel, $\mathsf{\Phi }$ is the measurement matrix determined by pilot placement, ${\mathbf{y}}_p$ represents the pilot observations, and $ϵ$ bounds the noise level. Neural unfolding methods map iterative CS algorithms (e.g., ISTA, ADMM) into trainable layers:

  $${\mathbf{h}}_s^{\left(k+1\right)}=f_{{\mathsf{\theta }}^{\left(k\right)}}\left({\mathbf{h}}_s^{\left(k\right)},{\mathbf{y}}_p\right),$$

  (38)

  where each iteration of the underlying optimization algorithm is unfolded into a corresponding neural network layer, allowing the model to learn regularization parameters and adaptive thresholding operators jointly from data.
• Learned ISTA (LISTA). A representative example of algorithm unfolding is the Learned Iterative Shrinkage-Thresholding Algorithm:

  $${\mathbf{h}}_s^{\left(k+1\right)}={\eta }_{{\lambda }_k}\left({\mathbf{W}}_1^{\left(k\right)}{\mathbf{h}}_s^{\left(k\right)}+{\mathbf{W}}_2^{\left(k\right)}{\mathbf{y}}_p\right)$$

  (39)

  where ${\eta }_{\lambda }\left(\cdot \right)$ denotes the soft-thresholding operator:

  $${\eta }_{\lambda }\left(z\right)=\mathrm{sign}\left(z\right)\cdot \mathrm{m}\mathrm{a}\mathrm{x}\left(\left|z\right|-\lambda ,0\right)$$

  (40)

The parameters ${\mathbf{W}}_1^{\left(k\right)}$, ${\mathbf{W}}_2^{\left(k\right)}$, and ${\lambda }_k$ are learned for each layer $k$, significantly accelerating convergence (typically 10–20×) compared to classical ISTA while maintaining or improving recovery accuracy. The training loss is:

$${\mathcal{L}}_{\mathrm{LISTA}}={\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{i=1}^{\left|\mathcal{D}\right|}{∥{\mathbf{h}}_s^{\left(K\right),i}-{\mathbf{h}}_{s,\mathrm{true}}^i∥}_2^2$$

(41)

where $K$ is the total number of unfolded layers (iterations).

3.4.2. Pilot-Free Channel Estimation via Deep Learning

Blind channel estimation schemes exploit the inherent structure of data signals to estimate the channel without allocating explicit pilot resources, thereby maximizing spectral efficiency. An autoencoder (AE) architecture can jointly learn encoding, channel estimation, and decoding:

$$\mathbf{z}=f_{\mathrm{enc}}\left(\mathbf{s};{\mathsf{\theta }}_{\mathrm{enc}}\right), \widehat{\mathbf{H}}=f_{\mathrm{est}}\left(\mathbf{y};{\mathsf{\theta }}_{\mathrm{est}}\right), \widehat{\mathbf{s}}=f_{\mathrm{dec}}\left(\mathbf{y},\widehat{\mathbf{H}};{\mathsf{\theta }}_{\mathrm{dec}}\right)$$

(42)

where $\mathbf{s}$ denotes the information bits, $\mathbf{z}$ the encoded transmission, and $\widehat{\mathbf{s}}$ the decoded output. The loss function combines data reconstruction accuracy and channel estimation quality:

$${\mathcal{L}}_{\mathrm{blind}}=\underset{\underset{\mathrm{reconstruction}}{⏟}}{{∥\widehat{\mathbf{s}}-\mathbf{s}∥}^2}+\lambda \underset{\underset{\mathrm{estimation}}{⏟}}{{∥\widehat{\mathbf{H}}-\mathbf{H}∥}_F^2}$$

(43)

where $\lambda$ balances the two objectives. The encoder implicitly learns to embed structural redundancy into the transmitted signal, facilitating blind channel recovery at the receiver.

• Deep Blind Equalization. DL extends classical techniques such as the Constant Modulus Algorithm (CMA) by allowing neural networks to learn adaptive cost functions:

  $${\mathcal{L}}_{\mathrm{CMA}}=\mathbb{E}\left[{\left({\left|f_{\mathsf{\theta }}\left(\mathbf{y}\right)\right|}^2-R_2\right)}^2\right]$$

  (44)

  where $R_2=\mathbb{E}\left[{\left|x\right|}^4\right]/\mathbb{E}\left[{\left|x\right|}^2\right]$ is the target constant modulus. Neural networks generalize this criterion to arbitrary constellations and nonlinear channel conditions, providing improved convergence and reduced steady-state error.

3.5. Transfer Learning and Meta-Learning for Channel Estimation

In practical wireless systems, training DL models from scratch for each specific channel scenario—characterized by distinct propagation conditions, carrier frequencies, antenna configurations, and deployment environments—is computationally prohibitive and data-intensive. Transfer learning and meta-learning strategies address this challenge by enabling rapid adaptation to new scenarios using limited target-domain data.

Transfer Learning. A model pre-trained on a diverse set of source–channel scenarios can be adapted to a target scenario through selective fine-tuning of higher network layers, while preserving generalizable low-level feature representations in frozen lower layers. The adapted parameters are obtained by

$${\mathsf{\theta }}_{\mathrm{target}}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathsf{\theta }}_{\mathrm{upper}}}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.33em}\mathcal{L}\left(f_{\left[{\mathsf{\theta }}_{\mathrm{lower}}^{\mathrm{frozen}},{\mathsf{\theta }}_{\mathrm{upper}}\right]}\left({\mathcal{D}}_{\mathrm{target}}\right)\right)$$

(45)

where ${\mathsf{\theta }}_{\mathrm{lower}}^{\mathrm{frozen}}$ denotes the frozen parameters of lower layers and ${\mathsf{\theta }}_{\mathrm{upper}}$ are the trainable parameters of upper layers. This approach typically reduces the required target-domain training samples by 5–10× and adaptation time by an order of magnitude, while preserving generalizable feature representations learned from diverse channel environments.

• Model-Agnostic Meta-Learning (MAML). MAML seeks a parameter initialization $\mathsf{\theta }$ that enables rapid adaptation to new tasks (channel scenarios) with only a few gradient updates:

  $${\theta }^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathsf{\theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{\mathsf{\theta }}_i\prime }\right), \mathrm{where} {\mathsf{\theta }}_i^{\prime }=\mathsf{\theta }-\alpha {\nabla }_{\theta }{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\mathsf{\theta }}\right)$$

  (46)

For channel estimation, each task ${\mathcal{T}}_i$ corresponds to a different propagation scenario (e.g., indoor office, urban macro, vehicular highway). MAML learns an initialization that generalizes across diverse channel distributions, enabling efficient adaptation under distribution shifts with as few as 5–10 gradient steps and 50–100 target-domain samples.

• Complexity and Practical Considerations. Transfer learning introduces minimal computational overhead during fine-tuning (typically 1–5% of full training cost), while MAML requires second-order gradient computation during meta-training, increasing training cost by approximately 2–3×. However, both approaches dramatically reduce deployment-time adaptation costs, which is the primary practical concern for real-world systems.

3.6. Performance Evaluation and Comparative Analysis

Extensive empirical studies have compared DL-based channel estimation methods with classical model-driven techniques across a variety of propagation environments [15,16]. The key findings are summarized below, with a structured comparison provided in Table 2.

Table 2. Comparative Analysis of Channel Estimation and Prediction Methods.

Method	Category	Statistical Assumptions	Training Data Req.	Complexity	NMSE vs. MMSE	Adaptability	Key Ref.
Least Squares (LS)	Classical	None (model-free)	None	$\mathcal{O}\left(N_t\right)$	+5–10 dB worse	None	[23]
MMSE	Classical	Known 2nd-order statistics (${\mathbf{R}}_{HH}$)	None	$\mathcal{O}\left(N_r^3\right)$ (matrix inversion)	Baseline (0 dB)	Low (requires ${\mathbf{R}}_{HH}$ update)	[23,25]
DNN-based (Fully Connected)	AI-native	None (learned from data)	${10}^4$–${10}^5$ samples	$\mathcal{O}\left(L\cdot n^2\right)$ (fixed at inference)	−2 to −4 dB	Moderate (retraining)	[28]
CNN-based (2D freq.–time)	AI-native	Spatial/freq. locality prior	${10}^4$–${10}^5$ samples	$\mathcal{O}\left(C_{\mathrm{in}}{C}_{\mathrm{out}}{K}^2{S}^2\right)$	−3 to −5 dB	Moderate	[16]
LSTM-based (Channel Prediction)	AI-native	Temporal stationarity prior	${10}^4$–${10}^6$ sequences	$\mathcal{O}\left(4n_h^{2}T\right)$ per step	−2 to −4 dB (prediction)	High (sequential adaptation)	[16]
Transformer-based (Attention)	AI-native	Long-range dependency prior	${10}^5$–${10}^6$ samples	$\mathcal{O}\left(N^{2}d\right)$ (self-attention)	−3 to −6 dB	High (pre-train + fine-tune)	[16]
Sparse Recovery (LISTA)	Model-aided AI	Sparsity in delay/angle domain	${10}^3$–${10}^4$ samples	$\mathcal{O}\left(K\cdot N_t{N}_r\right)$ per iteration	−1 to −3 dB	Moderate	[24]
Transfer/Meta-Learning (MAML)	AI-native	Few-shot adaptation prior	${10}^2$–${10}^3$ target samples	Base model + fine-tune	−2 to −5 dB	Very high (few-shot)	[29]

In large-scale MIMO configurations operating at mmWave frequencies, neural architectures consistently demonstrate improved estimation accuracy relative to LS and MMSE, particularly under low SNR conditions and limited pilot resources. In representative massive MIMO scenarios (e.g., 64-antenna arrays under standardized 3GPP 38.901 Urban Micro (UMi) channel models), FCN estimators typically outperform LS by 5–8 dB and MMSE by 3–5 dB in NMSE, while CNN-based architectures further enhance performance by 1–2 dB through exploitation of spatial and frequency-domain correlations. Transformer-based models provide additional gains of 0.5–1.5 dB by capturing long-range dependencies across subcarriers and antenna elements. These improvements are especially pronounced in high-mobility regimes, where model mismatch degrades the effectiveness of classical estimators [15].

For channel prediction in vehicular environments, LSTM networks substantially reduce multi-step prediction error compared to classical AR models. Attention-based transformers further improve prediction stability over longer horizons, enabling proactive CSI acquisition and reducing feedback latency [16].

However, performance gains must be evaluated alongside computational complexity and real-world deployment feasibility. The MMSE estimator scales as $\mathcal{O}\left(N_r^3\right)$ due to matrix inversion, whereas DNN complexity depends primarily on network depth and hidden layer width. CNN architectures reduce parameter count through weight sharing, and transformer models introduce $\mathcal{O}\left(N^2\right)$ attention complexity. Practical deployment therefore requires model compression techniques—including INT8/INT4 quantization, structured pruning, and knowledge distillation—to satisfy latency constraints (sub-millisecond for URLLC) and energy budgets on embedded hardware platforms [26,27].

Overall, while AI-native channel estimators offer improved robustness and adaptability in complex propagation environments, their benefits depend on the availability of representative training data, efficient hardware acceleration, and careful architectural choices that balance accuracy against inference cost.

Table 2 summarizes representative channel estimation and prediction approaches, highlighting trade-offs between statistical assumptions, data dependency, computational complexity, and adaptability across classical and AI-native methods.

4. Signal Detection Using Neural Networks

Signal detection is a critical function of the wireless receiver, responsible for estimating transmitted symbols or information bits from noisy, distorted received signals [30]. In high-dimensional Multiple-Input Multiple-Output (MIMO) systems, optimal Maximum Likelihood (ML) detection exhibits exponential computational complexity in the number of transmit antennas and constellation size, which motivates the development of suboptimal approximations with reduced computational cost [31]. Deep Learning (DL) provides an alternative paradigm by enabling detection architectures that achieve a favourable balance between detection performance and computational complexity through learned, data-adaptive detection structures [14]. This section reviews the detection problem formulation, neural network architectures for direct and structured detection, extensions to non-ideal channels and multi-user scenarios, and implementation considerations.

Figure 3 illustrates the signal detection framework based on DL techniques, depicting the mapping from received signals and Channel State Information (CSI) to detected symbols or bits through a trained neural architecture.

Figure 3. Signal detection framework based on deep learning techniques.

Figure 3 shows the receiver processing chain in a MIMO system, where the received signal vector $\mathbf{y}\in {\mathbb{C}}^{N_r}$ and the estimated channel matrix $\widehat{\mathbf{H}}\in {\mathbb{C}}^{N_r\times N_t}$ are jointly fed into a neural network detector. The architecture comprises multiple hidden layers with nonlinear activations, producing either hard symbol decisions or soft bit-level log-likelihood ratios (LLRs) at the output. The key insight is that neural detectors learn implicit interference cancellation and noise suppression strategies from training data, achieving near-ML performance at computational costs comparable to linear detectors. The takeaway is that DL-based detection is particularly advantageous in high-dimensional MIMO systems ($N_t\ge 16$) where the ML complexity becomes prohibitive, offering practical deployment on FPGA and ASIC platforms with sub-10 μs latency.

4.1. Detection Problem Formulation

System Model. Consider the standard MIMO signal model:

$$\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{n}$$

(47)

where $\mathbf{y}\in {\mathbb{C}}^{N_r}$ is the received signal vector, $\mathbf{H}\in {\mathbb{C}}^{N_r\times N_t}$ is the channel matrix, $\mathbf{x}\in {\mathcal{X}}^{N_t}$ is the transmitted symbol vector with $\mathcal{X}$ denoting the signal constellation (e.g., $M$-QAM), and $\mathbf{n}\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_n^2\mathbf{I}\right)$ is AWGN. The detection objective is to estimate $\mathbf{x}$ by minimizing the probability of detection error $P_e=\mathrm{P}\mathrm{r}\left(\widehat{\mathbf{x}}\ne \mathbf{x}\right)$.

• Maximum Likelihood Detection. The ML detector computes:

$${\widehat{\mathbf{x}}}_{\mathrm{ML}}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathbf{x}\in {\mathcal{X}}^{N_t}}{\mathrm{m}\mathrm{i}\mathrm{n}}{∥\mathbf{y}-\mathbf{H}\mathbf{x}∥}^2$$

(48)

The computational complexity is $\mathcal{O}\left({\left|\mathcal{X}\right|}^{N_t}\right)$, which grows exponentially with the number of transmit antennas, rendering ML detection infeasible for systems with $N_t\ge 8$ and higher-order modulations.

• Linear Detectors. Zero-Forcing (ZF) and MMSE detectors provide linear solutions with polynomial complexity $\mathcal{O}\left(N_t^3\right)$:

  $${\widehat{\mathbf{x}}}_{\mathrm{MMSE}}={\left({\mathbf{H}}^H\mathbf{H}+{\sigma }_n^2\mathbf{I}\right)}^{-1}{\mathbf{H}}^H\mathbf{y}$$

  (49)

These detectors exhibit suboptimal performance, particularly under high system loading conditions where $N_t/N_r\to 1$, due to residual inter-stream interference that linear processing cannot fully suppress.

4.2. Neural Networks for Direct Detection

• DNN Detector. A direct approach employs a Deep Neural Network to map the received signal and CSI to the detected symbols:

  $$\widehat{\mathbf{x}}=f_{\mathsf{\theta }}\left(\mathbf{y},\widehat{\mathbf{H}}\right)$$

  (50)

  where the architecture comprise: Input: Concatenation of real and imaginary parts $\left[\mathrm{Re}\left(\mathbf{y}\right),\mathrm{Im}\left(\mathbf{y}\right),\mathrm{vec}\left(\mathrm{Re}\left(\widehat{\mathbf{H}}\right)\right),\mathrm{vec}\left(\mathrm{Im}\left(\widehat{\mathbf{H}}\right)\right)\right]$; Hidden Layers: Multiple fully connected layers with Rectified Linear Unit (ReLU) activation functions and optional batch normalization; Output: For complex symbol detection: $2N_t$ real-valued outputs; for bit detection: $N_t{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}M$ outputs with sigmoid activation.
• Loss Functions. For symbol-level detection, the MSE loss is employed:

$${\mathcal{L}}_{\mathrm{sym}}={\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{i=1}^{\left|\mathcal{D}\right|}{∥f_{\mathsf{\theta }}\left({\mathbf{y}}^{\left(i\right)},{\widehat{\mathbf{H}}}^{\left(i\right)}\right)-{\mathbf{x}}^{\left(i\right)}∥}^2$$

(51)

For bit-level soft-output detection, the binary cross-entropy (BCE) loss is used:

$${\mathcal{L}}_{\mathrm{bit}}=-{\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{i=1}^{\left|\mathcal{D}\right|}\sum _{j=1}^{N_t{\mathrm{l}\mathrm{o}\mathrm{g}}_{2}M}\left[b_j^{\left(i\right)}\mathrm{l}\mathrm{o}\mathrm{g}{\widehat{b}}_j^{\left(i\right)}+\left(1-b_j^{\left(i\right)}\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\widehat{b}}_j^{\left(i\right)}\right)\right]$$

(52)

where $b_j^{\left(i\right)}$ and ${\widehat{b}}_j^{\left(i\right)}$ denote the true and estimated bit probabilities, respectively.

Training Methodology. Training data are generated by sampling channel realizations from the target channel model (e.g., 3GPP Spatial Channel Model), random symbol vectors from ${\mathcal{X}}^{N_t}$, and noise vectors. The dataset typically comprises ${10}^5$–${10}^6$ samples, split into training (80%), validation (10%), and test (10%) sets. The Adam optimizer with learning rate scheduling (initial rate ${10}^{-3}$, decay factor 0.5 every 50 epochs) is commonly employed.

4.3. Attention and Transformer-Based Detection

• Attention-Based Detection. Attention mechanisms enable the detector to focus on the most relevant components of the received signal for detecting each transmitted symbol:

  $${\widehat{x}}_i=\sum _{j=1}^{N_r}{\alpha }_{ij}\hspace{0.17em}{\mathbf{v}}_j, {\alpha }_{ij}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathbf{q}}_i^T{\mathbf{k}}_j/\sqrt{d}\right)}{\sum _{l=1}^{N_r}\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathbf{q}}_i^T{\mathbf{k}}_l/\sqrt{d}\right)}}$$

  (53)

  where ${\mathbf{q}}_i$ represents the query used to detect symbol $i$, while ${\mathbf{k}}_j$ and ${\mathbf{v}}_j$ correspond to key and value projections derived from the received signals, and $d$ is the projection dimension.

✓

Transformer-Based Detector. A complete transformer-based detection architecture consists of the following elements:

• Embedding: Projection of the received signals into a high-dimensional representation space
• Multi-Head Self-Attention: Captures interdependencies among transmitted symbols across spatial streams
• Feed-Forward Networks: Performs nonlinear feature transformation with residual connections
• Output Layer: Produces symbol probabilities or bit-level LLRs

The computational complexity per transformer layer is $\mathcal{O}\left(N_t^{2}d\right)$, which is significantly more efficient than ML detection for moderate values of $N_t$ (e.g., $N_t\le 32$). For a $16\times 16$ MIMO system with 64-QAM, the transformer detector achieves within 0.5 dB of ML performance while reducing complexity by over five orders of magnitude.

4.4. Detection Under Non-Ideal Conditions

• Fading Channels. For time-varying fading channels, the training procedure incorporates channel statistics to build robustness:

  $$\mathbf{y}=\left(\mathbf{H}\odot \mathsf{\alpha }\right)\mathbf{x}+\mathbf{n}$$

  (54)

  where $\mathsf{\alpha }$ represents fading coefficients and $\odot$ denotes element-wise multiplication. The neural detector implicitly learns robustness to fading variations by training over diverse channel realizations, without requiring explicit diversity combining mechanisms.
• Hardware Impairments. Neural detectors can also compensate for hardware imperfections—including in-phase/quadrature (IQ) imbalance, power amplifier (PA) nonlinearities, phase noise, and quantization effects—by learning directly from measured data:

  $$\mathbf{y}=g_{\mathrm{HW}}\left(\mathbf{H}\mathbf{x}\right)+\mathbf{n}$$

  (55)

  where $g_{\mathrm{HW}}\left(\cdot \right)$ encapsulates the aggregate hardware distortion. The neural detector implicitly learns to invert hardware-induced distortions, enabling improved detection under non-ideal operating conditions that are difficult to model analytically. This capability is particularly valuable for low-cost devices with significant hardware imperfections and for mmWave systems where PA nonlinearities are pronounced.

4.5. Multi-User Detection

• DL-Based Non-Orthogonal Multiple Access (NOMA) Detection. In NOMA systems, multiple users share the same time–frequency resources through power-domain multiplexing. The received signal is:

  $$\mathbf{y}=\sum _{u=1}^U\sqrt{p_u}\hspace{0.17em}{\mathbf{h}}_u{x}_u+\mathbf{n}$$

  (56)

  where $p_u$, ${\mathbf{h}}_u$, and $x_u$ denote the transmit power, channel vector, and symbol of user $u$, respectively. A neural detector can perform learned successive interference cancellation (SIC):

  $${\widehat{x}}_u=f_{{\mathsf{\theta }}_u}\left(\mathbf{y},{\widehat{x}}_1,\dots ,{\widehat{x}}_{u-1}\right), u=1,\dots ,U$$

  (57)

  enabling joint multi-user detection by implicitly learning interference structures and decoding order, without relying on explicitly designed SIC procedures.
• Multi-Cell Detection with Graph Neural Networks (GNNs). In multi-cell scenarios with strong inter-cell interference, GNNs model the interference structure by representing each user as a node and interference relationships as edges:

  $${\mathbf{m}}_u^{\left(l+1\right)}=\mathrm{AGG}\left(\left\{f_{\mathsf{\theta }}^{\left(l\right)}\left({\mathbf{m}}_u^{\left(l\right)},{\mathbf{m}}_v^{\left(l\right)}\right):v\in \mathcal{N}\left(u\right)\right\}\right)$$

  (58)

  where $\mathcal{N}\left(u\right)$ denotes the set of interfering neighbours of user $u$, ${\mathbf{m}}_u^{\left(l\right)}$ is the message embedding at layer $l$, and AGG denotes a permutation-invariant aggregation function. The GNN propagates information across the interference graph to enable cooperative interference mitigation with complexity scaling linearly in the number of edges.

4.6. Computational Complexity and Hardware Implementation

• Complexity Comparison. Table 3 summarizes the computational complexity and relative performance of representative detection algorithms, including ML, ZF, MMSE, sphere decoding, DNN, and transformer-based detectors.

Table 3. Computational complexity and performance trade-offs of model-driven and deep learning-based detectors.

Detector	Computational Complexity (FLOPs)	BER Gap to ML (dB)	$\mathbf{Scalability} \mathbf{with} {\mathit{N}}_{\mathit{t}}$	HW Acceleration	Latency	Key Ref.
ML (Exhaustive Search)	$\mathcal{O}\left({\left|\mathcal{X}\right|}^{N_t}\right)$—exponential	0 (optimal)	Poor ($N_t>8$ infeasible)	Not practical	Very high	[30]
Zero-Forcing (ZF)	$\mathcal{O}\left(N_t^3\right)$—cubic (matrix inversion)	3–8 dB	Good	Moderate	Low	[30]
MMSE	$\mathcal{O}\left(N_t^3\right)$—cubic	2–5 dB	Good	Moderate	Low	[23]
Sphere Decoding (SD)	$\mathcal{O}\left(N_t^3\right)$ avg., $\mathcal{O}\left({\left|\mathcal{X}\right|}^{N_t}\right)$ worst	0–0.5 dB	Moderate (variable)	Difficult (irregular)	Variable	[31]
DNN Detector (FC)	$\mathcal{O}\left(L\cdot n^2\right)$—fixed polynomial	0.5–1.5 dB	Very good (fixed cost)	Excellent (FPGA/ASIC)	<10 μs	[14]
DetNet (Unfolded PGD)	$\mathcal{O}\left(K\cdot N_t^2\right)-K$ iterations	0.3–1.0 dB	Good	Excellent	<10 μs	[14]
Transformer Detector	$\mathcal{O}\left(N_t^2\cdot d\right)$—quadratic attention	0.2–0.8 dB	Good ($N_t\le 64$)	Good (GPU/TPU)	<50 μs	[18]
GNN Detector (MU-MIMO)	$\mathcal{O}\left(\left|\mathcal{E}\right|\cdot d\right)$—linear in edges	0.5–2.0 dB	Excellent (distributed)	Good	<20 μs	[14]

Table 3 compares detection approaches along the dimensions of computational complexity (in floating-point operations), detection performance relative to ML (in dB gap), scalability with ${\mathrm{N}}_{\mathrm{t}}$, and suitability for hardware acceleration. DL-based detectors occupy a favourable region in the complexity–performance trade-off space, achieving near-ML accuracy with polynomial or fixed complexity.

• FPGA Implementation. Neural detectors are highly parallelizable, making them well suited for hardware acceleration. Field-Programmable Gate Array (FPGA) implementations have demonstrated: throughput exceeding 1 Gbps for $4\times 4$ MIMO systems, latency below 10 μs, and energy efficiency greater than 100 Mbps/W. These results confirm the feasibility of deploying DL-based detectors in real-time communication systems [32,33].
• Quantization. Model quantization techniques, such as reducing weights and activations to INT8 or INT4 precision, decrease memory footprint and power consumption by factors of 4–8$\times$, with minimal performance degradation (typically less than 0.5 dB in Bit Error Rate (BER)) [34]. Post-training quantization and quantization-aware training represent complementary strategies, with the latter yielding smaller accuracy losses at the cost of additional training effort.
• Interpretability Considerations. Unlike classical detectors whose behaviour can be analysed through well-understood algebraic relationships, neural detectors operate as learned nonlinear mappings with limited interpretability. Algorithm unfolding—where iterative detection algorithms (e.g., projected gradient descent, expectation propagation) are mapped to trainable neural layers—provides a principled approach to combining the interpretability of model-driven methods with the adaptability of data-driven learning [14].

5. Intelligent Beamforming and Beam Management

Beamforming is a fundamental technique in Multiple-Input Multiple-Output (MIMO) and millimeter-wave (mmWave) communication systems, enabling the spatial concentration of transmitted energy to improve the Signal-to-Noise Ratio (SNR) and reduce interference [35]. In next-generation wireless systems employing large-scale antenna arrays and operating in dynamic propagation environments, AI-native adaptive beamforming provides significant advantages over traditional model-based approaches by learning complex spatial relationships directly from data and enabling real-time adaptation [36]. This section reviews beamforming fundamentals, Deep Learning (DL)-based beam prediction and tracking, end-to-end beamforming design, multi-user and cooperative strategies, hybrid analog–digital architectures, and Reconfigurable Intelligent Surface (RIS)-assisted beamforming.

Figure 4 illustrates the AI-based intelligent beamforming and beam management framework in wireless systems.

Figure 4. AI-based intelligent beamforming and beam management in wireless systems.

Figure 4 depicts a base station equipped with a large-scale antenna array serving multiple users in a dynamic propagation environment. The AI-native beamforming module receives Channel State Information (CSI) or raw received pilot signals as input and produces beamforming weight vectors or codebook indices as output. The framework encompasses beam prediction (selecting optimal beam directions from codebooks), beam tracking (adapting beam directions under mobility), multi-user beamforming (jointly optimizing beams for multiple users), and RIS phase configuration. The key insight is that neural network-based beam management replaces computationally expensive exhaustive beam search with learned mappings, reducing beam alignment latency by one to two orders of magnitude. The takeaway is that AI-native beamforming is essential for mmWave and sub-THz systems where beam coherence times are short and codebook sizes are large.

5.1. Fundamentals of Beamforming

• Array Response. For a Uniform Linear Array (ULA) with $N$ antenna elements and inter-element spacing $d$, the steering vector for an angle of arrival or departure $\theta$ is [37]:

  $$\mathbf{a}\left(\theta \right)={\displaystyle \frac{1}{\sqrt{N}}}{\left[1,\hspace{0.33em}{e}^{j{\displaystyle \frac{2\pi d}{\lambda }}\mathrm{s}\mathrm{i}\mathrm{n}\theta },\hspace{0.33em}\dots ,\hspace{0.33em}{e}^{j{\displaystyle \frac{2\pi d}{\lambda }}\left(N-1\right)\mathrm{s}\mathrm{i}\mathrm{n}\theta }\right]}^T$$

  (59)

  where $\lambda$ denotes the carrier wavelength.
• Transmit Beamforming. The transmitted signal vector is expressed as

  $$\mathbf{x}=\mathbf{w}\hspace{0.17em}s$$

  (60)

  where $\mathbf{w}\in {\mathbb{C}}^{N_t}$ is the beamforming weight vector and $s$ is the transmitted data symbol. The resulting radiation pattern is

  $$P\left(\theta \right)={\left|{\mathbf{a}}^H\left(\theta \right)\hspace{0.17em}\mathbf{w}\right|}^2$$

  (61)
• Classical Design. To maximize SNR in a target direction ${\theta }_0$, Maximum Ratio Transmission (MRT) yields

  $${\mathbf{w}}_{\mathrm{MRT}}={\displaystyle \frac{\mathbf{a}\left({\theta }_0\right)}{\parallel \mathbf{a}\left({\theta }_0\right)\parallel }}$$

  (62)

To simultaneously suppress interference in directions ${\theta }_1,\dots ,{\theta }_K$, the beamforming vector is designed to satisfy the null constraint ${\mathbf{a}}^H\left({\theta }_k\right)\mathbf{w}=0$ for $k=1,\dots ,K$, which can be obtained through projection-based methods:

$${\mathbf{w}}_{\mathrm{null}}=\left(\mathbf{I}-{\mathbf{A}}_{\mathrm{int}}{\left({\mathbf{A}}_{\mathrm{int}}^H{\mathbf{A}}_{\mathrm{int}}\right)}^{-1}{\mathbf{A}}_{\mathrm{int}}^H\right)\mathbf{a}\left({\theta }_0\right)$$

(63)

where ${\mathbf{A}}_{\mathrm{int}}=\left[\mathbf{a}\left({\theta }_1\right),\dots ,\mathbf{a}\left({\theta }_K\right)\right]$ is the interference steering matrix.

5.2. Deep Learning for Beam Prediction and Tracking

5.2.1. Beam Prediction from CSI

• Instead of performing exhaustive search over beamforming codebooks—which requires $\left|\mathcal{W}\right|$ measurements and incurs proportional latency—DL approaches directly predict the optimal beam index:

  $$\widehat{b}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{i\in 1,\dots ,\left|\mathcal{W}\right|}{\mathrm{m}\mathrm{a}\mathrm{x}}{f}_{\mathsf{\theta }}{\left(\widehat{\mathbf{H}}\right)}_i$$

  (64)

  where $\widehat{b}$ denotes the predicted beam index within the codebook $\mathcal{W}$.

  ✓

  CNN-Based Architecture. Convolutional Neural Networks exploit the spatial structure of CSI:

  • Input: CSI matrix $\left[\mathrm{Re}\left(\widehat{\mathbf{H}}\right),\mathrm{Im}\left(\widehat{\mathbf{H}}\right)\right]\in {\mathbb{R}}^{N_r\times N_t\times 2}$.
  • Convolutional Layers: Extraction of spatial features using 2D kernels.
  • Global Pooling: Aggregation of spatial information.
  • Fully Connected Layers: Mapping to beam index classification.
  • Output: Probability vector $\widehat{\mathbf{p}}\in {\left[0,1\right]}^{\left|\mathcal{W}\right|}$.

• Loss Function. Cross-entropy loss for classification:

  $${\mathcal{L}}_{\mathrm{beam}}=-\sum _{i=1}^{\left|\mathcal{W}\right|}{y}_i\mathrm{l}\mathrm{o}\mathrm{g}{\widehat{p}}_i$$

  (65)

  where $y_i=1$ if beam $i$ is the ground-truth optimal beam, and $y_i=0$ otherwise. The ground truth is obtained from exhaustive search during training data generation.

5.2.2. Beam Tracking Under Mobility

• In high-mobility scenarios (e.g., vehicular communications at speeds exceeding 100 km/h and unmanned aerial vehicle (UAV) systems), the optimal beam direction varies rapidly. Proactive beam tracking reduces latency and signalling overhead associated with beam realignment.
• RNN/LSTM-Based Beam Tracking. Future beam indices are predicted from historical beam and feature observations:

  $$\widehat{b}\left[t+\Delta \right]=f_{\mathsf{\theta }}^{\mathrm{track}}\left(b\left[t\right],b\left[t-1\right],\dots ,b\left[t-\tau \right],\hspace{0.33em}\mathbf{z}\left[t\right]\right)$$

  (66)

  where auxiliary features $\mathbf{z}\left[t\right]$ may include user position, velocity estimates, and SNR measurements. These recurrent models capture temporal dependencies in beam evolution and enable predictive beam selection.

Sequence-to-Sequence (Seq2Seq) with Attention for Multi-Step Prediction. For multi-step beam forecasting, the encoder processes the historical sequence while the decoder generates future beam sequences:

$$\widehat{b}\left[t+k\right]=f_{\mathrm{dec}}\left({\mathbf{c}}_k,\widehat{b}\left[t+k-1\right]\right), {\mathbf{c}}_k=\sum _j{\alpha }_{kj}\hspace{0.17em}{\mathbf{h}}_j^{\mathrm{enc}}$$

(67)

where ${\mathbf{c}}_k$ is the context vector and ${\alpha }_{kj}$ are learned attention weights that allow the decoder to focus on the most relevant portions of the historical sequence. This framework enables the anticipation of beam transitions several steps ahead, which is critical for maintaining beam alignment in rapidly varying environments.

5.3. Beamforming Without Explicit CSI and End-to-End Design

• Direct Learning from Received Signals. Instead of explicitly estimating CSI and subsequently computing beamforming vectors, DL enables direct mapping from received pilot measurements to beamformers:

  $$\widehat{\mathbf{w}}=f_{\mathsf{\theta }}\left({\mathbf{y}}_p\right)$$

  (68)

  where ${\mathbf{y}}_p$ denotes the received pilot signals, eliminating the intermediate channel estimation step and its associated error propagation.
• End-to-End Beamforming Optimization. Beamformers can be optimized directly by maximizing a system-level performance metric such as the achievable rate:

  $${\mathsf{\theta }}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathsf{\theta }}{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}\mathbb{E}\left[R\left(f_{\mathsf{\theta }}\left({\mathbf{y}}_p\right)\right)\right]$$

  (69)

  where $R\left(\mathbf{w}\right)={\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(1+{\displaystyle \frac{{\left|{\mathbf{h}}^H\mathbf{w}\right|}^2}{{\sigma }_n^2}}\right)$ represents the achievable rate. This formulation offers several advantages: it avoids error propagation from channel estimation, learns implicit compensation of hardware impairments, and reduces computational complexity by bypassing intermediate processing stages. The loss function is the negative expected rate, enabling unsupervised training without ground-truth beamforming labels.

5.4. Multi-User and Cooperative Beamforming

• Multi-User MISO (MU-MISO) Problem. For a base station with $N_t$ antennas serving $K$ single-antenna users, the objective is to design beamforming vectors $\{{\mathbf{w}}_k{\}}_{k=1}^K$ that maximize the weighted sum rate:

  $$\underset{{\mathbf{w}}_k}{\mathrm{m}\mathrm{a}\mathrm{x}}\sum _{k=1}^K{\omega }_k{\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(1+{\displaystyle \frac{{\left|{\mathbf{h}}_k^H{\mathbf{w}}_k\right|}^2}{\sum _{j\ne k}{\left|{\mathbf{h}}_k^H{\mathbf{w}}_j\right|}^2+{\sigma }_n^2}}\right), \mathrm{s}.\mathrm{t}. \sum _{k=1}^K\parallel {\mathbf{w}}_k{\parallel }^2\le P_{\mathrm{m}\mathrm{a}\mathrm{x}}$$

  (70)

  where ${\omega }_k$ are user priority weights and $P_{\mathrm{m}\mathrm{a}\mathrm{x}}$ is the total transmit power budget. This optimization is non-convex and generally NP-hard.
• DNN-Based MU Beamforming. A neural network learns the direct mapping:

  $$\left[{\widehat{\mathbf{w}}}_1,\dots ,{\widehat{\mathbf{w}}}_K\right]=f_{\mathsf{\theta }}\left({\mathbf{h}}_1,\dots ,{\mathbf{h}}_K\right)$$

  (71)

  with a power normalization layer enforcing the constraint

  $${\tilde{\mathbf{w}}}_k=\sqrt{P_{\mathrm{m}\mathrm{a}\mathrm{x}}}\cdot {\displaystyle \frac{{\widehat{\mathbf{w}}}_k}{\sqrt{\sum _{j=1}^K\parallel {\widehat{\mathbf{w}}}_j{\parallel }^2}}}$$

  (72)
• The architecture employs shared layers for extracting common multi-user features, followed by user-specific output branches. Unsupervised training directly maximizes the sum rate, eliminating the need for ground-truth beamforming solutions generated by expensive iterative algorithms:

$${\mathcal{L}}_{\mathrm{MU}}=-\sum _{k=1}^K{\omega }_k{\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(1+{\mathrm{SIN}\mathrm{R}}_k\left(f_{\mathsf{\theta }}\left({\mathbf{h}}_1,\dots ,{\mathbf{h}}_K\right)\right)\right)$$

(73)

• Cooperative Beamforming with Graph Neural Networks. In multi-cell wireless networks, coordinated beamforming across base stations (BSs) mitigates inter-cell interference. The network topology is modelled as a graph where nodes represent BS–user pairs and edges capture interference relationships. GNNs propagate information across the graph:

  $${\mathbf{m}}_v^{\left(l+1\right)}=\varphi \left({\mathbf{m}}_v^{\left(l\right)},\hspace{0.33em}\mathrm{AGG}\left(\left\{\psi \left({\mathbf{m}}_v^{\left(l\right)},{\mathbf{m}}_u^{\left(l\right)},e_{vu}\right):u\in \mathcal{N}\left(v\right)\right\}\right)\right)$$

  (74)

  where $e_{vu}$ encodes the interference strength between nodes $v$ and $u$, and $\varphi$, $\psi$ are learned functions. GNN-based coordination strategies scale gracefully with network size and naturally handle variable topologies, enabling distributed execution after centralized training.

5.5. Hybrid Beamforming and RIS-Assisted Systems

5.5.1. Hybrid Analog–Digital Beamforming

• Hybrid architectures combine analog beamforming (implemented via phase shifters with constant-modulus constraints) and digital baseband beamforming to reduce the number of radio-frequency (RF) chains:

  $$\mathbf{x}={\mathbf{F}}_{\mathrm{RF}}\hspace{0.17em}{\mathbf{F}}_{\mathrm{BB}}\hspace{0.17em}\mathbf{s}$$

  (75)

  where ${\mathbf{F}}_{\mathrm{RF}}\in {\mathbb{C}}^{N_t\times N_{\mathrm{RF}}}$ satisfies the constant-modulus constraint $\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{i,j}\right|=1/\sqrt{N_t}$ and ${\mathbf{F}}_{\mathrm{BB}}\in {\mathbb{C}}^{N_{\mathrm{RF}}\times N_s}$ is the digital precoder, with $N_{\mathrm{RF}}\ll N_t$ RF chains and $N_s$ data streams.
• DL-Based Hybrid Design. Neural networks learn the optimal decomposition:

  $$\left[{\widehat{\mathbf{F}}}_{\mathrm{RF}},{\widehat{\mathbf{F}}}_{\mathrm{BB}}\right]=f_{\mathsf{\theta }}\left(\widehat{\mathbf{H}}\right)$$

  (76)

  with constraint enforcement through architectural design. Analog Constraint (Phase Projection): ${\left[{\widehat{\mathbf{F}}}_{\mathrm{RF}}\right]}_{i,j}={\displaystyle \frac{1}{\sqrt{N_t}}}{e}^{j\mathrm{\angle }{\left[{\widehat{\mathbf{F}}}_{\mathrm{RF}}\right]}_{i,j}}$. Power Constraint (Normalization):

  $${\widehat{\mathbf{F}}}_{\mathrm{BB}}\leftarrow \sqrt{P_{\mathrm{m}\mathrm{a}\mathrm{x}}}\cdot {\widehat{\mathbf{F}}}_{\mathrm{BB}}/\parallel {\mathbf{F}}_{\mathrm{RF}}{\widehat{\mathbf{F}}}_{\mathrm{BB}}{\parallel }_F$$

  (77)
• Unfolded Alternating Optimization. Iterative hybrid beamforming algorithms (e.g., alternating minimization between ${\mathbf{F}}_{\mathrm{RF}}$ and ${\mathbf{F}}_{\mathrm{BB}}$) can be unfolded into trainable neural layers:

  $${\mathbf{F}}_{\mathrm{RF}}^{\left(k+1\right)}=\mathrm{PhaseProject}\left(g_{{\mathsf{\theta }}_1^{\left(k\right)}}\left({\mathbf{F}}_{\mathrm{RF}}^{\left(k\right)},{\mathbf{F}}_{\mathrm{BB}}^{\left(k\right)},\widehat{\mathbf{H}}\right)\right)$$

  (78)

  $${\mathbf{F}}_{\mathrm{BB}}^{\left(k+1\right)}=g_{{\mathsf{\theta }}_2^{\left(k\right)}}\left({\mathbf{F}}_{\mathrm{RF}}^{\left(k+1\right)},\widehat{\mathbf{H}}\right)$$

  (79)

  where each unfolded layer corresponds to one iteration with learned parameters, accelerating convergence by 5–10$\times$ over classical alternating optimization.

5.5.2. Beamforming for RIS-Assisted Systems

• RIS Channel Model. With an RIS comprising $N_{\mathrm{RIS}}$ reflecting elements, the effective end-to-end channel is

  $${\mathbf{H}}_{\mathrm{eff}}={\mathbf{H}}_d+{\mathbf{H}}_r\hspace{0.17em}\mathsf{\Theta }\hspace{0.17em}{\mathbf{H}}_t$$

  (80)

  where ${\mathbf{H}}_d$ is the direct BS-to-user channel, ${\mathbf{H}}_r$ and ${\mathbf{H}}_t$ are the RIS-to-user and BS-to-RIS channels, and $\mathsf{\Theta }=\mathrm{diag}\left(e^{j{\varphi }_1},\dots ,e^{j{\varphi }_{N_{\mathrm{RIS}}}}\right)$ represents the RIS phase configuration.
• Joint Optimization. The goal is to jointly design the BS beamformer $\mathbf{w}$ and RIS phases $\mathsf{\Theta }$:

  $$\underset{\mathbf{w},\mathsf{\Theta }}{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{\left|{\mathbf{h}}_{\mathrm{eff}}^H\left(\mathsf{\Theta }\right)\hspace{0.17em}\mathbf{w}\right|}^2 \mathrm{s}.\mathrm{t}. \parallel \mathbf{w}{\parallel }^2\le P_{\mathrm{m}\mathrm{a}\mathrm{x}},\hspace{0.33em}\left|{\Theta }_{nn}\right|=1\hspace{0.33em}\forall n$$

  (81)

DNN-Based RIS Optimization. DL models jointly learn the optimal configuration:

$$\left[\widehat{\mathbf{w}},\widehat{\mathsf{\varphi }}\right]=f_{\mathsf{\theta }}\left({\widehat{\mathbf{H}}}_d,{\widehat{\mathbf{H}}}_r,{\widehat{\mathbf{H}}}_t\right)$$

(82)

where $\widehat{\mathsf{\varphi }}=\left[{\widehat{\varphi }}_1,\dots ,{\widehat{\varphi }}_{N_{\mathrm{RIS}}}\right]$ are the predicted RIS phase shifts [38].

• Reinforcement Learning (RL) without Perfect CSI. When multi-hop CSI (${\mathbf{H}}_r$, ${\mathbf{H}}_t$) is difficult or impossible to obtain, RL optimizes RIS configurations based on observed performance metrics [39]:
  • State: Received SNR and previous RIS configuration.
  • Action: Phase adjustments $\mathsf{\Delta }\mathsf{\varphi }$.
  • Reward: Improvement in received SNR or achievable rate.

Algorithms such as Deep Deterministic Policy Gradient (DDPG) learn optimal configuration policies without requiring explicit channel models, enabling practical RIS deployment in scenarios where cascaded channel estimation is infeasible [40].

• Trade-offs and Real-World Constraints. RIS-assisted beamforming introduces additional optimization dimensions ($N_{\mathrm{RIS}}$ phase variables), which increases the search space significantly. DL-based approaches reduce the per-inference optimization time from seconds (iterative algorithms) to microseconds (single forward pass), but require offline training with representative channel data. Energy-efficient RIS operation and low-resolution phase quantization (e.g., 1–3 bit phase shifters) are important practical considerations that can be incorporated into the training loss as quantization constraints.

6. Spectral Resource Management Using Reinforcement Learning

Efficient allocation of spectral resources—including transmit power, bandwidth, time slots, and resource blocks—is essential for maximizing spectral efficiency and satisfying heterogeneous Quality-of-Service (QoS) requirements in next-generation wireless systems [2]. The dynamic, combinatorial, and often non-convex nature of resource allocation problems in multi-user, multi-cell environments motivates the use of Deep Reinforcement Learning (DRL) techniques, which learn allocation policies through interaction with the environment without requiring closed-form objective functions or complete system models [22]. This section provides a systematic review of the Markov Decision Process (MDP) formulation, value-based and policy gradient DRL algorithms, multi-agent extensions, and representative applications.

Figure 5 illustrates the DRL-based spectral resource management framework for wireless communication systems.

Figure 5. Deep learning-based spectral resource management in wireless communication systems.

Figure 5 depicts a multi-cell, multi-user wireless network where a DRL agent observes the system state (including CSI, traffic demands, buffer levels, and QoS requirements) and takes resource allocation actions (power levels, subchannel assignments, modulation and coding scheme (MCS) selections). The agent receives a reward signal reflecting system-level performance (e.g., sum throughput, energy efficiency, or QoS satisfaction). The key insight is that DRL formulates resource allocation as a sequential decision-making problem, enabling the agent to learn policies that adapt to time-varying channel conditions and traffic patterns without requiring explicit optimization models. The takeaway is that DRL-based approaches are particularly advantageous in dynamic heterogeneous networks where classical optimization becomes intractable due to the curse of dimensionality and non-convexity.

6.1. MDP Formulation for Resource Allocation

The resource allocation problem is modelled as an MDP defined by the tuple $\left(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma \right)$ [19]:

• State Space $\mathcal{S}$. The state encapsulates the system information available to the agent:

  $${\mathbf{s}}_t=\left[{\mathbf{h}}_t,\hspace{0.33em}{\mathbf{d}}_t,\hspace{0.33em}{\mathbf{q}}_t,\hspace{0.33em}{\mathsf{\rho }}_t\right]$$

  (83)

  where ${\mathbf{h}}_t$ represents Channel State Information (CSI), ${\mathbf{d}}_t$ denotes traffic demand indicators, ${\mathbf{q}}_t$ captures buffer states and queue lengths, and ${\mathsf{\rho }}_t$ encodes QoS requirements.

• Action Space $\mathcal{A}$. The action comprises the resource allocation decisions:

  $${\mathbf{a}}_t=\left[{\mathbf{p}}_t,\hspace{0.33em}{\mathsf{\chi }}_t,\hspace{0.33em}{\mathbf{m}}_t\right]$$

  (84)

  where ${\mathbf{p}}_t$ specifies power levels, ${\mathsf{\chi }}_t$ encodes subchannel assignments, and ${\mathbf{m}}_t$ indicates the selected MCS per user.
  • Reward $\mathcal{R}$. A utility function quantifying system-level performance:

    $$r_t=\mathcal{R}\left({\mathbf{s}}_t,{\mathbf{a}}_t\right)=\sum _{k=1}^K{\omega }_k{R}_k\left({\mathbf{a}}_t\right)-{\lambda }_{\mathrm{QoS}}\sum _{k=1}^K\mathrm{m}\mathrm{a}\mathrm{x}\left(0,\hspace{0.33em}{\rho }_k^{\mathrm{m}\mathrm{i}\mathrm{n}}-R_k\left({\mathbf{a}}_t\right)\right)$$

    (85)
  where the first term is the weighted sum rate and the second penalizes QoS violations with penalty coefficient ${\lambda }_{\mathrm{QoS}}$.

Objective. The goal is to learn an optimal policy ${\pi }^{\ast }$ that maximizes the expected cumulative discounted return:

$${\pi }^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\pi }{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\mathrm{\infty }}{\gamma }^t\hspace{0.17em}{r}_t\right]$$

(86)

where $\gamma \in \left[0,1\right)$ is the discount factor that controls the trade-off between immediate and future rewards.

6.2. Value-Based and Policy Gradient Methods

6.2.1. Deep Q-Networks (DQN) for Discrete Resource Allocation

• Q-Learning. The optimal action-value function satisfies the Bellman optimality equation:

$$Q^{\ast }\left(\mathbf{s},\mathbf{a}\right)=\mathbb{E}\left[r+\gamma \underset{{\mathbf{a}}^{\prime }}{\mathrm{m}\mathrm{a}\mathrm{x}}{Q}^{\ast }\left({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime }\right)\hspace{0.33em}|\hspace{0.33em}\mathbf{s},\mathbf{a}\right]$$

(87)

• DQN. A Deep Neural Network (DNN) approximates $Q^{\ast }$:

  $$Q\left(\mathbf{s},\mathbf{a};\mathsf{\theta }\right)\approx Q^{\ast }\left(\mathbf{s},\mathbf{a}\right)$$

  (88)

  with the architecture comprising an input state representation, hidden layers with ReLU activations, and output Q-values for each discrete action. Training minimizes the Bellman error through experience replay:

  $${\mathcal{L}}_{\mathrm{DQN}}\left(\mathsf{\theta }\right)={\mathbb{E}}_{\left(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime }\right)\sim \mathcal{B}}\left[{\left(r+\gamma \underset{{\mathbf{a}}^{\prime }}{\mathrm{m}\mathrm{a}\mathrm{x}}Q\left({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime };{\mathsf{\theta }}^{-}\right)-Q\left(\mathbf{s},\mathbf{a};\mathsf{\theta }\right)\right)}^2\right]$$

  (89)

  where $\mathcal{B}$ is the replay buffer and ${\theta }^{-}$ are periodically updated target network parameters that stabilize training.
• Double DQN. Reduces Q-value overestimation by decoupling action selection and evaluation:

$$Q_{\mathrm{target}}=r+\gamma \hspace{0.17em}Q\left({\mathbf{s}}^{\prime },\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathbf{a}}^{\prime }}{\mathrm{m}\mathrm{a}\mathrm{x}}Q\left({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime };\mathsf{\theta }\right);{\mathsf{\theta }}^{-}\right)$$

(90)

• Duelling DQN. Decomposes the Q-function into a state-value component and action-advantage component:

  $$Q\left(\mathbf{s},\mathbf{a};\mathsf{\theta }\right)=V\left(\mathbf{s};{\mathsf{\theta }}_V\right)+A\left(\mathbf{s},\mathbf{a};{\mathsf{\theta }}_A\right)-{\displaystyle \frac{1}{\left|\mathcal{A}\right|}}\sum _{{\mathbf{a}}^{\prime }}A\left(\mathbf{s},{\mathbf{a}}^{\prime };{\mathsf{\theta }}_A\right)$$

  (91)

  improving learning efficiency in states where the action choice has limited impact on the value.

6.2.2. Policy Gradient and Actor–Critic Methods

• Limitation of DQN. DQN requires a discrete and finite action space. Many resource allocation problems involve continuous actions (e.g., power levels) or combinatorically large discrete spaces, rendering DQN impractical.
• Policy Gradient (REINFORCE). The policy is parameterized directly and optimized using the policy gradient theorem:

  $${\nabla }_{\mathsf{\theta }}J\left(\mathsf{\theta }\right)={\mathbb{E}}_{{\pi }_{\mathsf{\theta }}}\left[{\nabla }_{\mathsf{\theta }}\mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\mathsf{\theta }}\left(\mathbf{a}|\mathbf{s}\right)\hspace{0.17em}{G}_t\right]$$

  (92)

  where $G_t=\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{r}_{t+k}$ is the discounted return. The REINFORCE estimator suffers from high variance, which is mitigated through baseline subtraction.
• Actor–Critic. This combines the policy gradient (actor) with a learned value function (critic) to reduce variance: Actor update: ${\mathsf{\theta }}_{\mathrm{actor}}\leftarrow {\mathsf{\theta }}_{\mathrm{actor}}+{\alpha }_a\hspace{0.17em}{\delta }_t\hspace{0.17em}{\nabla }_{\mathsf{\theta }}\mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\mathsf{\theta }}\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)$.

  ✓

  Critic update (advantage estimation): ${\delta }_t=r_t+\gamma V\left({\mathbf{s}}_{t+1};{\mathsf{\theta }}_V\right)-V\left({\mathbf{s}}_t;{\mathsf{\theta }}_V\right)$

• The Asynchronous Advantage Actor–Critic (A3C) variant parallelizes training with multiple agents exploring simultaneously, improving sample efficiency.

6.2.3. Proximal Policy Optimization (PPO)

PPO is a policy gradient algorithm that constrains policy updates to prevent catastrophic performance degradation:

$${\mathcal{L}}_{\mathrm{PPO}}\left(\mathsf{\theta }\right)=\mathbb{E}\left[\mathrm{m}\mathrm{i}\mathrm{n}\left({\rho }_t\left(\mathsf{\theta }\right)\hspace{0.17em}{\widehat{A}}_t,\hspace{0.33em}\mathrm{clip}\left({\rho }_t\left(\mathsf{\theta }\right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right]$$

(93)

where ${\rho }_t\left(\mathsf{\theta }\right)={\pi }_{\mathsf{\theta }}\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)/{\pi }_{{\mathsf{\theta }}_{\mathrm{old}}}\left({\mathbf{a}}_t|{\mathbf{s}}_t\right)$ is the probability ratio and $ϵ$ (typically 0.2) controls the clipping range. PPO offers greater stability than vanilla policy gradients, simplicity relative to Trust Region Policy Optimization (TRPO), and effectiveness in both continuous and discrete action spaces, making it widely adopted for wireless resource management.

6.3. Multi-Agent Reinforcement Learning (MARL)

In multi-cell or multi-user networks, multiple agents (e.g., base stations, users, or network slices) must coordinate resource allocation decisions. Three principal paradigms exist:

• Centralized: A global agent controls all resources. This achieves globally optimal solutions but suffers from limited scalability and high communication overhead.
• Decentralized: Each agent decides independently based on local observations. This offers maximal scalability but may converge to suboptimal equilibria due to non-stationarity of other agents’ policies.
• Centralized Training with Decentralized Execution (CTDE): Agents are trained using global information but execute decisions based solely on local observations, providing an effective trade-off between performance and scalability.
• Multi-Agent DDPG (MADDPG). An extension of Deep Deterministic Policy Gradient for CTDE:

$${\nabla }_{{\mathsf{\theta }}_i}J\left({\mathsf{\theta }}_i\right)=\mathbb{E}\left[{\nabla }_{{\mathsf{\theta }}_i}{\pi }_i\left({\mathbf{a}}_i|{\mathbf{o}}_i\right)\cdot {\nabla }_{{\mathbf{a}}_i}{Q}_i\left(\mathbf{s},{\mathbf{a}}_1,\dots ,{\mathbf{a}}_K\right){|}_{{\mathbf{a}}_i={\pi }_i\left({\mathbf{o}}_i\right)}\right]$$

(94)

where each agent $i$ has an actor using only local observation ${\mathbf{o}}_i$ and a critic using the global state and all agents’ actions.

• QMIX. For cooperative scenarios, QMIX factors the global Q-function using a monotonicity constraint:

  $$Q_{\mathrm{tot}}\left(\mathbf{s},{\mathbf{a}}_1,\dots ,{\mathbf{a}}_K\right)=f_{\mathrm{mix}}\left(Q_1\left({\mathbf{o}}_1,{\mathbf{a}}_1\right),\dots ,Q_K\left({\mathbf{o}}_K,{\mathbf{a}}_K\right);\mathbf{s}\right), {\displaystyle \frac{\partial Q_{\mathrm{tot}}}{\partial Q_i}}\ge 0\hspace{0.33em}\forall i$$

  (95)

  where $f_{\mathrm{mix}}$ is a mixing network whose weights are generated by a hypernetwork conditioned on the global state. This allows centralized training with greedy decentralized execution.

6.4. Representative Applications

• Power Control. Continuous adjustment of per-user transmit power to maximize system Signal-to-Interference-plus-Noise Ratio (SINR) while minimizing inter-cell interference. DDPG or Twin Delayed DDPG (TD3) algorithms are effective for the continuous power action space, with reward defined as $r=\sum _k{\mathrm{l}\mathrm{o}\mathrm{g}}_2\left(1+{\mathrm{SIN}\mathrm{R}}_k\right)$ subject to per-user power constraints.
• Dynamic Spectrum Sharing. Assignment of frequency bands to users and cells, formulated as a discrete action problem suitable for DQN or PPO. The reward balances total throughput against collision penalties from overlapping allocations.
• Dynamic TDD Configuration. Optimization of uplink/downlink time-slot direction in each cell, employing MARL with coordination between neighbouring cells to manage cross-link interference.
• Network Slice Resource Allocation. In network slicing environments, resources are allocated to slices with different Service Level Agreements (SLAs). Hierarchical RL—where the high-level agent assigns resources to slices and the low-level agent distributes resources among users within each slice—provides an effective decomposition.
• Energy-Efficient Resource Allocation. Maximization of energy efficiency ${\eta }_{\mathrm{EE}}=\sum _k{R}_k/P_{\mathrm{total}}$, requiring multi-objective RL with learned trade-offs between achievable rate and power consumption.

6.5. Transfer Learning and Meta-RL for Adaptation

• Challenge. Training DRL agents from scratch for each deployment scenario is prohibitively expensive in terms of both computational cost and required interaction time with the real environment.
• Domain Adaptation. Agents are pre-trained in simulation and adapted to the real world through fine-tuning, employing domain randomization (training over a wide distribution of simulated parameters) and adversarial adaptation (training a discriminator to distinguish simulated from real experiences) to bridge the simulation-to-reality gap [41].
• Meta-RL: Learning policies that adapt rapidly to new tasks.
• RL² (Learning to Reinforcement Learn): An RNN acts as a complete RL algorithm, learning to explore and exploit within its hidden state dynamics [42].
• MAML for RL: Seeks a policy parameter initialization enabling rapid adaptation to new environments with only a few gradient updates [29].

$${\mathsf{\theta }}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathsf{\theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left({\pi }_{{\mathsf{\theta }}_i\prime }\right), {\mathsf{\theta }}_i^{\prime }=\mathsf{\theta }-\alpha {\nabla }_{\mathsf{\theta }}{\mathcal{L}}_{{\mathcal{T}}_i}\left({\pi }_{\mathsf{\theta }}\right)$$

(96)

These meta-learning approaches enable DRL agents to adapt to previously unseen network configurations—such as new cell layouts, traffic patterns, or propagation conditions—within tens of episodes rather than thousands, significantly improving deployment feasibility in dynamic wireless environments [43].

• Real-World Deployment Considerations. DRL-based resource management faces challenges including safety during online exploration (where poor actions can degrade service), non-stationarity of multi-agent environments, and computational requirements for real-time policy inference. Offline RL methods, which learn from pre-collected datasets without online interaction, and conservative policy evaluation techniques offer pathways to safer deployment.

7. Adaptive Modulation and Intelligent Waveforms

The selection of modulation schemes and waveform designs based on instantaneous channel conditions is critical for maximizing spectral efficiency and transmission reliability in next-generation wireless systems [44]. Traditional Adaptive Modulation and Coding (AMC) relies on threshold-based mappings from Channel State Information (CSI) to Modulation and Coding Scheme (MCS) selections, which are inherently limited in their ability to exploit temporal correlations, traffic patterns, and channel-specific characteristics. Deep Learning (DL) enables substantially finer-grained adaptation and proactive prediction, surpassing conventional rule-based approaches [45]. This section reviews classical link adaptation, DL-based MCS selection and prediction, learned neural constellations, intelligent waveform design including Peak-to-Average Power Ratio (PAPR) reduction, and waveform techniques for spectral coexistence.

Figure 6 illustrates the DL-based framework for adaptive modulation, prediction, and intelligent waveform design.

Figure 6. Deep learning-based MCS selection, proactive prediction, and intelligent waveform design.

Figure 6 shows a unified framework where the DL module takes as input multi-dimensional channel features (including instantaneous Signal-to-Interference-plus-Noise Ratio (SINR), Doppler spread, delay spread, and ACK/NACK history) and produces either an MCS selection, a predicted future MCS sequence, or an optimized waveform representation. The key insight is that neural classifiers exploit correlations across multiple channel quality indicators simultaneously, whereas classical AMC relies on a single SINR threshold, resulting in throughput gains of 10–20% in dynamic environments. For waveform design, end-to-end autoencoders learn time-domain signal representations that inherently reduce PAPR and inter-symbol interference without requiring cyclic prefixes. The takeaway is that DL-based link adaptation and waveform design provide joint optimization opportunities that classical block-based designs cannot exploit.

7.1. Classical Link Adaptation

• Adaptive Modulation and Coding (AMC). AMC adjusts the MCS based on estimated channel quality, typically quantified by the SINR [46]:

  $${\mathrm{MCS}}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathrm{MCS}\in \mathcal{M}}{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{R}_{\mathrm{MCS}} \mathrm{s}.\mathrm{t}. \mathrm{BLER}\left(\mathrm{MCS},\mathrm{SIN}\mathrm{R}\right)\le {\mathrm{BLER}}_{\mathrm{target}}$$

  (97)

  where $R_{\mathrm{MCS}}$ is the spectral efficiency of the candidate MCS, $\mathrm{BLER}$ denotes the block error rate, and ${\mathrm{BLER}}_{\mathrm{target}}$ is the target error rate (e.g., ${10}^{-1}$ for initial transmission). A typical threshold-based mapping employs fixed SINR boundaries corresponding to specific MCS levels, as summarized in Table 4.

Table 4. Typical SINR-to-MCS mapping for adaptive modulation and coding (AMC).

MCS Index	Modulation	Code Rate	Spectral Efficiency (bps/Hz)	SINR Threshold (dB)
0	QPSK	0.12	0.23	−6.7
1	QPSK	0.19	0.38	−4.7
2	QPSK	0.30	0.60	−2.3
3	QPSK	0.44	0.88	0.2
4	QPSK	0.59	1.18	2.4
5	16-QAM	0.37	1.48	4.7
6	16-QAM	0.48	1.91	6.5
7	16-QAM	0.60	2.41	8.6
8	16-QAM	0.66	2.64	10.3
9	64-QAM	0.46	2.73	11.7
10	64-QAM	0.55	3.32	14.1
11	64-QAM	0.65	3.90	16.3
12	64-QAM	0.75	4.52	18.7
13	64-QAM	0.85	5.12	21.0
14	256-QAM	0.65	5.16	22.7
15	256-QAM	0.75	5.99	25.0

Table 4 lists representative SINR threshold ranges (in dB) and the corresponding MCS indices, modulation orders (QPSK, 16-QAM, 64-QAM, 256-QAM), and code rates, illustrating the staircase adaptation function used in conventional systems.

• Limitations of Classical AMC: fixed thresholds do not adapt to channel-specific statistical characteristics; limited predictive capability in rapidly time-varying channels with significant feedback delay; lack of exploitation of temporal correlation, multi-dimensional channel features, and traffic patterns

7.2. Deep Learning-Based MCS Selection and Prediction

7.2.1. Neural MCS Classification

A neural network maps multi-dimensional channel feature to the optimal MCS:

$$\widehat{\mathrm{MCS}}=f_{\mathsf{\theta }}\left(\mathrm{SINR},\hspace{0.33em}{f}_D,\hspace{0.33em}{\tau }_{\mathrm{rms}},\hspace{0.33em}\mathrm{ACK}/\mathrm{NACK} \mathrm{history},\hspace{0.33em}{\mathbf{d}}_{\mathrm{traffic}}\right)$$

(98)

where $f_D$ denotes the Doppler spread (mobility indicator), ${\tau }_{\mathrm{rms}}$ is the root-mean-square delay spread (multipath dispersion), and ${\mathbf{d}}_{\mathrm{traffic}}$ captures traffic demand characteristics. The architecture comprises an input layer with concatenated features, hidden fully connected layers with ReLU activation and dropout regularization, and a softmax output over available MCS levels.

• Loss Function. A throughput-weighted cross-entropy loss encourages the network to favour higher-rate MCS selections when feasible:

  $${\mathcal{L}}_{\mathrm{MCS}}=-\sum _{i=1}^{\left|\mathcal{M}\right|}{R}_i\cdot y_i\mathrm{l}\mathrm{o}\mathrm{g}{\widehat{p}}_i$$

  (99)

  where $R_i$ weights the classification loss by the achievable rate of MCS $i$, $y_i$ is the ground-truth indicator, and ${\widehat{p}}_i$ is the predicted probability. Empirical studies report throughput gains of approximately 10–20% compared to conventional threshold-based AMC in dynamic channel conditions, due to the network’s ability to exploit multi-dimensional feature correlations.

7.2.2. Proactive MCS Prediction

• Challenge. Feedback latency (typically 1–4 ms in 5G New Radio (NR)) leads to a mismatch between the CSI used for MCS selection and the actual channel at transmission time, particularly in high-mobility scenarios.
• Long Short-Term Memory (LSTM)-Based Prediction. The optimal MCS is predicted $\Delta$ steps ahead:

  $$\widehat{\mathrm{MCS}}\left[t+\Delta \right]=f_{\mathsf{\theta }}^{\mathrm{pred}}\left(\mathrm{SIN}\mathrm{R}\left[t\right],\mathrm{SIN}\mathrm{R}\left[t-1\right],\dots ,\mathrm{SIN}\mathrm{R}\left[t-\tau \right]\right)$$

  (100)
• Seq2Seq for Joint Scheduling and MCS Prediction:

  $$\left[\widehat{\mathrm{MCS}}\left[t+1\right],\dots ,\widehat{\mathrm{MCS}}\left[t+\Delta \right]\right]=f_{\mathrm{seq}2\mathrm{seq}}\left(\mathbf{s}\left[t-\tau :t\right]\right)$$

  (101)

  enabling proactive scheduling and reduced effective feedback latency, improving throughput by 5–15% under high mobility conditions (vehicle speeds $>60$ km/h).

7.3. Learned Modulation: Neural Constellations

• Beyond QAM. Conventional constellations (e.g., $M$-PSK, $M$-QAM) are designed for Additive White Gaussian Noise (AWGN) channels and may be suboptimal under non-Gaussian noise, fading, or hardware impairments. DL enables the design of constellations tailored to specific channel conditions.
• Autoencoder-Based Constellation Design. The encoder learns the bit-to-symbol mapping and the decoder learns the inverse:

  $$\mathbf{z}=f_{\mathrm{enc}}\left(\mathbf{b};{\mathsf{\theta }}_{\mathrm{enc}}\right), \widehat{\mathbf{b}}=f_{\mathrm{dec}}\left(\mathbf{y};{\mathsf{\theta }}_{\mathrm{dec}}\right)$$

  (102)

  subject to the average power constraint $\mathbb{E}\left[\parallel \mathbf{z}{\parallel }^2\right]\le P_{\mathrm{avg}}$ and implicit symbol separability requirements. End-to-end training minimizes

  $${\mathcal{L}}_{\mathrm{const}}={\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{i=1}^{\left|\mathcal{D}\right|}\mathrm{CE}\left({\mathbf{b}}^{\left(i\right)},f_{\mathrm{dec}}\left(f_{\mathrm{channel}}\left(f_{\mathrm{enc}}\left({\mathbf{b}}^{\left(i\right)}\right)\right)\right)\right)$$

  (103)

  where $\mathrm{CE}$ denotes the cross-entropy loss and $f_{\mathrm{channel}}$ represents a differentiable channel model (or a learned channel approximation for non-differentiable channels). Learned constellations achieve gains of 1–2 dB over conventional QAM under severe Rayleigh fading and with hardware nonlinearities, by adapting the constellation geometry to minimize error probability under the true channel distribution.

7.4. Intelligent Waveform Design and PAPR Reduction

7.4.1. Learned Waveforms

• Beyond Orthogonal Frequency Division Multiplexing (OFDM). While OFDM is widely used due to its robustness to multipath propagation, it suffers from high PAPR, sensitivity to Carrier Frequency Offset (CFO), and cyclic prefix (CP) overhead that reduces spectral efficiency.
• DL for Waveform Design. An end-to-end autoencoder generates and recovers time-domain waveforms directly:

  $${\mathbf{x}}_{\mathrm{time}}=f_{\mathrm{enc}}\left(\mathbf{b};{\mathsf{\theta }}_{\mathrm{enc}}\right)$$

  (104)

  $$\widehat{\mathbf{b}}=f_{\mathrm{dec}}\left({\mathbf{x}}_{\mathrm{time}}\ast \mathbf{h}+\mathbf{n};{\mathsf{\theta }}_{\mathrm{dec}}\right)$$

  (105)

  where $\ast$ denotes convolution with the channel impulse response $\mathbf{h}$. The temporal autoencoder learns waveform representations with inherently reduced PAPR (no constructive subcarrier superposition), implicit inter-symbol interference (ISI) robustness (eliminating the need for CP), and adaptation to specific channel conditions. Challenges include synchronization (requiring jointly learned preamble and timing structures), generalization to variable block lengths, and implementation complexity on conventional hardware designed for OFDM processing.

7.4.2. PAPR Reduction

• PAPR Problem. OFDM signals exhibit high PAPR, causing distortion in nonlinear power amplifiers (PAs):

$$\mathrm{PAPR}={\displaystyle \frac{\underset{t}{\mathrm{m}\mathrm{a}\mathrm{x}}{\left|x\left(t\right)\right|}^2}{\mathbb{E}\left[{\left|x\left(t\right)\right|}^2\right]}}$$

(106)

Classical reduction techniques (clipping and filtering, Selective Mapping (SLM), Partial Transmit Sequence (PTS)) introduce trade-offs between BER degradation, side-information overhead, and computational complexity.

• DNN for PAPR Reduction. A neural network learns a signal transformation that reduces PAPR while maintaining BER:

$$\tilde{\mathbf{x}}=f_{\mathsf{\theta }}^{\mathrm{PAPR}}\left({\mathbf{x}}_{\mathrm{OFDM}}\right)$$

(107)

The multi-objective loss function balances PAPR reduction and signal fidelity:

$${\mathcal{L}}_{\mathrm{PAPR}}=\underset{\underset{\mathrm{PAPR} \mathrm{reduction}}{⏟}}{\mathrm{PAPR}\left(\tilde{\mathbf{x}}\right)}+\beta \underset{\underset{\mathrm{signal} \mathrm{distortion}}{⏟}}{\parallel \tilde{\mathbf{x}}-{\mathbf{x}}_{\mathrm{OFDM}}{\parallel }^2}+\gamma \underset{\underset{\mathrm{BER} \mathrm{penalty}}{⏟}}{{\mathcal{L}}_{\mathrm{BER}}}$$

(108)

where $\beta$ and $\gamma$ control the trade-off weights. This approach achieves 2–3 dB PAPR reduction with minimal BER degradation ($<0.5$ dB).

• Nonlinearity-Aware Autoencoder. Training with a differentiable nonlinear PA model in the loop:

  $$\mathbf{y}=g_{\mathrm{PA}}\left(f_{\mathrm{enc}}\left(\mathbf{b}\right)\right)\ast \mathbf{h}+\mathbf{n}$$

  (109)

  where $g_{\mathrm{PA}}\left(\cdot \right)$ models the amplifier nonlinearity (e.g., Rapp or Saleh model). The encoder implicitly learns pre-distortion, producing signals that remain within the PA’s linear operating region.

7.5. Waveform Design for Spectral Coexistence

In multi-service scenarios (enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communication (URLLC), massive Machine-Type Communication (mMTC)), different numerologies coexist within the same carrier bandwidth, leading to inter-numerology interference (INI).

• Multi-Numerology Challenge. Different subcarrier spacings are employed for different service types: eMBB: 15 kHz (high spectral efficiency); URLLC: 60 kHz (low latency); and mMTC: 3.75 kHz (extended coverage)

The non-orthogonality between numerologies generates INI, which degrades performance.

• DL for Inter-Numerology Interference Cancellation: Neural receivers are trained to suppress interference from neighbouring numerologies [47]:

  $${\widehat{\mathbf{b}}}_{\mathrm{eMBB}}=f_{\mathsf{\theta }}\left(\mathbf{y},\hspace{0.33em}{\mathbf{z}}_{\mathrm{URLLC}}\right)$$

  (110)

  where side information ${\mathbf{z}}_{\mathrm{URLLC}}$ about the URLLC signalling parameters aids interference cancellation.
• OFDM Filtering with Learned Filters: Per-subband filters are designed using DNNs to minimize spectral leakage [48]

  $${\widehat{\mathbf{g}}}_k=f_{\mathsf{\theta }}^{\mathrm{filter}}\left(k,\hspace{0.33em}\mathsf{\Delta }{f}_k,\hspace{0.33em}{\mathrm{SCS}}_k\right)$$

  (111)

  where $k$ indexes the subband, $\mathsf{\Delta }{f}_k$ is the subcarrier spacing, and ${\mathrm{SCS}}_k$ encodes numerology parameters. Learned filters achieve 10–15 dB improved Adjacent Channel Leakage Ratio (ACLR) compared to classical windowing techniques [49], enabling tighter spectral coexistence with minimal guard-band overhead.
• Energy Efficiency and Hardware Considerations: Intelligent waveform techniques must balance performance gains against implementation complexity on existing hardware platforms. Learned waveforms require departure from standardized OFDM processing chains, posing challenges for backward compatibility. PAPR reduction and learned filtering, by contrast, can be implemented as add-on modules within existing architectures, offering a more practical deployment pathway.

8. Semantic and Task-Oriented Communications

Semantic communication represents a fundamental departure from the classical Shannon paradigm by shifting the communication objective from accurate bit-level reconstruction to the faithful conveyance of meaning and the effective completion of downstream tasks [50,51]. Rather than treating all transmitted bits as equally important, semantic communication systems leverage Deep Learning (DL) to extract, encode, and transmit compact semantic representations that preserve task-relevant information while discarding redundant data. This approach enables exponential improvements in bandwidth efficiency for specific applications, making it particularly relevant for next-generation wireless systems with stringent latency and bandwidth constraints. This section reviews the theoretical foundations of semantic communication, DL-based semantic extraction and end-to-end system design, goal-oriented communication, representative applications, and key challenges.

Figure 7 illustrates the DL-based semantic communication framework.

Figure 7. Deep learning-based semantic communication framework.

Figure 7 depicts the end-to-end semantic communication architecture, comprising a semantic encoder (which extracts task-relevant latent representations from the source data), a channel encoder (which maps semantic features to channel-compatible symbols), the physical channel, a channel decoder (which recovers semantic features from noisy received signals), and a semantic decoder (which produces the task output). Unlike classical systems that optimize for bit-level fidelity, this architecture is trained end-to-end to minimize a task-oriented loss function. The key insight is that by transmitting compact semantic representations instead of raw data, the system achieves bandwidth reductions of 5–100$\times$ while maintaining or improving task performance (e.g., classification accuracy, control quality). The takeaway is that semantic communication redefines the communication design problem from “how to transmit bits reliably” to “how to transmit meaning efficiently”, with DL serving as the key enabler for semantic extraction and joint source–channel coding.

8.1. Foundations of Semantic Communication

• Shannon’s Classical Architecture: Traditional communication systems separate source coding and channel coding according to Shannon’s separation theorem [11], shown below.

$$\mathbf{s}\stackrel{\mathrm{Source} \mathrm{Enc}.}{\to }\mathbf{b}\stackrel{\mathrm{Channel} \mathrm{Enc}.}{\to }\mathbf{x}\stackrel{\mathrm{Channel}}{\to }\mathbf{y}\stackrel{\mathrm{Channel} \mathrm{Dec}.}{\to }\widehat{\mathbf{b}}\stackrel{\mathrm{Source} \mathrm{Dec}.}{\to }\widehat{\mathbf{s}}.$$

The design objective is to reconstruct the transmitted bits $\mathbf{b}$ with arbitrarily small error probability, which is optimal in the asymptotic regime of infinite block length. However, for finite block lengths and practical applications such as machine vision, natural language understanding, or control systems, exact bit reconstruction may be unnecessary and wasteful if the task objective can be achieved with a significantly compressed representation.

• Weaver’s Extended Communication Model: Weaver extended Shannon’s framework by distinguishing three levels of communication: Level A (Technical): Accuracy of transmitted symbols; Level B (Semantic): Accuracy of conveyed meaning; Level C (Effectiveness): Impact on intended behaviour or task completion.

Semantic communication directly optimizes Levels B and C, potentially bypassing exact reconstruction at Level A. This perspective has received renewed attention with the advent of DL, which provides the computational tools to learn semantic representations and task-oriented encoding strategies.

• Information-Theoretic Framework: From an information-theoretic perspective, semantic communication can be formalized through rate–distortion theory with task-relevant distortion measures. The semantic rate–distortion function characterizes the minimum transmission rate required to achieve a given level of task performance:

  $$R_{\mathrm{sem}}\left(D\right)=\underset{p\left(\widehat{\mathbf{s}}|\mathbf{s}\right):\mathbb{E}\left[d_{\mathrm{task}}\left(\mathbf{s},\widehat{\mathbf{s}}\right)\right]\le D}{\mathrm{m}\mathrm{i}\mathrm{n}}I\left(\mathbf{s};\widehat{\mathbf{s}}\right)$$

  (112)

  where $d_{\mathrm{task}}\left(\cdot ,\cdot \right)$ is a task-specific distortion measure (e.g., classification error, control cost) and $I\left(\mathbf{s};\widehat{\mathbf{s}}\right)$ is the mutual information. Cai et al. [13] have established information-theoretic frameworks for task-oriented semantic communications that formalize these concepts, providing fundamental limits and design principles for systems where the communication objective is aligned with the downstream inference or control task rather than data reconstruction.

8.2. Deep Learning for Semantic Extraction and End-to-End Transmission

8.2.1. Semantic Feature Extraction

• Latent Representations: Deep encoders extract compact semantic features from high-dimensional source data

  $$\mathbf{z}=f_{\mathrm{sem}}\left(\mathbf{s};{\mathsf{\theta }}_{\mathrm{sem}}\right)$$

  (113)

  where $\mathbf{s}$ is the source data (image, text, sensor signal) and $\mathbf{z}\in {\mathbb{R}}^d$ with $d\ll \mathrm{d}\mathrm{i}\mathrm{m}\left(\mathbf{s}\right)$ is the semantic representation.
• Variational Autoencoders (VAEs): VAEs learn a probabilistic distribution over semantic representation

  $$q_{\mathsf{\varphi }}\left(\mathbf{z}|\mathbf{s}\right)=\mathcal{N}\left(\mathbf{z};{\mathsf{\mu }}_{\mathsf{\varphi }}\left(\mathbf{s}\right),\mathrm{diag}\left({\mathsf{\sigma }}_{\mathsf{\varphi }}^2\left(\mathbf{s}\right)\right)\right)$$

  (114)

  trained by maximizing the evidence lower bound (ELBO), which balances reconstruction quality against regularization of the latent space. The learned representations capture semantic factors of variation while maintaining a structured latent space amenable to transmission.
• Contrastive Learning: Self-supervised contrastive learning trains encoders to produce representations that group semantically similar inputs

  $${\mathcal{L}}_{\mathrm{contrastive}}=-\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{sim}\left({\mathbf{z}}_i,{\mathbf{z}}_j^{+}\right)/\tau \right)}{\sum _{k=1}^{2N}{\mathbb{1}}_{\left[k\ne i\right]}\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{sim}\left({\mathbf{z}}_i,{\mathbf{z}}_k\right)/\tau \right)}}$$

  (115)

  where $\left({\mathbf{z}}_i,{\mathbf{z}}_j^{+}\right)$ are semantically similar (positive) pairs, the denominator includes negative pairs, $\mathrm{sim}\left(\cdot ,\cdot \right)$ is the cosine similarity, and $\tau$ is a temperature parameter. This approach does not require labelled data for pre-training, reducing the dependency on task-specific annotations.

8.2.2. End-to-End Semantic Communication Systems

The complete semantic communication pipeline transmits learned representations through the physical channel:

✓

Semantic Encoder: $\mathbf{z}=f_{\mathrm{sem}}\left(\mathbf{s};{\mathsf{\theta }}_{\mathrm{sem}}\right)$.

✓

Channel Encoder: $\mathbf{x}=f_{\mathrm{ch}\text{-}\mathrm{enc}}\left(\mathbf{z};{\mathsf{\theta }}_{\mathrm{ch}}\right)$.

✓

Physical Channel: $\mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{n}$.

✓

Channel Decoder: $\widehat{\mathbf{z}}=f_{\mathrm{ch}\text{-}\mathrm{dec}}\left(\mathbf{y};{\mathsf{\theta }}_{\mathrm{ch}\text{-}\mathrm{dec}}\right)$.

✓

Semantic Decoder: ${\widehat{\mathbf{s}}}_{\mathrm{task}}=f_{\mathrm{sem}\text{-}\mathrm{dec}}\left(\widehat{\mathbf{z}};{\mathsf{\theta }}_{\mathrm{sem}\text{-}\mathrm{dec}}\right)$.

• Task-Oriented Loss Function: Instead of minimizing pixel-level or bit-level reconstruction error, the system is trained end-to-end with a task-specific loss

  $${\mathcal{L}}_{\mathrm{task}}=\mathbb{E}\left[d_{\mathrm{task}}\left(\mathbf{s},\hspace{0.33em}{f}_{\mathrm{sem}\text{-}\mathrm{dec}}\circ f_{\mathrm{ch}\text{-}\mathrm{dec}}\circ \mathrm{Channel}\circ f_{\mathrm{ch}\text{-}\mathrm{enc}}\circ f_{\mathrm{sem}}\left(\mathbf{s}\right)\right)\right]$$

  (116)

  where $d_{\mathrm{task}}$ may be cross-entropy for classification, mean absolute error for regression, or a task-specific metric for control applications. This formulation jointly optimizes the source and channel coding, enabling the system to allocate communication resources (bandwidth, power) to the most task-relevant information components.
• Training Methodology: End-to-end training requires a differentiable channel model. For channels that are not analytically differentiable (e.g., discrete-input channels, channels with unknown statistics), techniques such as the straight-through estimator, REINFORCE gradient estimation, or learned channel models (trained from measured data) are employed. Training typically uses the Adam optimizer with a dataset of ${10}^4$–${10}^6$ source samples paired with diverse channel conditions.

8.3. Goal-Oriented Communication and Shared Knowledge

8.3.1. Goal-Oriented Communication

Rather than reconstructing data at the receiver, goal-oriented systems optimize task objectives directly through the communication link.

• RL Formulation: An agent learns a communication policy that maximizes the task reward

  $${\pi }^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\pi }{\mathrm{m}\mathrm{a}\mathrm{x}}\hspace{0.33em}{\mathbb{E}}_{\pi }\left[\sum _{t=0}^T{\gamma }^t\hspace{0.17em}{r}_{\mathrm{task}}\left({\mathbf{s}}_t,{\mathbf{a}}_t\right)\right]$$

  (117)

  where ${\mathbf{s}}_t$ includes the observed state, ${\mathbf{a}}_t$ encompasses both communication decisions (what to transmit) and control actions (how to act based on received information), and $r_{\mathrm{task}}$ measures task performance (e.g., navigation accuracy for autonomous systems, tracking error for remote control). The system jointly learns what information to transmit and how to act on received information, maximizing task reward under communication constraints (bandwidth, power, latency).

8.3.2. Shared Knowledge Base

When transmitter and receiver share prior knowledge—such as a pre-trained generative model $p_{\mathsf{\theta }}\left(\mathbf{s}|\mathbf{z}\right)$—the communication burden is dramatically reduced:

$$\mathrm{Transmitter} \mathrm{sends}: \widehat{\mathbf{z}}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathbf{z}}{\mathrm{m}\mathrm{i}\mathrm{n}}d\left(\mathbf{s},G_{\mathsf{\theta }}\left(\mathbf{z}\right)\right)$$

(118)

where $G_{\mathsf{\theta }}$ is a shared generative model (e.g., a pre-trained decoder or diffusion model) and $\widehat{\mathbf{z}}$ is a low-dimensional latent code. The receiver reconstructs the source as $\widehat{\mathbf{s}}=G_{\mathsf{\theta }}\left(\widehat{\mathbf{z}}\right)$. This enables massive compression when the shared model accurately captures the source distribution. For example, transmitting a 128-dimensional latent code instead of a $224\times 224\times 3$ image achieves a compression ratio exceeding $1000\times$.

• Trade-offs: Shared knowledge-based approaches require (i) synchronized model versions at transmitter and receiver, (ii) sufficient model capacity to represent the source distribution, and (iii) mechanisms for model updates over the air. Generalization degrades when the source distribution shifts away from the training distribution of the shared model.

8.4. Applications and Experimental Evaluation

Image Transmission for Classification: Semantic communication for image classification transmits only task-relevant features rather than full pixel data. The performance comparison is presented in Table 5.

Table 5. Bandwidth–accuracy trade-off in semantic image transmission.

Method	Bandwidth (Bits/Image)	Top-1 Accuracy (%)	Compression Ratio vs. JPEG+LDPC	SNR Regime	Key Ref.
JPEG + LDPC (Rate 1/2)	30,000	76.1	1× (baseline)	≥10 dB	[52]
JPEG + LDPC (Rate 3/4)	20,000	75.4	1.5×	≥15 dB	[52]
DeepJSCC (SNR 10 dB)	5000	74.3	6×	10 dB	[52]
DeepJSCC (SNR 20 dB)	5000	76.8	6×	20 dB	[52]
Task-Oriented Semantic Encoder	3000	75.2	10×	≥5 dB	[50]
Task-Oriented (with IB optimization)	2500	74.8	12×	≥5 dB	[13]
JPEG + LDPC (Low SNR, 5 dB)	30,000	52.3 (cliff effect)	1×	5 dB	[52]
DeepJSCC (Low SNR, 5 dB)	5000	68.7 (graceful)	6×	5 dB	[52]

Table 5 compares conventional JPEG+LDPC transmission with semantic communication approaches (DeepJSCC, task-oriented encoders) in terms of bandwidth usage (bits per image), classification accuracy (top-1 on ImageNet), and compression ratio. Semantic communication achieves comparable or superior classification accuracy with 5–10$\times$ bandwidth reduction [52].

• Video Transmission for Object Detection: Rather than transmitting raw video frames (requiring H.265/HEVC encoding), semantic approaches transmit intermediate feature maps from object detection networks, achieving up to $100\times$ compression relative to H.265 while maintaining detection performance (mAP within 2% of uncompressed baseline).
• Text Communication for Translation: Semantic embeddings are transmitted instead of character-level or word-level representations, providing robustness to channel errors and preserving meaning rather than exact syntax. Under severe channel conditions (SNR $<5$ dB), semantic text transmission maintains BLEU scores 20–30% higher than classical coded transmission.
• Drone Remote Control: The performance of goal-oriented communication in autonomous drone navigation is summarized in Table 6.

Table 6. Rate–performance trade-off in goal-oriented communication for remote control systems.

Method	Comm. Rate (Bits/Step)	Navigation Accuracy (%)	Collision Avoidance Rate (%)	Bandwidth Reduction	Control Latency	Key Ref.
Full State Vector (uncompressed)	256	97.2	99.1	1× (baseline)	Low	[53]
Quantized State Vector (8-bit)	64	95.8	98.5	4×	Low	[53]
Random Subsampling (25%)	64	88.3	91.7	4×	Low	[53]
Goal-Oriented (DRL-trained encoder)	32	96.1	98.7	8×	Low	[53]
Goal-Oriented (IB-optimized, $\beta =0.1$)	16	94.5	97.9	16×	Low	[13,53]
Goal-Oriented (aggressive, $\beta =0.01$)	10	91.8	96.2	25×	Low	[53]
Full State + Noisy Channel (SNR 5 dB)	256	82.4	87.3	1×	Low	[53]
Goal-Oriented + Noisy Channel (SNR 5 dB)	32	93.7	97.1	8×	Low	[53]

Table 6 compares conventional data transmission (full state vector) with goal-oriented communication in terms of communication rate (bits per control step), navigation accuracy, and collision avoidance performance. Goal-oriented communication achieves near-optimal control performance with up to 25$\times$ lower bandwidth requirements, as the system learns to transmit only the information necessary for effective control decisions [53].

8.5. Challenges and Open Issues

• Compatibility and Interoperability. Semantic communication requires aligned encoder–decoder models at the transmitter and receiver, raising challenges for standardization and interoperability across heterogeneous devices and vendors. Model versioning, over-the-air updates, and fallback to conventional transmission are essential for practical deployment.
• Generalization under Distribution Shifts. Semantic encoders trained on specific data distributions may perform poorly when the source statistics change (e.g., new scene types, languages, or sensor modalities). Domain adaptation, continual learning, and robust training techniques are active research areas [54].
• Security and Privacy. Semantic features may inadvertently encode sensitive information (e.g., identity, location) that is not relevant to the communication task, raising privacy concerns. Differential privacy mechanisms, adversarial training for privacy preservation, and secure semantic encoding are important considerations [54].
• Latency and Complexity. Deep semantic encoders and decoders introduce processing latency that may conflict with real-time requirements, particularly for control-oriented applications. Model compression, early-exit architectures, and hardware-accelerated inference are necessary for practical deployment [55].
• Theoretical Foundations. Despite significant empirical progress, the information-theoretic foundations of semantic communication remain an active area of investigation. The characterization of fundamental rate–distortion–task performance trade-offs, the development of semantic capacity measures, and the establishment of achievability and converse results for task-oriented communication represent open theoretical challenges [13].

9. Implementation Challenges and Practical Considerations

While deep learning (DL) offers substantial improvements in physical-layer design, practical deployment introduces significant challenges spanning computational complexity, hardware constraints, generalization, robustness, standardization, privacy, interpretability, and environmental sustainability [7]. This section consolidates these challenges and discusses the principal trade-offs that must be navigated to bridge the gap between research prototypes and production-grade AI-native wireless systems.

9.1. Computational Complexity, Latency, and Hardware Deployment

Deep neural networks (DNNs) require millions of floating-point operations (FLOPs) per inference, potentially exceeding the latency and energy budgets of real-time wireless systems [56].

For a fully connected DNN with $L$ layers of width $n_l$, the inference complexity is

$${\mathcal{C}}_{\mathrm{F}\mathrm{C}}=\sum _{l=1}^L{n}_{l-1}\cdot n_l$$

(119)

whereas a convolutional layer with $C_{\mathrm{i}\mathrm{n}}$ input channels, $C_{\mathrm{o}\mathrm{u}\mathrm{t}}$ output channels, kernel size $K$, and spatial resolution $S$ requires

$${\mathcal{C}}_{\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}}=C_{\mathrm{i}\mathrm{n}}\cdot C_{\mathrm{o}\mathrm{u}\mathrm{t}}\cdot K^2\cdot S^2$$

(120)

As a concrete example, a five-layer DNN with 512 neurons per layer requires approximately $5\times {512}^2\approx 1.3\times {10}^6$ FLOPs per inference, corresponding to roughly 130 μs on a 10 GFLOPS processor. While this latency is acceptable for 1 ms transmission time interval (TTI) systems, it becomes prohibitive for ultra-reliable low-latency communication (URLLC) scenarios demanding sub-100 μs processing [1].

The trade-off between model expressiveness and inference latency is therefore central to deployment. Larger, deeper networks improve estimation or detection accuracy but increase latency and energy consumption. Practical remedies to manage this trade-off include:

• Quantization: INT8/INT4 representations reduce memory footprint by $4\times$ and latency by $2$–$4\times$ with less than 1 dB performance degradation [57].
• Structured pruning: Removal of redundant parameters (up to $10\times$ reduction) with negligible accuracy loss.
• Knowledge distillation: Transferring the learned behaviour of a large teacher network into a compact student model.
• Neural Architecture Search (NAS): Automatically identifying hardware-efficient architectures, yielding $5$–$10\times$ speedups [58].
• Early-exit networks: Providing adaptive computation depth, reducing average latency for high-confidence inputs.

From a hardware perspective, different acceleration platforms present distinct trade-offs between throughput, latency, power consumption, and reconfigurability. GPUs deliver massive parallelism (throughput exceeding 1000 MIMO detections per second) but consume 100–300 W, rendering them impractical for edge deployment. Tensor Processing Units (TPUs) provide optimized systolic architectures with throughput exceeding 100 TOPS and superior energy efficiency relative to GPUs. Field-Programmable Gate Arrays (FPGAs) offer ultra-low latency (<100 μs), moderate power consumption (10–50 W), and post-deployment reconfigurability, making them suitable for MIMO detection and neural decoding in base stations. Application-Specific Integrated Circuits (ASICs) achieve maximum energy efficiency ($>1000$ GOPS/W) and minimum latency, but at the cost of high development expenditure and lack of post-fabrication flexibility. Table 7 provides a comparative analysis of these platforms for DL-based MIMO detection.

Table 7. Performance comparison of hardware acceleration platforms for deep learning-based MIMO detection.

Platform	Throughput	Latency	Power	Energy Efficiency	Reconfigurability
GPU (NVIDIA A100)	$>$1000 det/s	1–5 ms	100–300 W	Moderate	Software-level
TPU (Google Edge)	$>$100 TOPS	$<$1 ms	15–40 W	High	Limited
FPGA (Xilinx)	Moderate	<100 μs	10–50 W	High	Full (RTL)
ASIC (Custom)	Very High	<50 μs	1–10 W	$>$1000 GOPS/W	None

In summary, the selection of hardware platform and model compression strategy must be co-designed with the target deployment scenario, explicitly accounting for the latency budget, energy envelope, and required throughput of the wireless application.

9.2. Generalization, Robustness, and Adversarial Considerations

A DL model trained on a specific data distribution may fail under deployment conditions that differ from the training environment—a problem known as domain shift. Sources of distribution mismatch include discrepancies between simulated and measured channels, variations in signal-to-noise ratio (SNR), hardware impairments (phase noise, IQ imbalance), and changes in propagation scenario (indoor, outdoor, vehicular) [41].

Several complementary strategies address this challenge. Data augmentation enriches the training distribution through channel randomization, SNR variation, and hardware imperfection simulation. Domain adaptation retrains upper layers on target-domain data or employs adversarial feature alignment:

$${\mathcal{L}}_{\mathrm{D}\mathrm{A}}={\mathcal{L}}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}-\lambda \hspace{0.17em}{\mathcal{L}}_{\mathrm{d}\mathrm{o}\mathrm{m}\mathrm{a}\mathrm{i}\mathrm{n}}$$

(121)

where the feature extractor learns domain-invariant representations by deceiving a domain discriminator. Meta-learning approaches such as Model-Agnostic Meta-Learning (MAML) enable rapid adaptation to new channel conditions from only a few examples:

$${\mathsf{\theta }}^{\ast }=\mathsf{\theta }-\alpha {\nabla }_{\mathsf{\theta }}{\mathcal{L}}_{{\mathcal{T}}_i}\left(\mathsf{\theta }-\beta {\nabla }_{\mathsf{\theta }}{\mathcal{L}}_{{\mathcal{T}}_i}\left(\mathsf{\theta }\right)\right)$$

(122)

Ensemble methods combine diverse models to improve robustness:

$${\widehat{y}}_{\mathrm{e}\mathrm{n}\mathrm{s}}={\displaystyle \frac{1}{M}}\sum _{m=1}^M{f}_{{\mathsf{\theta }}_m}\left(\mathbf{x}\right)$$

(123)

and uncertainty quantification techniques—including Monte Carlo dropout and Bayesian neural networks—provide calibrated confidence estimates that are essential for safety-critical decisions in wireless systems.

Beyond natural distribution shifts, DNNs are vulnerable to adversarial perturbations. An adversary can craft a perturbation $\mathsf{\delta }$ satisfying $\parallel \mathsf{\delta }\parallel \le ϵ$ that maximizes the detector or decoder error:

$${\mathsf{\delta }}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\parallel \mathsf{\delta }\parallel \le ϵ}{\mathrm{m}\mathrm{a}\mathrm{x}}\mathcal{L}\left(f_{\mathsf{\theta }}\left(\mathbf{x}+\mathsf{\delta }\right),\hspace{0.17em}y\right)$$

(124)

In wireless systems, such attacks manifest as adversarial jamming—optimized interference that is far more damaging than Gaussian noise of equal power—as well as evasion attacks and backdoor poisoning of training datasets. Defences include adversarial training, which augments the empirical risk minimization objective:

$$\underset{\mathsf{\theta }}{\min }E!\left[\underset{\left|\mathsf{\delta }\right|\le \mathsf{ϵ}}{\max }\mathcal{L}\left(f_{\mathsf{\theta }}\left(x+\mathsf{\delta }\right),\hspace{0.17em}y\right)\right]$$

(125)

Randomized smoothing, which provides certified robustness guarantees within an ${\mathcal{l}}_2$-ball of radius $r$, and anomaly detection modules that flag adversarial inputs prior to processing. A key trade-off exists between adversarial robustness and nominal accuracy: hardened models typically incur a 1–3 dB degradation on clean data, necessitating application-specific calibration of the robustness–performance operating point.

9.3. Standardization and Interoperability

Deploying DL in commercial cellular networks requires standardized frameworks that ensure interoperability across equipment vendors and network operators. Standardization must address model architectures (layer types, dimensions, activation functions), weight representation formats (FP32, FP16, INT8), inference interfaces (input/output specifications, capability signalling), and over-the-air model update procedures, including versioning and fallback to traditional algorithms.

Several standardization initiatives are actively progressing. The Third Generation Partnership Project (3GPP) has introduced Study Items in Release 18/19 exploring AI/ML for beam management, CSI compression, and positioning, with particular attention to split-architecture learning where computation is partitioned between the user equipment (UE) and the base station (gNB). The Open Neural Network Exchange (ONNX) provides a vendor-neutral model interchange format enabling cross-framework deployment. The Open Radio Access Network (O-RAN) architecture defines the RAN Intelligent Controller (RIC)—both near-real-time and non-real-time variants—as a standardized platform for hosting ML-based xApps and rApps, establishing key performance indicator (KPI) definitions, training data formats, and an AI application marketplace. These efforts are critical for transitioning AI-native designs from laboratory demonstrations to large-scale commercial deployment, and alignment with ITU-R IMT-2030 requirements will be essential for global harmonization.

9.4. Data Privacy, Security, and Trustworthiness

Training DL models requires large volumes of data that may contain sensitive user information, raising privacy, security, and regulatory compliance concerns [54].

Privacy risks include membership inference attacks, where an adversary determines whether a specific sample was used for training; model inversion attacks, where training data are partially reconstructed from model outputs; and data poisoning, where adversarial samples are injected into the training set.

Differential privacy (DP) provides a principled defence by adding calibrated noise to stochastic gradient descent updates:

$${\tilde{\mathbf{g}}}_t={\displaystyle \frac{1}{B}}\sum _{i=1}^B\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}\left({\nabla }_{\mathsf{\theta }}{\mathcal{L}}_i,\hspace{0.17em}C\right)+\mathcal{N}\left(0,\hspace{0.17em}{\sigma }^2{C}^2\mathbf{I}\right)$$

(126)

where $C$ is the gradient clipping threshold and $\sigma$ controls the noise scale. The privacy budget $\epsilon$ (typically $1\le \epsilon \le 10$) quantifies the privacy–utility trade-off, with lower $\epsilon$ yielding stronger guarantees at the cost of 2–5% accuracy degradation.

Federated learning (FL) enables distributed model training without sharing raw data. User equipment trains locally and transmit only model updates to the base station, which aggregates them via

$${\mathsf{\theta }}_{t+1}={\mathsf{\theta }}_t+{\displaystyle \frac{1}{K}}\sum _{k=1}^K\Delta {\mathsf{\theta }}_t^{\left(k\right)}$$

(127)

Challenges in federated settings include non-independent and identically distributed (non-IID) data across clients, communication overhead, and resource-constrained devices. Secure multi-party computation and trusted execution environments (e.g., Intel SGX, ARM TrustZone) provide complementary hardware-level protections, and homomorphic encryption allows inference on encrypted data at the cost of substantially increased latency. Together, these mechanisms form a layered trustworthiness framework essential for regulatory compliance in commercial AI-native wireless systems.

9.5. Explainability and Interpretability

DL models are widely characterized as “black boxes,” which hinders trust, regulatory compliance, and effective debugging in operational wireless networks [7]. As AI-native designs assume greater decision-making responsibility—particularly in safety-critical applications such as autonomous vehicular communication—explainability becomes a prerequisite rather than an option.

Post hoc explanation techniques applicable to wireless DL models include:

• SHAP (Shapley Additive Explanations): Assigns a Shapley value ${\varphi }_i$ to each input feature, quantifying its marginal contribution:

  $${\varphi }_i=\sum _{S\subseteq \mathcal{F}\left\{i\right\}}{\displaystyle \frac{\left|S\right|!\hspace{0.17em}\left(\left|\mathcal{F}\right|-\left|S\right|-1\right)!}{\left|\mathcal{F}\right|!}}\left[f\left(S\cup i\right)-f\left(S\right)\right]$$

  (128)
• LIME (Local Interpretable Model-agnostic Explanations): Fits a locally faithful interpretable surrogate:

  $$\xi =\mathrm{a}\mathrm{r}\mathrm{g}\underset{g\in \mathcal{G}}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.33em}\mathcal{L}\left(f,g,{\pi }_{\mathbf{x}}\right)+\Omega \left(g\right)$$

  (129)
• Attention visualization: For transformer-based architectures, the learned attention weights ${\alpha }_{ij}$ reveal which input positions (subcarriers, antenna elements, time slots) are most influential for each output decision.
• Activation maximization and model distillation: Synthesizing maximally activating inputs reveals the patterns recognized by individual neurons, while distilling a DNN into a compact decision tree provides directly interpretable if-then rules at the expense of fidelity.

The inherent trade-off between model complexity and interpretability must be explicitly managed: simpler, more interpretable architectures may sacrifice 1–2 dB of performance relative to deep opaque models. Hybrid approaches—where a high-performance DL model is accompanied by an interpretable surrogate for auditing purposes—represent a pragmatic compromise for deployment.

9.6. Energy Efficiency and Environmental Sustainability

Training large DL models incurs substantial energy consumption and carbon emissions, raising sustainability concerns for AI-native wireless networks that require continuous training, fine-tuning, and distributed inference across large-scale infrastructures [59].

Mitigation strategies span multiple levels:

• Efficient training: Mixed-precision arithmetic (FP16/BF16), gradient checkpointing, and communication-efficient optimizers (AdamW, LAMB) reduce training FLOPs and memory requirements.
• Transfer learning: Pre-training a foundation model once on diverse data and fine-tuning for specific scenarios amortizes the computational cost across applications, reducing redundant training by orders of magnitude.
• Efficient architecture search: Methods such as ENAS and DARTS reduce NAS cost by $1000\times$ relative to brute-force search [58].

• Model sharing: Establishing repositories of pre-trained base models for wireless tasks avoids duplication of training effort across operators and research groups [60].
• Green AI metrics: Reporting energy efficiency alongside accuracy encourages the development of models that are Pareto-optimal in the performance–energy plane [59]

  $$\mathrm{Efficiency}={\displaystyle \frac{\mathrm{Performance}}{\mathrm{Energy} \mathrm{Consumed}}}$$

  (130)
• Edge computing: Shifting inference to edge devices reduces data transmission to centralized clouds, thereby decreasing communication energy and network congestion [61].

Collectively, these measures promote the design of AI-native systems that are not only high-performing but also environmentally responsible—an increasingly important requirement as sustainability targets become integrated into 6G standardization roadmaps.

10. Future Directions

The preceding sections have established that DL-based approaches consistently advance physical-layer performance across channel estimation, signal detection, beamforming, resource allocation, and semantic communication. This section identifies and prioritizes the most impactful research directions for the next generation of AI-native wireless systems, organized into five thematic areas.

10.1. Foundation Models and Self-Supervised Learning for the Physical Layer

Inspired by the success of large pre-trained models in natural language processing and computer vision, foundation models for wireless communications aim to learn general-purpose representations from diverse, large-scale datasets spanning multiple channel conditions, frequency bands, and network configurations [60]. A universal transformer backbone can process heterogeneous inputs—including pilot observations, CSI tensors, and network metadata—through a unified architecture:

$$\mathbf{z}=\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\left(\left[{\mathbf{x}}_{\mathrm{p}\mathrm{i}\mathrm{l}\mathrm{o}\mathrm{t}};\hspace{0.17em}{\mathbf{x}}_{\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{a}};\hspace{0.17em}{\mathbf{x}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}\mathrm{i}\mathrm{g}}\right]\right)$$

(131)

where the context vector $\mathbf{z}$ is subsequently fine-tuned for specific downstream tasks (detection, estimation, beamforming) with minimal labelled data.

Self-supervised learning reduces reliance on expensive labelled datasets by leveraging unlabelled over-the-air signals. Contrastive learning objectives encourage representations that distinguish semantically different signal conditions:

$${\mathcal{L}}_{\mathrm{C}\mathrm{L}}=-\mathrm{l}\mathrm{o}\mathrm{g}{\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathbf{z}}_i,{\mathbf{z}}_j\right)/\tau \right)}{\sum _{k=1}^{2N}\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({\mathbf{z}}_i,{\mathbf{z}}_k\right)/\tau \right)}}$$

(132)

where ${\mathbf{z}}_j$ is an augmented view of ${\mathbf{z}}_i$ and $\tau$ is a temperature parameter. Complementarily, masked autoencoders that reconstruct randomly masked subcarriers or antenna elements learn spatial–frequency representations transferable to downstream estimation tasks. These approaches promise to dramatically reduce per-task training costs while improving generalization across heterogeneous deployment scenarios.

10.2. AI-Native Air Interface Design and Standardization

The transition from block-optimized to end-to-end learned air interfaces constitutes a defining research frontier for 6G. In this paradigm, the encoder $f_{{\mathsf{\theta }}_{\mathrm{e}\mathrm{n}\mathrm{c}}}$ and decoder $f_{{\mathsf{\theta }}_{\mathrm{d}\mathrm{e}\mathrm{c}}}$ are jointly optimized over the physical channel, replacing conventional modulation, coding, and equalization blocks with a single differentiable pipeline [8,17]. Achieving this vision at scale requires addressing three interconnected challenges:

• Differentiable channel models that faithfully represent real propagation while permitting gradient backpropagation through the channel.
• Standardized neural codec interfaces within 3GPP Release 18/19 and ITU-R IMT-2030 frameworks, defining input/output formats, model update protocols, and fallback mechanisms to ensure interoperability.
• Split-architecture learning that partitions computation between the UE and gNB, balancing inference latency, signalling overhead, and model synchronization.

Realizing AI-native air interfaces will demand coordinated advances in network architecture standardization, over-the-air model distribution, and robust performance guarantees under diverse operating conditions.

10.3. Semantic and Goal-Oriented Communication Systems

Moving beyond bit-level fidelity, semantic communication transmits task-relevant meaning rather than raw symbols, achieving compression gains of $10\times$–$100\times$ in bandwidth-constrained scenarios [13,50]. Goal-oriented extensions further optimize transmission for downstream task performance (e.g., classification accuracy, control stability) rather than reconstruction quality.

Key open problems include the design of shared semantic knowledge bases that enable transmitter and receiver to agree on meaning representations, the development of universal semantic metrics that generalize across modalities (text, image, speech, control signals), and the establishment of information-theoretic foundations for semantic rate–distortion–perception trade-offs. Recent work on end-to-end learning for task-oriented transmission over MIMO channels [13] demonstrates that joint source–channel–task optimization can substantially outperform separated designs, particularly under severe bandwidth constraints. Scaling these approaches to multi-user, multi-modal, and heterogeneous-network settings represents a high-priority research direction.

10.4. Integration with Emerging Hardware: Neuromorphic and Quantum-Assisted Processing

Neuromorphic computing employs spiking neural networks (SNNs) that process information through discrete temporal events rather than continuous activations:

$$V_i\left(t+1\right)=\beta \hspace{0.17em}{V}_i\left(t\right)+\sum _j{w}_{ij}\hspace{0.17em}{s}_j\left(t\right), s_i\left(t\right)=\mathbf{1}\left[V_i\left(t\right)>V_{\mathrm{t}\mathrm{h}}\right]$$

(133)

where $V_i$ is the membrane potential, $\beta$ is a decay factor, and $s_i\left(t\right)$ is the binary spike output. Neuromorphic hardware offers extreme energy efficiency (event-driven computation), ultra-low latency, and natural temporal processing, making it particularly attractive for always-on signal detection and edge inference in resource-constrained devices.

Quantum-assisted optimization leverages parameterized quantum circuits $U\left(\mathsf{\varphi }\right)$ for combinatorial problems that arise in resource allocation and channel coding:

$$\left|\psi \left(\mathsf{\varphi }\right)⟩=U\left(\mathsf{\varphi }\right)\hspace{0.17em}\right|0{⟩}^{\otimes n}$$

(134)

While current noisy intermediate-scale quantum (NISQ) devices remain limited in qubit count and coherence time, hybrid quantum–classical algorithms show promise for specific sub-problems in wireless optimization. Both neuromorphic and quantum hardware are at relatively early stages of maturity; hence, near-term research should focus on identifying the PHY sub-problems where these architectures provide demonstrable advantages over classical DL accelerators.

10.5. Trustworthy, Explainable, and Sustainable AI for Wireless Systems

As AI assumes an increasingly central role in wireless network operation, trustworthiness emerges as a cross-cutting requirement encompassing robustness, fairness, privacy, explainability, and sustainability. Priority research directions include:

• Theoretical performance limits: Establishing fundamental bounds on the capacity of learned communication systems—particularly under channel uncertainty—and characterizing the sample complexity required to approach these bounds.
• Causal reasoning: Moving beyond correlational pattern recognition to structural causal models that support counterfactual analysis (e.g., “what would throughput be if beam selection were changed?”), enabling more reliable network optimization.
• Continual and lifelong learning: Developing algorithms that adapt to evolving environments without catastrophic forgetting, using techniques such as elastic weight consolidation and progressive network expansion.
• Pareto-optimal multi-objective design: Jointly optimizing throughput, latency, energy efficiency, and model complexity through multi-task learning with dynamically weighted objectives

  $${\mathcal{L}}_{\mathrm{M}\mathrm{O}}=\sum _{k=1}^K{\lambda }_k\hspace{0.17em}{\mathcal{L}}_k$$

  (135)

  where the weights ${\lambda }_k$ are learned to balance conflicting criteria.
• Global federated model ecosystems: Enabling cross-operator, cross-border collaborative training through federated learning with heterogeneity-aware aggregation, coupled with standardized model repositories (“model zoos”) that accelerate reproducible research and benchmarking.

Addressing these challenges in an integrated manner will be essential for building AI-native 6G systems that are not only performant but also safe, transparent, and aligned with societal and environmental objectives.

11. Conclusions

This article has presented a comprehensive survey of deep learning architectures and optimization paradigms for the transition from model-driven to AI-native physical-layer design in next-generation wireless communication systems. The review has systematically examined DL-based approaches across all critical PHY subsystems—channel estimation, signal detection, beamforming and beam management, spectral resource allocation, adaptive modulation and waveform design, and semantic and task-oriented communication—within a unified optimization framework grounded in empirical risk minimization, representation learning, and information-theoretic principles.

The key findings of this survey are summarized as follows:

• Performance gains: DL-based techniques consistently outperform traditional model-driven methods. Improvements of 3–5 dB in normalized mean square error (NMSE) have been demonstrated for channel estimation relative to MMSE baselines. Learning-based MIMO detectors achieve 90–98% of maximum likelihood performance at fixed computational complexity. Beamforming optimization via neural networks reduces latency by up to two orders of magnitude compared with iterative algorithms, and multi-agent reinforcement learning enables resource allocation strategies that surpass conventional centralized schemes.
• Implementation feasibility: Model compression techniques—including INT8/INT4 quantization, structured pruning, and knowledge distillation—reduce computational requirements by up to an order of magnitude with less than 1 dB performance degradation. Specialized hardware platforms (FPGAs and ASICs) enable sub-millisecond inference latency with high energy efficiency, validating the real-time deployment viability of AI-native designs.
• Adaptability and generalization: Transfer learning, meta-learning, and domain adaptation mechanisms enable rapid adaptation to new propagation environments with limited target-domain data, substantially reducing retraining costs. These capabilities are essential for maintaining performance across the heterogeneous and non-stationary conditions encountered in practical deployments.

Semantic communication: Task-oriented and semantic transmission paradigms achieve compression gains of $10\times$–$100\times$ by transmitting meaning rather than raw bits, representing a fundamental departure from classical source–channel separation toward end-to-end optimization of communication objectives.

Implementation challenges: Critical barriers to commercial deployment include standardization of AI/ML interfaces within 3GPP and O-RAN frameworks, robustness against adversarial threats, interpretability requirements for safety-critical applications, and the environmental footprint of large-scale model training and distributed inference.

Looking forward, we have identified five prioritized research directions: (i) foundation models and self-supervised learning for scalable, general-purpose PHY intelligence; (ii) AI-native air interface design integrated with evolving standardization efforts; (iii) semantic and goal-oriented communication systems with rigorous information-theoretic foundations; (iv) integration with emerging neuromorphic and quantum-assisted hardware; and (v) trustworthy, explainable, and sustainable AI frameworks for wireless systems.

Ultimately, the integration of AI and DL into the physical layer represents a fundamental evolution in wireless communication system design. AI-native 6G systems will not only advance traditional performance metrics—throughput, latency, and spectral efficiency—but will also enable qualitatively new capabilities, including real-time environmental adaptation, semantic-aware transmission, autonomous coordination in ultra-dense networks, and personalized communication services. Realizing this vision requires coordinated progress across theoretical foundations, representative datasets, standardized architectures and interfaces, efficient hardware, and appropriate regulatory frameworks. The technical foundations reviewed in this survey are firmly established; the central challenge ahead lies in translating these advances from controlled research environments into robust, large-scale, real-world deployment.