Machine Learning-Enabled 5G and 6G Networks: Methods, Challenges, and Opportunities

Abstract

Fifth-generation (5G) and sixth-generation (6G) wireless communications aim to achieve significantly higher data speeds, remarkably low latency, and substantial improvements in the efficiency of base stations. With the rapid increase in the utilization of broadband data driven by Internet of Things (IoT) gadgets, smart home systems, autonomous vehicles, and virtual reality devices, 5G and 6G networks are set to overcome the limitations of earlier telecommunication technologies and serve as key enablers for future IoT applications. Anticipated as the primary infrastructure for delivering emerging services, 5G cellular networks introduce new requirements and challenges that complicate the achievement of desired objectives. This paper provides a comprehensive overview of machine learning (ML) methods and their application in 5G and 6G wireless networks, covering supervised, unsupervised, and reinforcement learning (RL) approaches. ML is set to play a central and important role in 6G systems for these wireless networks. Subsequently, this paper thoroughly explores a series of challenges within the domain of 5G and 6G networks and examines research opportunities for applying ML techniques to address these challenges.

1. Introduction

The advent of cutting-edge mobile technology, such as cloud gaming, virtual reality apps, and video streaming, has led to an exponential increase in mobile network traffic [1]. Several challenges must be addressed in fifth-generation (5G) and sixth-generation (6G) wireless networks to meet the growing demand for ultra-low latency and high data rates [2]. These days, a lot of cutting-edge technology, including Internet of Things (IoT), autonomous vehicles, and unmanned aerial systems, rely on 5G and 6G networks [3]. To meet the increasing demands of diverse networks and users, 5G and 6G wireless networks need to demonstrate a high degree of flexibility in their design, resource management, and allocation [4]. The architecture of 5G and 6G networks is designed to support ultra-reliable, high-speed communication across diverse applications, from autonomous vehicles and smart cities to industrial IoT [5]. At a high level, these networks consist of multiple components that work together to provide connectivity, process data, and manage resources efficiently [6]. The Radio Access Network (RAN) manages wireless communication between user devices and base stations, allowing users to connect to the network [7]. Massive Multi-Input Multi-Output (Massive-MIMO) technique in 5G improves RAN by using numerous antennas to increase network capacity and coverage. In 6G, RAN will grow into an intelligent network enhanced by machine learning (ML). Figure 1 shows the evolution of mobile networks from 1G to 6G, emphasizing a shift from basic connectivity toward intelligent networking. Earlier generations focused on voice and broadband capacity; the figure highlights that 5G and especially 6G introduce increasing system complexity, service diversity, and performance demands. The explicit association of ML with 5G and 6G indicates that learning-based intelligence can become a core enabler rather than an auxiliary feature.

ML is the revolutionary technology set to deliver 5G and 6G with the required flexibility and intelligence. Large data quantities that may be beyond the systematic human capacity to learn, optimize, and evaluate are made possible by machine learning. It provides the ability to not just proactively prevent issues but also accurately predict their occurrence in real-time [9]. Machine learning is highly effective for addressing complex problems that demand significant manual adjustments in existing solutions, or for situations where traditional approaches provide no solution. These problems can be solved by using ML techniques that automatically learn from historical data, which can replace traditional software with long rule lists [10]. Mobile and wireless networks involve numerous parameters, many of which are determined using heuristic methods. This is often due to the lack of closed-form solutions or the high cost of extensive measurement campaigns. In scenarios like these, a machine learning algorithm can be valuable in forecasting the parameters and deriving estimations for functions using the data at hand [11]. Optimization approaches will also be needed for next-generation wireless communication technologies in order to optimize certain goal functions and either minimize or increase them. Many issues in mobile and wireless communications must be estimated since they are either polynomial or non-linear in nature [12,13]. Machine learning techniques are appropriate for expressing the objective functions of non-linear problems requiring estimation or optimization [14,15]. This paper presents the core concepts of 5G and 6G wireless networks and their applications. It also reviews machine learning techniques for 5G and 6G, including supervised learning, unsupervised learning, and reinforcement learning (RL). Figure 2 shows the hierarchical relationship of Artificial Intelligence (AI) technologies applied to 5G and 6G wireless networks, showing that practical intelligence is primarily achieved through machine learning and deep learning, which enable data-driven optimization and adaptive network control.

Motivation and Contributions

The integration of ML with 5G and 6G wireless networks is crucial for several reasons. ML can significantly enhance the efficiency and performance of these networks by optimizing resource allocation, managing interference, and ensuring robust security. Traditional methods fall short in addressing the challenges caused by the exponential growth in connected devices and the increasing complexity of network operations [16]. Therefore, ML techniques offer promising solutions to meet the high demands of future wireless communication. Several surveys have explored the intersection of 5G networks and machine learning. Siriwardhana et al. [1] analyze mobile augmented reality supported by 5G and edge computing, focusing on system architectures and applications. Erunkulu et al. [2] compare various use cases of 5G mobile communication applications but do not delve deeply into machine learning techniques. Arjoune et al. [3] discuss the opportunities and challenges of AI in 5G, with an emphasis on future research directions, but their survey lacks a comprehensive analysis of practical applications. Xu et al. [4] provide a thorough analysis of resource distribution for 5G networks, highlighting current research and future trends. However, their work is primarily focused on resource allocation, without covering the broader spectrum of ML applications in 5G. Morocho-Cayamcela et al. [10] explore the potential, limitations, and future directions of ML for 5G and 6G mobile and wireless communications, providing a good theoretical background but lacking practical implementation insights. Zhang et al. [11] provide a broad survey on deep learning (DL) in wireless networks, but their focus is more on the potential of deep learning rather than specific applications within 5G networks. In our survey provides a holistic view by covering supervised, unsupervised, and reinforcement learning techniques, and their practical applications in both 5G and 6G networks. This paper also addresses the integration challenges and potential solutions, providing a more comprehensive resource for researchers and practitioners. Moreover, unlike previous surveys, our paper dedicates a section to the integration challenges of ML with 5G and 6G networks, such as algorithm complexity, data gathering, resource allocation, reliability, security, and privacy. This comprehensive approach ensures that our survey not only reviews current technologies but also prepares for future developments. The contributions are as follows:

• We analyze the complexity of the systems and examine key technologies for advancing 5G and 6G networks.
• We provide a comprehensive overview of ML techniques.
• We offer a foundational overview of current machine learning solutions applied to tackle challenges within the context of 5G and 6G networks.
• We perform a comprehensive analysis of the applicability of supervised learning, unsupervised learning, and RL techniques in advancing 5G and 6G wireless networks.

The subsequent sections of this paper are arranged as follows: Section 2 presents an overview of 5G and 6G networks, and Section 3 explores ML techniques. Section 4 presents ML applications for 5G and 6G wireless networks. Section 5 presents ML challenges for 5G and 6G networks. Section 6 presents the future research directions and Section 7 concludes our paper.

2. Background of 5G and 6G Wireless Networks

5G and 6G communication systems are expected to support a wide range of applications. This presents significant challenges for mobile operators due to the rapid growth in connected devices and service demands. 5G and 6G are expected to provide support for various applications with a broad spectrum of communication needs. These demands are linked to three general service categories, namely, eMBB, mMTC, and URLLC. The goal is to enhance the performance of 5G and 6G applications by leveraging 5G and 6G technologies [17,18]. The foundational aspects of 5G and 6G wireless networks are well-established. eMBB and URLLC remain critical components, but innovative applications such as holographic communications and digital twins are pushing the boundaries of 5G and 6G [19]. The integration of 5G and 6G with edge computing and AI is creating new opportunities for ultra-low-latency applications, particularly in autonomous driving and industrial automation [2]. These advancements highlight the dynamic and evolving nature of 5G and 6G technology beyond its foundational principles [3]. Figure 3 shows the trade-off among eMBB, mMTC, and URLLC services in 5G and beyond, highlighting the diverse and often conflicting requirements of modern applications. This motivates the need for ML to dynamically optimize throughput, latency, and reliability across heterogeneous services. The key technologies of 5G and 6G are as follows:

2.1. Enhanced Mobile Broadband (eMBB)

eMBB is designed to meet the high data rate transmission requirements in 5G and 6G. This is critical for many human-interaction applications, such as video streaming, and virtual reality, and the objective for user data traffic latency is set at 4 milliseconds [21]. Two key technologies, mmWave and Massive MIMO, are employed to facilitate the higher data rates and enable ultra-dense networks to provide these data rates [22].

2.2. Massive Machine-Type Communications (mMTC)

The rise of massive IoT drives, mMTC, in 5G and 6G can link a significant number of devices. Numerous applications, including resource utility management, asset tracking, traffic monitoring, environmental observations, and the smart grid, among others, require extensive connection. This service has the potential to facilitate the adoption of the IoT by offering effective Internet connectivity [23]. Certain devices are battery-powered and are designed to operate for an extended period, for instance, a couple of years, without requiring battery replacements. Other devices, like smart meters, may be deployed in areas with significant penetration loss [24,25].

2.3. Ultra-Reliable and Low-Latency Communications (URLLC)

URLLC is a critical component of 5G and 6G that imposes strict criteria regarding availability, latency, and reliability [26]. In essence, the communication link must be consistently accessible to swiftly transmit data within a brief timeframe and ensure high reliability. This service is essential to the continued operation of some cutting-edge mission-critical applications, such as vehicle safety communications, and remote surgery. High reliability (i.e., ${10}^{-5}$ packet error probability), high availability (e.g., 99.9999%), and low latency (e.g., <1 ms) are required for URLLC [27,28].

2.4. Massive Multi-Input Multi-Output (Massive MIMO)

Massive MIMO possesses the ability to increase signal strength, improve performance at the cell edges, and enhance overall cell throughput [29]. Massive MIMO enhances the capacity and coverage of 5G using a large number of antennas, but it faces challenges regarding energy consumption and hardware complexity. ML, particularly supervised learning models such as SVM and NNs, are applied to dynamically optimize antenna configurations, reducing interference and improving signal strength. The models are trained on signal-to-noise ratio (SNR), user behavior, and environmental data, and their performance is evaluated using throughput, energy efficiency, and SNR metrics [30]. For 6G, RL will be incorporated into massive MIMO systems, enabling the AI-driven optimization of antenna configurations in real-time based on network conditions and user demand, with energy consumption and performance metrics like throughput and latency as key evaluation criteria. This technology has the capability to enhance both 5G and 6G, leading to improved energy efficiency and increased capacity. It decreases air interface latency and simplifies the transceiver design [31,32].

2.5. Orthogonal Frequency-Division Multiplexing (OFDM)

OFDM is an essential modulation and multiplexing method widely used in wireless networks, especially in cutting-edge 5G and 6G technology. OFDM is widely used in 5G for high-speed data transmission, but struggles at higher frequencies (e.g., mmWave) due to signal attenuation. To address this, ML, particularly supervised learning models such as SVM and DNNs, have been applied to optimize resource allocation and subcarrier assignment. These models are trained on network traffic data and CSI, enabling the dynamic adjustment of subcarriers based on real-time conditions. The performance of these models is evaluated using spectral efficiency, latency, and throughput metrics [33]. For 6G, ML techniques will further evolve to replace traditional OFDM with more efficient schemes like filter bank multi-carrier and universal filtered multi-carrier, optimizing the modulation schemes in real-time to handle high-frequency bands, with performance metrics focusing on spectral efficiency and energy consumption [34,35].

2.6. Beamforming

Within 5G and 6G wireless networks, beamforming is indispensable to improve communication efficiency and reliability. Beamforming in 5G improves network efficiency by focusing radio signals in specific directions, but traditional beamforming is static and cannot dynamically adapt to network changes. To overcome this limitation, RL has been applied to optimize beamforming directions. RL agents are trained on user mobility, signal strength, and network traffic data, adjusting beam directions to improve throughput and reduce interference. The models are evaluated using SNR, throughput, and latency metrics [36]. In 6G, AI-based beamforming will enable real-time adjustments based on user density and network conditions, using DRL to continuously improve beam direction and energy efficiency and evaluating through real-time performance [37,38,39].

2.7. Machine-to-Machine (M2M) Communication

M2M communication plays a central role in 5G and 6G networks, encapsulating the seamless interaction among devices without the need for human intervention. Leveraging the network’s ability to manage massive device connectivity concurrently, M2M communication ensures real-time communication for a variety of applications [40]. In practical situations, this synergy finds application in smart factories, smart cities, and healthcare environments, where devices communicate to enhance industrial processes, optimize urban management, and remotely monitor patients [41]. It meets diverse 5G and 6G requirements by supporting numerous devices at low data rates and achieving extremely low latency [42].

2.8. Device-to-Device (D2D) Communication

D2D communication is a foundational feature in 5G and 6G wireless networks, enabling direct communication between devices located in close proximity [43]. In contrast to conventional network communication, this method enhances efficiency by reducing latency and easing network traffic [6]. Taking advantage of device proximity, D2D communication proves especially beneficial in situations such as public safety, allowing for prompt coordination during emergencies. Furthermore, D2D communication plays a crucial role in content-sharing and collaborative edge computing, enabling direct and energy-efficient data exchanges. In the realm of 5G and 6G, it enhances network efficiency by optimizing resource utilization and fostering direct communication between devices. This approach not only improves overall connectivity but also reduces reliance on centralized network infrastructure [26,44,45].

2.9. Cloud Computing

Cloud computing is essential for the progress of 5G and 6G wireless networks, offering centralized processing, storage, and management capabilities. ML has been applied to optimize resource allocation between cloud computing to predict the optimal distribution of computational tasks.These models are trained on network load data and traffic patterns, and evaluated based on latency, throughput, and resource utilization metrics [45]. In 6G, AI-driven cloud-edge architectures will further optimize task offloading decisions, using RL to make real-time resource allocation decisions, aiming to reduce latency and improve energy efficiency and maintaining high data throughput. Ultimately, this contributes to the fulfillment of 5G and 6G vision for high-speed, low-latency, and flexible wireless communication [46].

2.10. Edge Computing

In 5G and 6G wireless networks, edge computing is essential because it moves processing power closer to consumers and devices, lowering latency and improving application performance overall [47]. Low latency is critical for 5G and 6G such as virtual reality, and real-time IoT services, edge computing enables faster data processing by siting computing resources at the network’s edge. ML, specifically RL, is applied to dynamically allocate resources at the edge based on network conditions. The models are trained using real-time traffic data, such as task requirements and network load, and evaluated using latency, throughput, and energy efficiency metrics. In 6G, AI-driven edge computing will enable further optimization of resource allocation by continuously adjusting resources based on real-time feedback. ML models will be trained on network load, device behavior, and application demands, with latency and data throughput as key performance metrics [48,49].

2.11. Wireless Network Virtualization

Wireless network virtualization enables heightened flexibility, resource efficiency, and the dynamic management of network services [50]. This technique provides virtualized resources for high-level applications by abstracting the actual hardware functionalities. In order to operate ultra-dense networks that provide high data rates for small cell users together with comprehensive coverage, mobility management, and monitoring, wireless network virtualization is an essential procedure [51]. In 5G and 6G, network virtualization is frequently accomplished through technologies such as network function virtualization (NFV), which enables the virtualization of network functions that were traditionally handled by dedicated hardware, transforming them into software-based components deployable on standard servers. In the context of 5G, NFV helps to reduce operational and capital [52,53]. NFV can effectively handle a high number of connections and manage spatiotemporal load changes, which will enable M2M communications and IoT in the 5G and 6G networks [54]. This architecture empowers operators to effortlessly control all network element [55]. This virtualized approach enables efficient resource allocation (RA) and the dynamic scaling of network resources. The software-defined networking (SDN) lowers the 5G network’s complexity by increasing the network’s intelligence [56,57]. Two essential technologies required to realize 5G networks are NFV and SDN.

2.12. Full Duplex Wireless Communication

Full-duplex (FD) technology allows data to be sent and received concurrently across the same frequency range. As a result, it is considered a promising technology in the context of 5G and 6G [58]. The ML model optimizes power control and interference cancellation parameters based on real-time feedback. Performance is evaluated using interference suppression gain, SINR, and bit error rate, demonstrating improved robustness compared with traditional cancellation techniques. In traditional half-duplex communication, a device can perform either data transmission or reception at any given time, but not both simultaneously [59]. FD communication improves spectral efficiency and throughput in wireless networks, thereby enhancing overall performance and capacity. Crucial requirements include high data rates and low latency [60].

2.13. Network Slicing

Network slicing is a foundational concept in 5G and 6G wireless networks, facilitating the virtualized, isolated networks tailored for applications [61]. This innovative approach facilitates the efficient sharing of a common physical infrastructure and providing dedicated and customized slices of the network. Each slice comes with unique characteristics that meet the diverse requirements of various use cases. Each network slice can be optimized for specific attributes, encompassing but not limited to latency, bandwidth, security, and reliability. Achieving programmability, flexibility, modularity, and robust virtualization capabilities primarily relies on technologies such as NFV, SDN, and network softwarization. This enables the division of the physical network into several logical virtual networks, commonly referred to as network slices [9,62]. ML have a vital role in the efficient management and orchestration of network slices, ensuring optimal performance and resource utilization [63].

2.14. Millimeter-Wave

Millimeter-wave (mmWave) technology serves as a fundamental element in the architecture of 5G and 6G wireless communications. Leveraging mmWave, 5G delivers unparalleled data rates, reduced latency, and enhanced capacity. ML is used to optimize beam selection and tracking. AI models are trained using CSI, signal strength, and user mobility data to predict optimal beam directions. Model performance is evaluated using throughput, beam misalignment probability, and handover latency, showing clear gains over static beamforming methods [64,65].

2.15. Terahertz (THz) Communications

For 6G to reach ultra-high-speed data transmission rates of 100 Gbps, terahertz frequencies that range from 100 GHz to 10 THz, will be essential. New uses like as holographic communication and real-time immersive experiences are made possible by these frequencies. However, because high frequencies cause signal deterioration over longer distances, issues like propagation loss and the requirement for sophisticated materials and beamforming techniques still exist [66].

2.16. Reconfigurable Intelligent Surfaces (RIS)

RIS allows for the dynamic management of electromagnetic waves to increase signal coverage, energy efficiency, and data throughput. By putting these surfaces on infrastructure, 6G networks will enhance performance in contexts with physical impediments or interference, such as crowded metropolitan regions and indoor environments [67].

3. Machine Learning Techniques

Machine learning techniques have been extensively studied, yet their application in 5G and 6G networks continues to reveal new insights and opportunities. Without machine learning, network operators may struggle to efficiently deliver 5G and 6G services with their diverse requirements [68]. Machine learning techniques have the most promise in their ability to predict future events, learn from system experiences on their own, and adjust to changing surroundings [69,70]. Supervised learning techniques, such as DNN, have shown promise in optimizing network performance and managing resources efficiently [4]. Recent research has focused on using reinforcement learning for dynamic spectrum management, which is crucial for maximizing the efficiency of 5G and 6G networks. For example, multi-agent RL has been applied to manage interference in dense urban environments, where traditional methods fall short [9]. Furthermore, the combination of machine learning with blockchain technology is emerging as a novel approach to enhance security and trust in 5G and 6G networks, addressing concerns related to data privacy and cyber threats [10]. These recent advancements demonstrate the potential of ML to transform 5G and 6G networks, offering new solutions to persistent challenges [11].

3.1. Supervised Learning

Training an algorithm on a labeled dataset, where each label on the input data corresponds to an output label is a type of ML known as supervised learning, as shown in Figure 4.

The goal of supervised learning is to instruct the algorithm on how to map or relate the input data to the intended output. This allows the algorithm to make predictions and find optimal solutions or classifications when presented with new, unseen data [10]. Regression and classification are the two main subtypes of supervised learning. Regression is an algorithm that predicts a continuous output [71]. The classification algorithm makes predictions about the class or category to which the input belongs [72]. Several widely used supervised methods are presented as follows:

• Artificial neural networks (ANNs): ANNs draw inspiration from the natural world and aim to replicate the intricate workings of biological neural networks (NNs). Through extensive training on intricate datasets, ANNs gain the ability to grasp the network architecture of wireless communication systems and make predictions about user behavior. ANNs prove invaluable in addressing a diverse array of challenges, including resource allocation, spectrum utilization, and cell association [63]. The emergence of deep neural networks (DNNs) has further amplified the capabilities and effectiveness of ANNs [73].
• Support vector machine (SVM): In supervised machine learning, SVM is used for both regression tasks and classification tasks. Its main objective is to identify the optimal hyperplane in a high-dimensional setting, effectively distinguishing data points belonging to different classes [74]. SVM plays a vital role in wireless networks by aiding in tasks such as signal classification and detecting interference. SVM competently categorizes signals, identifies and minimizes interference, improves channel allocation, and anticipates service quality by evaluating data trends. This improves network performance and efficiency [75].
• Naive Bayes: The probabilistic machine learning algorithm Naive Bayes calculates the likelihood of an event based on past knowledge of the circumstances surrounding it. Despite its apparent simplicity, Naive Bayes is a powerful and frequently employed algorithm for classification tasks, especially in fields like natural language processing and spam filtering. A significant number of independent continuous or categorical features can be effectively managed using Naive Bayes classifiers [76,77].
• Convolutional neural networks (CNNs): A CNN is a form of ANN designed specifically to process and evaluate complex problems such as images or videos. Neurons in these models are capable of self-optimization and unsupervised learning. The convolutional layer, the pooling layer, and the fully connected layer comprise the entire CNN architecture [78].
• Recurrent neural network (RNN): RNNs, unlike standard feedforward neural networks, feature directed cycle connections that allow them to maintain a memory of prior inputs within their internal state. As a result, RNNs can be specifically constructed to handle data sequences [79]. RNNs are frequently used to handle ordinal or temporal issues in a variety of disciplines, including language translation, natural language processing, photo captioning, and speech recognition. Long short-term memory (LSTM) network applications include assessing, categorizing, and predicting results from time series data [80].
• K-Nearest Neighbor (KNN): KNN is a flexible supervised ML method that may be used for both classification and regression issues. Predicting a data point’s class or value by taking into account the majority class or average value of its KNN in the feature space is the fundamental idea behind KNN. To classify a data point, the algorithm finds the k training cases whose feature values are nearest to the input data point. Then, choose from among these neighbors the class that shows up most frequently. Through the process of averaging the goal values of the KNN, regression forecasts the continuous output for the new data point [81,82]. The algorithm boasts numerous advantages, including its insensitivity to outliers, ease of implementation, and suitability for multi-class classifications. One notable disadvantage of this technique is that it becomes highly time-consuming, especially for large-input datasets [83].
• Decision tree: The decision tree iteratively separates the data into subsets based on the input feature values and is used for both classification and regression applications. At each node in the tree, a decision is made using a specific attribute, resulting in branches that represent alternate outcomes or more decision points. Homogeneous subsets are finally formed by making decisions at each node and building a tree that correctly predicts the target variable. The key advantages of this strategy are its ease of deployment and high classification accuracy [84].
• Random Forest: The Random Forest technique amalgamates predictions from numerous decision trees to enhance overall accuracy and resilience. Every tree within the forest is constructed separately using randomly selected input variables, and the ultimate forecast is established by combining the outcomes of these trees. Whereas the average of the individual tree forecasts is computed for regression tasks, the mode of individual tree predictions is taken into account for classification tasks [85,86]. To discover the ideal split, the algorithm evaluates only a subset of features, emphasizing the necessity of maintaining a low correlation between trees to avoid the dominance of a small number of relevant traits [87].

Supervised learning algorithms such as random forests and SVM are commonly used for traffic prediction and channel state estimation. These techniques can classify network traffic into predictable patterns, which is crucial for dynamic spectrum allocation in 5G and 6G networks. By analyzing historical data, these algorithms can predict periods of high traffic, allowing the network to proactively adjust resources and prevent congestion.

Table 1. Strengths and weaknesses of supervised ML techniques.

ML Method	Strengths	Weaknesses	Real-World Limitations
Linear Regression [89,92]	Simple and fast	Poor performance with non-linear data	Limited applicability in dynamic, non-linear network environments.
	Easy to interpret	Sensitive to outliers	Issues with scalability in high-dimensional network data.
	Performs well for linearly separable data
Logistic Regression [86,90]	Effective for binary classification	Assumes linear relationship	Has difficulty handling complex network dynamics.
	Outputs probability estimates	Limited to linearly separable classes	Faces challenges with data sparsity in large-scale networks.
Decision Tree [84,88]	Easy to understand and visualize	Prone to overfitting	Overfitting due to noisy, complex network data.
	Handles non-linear relationships	Unstable with small data changes	Requires frequent retraining in dynamic networks.
Random Forest [85,87]	Reduces overfitting	Less interpretable	High computational cost, making it impractical for real-time networks.
	Handles high-dimensional data	Computationally intensive	Performance can degrade in large-scale networks.
	Robust to noise
KNN [81,93]	Simple to implement	Slow for large datasets	Inefficient for large-scale network data.
	No training phase required	Sensitive to irrelevant features and noise	Computationally expensive during inference.
	Good for multi-class problems		Struggles with real-time applications with large data.
SVM [74,74]	Effective in high-dimensional spaces	Not suitable for large datasets	Faces scalability challenges in real-time network deployments.
	Works well with clear margin separation	Hard to tune kernel parameters	Parameter tuning is challenging in dynamic networks.
Naive Bayes [76,77]	Fast and scalable	Assumes feature independence	Has difficulty with correlated features in real-world data.
	Works well with high-dimensional data	Poor performance with correlated data	Assumes unrealistic feature independence in networks.
ANN [63,91]	Can model complex patterns	Requires large data	Demands large datasets.
	Scalable for large datasets	Computationally expensive	High cost limits use in edge computing or low-resource environments.
		Less interpretable	Interpretability issues in mission-critical applications.

provides a comparative overview of the strengths and weaknesses of the most widely used supervised ML methods [88,89,90,91].

3.2. Unsupervised Learning

Unsupervised learning, as applied in ML, refers to the process by which algorithms study and detect patterns in unlabeled data without explicit direction, as shown in Figure 5 [94].

When explicit labels are unavailable, unsupervised learning can be used to detect abnormalities, reveal hidden patterns in data, and conduct exploratory data analysis. Unsupervised learning may find patterns and correlations within datasets, which is useful for applications like pattern recognition, and data dimensionality reduction. Unsupervised learning strategies include dimensionality reduction, which streamlines complicated datasets by lowering the number of features, and clustering, which groups related data points based on internal patterns [95,96]. Clustering is an essential technique in unsupervised learning that groups related data points based on predefined criteria. The purpose of clustering is to find underlying structures or patterns in a collection when labels are not present. K-Means and others are significant clustering algorithms [97,98]. The main objective is to streamline intricate data and eliminate irrelevant or redundant features, leading to improved computational efficiency, noise reduction, and a decreased risk of overfitting in machine learning models [99,100]. A number of popular unsupervised methods are presented as follows [101,102]:

• K-means: K-means is one of the most commonly used clustering algorithms in unsupervised ML. Its purpose is to divide a dataset into distinct and non-overlapping groups or clusters. The algorithm operates iteratively by assigning data points to clusters based on their similarity and subsequently updating the cluster centroids through the calculation of the mean of the points within each cluster. Indeed, in K-Means, the “K” denotes the predetermined number of clusters that the algorithm aims to identify within the dataset. The process involves initializing centroids and assigning data points to clusters iteratively until convergence, where the goal is to reduce the total squared distance between each data point and the centroid of its assigned cluster [103,104].
• Autoencoders: Autoencoders serve as a neural network architecture employed in unsupervised learning and dimensionality reduction. Their fundamental objective is to acquire an efficient representation or encoding of the input data. The architecture comprises an encoder network responsible for compressing the input into a lower-dimensional representation, referred to as the encoding or bottleneck layer. The primary objective of the autoencoder is to reduce the reconstruction error, hence encouraging the model to effectively capture the most important properties of the input data [105,106].
• Self-Organizing Map (SOM): SOM, also referred to as a Kohonen map, is an unsupervised learning technique used for high-dimensional visualization of data and clustering. SOM is a form of ANN that organizes input data into a grid of neurons or nodes, maintaining the topological relationships of the input space. In this grid, each node represents a weight vector, and throughout training, the SOM adjusts these weight vectors based on the input data, ensuring that similar input patterns are mapped to nearby nodes. This process facilitates the creation of a meaningful and organized representation of the input data on the SOM grid [107,108].

Unsupervised learning methods, such as clustering algorithms like K-Means and DBSCAN, are used in anomaly detection within wireless networks. These algorithms can detect abnormal network behaviors or congestion patterns, helping to optimize the QoS and ensuring that the network runs smoothly even under heavy load. Table 2 provides a comparative overview of the strengths and weaknesses of the most widely used unsupervised ML methods [109,110,111,112].

3.3. Reinforcement Learning

RL is a type of ML approach in which an entity, referred to as an agent, acquires decision-making skills through its engagement with an environment, as shown in Figure 6.

The agent makes decisions according to its existing comprehension of the environment, obtains responses in the form of rewards or penalties, and refines its strategy, known as a policy, to optimize the accumulation of rewards throughout its learning process [113]. The procedure entails a delicate equilibrium between exploration, aimed at uncovering optimal actions, and exploitation focused on leveraging already-identified successful actions [114]. RL represents a fusion of supervised and unsupervised learning. It incorporates elements of supervised learning, as supervision is essential for the model to comprehend and learn the optimal performance of a system [83]. Numerous algorithms in the domain of reinforcement learning have been created to tackle diverse facets of learning and decision-making across a variety of environments [115]. Some RL methods are as follows:

Table 2. Strengths and weaknesses of unsupervised ML techniques.

ML Method	Strengths	Weaknesses	Real-World Limitations
K-Means [103,104]	Simple and fast	Assumes spherical clusters	Not suitable for irregular-shaped clusters in network data.
	Efficient for large datasets	Sensitive to initial centroids and outliers	Performs poorly with noisy or unclean data in networks.
	Easy to interpret		Limited in complex, non-linear network.
Hierarchical Clustering [116]	Dendrogram helps visualize data structure	Computationally expensive	Not scalable for large, real-time network data.
	No need to specify number of clusters	Not scalable to large datasets	Not efficient for high-dimensional network traffic.
DBSCAN [109,116]	Can find arbitrarily shaped clusters	Difficult to choose parameters	Sensitive to parameter selection in dynamic network data.
	Robust to outliers	Struggles with varying densities	Performance can drop in dense, diverse network.
PCA [111,117]	Reduces dimensionality	Linear method	May miss non-linear relationships in complex network data.
	Removes noise and redundancy	May lose interpretability of features	Can simplify critical data, losing important insights.
Autoencoders [105,106]	Effective for complex feature extraction	Requires tuning	Needs large datasets.
	Suitable for anomaly detection	Hard to interpret	Lack of interpretability makes troubleshooting difficult.
		Needs large amount of data	High computational cost, limiting use on edge devices.
t-SNE [112,118]	Great for visualizing high-dimensional data	Not scalable	Inefficient for large datasets in real-time applications.
		Not suitable for clustering	Primarily used for visualization, not clustering tasks.
GMM [110,112]	Provides soft clustering (probabilistic)	Sensitive to initialization	Assumes Gaussian distribution, limiting its use with non-Gaussian network data.
	Can model complex distributions	Assumes data is generated from Gaussian	Not suitable for all network data distributions.
SOM [107,108]	Intuitive visualization	Limited scalability	Inefficient for large-scale, real-time network data.
	Captures topological relationships	Hard to tune map size and learning rate	Requires extensive tuning, impractical for large datasets.
Isolation Forest [109,112]	Efficient anomaly detection	Less interpretable	Best for anomaly detection, not for clustering large datasets.
	Handles high-dimensional data	Not suitable for clustering	Limited use for tasks requiring data grouping in networks.

Table 2. Strengths and weaknesses of unsupervised ML techniques.

ML Method	Strengths	Weaknesses	Real-World Limitations
K-Means [103,104]	Simple and fast	Assumes spherical clusters	Not suitable for irregular-shaped clusters in network data.
	Efficient for large datasets	Sensitive to initial centroids and outliers	Performs poorly with noisy or unclean data in networks.
	Easy to interpret		Limited in complex, non-linear network.
Hierarchical Clustering [116]	Dendrogram helps visualize data structure	Computationally expensive	Not scalable for large, real-time network data.
	No need to specify number of clusters	Not scalable to large datasets	Not efficient for high-dimensional network traffic.
DBSCAN [109,116]	Can find arbitrarily shaped clusters	Difficult to choose parameters	Sensitive to parameter selection in dynamic network data.
	Robust to outliers	Struggles with varying densities	Performance can drop in dense, diverse network.
PCA [111,117]	Reduces dimensionality	Linear method	May miss non-linear relationships in complex network data.
	Removes noise and redundancy	May lose interpretability of features	Can simplify critical data, losing important insights.
Autoencoders [105,106]	Effective for complex feature extraction	Requires tuning	Needs large datasets.
	Suitable for anomaly detection	Hard to interpret	Lack of interpretability makes troubleshooting difficult.
		Needs large amount of data	High computational cost, limiting use on edge devices.
t-SNE [112,118]	Great for visualizing high-dimensional data	Not scalable	Inefficient for large datasets in real-time applications.
		Not suitable for clustering	Primarily used for visualization, not clustering tasks.
GMM [110,112]	Provides soft clustering (probabilistic)	Sensitive to initialization	Assumes Gaussian distribution, limiting its use with non-Gaussian network data.
	Can model complex distributions	Assumes data is generated from Gaussian	Not suitable for all network data distributions.
SOM [107,108]	Intuitive visualization	Limited scalability	Inefficient for large-scale, real-time network data.
	Captures topological relationships	Hard to tune map size and learning rate	Requires extensive tuning, impractical for large datasets.
Isolation Forest [109,112]	Efficient anomaly detection	Less interpretable	Best for anomaly detection, not for clustering large datasets.
	Handles high-dimensional data	Not suitable for clustering	Limited use for tasks requiring data grouping in networks.

• Q-learning: Q-learning is a model-free RL algorithm applied in situations where an agent engages with an environment to acquire an optimal policy. Functioning within discrete state and action spaces, Q-learning systematically updates its action-value function, Q(s,a), by considering observed rewards and transitions. The algorithm utilizes a straightforward update rule, involving a learning rate, immediate rewards, and a discount factor for future rewards [119,120].
• Double Q-learning: Double Q-learning serves as an extension of the conventional Q-learning algorithm, designed to counteract overestimation bias in the estimation of action values. In traditional Q-learning, a single set of Q-values is employed for both action selection and evaluation, potentially resulting in optimistic overestimation, especially during the initial learning phases. Double Q-learning resolves the issue by maintaining two sets of Q-values, employing one set for action selection and the other for action evaluation. This algorithm contributes to the enhancement of Q-value estimate accuracy and promotes learning stability, particularly in environments characterized by extensive and intricate state–action spaces [121,122].
• State–Action–Reward–State–Action (SARSA): SARSA is a model-free reinforcement learning technique used for environments featuring discrete state and action spaces. Employing an on-policy approach, SARSA systematically updates its state–action values through iterations that take into account the current state, the action taken, the immediate reward, the next state, and the subsequent action chosen by the policy [123,124].
• Deep Reinforcement Learning (DRL): DRL represents a subset of ML that integrates RL methods with DNN. This amalgamation empowers agents to learn and make decisions in environments characterized by intricate and high-dimensional state spaces. By utilizing deep learning (DL) techniques, these agents can automatically identify and represent hierarchical features from unprocessed sensory inputs, such as photos or sensor data, doing away with the necessity for manual feature engineering [125,126].
• Policy Gradient: The policy gradient method, a form of RL approach, directly enhances the policy—the decision-making strategy of an agent. To achieve this improvement, the policy parameters must be changed to maximize the projected cumulative reward. Unlike value-based methods, policy gradient techniques include calculating and updating the gradient of the predicted reward in relation to the policy parameters. The policy is guided toward acts that result in larger rewards through this iterative optimization process [127,128].
• Actor–Critic: Value-based and policy-based techniques are used in Actor–Critic, an RL architecture. The actor, who chooses acts in accordance with the present policy, and the critic, who evaluates the selected actions and provides input to improve the policy, are the two main components of this system. The actor’s objective is to acquire a policy dictating actions in various states, and the critic estimates the value function, serving as a gauge for the anticipated cumulative reward. The actor utilizes this feedback to enhance its policy, and the critic is updated to better approximate the true value function [129,130].

RL is particularly well-suited to resource management in network slicing. In 5G and 6G, network slices are created to cater to the specific needs of different services, such as IoT, autonomous driving, or critical communications. RL agents can autonomously decide how to allocate resources to different slices based on real-time feedback, thereby optimizing network performance without manual intervention.

Table 3. Strengths and weaknesses of reinforcement learning tchniques.

RL Method	Strengths	Weaknesses	Real-World Limitations
Q-Learning [119,120]	Simple to implement	Not suitable for large or continuous-state spaces	Slow convergence in dynamic networks.
	Model-free	Convergence can be slow	Not ideal for real-time, large-scale networks.
	Good for discrete action spaces		Limited in complex, evolving environments.
SARSA [123,124]	On-policy learning	Slower learning	Slow adaptation to network changes.
	Safer exploration	Sensitive to policy changes	Less efficient in unstable networks.
DQN [121,122]	Handles high-dimensional state spaces	Requires large memory	High memory demand limits use in resource-constrained devices.
	Effective in complex environments	Sensitive to hyperparameters	Requires significant computational resources.
		Requires significant computational power	Not suitable for real-time deployment.
Policy Gradient [127,128]	Suitable for continuous-action spaces	High variance in updates	High variance can lead to instability.
	Can learn stochastic policies	Requires careful tuning of learning rate	Sensitive to tuning parameters in real-time applications.
Actor-Critic [129,130]	Combines value-based and policy-based methods	Complex implementation	High computational cost and complexity.
	Lower variance than pure policy gradients	Stability issues during training	Can be unstable in dynamic environments.
PPO [131,135]	Stable and efficient	May be sample inefficient	Inefficient in real-time, large-scale networks.
	Easy to implement and tune	Computationally intensive	High computational cost for real-time deployment.
	Widely used in practice		Requires significant resources for adaptation.
Monte Carlo [132,134]	Simple to implement	High variance in returns	Inefficient for long-term network planning.
	No need for environment model	Inefficient for long episodes	Better suited for episodic tasks, not continuous networks.
	Suitable for episodic tasks		Not ideal for dynamic network environments.
Temporal Difference [131,133]	Learns online	May converge slowly	Slow convergence in rapidly changing networks.
	More efficient than Monte Carlo	Requires balanced exploration–exploitation	Challenging to balance in fluctuating network states.
Model-Based RL [133,134]	Sample efficient	Requires accurate environment model	Needs precise models, which may not be available.
	Can simulate future states	High modeling complexity	High complexity makes real-time deployment challenging.
	Useful for planning		Requires significant computational resources.

provides a comparative overview of the strengths and weaknesses of the RL [131,132,133,134].

4. Machine Learning Techniques for 5G and 6G Wireless Networks

ML methods are essential for improving the effectiveness, efficiency, and dependability of wireless communication in the domains of 5G and 6G wireless networks. The application of ML is evident in several critical domains within the context of 5G and 6G, influencing their performance and capabilities. The problems addressed by ML in these contexts include, but are not limited to, resource allocation, interference management, beamforming, and security. The following section presents a comprehensive discussion of significant problems.

4.1. Supervised Learning Methods

In 5G and 6G wireless networks, supervised learning methods can be utilized for optimizing networks, managing resources, and improving performance. Supervised learning can be deployed in the following domains.

4.1.1. Resource Allocation

The application of supervised learning for RA in 5G and 6G wireless communications entails utilizing historical datasets that contain labeled information regarding network conditions and the corresponding optimal resource allocation decisions. In [136], a resource allocation system is proposed that takes into account the uncertain connection between partial channel state information (CSI) and proportional fairness using DL. They gather data and build a dataset for supervised learning. The numerical results demonstrate that the suggested method outperforms the traditional resource allocation scheme in terms of performance. In [137], the author presented a supervised approach to addressing the problem using a framework built on graph neural networks. In terms of average sum rate and sample efficiency, the suggested framework performs better than benchmark systems. In [138], the authors introduce a novel approach, termed generalization-representation learning (GRL), to tackle the challenges associated with power allocation. The authors propose that there is a function that can be used to indicate the relationship between the parameters of the network and the best way to allocate resources, and they solve the issues by optimizing this function. This method combines training approaches that are data-driven (supervised) with model-driven approaches (unsupervised), leading to accurate predictions of the optimal function, with satisfactory results.

The author in [139] suggested a modified RA strategy that uses a learning-based resource segmentation method to solve the RA problem. This approach involves the use of a modified random forest algorithm and positional coordinates to derive the location coordinates of end-users. The simulation analysis illustrates the efficiency of the proposed method concerning throughput and energy efficiency. The author in [140], suggests a technique that utilizes DL for power allocation for massive MIMO. Supervised learning is employed to obtain power allocation through a particle swarm optimization-based algorithm. Power prediction is performed using the trained DNN. Deep learning improves the computational complexity and optimizes the sum-rate. The authors conduct a comparison of various supervised ML algorithms, including random forest, SVM, and, ANN to predict data rates. The findings indicate that the random forest technique achieves the minimum prediction error, with the error being more significant in the downlink transmission direction but notably lowered in the uplink transmission direction [141]. In [142], a combination of two renowned beamforming techniques, specifically, zero-forcing and maximum ratio transmission, are used in a MISO (Multiple-Input Single-Output) channel with K users. The suggested technique utilizes a deep neural network with a total rate of 99% and input nodes that consider transmit power and channel vector. The output provides the factors that are used to combine signals for the purpose of beamforming in the transmitter.

4.1.2. Channel Allocation

The problem of channel allocation with a focus on specific tasks is formulated as a decentralized partially observable Markov decision process. The authors present a supervised DNN approach for adaptive bit allocation in heterogeneous networks, considering imperfect CSI. Accurate CSI estimation in such networks plays a crucial role in system performance. Additionally, minimizing feedback overhead represents a notable challenge within the framework of heterogeneous networks [143]. In [144], the power and channel allocation challenges aim to optimize the sum rate and achieving the lowest user-achievable rate. Due to the non-convex nature of the construction model and the high-dimensionality of response variables, a DRL approach, specifically the distributed proximal policy optimization, is proposed for RA. Performance in heterogeneous networks is enhanced by the proposed method.

4.1.3. Interference Management

Interference is a significant issue in densely populated wireless environments. Effective interference management is crucial for maintaining the quality and reliability of communication. Leveraging ML techniques for interference management in 5G and 6G wireless communications marks a notable stride in wireless communication technology. In this context, interference management encompasses tackling challenges stemming from signal interference within the shared frequency spectrum. ML approaches, such as deep learning and multi-agent systems, have been used to predict and mitigate interference by learning from historical data and real-time network states [11]. In [145], the author suggests employing DRL for interference management by integrating joint power control and beamforming in multi-cell networks. The algorithm proposed is designed to operate effectively for any number of cells, aiming to enhance network performance, as assessed by the achievable signal-to-interference-plus-noise ratio (SINR) and sum-rate. In [146], a smart interference management approach is proposed, utilizing DRL. Power management and joint sub-band masking are used in this strategy. A Markov decision technique is used to formulate the sub-band resource masking problem. DRL is employed to approximate the policy functions, mitigating the computational and storage challenges associated with conventional tabular-based approaches.

4.1.4. Beamforming

Beamforming is essential for enhancing signal quality and extending the coverage area of 5G networks. ML algorithms, including supervised and reinforcement learning, have been employed to optimize beamforming parameters, resulting in improved signal strength and reduced interference [10]. Supervised learning for beamforming in 5G and 6G wireless networks involves utilizing labeled historical data to train models. These models are designed to predict optimal beamforming strategies based on specific network conditions. In [147], a pioneering deep learning approach is introduced, utilizing a RNN, for beam selection. The model is capable of predicting the serving base station (BS) and beam for individual drones by leveraging their previous locations and trajectories. According to the simulation data, the suggested method predicts beams with an accuracy of more than 90%. A deep neural network is introduced for optimizing downlink beams in mm-wave networks, aiming to improve data rate. The suggested model has improved performance and resilience, as demonstrated by the simulation results. Remarkably, traditional methods frequently rely significantly on sub-6GHz data, especially in areas with poor signal-to-noise ratios (SNRs) [148]. In order to minimize the space needed for the initial beam, they provide a DNN approach for mm-wave beam selection. The results demonstrate that the recommended technique for beam selection can reduce beam overhead by as much as 79.3% [149].

4.1.5. Security

Security in 5G networks encompasses various aspects, including authentication, data privacy, and protection against cyber-attacks. ML techniques, such as anomaly detection and adversarial training, are increasingly being integrated into security frameworks to identify and mitigate potential threats in real-time [17].Supervised learning for security in 5G and 6G wireless networks entails using labeled datasets to train models. These models are developed to predict and mitigate security threats based on specific network conditions. To identify various attacks on machine learning mechanisms, a hybrid approach is employed in this work. The study addresses threats such as unfair use of resources, Denial-of-Detection, and Denial-of-Service, utilizing an enhanced ML methodology. The suggested method incorporates a long short-term memory model to enhance accuracy in decision-making and the classification of attacks in 5G and 6G networks [150]. The authors use explainable AI in intrusion detection with a decision tree to improve trust management. The intrusion detection system uses a simple decision tree approach to decompose sub-choices, simulating a human decision-making process. The findings show that the suggested approach’s accuracy is on par with cutting-edge algorithms [151].

Table 4. Supervised machine learning techniques for 5G and 6G wireless networks.

Reference	ML Technique	Objective	Description
[137]	Graph Neural Network	Resource Allocation	Optimize efficiency and system sum rate
[139]	Random Forest Algorithm	Resource Allocation	Optimize throughput and energy efficiency
[140]	Deep Learning	Power Allocation	Optimize power allocation in massive MIMO systems
[144]	Deep Learning	Power and Channel Allocation	Improve power and channel allocation in heterogeneous networks
[145]	Deep Learning	Interference Management	Enhance overall network performance and sum rate
[147]	Recurrent Neural Network	Beam Selection	Optimize beam prediction accuracy
[149]	Deep Neural Network	Beam Selection	Enable efficient beam selection
[150]	Decision Tree	Security	Optimize trust management mechanisms

represents the supervised ML techniques for 5G and 6G wireless networks.

4.2. Unsupervised Learning Methods

Unsupervised learning techniques in machine learning are instrumental for uncovering patterns and relationships within data without labeled outputs. In 5G and 6G wireless communication, these unsupervised learning methods find application in various aspects. Here are some potential use cases:

4.2.1. Resource Allocation

Unsupervised learning for RA in 5G and 6G wireless communication involves utilizing unlabeled data to identify patterns and structures within the network. This approach enables more adaptive and data-driven resource management. In [152], the authors present a distributed resource allocation system that blends an unsupervised learning network with a deep Q network (DQN). An optimal channel power control method used a DNN for power control, utilizing an unsupervised learning technique. The objective is to maximize the sum-rate and accounting for the associated constraint handling. The methodology used in this study performed better than traditional algorithms and maximized the sum-rate. In [153], a graph NN-based strategy is used to adaptably optimize allocations of resources. An unsupervised approach is used to efficiently train the model. The algorithm presented efficiently addresses the resource allocation problem. Moreover, the suggested unsupervised training approach demonstrates superior convergence speed and performance when compared to training methods based on supervised learning and RL. In [154], a multi-agent RL algorithm has been proposed for RA, incorporating unsupervised learning. The reinforcement model utilizes a concise representation of Q-values to address the problem, reducing computational complexity expediting convergence, and optimizing energy efficiency.

In [155], the authors explored anomaly detection within an unsupervised framework and introduced algorithms based on LSTM NNs, optimizing the substantial performance improvements. The author in [156], devised an unsupervised learning approach utilizing aggregation graph neural (GNN) networks to address challenges in optimal resource allocation. The method relies on information present at each network node. Learning-based approaches are expected to replace traditional optimization methods because of their better performance and low computing complexity. In [157], the author focuses on addressing power allocation in cloud radio access networks (CRAN) using an unsupervised deep learning approach. This can potentially improve the effectiveness of power allocation approaches based on DL. The proposed method maintains minimal computing complexity and achieving performance very near to the optimal levels. To maximize transmit powers in cellular wireless networks, in [158], the author uses an unsupervised trained feedforward neural network, taking into account both uplink and downlink scenarios.

4.2.2. Channel Allocation

The allocation of channels in 5G and 6G wireless networks is challenging because of the increasing complexity associated with these advanced communication technologies. In [159], the author presents a DL-based channel estimation method for massive MIMO systems. The suggested estimator employs a DNN specifically designed based on the deep image prior network. It begins by denoising the received signal using this DNN and subsequently applies conventional least-squares estimation. The suggested approach eliminates the need for training and utilizes significantly fewer parameters compared to conventional DNNs. Furthermore, the proposed deep channel estimator demonstrates robustness against pilot contamination, and under specific conditions, it can effectively eliminate such contamination. In [160], the authors introduce a pilotless channel estimation scheme that does not rely on any pilot signals. The proposed scheme exclusively leverages data signals and incorporates the K-means clustering algorithm. In [161], the authors optimize RA for channel estimation in the context of URLLC. The objective is to create innovative unsupervised deep learning algorithms, both model-based and model-free, to train a DNN for the purposes of RA and data transmission. The deep learning algorithms outperform channel estimation. In [162], the authors introduce a novel channel estimation approach that incorporates deep learning to enhance the least squares estimation. The least squares estimation is a cost-effective method, which tends to incur relatively high channel estimation errors, and our proposed method aims to address this limitation. This objective is accomplished by employing a MIMO system featuring a multi-path channel profile, which is utilized for simulations within the context of 5G networks.

4.2.3. Interference Management

The effective management of interference is a pivotal consideration in the operation and design of wireless networks, particularly within the framework of 5G and 6G networks. Unsupervised learning emerges as a valuable tool for interference management, offering adaptive and efficient solutions. In [163], the authors explore the application of federated learning in a multi-cell wireless network to manage interference for uplink and downlink. To address the interdependence between downlink and uplink transmissions, along with the inter-cell coupling, a multi-cell federated learning optimization is introduced. This system’s objective is to effectively manage interference for both uplink and downlink communications. The author of [164] introduces an unsupervised learning algorithm designed for power allocation in interference management. Additionally, a transfer learning approach is outlined to acquire interference management, specifically in the context of transmit beamforming, and improve the overall performance. In [165], the author describes a unique technique for interference minimization based on DL. DL is used to decrease interference by learning interference characteristics directly from the data, avoiding the need for expert systems. In [166], the author focuses on creating enhanced inter-cell interference cooperation using DL. The simulation findings show that DL-based interference management outperforms expectations.

4.2.4. Beamforming

Beamforming stands as a pivotal technology within the realm of 5G and 6G wireless networks. This methodology involves directing the transmission of radio signals toward a specific direction, thereby amplifying signal quality and bolstering the overall performance of communication systems. In [167], the authors investigate an ML beamforming method based on the KNN approximation. This approach is designed to learn and produce appropriate beamforming setups by analyzing the distribution of throughput demand. The achieved energy and spectral efficiency values align with other approaches, demonstrating comparable performance. Additionally, the algorithmic complexity is reduced, as per-user beamforming calculations are omitted due to machine learning. In [168], a low-complexity orthogonal hybrid beamforming design is proposed, assisted by an ML-based beam-user selection scheme to optimize the performance.

4.2.5. Security

It is critical to ensure the security of 5G and 6G networks, given the heightened complexity, scale, and potential vulnerabilities associated with this next-generation wireless technology. The traditional security measures remain pertinent, and the application of unsupervised learning techniques can enhance the ability to address evolving threats and vulnerabilities in a more dynamic and adaptive manner. In [169], the authors used CNN and stacked autoencoders in an unsupervised manner to identify intrusions. In [170], the author suggests an unsupervised Gaussian mixture model technique for enhancing the performance of the physical layer security model. In [171], the author proposes the utilization of software-defined security (SDS) as a tactic to create an automated, adaptable, and scalable network defense system. SDS intends to make use of current advances in ML by creating a CNN, employing an NN architecture search to detect abnormal network traffic. The model attains a perfect accuracy rate of 100% in identifying benign traffic and demonstrates a detection rate of 96.4% for abnormal traffic, underscoring the promising potential of this approach. In [172], the author solves security challenges, and proposes a DRL technique that uses unsupervised learning to detect multiple attacking possibilities. In [173], the author introduces a blockchain-based secure federated learning approach for creating smart contracts and preventing unreliable participants and malicious activities. The suggested framework is adept at deterring poisoning and membership inference attacks, consequently enhancing the security of federated learning in 5G and 6G wireless networks.

Table 5. Unsupervised machine learning methods for 5G and 6G wireless networks.

Reference	ML Technique	Objective	Description
[152]	Unsupervised Deep Learning (objective-based DNN)	Resource Allocation	Maximize the system sum rate
[153]	Unsupervised Graph Neural Networks	Resource Management	Optimize resource allocation efficiency
[154]	Unsupervised Dynamic Clustering	Resource Management	Improve energy efficiency
[156]	Unsupervised Agg-GNN	Resource Allocation	Joint optimization of power and resource allocation
[157]	Unsupervised Deep Learning	Power Allocation	Optimize power allocation in cloud radio access networks (CRAN)
[160]	K-means Clustering	Channel Allocation	Channel estimation and grouping
[161]	Unsupervised Deep Learning	Channel Estimation	Optimize resource allocation for accurate channel estimation
[174]	Unsupervised Score Based Learning	Channel Estimation	Improve wireless channel estimation
[164]	Unsupervised Deep Learning	Interference Management	Enhance interference mitigation performance
[175]	Unsupervised Deep Learning	Beamforming	Improve energy and spectrum efficiency
[170]	Gaussian Mixture Model	Security	Enhance overall network security
[155]	Unsupervised LSTM	Anomaly Detection	Optimize anomaly detection

presents some unsupervised ML methods for 5G and 6G wireless networks.

4.3. Reinforcement Learning Methods

Reinforcement learning has been investigated as a potential strategy for enhancing the performance and administration of wireless networks. The challenges posed by 5G and 6G wireless networks include dynamic and diverse environments, fluctuating traffic patterns, and a range of Quality of Service (QoS) requirements. RL presents a viable solution to tackle these challenges, facilitating autonomous real-time decision-making and optimization. There are several applications of reinforcement learning within the domain of 5G and 6G wireless networks.

4.3.1. Resource Allocation

Resource allocation problems in 5G and 6G wireless communications can be successfully addressed through the application of RL. In [176], the authors introduce a scheme known as deep transfer RL, designed for the concurrent allocation of radio and cache resources to support a 5G network. The suggested schemes involve learner agents leveraging the knowledge of expert agents to enhance their performance in current tasks and improve the overall performance in the context of URLLC and eMBB. In [177], a dynamic resource allocation framework is put forward, which combines blockchain technology with multi-agent DRL. The aim is to optimize resource allocation in the context of 5G and 6G networks with multiple unmanned aerial vehicles (UAVs). The goal is to allocate resources efficiently to mobile users and minimize costs. In [125], a network slicing technique is developed based on DRL to determine the RA policy that maximizes long-term throughput and adherence to QoS in 5G and 6G networks. The introduced method demonstrates its efficacy in optimizing overall throughput over an extended period and effectively handling the coexistence of various use cases in 5G and 6G networks.

In [178], the author provides a DRL-based control strategy for meeting the demanding QoS criteria of URLLC and eMBB via resource allocation in the context of 5G. The suggested strategy aims to decrease user equipment energy usage and simultaneously optimize system utility performance. This performance metric is defined by its spectral efficiency, highlighting the efficiency of utilizing the available spectrum resources. The author in [179] presents a model based on DRL using an actor–critic architecture employed for optimizing resource allocation and addressing the joint network control challenges in IoT. Each IoT allotted data rate is reduced in part by the actor–critic algorithms. The proposed model demonstrates superior performance compared to conventional approaches across various network parameters and metrics. In [180], the author presents a DRL-based approach for maximizing the downlink signal-to-noise ratio in intelligent reflecting surface (IRS) communications. According to the simulation findings, the system not only minimizes time consumption but also reaches almost the maximum received signal-to-noise ratio. For resource allocation, a single-agent Q-learning approach is used in [181], which makes use of past performance to provide acceptable outcomes. A multi-agent Q-learning method is used to perform task offloading choices. The suggested algorithm’s efficacy is confirmed by contrasting its results with those of traditional algorithms in a range of network settings. The optimization issue is formulated in order to optimize the overall data rate across all D2D users in [182] for the power allocation and simultaneous uplink–downlink subcarrier assignment of D2D pairs. Using a model-free double-deep Q-network approach and a distributed DRL technique, each D2D pair solves the joint optimization problem as an agent. The proposed double deep Q-network approach exhibits minimal computational cost and rapidly converges to achieve a performance that is close to optimal, and is therefore suitable for implementation in D2D underlay wireless cellular networks.

4.3.2. Channel Allocation

Reinforcement learning can be successfully utilized to address the channel allocation problem in 5G and 6G networks. A hybrid optimization problem including power control, channel allocation, and beamforming is defined in [183], considering the constraints of minimal secrecy rate and SINR. A multi-agent RL method is used to achieve the highest long-term reward; the suggested approach improves performance overall. The authors of [184] provide a model-based approach for channel allocation that is based on RL. Simulations show that the method exhibits tremendous promise for 6G systems, performing brilliantly on genuine long-term evolution and 5G channels. The authors put forth a solution in the form of a multi-agent RL to address this challenge. When compared to baseline techniques, the suggested method has the potential to improve the efficacy and efficiency of multi-agent communication [185]. In [186], an accurate channel estimation technique for 5G networks is proposed using a Monte Carlo approach. The method combines the best aspects of both traditional and DL approaches by integrating conventional pilot-based channel estimations as a prior into the structure of the DL model. In [187], an innovative method is proposed for the allocation of pilots and estimation of channels in reconfigurable intelligent surface systems using learning techniques. The method involves training a masked autoencoder (MAE) to accomplish precise channel estimate with a constrained number of pilots. Subsequently, a DRL agent learns pilot allocation policies based on the insights gained from the MAE to optimize the channel estimation.

4.3.3. Interference Management

The utilization of RL stands out as a powerful strategy for efficiently addressing interference management challenges in 5G wireless networks. Describe interference management, power management, and beamforming joint optimization as a non-convex optimization problem with a maximizing SINR goal [188]. To tackle this challenge, employ DRL to provide a solution and maximize the sum-rate. In [189], the authors introduce a path planning scheme for a network of cellular-connected UAVs that takes into consideration interference awareness. A DRL is proposed, aiming to decrease the computational complexity associated with the proposed method and minimize interference. Enhancing downlink throughput in the 5G network is primarily reliant on the effective suppression of co-channel interference. To address this, interference whitening is proposed as a valuable, low-complexity linear technique aimed at mitigating colored interference within a MIMO-OFDM system. To tackle this issue, the author in [190], suggests a novel approach known as reinforcement learning-based interference whitening. This method dynamically regulates the interference whitening mode through a learning algorithm, adapting to the specific requirements of the environment.

4.3.4. Beamforming

In 5G and 6G wireless networks, where beamforming is an essential method to enhance communication between base stations and user devices, reinforcement learning may be applied to optimize beamforming. In [191], a multi-agent DRL framework was specifically proposed to increase knowledge of the simultaneous optimization of beamforming from the BS and the reflection beam from the reconfigurable intelligent surface. This optimization process is achieved solely based on received power measurements. The learning framework successfully learns optimized base station beamforming and reconfigurable intelligent surface configurations. The author introduces RL-based methods for beamforming and end-to-end channel prediction in a multi-user downlink system [192]. To enable autonomous learning of the beamforming policy with the aim of optimizing the transmission sum rate, the actor–critic-based beamforming layer is presented. In [193], the author proposes a DL-integrated RL approach designed to enhance the understanding of intelligent beam-steering in 6G networks. The main strategy is using RL between the user and the BS to optimize the beam direction, which significantly raises the SNR.

4.3.5. Security

RL has the potential to make a substantial contribution to enhancing the security of 5G and 6G wireless networks. It achieves this by facilitating intelligent and adaptive responses to emerging threats and vulnerabilities. The most recent advancements in 5G and 6G networks have expanded the capabilities of IoT applications by delivering unprecedented levels of connection, speed, and latency. These approaches also bring notable security risks that have the potential to result in widespread harm. DRL enables the construction of intelligent security solutions capable of adapting to the dynamic and complicated nature of IoT applications connected to 5G and 6G networks. In [194], the author proposes a novel approach aimed at enhancing the security of IoT applications in the 5G and 6G networks. This involves the creation of an intrusion detection system that utilizes the DRL method. In addressing attacks like dynamic jamming and swept jamming in [195], the authors employ a multi-agent RL method for an efficient defense strategy. Based on simulations, the algorithm is able to successfully avoid these sophisticated jamming attempts. Its agents work together to share the spectrum. In [196], the authors investigate a wireless secure communication system aided by the IRS. In order to ensure secure communication for numerous authorized users in the face of many eavesdroppers, the IRS is strategically placed to control its reflecting components. With the highly dynamic and complex nature of the system, tackling the non-convex optimization problem poses a challenge. Consequently, we introduce a pioneering approach that utilizes DRL for secure beamforming. In dynamic situations, this method seeks to determine the best beamforming policy to prevent eavesdroppers. The suggested secure beamforming approach, based on DL, markedly enhances the secrecy rate of the system and Quality of Service satisfaction in IRS-aided secure communication systems. The RL methods for 5G and 6G wireless networks are shown in

Table 6. Reinforcement learning techniques for 5G and 6G wireless networks.

Reference	ML Technique	Objective	Description
[176]	Deep Transfer Reinforcement Learning	Resource Allocation	Optimize resource allocation under URLLC and eMBB requirements
[185]	Multi-Agent Reinforcement Learning	Channel Allocation	Improve communication efficiency and system performance
[125]	Deep Reinforcement Learning	Resource Allocation	Learn resource allocation policies that maximize throughput
[178]	Deep Reinforcement Learning	Resource Allocation	Improve energy and spectrum efficiency
[179]	Deep Reinforcement Learning	Resource Allocation	Enhance overall network performance
[182]	Multi-Agent Deep Reinforcement Learning	Power Allocation	Optimize power allocation with reduced computational complexity
[183]	Multi-Agent Reinforcement Learning	Channel Estimation and Energy Efficiency	Joint optimization of beamforming, channel allocation, and power control
[187]	Deep Reinforcement Learning with Autoencoder	Channel Estimation	Accurate channel estimation in reconfigurable intelligent surfaces (RIS)
[188]	Deep Reinforcement Learning	Interference Management	Joint optimization of power control, beamforming, and interference mitigation
[197]	Lyapunov-Driven Deep Reinforcement Learning	Interference Management	Enable low-latency, energy-efficient, and accurate inference for RIS-aided networks
[198]	Deep Reinforcement Learning	Beamforming	Joint optimization of power control and beamforming
[193]	Deep Learning–Integrated Reinforcement Learning	Beamforming	Optimize beam direction selection
[194]	Deep Reinforcement Learning	Security	Enhance security in IoT-enabled 5G and 6G networks
[195]	Multi-Agent Reinforcement Learning	Security	Improves network resilience against security threats

.

4.4. Synthesis and Comparative Analysis

To synthesize the above techniques,

Table 7. Comparative analysis of ML paradigms across 5G and 6G wireless network optimization domains.

Optimization Problem	Recommended ML Method	Discussion
Link adaptation [26,140]	Supervised learning (DNN, tree-based)	Labels can be obtained from measurements/simulators; fast inference; learns nonlinear physical layer mappings better than hand-crafted rules
Channel estimation/CSI denoising [159,162]	CNNs/Transformers (supervised or self-supervised)	High-dimensional structured inputs; deep feature extraction handles noise/non-idealities better than linear estimators
Beam selection/beam management [149,168]	Supervised and sequence models (RNN/Transformer)	Decisions depend on context (mobility/blockage); supervised gives stable baseline; bandits adapt online with limited exploration
Scheduling [199]	Imitation learning/supervised from solvers; constrained RL when needed	Hard real-time constraints; supervised is stable and low-latency; RL helps when long-horizon objectives dominate
Power control/interference coordination [126,152,188]	GNNs and supervised; multi-agent RL for coordination	Interference is relational (graph-structured); GNN generalizes across topologies; multi-agent RL captures coupled multi-cell decisions
Load balancing/cell association [63,138]	Learning-to-optimize (supervised); RL (long-horizon)	Trades immediate throughput vs. long-term congestion; RL models delayed effects; L2O approximates solver outputs efficiently
Handover/mobility management [73,115]	RL/contextual bandits and sequence modeling	Sequential decision with delayed reward; bandits are lightweight; sequence models capture mobility patterns
Traffic prediction/demand forecasting [117,192]	LSTM, Transformers; hybrid statistical and ML	Strong temporal patterns; Transformers capture long dependencies; hybrids improve stability and robustness
Network slicing/admission control [125,130]	RL; supervised learning-to-optimize	Explicit service level agreement. constraints; constrained RL optimizes long-term utility under QoS; L2O enables fast allocation decisions
Fault detection/anomaly detection [155,171]	Unsupervised/self-supervised (autoencoders, contrastive); tree-based	Labels are scarce; anomaly discovery benefits from representation learning; tree models work well on tabular key performance indicators and are interpretable
Security (intrusion/attack detection) [150,194]	Supervised and self-supervised pretraining; graph-based	Attacks are rare/imbalanced; self-supervision improves features; graphs capture flow/host relationships
Edge offloading/computation placement [48,181]	RL/combinatorial bandits; supervised approximations	Stochastic and context dependent (channel, queue, energy); RL/bandits handle uncertainty; supervised models enable low latency
RIS/IRS [67,187,191]	Supervised and RL fine-tuning; model-based and learning	Huge action space; supervised reduces search; RL fine-tunes under real conditions; learning helps with imperfect CSI

maps representative 5G and 6G network tasks to the ML paradigm most aligned with their decision structure and data constraints. This comparison clarifies why certain methods are preferred in practice and provides a compact guide for selecting models under latency, observability, and control requirements.

5. Machine Learning Challenges for 5G and 6G Wireless Networks

The utilization of ML applications has the potential to introduce novel avenues for research and solutions within wireless networks. ML plays an important role in facilitating the implementation of 6G networks. Although considerable research has been conducted on ML in wireless networks, numerous challenges and unresolved issues persist. The incorporation of ML into 5G and 6G wireless communication systems encounters several obstacles, which can be outlined as follows:

5.1. Complexity

The eventual implementation of deep learning algorithms in wireless devices is imperative. Nonetheless, numerous wireless devices face constraints such as limited memory and computing capabilities, rendering them unsuitable for complex algorithms. Gathering large samples and training DL models is a time-intensive task, creating a notable challenge for implementing them on wireless devices with limited power and storage capabilities. In certain scenarios, an increase in the number of samples and extended training time correlates with improved accuracy in recognizing signal and network features. However, acquiring more samples and prolonging the training process results in slower feedback. Consequently, there is a need to design DL models that can achieve optimal accuracy with fewer samples and within a shorter timeframe. Recent studies highlight the need for optimization techniques to reduce computational overhead and maintaining model accuracy and efficiency [173]. Approaches such as model pruning, quantization, and efficient neural architectures are crucial for deploying ML [4].

5.2. Resource Allocation

Resource allocation remains a key challenge for ML in 5G and 6G networks due to dynamic traffic conditions and heterogeneous service requirements. ML-based approaches must operate under strict real-time constraints and limited computational resources, as well as ensuring scalability and reliable generalization across diverse network scenarios remains difficult.

5.3. Reliability

ML-based techniques may exhibit lower reliability than traditional methods for certain wireless communication tasks. For example, in massive MIMO, DL can rival least squares and minimum mean square error in wireless channel estimation. These approaches are distinguished by slow feedback. The system response time may be prolonged during deep learning inference. This delay arises due to the limited availability of cloud computing on most wireless devices. Even in cases where access to cloud servers exists, communication with them introduces additional delays. A network comprising a large and varied user population operates dynamically as users possess diverse QoS and Quality of Experience requirements. On the other hand, users conducting financial transactions through payment software prioritize high security. Developing a cross-layer, action-based machine learning protocol specific to various applications is crucial to fulfilling distinct requirements and maintaining a balance in network resource utilization [200].

5.4. Real-Time Processing and Latency

Real-time processing is critical for many 5G applications, such as autonomous driving and industrial automation. ML models must be capable of making rapid decisions with minimal latency. Edge computing, which brings computational power closer to the data source, plays a pivotal role in achieving low-latency ML inference [1]. Techniques like edge-cloud collaboration and distributed learning are essential for balancing the trade-offs between latency and computational load [2].

5.5. Scalability and Deployment

Scalability is a key concern when deploying ML models in large-scale 5G and 6G networks. The dynamic nature of these networks requires models that can adapt to changing conditions and scale efficiently with the number of connected devices. Hierarchical ML models and transfer learning are promising strategies to enhance scalability and adaptability [10]. Moreover, containerization and micro services architectures can facilitate the deployment and management of ML models in diverse network environments [63].

5.6. Data Availability and Quality

The performance of ML models heavily relies on the availability and quality of data. In the context of 5G and 6G, data can be sparse, heterogeneous, and subject to privacy constraints. Ensuring high-quality data collection and preprocessing is essential for training robust ML models. However, gathering extensive datasets for training AI models is often challenging due to the sensitive nature of user information contained within these datasets. Mobile service providers are frequently unable to release such data without risking consumer privacy violations [173]. Additionally, synthetic data generation and data augmentation techniques can help mitigate the challenges of limited data availability [11]. Even with transfer learning, leveraging models learned on previous datasets, adapting these models to specific networks and contexts demands re-training. These restrictions severely hinder the growth of wireless AI development.

5.7. Intelligent Reflecting Surface

The timely and accurate acquisition of CSI is vital in IRS-enhanced wireless networks, particularly in MIMO-IRS and MISO-IRS networks, and in wireless networks with IRS enhancements; however, obtaining CSI is a difficult process, involving significant training overhead. In IRS-assisted non-orthogonal multiple access networks, users within each cluster need to share CSI, presenting additional challenges because of the passive nature of the IRS. Using machine learning and DL methods for leveraging CSI becomes a challenging issue, especially in cases that go beyond linear correlations [201].

5.8. Security

Ensuring the security of DL models poses an additional challenge, as NNs are vulnerable to adversarial attacks. Attackers can tamper with the training process by adding fictitious training datasets, decreasing model accuracy, and creating erroneous designs that might negatively affect network performance. Research on deep learning security and ML security in general is still in its infancy. Robust defense mechanisms, such as adversarial training and anomaly detection, are crucial for enhancing the resilience of ML models [17].

5.9. Privacy

Safeguarding user privacy stands as the foremost priority for mobile and service providers. In wireless artificial intelligence, a major challenge lies in enabling training on user-owned datasets without divulging input data or jeopardizing personal user information. A robust security approach is essential to enhance the seamless integration of deep learning into wireless communications [10].

5.10. Non-Stationarity

One of the main problems with ML-enabled 5G and 6G management is that things are always changing. For example, channel statistics, user mobility, traffic demand, interference patterns, and service needs all change over time and location. As a result, the data distribution that was used for training may not be the same as the one that was used for deployment. This mismatch is not just a theory; in real networks, QoS goals vary from app to app and can change quickly (for example, streaming services prioritize throughput over latency, and financial services have strict security requirements). This means that the “optimal” policy changes over time and makes fixed models less reliable. Non-stationarity specifically presents as: (i) degraded inference accuracy when models are utilized on unobserved cells, user densities, or propagation environments; (ii) unstable control actions in response to rapidly changing CSI or mobility, leading to oscillatory scheduling and power decisions; and (iii) performance degradation during distribution shifts [21].

5.11. Data Scarcity

Despite massive amounts of network telemetry, many learning tasks still suffer from data scarcity in practice. High-quality labels for supervision (e.g., “ground-truth optimal allocation”, fine-grained congestion states, or security incident labels) are costly to generate, and data are typically diverse across vendors, deployments, and slices. Moreover, raw data centralization is typically hampered by privacy and by communication difficulties, which hinders broad data sharing and delays model progress. These restrictions often cause: (a) overfitting to constrained situations, (b) weak generalization to under-represented corner instances, and (c) biased learning under class imbalance and missing data [202].

5.12. Overhead

Beyond accuracy, ML includes overhead that directly conflicts with strict latency and efficiency requirements. Many wireless control issues involve regular CSI acquisition or coordination, producing considerable pilot/signaling overhead, particularly for RIS systems, where estimating cascaded channels and setting large numbers of reflecting elements can be expensive. The computational cost of training and the latency/energy cost of inference might become bottlenecks at the edge, rendering the naive deployment of big models infeasible for real-time control loops [91].

5.13. Critical Analysis

In this section, we provide a critical analysis of these challenges, offering insights into their real-world implications and trade-offs.

Table 8. Critical analysis of ML in 5G and 6G wireless networks.

Challenge	Critical Analysis	Discussion
Computational Complexity [4,63,173,200]	Deep learning models require substantial computational power, which is impractical for resource constrained wireless devices. Efficient models are needed to balance accuracy with real-time processing.	Optimization techniques like model pruning and quantization are crucial for minimizing the computational load and maintaining accuracy in resource-constrained environments.
Data Availability and Quality [8,11]	ML models depend on high-quality labeled data, which is difficult to obtain due to privacy concerns and data heterogeneity. More data-efficient methods are required for better performance.	Techniques like synthetic data generation and transfer learning could help mitigate data scarcity and improve model generalization.
Scalability [10,63,88]	As networks scale, ML models must adapt to dynamic conditions. Federated learning offers potential but faces challenges in data heterogeneity and model aggregation.	Future research should focus on scalable federated learning and decentralized model updates to address the growing scale of 5G and 6G networks.
Security [17,113]	ML models are vulnerable to adversarial attacks, posing risks to network integrity. Developing robust defenses and privacy-preserving solutions is critical.	Adversarial training and robust defense mechanisms are necessary to secure ML systems, especially in mission-critical applications like autonomous driving.
Real-Time Processing and Latency [112,121,161]	Real-time applications, such as autonomous driving, demand low-latency decisions. Edge computing helps, but balancing latency with accuracy is still a challenge.	Edge-based processing and low-latency inference models are essential to meet the stringent requirements of real-time applications.
Non-Stationarity [21,131]	Changing network conditions can degrade model performance. Adaptive models are needed to handle non-stationary environments effectively.	The development of self-adjusting models capable of learning in dynamic, real-time environments is needed to improve the robustness of ML systems in 5G and 6G networks.
Overhead and Efficiency [91,202]	ML models introduce computational and communication overhead, especially in RIS-assisted networks. Reducing latency and energy consumption is essential for real-time applications.	Efficient energy consumption strategies and reduced overhead models are necessary to optimize the performance of ML applications in resource-constrained environments.

summarizes each challenge and reflects on their impact on the practical deployment of ML models in next-generation networks.

6. Future Research Directions

Future research on ML-enabled 5G and 6G networks should move beyond identifying challenges and focus on concrete research questions, technical solutions, and measurable performance objectives.

6.1. Efficient and Lightweight ML Model Design

How to design lightweight ML models suitable for edge deployment remains an open question. Techniques such as model compression, pruning, and lightweight reinforcement learning should be explored and evaluated using latency, energy consumption, and convergence speed.

6.2. Data-Efficient and Privacy-Aware Learning

Future work should investigate how ML models can be trained effectively under limited, distributed, and privacy-sensitive data conditions. Techniques such as federated learning, self-supervised learning, and synthetic data generation offer potential solutions.

6.3. Real-Time and Low-Latency Learning Frameworks

Meeting ultra-low latency requirements remains an open challenge for ML-driven 5G and 6G systems. Research is needed on edge–cloud collaborative learning, online reinforcement learning, and distributed inference architectures that support real-time decision-making.

6.4. Scalability and Network Adaptability

Developing ML models that can dynamically adapt to changing network conditions and scale efficiently with the number of connected devices is crucial for future 5G and 6G deployments.

6.5. Security and Robustness of ML-Driven Networks

ML-enabled wireless networks are vulnerable to adversarial threats such as poisoning and evasion attacks. Future work should integrate adversarial training, secure model deployment, and anomaly detection into ML pipelines. Performance should be evaluated using attack success rate, robustness, and detection accuracy.

7. Conclusions

In this paper, we presented a structured and analytical overview of ML techniques applied to 5G and emerging 6G wireless networks. The key components of next-generation wireless systems were outlined, and representative applications of ML across different network functions and learning paradigms were discussed. Supervised, unsupervised, and reinforcement learning methods were reviewed in the context of addressing major challenges in 5G and 6G communications, with an emphasis on their role in improving system efficiency and supporting complex network operations. Furthermore, we highlighted several open challenges related to the deployment of ML in next-generation wireless networks, including algorithm complexity, data quality, real-time processing constraints, scalability, and security. By synthesizing existing studies and identifying these challenges, this work underscores the potential of ML-enabled 5G and 6G networks as an important direction for future research and development. Continued investigation into efficient learning frameworks, data management strategies, low-latency operation, scalability, and robust security mechanisms will be essential to fully realize the benefits of ML in next-generation wireless networks.