AI-Driven Pilot Overhead Reduction in 5G mmWaveMassive MIMO Systems

Abstract

The emergence of 5G technology promises remarkable advancements in wireless communication, particularly in the realm of mmWave (millimeter-wave) massive multiple input multiple output (m-MIMO) systems. However, the realization of its full potential is hindered by the challenge of pilot overhead, which compromises system efficiency. The efficient usage of pilot signals is crucial for precise channel estimation and interference reduction to maintain data integrity. Nevertheless, this requirement brings up the challenge of pilot overhead, which utilizes precious spectrum space, thus reducing spectral efficiency (SE). To address this obstacle, researchers have progressively turned to artificial intelligence (AI) and machine learning (ML) methods to design hybrid beam-forming systems that enhance SE while reducing changes to the bit error rate (BER). This study addresses the challenge of pilot overhead in hybrid beamforming for 5G mmWave m-MIMO systems by leveraging advanced artificial intelligence (AI) techniques. We propose a framework integrating k-clustering, linear regression, random forest regression, and neural networks with singular value decomposition (NN-SVD) to optimize pilot placement and hybrid beamforming strategies. The results demonstrate an 82% reduction in pilot overhead, a 250% improvement in spectral efficiency, and a tenfold enhancement in bit error rate at low SNR conditions, surpassing state-of-the-art methods. These findings validate the efficacy of the proposed system in advancing next-generation wireless networks.

1. Introduction

One of the major technologies of the next generation of wireless communication, leveraging millimeter-wave (mmWave) frequencies that enable high-speed, low-latency connectivity is the 5G/6G m-MIMO (massive multiple input multiple output) system. A large number of antennas at both the transmitter and receiver are utilized in m-MIMO systems to significantly enhance the spectral efficiency, energy efficiency, data rate, and reliability of wireless communication systems [1]. These systems are particularly effective in the mmWave frequency band, where the high free-space path loss necessitates large array gains to achieve sufficient signal-to-noise ratio (SNR) even at moderate distances, and this is where beamforming emerges as an essential block to achieve this technology. Beamforming is crucial in new-generation networks for targeting specific users/devices with focused beams to enhance signal quality. Digital beamforming offers advantages like multiple closely spaced beams, super resolution, and pattern correction. Converting n signals into digital data streams in the DBF approach leads to composite signals, but it is costly and power-consuming in multi-user m-MIMO systems [2]. Analog beamforming, however, addresses cost and power issues by phase-shifting signals using analog methods, resulting in a single digitized output. Despite its benefits, analog beamforming does not fully leverage the advantages of DBF due to signal imperfections and noise introduced during amplification. On the other hand, hybrid beamforming is a promising solution to these challenges. It leverages both analog and digital processing capabilities, optimizing the performance of m-MIMO systems without compromising on the benefits they offer [3].

The beamforming task involves converting channel state information (CSI) into a beam pattern, utilizing pilot bits. Pilot signals are special reference signals transmitted by the base station to the user equipment (UE) in wireless communication systems. These signals are crucial for several reasons, including channel estimation, synchronization, and beamforming. In the context of hybrid beamforming, pilot signals play a pivotal role in facilitating the estimation of the CSI between the transmitting and receiving antennas. The CSI is essential for determining the optimal beamforming weights that maximize the signal-to-interference-plus-noise ratio (SINR) and, consequently, the data rate and reliability of the communication link. One of the major strategies used to determine where and how often pilot signals should be transmitted is the pilot placement technique [4]. Although the use of pilot signals is crucial in such systems, it imposes additional overhead in terms of bandwidth and energy consumption, leading to reduced spectral efficiency (SE) and increased energy consumption [5]. Therefore, minimizing the pilot overhead while maintaining sufficient accuracy in channel estimation is an important challenge in communication system design.

Deep learning has revolutionized hybrid beamforming by enabling AI-driven optimization [6].Unlike traditional approaches, DNNs can learn from available data sources to minimize pilot overhead and maximize spectral efficiency, and this was our focus in this study. This innovative approach promises to significantly enhance the performance of wireless communication systems. Both unsupervised and supervised approaches can be used to achieve pilot overhead decrease while enhancing BER and SE.

Few works have been reported on this topic. A user clustering-based pilot allocation scheme demonstrated a 17% reduction in required pilot overhead while enhancing spectral efficiency by 8–14% through optimization techniques like integer linear programming and heuristics [7]. The authors in [8] used integrated sensing and communications (ISAC) to reduce overhead in high-mobility V2X networks by optimizing channel state information (CSI) acquisition. The implementation of ISAC can lead to a reduction of up to 43.24% in overhead, improving communication throughput and beam tracking performance. In [9], the authors proposed a novel technique for pilot contamination analysis in 5G MIMO systems, utilizing multi-user pilot scheduling and convolutional adversarial training to enhance spectral efficiency and reduce pilot overhead, achieving significant improvements in SINR and detection accuracy. The results achieved an SINR of 72% and spectral efficiency of 85%, while detection accuracy reached 95% with a normalized MSE of 73%. The authors of [10] developed a deep learning-based hybrid beamforming technique that improved SE by 20% compared to traditional methods, with a corresponding BER reduction of 10%. Additionally, in [11], the authors proposed a data-driven approach that enhanced SE by 25% and reduced BER by 15% in high-mobility scenarios. The work discussed in [12] introduced a machine learning framework for hybrid beamforming that achieved a 35% increase in SE and a 20% reduction in BER, demonstrating the efficacy of AI-driven approaches.Furthermore, the authors of [13] presented a deep neural network-based hybrid beamforming method that outperformed existing techniques by 2 dB in terms of BER performance.

Despite significant advancements, many studies have focused on isolated AI methods or relied on traditional optimization frameworks, limiting their scalability and generalizability. Our work bridges these gaps by integrating multiple AI techniques (k-clustering, linear regression, random forest regression, and NN-SVD) into a unified framework tailored for hybrid beamforming. Furthermore, we validated our approach using scalable configurations (up to 64 Tx/16 Rx antennas) and multiple modulation schemes, providing practical insights into its applicability.

This paper investigates the application of various artificial intelligence (AI) techniques for pilot overhead reduction in millimeter-wave (mmWave) massive multipleinput and multipleoutput (m-MIMO) systems employing hybrid beamforming. The primary goal of this research was to address the challenge of pilot overhead in hybrid beamforming for 5G mmWave m-MIMO systems. By leveraging advanced artificial intelligence (AI) techniques, this paper proposes and evaluates methods that significantly reduce pilot overhead while maintaining or improving key system performance metrics, including spectral efficiency (SE) and bit error rate (BER). Specifically, the paper investigates the application of k-clustering, linear regression, and random forest regression and finally combines random forest regression with the integration of neural networks and singular value decomposition (SVD) to optimize pilot placement and hybrid beamforming strategies. This research aimed to bridge the gap between traditional pilot management techniques and emerging AI-driven solutions, contributing to the advancement of next-generation wireless communication systems. This paper contributes to the state of the art by introducing a comprehensive AI-driven framework that combines traditional and advanced machine learning (ML) methods specifically tailored for reducing pilot overhead in 5G mmWave m-MIMO systems. Unlike existing approaches, which often focus on single techniques or limited scenarios, our work contributes to the field in the following ways:

• Develops an ensemble framework integrating multiple ML methods, leveraging their complementary strengths for optimal pilot placement and channel estimation;
• Proposes a hybrid beamforming system that combines NN-predicted SVD for adaptive optimization of beamforming matrices, reducing complexity and enhancing spectral efficiency;
• Extends the evaluation to include realistic channel conditions, advanced modulation schemes, and scalable antenna configurations, providing a more holistic and practical analysis;
• Demonstrates a significant reduction in pilot overhead (up to 82%) while maintaining or improving BER and SE compared to traditional and existing AI-based methods.

The paper is organized as follows: in Section 2, the system model and methods used are described; in Section 3, the system performance is analyzed; and finally, the paper is concluded in Section 4.

2. System Model

2.1. Hybrid Beamforming and Pilot Placement

2.1.1. Hybrid Beamforming System

Hybrid beamforming (HB) can be categorized according to parameters such as required CSI within the analog beamforming segment, carrier frequency, and complexity. The optimal design or algorithm for a specific application strikes a delicate balance between these factors.

Figure 1 illustrates the transmitter/receiver architecture for the hybrid beamforming system in the m-MIMO setup. The structure includes RF chains connected to the antenna array through constant modulus phase shifters. On the transmitter side, various operations are performed on the digital baseband signal to prepare it for transmission. These operations include modulation, encoding, multiplexing, and pilot placement, which are critical for efficient communication. The RF chains enable the hybrid beamforming process, combining analog and digital techniques to optimize performance while reducing hardware complexity.

In the proposed system, the signal received at a mobile station MS in a hybrid beamforming mmWave system is expressed as follows [14]:

$$y_{u_m}={(w}_{rf}^{u_m}) \times \{{(H}_{u_m}{F}_{rf}{F}_{bb}s)+n_{u_m}\}$$

(1)

where $u_m$ is the MS (the user) being served by the station, $y_{u_m}$ is the received signal at the MS $u_m$, $w_{rf}^{u_m}$ is the RF combiner for the $u_m$ MS, $H_{u_m}$ is the channel between the BS and MS $u_m$, $n_{u_m}$ is the noise component, and $F_{bb}$ and $F_{rf}$ are the digital and analog pre-coding weights. On the other hand, the received signal of the $jth$ user can be given as follows [2]:

$$Y_k=\sum _{k=1}^j{B}_k^H{H}_k{F}_{rf}{F}_{{bb}_k}{s}_k+B_k^H{n}_k$$

(2)

where $B_k^H$ is the linear receiver beamforming vectors used to detect the transmitted signal. Additionally, $N_{RF}^T$ represents the number of RF chains at the transmitter, $N_{RF}^R$ represents the number of RF chains at the receiver, and $W_{BB}$ represents the baseband combining matrix. These components collaboratively contribute to achieving efficient channel estimation and hybrid beamforming, which are integral to the proposed system’s superior performance.

The challenge with HB rises with the need of estimating the channel conditions between the transmitter and receiver. Channel estimation is crucial for adaptive modulation and coding techniques, which dynamically adjust the data rate and error correction based on the current channel state to optimize system performance. To achieve accurate channel estimation, effective pilot placement techniques are essential. This concept is discussed in the next paragraph.

2.1.2. Pilot Placement

One of the key aspects in modern communication systems are pilot signals. These are used for channel estimation, synchronization, and beamforming. In 5G communication systems, pilots play a crucial role in achieving high data rates, low latency, and reliable communications [4].

The pilot placement technique refers to the method or strategy used to determine the locations and allocation of pilot signals within a communication system [15]. A simplified version of the pilot placement equation is expressed as follows:

$$p_k=p_0+{\mathsf{\Delta }}_p$$

(3)

where $p_k$ represents the position of the $k\mathrm{th}$ symbol, $p_0$ is the starting position of pilot symbols, and ${\mathsf{\Delta }}_p$ denotes the spacing between consecutive pilot symbols. In practical implementations, pilot placement algorithms may involve more complex equations to adaptively adjust pilot positions based on real-time channel conditions and system requirements. This is shown in Equation (4) below:

$$p_k=p_0+\alpha ·k^2+\beta ·k+\gamma$$

(4)

where the pilot positions are determined using a quadratic function. The coefficients α, β, and γ can be optimized based on channel characteristics and system requirements in order to achieve better spatial diversity and interference mitigation.

Regardless of the methods used, there are several disadvantages associated with pilot placement in communication systems. One disadvantage is the increased pilot overhead. Due to the allocation of additional resources for pilot transmission, such as time or frequency slots, there is a reduction in the available resources for transmitting actual user-data. This can impose restrictions on the system’s ability to process data and handle workloads [16]. Another disadvantage is the potential for pilot pollution. Pilot pollution occurs when pilot signals from neighboring cells interfere with each other, leading to inaccurate channel estimation and degraded system performance [4]. To tackle these problems, several approaches have been explored. In [11], Hu et al. proposed a pilot reuse technique that exploits the spatial sparsity of channel responses to achieve pilot overhead reduction in m-MIMO systems. In [17], Liu et al. proposed a compressed sensing-based pilot design method that reduces the number of pilots required for channel estimation. We compare our results to these methods later.

Accurate pilot placement enables precise estimation of the channel matrix H, which is a prerequisite for effective beamforming. Leveraging SVD, the proposed system optimally utilizes the estimated channel information to reduce pilot overhead while enhancing performance metrics such as SE and BER. SVD is discussed in the next section.

2.1.3. Singular Value Decomposition

Singular value decomposition (SVD) in hybrid beamforming serves as a pivotal technique employed to enhance the performance of m-MIMO systems within the realm of modern networks. SVD, as a fundamental matrix factorization approach, dissects a given matrix into three distinct matrices, thereby simplifying the complexity of mathematical processes and improving the efficiency of beamforming algorithms. Within the domain of hybrid beamforming, the significance of SVD becomes pronounced in its important role concerning the accurate estimation of the CSI matrix and the computation of beamforming weights [16,18].The SVD of a matrix is represented as follows:

$$H=U\mathsf{\Sigma }{V}^T$$

(5)

where $H$ is an $(m\times n)$ matrix describing the relationship between the transmit and receive antennas, $U$ is an $(m\times m)$ orthogonal matrix (left singular vectors derived from the singular value decomposition (SVD) of the channel matrix H), $\mathsf{\Sigma }$ is an $(m\times n)$ diagonal matrix with non-negative real numbers on the diagonal called the singular values of $H$, and $V^T$ is an $(n\times n)$ orthogonal matrix (right singular vectors), where $V^T$ denotes the transpose of V. Mathematically, SVD can be expressed as follows:

$$H={\sum }_{i=1}^{\min (m,n)}{\sigma }_i{u}_i{v}_i^T$$

(6)

where ${\sigma }_i$ are the singular values, $u_i$ and $v_i$ are the corresponding left and right singular vectors, and $\min (m,n)$ denotes the minimum of the number of rows and columns of $H$. This decomposition helps in understanding the channel’s behavior and designing efficient transmission strategies by exploiting the channel’s spatial characteristics, such as optimizing the transmission strategies to exploit spatial multiplexing gains and mitigate interference and reducing the complexity of solving the HB matrix weights.

2.2. AI-Driven Pilot Placement

Recent developments in AI and machine learning provide robust capabilities for enhancing different facets of wireless communication systems, such as pilot placement, allocation, and processing, in order to reduce the pilot overhead size while maintaining good signal characteristics. Machine learning algorithms can analyze historical channel data and user mobility patterns to intelligently place pilots in areas where channel variations are significant or where users are densely located. Through the optimization of pilot positions guided by learned patterns, AI can effectively minimize pilot overhead without compromising the accuracy of channel estimation. The simplest form of utilizing machine learning in pilot placement is shown in the equation below:

$$p_k=f(x_k)$$

(7)

where $x_k$ represents a feature vector associated with the $kth$ pilot symbol, which may include parameters such as channel quality, user density, interference levels, etc. [6]. The function is learned using machine learning techniques to predict the optimal pilot positions based on the input features. In supervised learning, the process involves acquiring knowledge from data that are labeled in order to formulate predictions or decisions. Conversely, unsupervised learning involves working with data that lack labels to unveil patterns within the dataset. In this paper, methods of each category are used to reduce pilot overhead. The k-clustering technique is used as an unsupervised machine learning method; as for the supervised method, the linear regression and random forest regression methods are applied. Moreover, SVD is integrated with the supervised methods in order to further improve system performance.

2.2.1. Unsupervised K-Clustering ML

Based on real-time channel conditions, AI and machine learning models can efficiently allocate pilots in time, frequency, and spatial domains. Unsupervised ML empowers systems to autonomously identify patterns and structure within the data, enabling more efficient pilot allocation and reducing overhead while maintaining high SE. Notably, k-clustering divides a dataset into $k$ clusters based on the resemblance between data points, where $k$ represents a pre-determined number of clusters designated by the user. The algorithm iteratively enhances an objective function, typically aimed at minimizing the total sum of squared distances from every data point to its designated cluster centroid [16].

In this system, k-clustering is applied directly to the pilot signals rather than to users, as the study focuses on a single-user mmWave m-MIMO system. The purpose of clustering is to group pilot signals with similar characteristics, thereby reducing redundancy and optimizing the placement of pilots in the time, frequency, or spatial domain. The process involves the following:

• Feature selection: Input features for clustering include the temporal positions of pilot symbols, their spatial allocation, and inter-pilot spacing. These features reflect the structure of the pilot signals rather than channel metrics or user-specific parameters;
• Clustering process: Using k-means clustering, the algorithm partitions the pilot signals into k clusters. Each cluster represents a group of pilot signals with similar temporal or spatial properties;
• Pilot Allocation: A representative pilot signal is selected for each cluster, which reduces the number of total pilots needed while maintaining sufficient information for channel estimation.

Given a dataset $X=\left\{x_1,x_2,\dots ,x_n\}\right.$ consisting of $n$ data points and the desired number of clusters $k$, cluster centroids $u_1,u_2,\dots ,u_k$ are initialized as the initial positions. For each data point $x_i$ the distance from the centroid is calculated:

$$d(x_i,u_j)={‖x_i-u_j‖}^2$$

(8)

Then, each point is assigned to the nearest centroid:

$$c(x_i)=argmind(x_i,u_j)$$

(9)

After that, each cluster centroid is updated to be the mean of the data points assigned to it based on the below equation:

$$u_j={\displaystyle \frac{1}{⌈S_j⌉}}{\sum }_{x_iϵS_{ j}}{x}_i$$

(10)

where $S_j$ is the set of data points assigned to cluster $j$. The step in Equation (10) is repeated and updated until the assignment of the clusters stabilizes, or a convergence criterion is met. In our work, the k-means clustering algorithm minimizes the variance within clusters of pilot signalsrepresented by the following function:

$$J={\sum }_{j=1}^k{\sum }_{pϵC_i}{‖p-u_j‖}^2$$

(11)

where $C_i$ represents the i-th cluster of pilot signals, $u_j$ is the centroid of $C_i$, and p is the pilot signal parameters such as temporal positions or spacing. This represents the objective function, where J is the sum of squared distances between data points p and their corresponding cluster centers $u_j$. After clustering, pilot signals are reassigned as follows:

$$P_i=Representative(C_i), \forall C_i$$

(12)

where $P_i$ is the representative pilot signal for cluster $C_i$. This reassignment ensures that the clustered pilots provide the necessary information for accurate channel estimation while reducing overhead.

The used algorithm aims to optimize the aforementioned objective function by iteratively adjusting the cluster centroids until reaching convergence. Considering the above equations and the main goal of reducing the pilot overhead of the system, it is possible to apply the k-clustering AI method to cluster pilots to reduce the pilot overhead and study the effect of such reduction on SE and BER.

2.2.2. Supervised ML

In supervised machine learning, the model is trained on a labeled dataset. During the training process, the model learns from input data that are already tagged with the correct output. The goal is for the model to learn the underlying patterns in the data so that it can accurately predict the output for new, unseen data. In the context of reducing pilot overhead, a supervised machine learning approach can help reduce this overhead by leveraging prior observations of the channel to predict the optimal RF beamforming/combining vectors directly from the channel measurement vectors. This is achieved through a neural network architecture known as auto-pre-coder, which is trained end-to-end as a multi-task classification problem. The auto-pre-coder network is designed to optimize the transmitter/receiver measurement vectors in an unsupervised way to focus the sensing power on the most promising directions.

Linear Regression

Linear regression, in particular, aims to build a model that designs the correlation between one or more input characteristics (independent variables) and a continuous target variable (dependent variable). The algorithm forms a linear equation that optimally represents the dataset, enabling it to predict new and unobserved target variables on the input characteristics [14]. The objective of this method is to estimate the channel characteristics from a reduced set of pilot signals. This is achieved through a five-step process: data preparation, feature selection, model training, model prediction, and optimization. In the first step, data are collected. The data consist of pilot signals and corresponding channel characteristics such as channel capacity and number of streams. In the second step, relevant features such as pilot signals, SINR values, and other relevant parameters are identified. Then, linear regression is used to train a model. The model uses pilot signals to predict the channel characteristics. Based on the data, a relationship between the input pilot signals and the output channel characteristics is learned. Once the model is trained, it becomes capable of making predictions about channel characteristics for new pilot signals. Afterwards, these predictions are used for channel characteristics estimation to transmit data without requiring full pilot overhead.

Linear regression model is represented by the following equation:

$$\widehat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\dots +{\beta }_n{x}_n$$

(13)

where $\widehat{y}$ is the predicted channel characteristic; $x_1,x_2,\dots ,x_n$, are the input pilot signals (features); ${\beta }_0,{\beta }_1,{\beta }_2,\dots ,{\beta }_n$ are the coefficients or parameters that are learned by the linear regression model. Finally, for optimization, the coefficients ${\beta }_0,{\beta }_1,{\beta }_2,\dots ,{\beta }_n$ undergo a process of optimization throughout the training phase in order to achieve a reduction in the prediction error, which is calculatedby comparing the predicted channel characteristics and the actual channel characteristics observed in the training data.

Random Forest Regression

Random forest regression is a supervised machine learning algorithm that is part of the ensemble learning methods. Specifically tailored for regression tasks, it aims to predict a continuous outcome variable by leveraging one or more predictor variables. Random forest regression creates multiple decision trees, each trained on a random subset of the data and features. The final prediction is the average of all the trees’ predictions [19].

In the realm of minimizing pilot overhead in beamforming systems, random forest regression offers a viable approach for predicting optimal pilot arrangements based on input parameters like channel characteristics and system specifications [20]. By adapting the equations to the context of a multi-user MIMO system with M transmitter antennas, N receiver antennas, and Kdata streams, the following equation can be obtained:

$$G={\sum }_{i=1}^{K}p(i)(1-p(i))$$

(14)

where $G$ is the Gini impurity factor, which is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset, and $p(i)$ represents the probability of user $i$ at a node in the decision tree. The entropy, which is a measure of uncertainty or disorder in a system equation, is given by the following:

$$H={\sum }_{i=1}^{K}p(i){log}_2(p(i))$$

(15)

The mean square error MSE is also a useful parameter in random forest regression used as a loss function for assessing the accuracy of predictions made by the model given by the following equation:

$$MSE={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{(y(i)-\widehat{y}(i))}^2$$

(16)

where $n$ represents the number of samples in the training data, $y(i)$ represents the actual pilot overhead, and $\widehat{y}(i)$ represents the predicted pilot overhead. The final prediction is given by the equation below:

$$\widehat{y}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\widehat{y}(i)$$

(17)

In the equations provided, M (the number of transmitter antennas) does not directly appear in the decision tree splitting criterion or the aggregation of predictions. However, M indirectly affects the feature space used in training the random forest regression model. The features used in training the model include characteristics related to the transmitter antennas, such as antennas’ spatial distribution and transmit power levels. These features can indirectly influence the decision tree splitting process and the model’s predictions, which are noted in the Results Section of this study.

Neural Networks SVD Enhancement

Neural networks (NNs) are a cornerstone of machine learning, designed to mimic the human brain’s structure and function, where these networks consist of layers of interconnected nodes, giving an analogy of neurons, which process input data through a series of connections and transformations. They can learn and model complex relationships between inputs and outputs through training on datasets. The process of learning takes place through the adjustment of connection weights using back-propagation in order to reduce errors, wherein the neural network acquires knowledge from the training data by minimizing the difference between its predicted outcome and the real outcome [21,22].

Integrating SVD hybrid beamforming with NN regression can involve using NNs to predict the optimal beamforming vectors or other relevant parameters based on the channel’s characteristics. This prediction is based on historical data, where the NN learns the relationship between the channel conditions (input) and the optimal SVD-based beamforming strategies (output).

The input to the neural network can include features derived from the channel matrix H, which are singular values and angles of arrival in the proposed model. Let X represent the input matrix to the NN, with dimensions K × P and with P representing the number of input features. The neural network predicts the optimal beamforming parameters or vectors. Y denotes the output matrix from the neural network, with dimensions K × Q, where Q represents the number of output parameters or dimensions of the beamforming vectors.

The training dataset consists of pairs of input–output examples, where the input is derived from the channel conditions, and the output is the corresponding optimal beamforming parameters obtained through SVD-based hybrid beamforming. During training, the neural network minimizes a loss function, in our case the MSE, to optimize the predictions relative to the ground truth obtained from SVD-based hybrid beamforming.

In the next section, we present results of applying the above-mentioned AI methods (linear regression, random forest regression, and neural networks SVD enhancement) to reduce the pilot overhead size while aiming to maintain the BER and SE at acceptable levels.

3. Results

We next compare the performance of traditional pilot placement methods with ML-based approaches, including k-means clustering, linear regression, and random forest regression. Additionally, we explore the combination of SVD and machine learning for further performance improvements.

Simulation parameters are provided in

Table 1. Simulation setup.

Parameters	Values
Carrier frequency	28 × 109 Hz
Sampling frequency	100 × 106 Hz
Modulation technique	BPSK
Transmitter antennas	16
Receiver antennas	4
Number of streams	4
Number of symbols	1000
SNR	−10 dB to 30 dB

. Throughout the simulations, we compare the performance of each method, analyzing their advantages and disadvantages.

3.1. Classical Pilot Placement Technique

Figure 2 shows the pilot overhead key parameter when the pilot placement technique is applied. The results demonstrate a significant initial pilot overhead size of approximately 350 symbols at low signal-to-noise ratio (SNR) values. The pilot overhead ratio to overall symbols count is similar to the pilot overhead for downlink channel estimation in LTE-A standard with eight antennas, which exceeded 25% [23]. This overhead remains relatively constant as SNR increases until a threshold of 15 dB is reached. Beyond this point, the pilot overhead exhibits a marked decrease, reaching a minimum of around 63 symbols at an SNR of 30 dB.

Figure 3 illustrates the BER performance versus SNR for the proposed pilot placement technique. The BER achieved at −10 dB SNR is ${6\times 10}^{-3}$ and decreases to ${2\times 10}^{-3}$ at 30 dB SNR. While these results are comparable to those reported in [2,3], further optimizations are necessary to mitigate the impact of un-optimized pilot placement and imperfect channel estimation.

Figure 4 illustrates the system’s spectral efficiency (SE) as a function of signal-to-noise ratio (SNR) for the conventional pilot placement technique. While the SE increases with SNR, reaching a maximum of 12.5 bps/Hz at 30 dB SNR, this performance is suboptimal. To address this limitation, we propose using AI-based methods to optimize pilot placement and enhance system performance.

3.2. K-Means Clustering for Pilot Overhead Reduction

This section investigates the application of the k-means clustering algorithm (k-clustering) to optimize system performance by reducing pilot overhead. The effectiveness of k-clustering is evaluated across various cluster configurations (2, 4, 8, and 16 clusters). The impact of k-clustering on system parameters is analyzed, particularly focusing on pilot overhead reduction compared to the baseline scenario (classical pilot placement technique).

Figure 5 illustrates the impact of k-means clustering on pilot overhead reduction. By employing k-means clustering, a significant reduction in pilot overhead is observed, especially at low SNR values. The initial overhead of approximately 350 bits is reduced to around 63 bits, representing an impressive 82% reduction. This significant improvement demonstrates the effectiveness of k-means clustering in optimizing system performance under challenging conditions.

Figure 6 illustrates the BER performance of the hybrid beamforming system after applying k-means clustering for pilot placement optimization. The results demonstrate a significant improvement in BER, especially at higher SNR values. Notably, a configuration with 16 clusters yielded the lowest BER, achieving the best improvement at 30 dB SNR, aligning with the findings in [24].

Figure 7 illustrates the system’s spectral efficiency (SE) as a function of signal-to-noise ratio (SNR) after pilot placement optimization using k-means clustering. Compared to the baseline system (Figure 4), which achieved an SE of approximately 12.5 bps/Hz, the k-means-optimized system significantly improves SE to around 31 bps/Hz, representing a 2.5-fold increase. This result surpasses the clustering-based pilot allocation scheme in [7], which demonstrated a 17% reduction in pilot overhead, with an 8–14% SE improvement. In contrast, our proposed system offers a substantial 150% increase in SE and an 82% reduction in pilot overhead. It is also noted that the spectral efficiency reaches a near-optimal value at k = 4. This plateau can be attributed to the fact that increasing the number of clusters beyond this point does not significantly enhance pilot allocation efficiency. The diminishing returns indicate that the channel variations are effectively captured with fewer clusters, aligning with the observed pilot overhead reduction trends.

3.3. Linear Regression for Pilot Overhead Reduction

To further reduce pilot overhead while maintaining or improving spectral efficiency (SE) and bit error rate (BER), this study explored the application of supervised machine learning techniques. Initially, we investigated linear regression coupled with singular value decomposition (SVD) to optimize pilot placement. The training data for the model consisted of features such as the number of transmitter and receiver antennas, streams, pilot bits, data symbols, and channel capacity. The effectiveness of this method in reducing pilot overhead is presented in Figure 8.

Building upon the pilot overhead reduction achieved with k-clustering, the linear regression SVDHB system demonstrates comparable performance. Figure 9 illustrates the superior BER performance of the linear regression SVDHB system. The same improvement as in the case of k-clustering is observed, with a system accuracy of 88%.

Figure 10 shows the SE performance. Again, the performance is comparable to the k-clustering method using 16 clusters. The high accuracy of the linear regression model, as evidenced by the near-perfect match between actual and predicted SE, further validates its effectiveness.

3.4. Random Forest Regression for Pilot Overhead Reduction

In this section, we evaluate the performance of random forest regression SVDHB. The training model utilized features including the number of transmitter and receiver antennas, data streams, pilot bits, data symbols, and channel capacity.

As depicted in Figure 11, the random forest regression SVDHB system achieves pilot overhead reduction comparable to the linear regression.

Random forest regression outperforms k-clustering and linear regression in terms of BER performance. Figure 12 illustrates this improvement, with a significant 16% reduction in BER at −10 dB SNR compared to previous methods. The system’s robustness is further evident in its ability to maintain lower BER levels across a wide SNR range. These results highlight the potential of random forest regression for enhancing the reliability of wireless communication systems. It is also noted that the system accuracy reaches 99%, which is a major improvement over linear regression.

Figure 13 demonstrates that the proposed system, enhanced with random forest regression, maintains comparable spectral efficiency to the linear regression approach.

3.5. Random Forest Regression NNSVD for Pilot Overhead Reduction

This section presents the performance of the proposed random forest regression NNSVD (RFR-NNSVD) system, which combines SVD with a random forest regression neural network (NN) for optimal beamforming vector prediction. The proposed random forest regression–neural network-singular value decomposition (RFR-NNSVD) framework synergizes SVD-based hybrid beamforming with machine learning to enhance the performance of mmWave communication systems. SVD serves as a foundational step, enabling the decomposition of the channel matrix into analog and digital beamforming components that are mathematically aligned with the dominant eigenmodes of the channel. This ensures optimal hybrid beamforming, simplifying learning complexity, enhancing adaptation to channel conditions, and maximizing channel capacity and spectral efficiency while minimizing pilot overhead. A multi-layer perceptron (MLP regressor) is trained to predict the hybrid beamforming matrix by learning the mapping between phase information (input) and absolute values (target) derived from the SVD process, ensuring that the transmitted signals are optimally aligned with the channel conditions. Furthermore, random forest regressors are developed to model the relationship between system parameters, such as the number of antennas, data streams, pilot symbols, signal length, channel capacity, SNR, and key performance metrics, including pilot overhead, bit error rate, and spectral efficiency. By combining SVD’s mathematical precision for beamforming optimization with the predictive capabilities of machine learning, the proposed framework provides a robust mechanism to evaluate and enhance system performance under diverse SNR conditions. The results highlight the framework’s ability to achieve low pilot overhead, high spectral efficiency, and reliable signal reconstruction.

Figure 14 depicts that with RFR-NNSVD the system achieves a pilot overhead size close to the system without NN implementation (approximately 62.5 at low SNR and 62.38 at high SNR). On the other hand, the BER and SR are enhanced.

Figure 15 illustrates the significant BER reduction achieved by the RFR-NNSVD method compared to the original system and other ML-based approaches. At an SNR of –10 dB, the proposed method achieves a BER of ${1.05\times 10}^{-4}$ bits/symbol, about a tenfold improvement. Furthermore, the BER continues to decrease to ${1.1\times 10}^{-5}$ bits/symbol at 30 dB SNR. These results demonstrate the superior performance and robustness of the RFR-NNSVD system across a wide range of SNR conditions. Figure 16 demonstrates the improved spectral efficiency (SE) of RFR-NNSVD. The system achieves a maximum SE of 40 bps/Hz, representing a 25% increase compared to prior ML methods. This improvement is attributed to the system’s superior channel estimation capabilities and optimized beamforming vector adjustments facilitated by the RFR-NN architecture.

The simulation results throughout all configurations of this study show that the use of machine learning AI-driven unsupervised and supervised methods can yield a significant improvement in the pilot overhead reduction while maintaining the BER and SE of the system in some cases and even improving them in other cases, such as the NNSVD random forest regression.

Figure 17 presents a comparative analysis of the proposed systems in terms of pilot overhead, BER, and SE. The RFR-NNSVD system demonstrates superior performance across all metrics. It achieves the lowest pilot overhead of 62, a significant reduction from the original system’s 350. Additionally, RFR-NNSVD significantly outperforms other methods in terms of BER reduction, achieving a 250-fold improvement. While all systems exhibit improved SE, RFR-NNSVD further enhances SE to 40 bps/Hz, solidifying its position as the most effective system.

The comparison in Figure 18 among the four techniques [7,8,9] and RFR-NNSVD clearly shows the superior performance of the RFR-NNSVD method in terms of pilot overhead reduction, BER improvement, and SE. Regarding pilot overhead reduction, RFR-NNSVD achieves the highest reduction of 82.14%, slightly exceeding the 80% reduction of [9] while significantly outperforming ref. [7] at 17% and ref. [8] at 43.24%. In terms of SE improvement, RFR-NNSVD achieves an impressive 250% increase, far outpacing ref. [9] at 85%, ref. [8] at 43%, and ref. [7] at 14%. Moreover, RFR-NNSVD is the only method that demonstrates measurable BER improvement, reducing BER to ${5\times 10}^{-4}$ and showing an 82% reduction in BER from the original system, while the other techniques [7,8,9] did not take into consideration the BER study. This superiority is attributed to the integration of a neural network with SVD in RFR-NNSVD. The neural network effectively models the complex non-linear relationships of the hybrid beamforming matrix, while SVD ensures efficient matrix factorization, enhancing channel estimation and system performance. These attributes, combined with the robust regression capabilities of the RFR, leverage both supervised learning capabilities and optimized matrix factorization to enhance channel estimation, achieving significant reductions in pilot overhead and BER while maximizing spectral efficiency, making it the most effective technique in this comparison.

The results in

Table 2. Comparison of RFR-NNVSD parameters with those of all previous works.

	PO Reduction	BER Reduction	SE Improvement
RFR-NNSVD	82.14%	82%	250%
[7]	17%	NA	17%
[8]	43.24%	NA	43%
[9]	80%	NA	85%
[10]	NA	10%	20%
[11]	NA	15%	25%
[12]	NA	20%	35%

align with the findings illustrated in Figure 18, emphasizing the exceptional performance of the proposed RFR-NNSVD system. While the graph highlights the comparative improvements for systems that studied pilot overhead reduction as a main parameter, the table provides a broader perspective by incorporating additional references, showcasing that RFR-NNSVD outperforms all other techniques across the three key performance metrics, which are the percentages of PO reduction, BER reduction, and SE improvement. Notably, the table reiterates that RFR-NNSVD achieves the best pilot overhead reduction, marginally surpassing [9] and significantly outperforming the reductions reported in [7,8]. In terms of SE, the proposed system delivers improvements that are much greater than the referenced methods, reflecting its superior optimization capabilities. Furthermore, the inclusion of BER reduction, a metric neglected by the three prior studies that studied PO reduction, underscores the comprehensiveness of the RFR-NNSVD approach. Achieving an 82% BER reduction, RFR-NNSVD not only advances channel estimation accuracy but also addresses a critical aspect overlooked in the state-of-the-art techniques and overcomes the BER reduction in [10,11,12] by a wide margin. These results validate the significant enhancements introduced by integrating a neural network with SVD and leveraging RFR. This combination enables RFR-NNSVD to effectively model complex beamforming relationships, optimize matrix factorization, and achieve improvement in system performance, surpassing the limitations of previously proposed methods.

3.6. Random Forest Regression NNSVD with 64 Tx/16 Rx Antennas

In this section, we change the parameters of the simulation to study the effect of this change on the system, proving that the proposed system is valid for all possibilities through changing the antenna numbers and the modulation technique used. These changes were carried out with the RFR-NNSVD technique since it was shown to be the most effective pilot overhead reduction technique in our work. First, we changed the transmitter and receiver antenna numbers and studied the same parameters to observe the effect of this change. The baseline configuration of 16 transmitter (Tx) and 4 receiver (Rx) antennas was chosen for computational feasibility while maintaining a focus on pilot overhead reduction. To address scalability, an additional configuration with 64 Tx/16 Rx antennas was simulated. This setup aligns more closely with the requirements of m-MIMO systems and demonstrate the robustness of the proposed methods under varying antenna densities.

Figure 19 shows the pilot overhead after changing the number of antennas. It is clear that the pilot overhead size increased from 63 at −10 dB with Tx = 16 and Rx = 4 to 160 at the same SNR, and it decreases as SNR increases to reach 154 at 30 dB. Although there is an increase in pilot overhead size, this is the normal behavior of the system since as the number of antennas increases, the number of spatial channels that need estimation also increases. This typically requires more pilot symbols to estimate the channel accurately. And although the number of antennas are multiplied by 4, the pilot overhead size is less than multiplication by 4 from the previous result (160 is less than 63 × 4), leading to the conclusion that the proposed system optimizes the pilot overhead size by a ratio less than what is expected.

The BER of the system is shown in Figure 20. The results show that the BER for the increased number of antennas is enhanced, showing a decrease in BER at −10 dB from ${1.05\times 10}^{-4}$ in the previous antennas combination to around ${3\times 10}^{-5}$ at the same SNR level, reaching ${2\times 10}^{-5}$ at 30 dB. Also, the accuracy of the system is shown to be high at about 98% when comparing the predicted and the actual BER. This coincides with the theory that the increase in Tx and Rx antenna numbers leads to improved spatial diversity, reducing the likelihood of fading and improving beamforming gain since a system with more antennas can focus energy on the desired directions, leading to reduced interference and thus improving BER [25].

As for the SE in the modified antennas system shown in Figure 21, it is obvious that the system yielded the anticipated result, showing an increase of SE at 30 dB from 40 bps/Hz in the older antenna configuration to reach almost 70 bps/Hz in the increased antenna number configuration. This is due to the increased data rate transmission caused by increasing the antenna numbers and thereby significantly increasing the SE of the system. Of course, this increase in SE can be offset by the increase in pilot overhead, but as seen in Figure 19, the pilot overhead size did not increase linearly as the number of antennas increased, minimizing the effect of the pilot overhead increase and still leading to a significant increase in SE, which was discussed in [26].

3.7. Random Forest Regression NNSVD with QPSK Moduation

The next manipulated parameter in our simulation is the modulation technique. To enhance the realism of the simulation, we expanded the modulation schemes to include quadrature phase-shift keying (QPSK) instead of BPSK. This scheme reflects practical scenarios in mmWave communication systems, enabling us to analyze the impact of pilot overhead reduction across varying levels of modulation complexity.

Figure 22 shows a decrease in the pilot overhead of the system after applying QPSK, where size decreased from around 62 to around 53.7. This is expected due to the fact that BPSK transmits fewer bits per symbol, so it requires more symbols to carry the same amount of information.

Figure 23 depicts that with QPSK, the system gives a higher BER than BPSK, which is logical since BPSK uses only two symbols (0 and 1), making it more robust against noise and interference at lower SNR, while QPSK uses four symbols (00, 01, 10, and 11). While this increases the rate of data transmission, it also makes the system more susceptible to noise, as the symbols are closer in the constellation. It is clear that at a lower SNR of −10 dB, the BER was ${1.05\times 10}^{-4}$ using BPSK and became ${1.65\times 10}^{-3}$ at the same SNR.

As for the SE study using QPSK modulation instead of BPSK, the results are shown in Figure 24. It is clear that there is an increase in SE, reaching 55 bps/Hz at 30 dB compared to 40 bps/Hz using BPSK modulation.

The results obtained from varying the number of Tx and Rx antennas and from changing the modulation technique from BPSK to QPSK clearly shows that the system proposed in our work reflects practical scenarios in mmWave communication systems, enabling us to reduce the pilot overhead across varying levels of antenna numbers and modulation complexity and still reach realistic and improved results.

4. Conclusions

This study presents an AI-driven framework for reducing pilot overhead in 5G mmWave m-MIMO systems, integrating multiple machine learning techniques with hybrid beamforming. The proposed system offers a novel integration of k-clustering, linear regression, random forest regression, and neural network-enhanced singular value decomposition (NN-SVD) to optimize pilot placement and beamforming strategies. Unlike existing approaches, the application of k-clustering directly to pilot signals introduces a unique perspective in optimizing pilot signal allocation by minimizing redundancy without compromising channel estimation accuracy. Another novel approach is the combination of NN and SVD to optimize the beamforming matrix and their implementation in RFR to optimize the system already enhanced with RFR. Moreover, the study demonstrated scalability through simulations with different combinations of Tx and Rx antenna configurations, achieving an 82% reduction in pilot overhead, a 250% increase in spectral efficiency, and a tenfold improvement in bit error rate at low SNR conditions, all while achieving a system accuracy of 98%. By bridging traditional pilot management techniques with emerging AI-driven solutions, this research advances the understanding and application of AI in reducing pilot overhead while enhancing system performance in next-generation wireless networks. Future work will explore additional system parameters and alternative AI techniques to further optimize pilot overhead reduction and overall system efficiency.