Performance Evaluation of Single-Carrier and Orthogonal Frequency Divison Multiplexing-Based Autoencoders in Comparison with Low-Density Parity-Check Encoder

Abstract

Recently, the growing demands for ultra-high speed applications require more advanced and optimal data transmission techniques. Wireless autoencoders have gained significant attention since they provide global optimization of the transceiver structure. This article explores the application of autoencoders to enhance the performance of wireless communication systems. It provides the performance evaluation of the systems using single-carrier and OFDM-based autoencoders under the conditions of AWGN and fading channels. Then, in terms of the BLER metric, the wireless systems with autoencoders are compared with conventional systems using LDPC coding and quadrature amplitude modulation for various configurations. Simulation results indicate that for high-modulation orders (QAM-64 or QAM-256), communication systems employing autoencoders provide superior performance compared to systems using LDPC channel encoding in regions with a low signal-to-noise (SNR) ratio. Specifically, a gain of 1–2 dB in signal power is obtained for single-carrier autoencoders and 0.3–2 dB is obtained for OFDM-based autoencoders. Therefore, wireless communication systems utilizing autoencoders can be considered as a promising candidate for future wireless communication systems.

1. Introduction

In recent years, wireless communication networks in general, and mobile networks in particular, have undergone rapid development, becoming an essential part of our daily lives. They enable easy and convenient connectivity and information exchange. To meet the growing demands of users, researchers have been continuously exploring innovative methods to address the current challenges and improve the performance of these networks [1,2]. Currently, several approaches are being pursued to tackle these issues. The first approach is utilizing various channel coding techniques such as Turbo, Polar, and LDPC codes [3,4]. The use of these techniques can improve the reliability of data transmission. Secondly, efficient spectrum utilization can be provided by using different types of multicarrier modulation schemes such as orthogonal frequency division multiplexing (OFDM), filter-bank multicarrier (FBMC), generalized frequency sivision multiplexing (GFDM), and spectrally efficient frequency division multiplexing (SEFDM) [1,2]. However, these two approaches make the design of the transmitter and receiver much more complicated. Therefore, a promising and emerging direction of applying machine learning techniques into wireless communication systems becomes more and more popular. This direction introduces a new paradigm that holds great promise, not only due to its simplicity in design but also due to its expected capability to break through common limitations in communication systems, such as the Shannon limit. In conjunction with advancements in the field of deep learning, numerous studies have proposed the use of deep learning for end-to-end optimization of communication systems [5,6,7,8]. In addition, there are several advanced technologies recently proposed for use in 5G wireless networks, such as hybrid beamforming [9], rate splitting multiple access (RSMA) [10,11], and non-orthogonal multiple access (NOMA) [12]. These advanced techniques have the potential to revolutionize the operation of 5G networks, providing higher data rates, improved connectivity, and enhanced user experiences, making 5G technology even more powerful and versatile.

Among these approaches, autoencoders have gained significant attention [5,6]. In contrast to conventional communication systems, autoencoders enable global optimization of both the transmitter and receiver for any channel model, without being constrained by separately optimizing individual components such as channel coding, modulation, and channel equalization. The fundamental idea behind autoencoders is to learn compact representations of data by training a neural network to reconstruct its input at the output layer. In the context of communication systems, this concept is extended to optimize the entire transmission process, from the encoding of information at the transmitter to the decoding at the receiver, in an integrated and adaptive manner. By treating the entire communication system as a single neural network, autoencoders have the potential to overcome the limitations imposed by traditional modular approaches, where each component is optimized independently, often resulting in suboptimal performance. One of the significant advantages of using autoencoders in communication systems is their ability to capture complex dependencies and adapt to different channel conditions. Conventional communication systems often rely on handcrafted algorithms and mathematical models to deal with various impairments in the transmission channel. However, these models may not fully capture the intricate dynamics of real-world channels, leading to suboptimal performance. Autoencoders, on the other hand, have the inherent capability to learn and adapt to channel characteristics through training on large amounts of data. This adaptability enables them to effectively deal with channel variations and imperfections, resulting in improved performance and robustness.

Despite the great advantages of autoencoders, the number of studies evaluating the performance of autoencoders for specific wireless systems is still limited. Some of them are highlighted as follows. In [6], the authors presented the fundamental concepts of autoencoders and apply them to single-carrier modulation systems such as QPSK, and BPSK, in multipath channels. In [5], a combined scheme between autoencoders and OFDM modulation was proposed. This combination had improved the BLER performance by 1–2 dB depending on the constellation used. In [13], an autoencoder using convolutional neural networks (CNN) and the simultaneous perturbation stochastic approximation algorithm (SPSA) in place of backpropogation was proposed. This combination can potentially enhance the performance of the autoencoder by leveraging the strengths of both CNNs and SPSA. In addition, [14] proposed low-complexity autoencoder-based end-to-end learning of coded communications systems. The use of the proposed autoencoder helps reduce complexity in the design of transmitters and receivers without compromising system performance.

For high-speed wireless communication standards such as 4G LTE, 5G NR, and WiFi, the LDPC coding is chosen. However, the LDPC coding block and the modulation block are separately optimized. There is an idea of replacing the combination of LDPC coding and modulation scheme by an single autoencoder. It is unclear whether these autoencoders can take place of the state-of-the-art LDPC coding under the conditions of additive white Gaussian noise and various fading channels (especially in low-signal-to-noise-ratio scenarios). This study aims to answer the question by conducting the performance evaluation of autoencoders and LDPC coding in single-carrier and OFDM-based wireless communication systems. The main contributions of this paper are summarized as follows. For single-carrier systems, we present the performance evaluation of autoencoders compared with LDPC coding in terms of BLER metrics in the conditions of AWGN. For OFDM-based systems, the performance comparison of OFDM-based autoencoders is conducted in 5G NR fading channels (TDLA30-10). The experiments are conducted for various combinations of code rate, modulation order, and pilot arrangement. The performance comparison in this study provides useful information for industry engineers and standardization organizations developing future wireless technologies.

This paper is organized as follows. In Section 2, the structures of single-carrier autoencoders and OFDM-based autoencoders are presented. The performance evaluation of single-carrier autoencoders and OFDM-based autoencoders in different channel models are shown in Section 3. Finally, the conclusion of this study is given in Section 4.

2. Application of Autoencoders for Wireless Communications

2.1. Single-Carrier Autoencoders

A general wireless communication system consists of separated blocks, illustrated in Figure 1 [15]. Each block in the system performs a distinct and independent function, such as channel encoding/decoding, digital modulation/demodulation, channel estimation, and equalization. This enables the wireless communication system to operate efficiently and flexibly. Indeed, many advanced techniques are used for the performance of each block individually [16,17,18,19,20,21]. Among these blocks, the channel encoding/decoding blocks play a crucial role to ensure transmission reliability under AWGN noise and frequency fading channels. Various schemes of channel coding are proposed, aiming to enhance the error-correcting ability while keeping minimal implementation complexity: Hamming codes, Reed–Muller codes, LDPC codes, and Polar codes. The LDPC codes are now widely used in popular advanced wireless communication systems such as 4G LTE, WiFi, 5G NR…

However, one of the major challenges in this system is optimizing the performance of individual blocks, which may not necessarily result in optimal performance for the entire system. To solve this issue, the proposed approach of using deep learning to optimize the entire system is being considered and researched. Deep learning is expected to improve performance better than conventional wireless communication systems.

An autoencoder is an artificial neural network (ANN) model primarily used for unsupervised learning tasks, especially in the field of deep learning. As shown in Figure 2, the structure of an autoencoder is divided into an encoder and a decoder, which include the input layer, hidden layers, and output layer [6]. An autoencoder can have one or multiple hidden layers that serve the encoding function to generate data that capture the most fundamental attributes necessary to fully describe the input data. Then, the decoder reconstructs an approximation of the encoder to generate an output that closely resembles the input. Autoencoders are considered unsupervised learning techniques because they do not require labeled data for training. However, to be more precise, we can use a form of self-supervision by creating our own labels from the training data.

The primitive function of AE is to transform data into another signal form, so that the new signal can be restored back to the most similar to the original data. In this study, we take advantage of AE to convert bit streams into complex signals, so AE can be meaningfully representative of the whole physical layer in wireless communications. However, a model that is too large will be challenging to train well. We proposed to train an AE model that can replace the channel encoder and modulator on the principle of ensuring fairness in comparison. More details on wireless autoencoder idea can be found in [6].

The input bits to the autoencoder, denoted as vector x, are represented as a one-dimensional vector where the s-th element of the vector has a value of one and all other elements have a value of zero. That vector is called a one-hot vector, and is symbolized as ${\mathbf{1}}_s$. On the transmitter side, a neural network with multiple layers and a normalization layer ensures energy or power constraints on x. The receiver side also includes a similar neural network with corresponding layers as the transmitter. The final layer of the system utilizes the softmax activation function, producing an output vector $\mathbf{p}\in (0,1)$, representing the probabilities of all possible characters. The decoded symbol s-th corresponds to the index of the element in ${\mathbf{p}}_i$ (where i = 1, 2, …, M) with the highest probability. Afterward, the autoencoder can be trained end-to-end by using stochastic gradient descent (SGD) on the set of all possible symbols $\mathbf{s}\in \mathbb{M}$, utilizing the cross-entropy loss function to appropriately classify the difference between ${\mathbf{1}}_s$ and p.

Regarding the training process, AE is trained by the SGD (stochastic gradient descent) algorithm combined with Adam. In it, each pair of training data consists of a symbol converted to a one-hot vector, as well as its probabilistic output after passing through the encoder, noise, and decoder. The loss function used is the following cross-entropy: Using the common backpropagation mechanism, the gradients are calculated, and the network layer weights at the encoder and decoder are updated through each loop in order to optimize the loss function. The visual results of the training process are shown in the next sections.

2.2. OFDM-Based Autoencoders

Recently, the OFDM technique has been an appropriate choice for wireless communications (such as 4G LTE and 5G NR mobile communications), since it is resilient in frequency-selective fading conditions and provides reliable synchronization ability. For that reason, the study in [5] proposed a system utilizing OFDM modulation based on autoencoders. The block diagram of the system is illustrated in Figure 3.

In contrast to the scheme of autoencoders for single-carrier modulation, a discrete Fourier transform (DFT) with a length of $N_{FFT}$ is applied to a set of symbols from the output of the autoencoder the length of $N_{FFT}$, resulting in $N_{FFT}$ equivalent independent subchannels, where each symbol from the autoencoder is assigned to each subcarrier. To avoid intersymbol interference (ISI), a cyclic prefix (CP) of length $N_{CP}$ is added, meaning that independent $N_{FFT}$-encoded symbols (in the frequency domain) form a single OFDM symbol (in the time domain) with a total length of $N_{FFT}$ + $N_{CP}$ samples. Therefore, a sequence consisting of ($N_{FFT}$ + $N_{CP}$) $n/2$ complex-valued symbols is transmitted through the transmission channel.

A structure of autoencoders in the OFDM communication system is presented in Figure 4. In this system, the processed symbols are mapped to OFDM subcarriers after passing through a normalization layer and transmitted through the transmission channel. Two fully connected layers map k (in the form of a one-dimensional network with length M) to n real numbers. After the normalization layer, the OFDM modulation layer maps these n real numbers to $n/2$ complex symbols and assigns each symbol to a subcarrier. To ensure that the OFDM modulation layer outputs the complete set of OFDM symbols, the minimum input length is $N_{FFT}$. Thus, the input to the neural network is a sequence of one-hot values with a size of $M\times N_{FFT}$.

3. Performance Evaluation

3.1. Simulation Framework

This section presents the simulation framework for obtaining the BLER metrics for system comparison. The simulation program was implemented in MATLAB by the Monte Carlo method. In this study, the block error ratio (BLER) is calculated as the sum of differences between transmitted and received data symbols divided by the product of the number of transmitted blocks and the number of bits per symbol. The data stream is transmitted in blocks, and the block length is determined by the system parameters.

In this study, we propose a method to fairly compare the conventional system and the system with autoencoders considering various code rate and modulation orders. We denote the number of input bits and output real numbers of an autoencoder as $k_a$ and $n_a$, respectively. As a consequence, we obtain $n_a/2$ complex numbers to send to further steps for $k_a$ input bits. For a conventional system, the LDPC encoder generates $n_{LDPC}$ bits for $k_{LDPC}$ input bits. Then, a modulator converts $n_{LDPC}$ bits into $n_{LDPC}/lo{g}_2\left(M_{QAM}\right)$ complex numbers to send to further steps (where $M_{QAM}$ is the modulation order of the modulator). To fairly compare the performance of the two systems, it is necessary to ensure that:

$$\begin{array}{c}\hfill \frac{k_a}{n_a/2}=\frac{k_{LDPC}}{n_{LDPC}/lo{g}_2\left(M_{QAM}\right)}.\end{array}$$

(1)

From Equation (1), we present

Table 1. Configurations of autoencoders ($n_a$, $k_a$) compared with conventional QPSK/QAM16 without LDPC codes.

Autoencoder		LDPC Codes		Description
$k_a$	$n_a$	$k_{LDPC}$	$n_{LDPC}$
2	2	-	-	Autoencoder (2, 2) compared with QPSK without LDPC coding (Figure 7).
4	2	-	-	Autoencoder (4, 2) compared with QAM16 without LDPC coding (Figure 7).

,

Table 2. Configurations of autoencoders ($n_a$, $k_a$), LDPC code rate, and QAM16 for comparison.

Autoencoder		LDPC Codes		Description
$k_a$	$n_a$	$k_{LDPC}$	$n_{LDPC}$
4	3	43,200	64,800	Autoencoder (4, 3) compared with QAM16 using LDPC code rate 2/3 (Figure 8a).
4	4	32,400	64,800	Autoencoder (4, 4) compared with QAM16 using LDPC code rate 1/2 (Figure 8a).
5	4	25,920	64,800	Autoencoder (5, 4) compared with QAM16 using LDPC code rate 2/5 (Figure 8a).
6	4	21,600	64,800	Autoencoder (6, 4) compared with QAM16 using LDPC code rate 1/3 (Figure 8a).
8	4	16,200	64,800	Autoencoder (8, 4) compared with QAM16 using LDPC code rate 1/4 (Figure 8a).

,

Table 3. Configurations of autoencoders ($n_a$, $k_a$), LDPC code rate, and QAM64 for comparison.

Autoencoder		LDPC Codes		Description
$k_a$	$n_a$	$k_{LDPC}$	$n_{LDPC}$
3	6	43,200	64,800	Autoencoder (3, 6) compared with QAM64 using LDPC code rate 2/3 (Figure 8b).
4	6	32,400	64,800	Autoencoder (4, 6) compared with QAM64 using LDPC code rate 1/2 (Figure 8b).
5	6	25,920	64,800	Autoencoder (5, 6) compared with QAM64 using LDPC code rate 2/5 (Figure 8b).
6	6	21,600	64,800	Autoencoder (6, 6) compared with QAM64 using LDPC code rate 1/3 (Figure 8b).
8	6	16,200	64,800	Autoencoder (8, 6) compared with QAM64 using LDPC code rate 1/4 (Figure 8b).

and

Table 4. Configurations of autoencoders ($n_a$, $k_a$), LDPC code rate, and QAM256 for comparison.

Autoencoder		LDPC Codes		Description
$k_a$	$n_a$	$k_{LDPC}$	$n_{LDPC}$
3	8	43,200	64,800	Autoencoder (3, 8) compared with QAM256 using LDPC code rate 2/3 (Figure 8c).
4	8	32,400	64,800	Autoencoder (4, 8) compared with QAM256 using LDPC code rate 1/2 (Figure 8c).
5	8	25,920	64,800	Autoencoder (5, 8) compared with QAM256 using LDPC code rate 2/5 (Figure 8c).
6	8	21,600	64,800	Autoencoder (6, 8) compared with QAM256 using LDPC code rate 1/3 (Figure 8c).
8	8	16,200	64,800	Autoencoder (8, 8) compared with QAM256 using LDPC code rate 1/4 (Figure 8c).

for various configurations of autoencoders ($n_a$, $k_a$), LDPC code rate, and modulation order $M_{QAM}$. We also present the validation accuracy and loss of autoencoder (2, 4) in Figure 5 and the received constellations in Figure 6.

3.2. Performance of Single-Carrier Autoencoders

The autoencoder in the single-carrier communication system consists of nine layers, and is distributed as shown in Figure 2. In the simulation, two fully connected layers are added for both the encoder (transmitter) and decoder (receiver) in order to achieve the best results with the minimum system complexity. The feature input layer receives a one-hot vector of length M, where M is the number of symbols generated from k input bits, and M = $2^k$. Next is the ReLU layer (ReLU Layer). The second fully connected layer has M input parameters and n output parameters, followed by the normalization layer. There are two normalization methods used in this layer: one is the normalization based on the average power of the signal, and the other is the normalization based on the energy of the signal. The encoding layers are followed by the channel layer. The first layer in the decoding part is a fully connected layer with n input parameters and M output parameters, followed by the ReLU layer. The second fully connected layer has M input parameters and M output parameters. Next is the softmax layer, which provides probabilities for each of the M symbols. The classification layer then outputs the most probable transmitted symbol ranging from 0 to $M-1$. The simulation parameters for single-carrier autoencoders are provided in

Table 5. Parameters of the simulation for single-carrier configuration.

Parameter	Value
Optimization algorithm	SGD combined with Adam
Initial learning rate	0.08
Maximum epochs	10
Minibatch size	100 M
Learning rate drop factor	0.1
LDPC code rate	2/3, 1/2, 2/5, 1/3, 1/4
Number of frames	200

.

Validation accuracy and loss during the training process for autoencoder (2, 4) with two normalization methods (average signal power and energy normalization) are presented in Figure 5a,b. From these figures, it can be observed that the validation accuracy and validation loss during the training process, with a sufficiently large training data set, approximate values above 90% and below 0.5%, respectively. This indicates that the results obtained from the training process of autoencoders with the mentioned parameters are reliable enough to evaluate the performance of the system utilizing autoencoders.

Regarding to the obtained signal constellations, from Figure 6a,b, it can be observed that different signal normalization methods lead to distinct received constellations after the learning process. When employing the energy-based signal normalization method (Figure 6a), the obtained constellation exhibits similarities to a conventional PSK-modulated signal. On the other hand, when using the average power-based signal normalization method, the resulting constellation resembles the characteristics of a QAM-modulated signal. This can be explained by the fact that, for the energy-based normalization method, the signal’s amplitude is normalized, causing the constellation points to be distributed on a circle with a radius of 1.

Figure 7 compares the BLER performance of systems utilizing autoencoders (2, 2) and (2, 4) with conventional single-carrier modulation schemes. This figure corresponds to the configurations in Table 1. From Figure 7, it is evident that the systems employing autoencoders achieve comparable BLER performance to those using conventional modulation schemes. This implies that the performance of both systems is equivalent in terms of BLER. This occurs because the received constellations after the training process of the autoencoder also resemble the constellations of systems employing traditional modulation schemes, such as QPSK or 16-QAM.

It is interesting to examine the simulation results when the LDPC codes are applied. Figure 8 compares the BLER performance of systems employing autoencoders and various modulation schemes combined with LDPC encoding. From Figure 8a, which corresponds to the configurations in Table 2, we can observe that the system employing 16-QAM modulation combined with LDPC encoding, still provides better performance compared to the system using an autoencoder. However, according to Figure 8b, which corresponds to the configurations in Table 3, the system using an autoencoder yields slightly better BLER performance compared to the system utilizing 64-QAM modulation combined with LDPC encoding. Particularly at LDPC encoding rates of 1/3 and 1/4, based on the BLER metric, the system employing an autoencoder with the corresponding parameter set offers a gain of 2 to 2.5 dB at BLER = 0.1. Moreover, for the case of 256-QAM modulation, the system using an autoencoder exhibits superior performance compared to the conventional system, as seen in Figure 8c, which corresponds to the configurations in Table 4. From Figure 8c, it can be observed that at BLER = 0.1, the systems employing autoencoders consistently provide a better performance, with a gain ranging from 1.5 to 3 dB. However, when the BLER = 0.05, the combined scheme of conventional modulation combined with LDPC encoding yields better results in the case of 16-QAM and 64-QAM. When compared to the system using 256-QAM modulation, the system utilizing the autoencoder still achieves better performance (except for the 2/3 encoding rate). A summary of the performance of the systems is visually demonstrated in Figure 9 (for the case of BLER = 0.05).

3.3. Performance of OFDM-Based Autoencoders

In this section, the scheme of the autoencoder in the OFDM modulation system is compared to the conventional OFDM modulation system. The diagram scheme and simulation parameters are presented in Figure 3 and

Table 6. Parameters of the simulation.

Parameter	Value
Optimization algorithm	SGD combined with Adam
Initial learning rate	0.02
Maximum epochs	10
Minibatch size	100 M
Learning rate drop factor	0.1
LDPC code rate	2/3, 1/2, 2/5
$N_{FFT}$	256
Channel estimation method	Ideal/using pilots
Pilot spacing	2/8/16/64
Channel model	TDLA-30/10
Interpolation method	Spline

, respectively.

From Figure 10, in the TDL-A30/10 channel model, we observe that when the system utilizes the combined autoencoders with OFDM and the system employs OFDM + LDPC with a coding rate of 2/5, the system using OFDM combined with LDPC encoding yields better performance than the system using autoencoders. However, in the remaining cases, the performance of the system using combined autoencoders with OFDM outperforms the system using OFDM combined with LDPC by a margin of 0.3 to 2 dB at a BLER value of 0.1.

In practice, the transmitter side sends pilot signals to the receiver to estimate the channel. Subsequently, the receiver side performs channel estimation at the positions where pilot signals are placed and then interpolates to obtain the evaluation results. Depending on the channel models, the number and placement of pilot signals are considered and adjusted accordingly. Within the scope of this article, we will evaluate the performance of systems using combined autoencoders with OFDM with different numbers of pilot signals.

Figure 11 represents the performance of the system using autoencoders combined with OFDM in the TDL-A30/10 channel with different distances between pilot signals. From Figure 11, it can be observed that when the distance between pilot signals is 2, the performance of the system using pilot signals combined with the spline interpolation method to estimate the transmission channel achieves performance almost equivalent to the ideal case. This can be explained by the fact that the frequency response of the estimated and evaluated transmission channel closely resembles the ideal frequency response. However, adding pilot signals to the transmission frame reduces the number of useful signals transmitted. As the pilot spacing increases from 2 to 64, indicating a decrease in the number of pilots, the performance of the aforementioned systems also decreases because the frequency response of the evaluated channel deviates increasingly from the ideal frequency response. Additionally, the validation accuracy also decreases due to the deviation between the ideal frequency response of the channel and the frequency response obtained during the channel estimation process using pilot signals in the training process. As can be seen from Figure 12, when increasing the pilot spacing from 2 to 64, the validation accuracy of the training process decreases from 98.12% to 78.25%.

4. Conclusions

In this study, we developed and evaluated the performance of wireless communication systems using single-carrier and OFDM-based autoencoders in a framework of the 5G NR system. The BLER performance of the system was considered compared with a conventional system with various modulation orders and LDPC code rates.

The simulation results show that the system with autoencoders provides superior performance compared to systems using LDPC channel encoding in low-signal-to-noise-ratio regions and high-modulation orders (64QAM and 256QAM). Specifically, a gain of 1–2 dB in signal power is obtained for single-carrier autoencoders and 0.3–2 dB is obtained for OFDM-based autoencoders. Since the complexity of implementing autoencoder schemes in communication systems is lower compared to designing LDPC encoding and decoding schemes, autoencoder is a promising candidate for enhancing wireless communications systems.

For further research directions, more optimal architecture of autoencoders should be investigated. Moreover, the impact of timing synchronization, frequency errors, and number quantization noise should be addressed.