Probabilistic Shaping Based on Single-Layer LUT Combined with RBFNN Nonlinear Equalization in a Photonic Terahertz OFDM System

Abstract

We propose a probabilistic shaping (PS) scheme based on a single-layer lookup table (LUT) that employs only one LUT for symbol mapping while achieving favorable system performance. This scheme reduces the average power of the signal by adjusting the symbol distribution using a specialized LUT architecture and a flexible shaping proportion. The simulation results indicate that the proposed PS scheme delivers performance comparable to that of the conventional constant-composition distribution-matching-based probabilistic shaping (CCDM-PS) algorithm. Specifically, it reduces the bit error rate (BER) from 1.2376 $\times {10}^{-4}$ to 6.3256 $\times {10}^{-5}$, corresponding to a 48.89% improvement. The radial basis function neural network (RBFNN) effectively compensates for nonlinear distortions and further enhances transmission performance due to its simple architecture and strong capacity for nonlinear learning. In this work, we combine lookup-table-based probabilistic shaping (LUT-PS) with RBFNN-based nonlinear equalization for the first time, completing the transmission of 16-QAM OFDM signals over a photonic terahertz-over-fiber system operating at 400 GHz. Simulation results show that the proposed approach reduces the BER by 81.45% and achieves a maximum Q-factor improvement of up to 23 dB.

1. Introduction

With the rapid advancement of virtual reality, big data, and ultra-high-definition-video services, meeting the demand for high-speed, large-capacity, and broadband services has emerged as a key objective in the development of communication networks [1,2]. The use of higher carrier frequencies is an effective approach to increasing bandwidth and enhancing transmission capacity and thus offers a promising solution for high-speed communication [3]. The terahertz band, which falls in the range 0.1–10 THz, has significant advantages in alleviating shortages of spectrum resources and capacity bottlenecks in communication systems owing to its wide bandwidth and capacity for ultra-high-speed data-transmission [4]. Moreover, terahertz signals exhibit good directivity and anti-interference characteristics, making them well suited for use in short-range, high-density wireless communication. As a result, the terahertz band is widely regarded as a core technology that can be used to promote the development of future 6G mobile communication and one that shows great potential for use in ultra-high-frequency-band communication, ultra-dense networks, and emerging application scenarios [5,6].

However, a terahertz signal generated in electronic constrained by limitations such as narrow frequency bandwidth, significant signal loss, and low transmission rate, constraining its application in long-distance transmission [7]. As optical-fiber communication technology has matured, the photon-assisted generation of terahertz signals has become a more efficient option. There are two common methods used for the generation of photonic terahertz signals. One approach employs an optical frequency comb (OFC) to generate a highly stable terahertz carrier [8,9]. The other utilizes two narrow linewidth lasers with fixed frequencies, where the terahertz signal is produced via the heterodyne beating process in the photodiode. This method enables the generation of a terahertz signal with a wide tuning range and extremely high bandwidth, which give it the potential to meet the demand for high-speed and large-capacity transmission in future 6G and higher-generation communications systems [10,11]. In recent years, photon-assisted technology has opened up a new path for high-speed terahertz communication. Nevertheless, advanced signal-modulation technology is still necessary to achieve efficient and reliable data transmission. Orthogonal frequency division multiplexing (OFDM), a well-established multi-carrier modulation technology, is widely used in optical communication [12,13]. OFDM converts high-speed serial data streams into orthogonal low-speed parallel sub-data streams, effectively reducing intersymbol interference (ISI) and improving signal stability and transmission efficiency. Due to these advantages, OFDM has attracted considerable research attention and has been widely studied in recent years [14,15].

To further enhance the transmission performance of OFDM signals, probabilistic shaping (PS) technology has garnered increasing attention in both research and practical applications [16,17,18]. PS adjusts the probability distribution of constellation points by transforming them from a uniform distribution to one that approximates a Gaussian distribution. This technique increases the probability of selecting constellation points closer to the center while reducing the probability of selecting points farther from the center, thereby effectively reducing the signal’s average power. Research on PS algorithms has attracted widespread attention and has been progressing rapidly in recent years. In [19], a PS scheme based on trellis-coded modulation (TCM) is presented; this scheme enhances the efficiency of PS signals within bandwidth-constrained intensity-modulation and direct-detection (IMDD) systems. A hierarchical distribution matcher (HiDM) scheme with parallel input and output is proposed in [20]; this scheme simplifies the operation compared to constant component distribution matchers. A novel Hi-DM method is proposed in [21]; this scheme combines multiple short DMs into a hierarchical structure to create a longer one. This approach effectively achieves a balance between the advantage of long DMs and the simplicity of short DMs. However, these two methods still face limitations in terms of high implementation complexity and insufficient flexibility due to their multilayer design. Inspired by existing Hi-DM methods, this work aims to investigate a simpler shaping structure that employs only a single-layer lookup table as a distribution matcher and offers advantages in terms of implementation simplicity.

Nonlinear distortions caused by the nonlinear effects of optical fibers and optoelectronic devices represent another critical factor influencing transmission performance. Neural network-based nonlinear equalization algorithms, owing to their superior adaptive capability and nonlinear approximation performance, offer a novel approach to addressing complex nonlinear distortion problems [22,23]. A variety of neural network-based schemes have been proposed. These include a multiple-input multiple-output (MIMO) deep neural network [24], complex-valued convolutional neural networks (CV-CNNs) [25] and a pruning I/Q-joint deep neural network [26]. However, radial basis function neural networks (RBFNNs) have received relatively little attention in existing research. Given their fast training speed, simple structure, and strong localized response characteristics, RBFNNs are particularly well-suited for nonlinear equalization tasks. Therefore, further investigation is warranted to explore their potential and effectiveness in this context.

In [27], the authors completed the simulation-based verification of the joint optimization of probabilistic shaping and RBFNN nonlinear equalization in a photonic terahertz OFDM system, achieving a 61.9% reduction in bit error rate (BER). The main contributions of this paper are as follows:

• We present a PS scheme based on a single-layer lookup table (LUT-PS) that adjusts the symbol distribution by using a specific LUT architecture and a flexible shaping proportion to realize probabilistic shaping.
• We propose a joint scheme combining LUT-PS and RBFNN equalization to further mitigate nonlinear distortion.
• We demonstrate the transmission of 16-QAM OFDM signals over a photonic terahertz-over-fiber system at 400 GHz.

Section 2 explains the principle of LUT-PS and RBFNN nonlinear equalization. Section 3 describes in detail the setting of the simulation. Section 4 provides a detailed presentation and discussion of the simulation results. To conclude, we summarize the results of the above-described work in Section 5.

2. Principle

2.1. PS Based on Single-Layer LUT

We propose the LUT-PS scheme that employs a single-layer LUT for symbol mapping in both QAM modulation and demodulation. This scheme shapes the probability distribution into a quasi-Gaussian form. The schematic diagram of the proposed LUT-PS is shown in Figure 1. The operation principle of the LUT-PS involves mapping a uniform pseudo-random binary sequence (PRBS) tx_bits according to a specific regulation, using a fixed LUT and a flexible shaping proportion $\mu$, which describes the degree of non-uniformity in the symbol probability distribution. A flag signal is generated to record the mapping state. After mapping, the data are modulated into 16QAM symbols, and this step is followed by OFDM modulation. During PS decoding, the flag generated in the encoding process serves as a reference to allow accurate recovery of the original data using the inverse LUT mapping. This work is based entirely on a simulation framework and was conducted with the aim of evaluating the performance gain of the proposed LUT-PS scheme. In the simulation, the flag information is directly shared between the transmitter and receiver without being transmitted over the channel. This allows us to focus on assessing the performance of the probabilistic shaping itself without the results being affected by additional channel impairments. In practical communication systems, the flag can be transmitted by embedding it into the frame structure or through joint encoding with the data, which will be an important direction for our future work.

Taking the 16QAM signal as an example, the contents of the LUT are shown in

Table 1. The contents of the LUT.

Input	$\mathit{E}\left[\right|\mathit{X}{|}^2]$	Output	$\mathit{E}\left[\right|\mathit{Y}{|}^2]$
0101	2	0000	18
0111	2	0010	18
1101	2	1000	18
1111	2	1010	18
0001	10	0001	10
0011	10	0011	10
0100	10	0100	10
0110	10	0110	10
1001	10	1001	10
1011	10	1011	10
1100	10	1100	10
1110	10	1110	10
0000	18	0101	2
0010	18	0111	2
1000	18	1101	2
1010	18	1111	2

and include the input, output, and symbol energy. Each set of 4 bits is treated as the input to the LUT, resulting in 16 possible combinations. The energy for each input is calculated and arranged in ascending order, while the corresponding outputs are arranged in descending order of energy. The mapping process is governed by a comparison between the input energy $E\left[\right|X{|}^2]$ and the output energy $E\left[\right|Y{|}^2]$. If $E\left[\right|X{|}^2]$≤$E\left[\right|Y{|}^2]$, the input symbol remains unchanged. If ${E\left[\right|X|}^2{]>E[\left|Y\right|}^2]$, the input is mapped to the corresponding output value based on the lookup table (LUT). This approach effectively maps high-energy constellation points to low-energy ones, thereby reducing the average power of the signal. However, relying solely on the LUT causes all outer constellation points to be mapped onto the inner ones, resulting in a limited and inflexible shaping effect. To overcome this limitation, we introduce a shaping ratio parameter $\mu (\mu =1/i)$, which controls the proportion of symbols subjected to shaping. Specifically, it indicates that one out of every i symbols will remain unshaped. A smaller value of $\mu$ corresponds to a greater shaping depth, resulting in constellation points being more concentrated in the inner rings. This reduces the probability of symbol errors and enhances overall system performance. In this work, the value of the shaping parameter $\mu$ is selected based on simulation results. Specifically, we manually adjust the value of $\mu$ to observe its impact on system-performance metrics such as BER. As shown in

Table 2. The impact of the parameter $\mu$ on BER performance.

μ	1/11	1/9	1/7	1/5	1/3	1
BER	3.3003 $\times {10}^{-5}$	3.3003 $\times {10}^{-5}$	4.1254 $\times {10}^{-5}$	3.3003 $\times {10}^{-5}$	4.4004 $\times {10}^{-5}$	6.6007 $\times {10}^{-5}$

, a smaller $\mu$ generally leads to a lower BER, although the relationship is not strictly monotonic. A possible explanation is that when $\mu$ becomes too small, the symbol probability distribution becomes highly skewed—some symbols are transmitted far more frequently, while others are rarely used. This imbalance makes the receiver more sensitive to noise affecting the rarely occurring symbols, thereby resulting in an increase in BER. However, the parameter $\mu$ is essentially designed to flexibly adjust the shaping depth rather than serve as a direct performance-optimization variable. This flexibility offers the potential to adapt to different channel conditions or system requirements, which may be valuable in practical implementations.

Here, we define a flag signal to indicate the mapping state of each symbol. A flag value of 0 indicates an unshaped symbol, while a value of 1 represents a shaped symbol. This flag signal plays a key role in the probability decoding at the receiver. When the flag is 0, the symbol remains unchanged. When the flag is 1, the input and output energy must be evaluated. Notably, the LUT is used in reverse in this case, meaning that in the invLUT, the ‘output’ corresponds to the input for the current step, while the ‘input’ represents the output. When ${E\left[\right|Y|}^2{]<E[\left|X\right|}^2]$, the low-energy symbol is converted back into the corresponding high-energy counterpart.

Similarly, the LUT–PS method can be extended to higher-order constellations, such as 64QAM. The main idea is to divide all constellation points into three layers based on their symbol energy levels, then map the outermost layer to the innermost layer while keeping the middle layer unchanged. Figure 2a shows the mapping rules of the second quadrant of the 64QAM constellation diagram, and the same principle applies to the other three quadrants. It results in the corresponding probabilistic distribution for 64QAM, as illustrated in Figure 2b.

2.2. RBFNN Nonlinear Equalization

Neural networks are utilized to process complex input data and make corresponding predictions. They consist of numerous neurons and the connections between them. Each neuron receives input from the neurons in the previous layer, which is weighted and then combined with a bias term. The output is subsequently computed through a specified activation function, including the sigmoid function, the hyperbolic tangent (tanh) function, and the rectified linear unit (ReLU) function. The input data undergoes forward propagation, passing layer-by-layer through the neural network to generate the output. Subsequently, optimization techniques, including gradient descent, are employed to optimize the weights and biases of the network in order to minimize the error.

RBFNN is a form of feedforward neural network with a three-layer topology. Figure 3 illustrates the construction of the RBFNN. The input layer consists of nodes that receive the input signal and serve solely as conduits for data transmission without making any transformations to the input information. The second layer, known as the hidden layer, completes a spatial mapping transformation for the input data, with the number of nodes determined based on specific requirements. The activation function in the hidden layer neurons is RBF, typically a Gaussian kernel function, which depends only on the Euclidean distance between the input data and the center point. The corresponding formula is as follows:

$${\varphi }_i\left(x\right)=exp\left(-{\displaystyle \frac{\parallel x-c_i{\parallel }^2}{2{\sigma }_i^2}}\right)$$

(1)

where x is the input vector, $c_i$ is the ith center point of the RBF, and ${\sigma }_i$ is the width parameter of this RBF, which controls the sphere of action of the function. The third layer is the output layer, which employs a linear activation function. It computes the weighted sum from the outputs of the neurons in the hidden layer and generates the neural network’s final output. The expression is as follows:

$$y_i=\sum _{j=1}^N{\omega }_{jk}{\varphi }_i\left(x\right)$$

(2)

where ${\omega }_{jk}$ represents the weight between the hidden layer and the output layer and N signifies the quantity of hidden layer nodes.

A complete RBFNN operates in two stages: the training stage and the testing stage. During the training stage, the network learns the nonlinear functional relationship between the input and the target by training on a large dataset without requiring prior information and subsequently generates the network model. In the testing stage, the trained network model is used to make predictions on the entire dataset. To effectively train the RBFNN, the Levenberg–Marquardt algorithm is employed. This is an iterative optimization technique commonly used to minimize the error function. This algorithm combines the advantages of gradient descent and Newton’s method, demonstrating excellent performance in solving small-scale nonlinear problems. The specific training process is as follows: In this work, 40% of the received data is randomly selected as the training set, and an additional 20% is used as the testing set. The corresponding original data serve as the target set. The complex-valued signal is separated into its real and imaginary parts, which are fed independently into the RBFNN-based nonlinear equalizer. Based on the constructed network model, the output is computed and compared with the target values to evaluate the prediction error. If the error is below the predefined threshold, the training is terminated; otherwise, the algorithm checks whether the maximum number of iterations has been reached. If so, training stops; if not, the parameters are further optimized until convergence is achieved. The number of neurons is determined through parameter tuning, and a value that yields favorable training performance is selected. As shown in Figure 4, the training and testing results of the proposed RBFNN indicate fast convergence, achieving a low mean squared error within only seven iterations. Furthermore, the predicted values closely match the true values, demonstrating the effectiveness of the network’s convergence.

3. Simulation Setup

The simulation setup for 16-QAM OFDM signals over a photonic terahertz-over-fiber system at 400 GHz is given in Figure 5.

At the transmitting end, two external cavity lasers (ECLs) with 100 kHz linewidth, operating at 1552 nm and 1555 nm, respectively, generate two continuous waves at frequencies of 192.9 THz and 193.3 THz, respectively. A probability-shaped baseband OFDM signal using the LUT-PS technique is generated with the parameter $\mu =1/9$, resulting in a signal entropy of 3.66 bits/symbol. A total of 65,536 OFDM symbols modulated with 100 data subcarriers using a 16-QAM format are produced by a 128-point IFFT module. In order to compensate for the intersymbol interference (ISI), a cyclic prefix of length 16 was added. The real and imaginary parts of the baseband OFDM signal are used to drive the IQ modulator, which modulates the optical carrier provided by ECL1 operating at 1552 nm with an output power of 11.8 dBm. To ensure the signal achieves sufficient intensity, it is processed through a pair of electrical amplifiers (EA) a gain of 25 dB prior to being loaded into the I/Q modulator. Then, the optical signal from ECL2, operating at 1555 nm with an output power of 9.5 dBm, is coupled with the output of the IQ modulator using an optical coupler (OC). After that, the PS-16QAM OFDM signal is delivered to the erbium-doped fiber amplifiers (EDFA) across a 10-km SSMF fiber link. Subsequently, an attenuator (ATT) is used to control the input optical power of a uni-traveling-carrier photodiode (UTC-PD), converting the optical signal into a 400 GHz electrical signal through heterodyne beating.

At the receiving end, the terahertz signal is downconverted to the intermediate frequency (IF) domain after being mixed with the signal generated by the local oscillators (LO). The signal is then captured and stored by a digital storage oscilloscope (DSO) for subsequent digital signal processing. In digital signal processing, we use RBFNN to compensate for nonlinear distortion and demap the shaped signal back to a uniform distribution. To strengthen the network’s ability to generalize, a term that refers to the ability of the network to maintain good predictive performance when processing unseen data, we set the number of neurons in the hidden layer to 32 after iterative adjustment, with training conducted over 2000 iterations. A learning rate of 0.1 was used to guarantee that the weight updates were neither too fast nor too slow.

4. Analysis of Results

In this paper, we discuss the bit error rate (BER) and Q-factor performance of LUT-PS and the conventional constant composition distribution matching-based probabilistic shaping (CCDM-PS) scheme at an information entropy of 3.66 bits/symbol. The Q-factor is commonly used as a metric for evaluating system performance and serves as an effective indicator of signal quality. The Q-factor used in this paper is defined based on the error between the recovered received signal and the ideal transmitted signal. Specifically, Q is calculated as follows:

$$Q=-20·{log}_{10}\left({\displaystyle \frac{\sqrt{\frac{1}{N}{\sum }_{i=1}^N{|a_i-b_i|}^2}}{P_{ave}}}\right)$$

(3)

where $a_i$ represents the normalized received symbols, $b_i$ denotes the transmitted symbols, and $P_{ave}$ is the target average power used as the normalization reference. A higher Q value indicates better signal quality with less distortion and noise. The distributions of LUT-PS and CCDM-PS are displayed in Figure 6. It can be observed that under the same conditions, the probability distributions of the two schemes differ. The CCDM-PS scheme focuses more on concentrating the constellation points in the innermost ring, while the LUT-PS scheme emphasizes reducing the probability that the constellation points will be found in the outermost ring.

Figure 7 shows the simulation results for the BER and Q-factor versus the received optical power (ROP) of the UTC-PD for 400 GHz photonic terahertz OFDM signals under different schemes in the back-to-back (BTB) configuration without forward error correction (FEC). The ROP values used here are higher than those typically employed in conventional fiber-optic links because the photonic terahertz communication system simulated in this work adopts a receiver architecture that differs from traditional photodiode-based systems and can tolerate higher input power. It is clear from Figure 7a that the BER performance of the system is enhanced with the application of PS, and the degree of improvement becomes more obvious as the ROP values increase. When ROP is −0.64 dBm, the BER decreases from 1.2376 $\times {10}^{-4}$ to 6.3256 $\times {10}^{-5}$, representing a performance enhancement of 48.89%. In addition, the BER graph of LUT-PS basically coincides with that of CCDM-PS, indicating that their performances are comparable. Figure 7b illustrates the relationship between the Q-factor and the ROP. It reveals that the application of PS results in an approximate improvement of 0.93 dB in the Q-factor, and the results of the LUT-PS and CCDM-PS schemes are quite similar. By integrating Figure 7a,b, it can be seen that the presented LUT-PS scheme can be on par with CCDM-PS in performance, demonstrating the feasibility of LUT-PS.

Building upon the use of LUT-PS, we further investigate the performance of 5 Gbaud signal transmission over a 10 km fiber link with the application of the RBFNN nonlinear equalizer. The results were compared with those obtained using LUT-PS in combination with a decision feedback equalizer (DFE). The results are depicted in Figure 8, here Figure 8a,b are the BER and Q-factor versus ROP, respectively. The images in Figure 8c–e, respectively, depict the received constellations of the traditional system, the system using LUT-PS and DFE (LUT-PS-DFE), and the system using LUT-PS and RBFNN (LUT-PS-RBFNN). As shown in Figure 8a, LUT-PS improves the BER performance to a certain extent. Furthermore, compared to the system that does not use RBFNN, the BER performance of LUT-PS-RBFNN and LUT-PS-DFE shows further improvement, with both exhibiting similar enhancement. As ROP increases, the reduction in BER becomes more significant. When ROP exceeds −0 dBm, LUT-PS-RBFNN reduces BER by 81.45% compared to the conventional system, outperforming LUT-PS-DFE, which achieves a reduction of 67.54%. The red region in Figure 8d,e represents the constellation diagrams after nonlinear equalization. It is evident that the RBFNN achieves significantly better equalization performance. According to Figure 8b, both schemes lead to an improvement in the Q-factor. For the woRBFNN system, the Q-factor increases by 1.21 dB, indicating that LUT-PS provides a degree of performance improvement. For the LUT-PS-DFE system, the Q-factor gain remains relatively constant at approximately 2.07 dB. In contrast, the LUT-PS-RBFNN system exhibits an increasing Q-factor gain with increasing ROP, reaching a maximum of 23 dB, which is 21.79 dB and 20.93 dB higher than that of the woRBFNN system and that of the LUT-PS-DFE system, respectively.

5. Conclusions

In this work, we propose a PS scheme based on a single-layer LUT that requires only one LUT for mapping and achieves favorable system performance. Simulation results show that the proposed method achieves a performance improvement comparable to that of CCDM-PS, reducing the bit error rate (BER) from 1.2376 $\times {10}^{-4}$ to 6.3256 $\times {10}^{-5}$, a difference that corresponds to a 48.89% enhancement. Furthermore, the Q-factor is enhanced by approximately 0.93 dB. To further mitigate nonlinear distortion while simultaneously reducing the average signal power, we integrate the LUT-PS scheme with RBFNN-based nonlinear equalization. We conducted a simulation experiment in an OFDM photonic terahertz communication system at 400 GHz, successfully achieving the transmission of a 5 GBaud signal over a 10 km optical fiber. The results show that this combined scheme reduces the BER by 81.45% and achieves a maximum Q-factor improvement of up to 23 dB.