Advanced DFE, MLD, and RDE Equalization Techniques for Enhanced 5G mm-Wave A-RoF Performance at 60 GHz

Abstract

This article presents the decision feedback equalizer (DFE), the maximum likelihood detection (MLD), and the radius-directed equalization (RDE) algorithms designed in MATLAB-R2018a to equalize the received signal in a dispersive optical link up to 120 km. DFE is essential for improving signal quality in several communication systems, including WiFi networks, cable modems, and long-term evolution (LTE) systems. Its capacity to mitigate inter-symbol interference (ISI) and rapidly adjust to channel variations renders it a flexible option for high-speed data transfer and wireless communications. Conversely, MLD is utilized in applications that require great precision and dependability, including multi-input–multi-output (MIMO) systems, satellite communications, and radar technology. The ability of MLD to optimize the probability of accurate symbol detection in complex, high-dimensional environments renders it crucial for systems where signal integrity and precision are critical. Lastly, RDE is implemented as an alternative algorithm to the CMA-based equalizer, utilizing the idea of adjusting the amplitude of the received distorted symbol so that its modulus is closer to the ideal value for that symbol. The algorithms are tested using a converged 5G mm-wave analog radio-over-fiber (A-RoF) system at 60 GHz. Their performance is measured regarding error vector magnitude (EVM) values before and after equalization for different optical fiber lengths and modulation formats (QPSK, 16-QAM, 64-QAM, and 128-QAM) and shows a clear performance improvement of the output signal. Moreover, the performance of the proposed algorithms is compared to three commonly used algorithms: the simple least mean square (LMS) algorithm, the constant modulus algorithm (CMA), and the adaptive median filtering (AMF), demonstrating superior results in both QPSK and 16-QAM and extending the transmission distance up to 120 km. DFE has a significant advantage over LMS and AMF in reducing the inter-symbol interference (ISI) in a dispersive channel by using previous decision feedback, resulting in quicker convergence and more precise equalization. MLD, on the other hand, is highly effective in improving detection accuracy by taking into account the probability of various symbol sequences achieving lower error rates and enhancing performance in advanced modulation schemes. RDE performs best for QPSK and 16-QAM constellations among all the other algorithms. Furthermore, DFE and MLD are particularly suitable for higher-order modulation formats like 64-QAM and 128-QAM, where accurate equalization and error detection are of utmost importance. The enhanced functionalities of DFE, RDE, and MLD in managing greater modulation orders and expanding transmission range highlight their efficacy in improving the performance and dependability of our system.

1. Introduction

Orthogonal frequency division multiplexing (OFDM) is a widely used modulation technique in optical communication systems due to its high bandwidth efficiency, ability to mitigate narrowband interference and dispersion effects, and suitability for high data rate transmission. With a cyclic prefix, OFDM can combat both inter-carrier interference (ICI) and inter-symbol interference (ISI) [1]. When higher-order quadrature amplitude modulation (QAM) is used, it improves link efficiency by allowing for high data rates with a low bit error rate (BER) [2].

A radio-over-fiber (RoF) system is capable of overcoming the limitations caused by the impairments in the optical domain since it leverages the low-loss transmission properties of optical fiber to transmit the mm-wave signals over long distances with minimal signal degradation [3], and it has several benefits, including excellent spectral efficiency, fast signaling, and low functional expenses [4]. End-to-end communication in a mm-wave A-RoF system necessitates the integration of both wireless and optical domain advancements, which greatly increases the link distance [5] in joint optimization of the 5G fronthaul (FH) for end-to-end C-RAN architectures [6], achieving data rates up to 80 Gbit/s [7]. The V-band frequency range enables high bandwidth [8], high throughput, and cost-efficient solutions for delivering high data rate [9] communications in optical networks, which are made possible by converged mm-wave A-RoF systems. However, chromatic dispersion in the standard single-mode fiber (SSMF) is creating ISI, especially in high-bit-rate systems (≥10 Gbit/s) using OFDM. While narrowband transmitters can solve dispersion-related issues for data rates up to 2.5 Gbit/s, chromatic dispersion limits the transmission distance in high-data-rate systems [10]. Therefore, it is necessary to prevent the deterioration of the output signal due to ISI, which could be achieved either in the optical domain or using equalization techniques in the electrical domain.

The intensity-modulation and direct-detection (IM-DD) design with external modulators is the most commonly used technique for RoF transmissions [11,12]. In [13], authors demonstrate nonlinear post-equalizers, such as a Volterra series-based equalizer and a neural network-based (NN) equalizer, to combat signal degradation in a 28 GHz mm-wave A-RoF FH link with transmission over a 10 km fiber using an optical single-sideband modulation format. In 64-QAM, both equalizers achieve BER < 3.8 × 10⁻³; additionally, when the NN equalizer is used after OFDM 16-QAM along with least-squares equalization, it obtains better BER.

The proposed solution in [14,15] describes an artificial neural network nonlinear equalizer (ANN-NLE) for single-carrier 16-QAM and 64-QAM signals in a 60 GHz RoF communication system. It validates the approach to ensure effectiveness in equalizing the signal and improving the transmission performance. Moreover, an experimental demonstration of the 60 GHz RoF system is conducted utilizing an adaptive activated ANN-NLE to enhance the BER performance and effectively transmit a 5 Gbit/s Binary Phase Shift Keying (BPSK) signal across a 10 km fiber while maintaining a forward error correction limit of 10⁻³. The authors in [16] demonstrate the feasibility of transmitting and receiving a 100 Gbit/s data rate link at 28 GHz. The performance of three modulation formats (16-PSK, 16-QAM, and 64-QAM) is tested for optical fiber lengths ranging from 5 km to 35 km using two detection systems: coherent and direct detection with a DSP block implemented in simulation software. In [17], the authors introduce an iterative block (IB) decision feedback equalization (DFE) method for an intensity modulation and direct-detection (IM/DD) based optical code division multiplexing (OCDM) system for a 50 km SSMF link. This method effectively reduces signal degradation caused by chromatic dispersion (CD) in the fiber. In [18], an optical phase-locked loop (PLL) technique to generate mm-waves is proposed, and a pilot-assisted radio frequency (RF) method is used for phase offset equalization at the receiver side for 16-QAM modulation. In [19], the authors present and demonstrate convolutional neural network (CNN) and binary convolutional neural network (BCNN) decision schemes that effectively address both linear and nonlinear impairments resulting from signal modulation, transmission, and detection. The proposed schemes are very computationally efficient, and the CNN and BCNN decision methods are showcased in a 5 Gb/s 60 GHz RoF system, covering a fiber reach of up to 20 km.

In [20], the authors propose a Volterra Nonlinear Equalizer (VNLE) for directly modulated laser (DML)-based IM/DD systems and a simulation of Pulse Amplitude Modulation (PAM-4) signals for high baud rate transmission over 10 km SSFM. In [21], the authors are experimentally transmitting a single-channel 112 Gb/s PAM-4 direct detection signal, and a high receiver sensitivity is achieved by the maximum likelihood sequence estimation (MLSE) algorithm. In [22], the authors experimentally demonstrated the generation, detection, and transmission of a 112 Gb/s dual-polarization, single- and dual-carrier 16-QAM half-cycle Nyquist subcarrier modulation, specifically tailored for short-reach IM/DD systems. High-bandwidth direct modulation passive feedback lasers are utilized for the generation of optical signals. Pre-amplified receivers identify the received signals in a back-to-back configuration following transmission over a distance of 4 km using SMF. Equalization was conducted utilizing a CMA for pre-convergence, followed by an RDE. The EVM values are 10.23% for single carrier and 9.55% for dual carrier in a 16-QAM half-cycle Nyquist subcarrier modulation over a distance of 4 km of SMF. In [23], the authors conducted experiments to demonstrate a 32 Gbaud polarization-multiplexed carrier-based self-homodyne (PMC-SH) system designed for data center interconnects (DCIs), capable of supporting higher data rates. Simulation results confirm the validity of the proposed system for a 200 Gbps (50 Gbaud) PMC-SH 16-QAM 10 km SSMF link. Offline equalization is executed by cascading RDE with a DFE, leading to enhanced performance. Additionally, it was demonstrated through analysis that the PMC-SH scheme achieves a markedly improved BER for a specified transmission bit-rate or can effectively double the data rate for a given bandwidth of the electronics, in comparison to the PAM-4 system. In [24] a coherent receiver is presented based on 12.5 GBd and utilizes 16-QAM analog processing. It has been designed and simulated employing 130 nm bipolar CMOS (BiCMOS) technology. The receiver employs RDE equalization along with Costas loop-based carrier phase recovery and compensation mechanisms. Simulations demonstrate that the receiver can effectively mitigate a dispersion of up to 160 ps/nm and a frequency offset of up to 10 MHz. In a transmission scenario involving a 10 km-long SSMF link, the output EVM reported by the receiver is 23.2% with a laser linewidth of 200 kHz. In [25], the authors propose a sparse Volterra nonlinear equalization (SVNE) and gradient-descent noise whitening (GD-NW) digital signal processing method to reduce impairments. To the best of their knowledge, experimental verification yields the maximum net data rates of 142.3/143/134.6/102.8/73.7/61.4 Gb/s across 0/5/10/20/40/60 km SSMF for C-band DML-based IM-DD links without requiring chirp and dispersion management.

In [26], an experimental demonstration of a high-speed 131.4 Gb/s IM/DD optical link is presented operating at the 2-micron waveband using an offline DSP at the receiver. A 3rd-order VNLE is utilized to address both linear and nonlinear impairments. Following this, a simple 2-tap noise whitening filter (NWF) working at 1 sample per symbol (sps) is employed to mitigate the equalization-enhanced colored noise, resulting in a 118.2 Gb/s probabilistically constellation-shaped (PCS)-PAM-8 signal and a 131.4 Gb/s PCS-PAM-16 signal transmission over 100 m solid core fiber (SCF), respectively, at a threshold BER of 4 × 10⁻², by considering a 25% overhead of soft-decision forward error correction (SD-FEC). In [27], the authors conducted an experimental comparison of uniform PAM-8 and cap-shaped Maxwell–Boltzmann-distributed PAM-8, utilizing various Gaussian orders processed with a VNLE to attain a net data rate of 200 Gb/s. A memory length of 311 taps was employed for linear equalization to reduce the impact of reflections from the discrete component setup, copper cables, and connectors. Alongside linear taps, second- and third-order kernels of VNLE with a memory length of 11 were employed to address nonlinearities caused by an arbitrary waveform generator (AWG), an electro-absorption modulated laser (EML), a semiconductor optical amplifier (SOA), an electrical amplifier (EA), and square law detection. In [28], experimental demonstrations are carried out with a 40 Gb/s PAM-4 signal transmission over a 20 km (30 km) SMF with a link power budget of 38 dB (30.7 dB) and a pre-forward error correction (FEC) BER of 10⁻³ using a 17-tap tunable adaptive NLE with noise reduction to compensate for various linear and nonlinear effects from the elements and the SMF. In [29], the performance of several DSP techniques for short-reach high-speed IM/DD systems using different schemes of PAM, such as PAM-4, PAM-6, and PAM-8, are experimentally compared at net data rates of 180 Gb/s to 300 Gb/s. Results showed that PAM-4 duobinary (DB) VNLE + Viterbi equalization and PAM-6 with VNLE + 2-tap post-filter + Viterbi equalization are ideal for a system that uses commercially accessible components, has a 3 dB bandwidth of 20 GHz, and achieves a net rate of 200 Gb/s per lane while minimizing chromatic dispersion. In [30], the authors proposed a fair comparison and thorough study of the complexity of a 56 Gb/s multi-band carrier-less amplitude and phase (CAP) and discrete multi-tone (DMT) across 80 km dispersion compensation fiber links based on IM/DD for data center interconnects. The results show that the matched finite impulse response filters and the inverse fast Fourier transform (IFFT) and fast Fourier transform (FFT) account for the majority of the complexity in multiband CAP and DMT, respectively. The selection of the multi-band CAP sub-band count and the DMT IFFT/FFT size significantly affects system complexity and performance, necessitating careful consideration of trade-offs. The multi-band CAP exhibits significant complexity sensitivity to the number of sub-bands, indicating a preference for a limited sub-band count. The DMT requires a moderate IFFT/FFT size to optimize the balance between complexity and OSNR performance.

Moreover, in [31,32], the least mean square algorithm (LMS), the constant modulus algorithm (CMA), and the adaptive median algorithm (AMF) are implemented for equalization in a converged OFDM 5G mm-wave A-RoF system at 60 GHz. Quadrature phase shift keying (QPSK) and 16-QAM are used as modulation formats in the OFDM subcarriers. The maximum achieved transmission distance is 100 km after equalization when QPSK is used, while a transmission distance of 50 km is achieved after equalization for the 16-QAM case.

Table 1. Comments on the algorithms discussed in the literature review.

Literature Review	Approach/Methods Used	Comments
[13]	Nonlinear post-equalizers, such as a Volterra series-based equalizer and a neural network-based (NN) equalizer to combat signal degradation in a 28 GHz mm-wave A-RoF 10 km fiber link for OFDM 16-QAM, and 64-QAM.	- 28 GHz mm-wave frequency. - Short reach and complex algorithm.
[14]	Artificial neural network nonlinear equalizer (ANN-NLE) for single-carrier 16-QAM and 64QAM signal transmissions in the 60 GHz RoF transmission system of length 15 km.	- Short reach and complex algorithm.
[16]	DSP unit in simulated software for 5 to 35 km direct and coherent detection 16-PSK, 16-QAM, and 64-QAM 100 Gbit/s data rate link at 28 GHz.	- Simple technique. - 28 GHz mm-wave frequency.
[17]	Use of the iterative block (IB) decision feedback equalization (DFE) method for an intensity modulation and direct-detection (IM/DD) based optical code division multiplexing (OCDM) system.	- Effectively compensates for chromatic dispersion in an IM/DD-based system. - Complex receiver structure for the IM/DD-OCDM system using costly hardware.
[18]	Adaptive activated artificial neural network nonlinear equalizer (ANN-NLE) to enhance BER performance.	- Complex algorithm.
[19]	Convolutional neural network (CNN) and binary convolutional neural network (BCNN)-based decision schemes.	- Complex algorithm.
[21]	Transmission of a single channel 112 Gb/s PAM-4 direct detection signal using the maximum likelihood sequence estimation (MLSE) algorithm.	- Achieved the highest sensitivity for 112 Gb/s transmission. - Complex receiver architecture.
[22]	Experimental generation, detection and transmission of a 112 Gb/s dual-polarization, single- and dual-carrier 16-QAM half-cycle Nyquist subcarrier modulation, over 4 km of SMF length for short-reach IM/DD systems using CMA and RDE algorithms for equalization	- Not suitable for higher-order QAM. - Costly receiver.
[23]	Experimental demonstration of 200 Gbps 32 Gbaud polarization-multiplexed carrier-based self-homodyne (PMC-SH) system using 16-QAM 10 km SSMF link. Offline equalization is executed by cascading RDE with a DFE.	- Short reach and costly hardware used
[24]	12.5 GBd 16-QAM analog processing-based coherent receiver built and simulated in 130 nm BiCMOS. RDE-based equalization and Costas loop-based carrier phase compensation are used in the receiver for a transmission over 10 km-long fiber.	- Short reach and costly hardware used. - Not suitable for higher-order modulation formats.
[28,29]	LMS, CMA, and AMF-based equalization in converged mm-wave A-RoF system at 60 GHz.	- Not suitable for higher-order modulation formats.
[This Work]	Converged OFDM-based mm-wave A-RoF system at 60 GHz with signal processing using DFE, MLD, and RDE algorithms.	- Electronic dispersion Compensation for higher-order modulation formats.

delineates the various techniques examined in the literature review to address ISI and nonlinearities, along with some notes related to their advantages and disadvantages. In addition,

Table 2. Literature review comparisons over specific parameters.

Literature Review	Frequency (GHz)	Maximum Fiber Length (km)	Modulation Format	Algorithm	Computational Complexity
[13]	28	10	OFDM 16-QAM	Voltera and neural network-based equalizers	High
[14]	60	20	OFDM 16-QAM and 64-QAM	Complex valued (ANN-NLE)	High
[16]	28	5–35	16-PSK, 16-QAM and 64-QAM	Built-in DSP unit in simulation software	Low
[18]	60	10	BPSK	ANN-NLE	High
[19]	60	20	2-PAM	Convolutional neural network (CNN) and binary convolutional neural network (BCNN)-based decision schemes	High
[28]	60	25	QPSK and 16-QAM	LMS Algorithm	Low
[29]	60	100	QPSK and 16-QAM	LMS, CMA, and AMF Algorithms	Low
[This work]	60	0–120	QPSK, 16-QAM, 64-QAM and 128-QAM	DFE, MLD, and RDE Algorithms	Moderate

elaborates and compares specific parameters employed in the literature review articles.

To that end, in the present paper, we propose three new algorithms, i.e., the decision feedback equalizer (DFE), the maximum likelihood detection (MLD), and the radius-directed equalization (RDE) algorithms for equalization in a converged OFDM 5G mm-wave A-RoF system at 60 GHz. The modulation schemes being used in this system are QPSK, 16-QAM, 64-QAM, and 128-QAM in all the OFDM subcarriers. The proposed algorithms are developed in MATLAB-R2018a [33] to equalize the distorted subcarriers. In this work, we use an SSMF of length up to 120 km since DFE and MLD are more adept at managing fiber distortions and noise. DFE effectively mitigates inter-symbol interference (ISI) by leveraging past decisions, rendering it particularly beneficial in dispersive SSMF channels. Meanwhile, MLD offers near-optimal performance by evaluating all potential signal sequences and selecting the one that maximizes the probability of accurate detection, resulting in enhanced transmission distance relative to other known algorithms such as LMS and CMA. Finally, RDE performs best for QPSK and 16-QAM constellations.

The rest of the paper is organized as follows: Section 2 provides the details of the different equalization algorithms used. Section 3 briefly provides the simulation setup for a converged mm-wave A-RoF system at 60 GHz for testing the algorithms. Subsequently, Section 4 compares the DFE, MLD, and RDE algorithms with different modulation formats along the maximum fiber lengths achieved, while Section 5 provides details of the assessment of the proposed work against different well-known equalization algorithms. The relationship between RF input power and EVM for QPSK, 16-QAM, and 64-QAM modulation formats is considered in Section 6, and Section 7 presents the conclusions.

2. Proposed Equalization Algorithms

2.1. Decision Feedback Equalizer

A decision feedback equalizer (DFE) is an equalizer that offers superior performance compared to a linear equalizer. The use of DFE can reduce noise enhancement and improve the forward linear filter’s ability to handle ISI and co-channel interference. A DFE usually has more taps than an FFE, as it requires taps in both the feedforward and feedback filters. The DFE’s feedback portion takes into consideration earlier symbol decisions, which increase complexity and tap into the entire structure, but it is particularly beneficial when addressing significant channel distortion.

The DFE structure, as mentioned above, consists of a feedback filter and a decision function (Figure 1). The feedforward filter processes the current received symbols, passing them through the decision function. The received symbols are then refined by passing through the feedback filter, which utilizes previously identified symbols that ideally have minimal or no noise, allowing for precise estimation and cancellation of ISI without enhancing the noise, provided that the previous decisions are correct. If the decisions are incorrect, the error propagates to the next decisions. One method to reduce error propagation is through decision-directed adaptation, in which the DFE updates the feedback filter solely when the decisions are accurate [34].

One notable advantage of the DFE over linear equalizers is its ability to remove ISI without enhancing noise, because a linear equalizer handles noise and signal equally, which might result in the amplification of noise [35,36,37]. The following equations describe the DFE [38]:

$$\mathrm{y}[\mathrm{n}]={\sum }_{k=-{{\rm N}}_{f+1}}^0{\mathrm{f}}_{\mathrm{k}}\left(n\right)·x\left[n-k\right]+{\sum }_{k=1}^{Nb}{\mathrm{b}}_{\mathrm{k}}\left(n\right)·\widehat{s}[n-k]$$

(1)

t(n) = f{y(n)}

(2)

e(n) = y(n) − t(n)

(3)

where y[n] is the equalized output signal of the nth symbol.

• $x$[$n-k$]: the equalizer input sequence.
• ${\mathrm{f}}_{\mathrm{k}}$: the coefficients for the feedforward filter.
• ${\mathrm{b}}_{\mathrm{k}}$: the coefficients for the feedback filter.
• $N_f$: the order length of the feedforward filter.
• $N_b$: the order length of the feedback filter.
• $\widehat{s}[n-k$]: the previously detected symbols.
• t(n): the equalizer decision sequence.
• f{n}: the decision function.
• e(n): the error signal.

2.2. Maximum Likelihood Sequence Detection Algorithm

Maximum likelihood detection (MLD) is a widely recognized and highly effective approach suitable for practical applications that offer optimal detection performance. The idea is essential in signal processing, especially when identifying and interpreting symbols despite interference from noise and distortions. MLD is founded on the fundamental concept of statistical inference. It utilizes the likelihood function of detecting the received symbol based on a given set of potentially transmitted symbols. The objective is to optimize this likelihood to ascertain the most probable transmitted symbol [39,40,41]. However, the main drawback of the MLD algorithm is that it needs accurate information on the potentially transmitted symbols to compute the likelihood of each possible transmitted symbol. If the potentially transmitted symbol is not correct, the performance of MLD may deteriorate dramatically, possibly leading to erroneous decisions and detection errors. Also, its complexity increases exponentially for higher-order constellations. A low-complexity MLD algorithm, referred to as simplified MLD, can be employed to approximate the MLD solution, thereby alleviating this issue [42,43].

In our work, we employ an optimal maximum likelihood detection algorithm that accurately estimates the transmitted data sequence t = (t⁽⁰⁾, t⁽¹⁾, t^(k−1))^T. The potentially communicated data symbol vectors are t_u, u = 0, … M^k−1, where M^k represents the total number of possible transmitted data symbol vectors and M denotes the number of potential realizations of t^(k). The symbol ${(·)}^T$ denotes the transpose of the vector.

The MLD approach employed here is the maximum likelihood sequence estimation (MLSE). MLSE reduces the sequence error probability, specifically the probability of error in the data symbol vector, which corresponds to maximizing the conditional probability P{t_u|r}, indicating the likelihood that t_u was sent given the received vector r. The estimate of t (most likely transmitted data vector) derived from MLSE is calculated as follows:

$$\widehat{t}={arg}_{\mathrm{t}\mathrm{u}}\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{P}\left\{{\mathrm{t}}_{\mathrm{u}}\right|\mathrm{r}\}$$

(4)

with arg being the function’s argument. If the noise N_t is additive white Gaussian, Equation (4) will correspond to identifying the data symbol vector t_u that minimizes the squared Euclidean distance, written as follows:

$${\Delta }^2({\mathrm{t}}_{\mathrm{u}},\mathrm{r})={\Vert \mathrm{r}-\mathrm{A}{\mathrm{t}}_{\mathrm{u}}\Vert }^2$$

(5)

between the received sequences and all the potentially transmitted sequences. The most likely transmitted data vector is calculated as follows:

$$\widehat{t}={arg}_{\mathrm{t}\mathrm{u}}\mathrm{m}\mathrm{i}\mathrm{n}{\Delta }^2({\mathrm{t}}_{\mathrm{u}},\mathrm{r})$$

(6)

MLSE necessitates the assessment of the obtained Euclidean distances for the estimation of the data symbol vector $\widehat{t}$ [44].

2.3. Radius-Directed Equalization

The radius-directed equalizer (RDE) is proposed to improve the equalization performance for the QAM constellations. Since QAM constellation does not present a constant modulus or magnitude, the RDE is implemented as an alternative algorithm to CMA-based equalizers, demonstrating potential enhancements in steady-state performance. In RDE, the idea is to adjust the amplitude of the received, possibly distorted, QAM symbol so that its modulus is closer to the ideal value for that symbol.

The RDE optimization relies on the output from the equalizer and the nearest constellation radius. The error criterion for the RDE is defined as follows [45].

$$e\left(\mathrm{k}\right)=u\left(\mathrm{k}\right){(R-|y\left(\mathrm{k}\right)|}^2)$$

(7)

where e(k) is the error signal, R is the radius of the nearest constellation symbol, u(k) is the equalized signal, and ${\left|y\right(\mathrm{k}\left)\right|}^2$ is the squared magnitude of the distorted signal.

RDE is the best-performing algorithm in our setup for QPSK and 16-QAM constellations, as will be shown in Section 4.3, and provides a faster conversion than the CMA algorithm for QAM signals [46].

3. Simulation Setup

Figure 2 describes in block diagrams (a) OFDM Tx, (b) OFDM Rx, and (c) a complete simulated setup of a converged 5G mm-wave A-RoF system at 60 GHz designed in the VPI Photonics design suite [47] and used to test the developed algorithms. The OFDM Tx is depicted in Figure 2a, where the data stream is converted from serial to parallel form, creating multiple parallel data streams. This procedure allows the data to be transmitted simultaneously on various subcarriers. Next, the inverse fast Fourier transform (IFFT) is performed, transforming a complex frequency domain signal into a time domain signal, which is then used to modulate the signal onto orthogonal subcarriers in the digital domain. A cyclic prefix (CP), introducing a guard time interval between symbols, is used to minimize intersymbol interference (ISI). The parallel data streams are converted back to serial, and the output signal through a digital-to-analog converter (DAC) is transformed into an analog signal.

Using a Mach–Zehnder modulator (MZM), the signal is modulated, while an optical bandpass filter can be used to suppress the amplified spontaneous emission (ASE) noise (Figure 2c). The produced 60 GHz mm-wave analog signal [31,32] is transmitted through an SSMF of different lengths with a 16 ps/nm/km dispersion, while a photodiode is utilized to capture the optical signal, which is then converted into electrical and fed into an OFDM receiver for subcarrier recovery, as shown in Figure 2c. The electrical signal undergoes down-conversion to a baseband signal, followed by an OFDM decoder, as shown in Figure 2b. The distorted electrical symbols from the OFDM constellations are captured and subsequently equalized in MATLAB using the different algorithms.

Table 3. Simulation parameters.

Link Design Components	Values
Carrier Frequency	7.5 GHz
Laser CW	10 dBm
Wavelength	1553 nm
RIN	−130 dB/Hz
Radio Frequency	60 GHz mm-wave
Bit Rate Default	40 Gbit/s
Bits per Symbol	2, 4, 6, 7
SSMF Length	Up to 120 km
Dispersion	16 ps/nm/km
Dispersion Slope	0.08 × 103 s/m3
SSMF Attenuation Coefficient	0.2 dB/km
Photo Diode Model	PIN
Responsivity	0.8 A/W
Thermal Noise	10−12 A/Hz1/2
Shot Noise	ON
Cyclic Prefix	0.125

provides a summary of the simulation parameters used in this work.

4. Comparisons of Results for DFE, MLD Algorithms

The DFE and MLD algorithms are first compared regarding their complexity levels in time and space. Time complexity describes the duration an algorithm requires to be executed as a function of the input size. Within the framework of DFE and MLD, it provides an estimate of the duration required for the system to process a set of signals or symbols. DFE complexity grows linearly over time since the time complexity of the DFE is reliant upon the number of symbols in the input signal and the number of taps or delays in the feedback filter. For a signal of N symbols, the total time complexity is O(N) because DFE processes each symbol in sequential order. If T feedback taps are present in an algorithm, processing the feedback taps for each symbol might be necessary for the DFE, leading to a time complexity of O (N × T) [48,49,50,51].

In MLD, each possible symbol sequence is compared with the received symbol sequences. An exhaustive search needs to be applied to examine all the feasible transmitted symbol sequences; the time complexity is O(2^N) × T, as MLD necessitates a comparison of each possible symbol sequence across all symbols in the signal. In 64-QAM, there are 64 potential symbols; the time complexity is O (2⁶⁴) × T, indicating significant computational expense [49,50,51].

Space complexity describes how much memory or storage is utilized by an algorithm as a function of the input size. The space complexity of the DFE is dictated by the number of prior symbols stored in the feedback filter (e.g., past decisions) and the magnitude of the current input data. The space complexity of DFE is O(L), as it necessitates the storage of the past L symbols and processes each symbol independently.

In the case of MLD, for each symbol in the sequence, the space complexity for MLD is O(M^K), as it retains all potential combinations of the symbols for K received symbols and M potentially transmitted symbols, resulting in an exponential increase relative to the number of symbols.

Next, the received signal of the used setup shown in Figure 2 is processed offline using the three proposed equalization algorithms and applying different modulation formats in the OFDM subcarriers to minimize the ISI caused by the dispersive transmission.

4.1. Equalization by Maximum Likelihood Detection (MLD) Algorithm

The MLD algorithm is applied to the distorted received data after the photodiode in the OFDM analyzer. MLD incorporates the entire sequence of received symbols for the channel memory length. In the case of QPSK, there are four sequences; for 16-QAM, 16 possible symbol sequences; for 64-QAM, 64 symbol sequences; and for 128-QAM, 128 symbol sequences.

The distorted OFDM subcarrier signals with the QPSK, 16-QAM, 64-QAM, and 128-QAM modulation formats result in the constellation diagram and EVM calculations that are shown in Figure 3 before (a, b, c, and d) and after equalization (e, f, g, and h) for the respective modulation formats and at the maximum attained fiber lengths for each case.

Maintaining the performance criteria of modern communication systems across different modulation schemes requires keeping the error vector magnitude (EVM) of the received signal below the proposed 3GPP thresholds, ensuring the quality and reliability of the signal, particularly as the complexity of the modulation format increases. The implementation of the proposed maximum likelihood detection (MLD) algorithm is tested across different fiber lengths. For QPSK modulation on OFDM subcarriers, an EVM of 17.06% is achieved after equalization at a maximum fiber length of 120 km. When using 16-QAM modulation, the fiber length is reduced to 100 km, and the EVM to 12.40% after equalization. For higher-order modulations, such as 64-QAM and 128-QAM, the fiber lengths are 50 km and 40 km, respectively, to maintain compliance with the 3GPP threshold limits, and EVM values of 7.35% for 64-QAM and 4.85% for 128-QAM are obtained.

All equalized EVM values successfully meet the stringent 3GPP threshold limits (17.5% for QPSK, 12.5% for 16-QAM, 8% for 64-QAM, and 5% for 128-QAM), demonstrating the effectiveness of the proposed MLD in maintaining signal integrity across various modulation formats and fiber lengths.

4.2. Equalization by Decision Feedback Equalizer

The DFE equalization is performed on the distorted data obtained at the OFDM receiver. Two (2) feedback taps are used for QPSK and 16-QAM distorted constellations, while four (4) and six (6) feedback taps are used for 64-QAM and 128-QAM distorted constellations for equalization.

Figure 4 shows the constellation diagrams of QPSK, 16-QAM, 64-QAM, and 128-QAM obtained before and after the equalization at the maximum fiber length that can be achieved before (a, b, c, and d) and after equalizing (e, f, g, and h) for the respective modulation formats. After equalization, the EVM of the equalized constellations is computed.

The proposed DFE performance is evaluated over various fiber lengths. For QPSK modulation in OFDM subcarriers, with a maximum fiber length of 120 km, an EVM of 16.60% is achieved, while for 16-QAM modulation, at a maximum fiber length of 100 km, the EVM is 12.32% after equalization. In the cases of 64-QAM and 128-QAM modulations, EVMs of 7.19% and 4.73% are achieved for fiber lengths of 50 km and 40 km, respectively. Notably, all the equalized EVM values are below the 3GPP threshold.

Table 4. EVM improvement of DFE vs MLD across various modulation schemes and fiber lengths.

Modulation Scheme	EVM for MLD After Equalization (%)	EVM for DFE After Equalization (%)	EVM Improvement (%)	Max Fiber Length (km)
QPSK	17.06	16.60	0.46	120
16-QAM	12.40	12.32	0.08	100
64-QAM	7.35	7.19	0.16	50
128-QAM	4.85	4.73	0.12	40

presents a comparison of MLD and DFE algorithms in terms of EVM for different modulation formats. DFE demonstrates marginally superior performance compared to MLD regarding EVM. The primary reason for this is DFE’s capacity to utilize input from previously identified symbols, which can statistically reduce the variance of the error to mitigate interference, leading to a cleaner signal and reduced EVM. However, the enhancement in performance may be minimal, as MLD remains an ideal detection method.

Statistically, MLD identifies the greatest likelihood sequence that corresponds to the received signal, based on the channel model. In this regard, MLD can rectify errors utilizing all available information simultaneously; however, it lacks the advantages of feedback. In a noisy channel with ISI, the MLD is ideal as it reduces the probability of bit errors by evaluating all potential symbol sequences, but its statistical robustness can be affected by errors in detecting symbols that introduce incorrect decisions. It doesn’t use prior decisions to correct subsequent symbol detection errors. Since DFE leverages feedback, it can statistically reduce the impact of previous symbol errors on future decisions, hence slightly better EVM than that of MLD.

4.3. Equalization by Radius-Directed Equalizer

As the RDE algorithm is independent of the carrier offset and is ideal for QAM signals, achieving minimum error when the signal is equalized [45,46].

RDE is executed on the distorted data acquired at the OFDM receiver. The constellation diagrams for QPSK and 16-QAM were acquired following equalization at the maximum fiber lengths of 120 km and 100 km, respectively, resulting in an EVM of 11.35% for QPSK and 9.41% for 16-QAM. In our proposed setup, RDE is not effective for higher-order modulation formats such as 64QAM and 128QAM.

Figure 5 shows the constellation diagrams of QPSK and 16-QAM, obtained before and after the equalization at the maximum fiber length of 120 km and 100 km that can be achieved before equalization (a and b) and after equalization (c and d) for the respective modulation formats.

The RDE algorithm is based on the equalizer output and the nearest constellation radius. The running time increases linearly with the input size (N symbols). Thus, the complexity is O(N), and for time T, the total time complexity of the algorithm is O(N) × T.

Space complexity measures the amount of memory an algorithm uses as a function of the input size. Since RDE finds the closest constellation point within the decision radius, not in the entire constellation of points, we can say that its space complexity is linear [45].

4.4. EVM Performance as a Function of Fiber Length

Figure 6 illustrates the EVM performance as a function of fiber length for both QPSK and 16-QAM modulation schemes without equalization and after equalization with the MLD, DFE, and RDE algorithms. These results demonstrate the efficacy of MLD, DFE, and RDE techniques in compensating ISI within a converged mm-wave A-RoF system operating at 60 GHz. A comparative analysis of these two equalization algorithms (for QPSK and 16-QAM) reveals that the DFE slightly outperforms the MLD, as shown in Table 4, although RDE, as mentioned above, outperforms both of them.

Figure 7 presents the EVM results versus the fiber length for 64-QAM and 128-QAM. A comparative analysis of these two equalization algorithms (for 64-QAM and 128-QAM) reveals that the DFE slightly outperforms the MLD, as shown in Table 4. In our proposed setup, RDE is working for higher-order (more than 16-QAM) modulation formats and thus is not plotted.

Moreover, when we compare the two algorithms in terms of computational complexity, the DFE is performing better than the MLD because the latter assesses all potentially transmitted symbol sequences to identify the one that optimizes the likelihood of generating the observed received sequence, and MLD’s complexity increases exponentially with the size of the signal constellation and the number of symbols in the sequence [41], rendering it less efficient for systems with high modulation orders.

5. Assessment of Algorithms Against Prior Research Results

The performance of the proposed DFE, MLD, and RDE algorithms is compared against the least mean square (LMS) [52] algorithm, the constant modulus algorithm (CMA) [53,54,55], and the adaptive median filtering (AMF) [56] algorithm, as reported in our previously published work [31,32]. Although the LMS is simpler and more efficient for linear channels, it is inadequate for severely dispersive channels. At the same time, AMF, primarily intended for reducing noise, is less appropriate for equalization tasks in high-frequency, high-data-rate situations.

For QPSK modulation across all OFDM subcarriers, specifically, when comparing MLD, DFE, and RDE with LMS, CMA, and AMF for an SSMF of length 120 km, as shown in

Table 5. EVM and fiber length comparisons for the different equalization algorithms.

Algorithm	QPSK		16-QAM
	EVM (%)	Fiber Length (km)	EVM (%)	Fiber Length (km)
MLD	17.06	120	12.4	100
DFE	16.6	120	12.32	100
LMS	-	120	23.75	100
CMA	-	120	-	100
AMF	-	120	23.00	100
RDE	11.35	120	9.41	100

and Figure 8a, the LMS, CMA, and AMF algorithms do not provide an EVM value for QPSK, indicating that they failed to perform effectively under the given conditions or configuration. This suggests that LMS, CMA, and AMF may struggle to adapt to QPSK modulation over long fiber distances, making it unsuitable for this particular setup.

For QPSK, the MLD algorithm achieves an EVM of 17.06% over a fiber length of 120 km, while the DFE algorithm slightly outperforms MLD in terms of EVM, with a value of 16.6%. However, both MLD and DFE can manage fiber lengths up to 120 km, indicating that both algorithms can maintain signal quality over similar transmission distances. The small improvement in EVM for DFE suggests that this algorithm may be better suited for compensating for distortion in the dispersive channel compared to the MLD. Moreover, RDE surpasses MLD and DFE in EVM values for both QPSK and 16-QAM modulation formats, indicating that it is a suitable algorithm for specific formats.

For 16-QAM modulation, implementing MLD and DFE algorithms allows for an extended transmission distance of up to 50 km beyond what is achievable with LMS and AMF algorithms while maintaining EVM values below the 3GPP threshold, as shown in Figure 8b.

Specifically, when comparing MLD with LMS and AMF after equalization for 16-QAM modulation, as shown in Figure 8b, MLD achieves an EVM improvement of 11.35% over LMS and a 10.60% improvement over AMF at a fixed fiber length of 100 km. The CMA algorithm is not employed in this scenario due to its limitations with multiple radii in the constellation points, rendering it ineffective for 16-QAM modulation [57].

Further analysis reveals that DFE slightly outperforms MLD and outperforms LMS and AMF for 16-QAM modulation in OFDM subcarriers, demonstrating an EVM improvement of 0.80% over MLD, 11.43% over LMS, and 10.68% over AMF for the same fixed fiber length of 100 km. For 16-QAM modulation over a transmission distance of 50 km, a comparative analysis reveals significant improvements in EVM when using MLD and DFE algorithms.

In the context of QPSK and 16-QAM modulation across all OFDM subcarriers, a comparison of RDE with MLD, DFE, LMS, CMA, and AMF reveals that RDE attains an EVM of 11.35% for a fiber length of 120 km in the case of QPSK and an EVM of 9.41% for a fiber length of 100 km for 16-QAM, as illustrated in Table 5 and Figure 8a,b. The results indicate that RDE outperforms MLD and DFE algorithms in terms of EVM, demonstrating an improvement of 5.71% over MLD and 5.25% over DFE for QPSK. LMS, CMA, and AMF proved ineffective for a 120 km SMF length for our proposed setup.

In the case of 16-QAM, RDE demonstrates superior EVM performance compared to MLD, DFE, LMS, and AMF algorithms. RDE demonstrates an improvement in EVM of 2.99% compared to MLD, 2.91% compared to DFE, 14.34% compared to LMS, and 13.59% compared to the AMF algorithm, whereas CMA was ineffective for 16QAM.

These findings highlight the superior performance of MLD, DFE, and RDE algorithms not only in improving signal quality (EVM for QPSK and 16-QAM) but also in extending the transmission distance and demonstrating robustness and effectiveness in high-performance optical communication systems.

Moreover, when higher-order modulation schemes, such as 64-QAM and 128-QAM, are used for equalization approaches, like LMS, CMA, RDE, and AMF, do not give meaningful results. At the same time, the proposed MLD and DFE algorithms demonstrate their effectiveness in managing the challenges posed by higher modulation formats and dispersion in optical fibers, preserving signal quality and meeting the 3GPP EVM threshold limits.

Table 6. Comparisons for the different performance indicators of different equalization algorithms.

Performance Indicator	LMS	CMA	DFE	MLD	RDE	AMF
Error Vector Magnitude (EVM)	Moderate	Moderate	Low	Low	Very low (with restrictions in our system)	Moderate
Convergence Rate	Slow	Moderate	Moderate	Slow	Fast	Fast
Time Complexity	Low	Moderate	Moderate	High	Moderate	Low
Space Complexity	Low	Low	Moderate	High	Moderate	Low
Adaptability	Moderate	Moderate	Moderate	Moderate	High	Moderate
Memory Usage	Low	Low	Moderate	High	Moderate	Low
Convergence Time	Moderate	Moderate	Fast	Slow	Moderate	Fast
Number of Taps	Low	Moderate	High	High	Moderate	Low
Computational Complexity	Low	Low	Moderate	Moderate	Low	Low

provides a comprehensive summary of the various performance indicators associated with all equalization algorithms implemented in the simulation setup.

6. Relationship Between RF Input Power and EVM for Different Modulation Formats

The correlation between RF input power and EVM is crucial in optical communications, as it impacts signal quality, system performance, and overall efficiency. Increasing the RF input power can enhance the quality of the signal until a certain threshold, beyond which high power might cause nonlinearities.

In general, in scenarios in which the power levels are low, the signal may be weak and susceptible to noise, which can result in high EVM values. The signal-to-noise ratio (SNR) can be improved by increasing the input power, which can first increase the signal quality by overcoming noise and improving the output signal, leading to reduced EVM values and improvement in the demodulation performance.

Various modulation schemes have varied criteria for the quality of the signal and the required power level. Understanding the relationship between RF input power and EVM can help optimize the power levels and achieve the desired signal quality without consuming any additional power. Examining EVM across various RF input power levels can help determine the performance limits of every modulation scheme.

Figure 9 presents the EVM performance as a function of RF input power for the different modulation formats and the relevant maximum fiber lengths. For QPSK (100 km), the EVM is high at lower RF input power levels (3 dBm to 9 dBm), and the same applies for 16-QAM and 64-QAM. This indicates poor signal quality due to the significant effect of ISI and noise that are not adequately compensated for. For RF input power levels from 9 to 11 dBm, there is a substantial decrease in EVM, which suggests that increasing power helps to improve signal quality by overcoming, to some extent, the impairments. Beyond 11 dBm, further increases in RF input power continue to reduce EVM, but at a slower rate, indicating marginal benefits. Also, EVM, in all cases, does not fall below the corresponding 3GPP thresholds except for QPSK, which at 11.5 dBm input power is just below the threshold.

On the contrary, Figure 10 presents the EVM performance as a function of RF input power after equalization for the different modulation formats, indicating that although the trend of the curves is similar to that above, the system can perform efficiently with less input power; for example, for QPSK and 100 km fiber length, when the DFE algorithm is used, a 9 dBm input is sufficient, while in the case of MLD, the corresponding value is 9.5 dBm.

Within this scenario, the significance of equalization algorithms is of utmost importance since it is obvious from Figure 8 that the signal in higher modulation formats such as 16-QAM, 64-QAM, or 128-QAM can only reach the corresponding 3GPP thresholds with equalization. The proposed algorithms, maximum likelihood detection (MLD) and decision feedback equalizer (DFE), are specifically developed to more efficiently reduce the impact of ISI and noise. By implementing these techniques, it is feasible to attain reduced EVM values and enhanced signal quality, which is crucial for maximizing the performance of optical communication systems.

7. Conclusions

The DFE, MLD, and RDE algorithms are implemented in MATLAB to eliminate the ISI in a dispersive channel in an SSMF, which improves the system performance. A converged 5G mm-wave A-RoF system at 60 GHz is used as an example to evaluate the four equalization algorithms. When assessing these algorithms in the case of QPSK, 16-QAM, 64-QAM, and 128-QAM as a modulation format in OFDM subcarriers, the DFE algorithm performs slightly better than the MLD with an EVM improvement of 0.46%, 0.08%, 0.16%, and 0.12%, respectively, while RDE performance is the best among all for QPSK and 16-QAM modulation formats. When comparing MLD and DFE in terms of computational resource requirements, MLD searches all transmitted sequences for an M-ary modulation, and the complexity is proportional to $M^N$ where N is the number of transmitted symbols, which grows exponentially with signal space, requiring greater computational resources. However, DFE is more computationally efficient because it has linear complexity in terms of the number of symbols, as it only depends on the number of feedback taps. Thus, it reduces the problem to a feedback mechanism that uses previously decoded symbols to decode the current symbol, avoiding exhaustive searches. Consequently, it performs better regarding the EVM than MLD and has less hardware complexity.

The proposed algorithms, in comparison to the three most popular equalization algorithms, LMS, CMA, and AMF, achieved better EVM values and a fiber length that can extend up to 120 km. Moreover, algorithms such as LMS, CMA, RDE, and AMF fail miserably when applied to higher-order QAM constellations, while the DFE and the MLD allow us to reach EVMs of 7.19% and 7.35% for 64-QAM and 50 km SSMF distances, respectively, and 4.73% and 4.85% for 128-QAM and 40 km SSMF distances, respectively.

Lastly, it is proven that if equalization algorithms are used, there is no need to increase the input power to achieve EVM values below the corresponding 3GPP thresholds, especially in higher modulation formats.

Moreover, in the last few years a separate class of machine learning (ML) algorithms [58,59] has been developed that can be used to redefine the design of the coding and modulation either at the transmitter or the receiver of a communication system, showing promise in delivering lower bit error rates and better robustness to the wireless channel impairments. However, according to the literature, to handle larger constellation diagrams efficiently, the neural network may require more neurons in the input layer to represent them, and as the number of constellation points increases, the neural network needs more training data to correctly map the noisy received symbols to their correct positions in the constellation. Taking all the above into consideration, our next research endeavor will focus on the implementation of these algorithms and the comparison with the algorithms presented in this work.