Optimized DSP Framework for 112 Gb/s PM-QPSK Systems with Benchmarking and Complexity–Performance Trade-Off Analysis

Abstract

In order to enhance the performance of 112 Gb/s polarization-multiplexed quadrature phase-shift keying (PM-QPSK) coherent optical receivers, a novel digital signal processing (DSP) framework is presented in this study. The suggested method combines cutting-edge signal processing techniques to address important constraints in long-distance, high data rate coherent systems. The framework uses overlap frequency domain equalization (OFDE) for chromatic dispersion (CD) compensation, which offers a cheaper computational cost and higher dispersion control precision than traditional time-domain equalization. An adaptive carrier phase recovery (CPR) technique based on mean-squared differential phase (MSDP) estimation is incorporated to manage phase noise induced by cross-phase modulation (XPM), providing dependable correction under a variety of operating situations. When combined, these techniques significantly increase Q factor performance, and optimum systems can handle transmission distances of up to 2400 km. The suggested DSP approach improves phase stability and dispersion tolerance even in the presence of nonlinear impairments, making it a viable and effective choice for contemporary coherent optical networks. The framework’s competitiveness was evaluated by comparing it against the most recent, cutting-edge DSP methods that were released after 2021. These included CPR systems that were based on kernels, transformers, and machine learning. The findings show that although AI-driven approaches had the highest absolute Q factors, they also required a large amount of computing power. On the other hand, the suggested OFDE in conjunction with adaptive CPR achieved Q factors of up to 11.7 dB over extended distances with a significantly reduced DSP effort, striking a good balance between performance and complexity. Its appropriateness for scalable, long-haul 112 Gb/s PM-QPSK systems is confirmed by a complexity versus performance trade-off analysis, providing a workable and efficient substitute for more resource-intensive alternatives.

1. Introduction

Recent advances in digital signal processing (DSP) frameworks have significantly improved the spectral efficiency and performance of coherent optical receivers that use 112 Gbit/s polarization-multiplexed quadrature phase-shift keying (PM-QPSK). A better bit error rate (BER) and reach were achieved by Hazzouri [1] by introducing an integrated DSP framework that combined chromatic dispersion compensation, a constant-modulus algorithm (CMA) for polarization mode dispersion (PMD), and a modified Viterbi-Viterbi algorithm for phase recovery [1]. The authors of [2] created a unique spectral clustering-based unsupervised DSP approach to address nonlinear impairments. This scheme effectively suppresses fiber impairments and extends transmission distances for PM-16QAM coherent connections. Singh demonstrated an enhanced BER and system stability in a free-space optics scenario by using DSP to reduce channel effects in a 112 Gbit/s DP-16QAM system under changeable atmospheric conditions [3]. Using sophisticated DSP with integrated grating couplers and high-speed photodiodes, Peng developed a small 100 Gbit/s silicon photonic DP-QPSK receiver [4]. Back-to-back tests confirmed that Sun used a low-complexity DSP chain to develop a real-time FPGA-based flexible coherent receiver for QPSK and 16QAM [5].

When Hamadamin [6] used DSP-based PMD mitigation to test a 100 Gbit/s DP-QPSK link, they achieved outstanding Q-factor and OSNR tolerance over 2000 km, which is beneficial for long-haul WDM systems. A 10 Gbit/s burst mode coherent TDM-PON receiver with real-time DSP was introduced by Kanai [7] in high-splitting PON systems, attaining extremely high power budgets and precise receiver sensitivity control [7]. In a similar vein, Sivakumar [8] suggested a 160 Gbit/s intersatellite PDM-QPSK link that uses DSP for OSNR optimization and phase noise reduction in long-distance optical wireless communication. An e-based carrier phase estimator using GPU-based DSP for QPSK signals was constructed by Kim to address the problem of flexible carrier phase recovery, and it operated in real time with exceptional sensitivity [9]. Lastly, for reliable high-speed links, Bidan et al. [10] suggested a GEO feeder link DSP receiver design for 56 GBaud DP-QPSK, integrating data-aided phase recovery with blind CMA-based equalization.

Because of their exceptional spectral efficiency, adaptable modulation capabilities, and resistance to fiber impairments, coherent optical communication technologies continue to lead the way in high-capacity optical networking. These systems make use of the optical field’s amplitude and phase, allowing for notable enhancements in transmission efficiency via sophisticated digital signal processing (DSP) methods. A dependable option for long-haul and metro core applications, polarization-multiplexed quadrature phase-shift keying (PM-QPSK) at 112 Gbit/s provides a great balance between implementation complexity and data throughput efficiency [11,12,13]. The 112 Gb/s PM-QPSK option is appropriate for metro and regional optical networks because it balances bandwidth efficiency, noise tolerance, and hardware simplicity. Greater spectral efficiency is provided by more sophisticated formats (such as 16-QAM and 64-QAM), but this work attempts to avoid their stricter OSNR requirements, more complex DSP, and greater coding overheads.

In order to increase the transmission range and boost system resilience, recent advancements have integrated sophisticated DSP-based linear and nonlinear impairment mitigation procedures with adaptive phase recovery approaches, further improving the performance of coherent systems. Notably, recent research showed that AI-optimized DSP modules and hardware-efficient implementations can improve the BER and OSNR performance of 112 Gbit/s and multi-100G coherent systems. These systems can be used for a variety of applications, from metro access convergence and turbulence-resilient long-distance transmission [14] to underwater free-space optics [15,16].

Adjusting for polarization mode dispersion (PMD) and chromatic dispersion (CD) is a significant difficulty in PM-QPSK coherent optical systems. The computational overhead of traditional time-domain equalization (TDE) increases with data rates, despite its effectiveness [17]. One alternative is overlap frequency domain equalization (OFDE), which processes signals in the frequency domain and offers a reduction in computational complexity. With this method, CD may be compensated for effectively through block processing without seeing a major drop in performance [18]. Recent advances integrate OFDE within CO-OFDM systems, enabling dynamic DSP-based compensation for both CD and PMD at high data rates [19]. Additionally, density matrix-based PMD monitoring [20] and AI-driven joint impairment compensation using parametric networks [21] further improve system robustness and efficiency. The general block diagram of DSP processes in a PM-QPSK receiver is illustrated in Figure 1.

Another critical component of coherent receivers is carrier phase recovery (CPR), which corrects phase noise to enable accurate demodulation of the received signal. The Viterbi-Viterbi (V-V) algorithm and other traditional CPR techniques work well in situations with slow or static phase changes, but they have trouble with dynamic impairments such as cross-phase modulation (XPM), which is commonly seen in dense wavelength-division multiplexed (DWDM) systems [22]. Accurate phase estimation requires adaptive filtering techniques because XPM introduces fast, time-varying phase noise. In the presence of XPM, early methods such as the mean-squared differential phase (MSDP) estimator enhanced the adaptability of CPR filtering [23]. Recent advancements include joint compensation systems that include phase error correction and nonlinear distortion reduction in one step [24] and low-latency, hardware-optimized CPR architectures [25]. Furthermore, to achieve significant gains in bit error rate (BER) and system robustness, kernel-based online phase recovery techniques have been proposed to dynamically track phase noise in systems using cascaded parametric amplifiers [26].

Although adaptive carrier phase recovery (CPR) and overlap frequency domain equalization (OFDE) are both well established in coherent optical receivers, their combined use in high data rate systems is still relatively new. A unified digital signal processing (DSP) architecture that can effectively handle both linear and nonlinear impairments is provided by combining adaptive CPR with cross-phase modulation (XPM) mitigation and OFDE for chromatic dispersion (CD) compensation. Low-latency, hardware-efficient CPR implementations in high-speed coherent systems [25], joint compensation schemes that integrate phase noise and nonlinear distortion recovery [24], and sophisticated carrier phase estimation algorithms for next-generation systems [27] have all been shown to have substantial advantages in recent works.

This work investigates the performance of this integrated approach, demonstrating enhanced Q factor performance across varying FFT block and overlap lengths, with experimental results indicating optimal configurations for fiber spans of up to 2400 km. The rest of this paper is organized as follows. Section 2 details the OFDE methodology for CD compensation, which is the existing limitation; Section 3 explores our contribution of the proposed method for improving digital phase carrier recovery by using adaptive CPR with XPM filtering; and Section 4 presents the Comparative Benchmarking and Complexity–Performance Trade-off Analysis section. Finally, our conclusions are drawn in Section 5 regarding the effectiveness of this integrated DSP approach.

2. Efficient Chromatic Dispersion Compensation

Chromatic dispersion (CD) represents a major signal impairment that limits performance in long-haul optical communication systems, especially at high bit rates. Traditionally, systems operating at 10 Gbit/s and 40 Gbit/s have relied on in-line optical compensation techniques, such as dispersion-compensating fibers (DCFs), fiber Bragg gratings (FBGs), and etalon-based structures, to mitigate CD. In coherent systems, however, CD compensation is more effectively performed in the digital domain [28]. This approach employs an inverse linear finite impulse response (FIR) filter, eliminating the need for complex optical dispersion compensation elements. Despite its advantages, DSP-based CD correction dramatically raises circuit complexity and power consumption, particularly as the adjusted dispersion lengthens [17]. Recent research has addressed this by introducing computationally efficient alternatives that offer significant complexity and energy savings, such as clustered time-domain equalizers validated via FPGA implementations [17] and high-radix Fermat number transform-based equalizers [29].

Typically, in DSP-based coherent receivers, CD compensation is performed before polarization mode dispersion (PMD) compensation and is achieved through time-domain equalization (TDE) using FIR filters. Although TDE provides accurate CD correction, it requires a large number of taps, making it computationally intensive for long-haul communications. As an alternative, frequency-domain equalization (FDE) reduces computational complexity by processing signals in blocks using a fast Fourier transform (FFT) with an added guard interval to eliminate interference from the cyclic nature of FFTs [30,31]. While FDE is less computationally demanding than TDE for large tap sizes, it necessitates larger block sizes in long-haul systems to handle extensive CD. However, the required guard intervals in FDE can degrade transmission efficiency, and large block-to-block operations increase complexity and introduce potential synchronization errors [12].

To address these limitations, Kudo et al. proposed overlap frequency domain equalization (OFDE) as a solution [17]. Unlike standard FDE, OFDE does not require a guard interval, thus preserving the transmission efficiency. By carefully setting the overlap size of FFT windows, OFDE improves transmission performance by reducing inter-block interference (IBI). A block diagram illustrating the OFDE process is shown in Figure 2.

The efficiency of OFDE, and frequency-domain equalization in general, lies in its compatibility with application-specific integrated circuits (ASICs) using power-of-two FFT implementations. For FIR filters with more than four taps, frequency-domain implementations can be highly efficient when using FFT block lengths that are powers of two, optimized for both the block size and overlap length. By applying the radix-2 FFT algorithm, the FFT operation can be implemented with $N{log}_2\left(N\right)$ real multiplications, with additional real multiplications required for FDE. For a complete FDE implementation, this amounts to $2N{log}_2\left(N\right)+4N$ real multiplications per polarization, as shown in Equation (1) [18]:

$$\mathrm{Complexity}=2N{log}_2\left(N\right)+4N$$

(1)

In the OFDE approach, an FFT is applied to $N_c$ symbols in each received signal block, with $N_e$ overlap symbols and $N_0$ symbols transferred to the equalized output. The FDE block multiplies the frequency-domain representation of the received signal by a transfer function based on the fiber’s dispersion characteristics. This function, shown in Equation (2), compensates for the phase shift introduced by CD:

$$H\left(f\right)=exp\left(-j{\displaystyle \frac{\pi D{\lambda }^2{f}^{2}L}{c}}\right)$$

(2)

where D is the dispersion coefficient, $\lambda$ is the optical wavelength, L is the transmission distance, f is the baseband frequency, and c is the speed of light. The dispersion slope is generally negligible, as it has a minimal effect compared with primary CD [17].

This study evaluates OFDE performance for CD compensation in a 112 Gbit/s PM-QPSK signal. Unlike previous studies, where residual inter-symbol interference (ISI) was managed using a small TDE filter, this work utilizes an adaptive equalizer to mitigate ISI. The experimental data from J. C. Cartledge et al. includes transmission tests with 16 distributed feedback (DFB) lasers separated by 50 GHz and modulated by QPSK signals at 28 Gsym/s. These signals were polarization-multiplexed and transmitted over a 100 km span of Corning® LEAF® fiber, featuring a dispersion parameter of $D=4$ ps/nm/km. The Q factor was calculated from the bit error rate (BER) via direct error counting, with the total fiber spans tested up to 2400 km [32].

The overall experimental set-up of the system is described in Figure 3 [33].

Figure 3 shows the experimental configuration for nine-channel DWDM transmission with a 25 GHz spacing. The 128 Gb/s DP 8-QAM signals were generated at a baud rate of 28 Gbaud by mapping a $2^{19}$ de Bruijn bit sequence to symbols. A Ciena WaveLogic 3 transceiver with four synchronized 39.4 GSa/s DACs and a DP IQ modulator acting as an arbitrary optical waveform generator was used to store the waveforms for the observed channel (channel 5). A second WaveLogic 3 transceiver with eight CW input signals was used to generate the neighboring channels independently. Using a 64-tap impulse response and an oversampling rate of 1.85 samples per symbol, digital pulse shaping was carried out offline. The DACs’ inverse frequency responses were used to offset the signal sample values. The produced signals were fed into a gain equalizer $GE$, a loop synchronous polarization scrambler (LSPS), and a recirculating loop with four spans. Each span included an optical bandpass filter (OBPF) with a bandwidth of 18 nm, 100 km of standard SMF, and an erbium doped fiber amplifier (EDFA) with a noise figure of 5 dB. The attenuation per span was 15.5 dB, and the dispersion was 1275 ps/nm. The OBPFs kept out-of-band amplified spontaneous emission noise from overwhelming the EDFAs in the loop. A polarization- and phase-diverse coherent receiver with a 32 GHz bandwidth detected the received signal after it had been amplified and filtered (1.3 nm bandwidth). The nominal linewidth of the local oscillator laser was 100 kHz. Two synchronized real-time sampling oscilloscopes were used by 80 GSa/s ADCs to digitize the four signals from balanced photodetectors [33].

As illustrated in Figure 4, increasing the FFT block length generally enhanced OFDE accuracy and Q factor performance up to an optimal overlap size. Beyond this optimal overlap length, performance declined due to the cyclic properties of the FFTs at the block edges. The adaptive equalizer, though effective, was unable to fully correct noise when the overlap was less than 256 samples, resulting in a degraded constellation. Conversely, with $N_e=256$, the adaptive equalizer effectively compensated for noise, as can be seen on the right side of Figure 4.

Further analysis involved varying FFT block lengths from 256 to 2048 samples with a fixed overlap length of 128 samples for distances of 1600 km, 2000 km, and 2400 km. The results are shown in Figure 5, indicating that OFDE performance marginally improved with longer FFT blocks at fixed overlap lengths.

In summary, OFDE provides a computationally efficient method for CD compensation, balancing processing complexity and Q factor performance. By combining OFDE with an adaptive equalizer, coherent optical systems achieve high data rates and extended reaches, minimizing power consumption and hardware requirements for 112 Gbit/s PM-QPSK transmissions.

3. Improvements to Digital Phase Carrier Recovery

Carrier phase recovery (CPR) is a crucial signal processing step in coherent optical receivers, enabling accurate demodulation of phase-modulated signals. CPR techniques can generally be categorized into two main approaches: backward feed phase-locked loops and forward feed phase estimation methods. Among these, the Viterbi-Viterbi (V-V) algorithm has gained popularity and is widely used in QPSK coherent receivers due to its effectiveness and low computational complexity [1,22,25].

The V-V algorithm is a forward feed phase estimation method which has advantages over backward feed algorithms as it avoids loop delay issues. The V-V algorithm operates under the assumption that the carrier phase remains relatively constant over a certain averaging period. It estimates the carrier phase by raising the received symbol to the fourth power (for QPSK signals), which suppresses phase noise through an averaging process. The final carrier phase estimate is obtained by dividing the argument of the average by four. The basic steps of the V-V algorithm are illustrated in Figure 6.

The performance of the V-V algorithm depends on the choice of the averaging length. A longer averaging length enhances noise suppression but reduces phase tracking capability, while a shorter averaging length improves phase tracking but is less effective in noise reduction. The V-V algorithm’s assumption of a constant carrier phase is, however, invalid in scenarios with frequency offset and cross-phase modulation (XPM) [1,22].

To address these limitations, a frequency offset estimation step, as shown in Figure 1, is typically applied prior to CPR. However, XPM remains a significant nonlinear effect that limits long-haul transmission performance in dense wavelength-division multiplexed (DWDM) systems. XPM-induced phase noise is influenced by various factors such as the dispersion, data rate, and especially launch power. Yan et al. introduced an XPM monitor based on a mean-squared differential phase (MSDP) estimator, which accurately determines XPM autocorrelation with minimal complexity [23]. This XPM monitor can dynamically adjust filter coefficients for optimal phase noise mitigation, enhancing the basic V-V algorithm, as illustrated in Figure 7.

3.1. Optimal Filtering Using XPM and ASE Noise Correlation

Optimal filtering in the presence of XPM and amplified spontaneous emission (ASE) noise can be achieved by calculating a set of filter coefficients H based on the autocorrelation and cross-correlation matrices R and X as shown in Equation (3):

$$H=R^{-1}X$$

(3)

where H is the vector of filter coefficients, R is the autocorrelation matrix of the input, and X is the cross-correlation vector between the input and desired output. These parameters are defined by

$$\begin{array}{cc}\hfill H& =\left[\begin{array}{c}h(-L)\\ h(-L+1)\\ ⋮\\ h\left(L\right)\end{array}\right]X=\left[\begin{array}{c}{R}_{xy}\left(L\right)\\ R_{xy}(L-1)\\ ⋮\\ R_{xy}(-L)\end{array}\right]\hfill \\ \hfill R& =\left[\begin{array}{cccc}{R}_{xx}\left(0\right)& R_{xx}\left(1\right)& \dots & R_{xx}\left(2L\right)\\ R_{xx}\left(1\right)& R_{xx}\left(0\right)& \dots & R_{xx}(2L-1)\\ ⋮& ⋮& \ddots & ⋮\\ R_{xx}\left(2L\right)& R_{xx}(2L-1)& \dots & R_{xx}\left(0\right)\end{array}\right]\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill R_{xx}\left(k\right)& =exp\left\{M^2{\sigma }_c^2\left[\rho \left(k\right)-1\right]\right\}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill R_{xy}\left(k\right)& =exp\left\{M^2{\sigma }_c^2\left[\rho \left(k\right)-1\right]\right\}+M^2{\sigma }_n^2\delta \left(k\right)\hfill \end{array}$$

(6)

where ${\sigma }_c^2$ is the variance of XPM-induced phase noise, ${\sigma }_n^2$ is the variance of ASE-induced noise, M is the modulation index (four for QPSK), $\rho \left(k\right)$ is the normalized autocorrelation of the XPM noise, and $\delta \left(k\right)$ is the Kronecker delta function.

Amplified spontaneous emission (ASE) and cross-phase modulation (XPM)-induced phase noise for polarization division multiplexed (PDM) signals show cross-polarization correlations, necessitating an effective filtering technique that takes inter-polarization noise dependencies into consideration. However, this study applies optimum filtering independently to the X and Y polarizations and assumes independence between the polarization noise components for analytical tractability. García-Gómez and Kramer [34] provided a more rigorous model that specifically tackles these inter-polarization dependencies and their effect on achievable information rates.

In this investigation, XPM autocorrelation is calculated using both MSDP and offline phase error estimation over 56,000 received symbols. In our offline analysis, we assumed that variations in phase noise autocorrelation were attributed to XPM effects. This assumption allowed estimation of XPM noise autocorrelation, assuming dependencies only within the CPE block symbols. The experimental results, as shown in

Table 1. Performance of the V-V algorithm with and without optimal filtering for various transmission distances.

Distance (km)	V-V Algorithm (Q, dB)	Optimal Filtering (AutoCorr)	Optimal Filtering (MSDP)
1600	11.60	11.68	11.53
2000	10.36	10.33	10.32
2400	9.93	9.92	9.83

, yielded a comparison of the Q factor performance of the V-V algorithm with and without optimal filtering for various transmission distances.

As shown in Figure 8, the autocorrelation of XPM noise for a 2400 km span suggests that the XPM-induced noise exhibited dependencies across both polarizations. Therefore, an enhanced model that includes statistical dependencies across X and Y polarizations may yield further performance gains.

In another scenario, optimal filtering was tested under high launch powers to evaluate the performance of the modified algorithm. Carrier phase error (CPE) block lengths of 11, 21, and 41 symbols were applied, along with OFDE compensation at (2048, 256) and (512, 256). The Q factor performance for 2400 km transmission when utilizing FDE compensation is shown in

Table 2. Performance of the V-V algorithm with and without optimal filtering for various launch powers.

Launch Power (dBm/ch)	V-V Algorithm (Q, dB)	Optimal Filtering (AutoCorr)	Optimal Filtering (MSDP)
5	11.76	11.76	11.72
7	11.14	11.08	11.04
9	9.49	9.47	9.45

at launch powers of 5, 7, and 9 dBm/ch. The maximum Q factor was recorded at 5 dBm/ch. In the next paragraph, the effects of the FFT block length and overlap are covered individually.

A closer analysis of the XPM autocorrelation (Figure 9) revealed minimal gains from optimal filtering at higher powers, potentially due to underlying dependencies in the XPM noise across polarizations. Further modeling is necessary to fully account for these dependencies.

It is evident that while optimal filtering with XPM monitoring enhances phase noise mitigation in CPR, the full benefits in dual-polarization transmission require advanced models that include cross-polarization noise dependencies.

The results from optimum filtering using the autocorrelation method showed minimal deviation, remaining within ±0.08 dB of the uniform averaging approach used in the conventional V-V algorithm. Additional simulations with a carrier phase error (CPE) block length of 41 symbols, combined with OFDE compensation settings of $(N_c,N_e)=(2048,256)$ and $(512,256)$, yielded similar findings, although their detailed results are omitted here. Typically, the performance of the mean-squared differential phase (MSDP) method was observed to be 0.03–0.3 dB lower than that of the autocorrelation method across various configurations.

However, multiple failures in MSDP optimal filtering were recorded, particularly with a CPE block length of 41 symbols. In these cases, the phase estimator for one polarization failed to accurately estimate the XPM autocorrelation, resulting in an irrelevant optimal filter and a significant error in carrier phase estimation. Consequently, the bit error rate (BER) approached one for the affected polarization, while the opposite polarization maintained adequate performance. The lower computational requirements of the MSDP framework allow it to capture XPM noise statistics more gradually, which explains its occasionally unreliable performance compared with the autocorrelation method. In scenarios with slowly varying XPM noise statistics, the MSDP technique, when applied adaptively over long data sequences, demonstrated performance similar to the autocorrelation method. Despite the simulations’ assumption of polarization-independent noise to isolate DSP performance, further testing showed that the MSDP-CPR algorithm was more sensitive to polarization misalignment, especially at higher launch powers. Subsequent research will examine these impacts in mixed fiber types and span combinations to more accurately depict cross-polarization behavior.

The FFT block length selection had a significant impact on the MSDP-based carrier phase recovery (CPR) algorithm’s performance. Our simulations show that MSDP became unstable at longer block lengths, especially when the number of symbols exceeded 41. This is because the MSDP filter became less sensitive to abrupt phase changes and inter-symbol phase noise as block sizes increasd due to the decreased temporal resolution. Furthermore, in dynamic phase environments, larger blocks tend to average out short-term phase variations, which might cause tracking failure or delay convergence. The reduced Q factor values at large block lengths (see Table 2) clearly demonstrate this pattern. On the other hand, while smaller block sizes increase computational load and noise susceptibility, they also improve responsiveness. Thus, choosing the right block length is essential to preserving a steady trade-off between noise resilience and tracking agility. To increase MSDP stability across a larger parameter range, future research might investigate hybrid filtering techniques or adaptive block sizes.

These findings show that in long-haul dual-polarization optical transmission systems, the generally accepted independence of polarization noise does not always translate into appreciable performance improvements in low cross-phase modulation (XPM) regimes. Even when the XPM effects are minimal, nonlinear interactions cause residual correlations between polarization components, as demonstrated by García-Gómez and Kramer [34]. This may be seen in the nonzero ideal filter coefficients at the tap weights’ edges, which are characteristic of single-channel transmission situations when nonlinear crosstalk is not present.

3.2. System Parameter Sensitivity and Physical Layer Considerations

In order to gain a better understanding of the robustness and practical limitations of the proposed DSP framework, we examined two important aspects: (1) how system performance is affected by DSP configuration parameters like the overlap factor and FFT block length and (2) the possible impact of physical layer effects, especially nonlinear dispersion, which were not explicitly modeled. This section offers a more thorough examination in order to put the observed Q factor trends in context and make the modeling assumptions used in this work clearer.

3.2.1. Sensitivity to FFT Block Length and Overlap Factor

In order to assess the robustness of the suggested framework, we performed a sensitivity analysis of the overlap percentage and FFT block length selection. Table 1 and Table 2 demonstrate that across a range of launch powers and transmission distances, the best Q factor performance was obtained with an FFT size of 512 and 50% overlap. The trends in Figure 4 and Figure 5 further support the idea that extending the FFT block length over 512 results in minimal improvements and may lead to problems with convergence during the CPR stage. However, dispersion compensation noticeably degraded when the FFT size decreased below 256. In a similar vein, overlap factors below 25% resulted in more inter-block interference, but overlap factors above 50% provided rather little improvement at a considerable computational cost. These results emphasize how crucial it is to choose FFT values that strike a compromise between implementation viability and performance.

3.2.2. Impact of Nonlinear Dispersion

Long-haul coherent optical systems are known to be affected by nonlinear dispersion effects, such as self-phase modulation (SPM) and cross-phase modulation (XPM). Nevertheless, nonlinear dispersion was not specifically considered in this investigation. Our calculations’ moderate launch power domain (5–9 dBm), where linear impairments like chromatic dispersion (CD) and polarization mode dispersion (PMD) predominate the system behavior, justifies this reduction. Furthermore, rather than thoroughly modeling all physical layer nonlinearities, the goal of this work was to assess the DSP framework’s performance and complexity trade-off under realistic transmission scenarios. However, the Q factor measurements showed patterns of signal degradation that implicitly accounted for the residual effects of nonlinearity (see Table 1 and Figure 5). Full nonlinear Schrödinger equation-based simulations will be used in future research to examine the relationship between digital compensation methods and nonlinear dispersion.

4. Comparative Benchmarking and Complexity–Performance Trade-Off Analysis

A comparative benchmark against contemporary state-of-the-art methods published after 2021 was carried out in order to contextualize the effectiveness and performance of the suggested DSP framework. This covered CPR methods that are based on kernels, transformers, and machine learning. A complexity versus performance trade-off study and the test results demonstrated the usefulness of the suggested approach, which provides long-haul coherent optical systems with a balanced combination of high Q factor performance and less computational burden.

4.1. Comparative Benchmarking with Latest Results

In order to assess the effectiveness of current digital signal processing (DSP) methods for coherent optical communication systems, we carried out a comparison benchmark with an emphasis on peer-reviewed research after 2021. In the benchmark, different DSP frameworks for higher-order modulation formats and polarization-multiplexed quadrature phase-shift keying (PM-QPSK) in long-distance and metro-distance systems were compared. Our study, which combined adaptive carrier phase recovery (CPR) with cross-phase modulation (XPM) filtering and overlap frequency domain equalization (OFDE) for chromatic dispersion (CD) compensation, showed a competitive Q factor performance of 9.9–11.7 dB over distances up to 2400 km while keeping the computational complexity low. In contrast, Kherici et al. [19] used a CO-OFDM-based method to attain comparable Q factors (10.2–11.3 dB), albeit with more complicated equipment. Superior Q factors of 11–12.8 dB over long distances were achieved by more sophisticated machine learning (ML) and kernel-based CPR methods put forth by Neves et al. [27] and Nguyen et al. [26], but at the expense of noticeably greater DSP complexity and implementation overhead. In the meantime, Wang et al. [35] presented a highly effective DSP strategy that uses non-integer oversampling to achieve significant complexity reductions without sacrificing sensitivity for 400 Gb/s coherent PON systems. The expanding importance of deep learning models in optical DSP was highlighted by Gautam et al. [36], who showed that transformer-based nonlinear equalizers performed better than neural networks and traditional digital backpropagation (DBP) for dual-polarization 16QAM long-haul systems. To further highlight the significance of XPM-aware compensation schemes, Chen et al. [37] introduced a dual-OSC coding strategy that mitigates XPM-induced penalties in DWDM coherent connections, delivering a 1.3 dB Q factor improvement for 200 Gb/s DP-16QAM across 1618 km. This benchmark demonstrates that although kernel-based and AI-optimized approaches provide the best absolute Q factor performance, solutions such as our OFDE with adaptive CPR continue to be quite competitive for 112 Gb/s PM-QPSK systems because of their advantageous trade-off between long-distance performance and computational efficiency.

Table 3. Comparative benchmark with recent DSP studies.

Study and Year	Rate/Modulation	Max Distance	DSP Method	Q Factor (dB)
Our study	112 Gb/s PM-QPSK	2400 km	OFDE + MSDP-CPR (adaptive)	9.9–11.7
Wang [35]	400 Gb/s coherent PON via SCM	Metro (<100 km)	Non-integer oversampling DSP	N/A (maintained)
Gautam [36]	DP-16QAM (long-haul)	≥2000 km	Transformer-based nonlinear equalizer	>DBP baseline
Kherici [19]	112 Gb/s CO-OFDM QPSK	2000 km	CO-OFDM + adaptive CD/PMD/CPR	10.2–11.3
Neves [27]	100–400 Gb/s DP-QPSK/16QAM	3000 km	ML-based CPR	11–12.8
Lin [25]	200 Gb/s DP-QPSK	1600 km	Low-latency FPGA-based CPR hardware	11.5–12.5
Chen [37]	200 Gb/s DP-16QAM	1618 km	Dual-OSC coding XPM mitigation	+1.3 dB gain
Karar [33]	112 Gb/s PM-QPSK	2000 km	Polynomial pulse shaping for NL mitigation	∼11.0

provides a comparative benchmark with recent DSP studies.

The computational complexity of each key DSP module was quantitatively analyzed in order to support the performance–complexity trade-off that is highlighted in this work. While the MSDP-based CPR introduced a cost of $\mathcal{O}\left(N^2\right)$ per symbol due to matrix-based filtering, the OFDE displayed a complexity of $\mathcal{O}\left(2N{log}_{2}N\right)$. The complexity of kernel-based or ML-driven CPR techniques could reach $\mathcal{O}\left(M^3\right)$, while that of standard autocorrelation-based CPR was $\mathcal{O}\left(N{log}_{2}N\right)$. Table 3 highlights the proposed DSP architecture’s ability to balance efficiency and signal fidelity by summarizing these findings and comparing our method to recent research like Lin et al. [25] and Karar et al. [33].

Our proposed system achieved competitive Q factors while maintaining manageable complexity, especially compared with high-complexity ML approaches like those in [27,36]. These improvements support our aim to deliver a scalable, real-time-compatible DSP framework suitable for long-haul coherent optical networks.

4.2. Complexity Versus Performance Trade-Off Analysis

To contextualize the effectiveness of the suggested DSP framework, a complexity versus performance trade-off study was conducted in addition to Q factor performance. Although Neves et al. (2024) [27] and Nguyen et al. (2024) [26] demonstrated that AI-enhanced and kernel-based CPR techniques had higher Q factors in the range of 11.5–12.8 dB, they are less feasible for hardware-constrained long-haul systems because they require significantly more computational resources due to iterative phase estimation, large training datasets, and inference overhead. By using OFDE and adaptive CPR with XPM filtering, our approach, on the other hand, maintains relatively low DSP complexity through FFT-based block processing and lightweight adaptive phase recovery algorithms while achieving competitive Q factor values up to 11.7 dB over 2400 km. Though restricted to short distances and sub-400 Gb/s rates, techniques such as those of Wang et al. (2024) [35], which were aimed at metro access networks, achieved excellent complexity reductions via non-integer oversampling. Furthermore, a low-complexity dual-OSC coding technique that successfully reduces XPM-induced penalties in DWDM systems was presented by Chen et al. (2023) [37]. This scheme achieved a 1.3 dB Q factor gain without additional DSP overhead. This analysis shows that although advanced AI models can achieve maximum Q factor performance, systems like those developed by our study achieve a critical balance between DSP burden and performance, which makes them more appropriate for long-distance, scalable 112 Gb/s PM-QPSK installations.

Table 4. Complexity versus performance trade-off comparison for recent DSP techniques.

Study and Year	Q Factor (dB)	DSP Complexity	Scalability	Notes
Our study	9.9–11.7	Low	Excellent	FFT-based CD compensation + adaptive CPR
Wang [35]	N/A (maintained)	Very Low	Limited (metro only)	Non-integer oversampling DSP for PON
Gautam [36]	>DBP baseline	High	Moderate	Transformer-based nonlinear equalizer
Kherici [19]	10.2–11.3	Moderate	Good	CO-OFDM with adaptive CD/PMD/CPR
Neves [27]	11–12.8	Very High	Moderate	ML-based CPR, data- and computation-heavy
Nguyen [26]	11.5–12.5	High	Good	Kernel-based online phase recovery
Chen [37]	+1.3 dB gain	Low	Good	OSC-based XPM suppression without DSP overhead
Karar [33]	∼11.0	Low	Good	Polynomial pulse shaping for nonlinear mitigation

summarizes the complexity versus performance trade-off comparison for recent DSP techniques.

5. Conclusions

This study evaluated an integrated DSP approach combining OFDE for CD compensation with adaptive CPR utilizing XPM monitoring in coherent optical systems. The proposed method addresses both linear and nonlinear impairments in 112 Gbit/s PM-QPSK systems, which are critical for high-capacity, long-haul optical communications.

The experimental results demonstrate that OFDE is an effective alternative to conventional TDE, achieving CD compensation with reduced computational complexity. When combined with adaptive CPR that dynamically responds to XPM effects, this integrated DSP framework enhances phase noise mitigation, especially in DWDM systems, where nonlinear phase noise poses a significant challenge. Our findings indicate that this approach maintains high Q factor performance over transmission distances up to 2400 km, highlighting its suitability for long-haul, high data rate applications.

However, marginal gains in high-power scenarios suggest that optimal filtering with XPM monitoring does not fully capture the dependencies between XPM-induced noise across polarizations. This implies that dual-polarization systems may benefit from DSP models incorporating statistical dependencies between X and Y polarization channels, potentially enhancing robustness under high launch power conditions.

The suggested DSP system is scalable to data rates beyond 112 Gb/s, since its modular structure allows adaptation to higher symbol rates through increasing FFT precision and processing speeds. While this imposes additional computing demands, the architecture can be implemented on parallel hardware platforms such as FPGAs or GPUs.

A comparative benchmark and complexity trade-off analysis further validated the proposed framework, confirming its ability to deliver strong Q factor performance with significantly lower DSP complexity than recent AI-based and kernel-based approaches. This positions the solution as a practical and scalable option for long-haul 112 Gb/s PM-QPSK systems, balancing performance, efficiency, and implementation feasibility.

In conclusion, combining OFDE with adaptive CPR and XPM monitoring offers a computationally efficient solution for advanced optical networks, balancing processing demands with performance gains. Building on the current findings, future research will focus on four key directions: incorporating full nonlinear fiber effects using the nonlinear Schrödinger equation to evaluate DSP robustness under stronger nonlinearities, improving the stability and adaptivity of the MSDP-CPR through dynamic block sizing or hybrid filtering methods, and extending the proposed framework to support higher-order modulation formats (e.g., 16-QAM) and data rates beyond 200 Gb/s using optimized hardware platforms, such as FPGAs or GPUs. Additionally, we plan to validate the proposed algorithms using industry-grade simulation environments such as Optiwave OptiSystem, enabling the inclusion of physical layer impairments such as hardware jitter, modulator bandwidth limitations, and fiber nonlinearities. This will help assess the robustness and practical deployment feasibility of the system under real-world conditions. These efforts aim to enhance the practical applicability of the system in next-generation high-speed optical networks.