Enhanced Machine Learning-Based SDM-QAM Transmission Using Low-Cost Fast-OFDM

Abstract

This paper presents a novel integration of quadrature amplitude modulation (QAM)-based fast optical orthogonal frequency-division multiplexing (F-OFDM) with machine learning (ML)-based equalization in spatial division multiplexing (SDM) applications, using few-mode fibers (FMFs). The FMFs support four LP modes, resulting in a total of 12 orthogonal modes, each accommodating two polarizations. A digital multiple-input multiple-output channel equalizer is employed at the receiver’s digital signal processing (DSP) unit to effectively mitigate channel crosstalk. The study harnesses supervised ML-DSP techniques, in particular recurrent neural networks (RNNs) and deep neural networks (DNNs), achieving substantial reductions in bit error rates (BERs). In addition, higher-complexity architectures, namely convolutional neural networks (CNNs) and long short-term memory (LSTM) networks, are evaluated to assess the impact of advanced spatial and temporal feature extraction. It is shown that F-OFDM demonstrates superior performance over conventional optical OFDM, particularly when supported by ML techniques. Simulation results reveal that RNNs achieve a BER of 0.0019 over 15 km at 12 Gbaud (worst-case selected channel), showcasing a remarkable 52.5% improvement compared to linear equalization. DNNs achieve a BER of 0.0025, reflecting a 37.5% enhancement. While RNNs perform better, their computational demands pose challenges for real-time applications, and the more complex models (CNN and LSTM) do not provide additional performance gains. The paper also explores cyclic prefix management and subcarrier number strategies in F-OFDM to optimize performance, paving the way for future advancements in SDM networks.

1. Introduction

OFDM has been standardized in 5G wireless communications, exhibiting greater resistance to frequency-selective fading compared to single-carrier systems [1]. In contrast to single-carrier systems, which have a broad and leaky spectrum (such as raised cosine shapes with an excess factor), the OFDM spectrum is rectangular [1]. This rectangular configuration confines nearly all the energy within the defined spectrum, resulting in a narrower, higher spectral efficiency. On the other hand, optical-orthogonal frequency division multiplexing (O-OFDM) can effectively eliminate intersymbol interference (ISI) caused by fiber chromatic dispersion (CD) using simple, single-tap equalization [2,3,4,5]. Consequently, O-OFDM reduces computational complexity and cost compared to single-carrier m-QAM. The latter requires multi-tap equalizers, such as feed-forward equalization or time-domain equalization (TDE-MIMO), implemented within a more complex DSP unit [3]. In DP-QAM systems, polarization mode dispersion (PMD) is stochastic. Therefore, the finite impulse response (FIR) taps in TDE-MIMO must be continuously updated for each received symbol. In contrast, O-OFDM incorporates a straightforward cyclic prefix (CP) for each symbol to minimize ISI resulting from either CD or PMD. This enhancement demonstrates the potential of DP O-OFDM to serve as a replacement for traditional DP-QAM systems. Other benefits of OFDM include the parallelization with fast Fourier transform—a benefit for field programmable gate arrays (FPGAs)—and signal-to-noise ratio (SNR) monitoring in MHz or kHz granularity at any point in the spectrum [6].

Optical fast-OFDM (F-OFDM) has emerged as a promising updated multicarrier solution to OFDM, in order to further reduce the subcarrier spacing to half the symbol rate per subcarrier [7,8]. This innovative approach offers significant advantages over conventional OFDM, including improved spectral efficiency, lower energy consumption, and reduced latency [7,8]. Additionally, coherent optical F-OFDM enhances the frequency offset compensation in coherent detection and boosts resilience to phase noise in self-coherent detection [7]. In traditional OFDM, MHz-spaced parallel subcarriers form low-capacity data transmission signals. On the other hand, F-OFDM, which utilizes the compression properties of the inverse fast cosine transform (IFCT), reduces the frequency spacing between subcarriers, offering improved bandwidth efficiency compared to standard O-OFDM, as it employs half the subcarrier spacing [7,8]. However, the sub-MHz spacing can cause substantial distortion in phase-modulated signals, making it crucial to use single-dimensional modulation formats like amplitude shift-keying (ASK) [9]. Previous demonstrations have shown IFCT-based optical F-OFDM for long-haul coherent transmission using optical double-sideband signals [7], using a special configuration for 16QAM modulation [10], and alternatively using multi-tap equalization for performance improvement, which sacrifices complexity [11]. As a more straightforward implementation, direct-detected optical F-OFDM has also been applied in cost-sensitive local networks utilizing multimode fiber links, given the technique’s cost-effectiveness [8,9]. Nonetheless, like conventional O-OFDM, F-OFDM signals suffer from a high peak-to-average power ratio, leading to comparable transmission performance under similar signal capacities. In the work reported by Giacoumidis et al. [12], the dynamic parameter requirements of sub-ranging quantizers—identified as digital-to-analog and analog-to-digital converters (DACs/ADCs)—involved in optical F-OFDM signals were also analyzed, serving as a proof-of-concept for future real-time implementations.

To further increase signal capacity, space division multiplexing (SDM) employing m-QAM has been proposed mainly in the realm of few-mode fibers (FMFs) and multicore fibers [13,14,15]. By leveraging multiple spatial modes within a single fiber, SDM-QAM and SDM-OFDM [13,14,15,16] enhance data transmission capacity while maintaining high spectral efficiency. This approach is particularly beneficial for expanding bandwidth in optical communication systems, with direct relevance to high-capacity short-reach scenarios such as data center interconnects, as well as scalable metro and backbone networks envisioned for future 6G infrastructures [17,18]. Additionally, it promotes the efficient use of existing fiber infrastructure, reducing costs and power consumption while increasing overall network throughput. These are key requirements for both energy-efficient data centers and ultra-dense 6G communication frameworks. O-OFDM for SDM applications has been implemented for 314-Tbit/s (576 × 380.16 MHz 5G OFDM signals) wavelength-division-multiplexing (WDM)-based intermediate frequency (IF)-over-fiber transmission for analog mobile fronthaul [14]. SDM superchannel transmission over FMF with a data rate of 100 Gb/s and beyond has been recently demonstrated through coherent O-OFDM and single-carrier superchannel transmission [15]. For equalization purposes in SDM, Volterra-based nonlinear equalization has been proposed in [19] to reduce inter-modal nonlinear penalties in two-mode fiber (TMF). It was shown that for ~260 Gb/s DP coherent O-OFDM at 1040 km, the TMF inter-modal nonlinear penalty can be reduced by up to ~4 dB [19].

On the other hand, machine learning (ML) has become prevalent across various fields, and its use in optical communications is rapidly increasing due to the growing global internet traffic. Specifically, in the task of enhanced equalization, it has been utilized for coherent and direct-detection optical systems. In O-OFDM systems, unsupervised clustering techniques and advanced supervised ML techniques have been used for enhanced equalization, such as deep neural networks (DNNs) and support vector machines (SVMs) [20,21,22,23]. In [23], direct-detection SDM-OFDM was considered using an SVM-based equalizer. A convolutional neural network (CNN) for nonlinear equalization has also been implemented in conventional 16QAM OFDM, as reported in [24]. For F-OFDM, a low computational load artificial neural network (ANN) design was compared to the benchmark inverse Volterra series transfer function (IVSTF)-based compensator for long-haul single-mode fiber links at 9.69 Gb/s [25]. Results showed that ANN enabled an 80 km extension in transmission reach, effectively addressing stochastic nonlinear impairments like parametric noise amplification [25]. Additionally, ANN relaxed the dynamic parameter requirements of sub-ranging quantizers, reducing the optimum clipping ratio and quantization bits by 2 dB and 2 bits compared to both linear equalization and the IVSTF equalizer [25].

Despite the growing body of work on ML-based equalization in optical systems, prior studies have primarily focused on single-mode fiber (SMF) transmission or conventional O-OFDM systems [20,21,22,23,24,25]. Similarly, while F-OFDM has been investigated in both direct-detected and coherent optical links [7,8,9,10,11,12,25], its application has largely been limited to single-channel or single-mode scenarios, without considering the additional complexity introduced by SDM over FMFs. To the best of our knowledge, this work represents the first comprehensive investigation of coherent optical F-OFDM combined with ML-based equalization in an SDM-FMF transmission environment. The novelty of this work lies in three key aspects:

(i)

The integration of coherent F-OFDM with SDM over FMFs, enabling the evaluation of multicarrier transmission under strong inter-modal crosstalk and modal dispersion;

(ii)

The application and comparison of supervised ML equalizers, specifically DNN, CNN, and bidirectional recurrent neural networks (RNN) and long-short term memory (LSTM) architectures within a high-dimensional MIMO-SDM receiver framework;

(iii)

A systematic performance and complexity analysis highlighting the trade-off between BER improvement and computational cost in SDM systems, which is not addressed in prior ANN/DNN-based optical equalization studies.

Unlike previous ML-based equalization approaches that target predominantly temporal impairments in SMF systems, the proposed approach addresses both temporal and spatial impairments inherent to SDM transmission, including inter-modal coupling and differential mode delay. This distinction is critical in demonstrating the effectiveness and limitations of ML equalizers in next-generation high-capacity optical networks.

2. Methods

A coherent optical F-OFDM system was generated, similar to that described in [10] for 16QAM. Two bipolar 4ASK data sets were encoded using Gray code in Matlab (Matlab^® R2023b). A general transmission link diagram for the x-polarization is shown in Figure 1a for both coherent optical OFDM and F-OFDM cases. The IFCT and FCT processes implemented 128 points, with 100 subcarriers designated for F-OFDM data transmission and 6 for phase estimation. The first two subcarriers were left unmodulated to facilitate the use of AC-coupled drive amplifiers and receivers. The generation and decoding processes were conducted using Matlab and integrated into the VPIphotonics system for the optical components and the fiber. Key steps in the F-OFDM signal creation included serial-to-parallel (S/P) conversion, symbol mapping, and an optimum 10% CP was inserted to mitigate ISI. For fair comparison, identical parameters were selected for the conventional coherent optical OFDM system.

The F-OFDM/OFDM system generated an optical polarization-multiplexed QAM signal using a pair of single-drive MZMs [26] to independently modulate each polarization component of the carrier wave as depicted in Figure 1a. The same constellation was used for both polarizations. An ideal polarization beam splitter was used to maintain orthogonality among X- and Y-polarizations, and then an ideal polarization beam combiner. The bit sequence generation (pseudo-random binary sequence, PRBS) for the encoding of the m-QAM F-OFDM/OFDM was set at 2²¹, employing a virtually random sequence using Mersenne Twister in Matlab to avoid bit pattern recognition by the ML algorithms. For optical modulation, the in-phase/quadrature (IQ) single-drive architectures implemented for the X- and Y-polarization transmitters permitted advanced shaping of the modulator’s driving signals. The square-root-raised-cosine filtering at the transmitter side was selected, together with the matched filter at the receiver, to reduce the amount of detected noise, keeping ISI at a minimum. For the transmitter’s laser continuous wave source, a 300 kHz linewidth was set. In the inset of Figure 1a, the 4QAM for OFDM and the combined 4ASK for the F-FODM constellation diagrams are illustrated (qualitative showcase).

For the coherent receiver as depicted in Figure 1b, an optical coherent quadrature receiver model was used for signal detection. It incorporated a local oscillator (LO), optical hybrids, post-detection electrical filters, and ADCs. Enhanced accuracy was provided via clock recovery. Clock recovery ensured optimal sampling times in back-to-back configurations. The electrical received signal was digitized using four ADCs, outputting them as an N × 4 float matrix, where N was the product of the simulation time window and the sampling rate. Ideal optical hybrids were used without considering imbalances and losses. The ADC resolution for each channel was set at 8 bits. The LO laser linewidth was also set at 300 kHz with a frequency offset from the transmitter laser of 500 kHz. Balanced PIN photodiodes were used with the following parameters: a responsivity of 0.9 A/W, dark current of 10 nA, and thermal noise of 20 pA/Hz^0.5, including shot noise. Low-pass filters (LPFs) were implemented using a Bessel transfer function, featuring a bandwidth of 0.8 times the symbol rate. This bandwidth was selected based on a symbol rate of 12 Gbaud per channel, utilizing a 4th-order filter. At the digital receiver side, a MIMO processor was implemented to effectively mitigate channel crosstalk in SDM systems before OFDM/F-OFDM demodulation, as shown in Figure 1b.

In Figure 2, the total SDM F-OFDM QAM transceiver system is illustrated, where the principles of SDM over FMFs are shown, which support four LP modes, totaling twelve orthogonal modes (LP01, LP11a, LP11b, LP21a, LP21b, and LP02), each accommodating two polarizations. Furthermore, the Erbium-doped fiber amplifier (EDFA)-SDM model effectively accounts for mode-dependent gain and noise, enhancing the overall performance of the transceiver system. Ideal polarization coupling and splitting were used for the modes. For the FMF, the following parameters were utilized: The supported modes were (0, 1), (1, 1), (2, 1), and (0, 2). The attenuation was set to 0.2 dB/km. The relative mode delay was accounted for, and the CD was set at 20 ps/nm/km. The dispersion slope was set at 0.06 s/km³. The differential group delay (DGD) value for each mode was 0, 1.3 × 10⁻¹³, 1.5 × 10⁻¹⁴, and 2 × 10⁻¹³ s/m, respectively. The PMD was set at 0.01 s/km^0.5, while the intra-mode group delay deviation was 5 × 10⁻¹⁵ s/m. The simulation study also includes mode coupling. The reference frequency set at the FMF was 1552 nm (193.1 THz). The mean section length in the FMF model was set at 500 m, with a section length deviation of 5 m. The inverse transmission matrix of the link was used, which is harnessed in the MIMO equalizer. The optical spectrum of the SDM channels at the reference frequency after the optical multi-mode amplifier and before FMF transmission is shown in Figure 2b. Where “SB” denotes single-band simulation in the VPIphotonics simulator, which considers all channels collectively in the frequency domain, thereby addressing inter-channel crosstalk effects.

For the ML algorithms, both DNN and RNN models were employed in conjunction with the DSP, as illustrated in Figure 1b. The DNN model was based on a fully connected architecture, as described in [27]. This model utilizes the sliding window method to preprocess time-domain signals. The DNN structures are designed to operate with real numbers; consequently, arrays of complex numbers are converted into arrays of real numbers by stacking the real and imaginary parts into a single dimension. The initial hyperparameters selected for the model include: number of hidden of layers set to 4, number of neurons set to 8, using ReLU activation function, Adam-based stochastic gradient descent for the back-propagation adaptation, a learning rate of 0.001, loss function defined as minimum mean squared error (MMSE), a batch size of 32, the number of epochs set to 30, Kernel initializer based on He-Uniform, and a zeros bias initializer. Both DNN and RNN used 75% training data, 15% for testing, and 10% for validation. The RNN model was built upon a standard architecture as outlined in [27]. Similar to the DNN model, it employs the sliding window technique for pre-processing time domain signals. Utilizing the Keras framework, the RNN adopted a bidirectional RNN architecture, which processes input sequences in both forward and backward directions. This design enables the model to capture information from both past and future contexts. The architecture is structured as many-to-one, where multiple input values are used to predict a single output value. The inclusion of average nodes allows the output from different time steps to be combined, thereby enhancing the model’s capacity to understand and learn from the sequence data [27]. Identical hyperparameters to the DNN model were adopted for fair comparison. To ensure reproducibility, the input to the model consists of complex 16QAM samples, which are decomposed into their in-phase and quadrature components and concatenated into real-valued input vectors. A sliding window of length 5–7 samples (empirically optimized) is used to capture short-term temporal dependencies in the signal. The dataset was also split into training, validation, and testing subsets using a 75%/10%/15% ratio across all models for fair comparison. Training was performed using the Adam optimizer with a learning rate of 0.001, batch size of 32, and 30 epochs, while the loss function was defined as the mean squared error (MSE). Identical hyperparameters to the DNN model were adopted for fair comparison. The CNN architecture was designed following standard deep learning practices, including the use of small convolutional kernels (size 3), 32 filters, and stacked convolutional layers with max-pooling operations for hierarchical feature extraction, as reported in [28]. The LSTM model was implemented based on the standard architecture proposed by Hochreiter and Schmidhuber [29], as well as its application in coherent optical systems reported in [30]. The LSTM model incorporates input, forget, and output gates to regulate information flow within the memory cell. A configuration of 64 units with dropout regularization (0.2) was adopted following hyperparameter tuning to ensure stable training behavior [30] and consistent comparison with the simpler RNN architecture.

3. Results

It is important to note that for any configurations in the adopted system, the LP02X channel has demonstrated the most significant degradation. Therefore, this will be analyzed in the current Section 3 as a worst-case scenario.

3.1. Conventional Coherent Optical OFDM Transmission Performance

In Figure 3a, the FMF transmission performance of conventional coherent O-OFDM is illustrated for the most severely degraded channel, represented by the LP02X channel, in terms of BER. This comparison includes linear equalization, DNNs, and RNNs for both QPSK and 16QAM OFDM. The baud rate is maintained at 12 Gbaud per channel for both modulation formats. The results show that RNN consistently delivers the best performance for transmission distances greater than 10 km and up to 100 km of FMF for both QPSK and 16QAM. The RNN outperforms the DNN primarily due to its ability to handle sequential data and capture temporal dependencies effectively, which is crucial for managing the ISI in high-speed transmission. RNNs maintain a form of memory that allows them to consider past inputs when processing current data. This enhances their adaptability to varying channel conditions and complex dynamics, such as nonlinearities and crosstalk in FMF environments. Additionally, RNNs inherently learn relevant features from the data without extensive pre-processing, making them more effective in scenarios where the signal representation is complex.

Conversely, while the DNN performance is suboptimal, it still surpasses linear equalization and offers lower complexity compared to the RNN, as discussed in Section 3.4. Figure 3b depicts the BER as a function of baud rate for the same channel at a fixed distance of 15 km, comparing the DNN and RNN with both QPSK and 16QAM OFDM formats. The plot clearly indicates that as the baud rate increases, the performance gap between the RNN and DNN widens slightly for both modulation formats. This occurs because higher baud rates lead to increased data transmission speeds, which result in greater ISI and noise levels. RNNs, with their inherent ability to manage temporal dependencies and maintain a memory of past inputs, are better equipped to handle these complexities, allowing for more effective error correction as the baud rate rises. Moreover, at higher baud rates, the signal becomes more susceptible to FMF-induced impairments, such as nonlinearities, inter-modal crosstalk, modal coupling, and accumulated modal dispersion. RNNs can adapt dynamically to these changing conditions, leveraging their architecture to model complex interactions more effectively than DNNs, which may not capture temporal aspects as robustly.

3.2. Fast Coherent Optical OFDM (F-OFDM) Transmission Performance

Figure 4a presents a performance comparison in terms of the BER between conventional optical OFDM and F-OFDM for 16QAM, both with and without ML processing. For transmission distances ranging from 10 km to 100 km in FMF, the RNN applied to F-OFDM demonstrates the best performance, highlighting its superior capacity to mitigate ISI among subcarriers. This is also evident in the received constellation diagrams of Figure 5, showcasing the performance benefit of the RNN over RNN and linear equalization for the case of 12 Gbaud at 15 km of the LP02X channel under test. While linear equalization performs similarly in both conventional OFDM and F-OFDM, and DNNs show comparable results in both systems, the implementation of the RNN clearly exhibits better performance for the LP02X channel under tests, as measured by the BER. The improved performance of RNNs when applied to F-OFDM, in contrast to conventional optical OFDM, can be attributed to the following reasons: Firstly, RNNs are specifically designed to handle sequential data, enabling them to effectively capture temporal dependencies. Additionally, F-OFDM utilizes combinations of FCTs instead of the FFT employed in OFDM. This enhances spectral efficiency but decreases ISI, allowing the signal to be more amenable to the analysis and correction capabilities of RNNs. The FCT effectively clusters signal energy within fewer frequency bins, which RNNs can leverage to learn more effective representations of the data, thereby contributing to improved BER performance.

In Figure 6a, the BER versus transmission distance is shown for different numbers of subcarriers. In particular, 64, 128, and 256 subcarriers are compared for both optical OFDM and F-OFDM cases. The comparison of BER performance between them confirms that F-OFDM outperforms OFDM, especially as the number of subcarriers increases. Higher subcarrier counts (128 and 256) enhance resilience against ISI. While F-OFDM demonstrates a lower BER due to reduced inter-carrier interference and better handling of FMF impairments, increasing the number of subcarriers also brings challenges such as increased sensitivity to phase noise. In Figure 6b, the CP length is adjusted in the F-OFDM LP02X channel and tested over different FMF distances. Generally, as transmission distance increases, the BER also rises, indicating a higher likelihood of errors over longer distances. Notably, increasing the CP percentage helps mitigate the BER, suggesting that a longer CP can effectively counteract channel impairments and ISI in long FMF distances. The regions with darker blue colors indicate the optimal configurations for CP length, while noticeable transitions highlight critical distances where performance degrades significantly. Overall, the results emphasize the importance of carefully selecting CP lengths to enhance the robustness of F-OFDM systems in SDM applications, balancing improved performance against potential overhead.

In Figure 7, the performance of the LP02X channel in terms of the BER is evaluated across various combinations of the number of neurons and hidden layers in the RNN. This assessment is conducted using a 16QAM F-OFDM RNN system operating at a distance of 10 km and the same baud rate of 12 Gbaud. It is generally observed that an increase in the number of neurons correlates with lower BER values, particularly at lower hidden layer counts, indicating that more neurons enhance the model’s capacity to learn complex patterns. While adding hidden layers initially improves performance, there is a noticeable diminishing return effect beyond a certain point, where further layers either plateau the performance or degrade it, indicating potential overfitting. Optimal configurations, such as using 13 neurons with fewer hidden layers, result in the best BER outcomes.

3.3. Evaluation of Higher-Complexity ML Models for F-OFDM

Additional simulations have been conducted to compare the DNN and RNN models with their more advanced and computationally complex counterparts, namely CNN and LSTM architectures, respectively. Following hyperparameter optimization, the configurations for both the LSTM network and the CNN were selected as follows: the RNN (SimpleRNN) and LSTM are both “recurrent” neural networks, so they both process sequences and pass a hidden state forward in time. However, the LSTM network is a different cell type with gates (input/forget/output gates) and a separate cell state. That generally makes LSTM networks better at learning long-term dependencies than vanilla SimpleRNN. After sweeping all the key hyperparameters in the LSTM network, the final ones used were as follows: 64 units with 1 LSTM layer, dropout = 0.2, recurrent_dropout = 0, Adam learning rate = 1 × 10⁻³, batch size = 32, and return_sequences = True for many-to-many (prediction at each timestep). For the CNN model, after systematic hyperparameter tuning, the following configuration was adopted: Conv filters = 32, kernel size = 3, stride = 1, padding = “same”, 2 Conv1D/2D blocks with MaxPooling = 2 × 2 (pool_size = 2, stride = 2), activation = “relu”, GlobalAveragePooling layer, Dense units = 128, trained with Adam learning rate = 1 × 10⁻³ and batch size = 32. Final layer activation was linear (i.e., no activation). There were two final layer units (prediction for the in-phase and quadrature components of the QAM signal).

In Figure 8, the LSTM and CNN models are compared with the RNN and DNN, respectively. For both transmission distance (Figure 8a) and varying baud rates (Figure 8b), the LSTM model provides nearly identical performance to the vanilla RNN. This indicates that the additional gating mechanisms of the LSTM model do not provide a significant advantage for the considered SDM-QAM scenario. This can be attributed to the relatively limited temporal dependencies in the signal after standard DSP, where the simpler RNN is already sufficient to capture the underlying dynamics. On the other hand, the CNN shows slightly worse performance compared to the DNN. This behavior is expected, as CNNs are primarily designed to capture local spatial features, whereas the equalization task in this work benefits more from global feature representation, which is more effectively handled by fully connected DNN architectures. Consequently, the results confirm that the increased architectural complexity of the LSTM model and the CNN does not translate into performance gains, reinforcing the suitability of the lower-complexity RNN and DNN models for the targeted application.

The LP02x mode is considered the worst-case propagation scenario, exhibiting the strongest mode coupling and nonlinear impairments in the system. To further evaluate the robustness of the proposed ML-based equalization, additional higher-order spatial modes (LP01x, LP11x, and LP21x) are also considered for comparative analysis in Figure 8c, all of which exhibit slightly improved propagation characteristics relative to LP02x. Similar trends are observed for DNN, LSTM network, CNN, and RNN equalizers across all considered LP modes. However, the relative performance improvement provided by ML-based equalization is slightly reduced for LP01x, LP11x, and LP21x compared to the LP02x worst-case scenario. This is because the propagation impairments are less severe in these modes. It is also observed that the BER curves for the LP01x, LP11x, and LP21x modes exhibit a smoother variation with transmission distance compared to the LP02x worst-case scenario. This behavior is attributed to the reduced severity of mode coupling and nonlinear impairments in these modes, which leads to a less pronounced fluctuation in the effective channel response and consequently more gradual BER evolution. In the strong coupling regime (LP21x), ML-based equalizers provide more pronounced BER gains compared to LP11x and LP01x, indicating their enhanced capability to mitigate severe inter-modal interference and nonlinear effects.

3.4. ML Computational Complexity

Figure 9 illustrates the learning curves during the training stage, expressed as loss (mean square error (MSE)) versus epoch number, for the RNN and the DNN, applied to the 12 Gbaud LP02X F-OFDM channel over a transmission distance of 10 km. As observed, both models converge after a similar number of epochs, indicating comparable training stability. However, a clearer distinction emerges when considering the computational complexity, as summarized in

Table 1. Computational complexity comparison in terms of FLOPS.

FLOPS		Total Params	Trainable Params
RNN	262,713	4031	4031
DNN	2581	1341	1341
CNN	112,732	10,202	10,202
LSTM	819,423	16,601	16,601

, in terms of floating-point operations (FLOPS) and parameter count. Specifically, the RNN requires 262,713 FLOPS, whereas the DNN requires only 2581 FLOPS, resulting in approximately 101 times higher computational complexity for the RNN. In terms of model size, the RNN consists of 4031 trainable parameters compared to 1341 for the DNN, making it approximately three times more complex in parameter count. When extending the comparison to more advanced architectures, the CNN and LSTM models exhibit significantly higher computational demands. The CNN model requires approximately 112,732 FLOPS and around 10,202 parameters, placing it at an intermediate complexity level (substantially higher than the DNN but lower than the RNN in FLOPS). In addition, the CNN is still considerably larger in parameter count. In contrast, the LSTM model demonstrates the highest complexity due to its gated structure, requiring approximately 819,423 FLOPS and 16,601 parameters. This corresponds to roughly three times higher complexity than the RNN in FLOPS and about six times more parameters, while being more than two orders of magnitude more complex than the DNN.

4. Discussion

In this paper, the potential of coherent optical F-OFDM was explored as an advanced multicarrier solution, demonstrating its superiority over traditional coherent optical OFDM. The results highlighted the advantages of utilizing F-OFDM in combination with supervised ML equalization techniques, particularly in improving BERs in SDM-based FMF m-QAM systems. The findings indicate that while RNNs outperform DNNs in terms of BER under certain conditions, the complexity of RNNs presents a notable challenge. Specifically, the computational demands of RNNs, which demonstrated approximately 101.3 times more FLOPS compared to DNNs, make them less practical for real-time applications, particularly in cost-sensitive environments. On the other hand, as the number of neurons increases, the RNN shows improved performance; however, the diminishing returns in accuracy beyond a certain threshold suggest that simpler architectures like DNNs may be more efficient. It also may be prudent to consider adaptive CP length management [31] based on the evolution curve against the FMF distance, highlighting that optimal configurations can enhance robustness while also accommodating the limits of system complexity. This adaptive management helps balance system performance and resource consumption. It also enables more efficient and scalable optical communication networks.

Furthermore, while the simulation framework incorporates realistic component parameters (e.g., DAC/ADC resolution, laser linewidth, fiber impairments), real-time deployment would introduce additional constraints such as processing latency, memory requirements, and hardware resource limitations in FPGA/application-specific integrated circuit (ASIC) platforms. The results indicate that although RNN- and LSTM-based equalization provides a superior BER performance, its significantly higher computational complexity (e.g., >100 × FLOPS compared to the DNN) may limit its immediate applicability in real-time systems, especially in power- and cost-constrained environments. In contrast, DNN-based equalizers offer a more favorable trade-off between performance and implementation complexity, making them more suitable for near-term hardware realization. Therefore, this work should be viewed as a proof-of-concept study that establishes performance bounds and design trade-offs, while future work will focus on experimental validation and hardware-oriented optimization, including reduced-complexity ML architectures and FPGA/ASIC implementation strategies.

To further justify the selection of ML architectures and evaluate the impact of model complexity on F-OFDM performance, an ablation-style comparison is presented in

Table 2. An ablation-style comparison of the ML architectures for F-OFDM equalization.

Model	Architecture Role	Key Modification (Ablation Dimension)	Relative Performance (BER/MSE Trend)	Observation/Insight
DNN	Baseline spatial model	Fully connected layers only	Strong baseline	Efficient global feature learning; the lowest complexity
RNN	Baseline temporal model	Adds recurrence (no-gating)	≈LSTM (similar)	Captures temporal dependence but high computational cost
CNN	Local feature extractor	Convolution + pooling layers	Slightly worse than the DNN	Local feature bias not optimal for global signal equalization
LSTM	Enhanced temporal model	Adds gating mechanisms to RNN	≈RNN (no gain)	Gating does not improve performance; the highest complexity

. Table 2 is related to the summary of the F-OFDM results shown in Figure 8. In this context, the term “ablation-style” refers to a systematic comparison across different neural network architectures, where each model introduces a distinct structural modification (e.g., recurrence, convolution, or gating mechanisms) relative to a fully connected baseline network. This approach enables the isolation of the contribution of each architectural feature to the overall system performance. Specifically, the DNN serves as the baseline model, providing global feature learning through fully connected layers. The RNN extends this baseline by incorporating temporal dependencies via recurrent connections, while the LSTM model further enhances the recurrent structure by introducing gated memory mechanisms designed to capture long-term dependencies. In contrast, the CNN represents a fundamentally different approach, focusing on localized feature extraction through convolutional operations. As shown in Table 2, the results demonstrate that increasing the architectural complexity does not necessarily lead to performance improvements in the considered F-OFDM equalization task. While the RNN achieves better performance than the DNN, it does so at a significantly higher computational cost. Similarly, the LSTM model, despite its more advanced memory structure, does not provide measurable gains over the simpler RNN, indicating that long-term dependencies are limited in this application. The CNN, on the other hand, exhibits inferior performance, suggesting that local feature extraction is less effective than global feature learning for this specific signal processing task. In general, the ablation-style comparison highlights a clear complexity–performance trade-off, confirming that lower-complexity models such as the DNN and RNN are sufficient to capture the underlying signal characteristics, while more sophisticated architectures introduce unnecessary computational overhead without the corresponding performance benefits.

5. Conclusions

This paper demonstrates the viability and advantages of optical F-OFDM as a next-generation multicarrier transmission solution. The innovative approach of reducing subcarrier spacing, combined with the application of ML techniques, specifically RNNs and DNNs, has shown promising results in enhancing spectral efficiency and improving the BER under various channel conditions, respectively. While the RNNs exhibited superior performance in managing ISI and inter-carrier interference and adapting to complex channel dynamics, their high computational complexity poses challenges for potential future real-time implementation. On the other hand, DNNs, with their simpler architecture, provide a compelling alternative that balances performance with efficient processing requirements.

To further assess the impact of model complexity, higher-complexity architectures, namely the CNN and the LSTM network, were also investigated. The results indicate that, despite their advanced structural capabilities for spatial and temporal feature extraction, these models do not provide additional performance gains for the considered F-OFDM scenario, while introducing significantly higher computational complexity.

This study also highlighted the importance of selecting appropriate system configurations, such as adaptive CP management and subcarrier number selection, to optimize performance while minimizing complexity. Overall, the findings reinforce the potential of F-OFDM integrated with ML strategies to meet the demands of modern SDM-based optical communication systems, while emphasizing that lower-complexity models can offer the most practical and efficient solutions. This paves the way for more efficient, scalable, and robust network implementations.