Machine Learning Applied to Optical Communication Systems

1. Introduction

As the global demand for high-speed and high-capacity communication continues to surge, driven by cloud computing, artificial intelligence, 5G, virtual reality, and the Internet of Things (IoT), optical communication systems have emerged as the backbone of modern digital infrastructure [1,2,3,4,5]. However, the increasing complexity, performance requirements, and operational scale of these systems have begun to exceed the capabilities of traditional analytical and rule-based design methods. In this context, machine learning (ML) has become a transformative tool, enabling data-driven solutions that can adapt to dynamic conditions, extract hidden patterns, and optimize performance across the optical communication stack.

In long-haul coherent optical systems, where signal integrity is challenged by fiber non-linearities and polarization effects over hundreds of kilometers, ML techniques such as neural networks (NNs) and generative models have been deployed for non-linear compensation, channel equalization, and signal reconstruction [6,7,8,9]. For instance, deep learning models trained on simulated or real-world data can outperform conventional Volterra filters or digital backpropagation, offering both improved accuracy and flexibility [10,11,12,13]. In short-reach and direct-detection systems, including intensity-modulated direct-detection (IM/DD) links widely used in data centers, ML is increasingly used to counteract combined linear and non-linear impairments [14,15,16,17]. For example, NNs and support vector machines (SVMs) have been shown to enhance equalization, improve bit error rate (BER) performance, and adapt to non-idealities in low-cost hardware implementations, such as directly modulated lasers (DMLs), vertical cavity surface emitting lasers (VCSELs), and silicon photonic transceivers [18,19,20,21]. Passive optical networks (PONs) and optical access systems demand high bandwidth at a low cost, with minimal latency and high reliability. Here, ML is also employed for intelligent receiver implementation [22,23,24,25], enabling better system resilience and performance optimization.

Visible light communication (VLC) and optical wireless communication (OWC) are emerging as complementary technologies for indoor environments and last-meter access [26,27,28]. ML has proven particularly effective in these domains, supporting signal demodulation, distortion compensation, and positioning through image-based or channel-aware learning techniques. End-to-end learning using autoencoders and convolutional neural networks (CNNs) has shown strong potential to optimize the optical-to-electrical (O/E) conversion and enhance signal robustness [29,30,31,32]. In more advanced and non-traditional systems, such as chaos-based secure communication and photonic reservoir computing, ML enables reliable chaos recovery, signal decoding, and multi-channel synchronization [33,34,35,36]. These systems benefit from the adaptability and high dimensionality of learning-based methods, which can capture complex temporal and spectral features beyond the reach of conventional models. Finally, optical network management has also embraced ML in the form of AI-enhanced network orchestration, software-defined control, and optical performance monitoring [37,38,39,40]. These applications support real-time optimization, fault localization, and predictive maintenance, aligning with the goals of intelligent and autonomous network operation [41,42,43,44].

By covering this rich spectrum of systems and applications, the papers in this Special Issue collectively demonstrate the vast potential of ML in shaping the next generation of optical communication networks. From physical-layer enhancements to system-level intelligence, ML is not only a tool for performance improvement, but also an enabler of fundamentally new architectures and functionalities.

2. An Overview of the Published Articles

This Special Issue, “Machine Learning Applied to Optical Communication Systems”, brings together a diverse collection of research contributions that highlight the synergy between ML and a wide range of optical communication technologies. This Special Issue collects 14 diverse contributions, including three comprehensive review papers and 11 original research articles that exemplify the growing synergy between machine learning and optical communication systems. The selected works span various system types, including long-haul coherent transmission, short-reach interconnects, PONs, OWC, and chaos-based secure transmission systems. Each type of system poses distinct challenges ranging from non-linear impairments and chromatic dispersion to noise accumulation and hardware limitations, and ML offers promising techniques to address them. Below is a brief overview of each contribution.

For the research articles, in contribution 1, He et al. propose a modified regular perturbation (MRP) model enhanced with trainable parameters to improve the accuracy of fiber transmission modeling under dispersion and non-linearity. This hybrid physical–ML model effectively reduces fitting error, even under high launch power and dual-polarization transmission scenarios. In contribution 2, Freitas and Pires present an NN-based framework for the rapid estimation of capacity and cost in large-scale multi-fiber optical networks. The model achieves high accuracy with significant computational efficiency, serving as a practical tool for real-time network design and planning. In contribution 3, Vu et al. introduce DeepChaos+, a deep learning-based framework for chaos signal removal in wavelength division multiplexing (WDM) systems. Their model enhances detection performance while reducing the BER by about three orders of magnitude, offering a robust solution for chaos-based secure communications. In contribution 4, Srinivasan et al. propose a novel ML-driven equalization method using gradient-based optimization for VCSEL-based transceivers. The technique improves signal integrity in thermally unstable environments, a key need for data center and automotive interconnects. In contribution 5, Liem et al. present an ML-enhanced resilience mechanism for NG-EPONs to support ultra-reliable tactile Internet applications. The system utilizes ML for fault detection and software-defined networking (SDN) for proactive recovery, achieving excellent performance in reliability metrics. In contribution 6, Ji et al. apply attention-based CNNs to mitigate angle-induced distortion in camera-based visible light positioning systems. Their model significantly reduces the number of positioning errors, paving the way for practical and precise indoor navigation. In contribution 7, Osahon et al. demonstrate that a multilayer perceptron-based decision feedback equalizer (DFE) significantly outperforms the traditional methods in OWC links, particularly under high non-linear distortions. This paves the way for higher data rates in safe, short-range optical wireless links. In contribution 8, Luna-Rivera et al. address the often-neglected role of optical-to-electrical conversion in VLC systems. Using autoencoder architectures, they improve robustness and system performance, offering insights into practical VLC design for 5G and IoT networks. In contribution 9, Hung et al. propose a Mach–Zehnder interferometer (MZI)-based optical NN to regenerate the intensity-modulated signals distorted by the bandwidth limitations of Silicon micro-ring modulators. Their photonic NN matches digital NN performance, underscoring the promise of optical ML hardware for high-speed systems. In contribution 10, Zhong et al. present a photonic reservoir computing system using quantum dot spin-VCSELs for coherent optical chaos-based secure communication. They achieve robust demodulation of complex modulated signals, validating the potential of optical chaos and reservoir computing. In contribution 11, Guo et al. tackle interference from auxiliary management channels in PON systems using Gaussian mixture model (GMM)-based probabilistic shaping. The joint optimization at transmitter and receiver significantly reduces the bit error rates, improving system robustness.

For the review articles, in contribution 12, Xu et al. review NN-based equalizers for IM/DD systems, addressing their ability to mitigate non-linear impairments while highlighting the challenge of computational complexity. The paper offers a comparative analysis of network types and proposes strategies for reducing model size and power consumption, paving the way for practical deployments. In contribution 13, Shao et al. survey ML applications in short-reach systems, focusing on digital signal processing (DSP), monitoring, and control. A key contribution is their taxonomy of time series models, offering a structured view of recent advances. The review also discusses the limitations in the current methods and suggests directions for more efficient and scalable ML integration. In contribution 14, Wu et al. provide an overview of the ML techniques in self-coherent systems, emphasizing improvements in signal recovery and phase tracking. The paper discusses how ML enhances self-coherent detection performance while reducing hardware complexity, and outlines future trends in low-cost, ML-assisted self-coherent designs.

3. Conclusions

In summary, this Special Issue highlights the growing synergy between ML and optical communication systems, showcasing advancements across different optical communication applications. The eleven original research articles and three review papers collectively demonstrate how ML techniques can effectively tackle system impairments, optimize performance, and enable intelligent network management. These contributions underline the increasing role of ML as a core technology in optical systems, while also pointing toward key future challenges such as complexity reduction and real-time adaptability. ML is no longer a peripheral or auxiliary tool for optical communication; it is becoming a fundamental enabler of the next generation of optical systems. We hope that this Special Issue serves not only as a record of current advancements, but also as a roadmap for future exploration.