Machine Learning for Self-Coherent Detection Short-Reach Optical Communications

Abstract

Driven by emerging technologies such as the Internet of Things, 4K/8K video applications, virtual reality, and the metaverse, global internet protocol traffic has experienced an explosive growth in recent years. The surge in traffic imposes higher requirements for the data rate, spectral efficiency, cost, and power consumption of optical transceivers in short-reach optical networks, including data-center interconnects, passive optical networks, and 5G front-haul networks. Recently, a number of self-coherent detection (SCD) systems have been proposed and gained considerable attention due to their spectral efficiency and low cost. Compared with coherent detection, the narrow-linewidth and high-stable local oscillator can be saved at the receiver, significantly reducing the hardware complexity and cost of optical modules. At the same time, machine learning (ML) algorithms have demonstrated a remarkable performance in various types of optical communication applications, including channel equalization, constellation optimization, and optical performance monitoring. ML can also find its place in SCD systems in these scenarios. In this paper, we provide a comprehensive review of the recent progress in SCD systems designed for high-speed optical short- to medium-reach transmission links. We discuss the diverse applications and the future perspectives of ML for these SCD systems.

1. Introduction

With the advent of the 6G era, the Internet of Things, and the metaverse, there has been an explosive growth in data traffic in recent years, which poses higher requirements for current optical interconnects in terms of capacity and reliability. Coherent detection transceivers were introduced in 2006, which have been widely utilized in optical communication systems spanning thousands of kilometers, such as transoceanic, transcontinental, and metropolitan networks. In a coherent system, a local oscillator (LO) laser is employed to linearly map the received optical field into the electrical domain. Linear mapping allows for the obtainment of the amplitude, phase, and polarization information of the optical signal and for compensation against a number of transmission impairments, including fiber chromatic dispersion (CD), nonlinearity, random polarization rotation, and polarization mode dispersion using advanced digital signal processing (DSP) techniques [1,2,3,4,5,6,7,8,9]. Consequently, coherent detection enables large-capacity and high-spectral-efficiency (SE) long-haul optical communications.

On the other hand, short-to-medium distance optical networks mainly encompass data-center interconnects, passive optical networks, mobile front-haul, and industrial internet. These networks typically involve a great number of optical connections, making cost sensitivity a crucial factor for the deployed optical devices [10]. However, the utilization of LO in coherent detection necessitates temperature control circuits at the receiver to align with the frequency of the laser at the transmitter. This significantly increases the manufacturing cost of optical devices and hinders the deployment of coherent transceivers in cost-sensitive and large-scale short-to-medium distance optical links [11]. Furthermore, since the LO and the transmitter laser are different, phase noise and frequency offset estimation need to be performed in DSP, leading to the increased power consumption of the DSP chips. In contrast, direct detection systems have a natural structural advantage over coherent detection systems as they do not require a narrow-linewidth and high-stable LO at the receiver. This eliminates the need for complicated temperature control circuits, frequency offset estimation, and carrier phase recovery [11]. As a result, the manufacturing cost of direct detection transceivers is lower, making them promising for short-to-medium distance optical networks over the past decade.

The intensity modulation and direct detection (IMDD) scheme, as a classic direct detection system, encodes information directly onto the optical intensity. At the receiver, the optical intensity is converted into photocurrents through square-law detection using a single photodetector (PD), achieving the mapping from the optical domain to the electrical domain. While the IM-DD system is simple and practical, its transmission performance is limited by power fading caused by fiber CD [12]. The frequency-selective fading limits its applications for high data rates or long-distance transmission [12].

To address the issue of power fading, researchers have proposed to use vestigial sideband (VSB) modulation systems. One approach is to utilize an optical filter to eliminate one of the sidebands of the real-valued double-sideband (DSB) signal [13], reducing the influence of the fiber CD. While VSB modulation enhances the system’s resistance to CD, it also introduces nonlinear impairments due to the presence of an incompletely suppressed sideband. As a result, single-sideband (SSB) modulation systems without vestigial components have been developed as an alternative [14,15,16,17], which can be achieved using IQ modulators or optical frequency shifters. To further improve the electrical SE and transmission capacity beyond the SSB systems, single-polarization phase retrieval (PR) receiver [18,19,20,21], carrier-assisted differential detection (CADD) receiver [10,22,23], and asymmetric self-coherent detection (ASCD) receiver [24] have been proposed to achieve linear detection of complex-valued DSB signals, effectively doubling the electrical SE with respect to SSB and IM-DD systems. Additionally, polarization-division-multiplexing can double the capacity and SE of single-polarization direct detection systems. However, the random birefringence of the fiber leads to polarization rotation, resulting in polarization fading [25,26,27,28,29,30] in direct detection systems with a co-propagating optical carrier. In order to deal with this effect, Stokes-vector receiver (SVR) [25] and Jones-space field recovery (JSFR) [30] schemes have been proposed. SVR performs polarization rotation in Stokes space, allowing for up to three-dimensional real-valued modulation. The JSFR scheme, however, first recovers the optical field and then performs polarization rotation in Jones space, enabling four-dimensional modulation including the amplitude and phase of two polarizations [30]. The above-mentioned schemes in which the optical carrier and the signal are transmitted together, allowing for phase- or polarization-diversity, are commonly known as self-coherent detection (SCD) systems. SCD systems recover the optical field in the receiver DSP, allowing compensation for the CD similar to coherent detection. The power fading effect induced by the traditional IMDD channel will no longer be a problem in SCD systems.

Although SCD has numerous advantages, there are still several issues in SCD systems that need to be addressed, such as signal-to-signal beating interference (SSBI) and optical field reconstruction. In the past decade, machine learning (ML) technology has rapidly advanced, and its applications have spread across various fields, including image recognition [31], natural language processing [32], medical diagnosis [33], and optical fiber communications [34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78]. ML techniques often achieve a higher accuracy or lower complexity compared to traditional approaches in many scenarios. In optical fiber communications, ML has been extensively studied and has shown a promising performance in optical performance monitoring [34,35], modulation format recognition [36,37], channel equalization [38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72], and constellation shaping [73,74]. In this paper, we provide a comprehensive overview of the application of ML techniques in SCD communication systems, with a particular focus on their applications in nonlinear impairment compensation, IQ imbalance correction, PR, polarization demultiplexing, and optical signal processing. In Chapter 2, we provide a brief introduction to the principles and challenges of various self-coherent systems. In Chapter 3, we provide extensive applications, as well as a detailed analysis of the performance of ML techniques in SCD systems. Finally, in Chapter 4, we summarize the findings and provide an outlook on the future development of ML technology in SCD systems. All the abbreviations used in this paper are listed in Appendix A.

2. SCD Systems

2.1. VSB System

To mitigate the impact of CD-induced power fading impairment, a VSB system is proposed, employing a simple receiver configuration as depicted in Figure 1a. This receiver setup requires only a single PD and an analog-to-digital converter (ADC). The modulated optical spectrum of the VSB signal is shown in Figure 1b. The most commonly-used method in VSB systems is to employ an optical filter to remove unwanted spectral components in the optical domain. By selectively filtering out specific frequency components, the spectral shape of the VSB signal can be modified [13], allowing for effective suppression of the vestigial sideband. In addition to optical filters, VSB modulation can also be achieved through dual-drive Mach–Zehnder modulators (MZMs) and radio frequency delays. In the VSB system, the dominant impairment originates from the unfiltered residual spectral components, which can be expressed as in [21]:

$$E(t)=C+S_s(t)+S_r(t)$$

(1)

$${\left|E(t)\right|}^2={\left|C\right|}^2+C^{*}\left(S_s(t)+S_r(t)\right)+C{\left(S_s(t)+S_r(t)\right)}^{*}+{\left|S_s(t)+S_r(t)\right|}^2$$

(2)

where C, S_s(t), and S_r(t) denote the co-propagating optical carrier, the desired SSB signal, and the residual sideband signal, respectively. The superscript * denotes conjugate operation. After square-law detection as shown in Equation (2), the mirror image of residual components can cause nonlinear distortion to the signal, which can be compensated for by using nonlinear equalizers such as neural networks (NNs).

2.2. SSB System

To suppress the residual signal S_r(t) and nonlinear impairment, an SSB system is proposed based on the Hilbert transformation and IQ electrical-to-optical modulator [14,15]. The Hilbert transformation enables us to remove the unwanted sideband, generating a complex-valued electrical signal which drives the IQ modulator to convert into an optical signal, as shown in Figure 1c. The receiver structure of the SSB system is the same as the VSB system. After optical-to-electronic conversion, the dominant distortion is called SSBI, denoted by |S_s(t)|². For the SSB system, the received signal can be expressed as in [21]:

$${\left|E(t)\right|}^2={\left|C\right|}^2+C^{*}{S}_s(t)+C{S}_s{(t)}^{*}+{\left|S_s(t)\right|}^2$$

(3)

Fortunately, for the SSB signal, the impact of SSBI can be mitigated by employing phase recovery algorithms based on the minimum phase condition [79,80,81] or deep learning techniques. These methods help in recovering the phase information lost in optical-to-electronic conversion and enable the compensation of CD in the DSP, avoiding the impact of power fading.

2.3. PR Receiver

Although the resistance to CD is improved in VSB and SSB systems, the electrical SE of these systems is the same as the IM-DD system, defined as the achieved date rate divided by the electrical bandwidth of the receiver. To increase the SE, a PR receiver [18,19,20,21] is proposed to detect a complex-valued DSB signal, as shown in Figure 2a. The PR receiver consists of two PDs and one dispersive element, as shown in Figure 2b. The two detected photocurrents i₁(t) and i₂(t) are expressed as [82]

$$i_1(t)={\left|C+S_d\left(t\right)\right|}^2,i_2(t)={\left|\left(C+S_d\left(t\right)\right)\otimes h_D(t)\right|}^2,$$

(4)

where S_d(t) and h_D(t) are the DSB signal and the transfer function of the dispersive element. Using a fully-connected convolutional NN (CNN), or other nonlinear equalization algorithms, the optical field could be reconstructed in the receiver DSP [83,84]. Note that the PR receivers also enable to recover the phase of optical SSB signal, which will be introduced in Section 3.3.

2.4. CADD

Another kind of receiver used to detect a complex-valued DSB signal is the CADD receiver [10]. The receiver structure is shown in Figure 2c, consisting of one optical hybrid, one PD, and two balanced photodetectors (BPDs), which is more complex than the PR receiver. However, it can achieve a higher modulation bandwidth and electrical SE than the PR receiver. In the receiver DSP, certain SSBI iterative cancellation algorithms and deep NNs are also used for optical field reconstruction [10,85]. With the help of ML techniques, the channel parameters such as the optical delay values and the carrier-to-signal power ratio (CSPR) can be optimized accurately to achieve a better system performance than the SSBI iterative cancellation algorithm.

2.5. SVR

The direct detection system has been pursuing polarization division multiplexing because it can double its capacity and SE. However, for the optical field where the signal and carrier are transmitted together, the optical signal suffers from polarization fading after passing through a polarization beam splitter (PBS). Polarization fading can result in the failure of optical field recovery on random X- or Y-polarization, making it hard to achieve polarization demultiplexing using multi-input multi-output (MIMO) equalization. Thus, the famous SVR [25] was proposed to combat polarization fading in Stokes space. The receiver structure is shown in Figure 3a, where three received Stokes vectors S₁, S₂, and S₃ are used to address the polarization rotation. The transmitted Stokes vectors could be recovered using S₁, S₂, and S₃ and a de-rotation matrix. Thus, the polarization diversity of the DD system is successfully accomplished.

2.6. JSFR

Although the polarization fading issue is solved, at most three modulation dimensions are supported in the real-valued three-dimensional Stokes space. Great efforts are made to exploit the fourth modulation dimension, but these fail to compensate for CD. More recently, the JSFR scheme was proposed to realize polarization diversity for a direct detection system with a co-propagating optical carrier, as shown in Figure 3b. It utilizes the optical coupler to mix the two polarizations to eliminate the impact of the polarization fading effect. The generalized receiver (GR) in this scheme can be implemented using one PD, PR receiver, CADD, and ASCD, according to different modulation formats. Using JSFR, the amplitudes and phases of both X- and Y-polarizations can be recovered, which provides the potential of realizing high-SE and large-capacity optical interconnects for short-reach optical networks. For these polarization-diverse SCD systems, ML can be used to handle the coupling between the polarization modes, namely polarization tracking and polarization mode demultiplexing [82,85].

3. ML Techniques in SCD System

In this section, we will introduce the applications of ML techniques in SCD systems including nonlinearity compensation [86,87], IQ imbalance correction [88], PR in SSB [89,90,91], optical field recovery in PR receiver [83,84] and CADD schemes [82,92], and polarization tracking and demultiplexing in JSFR schemes [85]. In addition, the transfer learning [93,94,95] technique has been employed to realize fast remodeling in SSB system, which could be scalable to other DD systems. Finally, we briefly introduce the photonics neuromorphic computing [96] used in SCD systems to extract the phase information and demodulate the quadrature amplitude modulation (QAM) formats.

3.1. Nonlinear Compensation

3.1.1. Fiber Nonlinearity

In optical communication, the electrical field evolution of light in a single-mode fiber can be described by the well-known nonlinear Schrödinger equation (NLSE) [1], which takes the following form:

$$\frac{\partial A}{\partial z}+\frac{i{\beta }_2}{2}\frac{{\partial }^{2}A}{\partial t^2}=-\frac{\alpha }{2}A+i\gamma {\left|A\right|}^{2}A,$$

(5)

where z, α, β₂, and γ are, respectively, the propagation distance, the loss coefficient, the group-velocity dispersion (or second-order dispersion) coefficient, and the fiber nonlinear Kerr coefficient. The NLSE is a nonlinear partial differential equation that does not have an analytical solution, and the nonlinear parameter γ describes the effects of self-phase modulation and cross-phase modulation. In the case of SCD systems, the transmitted optical field has a strong optical carrier, making it more susceptible to fiber nonlinear impairments. It is widely known that NNs have powerful nonlinear fitting capabilities. Therefore, researchers have proposed the use of NNs to compensate for fiber nonlinearity, including various types of NNs such as artificial neural networks (ANNs) [86], long short-term memory networks (LSTMs) [87], and others, showing a better performance compared to traditional digital back-propagation and perturbation algorithms. LSTMs are a specific type of recurrent NN (RNN) model designed to mitigate the vanishing gradient problem commonly encountered in traditional RNNs. LSTMs have proven to be effective tools for mitigating transmission impairments, including both linear and nonlinear distortions, making them valuable for various applications in signal processing and communication systems. In [87], a linear network-assisted LSTM is proposed to mitigate the fiber nonlinearity in the wavelength-division-multiplexing (WDM) SSB system. Figure 4 depicts the architecture of a linear network-assisted LSTM.

The output $\widehat{y}$ can be expressed as [87]:

$$\widehat{y}=W_{L}X+W_{NL}Bi-LSTM(X_{NL})+b_{NL},$$

(6)

where X, X_NL, W_L, W_NL, Bi-LSTM, and b_NL are, respectively, the linear input vector, nonlinear vector, the weight matrix for the fully-connected layer of the linear module, the weight matrix for the nonlinear module, the one-layer Bi-LSTM network operations, and the bias vector of the nonlinear modules. Compared to conventional Bi-LSTM, the linear network-assisted LSTM achieves a significant improvement in terms of the Q-factor while also significantly reducing computation complexity.

3.1.2. Device Nonlinearity

Apart from the fiber nonlinearity, another nonlinear impairment comes from the electro-optic modulation. When the dual-drive MZM or IQ modulator is used for complex-valued QAM formats, the modulation nonlinearity will be enhanced with an increase in the peak-to-peak voltage. Figure 5a shows the bias point of the MZM and the modulation nonlinearity induced by the function of sin(·). Additionally, other device nonlinearity such as the responsibility curve of PD also deteriorates the system performance. In scenarios involving multiple nonlinear impairments, traditional methods face challenges in accurately estimating the channel parameters and compensating for the mixed nonlinear effects. However, ML demonstrates its excellent capability for parameter optimization in such complex channels. In [86], a sparsely connected ANN is proposed to address the fiber nonlinearity and modulation nonlinearity. The principle of ANN pruning is shown in Figure 5b. A weight threshold is set, and connections with weights below this threshold are pruned, thereby reducing the complexity of the NN. The pruned sparsely connected ANN is shown in Figure 5c. By implementing this method, the number of connections in the ANN is reduced by an order of magnitude, while maintaining the bit-error-rate (BER) performance without significant degradation.

3.1.3. SSBI Cancellation

Unlike coherent detection, the direct detection system does not utilize LO and BPD to cancel the common-mode component inside the photocurrent, known as SSBI. Therefore, for direct detection systems, SSBI generated by the PD becomes the primary impairment limiting the system’s transmission capacity. As observed from the fourth term in Equation (3), SSBI takes the form of a quadratic term of the original signal. The spectra of the signal and its SSBI are depicted in Figure 6a, illustrating that the bandwidth of SSBI is twice that of the original signal in the electrical domain. Consequently, SSBI distorts the signal, degrading system performance. Certain methods have been proposed to handle SSBI in direct detection systems such as the Volterra nonlinear equalization and SSBI iterative mitigation methods. Additionally, ML methods such as NNs can also play a significant role in SSBI cancellation. Compared to traditional algorithms, an NN-based equalizer offers tremendous improvements in SSBI elimination, improving the performance of the transmission system. In [81], a soft-combined ANN was proposed and its structure is shown in Figure 6b. The output of the soft-combined ANN is an average of the outputs of all of the ANNs. The results reveal that the soft-combined ANN exhibits a superior performance compared to a single ANN in compensating for both linear and nonlinear SSBI impairments in the signal. Remarkably, this improved performance is achieved while maintaining the same symbol length of the required training sequence.

3.2. IQ Imbalance Correction

For complex QAM modulation, IQ imbalance and crosstalk can lead to an incorrect signal decision resulting in a degraded BER performance. Therefore, in classical DSP steps, IQ orthogonalization algorithms are commonly used for compensation. In optical communication systems where both nonlinear impairments and IQ imbalance exist, the traditional DSP algorithms used for compensation can be replaced by a MIMO-ANN. The joint compensation approach, which addresses both types of impairments simultaneously, generally yields superior results compared to using separate compensation for each impairment individually. In [88], a MIMO-ANN is proposed to compensate for the fiber nonlinearity, SSBI, and IQ imbalance simultaneously. Figure 7 displays its structure, consisting of two ANNs. The in-phase and quadrature components, X_I and X_Q, and their delay copies are fed into these two ANNs. Y_I and Y_Q are the outputs of the MIMO-ANN. In order to minimize the cost function, the back-propagation algorithm is employed to update the weights and biases of layers. After the training processes, the optimized ANNs are used to equalize the received data. The experimental results confirm the outstanding performance of the MIMO-ANN scheme in mitigating interference between two orthogonal signals.

3.3. PR and Optical Field Recovery

In a DD system, the phase information of the optical signal is lost during envelope detection by the PD while the intensity information is retained in the photocurrent. To recover the phase information, the Kramers–Kronig (KK) receiver algorithm was proposed for DD in 2016 [79], which relies on the minimum phase condition. If the SSB signal satisfies the minimum phase condition, the phase can be extracted from the intensity information using a Hilbert transformation. To successfully apply the KK receiver algorithm, a high CSPR is required. However, achieving a high CSPR comes with certain challenges. It introduces an additional sensitivity penalty and increases the impact of nonlinear fiber propagation effects. These factors need to be carefully considered when implementing the KK receiver algorithm in DD systems. To alleviate the CSPR requirement, a supervised learning CNN model was proposed in [89,90] to emulate the KK algorithm for the PR task. The architecture of the NN model is illustrated in Figure 8a. The input of the NN is the received photocurrent, namely |E(t)|² in Equation (2). The down-sampling blocks, labeled as D_i (i = 1, 2, 3), consist of a convolutional layer followed by the Rectified Linear Unit (ReLU) activation function. The up-sampling blocks, labeled as U_i (i = 1, 2, 3), incorporate a combination of convolutional layers, transposed convolutional layers, and ReLU activation functions. In this NN model, the target outputs are selected as the in-phase and quadrature components, rather than the amplitude and phase. Through simulations, it has been demonstrated that the ML-based PR scheme accurately reconstructs the phase of a modulation phase signal even at weak carrier power levels. This ML-based approach relaxes the CSPR requirement and improves the receiver sensitivity compared to the original KK algorithm. Overall, the proposed NN model provides a promising solution for PR, leveraging the power of deep learning techniques to enhance the performance of SCD systems.

In addition to constructing SSB signals that do not satisfy the minimum phase condition, PR receivers and their corresponding algorithms can be utilized to restore the phase of the optical field. It can be applied to phase recovery of SSB signals or complex-valued DSB signals. Gerchberg–Saxton algorithm is most commonly employed for PR [18,19,20,21], but it requires multiple iterations to converge. For SSB signals, the received two optical photocurrents (i₁(t) and i₂(t)) can be fed as inputs to an NN [91], as shown in Figure 8b. This NN consists of eight convolutional blocks aimed at down-sampling and up-sampling. The outputs of the NN are the real and imaginary parts of the optical field. By implementing this NN to achieve PR, the required dispersion value of the dispersive element is significantly decreased and the computational complexity is also reduced by 30%. Most importantly, the SSB signal no longer requires a strong optical carrier to satisfy the minimum phase condition. With the same Erbium-doped fiber amplifier launch power, it is possible to increase the number of WDM channels and reduce nonlinear fiber impairments, which potentially provides a larger capacity.

On the other hand, the optical and electrical SE could be improved if the PR receiver is utilized to detect the complex-valued DSB optical signal. The deep-learning-enabled direct detection scheme [83,84] was proposed to recover the optical field at a low CSPR, which is shown in Figure 9. Similarly, the inputs are two samples of photocurrents. The NN based on deep residual learning blocks consists of two convolutional layers and several residual modules. Its output is the desired phase information of the optical field. Residual learning is a technique that introduces shortcut connections into the traditional CNN structure, providing benefits in terms of training speed and network performance. The deep residual network architecture is built around stacked residual blocks, with each block consisting of two convolutional layers and a shortcut connection. The shortcut connections enable the direct propagation of information from one layer to another, bypassing intermediate layers. The integration of shortcut connections and stacked residual blocks improves the training efficiency and enables the effective learning of deep CNN models. This architecture has been proven highly effective in various computer vision tasks, enabling the construction of deeper networks without the issues of vanishing or exploding gradients. In [83], the residual learning technique is applied to accurately recover the transmitted signal in the presence of a large SSBI under the low CSPR condition. Compared with the conventional SSBI cancellation scheme, the deep-learning-enabled DD receiver shows a significant reduction of 8 dB in the optimum CSPR when detecting a complex-valued DSB signal.

3.4. Polarization Demultiplexing

For polarization-multiplexed optical communication systems, random birefringence in optical fibers can lead to random coupling between polarization modes. Therefore, at the receiver end, DSP algorithms are required to accomplish polarization demultiplexing. Additionally, the coupling of polarization states varies over time, necessitating algorithms with the ability to track polarization changes. In the phase- and polarization-diverse JSFR scheme, a MIMO-NN was proposed [81,85] to simultaneously achieve linear optical field recovery, polarization demultiplexing, and non-linear SSBI mitigation. The receiver structure, along with the MIMO-NN, is depicted in Figure 10. The MIMO-NN consists of four layers and takes the six digital waveforms as inputs. It first extracts the in-phase and quadrature components of the dual-polarization optical field. Then, the MIMO-NN performs polarization mode demultiplexing by utilizing the inverse matrix of the polarization rotation unitary matrix. This integrated scheme enables the reconstruction of the optical field, the demultiplexing of the polarization modes, and the mitigation of nonlinear SSBI effects. By harnessing the capabilities of the MIMO-NN, the receiver achieves the detection of four-dimensional modulated signals, encompassing the amplitudes and phases of both polarizations. This advanced technique significantly enhances the SE of the DD system, approaching the performance levels of coherent detection systems.

3.5. Fast Remodeling

Transfer learning (TL) refers to the process of leveraging knowledge and experience gained from previous tasks to improve performance on new target tasks. In TL, the source task and the target task may not be consistent, meaning that they may differ in terms of data distribution, input/output spaces, or even objectives. In optical fiber communications, to reduce the number of training symbols and epochs, TL has been introduced and proven to enable fast remodeling [93,94], nonlinear equalization, and optical signal-to-noise ratio estimation [95]. In [93], a TL-assisted ANN was proposed for a multi-channel nonlinearity mitigation scheme in an SSB system. In the case of multi-channel transmission where multiple channels co-propagate in the same fiber, there exists a correlation of nonlinear distortion. This means that the nonlinear effects introduced by one channel can impact the other channels. Understanding and accounting for this correlation is crucial in designing and optimizing multi-channel transmission systems. By considering the correlation of nonlinear distortion, more accurate modeling and compensation techniques can be developed to mitigate the impact of nonlinearities and improve the overall system performance.

The principle of TL-ANN for multiple channels is shown in Figure 11. At the initial stage, an ANN is trained using labeled training data that have been collected. Once the initial training is complete, the prior distribution of parameters from the trained source model is transferred to accelerate the remodeling process. This parameter transfer avoids the need for re-initialization in the retraining method. By leveraging the learned knowledge from the source model, the remodeling process can be accelerated and potentially achieve a better performance. Subsequently, a few samples are used to train the parameters of the TL-ANN so that it can converge and accurately compensate for the impairments in the current channel. The experimental results show that the required training epochs can be reduced by 80% without BER performance degradation, saving considerable computational complexity.

3.6. Optical Signal Processing

Photonic NNs, also known as optical NNs, are a class of NNs that utilize the principles of photonics to perform signal processing in the optical domain. Instead of relying on traditional electronic components, these networks employ optical elements for computation and communication. One specific implementation of photonic NNs is photonic reservoir computing (RC), which is an ML framework that utilizes a fixed, random dynamical system called the reservoir to process data. In the case of a photonic RC, the reservoir is implemented using photonic components and principles. In [96], a recurrent optical spectrum slicing (ROSS) neuromorphic accelerator was proposed to realize an SCD receiver. This network aimed to extract phase information and demodulate QAM formats while simultaneously mitigating CD. The structure of the neuromorphic receiver based on ROSS is illustrated in Figure 12.

At the receiver side, a semiconductor optical amplifier compensates for the transmission and insertion losses of the integrated chip. The structure includes three recurrent nodes, each consisting of a Mach–Zehnder delay interferometer (MZDI) in a loop equipped with variable optical attenuators, phase shifters, and optical delays. PDs follow these nodes and are then connected to transimpedance amplifiers and ADCs. The subsequent DSP includes a light-based RNN for each quadrature. This configuration enables the extraction of phase information, demodulation of QAM formats, and effective mitigation of CD using the photonic components and principles employed in the ROSS structure. The photonic RC contributes to reducing the power consumption associated with high-bandwidth PDs/ADCs and heavy digital equalization algorithms.

4. Conclusions

This paper introduced the challenges and advancements in SCD systems and reviewed the application of ML techniques in addressing these challenges. The utilization of ML algorithms has exhibited promising results in compensating for various impairments such as fiber nonlinearity, IQ imbalance, SSBI, PR, polarization demultiplexing, and fast channel remodeling. CNNs, LSTMs, sparsely connected ANNs, and MIMO-NNs have been successfully employed to achieve accurate nonlinear impairment compensation and efficient signal processing. Furthermore, transfer learning has been utilized to reduce training time and improve modeling in multi-channel scenarios, while the residual learning method combined with a CNN has been proven effective for optical field recovery. Additionally, the emergence of photonic NNs, such as photonic reservoir computing, harnesses the advantages of photonics for information processing in SCD systems. The integration of ML techniques into SCD systems has resulted in significant enhancements in modulation dimensions, SE, transmission performance, and capacity. Integrating machine learning into direct detection systems may also raise costs due to specialized hardware needs for efficient computation. The actual impact varies with the technology advancements and performance benefits gained. However, further research is necessary to optimize ML models, explore novel network architectures, and address practical implementation challenges to fully leverage the potential of ML in SCD systems. In the context of SCD systems, machine-learning techniques are increasingly favored for tasks such as optical field recovery or phase retrieval: tasks that traditional nonlinear equalization algorithms struggle to achieve. Regarding challenges linked to applying ML in SCD systems, these involve concerns about computational complexity and hardware requirements, especially for ASIC chips. Consequently, it is essential to focus future endeavors on exploring and resolving the intricacies of ML algorithms to facilitate their practical implementation.

In summary, the combination of SCD systems and ML techniques holds tremendous promise for enabling high-capacity, cost-effective, and reliable optical communication networks in the 6G era and beyond. The advancements in ML algorithms offer new avenues for overcoming the challenges and improving the overall performance of SCD systems.