Machine Learning Applications for Short Reach Optical Communication

Abstract

With the rapid development of optical communication systems, more advanced techniques conventionally used in long-haul transmissions have gradually entered systems covering shorter distances below 100 km, where higher-speed connections are required in various applications, such as the optical access networks, inter- and intra-data center interconnects, mobile fronthaul, and in-building and indoor communications. One of the techniques that has attracted intensive interests in short-reach optical communications is machine learning (ML). Due to its robust problem-solving, decision-making, and pattern recognition capabilities, ML techniques have become an essential solution for many challenging aspects. In particular, taking advantage of their high accuracy, adaptability, and implementation efficiency, ML has been widely studied in short-reach optical communications for optical performance monitoring (OPM), modulation format identification (MFI), signal processing and in-building/indoor optical wireless communications. Compared with long-reach communications, the ML techniques used in short-reach communications have more stringent complexity and cost requirements, and also need to be more sensitive. In this paper, a comprehensive review of various ML methods and their applications in short-reach optical communications are presented and discussed, focusing on existing and potential advantages, limitations and prospective trends.

1. Introduction

In recent years, short-reach optical communication systems have attracted intensive interests from both the industry and the academia. The criteria we use in this paper to determine whether an optical communication system is a short-reach or a long-reach is based on the transmission distance. We use the short-reach to refer to the communication system and network with a transmission distance of shorter than 100 km, which are cost-sensitive due to their large deployment scale [1]. In short-reach optical communications, in addition to broadband optical access networks, the emergence of 5G and future beyond-5G (also called 6G), new distributed edge cloud computing networks, large-scale machine-to-machine communications, inter- or intra-datacenter communications and mobile fronthaul, also impose new stringent requirements [1,2,3]. The typical architecture and application of the short-reach communications is shown in Figure 1. For the ease of discussion, based on the transmission distance and function, here we divide short-reach communication systems and networks into five categories: inter-datacenter networks, intra-datacenter networks, optical access networks, in-building and indoor optical wireless communications, and mobile fronthaul communications.

In the past decade, large-scale data centers that promoted the development of information technology has greatly changed our lives [4]. Internet traffic within and between data centers is growing rapidly, particularly driven by the demand for bandwidth-intensive applications such as cloud computing, live streaming, and online games. Therefore, there is an urgent need for the design and deployment of higher capacity optical communication links in inter- and intra-datacenter networks (DCNs), where the intra-DCN link is usually up to hundreds of meters long, and the length of datacenter inter connection (DCI) link is usually up to tens of kilometers [5]. For the sake of solving the bandwidth limitation of traditional wavelength division multiplexing (WDM) networks, flexible bandwidth optical networking has been applied to intra and inter datacenter networking. Due to the low-cost requirement and the short transmission distance property, the multi-mode fiber has been widely used in intra-data center links. To enhance the channel capacity in multi-mode fiber intra-data center links, higher symbol rates and multi-level modulation formats have been used [6]. In addition, the latest standard can use four-level pulse amplitude modulation (PAM-4) to achieve a signal transmission rate of 25 Gbaud to support shortwave wavelength division multiplexing (SWDM) for intra-DCN [5]. Multi-mode fiber links with SWDM technology can help data centers to further upgrade to 40 G and 100 G Ethernet by using vertical-cavity surface-emitting lasers (VCSELs) and lasers optimized OM3/OM4 multi-mode optical fibers [7]. On the other hand, for DCI (with longer transmission distance and higher data rate requirements compared with intra-data center links), 100-Gbaud PAM-4 intensity-modulation direct detection (IM/DD) transceiver has been developed, which supports up to 200 Gb/s data rate [8]. Whilst being cost-effective, IM/DD transmission technologies struggle to further increase the capacity and reach of DCI links to meet even higher requirements, whilst completely coherent solutions are currently too costly. Gratifyingly, in 2020, the use of single side band (SSB) direct detection reached a data rate of more than 400 Gb/s per channel, where 5 WDM channels each with over 400 Gb/s (64-QAM at 85 GBaud/s) have been realized for the first time [9].

The information processed and exchanged in the data center needs to be transmitted to users through the optical access network. As shown in Figure 1, passive optical network (PON) is a typical optical access network architecture that uses un-powered optical splitters for the fiber to premises connections [10]. A PON consists of an optical line terminal (OLT) located at the central office of the service provider and several optical network units (ONUs) located close to customers. High-speed PON can increase the network capacity and improve QoS metrics [11]. ITU-T and IEEE have boosted the nominal line rate of PON systems to more than 10 Gbps (XG(S)-PON) in recent years [12]. Following XG-PON, 40 Gbps-capable PON (NG-PON2) have added solutions to meet higher speed, and larger bandwidth demands [13]. There are three types of higher-speed PON systems: time division multiplexing PON (TDM-PON), time and wavelength division multiplexing PON (TWDM-PON), and WDM-PON [12]. TDM-PON uses time division multiple access (TDMA) to connect multiple ONUs to the same OLT. The commercially widely used TDM-PON technology can provide a symmetrical bit rate of up to 10 Gbps [14]. According to IEEE and ITU definitions, the next-generation TDM-PON will deliver bit rates of 25 Gbps and 50 Gbp, which can cost-effectively accommodate xhaul traffic [14]. To further increase the capacity, WDM-PON has been proposed and studied. The ONUs coexist in the same fiber with light sources operated at various wavelengths, boosting the overall network bandwidth and the number of users serviced in the optical access network [15]. In terms of network connections, the WDM-PON may employ point-to-point, point-to-multipoint, or hybrid systems, which is widely used for FTTx (Fiber to the x, x = home, building, curb or node) [16]. The spectral efficiency of WDM-PON would also have to be improved in order to enhance system capacity and the number of users serviced, where narrowing the inter-channel spacing and reusing WDM channels are two possible strategies [17]. On the other hand, different from TDM-PON and WDM-PON, TWDM-PON is a multi-wavelength PON that uses TDM/TDMA to share each wavelength channel across multiple ONUs. The details of 50G TWDM-PON such as the wavelength plan and channel information are gradually being standardized under the research of ITU-T [12]. Recently, several studies have focused on the energy efficiency of TWDM-PON equipment such as the protocol design for energy efficient OLT [18,19].

The radio access network (RAN) adds a new network section known as fronthaul, which is the link between a radio digital unit (DU; also called a baseband unit, BBU) and a remote radio unit (RRU) [20]. Various functional splits are being specified in the recently proposed next-generation fronthaul interface (NGFI) to allow varied trade-offs between the RRU complexity, fronthaul bandwidth-efficiency, and system performance [20]. Mobile-PON, a high bandwidth efficiency, low latency mobile fronthaul architecture based on functional split and TDM-PON with combined mobile and PON scheduler, has also been presented [21]. In addition, ref. [22] presents a type of mobile fronthaul architecture design plan that uses WDM-PON in addition to the wireless multi-hop and point-to-point optical links to lower fronthaul deployment costs. Furthermore, the 5G RAN architecture of a optical fronthaul network using ultra dense WDM-PON (UDWDM-PON) combined with phased array fed (PAF) reflector antenna systems for radio frequency transmission and reception is also under active research [23].

With broadband access delivered to the doorstep, high-speed in-building and indoor communications have attracted intensive interests. Compared with wired connections, the wireless in-building and indoor communications are more attractive due to the mobility feature. The optical wireless communication (OWC) has been widely used in indoor and in-building communications. As a new type of communication technology, compared with conventional RF wireless technologies, the OWC has the advantages of realizing broadband transmission, flexible networking, low latency, anti-electromagnetic interference and high communication security [24,25,26]. Both the visible range and near-infrared wavelength band have been studied in OWC systems, and both line-of-sight (LOS) and non-line-of-sight (NLOS) links have been explored [27]. In addition, the use of OWC technology for in-building and indoor user positioning has also been proposed and studied [28,29,30,31]. Recently, instead of using dedicated photo-detectors (PDs), the use of optical camera as the receiver in in-building and indoor OWC systems has attracted intensive interests as well [32,33,34]. Using OWC in in-building and indoor environments, over Gbps wireless connections have been demonstrated [35,36].

In the various types of short-reach optical communication systems and networks described above, the communication requirement becomes increasingly stringent and the complexity and requirements of the system increases substantially. To cope with the increasing complexity of various short-reach communication applications, the use of artificial intelligence (AI) technologies in short-reach optical communications has been proposed and widely studied. AI simulate sophisticated biological processes such as learning, extrapolating, and self-correction to enable computer programming to solve problems. In the past few decades, improving the performance of optical communication systems and networks through the application of AI-based technologies has become a field of extensive research, including the fields of transmission, switching, and network management. In particular, machine learning, as a branch of the AI learning method, has been widely studied as it enables the agent to improve its performance in future tasks through learning from the experience gained [37].

In general, there are four types of machine learning algorithms: supervised, unsupervised, semi-supervised and reinforcement learning. The most commonly used ML algorithm in the field of optical communication is the supervised learning, which uses known output features to derive the computational relationship between the input training data and the output data [38]. The neural networks as a supervised machine learning method, such as the artificial neural networks (ANN), convolutional neural network (CNN), and recurrent neural network (RNN), have been widely applied in optical communication systems and networks.

The ML and neural networks have been used in different tasks in optical communication systems and networks. A summary is shown in Figure 2. At the system level, ML is mainly applied in the signal processing to optimize the signal reception and detection after propagating through the highly nonlinear optical communication channels. On the other hand, at the network level, optical networks are facing more challenges due to increasing data switching and routing. From the optical network point of view, the data plane and the control plane both contribute to the increased complexity [39]. The optical network transforms from the traditional static network to the dynamic network, with no predefined spectrum grid or modulation format, and with reconfigurable network topologies, such as the elastic optical networks (EON). Although the overall performance, flexibility and efficiency of the new dynamic network have been greatly improved, it is facing challenges in terms of network reliability, heterogeneity, latency, availability, complexity, quality assurance, capacity, etc. [38]. With the ability of learning and adapt to changes in the environment, the ML technology can well cope with the challenges brought by dynamic optical networks and can even deal with unforeseen scenarios when the ML algorithm is designed [40]. Therefore, the ML algorithm is a promising solution to dynamic optical networks, with applications in optical performance monitoring [41,42,43,44,45,46,47,48,49], modulation format identification [50,51,52] and nonlinearity mitigation [53,54]. In the control plane, ML techniques have been used to achieve the highly demand dynamic control, reconfiguration, and optical network virtualization [39]. For example, the support vector machines (SVM) and clustering algorithms have been employed for detecting soft failures, and data virtualization of optical networks [55]. Meanwhile, the software-defined networking (SDN) has become an important tool for network virtualization and a flexible network traffic controller in recent years. Therefore, ML techniques have been combined with SDN to solve routing and wavelength assignment problems in the WDM systems [56].

In the field of short-reach optical communication, the EON is a promising flexible grid technology that efficiently allocate the spectrum resource to satisfy the ever-increasing end-user bandwidth demand [57]. The application of EON makes the short-reach optical networks more transparent and dynamic. However, in the meantime, it also leads to more complicated network architectures and operations. Therefore, the real-time monitoring of miscellaneous channel impairments ubiquitous in the network, also known as OPM, is essential for the reliable operation and efficient management of such complex optical networks. Unfortunately, independently monitoring multi-channel is a difficult task because the effects of various impairments are usually mixed and they are almost physically indivisible [58]. Therefore, the use of ML techniques has been widely investigated in OPM, focusing on different key parameters, such as the chromatic dispersion (CD) and polarization-mode dispersion (PMD) monitoring [59,60], fiber nonlinearity monitoring [61], bit error rate (BER) estimation [62], and optical signal to noise ratio (OSNR) monitoring [63,64,65,66].

Another key requirement in short-reach optical communication is the ability to autonomously identify the optical modulation formats deployed. This is particularly required by the OPM units in the intermediate network nodes, because the algorithms used in these devices are dependent on the modulation format utilized [42]. Thus, reliable and accurate MFI is highly demanded. However, the analytical modeling and analysis of MFI in short-reach communication is challenging due to the high hardware cost and complex algorithms. On the other hand, ML algorithms can learn from historical data without the need of accurate analytical framework. As a result, we have also seen several attempts to apply ML methods for the MFI in short-reach optical communication networks [51,52,67,68,69].

When applying ML techniques for OPM and MFI, the requirements and focus in short-reach and long-reach scenarios are different. First, the ML-based OPM in short-reach optical communications need to be less complicated and have higher sensitivity. For example, the CD and OSNR estimations in short-reach systems require higher accuracy than long-reach due to the reduced transmission distance. In addition, the major impairments that can affect the BER are different. Long-reach systems are mainly affected by the amplified spontaneous emission (ASE) noise, accumulated dispersion and nonlinear interference generated by the fiber Kerr effect [70,71]. However, in many short-reach systems, optical amplifiers are not used, and hence, the dominating noise source is no longer ASE. Hence, the system configuration should be considered while determining which impairment parameters are the priorities, since nonlinearity, fiber dispersion, laser phase noise, and relative intensity noise (RIN) can all be the dominant impairment [1]. In terms of MFI, most ML-based MFI technologies are deployed in coherent communication systems to identify modulation formats, such as QPSK, 16-QAM and 64-QAM. These ML based MFI models are normally trained with phase constellation diagrams, which are suitable for both long-reach systems and short-reach systems with coherent receivers. However, the intensity modulation with direct detection is also widely used in short-reach communications. Hence, ML-based MFI technologies in short-reach optical communications also need to be able to recognize intensity-only modulation formats, such as non-return-to-zero (NRZ) and return to zero (RZ) on-off keying (OOK), PAM-4 and PAM-8. Thus, asynchronous amplitude histograms (AAH) [72] and eye-diagram [65] based ML technologies have been proposed for the MFI in short-reach optical systems.

In addition to the application of ML in the field of OPM and MFI, ML techniques are also promising solutions in the signal processing for short-reach optical communications. In optical communication systems, digital signal processing (DSP) is widely used to compensate various transmission impairments, such as the linear impairments (e.g., CD and PMD) [53,73]. For example, the ML-based equalizer has been used as the digital part of a tunable optical signal processor for CD compensation, which can reduce the footprint and attenuation compared with the solution based on fiber Bragg grating and dispersion compensation fibers [74]. In addition to the linear impairments, the nonlinear effects, such as stimulated Brillouin scattering (SBS), four-wave mixing (FWM) and self-phase modulation (SPM), also limit the performance of high-speed optical communication systems. For example, the interactions between laser phase noise and CD generate equalization enhanced phase noise (EEPN) [75], which is challenging to compensate. Furthermore, impairments caused by optical/electrical components with nonlinear memory effects are also difficult to compensate [76]. Therefore, high-speed DSP has been developed for the compensation of nonlinear impairments, such as the Volterra nonlinear equalizers [77]. While Volterra equalizers are known to be effective for removing nonlinear distortions, the implementation and computation complexity is very high. In order to find an alternative solution, the ML-based equalization has been proposed to provide better system performance or reduced complexity with comparable performance [78], since ML techniques can effectively reduce nonlinear distortion and accurately estimate various key parameters in optical networks [58].

In even shorter-reach in-building/indoor OWC systems, the ML technique has also been widely studied. In both visible light communication (VLC) and near-infrared indoor OWC systems, the ML technique has been applied to improve the data transmission performance, mainly by suppressing the inter-symbol interference and nonlinear effects existing in the system [79,80]. In addition, the ML technique has also been widely applied in OWC-based in-building/indoor positioning systems, where better positioning accuracy, as well as simultaneous positioning and receiver orientation estimations, have been realized [81].

In recent years, several reviews and surveys about ML or short-reach communications have been published, such as the ML for OPM and MFI [37], recent developments of shot-reach optical communications [1] and ML-based equalization for short-reach transmission [82]. Unlike the previous survey papers, our survey provides a comprehensive review focusing on various ML-based applications in short-reach communications, including ML-based OPM and MFI techniques, signal processing techniques and in-building/indoor OWC systems. Compared with [37], our paper focuses on short-reach communications, and we not only review ML-based applications for OPM and MFI, but also review the application of ML in signal processing and OWC in-building and indoor positioning. Compared with [1], our paper reviews the recent developments of ML-based techniques in short-reach communications instead of introducing various impairments and related theories that affect the transmission of short-reach systems. Compared to [82], our paper not only covers ML-based equalization techniques in short-reach communications, but also introduces the application of ML in OPM, MFI, and in-building and indoor positioning. The following is a list of our contributions:

• Review the state-of-the-art of short-reach optical communication systems and networks including inter-datacenter networks, intra-datacenter networks, optical access networks, mobile front-haul communications, and in-building and indoor optical wireless communications.
• Comprehensively review ML-based OPM and MFI related research in short-reach optical communications, including the comparison of the different ML techniques, how the OPM task is formulated and the ML technique is utilized, and our analysis of, advantages and limitations of these studies.
• Comprehensively review the recent progress on ML-based signal processing techniques in short-reach optical communications, including ML-based equalization techniques as well as other ML-based signal processors, such as auto encoder/decoder, digital pre-distortion and soft-demapping applications.
• Review the ML-based applications in in-building/indoor OWC systems, including the ML-based signal processing techniques, and the application of ML algorithms in indoor user positioning.

The remainder of this survey is organized as follows. In Section 2, we first describe and compare different ML-based OPM and MFI techniques. Then, we discuss the applications in direct detect and coherent short-reach optical communications in two subsections. In Section 3, we provide a review of recent progress on ML-based signal processing techniques for short-reach optical communications, including different ML-based linear and nonlinear equalizers, auto-encoder, digital predistortion and soft demapping. In Section 4, we review and discuss recent ML-based applications in in-building/indoor OWC systems, including both wireless signal processing and user positioning. Finally, in Section 5, we summarize this paper and provide our views on the potential future research directions.

2. Machine Learning for OPM and MFI in Short Reach Optical Communications

With the development of advanced optical communication systems and networks, the critical role of OPM in the system and network operation has emerged. OPM can optimize the system performance, shorten the system installation time, increase the transmission rate, and reduce operation costs. Advanced OPM methods also have the potential to enable fault management and enhance the quality of service (QoS) in the optical domain [83]. In optical communications and networks, the key parameters that need to be monitored in the optical domain include the total power, OSNR, CD, PMD and fiber nonlinearity [62]. Whilst the transmission distance is shorter, the OPM in short-reach needs to be less complicated and more sensitive. Recently, ML techniques, particularly the deep learning for implementing OPM have been proposed. Deep learning represents a ML method based on the concept of a deep and hierarchical representation of data, which can automatically learn the function of data at multiple levels of abstraction [84]. Once the deep learning algorithm, typically based on neural networks, is fully operational, the difference between the received system feature (including eye diagram, histogram, constellation diagram, power spectrum, and time/frequency domain signal) and the ideal data/ground truth can be attributed to specific physical parameters, which can be derived from previous training process, to realize the OPM function [85]. In addition, the parameters mentioned before (e.g., OSNR), MFI is an important parameter that needs to be monitored during the OPM implementation. Most of the non-machine learning technologies for MFI, such as AAH [86] and asynchronous delayed tap sampling (ADTP) [87], require additional hardware components, which can significantly increase the complexity and cost of implementation. Since neural networks can perform automatic feature extraction and conversion of the channel information required by MFI, the neural networks such as ANN, CNN, and RNN are also good candidates for MFI.

Figure 3 summarizes the generic approach for conducting OPM and MFI using ML techniques. The first step is typically creating a data set for training the deep learning algorithm. Traditionally, the optical signal is converted to the electrical domain at the receiving end and then sampled to construct a data set. Another method to obtain the data set is directly monitoring the optical signal from the transmission link, such as measuring the OSNR using an optical spectrum analyzer (OSA), which does not require optical-to-electrical conversion and can be detected online without interrupting the regular data transmission. The second step is to extract the specific characteristics of the signal, which contains information about the impairments in the system or the modulation format. The eye diagram, power spectrum, constellation diagram, and histogram are examples of such features. These features are then employed in the final step for ML algorithm training to execute OPM, or MFI [37]. Currently, the training is typically implemented offline and after training, the trained model and feature extraction steps can be used for online OPM parameters estimation and MFI. During the system or network operation, the transmission conditions and the corresponding impairments may change over time, and hence, the ML models used may need to be retrained using a newly collected training dataset [88]. Therefore, the effective retraining of ML models is also highly desirable.

The most popular deep learning is now based on the use of neural networks, so here we start with a summary of different neural networks that have been widely used in OPM and MFI in short-reach optical communications. ANN can be used to model the general relationship between the input and output data sets and are simple to implement [84,89]. The initial application of ANN in OPM can be traced back to 2006, which was used for the OSNR monitoring. It combines wavelet transform (WT) and ANN attributes to detect and classify fast power transients in optical networks [90]. The typical structure of an ANN is shown in Figure 4. The ANN model simulates the electrical activity of neurons in brains and the nervous systems. Generally, processing elements (neurons) are arranged in a layer or vector, and the output of one layer is used as the input of the next layer. A neuron can connect to all or part of the neurons in the next layer. Synaptic connections in the brain are modeled by these connections. The weighted data signal that enters the neuron mimics the nerve cell’s electrical excitation to simulate information transfer in the network. The input value $X_i$ of the processing unit is multiplied by the connection weight $w_{j,i}$, which simulates the learning in ANN by adjusting the strength or weight of the connection. The input-output relationship is then modeled as a more complicated model in the hidden layer under the influence of the activation function (e.g., sigmoid, tanh and Relu), whose input is multiplied by the weight and shifted by the bias, allowing the ANN algorithm to learn almost any function. The activation function f is applied as follows:

$$Z_j=f\left(\sum W_{ji}{X}_i\right)+bias$$

(1)

Then, a vector of summation function ($y_j=\sum W_{kj}^{\prime }{Z}_j$) is used to aggregate all of the weight-adjusted values to generate the output vector.

An ANN with many hidden layers between its input layer and output layer is called a deep neural network (DNN). With the increase of unstructured data such as images and videos that need to be processed, feature engineering becomes more complex and time-consuming, making DNN a more powerful tool than the single hidden layer ANN. This is because the DNN can eliminate the need for manual feature engineering by self-learning through training. In 2016, a study showed that a DNN could learn the function of the feature extractor of the incoming digital-to-analogue converters (DAC) sampled signal without manual feature engineering and form itself as an OSNR estimator [63].

The ANN discussed above is also called the multi-layer perception (MLP) neural network. When the node in each layer is connected to all nodes in the next layer, it is referred to as the fully-connected neural network. In addition to MLP, another type of neural network that has been widely studied for high-speed OPM and MFI is the CNN, which uses one or more layers of convolution units. As shown in Figure 5, CNN is composed of several pairs of convolutional and pooling layers.

The function of the convolutional layer is to extract and learn spatial features in the input data, which is typically arranged into the matrix format. This layer is generated by applying a series of different filters to the input. These filters are called convolution kernels. The filter slides element by element on the input matrix to generate a filtered output of the same size as the input. Multiple kernels will produce different filtered outputs. For the convolution of an input matrix A with kernel F to produce a convolved output O, the process can be expressed as [91]:

$$O_{i,j}=\sum _{k=1}^R\sum _{l=1}^R{F}_{k,l}{A}_{i+k,j+1}$$

(2)

CNN has been shown to be particularly suitable for image recognition and classification due to its ability of self-learning, which can automatically extract the features from different patterns without human supervision and prior knowledge [65,92]. Hence, CNN has been widely applied in OPM and MFI to process images such as eye diagrams, histograms and constellation diagrams.

In addition to the MLP and CNN discussed above, RNN has also been studied for OPM and MFI in short-reach optical communication systems and networks. Different from the previous feedforward neural networks, RNN can maintain information from the previous input using its internal memory to affect the current input and output. In addition, RNN has good performance in the correlation modeling of time-varying data and is suitable for processing the signals of optical fiber communication systems received in the time sequence format. RNN has different architectures, and the long short-term memory (LSTM) is a special RNN that can model both long-term and short-term time dependence of time-varying data. The basic structure of LSTM is shown in Figure 6. At time t, the hidden state $h_t$ is updated according to the current input $x_t$ and the hidden state of the previous time step $h_{t-1}$, which can be expressed as [89]:

$$h_t={\sigma }_h\left(W_{xh}{x}_t+W_{hh}{h}_{t-1}+b_t\right)$$

(3)

where the weight between two successive hidden states is $W_{hh}$, the weight at the input neuron is $W_{xh}$, ${\sigma }_h$ is the activation function and $b_t$ is the connection bias vector. LSTM uses three different gates (a forget gate, an input gate, and an output gate) in the LSTM cell (Figure 6) to control the information transmission, and internal self-loops to retain the long-term dependencies already learned from prior input data. Therefore, it can boost the modeling accuracy of long-term dependencies. The forget gate $f_t$ calculates the information about the cell state $C_{t-1}$ at time $t-1$ that needs to be forgotten at time t, according to $h_{t-1}$ and $x_t$. The input gate then identifies what information needs to be updated, and the $tanh$ nonlinear function produces the unit input state $\widetilde{C_t}$. Then, the updated cell state at time t $C_t$ is calculated. Finally, the cell state $C_t$ is used to determine the output $h_t$, which is then filtered by the output gate $o_t$ to produce the output sequence $y_t=[h_1,h_2,\dots \dots H_T]$. The forget gate ($f_t$), input gate ($i_t$), output gate ($o_t$), input state ($\widehat{C_t}$), cell state ($C_t$) and hidden state ($h_t$) calculations at time t in the forwards pass of an LSTM cell are [89]:

$$f_t={\sigma }_g(W_f[h_{t-1},x_t]+b_f)$$

(4)

$$i_t={\sigma }_g(W_i[h_{t-1},x_t]+b_i)$$

(5)

$$o_t={\sigma }_g(W_o[h_{t-1},x_t]+b_o)$$

(6)

$$\widetilde{C_t}=tanh(W_c[h_{t-1},x_t]+b_c)$$

(7)

$$C_t=f_t\ast C_{t-1}+i_t\ast \widetilde{C_t}$$

(8)

$$h_t=o_t\ast tanh\left(C_t\right)$$

(9)

where matrices $W_f$, $W_i$, $W_o$ are the weights between the hidden layer and the three gates, $W_c$ is the weight between the hidden layer and the input state, $b_f$, $b_i$, $b_o$ and $b_c$ are the bias, and ∗ is the convolutional operator. The activation function of the three gates are ${\sigma }_g$, which is typically based on a sigmoid function. In 2019, an OSNR monitoring scheme based on LSTM to evaluate the relationship of the time-varying data and the OSNR without the need for extracting features manually has been proposed and demonstrated [89].

In addition to long-haul systems and networks, OPM and MFI technologies based on machine learning have also gained popularity in short-reach optical communication systems and networks recently. Due to the channel configuration, where the modulation format, symbol rate, interval and transmit power have become diversified, the ML technology has become a reliable tool, under the accumulated fiber propagation impairments during signal propagation [93].

Table 1. OPM and MFI applications based on machine learning.

System Info.	Features Type	ML Techniques	Output Results	Advantages	Limitations
Direct detect 2 km link 40 Gb/s DPSK Sim. [84]	-Eye diagram -Eye histogram	-ANN	-OSNR -CD -PMD	-Simultaneous identification of CD PMD	-Need clock recovery -Feature engineering dependency
Coherent EON 10–100 km 35,70,90 GBd DP-QAM Sim. [93]	-Fiber nonlinear interference -Power spectral density	-ANN	-SNR -Phase noise	-ML-aided nonlinearity modeling -Power uncertainty tolerance	-Limited to few layers -High system complexity
Coherent 16 GBd DP-QPSK 64 GSa/s DAC Sim. [94]	-DAC sampled data	-DNN	-OSNR	-Learn from raw data -Save domain expertise	-Limited measurement range -Only considered ASE noise
Direct detect 20, 30 GBd PDM-QAM Exp. [95]	-AAH	-DNN	-MFI -BRI -OSNR	-Multitask handling -High MFI and BRI accuracy	-Feature engineering dependency
Coherent 80 km 20 Gbps QPSK Exp. [49]	-Spectral data	-ANN -KNN -SVM -DT	-OSNR	-High accuracy -Multifuctional spectrum analysis	-ANN-low accuracy -KNN-long training time -DT-complex structure
Coherent 14, 16 GBd DP-QAM Exp. [66]	-DAC sampled data	-CNN	-OSNR	-Without feature engineering	-High complexity
Direct detect PAM-4, RZ-DPSK, OOK Exp. [65]	-Eye diagram	-CNN	-MFI -OSNR	-Can handle complex raw eye diagram	-Feature engineering dependency
Coherent 32 GBaud QAM Exp. [96]	-Constellation images	-CNN	-OSNR -BER -MFI	-Multi-functional	-Feature engineering dependency
Coherent 4, 16, 64 QAM 100 km Sim. [97]	-Frequency domain sampled data	-LSTM	-OSNR -Nonlinear noise power	-Nonlinearity-insensitive OSNR estimation	-Feature engineering dependency

AAH: asynchronous amplitude histogram, BRI: baud rate identification, Ch.: channel, PDM: polarisation division multiplexed, QAM: quadrature amplitude modulation, SVM: support vector machine, KNN: k-nearest neighbors, DT: decision Tree.

summarizes recent studies on ML-based OPM and MFI in short-reach optical communication systems and networks. Specifically, we listed key information of the system/network considered, the extracted feature used as the input to the ML models, and the ML model structures. In addition, key advantages and limitations of these studies are also summarized in the table.

For easier overview and discussion, we classify the above approaches into direct detection and coherent detection categories according to their receiver detection method. The typical application of ML models in OPM and MFI in short-reach optical communications is summarized in Figure 7. At the transmitter side, the data to be transmitted is modulated onto the optical carrier, where both single optical carrier or multiple optical carriers via WDM can be used. The optical carriers are normally generated by laser diodes (LDs) and modulated using Mach-Zehnder modulators (MZMs). When WDM is used, the multiple output signals are then combined by the multiplexer (MUX) and can be optionally amplified by Erbium-doped fiber amplifier (EDFA) before being transmitted through the fiber or fiber network (e.g., PON). During the fiber transmission, the optical signals experience different types of impairments, such as CD and PMD dispersion, cross-phase modulation (XPM), FWM and SPM. After fiber transmission, the data-carrying optical signal is then demultiplexed (if needed in WDM) and detected by the photodetector using either direct or coherent receiver. The converted electrical signal is then sampled for processing via the ML model.

The ML studies can be split into two cases: the first case is using in-line data, such as OSNR, and the second case is using the received data at the receiver side. In the second case, most of the approaches require feature extraction generated diagrams such as the eye diagrams, histograms and constellation diagrams to train the ML algorithm. Some schemes do not require this step, where the raw received data are directly imported to the neural networks [92].

2.1. Direct Detection System

Direct detection is commonly used in short-reach optical communications, such as intra-/inter-datacenter linkages, metro, and access networks, due to its convenience for large-scale deployment, low-cost optical transceivers, and low power consumption [1,98]. The initial attempt to apply ML technology to the OPM in short-reach communications can be dated back to 2006. The authors apply the SVM and pattern recognition methods on eye diagrams to diagnose optical impairments, including CD, PMD and non-coherent crosstalk [50]. However, this approach requires noise filtering on the eye diagram before training, which increases the system’s complexity. Then in 2009, ref. [84] proposed a ML application on a direct detection system using ANN for OPM. 40 Gb/s RZ-OOK/RZ-DPSK signal is transmitted over a direct detection system. Eye diagrams of the received signal sampled by the oscilloscope are then generated, which are used as the input to the ML model. The ML model is based on the ANN, which has one hidden layer with 8 hidden neurons. The eye diagram is used to estimate OSNR, CD, and PMD. The mean squared error (MSE) is used during the training process, which can be expressed as [84]:

$$E_{T_r}\left(\omega \right)=\frac{1}{2}\sum _{n\in T_r}\sum _{k=1}^K{|y_k(x_n,\omega )-d_{kn}|}^2$$

(10)

where $x_n$ and $d_n$ are the inputs and the desired outputs of the neural network, $T_r$ is the index sequence of the training data, $d_{kn}$ is the $k^{th}$ element of $d_n$ and $y_k(x_n,\omega )$ is the neural network output for input $x_n$. Based on the experimental verification, the authors extended this approach to a WDM system for OSNR, fiber nonlinearity and dispersion measurement, where the average measurement errors of OSNR, CD and DGD are 0.77 dB, 4.74 ps/nm and 0.92 ps Gb/s for 40 Gb/s RZ-DPSK signal, respectively [84]. However, the input of the ANN comes from the eye diagram, which requires clock recovery and feature extraction, resulting in higher cost and complexity.

Based on prior studies that verified the feasibility of applying ML techniques for OPM and MFI, several multi-functional models that can provide both OPM and MFI simultaneously have been proposed. Ref. [99] proposed a multi-task model using SVM to perform MFI and ANN for OSNR monitoring on a 32 GBaud/s 64-QAM direct detection system. The SVM part for MFI is trained by the direct detected data, due to its kernel-based functions, which capture the similarities between any two pairings of raw input data. Meanwhile, the neural network part for OSNR monitoring is trained by eye-diagrams. Results show that the MFI accuracy is 94% and within the OSNR range of 4–17 dB, the mean OSNR estimation error is 0.2 dB.

Works like [99] require different ML models to implement MFI and OSNR monitoring separately. Hence, they have the drawbacks of high computational complexity and slow response. In [95], a multi-task deep neutral network (MT-DNN) has been proposed to monitor multiple optical performances in the system simultaneously. In this work, polarization division multiplexed QAM (PDM-QAM) signals are considered, and the asynchronous amplitude histogram after direct detection is selected as the feature input to the ML model. Using MT-DNN, simultaneous MFI, baud rate identification (BRI) and OSNR monitoring are realized. Since the OPM of this work contains multiple tasks, a weighted and hybrid loss function L is used in the MT-DNN, where the loss function of each OPM parameter is based on ether cross-entropy or MSE [95]:

$$L={\lambda }_1{L}_1+{\lambda }_2{L}_2+{\lambda }_3{L}_3$$

(11)

$$L_1=L_2=-\frac{1}{m}\left[\sum _{i=1}^m{y}_i{log}_2\widehat{y_i}+(1-y_i){log}_2(1-\widehat{y_i})\right]$$

(12)

$$L_3=\frac{1}{m}{\left(\sum _{i=1}^m{y}_i-\widehat{y_i}\right)}^2$$

(13)

where $L_1$, $L_2$ and $L_3$ are the loss functions of MFI, BRI and OSNR monitoring, respectively, ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ are the adjustable weights for the three tasks to optimize the overall monitoring results, m is the number of samples, $y_i$ is the actual output, and $\widehat{y_i}$ is the prediction from MT-DNN. The MT-DNN includes 5 layers with 80, 160, 80, 40, 6 neurons in each layer. The first three layers’ activation function is ReLU, whereas in the last two layers, softmax is used for BRI and MFI, and the linear function is used for OSNR monitoring. The accuracy rates of MFI and BRI can reach 100% and 96.8%, respectively [95]. However, the OSNR measurement range of this work is reduced when compared with previous works, limiting to 10–20 dB.

Meanwhile, ref. [65] also proposed a multi-functional ML model based on CNN, where MFI and OSNR estimation are realized simultaneously. Different from the use of asynchronous amplitude histogram [95], the eye diagram of the received signal is used as the feature input. Since CNN has the ability of self-feature extraction, it can process the eye diagram in its original form (the pixel value of the image) without having to know other eye diagram parameters (i.e., amplitude, eye histogram, jitter, rise time, fall time) or the original bit information. Therefore, the cost of manual intervention is reduced. In this scheme, the CNN consists of two convolution layers and two pooling layers. The outputs are then mapped into a one-dimensional layer, which is further fully connected to the output layer with 20 neutrons (4 for MFI and 16 for OSNR estimation) [65]. Results show that with 1600 training data, the OSNR estimation accuracy can reach 100%. In addition, 100% accuracy for MFI is also obtained with 800 training data.

2.2. Coherent Detection System

With the rapid technology development, the cost of coherent receivers is declining. In recent years, the application of coherent detection in short-reach optical communication systems and networks to improve spectral efficiency and data rate has also become a new development direction. For example, due to the high receiver sensitivity, frequency selectivity, and scalability, the coherent long-reach PON (LRPON) is regarded as a good choice for providing beyond 100 Gbit/s access, which typically targets at up to 20 km distance connection between the OLT and the ONU [59].

Several reported studies are showing the capability of ANN-based OPM and MFI for short-reach coherent systems [100,101]. In [101], the authors proposed a method based on the combination of ANN and AAH evaluation for OPM, and it can also identify the symbol rate and modulation format at the same time. The ANN model of this work consists of three layers, where there are 200 neurons and 25 neurons in the input layer and the hidden layer, respectively. The MSE was used to train the neural network to improve classification performance. The results reveal that this scheme can monitor CD in the range of 200–1600 ps/nm, with a DP-QPSK OSNR of 9–19 dB and a DP-16QAM OSNR of 14–26 dB. The estimation accuracy can reach $1.317\times {10}^{-6}$ after 40 training epochs.

However, the data set for training typical ANN-based OPM techniques is often manually selected [93,101], which limits the scalability to more broad signal sets with different symbol rates and modulation formats. The authors of [49] compared the effectiveness of various ML models for OSNR estimation and center wavelength detection of a QPSK coherent system with up to 80 km transmission distance by analyzing the OSA detected optical spectrum. Thirty optical pre-processed spectral data from 1551 to 1554 nm are collected as the training data set. Each spectral consists of 4096 samples. This work compared the training time and detection accuracy of four ML algorithms (k-nearest neighbor (KNN), decision tree (DT), SVM, ANN) with the same parameters. Results show that SVM has the highest estimation accuracy and lowest training time compared with KNN, ANN and DT. However, the author did not give the details of these ML models, such as the number of nodes in each layer of ANN and the loss function used.

In order to save the cost of feature extraction, a WDM system using the DNN trained with time-domain raw data for OSNR monitoring is proposed [94]. Since the multiple hidden layer units of DNN can automatically extract the input features for modeling complex data, the asynchronously sampled raw data from the coherent receiver is used as a feature vector for DNN training. The five layer DNN contains 2048 neurons in the input layer and is trained with 400,000 data obtained through experiments. Results show that the OSNR of a 16 GBaud/s DP-QPSK signal is successfully estimated with an average error of 1.23 dB and a range of 12 to 32 dB [94]. However, compared with the method based on the eye-diagram [101], the OSNR estimation error of this method is higher. In addition to the time domain data, the frequency domain raw data has also been explored for OSNR monitoring directly [66], where a CNN model is utilized in a single channel cohere system. The total number of training data is 1,000,000. The structure is a typical CNN consisting of an input layer, two convolutional layers, two pooling layers, and two fully connected layers (2048 nodes). The loss function for training is based on MSE. For different combinations of modulation formats and symbol rates (14- and 16-GBaud/s DP-QPSK, 16-QAM, and 64-QAM signals), the bias errors and standard deviations of the OSNR estimator are analyzed. Results show that the proposed OSNR estimator can achieve accurate estimation results with a 0.4 dB average error.

However, many ANN-, DNN- or CNN-based studies only considered the ASE noise or CD without the effects of fiber nonlinearity, such as SPM, XPM and FWM [66,84,94]. Furthermore, some of these studies are limited to single-channel systems [49,101]. Nonlinear distortion will accumulate as the fiber transmission distance increases, which is usually manifested as the crosstalk between symbols in the time domain and the spectrum broadening in the frequency domain. A method based on the LSTM network to estimate OSNR and nonlinear noise power simultaneously by using the strong long-term dependency capturing and learning capabilities of LSTM is reported, which is capable of extracting the relationship between historical and current data [97]. The proposed LSTM network has two hidden layers, each of which has double LSTM memory cells, and each cell contains 15 nodes. The coherent detected signal is translated to the frequency domain by fast Fourier transform (FFT) and then fed to the LSTM network to account for the interdependence of frequency components in a signal with nonlinear distortion. The linear function is employed as the activation function in the output layer of the LSTM network to produce the estimated OSNR and nonlinear noise power. The 5-channel system is simulated through VPI software, and results show that the OSNR estimation error is less than 1.0 dB, with the reference OSNR ranging from 15 to 30 dB for QPSK, 16-QAM and 64-QAM signals [97].

3. Machine Learning for Short-Reach Optical Communication System Signal Processing

DSP in optical communications has seen rapid and significant developments during the past couple of decades to compensate for various types of impairments, and the performance of optical communications has been substantially improved [78]. For short-reach systems, due to the widely use of receiver configurations based on the direct detection and other low-cost and small-size components, innovative signalling and DSP are critical to allow these components to reach their full potential and to meet the data rate demand in a cost-effective manner. ML-based techniques are considered to be a viable alternative to classic DSP due to their capability in solving nonlinear problems. For example, the neural network can be used to optimize the DSP function of the transmitter or the receiver [47]. In particular, in low-cost IM/DD short-reach optical fiber communication systems, where the combination of dispersion that introduces intersymbol interference (ISI) and nonlinear signal detection by a photodiode imposes severe impairments, the ML technique has been seen as a viable DSP solution to improve the system performance [1,102], where it has been applied in different manners, mainly including linear and nonlinear equalization, dispersion compensation, auto-encoder, and nonlinear predistortion, as summarized in

Table 2. Signal processing applications based on machine learning.

Applications	Features Type	ML Techniques	System Info.	Ref.
Nonlinear equalization	DSO sampled digital signal	FFNN (ANN)	32 GBd, PAM-8, IM/DD, 4 km SSMF	[103]
Linear and nonlinear equalization	DSO sampled digital signal	Compressed FFNN	112 Gbps PAM-4 PAM-8 VCSEL, 100 m MMF	[104]
Adaptive channel equalization	Training error sequence	RNN	SUI channels BPSK	[105]
Nonlinearity equalization	Frequency domain signal after CD compensation	LSTM	25 Gbaud, 16 QAM, 50 km	[54]
Signal detection and classification	Time domain Analogue pattern	ELM & RC	25 Gb/s NRZ-PAM, 56 Gb/s PAM-4, 17–55 km	[106]
Carrier recovery based on nonlinear equalization	Phase & frequency noise	Bayesian inference	50 MBaud CV-QKD system, 500 MSa/s sampling rate, 20 km SMF	[107]
Digital predistortion	Frequency-domain sampled signal	ELM	100 GSa/s sampling rate, 40 GHz bandwidth, 40 km SSMF	[108]
Auto-encoder	Time domain signal	LSTM	PAM2 and PAM4 20–80 km	[109]
Soft-demapping for nonlinear channels	Soft decision mapping relationship	DNN	800 Gb/s BtB coherent system, DP-32QAM	[110]
CD compensation	Frequency domain spectrum	DNN	32 GBd QPSK 16 QAM, 12 km 25 km SSMF	[74]
CD mitigation	Time domain digital signal	RC	32 GBd, OOK, 80 km	[111]
PMD compensation	Frequency domain sampled data	LDBP	128 Gsa/s sampling, MIMO-FIR filter, 100 km SSMF	[53]

FFNN: feed-forward neural network, SSMF: standard single mode fiber, SMF: single mode fiber, DSO: digital storage oscilloscope, MMFs: multi-mode fibers, SUI: Stanford University Interim, ELM: extreme learning machines, RC: reservoir computing, CV-QKD: continuous-variable quantum key distribution, BtB: back to back, DP: dual polarization, LDBP: learned digital back propagation, FIR: finite impulse response.

. In this section, we review recent related studies.

3.1. Machine-Learning-Based Equalizer for Short-Reach Optical Communications

With the increasing demand for higher speed and longer transmission distance, the impact of ISI, which can be caused by the bandwidth limitation of various devices/components and fiber dispersion, has become a critical limit in short-reach optical communication systems that leads to detection errors [82]. Therefore, a powerful equalizer is needed for high data rate applications. The digital feed-forward equalizers (FFEs) and decision-feedback linear equalizers (DFEs) are typically used. However, in IM/DD systems, due to the interaction among the square-law detection of PD, the laser/modulator chirp, the fiber dispersion and the additive noise, the overall channel response becomes nonlinear, making it difficult for the linear equalizer to perform accurate channel estimation and impairment compensation [103].

As a result, more complex approaches such as the maximum likelihood sequence estimation (MLSE) [112] and ML-based equalizers have been studied. ML techniques, especially neural networks, are now considered as a powerful equalization tool to compensate both linear and nonlinear distortions in optical communications. Some of them use neural networks as an aid to other signal processing steps, while others use the neural network directly as a one-stop equalizer. In these studies, the neural network has shown better performance than traditional equalizers, and hence, it becomes a very useful and popular technology for short-reach applications [113]. Among various proposed neural network architectures, the feedforward neural network (FFNN) [103,104,114], reservoir computing (RC) [82,106,111,115,116,117,118], and RNN [54,105] allow time-dependent processes (e.g., ISI) to be captured and learned effectively. Therefore, they are good candidates for the linear and nonlinear equalization of memory communication systems such as IM/DD systems [82]. Hence, this section will focus on the recent studies of using these neural network techniques for signal equalization in short-reach systems.

3.1.1. Feed Forward Neural Network Based Equalizer

FFNN has been widely used as the equalizer in short-reach optical communication systems. The FFNN used for equalization applications has the same neural network architecture as the ANN shown in Figure 4, but with a unique input data arrangement. The input of FFNN comes from a tapped delay line, which generates a delayed input time series data. Hence, we use FFNN to refer to the ANN with delayed time series data input when discussing ML-based signal equalization. Figure 8 shows the structure of an FFNN designed for signal equalization, in which the input of the ANN is fed with a time-delayed series of data received in the optical communication system. The FFNN consists of the input layer, the hidden layer and the output layer. Instead of feeding received data to the input layer directly, in FFNN, the sampled signals are fed into the input layer via the time delay array. Such an array creates different time delays, ensuring that FFNN understands both the current and the past information to conduct effective equalization. After the input layer, the received signal is further processed by the hidden layer with a nonlinear activation function to compensate for the nonlinearity in the optical communication system. Then, the distorted signals are reconstructed, and equalization coefficients are obtained at the output layer. The mathematical expression of the FFNN process is:

$$y\left(k\right)=f\left(X_k^T{W}_{ik}+B_h\right)·W_{ho}+B_o$$

(14)

where $X_k$ is the input signal sequence, $W_{ik}$ is the weight matrix between the input layer and the hidden layer, $W_{ho}$ is the weight matrix between the hidden layer and the output layer, $B_h$ and $B_o$ are the bias metrics of hidden layer and output layer, and f is the activation nonlinear function of the hidden layer.

A nonlinear equalizer based on FFNN has been proposed in an optical communication system with direct detection receivers. A radial basis function (RBF) network, which is based on the FFNN structure, is utilized to perform signal equalization to compensate the dispersion in the system [114]. The RBF method is a nonlinear equalization technology, and it takes the nonlinear transformation function of the photodetector into consideration. Hence, it is well suited for short-reach optical communications, where low cost square-law direct detection is normally used. In this study, the input function of the RBF model is expressed as:

$$r={\left[\begin{array}{cccc}y(i-1)& y(i-2)& \dots & y(i-m)\end{array}\right]}^T$$

(15)

where r is a real vector space with m dimensions expressed as $R^m$. Then, the expression of output layer after hidden layer is obtained by a $R^m$ to R (scalar) mapping:

$$\widehat{y}\left(i\right)={\omega }_0+\sum _{j=1}^{K-1}{\mathsf{\Phi }}_j\left(r\left(i\right)\right)·{\omega }_j$$

(16)

where ${\omega }_0$ … ${\omega }_{K-1}$ are the weights of the connections between the hidden layer and the output layer, and the nonlinear activation function is implemented by the hidden nodes ${\mathsf{\Phi }}_j(.)$, $1\le j\le K$. The nonlinear function selected by this work is:

$${\mathsf{\Phi }}_j\left(r\right)=exp(-{\widehat{r}}^2/{\alpha }_j^2)$$

(17)

where $\widehat{r}=\left|\right|r\left(i\right)-s_j\left|\right|$ is nonlinear function’s input parameter, $s_j$ are the RBF centers, and ${\alpha }_j$ are positive scalars known as width. In the $R^m$ space, the characteristic of nonlinearity ${\mathsf{\Phi }}_j(.)$ has a response with a radially symmetric shape around the center. In this study, the RBF equalizer is implemented offline and experimentally verified in the on-off keying (OOK) system. Results show that RBF equalizer obtained approximately one order of magnitude lower BER than FFE at 20 dB SNR over up to 100 km fiber length.

Whilst the study in [114] demonstrates the capability of FFNN, in a high-speed short-reach system, multi-level PAM is preferred compared with OOK due to the higher spectral efficiency. Hence, the capability of FFNN in the PAM-based high-speed short-reach optical communication system has also been studied [103], where 32 GBaud/s PAM-8 signal generated by an integrated electro-absorption modulated laser and transmitted over 4 km single mode fiber (SMF) has been equalized. The structures of the neural network-based equalizer is also an FFNN similar to Figure 8. The normalized least-mean-square (NLMS) algorithm minimizes the MSE of the training symbols. In this work, an FFNN with 9 input nodes is used. Results show a better equalization performance (50% lower BER at 2 dBm output power) is achieved compared with the 21-tap conventional symbol space linear FFE. However, this work assumes the channel is static, and the weights of the neural network and input layer time delay are also kept static. Thus, in reality, this FFNN model cannot adapt to possible channel changes over time.

Whilst the capability of FFNN as the signal equalizer has been demonstrated, the complexity of the neural network is an essential practical hurdle that needs to be considered. This is particularly critical in short-reach optical communications due to the more stringent complexity requirement than long-haul transmissions. To reduce the complexity, a compressed neural network equalizer using the iterative pruning approach has been proposed and demonstrated [104]. The pruning procedure is shown in Figure 9, and it reduces the network complexity, whereas the retraining procedure restores performance lost as a result of the pruning. This process of pruning and retraining is conducted recursively, and a sparse neural network structure with few connections can be generated after a number of iterations.

The compressed neural network is used as the equalizer in a VCSEL-based short-range system with 100 m MMF, and up to 112 Gbps PAM-4 or PAM-8 signals are transmitted. In the compressed neural network, the input and hidden layers have 51 nodes, and the output layer scale is decided by the modulation format, where 4 and 8 output nodes are used for PAM-4 and PAM-8 formats, respectively. The cross-entropy loss function was used to train the neural network to improve the classification performance. Rectified Linear Unit (ReLU), a prominent nonlinear activation function, was chosen as the activation function of a single hidden layer. Results reveal that, using the iterative pruning technique instead of fully connected FFNN, can result in a connection decrease of up to 71% at a BER threshold of $3.8\times {10}^{-3}$ [104]. When 2000 connections of the neural network are pruned, the BER of the one-time pruning and iterative pruning algorithms ($2\times {10}^{-4}$) is lower than that of a fully connected FFNN ($1\times {10}^{-3}$) in a PAM-8 back-to-back system. This is because that the network’s redundancy is reduced, and hence the network’s generalization ability is improved.

3.1.2. Reservoir Computing Based Equalizer

In addition to the FFNN-based equalizers, other types of neural networks have also been studied for signal equalization in short-reach optical communications. In this section, we review the use of RC, which can be considered as a highly redundant RNN with untrained input and hidden parts of the network (reservoir) [82]. RC enables the effective training of the entire neural network by optimizing only the output weights with a single linear regression step, and hence, it avoids typical training problems such as the vanishing gradient problem of typical RNN. The typical structure of the RC based equalizer is shown in Figure 10, and it also consists of the input layer, the hidden layer and the output layer.

Because it resembles nonlinear dynamical systems, the hidden layer is called a reservoir. The inputs $u\left[n\right]$ are injected into the reservoir via the linear transformation $W_{in}$ and they drive the reservoir states $x\left[n\right]$, which are characterized by the recurrent interconnections within the reservoir $W^{res}$ and output interconnections $W_{res}^{out}$ [82,116]. Therefore, the reservoir states $x\left[n\right]$ can be expressed as:

$$x\left[n\right]=\alpha ·f_{act}\left(W^{in}·u\left[n\right]+W^{res}·x[n-1]\right)+(1-\alpha )·x[n-1]$$

(18)

where $f_{act}$ is the chosen nonlinear activation function and $\alpha$ (0 < $\alpha$ < 1) is employed to simulate an exponential decay within the reservoir to meet the echo state requirement and replicate realistic physical implementations [118]. The direct connection between the input and the output layers is characterized by $W_{in}^{out}$. Then, the expression of output layer vector $y\left[n\right]$ is derived as:

$$y\left[n\right]=W_{res}^{out}·x\left[n\right]+W_{in}^{out}·u\left[n\right]$$

(19)

RC are promising tools for channel equalization in short-reach optical communications. In [116], RC are combined with the optical pre-processing, which divides the received optical signal into small sub-bands using an arrayed waveguide grating before entering the PD. RC is then utilized for digital post-processing at the receiver after photodetection. The output layer of this RC model is trained through linear regression with time domain sequences. In this study, 32 GBaud/s OOK modulated signal was transmitted over up to 60 km optical fiber. Compared with the more complex MLSE or FFNN based equalizers, results show that the proposed RC scheme can achieve a compensation distance increase from 10 km to 40 km under 1 dB SNR penalty threshold.

In [117], RC has also been used to achieve channel equalization in a WDM-RoF system. In WDM-RoF systems, due to the linear fiber distortion caused by dispersion, the fiber nonlinearity due to the Kerr effect, and the inelastic scattering of signals, the adjacent channel power ratio (ACPR) performance typically becomes worse after transmission. ACPR refers to the ratio of the overall power of the adjacent channels (i.e., inter-modulation signal) to the power of the main channel (i.e., useful signal), and it is a widely used parameter to characterize the performance of the WDM-RoF system. To overcome the degraded ACPR, RC has been utilized. According to the simulation results, the RC model with 500 nodes can generate 6∼9 dB ACPR gain in a WDM-RoF system. More significantly, the gain is directly proportional to the number of reservoir nodes within a range of 400–800 nodes. As a result, merely increasing the reservoir size can increase performance even further.

3.2. Other Machine-Learning Based Signal Processing Applications for Short-Reach Communication Systems

In addition to serving as the signal equalizer, the ML-based technique has also been extended to other types of signal processing applications, such as auto-encoder [67,109,119,120], digital predistortion [108,121] and soft-demapping [110,122]. In this section, we review recent studies of these ML signal processing applications in short-reach optical communications.

3.2.1. Auto-Encoder

In addition to dealing with the impairments of individual components or sections in optical communication systems, the ML technology can also enable us to construct a communication system by jointly optimizing the transmitter, the receiver and the communication channel in a single end-to-end process. This provides a method to interpret the entire communication system as an auto-encoder [109]. The structure of a fully-connected neural network based auto-encoder is shown in Figure 11.

The encode and decode sections are the two main parts of the auto-encoder. Encode is used to lower the input dimension, while decode is used to restore the reduced dimension code. The coding process keeps the primary features after dimension reduction, which can effectively reduce the transmission rate and improve the communication system’s reliability [67].

An auto-encoder for the IM/DD systems has been proposed and demonstrated based on the fully connected neural network and the bidirectional LSTM (BiLSTM) channel model [109]. The time domain optical signal characteristics are used, and various factors in the system are incorporated, including the nonlinearity, dispersion, attenuation, and optical-to-electrical square-law conversion. The BiLSTM, which consists of two LSTM sequences in opposite directions, allows bidirectional information to be captured. The BiLSTM network consists of four layers: the input layer, the BiLSTM layer, the hidden layer (a fully connected layer), and the output layer. The input layer consists of 51 neurons for seven classes of optical signals at different transmission distances from 20 to 80 km. The BiLSTM layer has three sub-layers, with each layer containing 32 neurons (16 in each direction). The loss function for training is based on cross-entropy. Using the time domain data for training, the proposed auto-encoder successfully models the channel with variable fiber transmission distance. Simulation results show that the proposed auto-encoder model can achieve one order-of-magnitude lower BER than the traditional PAM-2 and PAM-4 systems with a simple FFNN equalizer at the fiber length of 80 km.

3.2.2. Digital Predistortion

Digital predistortion (DPD) is a method widely used in power amplifiers for both linear and nonlinear impairments compensations. DPD is often realized using Volterra-based algorithms that employ sophisticated indirect and direct learning architectures [121].

In order to find an alternative to the Volterra-based DPD method, a low-complexity DPD based on Extreme Learning Machine (ELM) is proposed, which has the capability of fast adaptive estimation and compensation for any MZM transfer function [108]. The low-complexity ELM-based DPD has been experimentally demonstrated based on an unamplified 40 km DCI link with 400 Gb/s dual-polarization 16-QAM signal. The ELM-DPD achieves the same link budget as the more sophisticated third order Volterra-based nonlinear equalizer. Additionally, an overall link budget gain of 0.85 dB, resulting in a total link budget of 19.1 dB, has been achieved over an unamplified 40 km link when ELM-DPD and Volterra post-equalizer were combined.

ML-based (FFNN-based) DPD has also been used to improve the power efficiency of RoF links for analogue optical front-haul applications [121]. The FFNN-DPD model is trained using the received signal after ADC. The capability of FFNN-DPD is compared with the Volterra-based DPD method in this work. Results show that when the input power is 0 dBm, the FFNN-based DPD reduces the ACPR required in the system by 22 dB, while Volterra based method only reduces it by 14 dB. Since a lower ACPR means the better efficiency of the power amplifier, the FFNN based DPD method improves the RoF link performance with lower complexity compared with the Volterra-based approach.

3.2.3. Soft Demapping

The soft-decision forward error correction (FEC) and higher-order QAM data transmission have been considered as feasible technologies for achieving high-speed and high-spectral efficiency in optical communication systems. In a fading transmission channel such as the optical fiber channel, the bit interleaved code modulation (BICM) is a feasible approach for integrating channel coding and digital modulation, where the modulation constellation can be chosen irrespective of the coding rate. The BICM block’s structure is made up of a series of forwards error correction (FEC) codes, bit interleavers, and constellation mappers [123]. The soft demapper, which evaluates the soft bit (L value), is an important part of the soft decision FEC-BICM system [110]. Due to the high complexity of the Volterra-based equalization, a soft neural network-based method for nonlinear equalization and soft-decision demapping has been presented as an alternative [76].

In [76], the soft DNN (SDNN) architecture is investigated as a solution for nonlinear equalization and soft demapping. The soft DNN design consist of 17 input nodes (8 previous symbols + the current symbol + 8 future symbols) and two hidden layers with 26 and 25 nodes, respectively. The outcomes are then fed into 3 output nodes. The training process is shown in Figure 12. In the training process, symbols $x\left(i\right)$, $i=1\cdots n$ are transmitted through a coherent short-reach optical communication system. An observation value $y\left(i\right)$ is presented to the SDNN-based demapper, which represents the output of several consecutive channels before and after the ith channel. The calculated soft bit ${\ell }_i$ represents the hard decision for soft-decision FEC decoding. If ${\ell }_i>0$, then the bit is decided to be 0, and if ${\ell }_i<0$, then it is decided to be 1. The absolute value of ${\ell }_i$ indicates the degree of confidence of the soft demapper in its decision. In this work, the bit-wise equivocation is used as the loss function [76]:

$$\mathcal{L}(b,{\ell }_i)={log}_2[1+exp(-(1-2b){\ell }_i)]$$

(20)

where b is the transmitted bit and ${\ell }_i$ is the output L value of the neural network. Experiments are conducted to assess the performance and complexity of the proposed SDNN. Back-to-back transmitted 92 GBaud/s data with dual-polarized 64-QAM is utilized. Under the 15% overhead FEC restriction, the proposed bit-wise SDNN equalizer is compared to the fifth-order Volterra equalizer. Results show that the computational complexity is lowered by 65% when the same performance is achieved. Under the same complexity, an OSNR gain of 0.35 dB is achieved using the proposed SDNN, and hence, the system performance is improved.

A bi-directional RNN based soft demapper (BRNN-SD) has also been proposed in [122], which has shown better functionality than the ANN based soft demapper [76,110] on modeling the nonlinear infinite impulse response due to its ability to exhibit temporal dynamic behavior of time sequences. The BRNN-SD model is trained with pre-processed digital signal (a repeated 4096 symbols constant amplitude zero auto-correlation based training sequence). The iterative training is based on gradient descent and the back-propagation method using the adaptive moment estimation (ADAM) optimizer. The loss function is also based on the bit-wise equivocation. The capability of the BRNN has been compared with the reference Volterra based DSP. In the back-to-back system with 92 GBaud/s dual-polarization 32-QAM signals, the proposed BRNN-SD matches the performance of the reference DSP with less complexity. When a 32 channels DWDM system is considered, where each channel has a data rate of 800 Gb/s (DP-32 QAM at 96 GBaud/s), the proposed BRNN-SD method outperforms the reference DSP on the signal recovery.

4. Machine Learning Applications for Indoor Optical Wireless Communication and Positioning

Indoor OWC technology emerges as a complementary method to RF based wireless communication systems with a number of attractive features, such as extremely broad bandwidth, license-free spectrum, high communication security and anti-electromagnetic interference [124]. Among indoor OWC technologies, both the visible and near-infrared spectra have been investigated [125], corresponding to the VLC system and the infrared system, respectively. The infrared system typically adopts the near-infrared LD as the light source and PD as the detector, which have broad modulation bandwidth. However, the deployment of additional infrared transceivers increases the cost, and the maximum power of LD is limited by the laser eye and skin safety regulations [31]. Furthermore, strict alignment between the infrared transmitter and the receiver is typically needed, due to the limited coverage area of light beam and the relatively small reception angular range of the receiver. In contrast, in VLC systems, LED is adopted as the transmitter, and hence, the existing infrastructures can be used to realize illumination and communication simultaneously. Among different types of VLC technologies, optical camera communication (OCC) has recently attractive a lot of interests since it harnesses camera as the receiver with unique features, such as the larger receiver field of view (FOV), spatial and wavelength separation ability and low cost [126,127].

4.1. Machine Learning Applications for Indoor Optical Wireless Communication System

The typical structure of an indoor OWC system is shown in Figure 13, where both LOS signals and non-LOS signals can be used for the wireless data transmission. The background light normally comes from the sunlight or illumination lamps, and it leads to the additional noise of the received data. The limitations imposed by the channel, such as the path loss, ambient noise, inter-symbol interference and multipath propagation, normally affect the system performance. To solve these limitations and to improve the system performance, recently the use of ML technique has been proposed and studied, which has shown outstanding capability.

The typical applications of ML in indoor OWC systems are shown in Figure 14. In indoor OWC systems, the ML technique has mainly been used in two ways: ML-based encoder/decoder and ML-based signal processor.

4.1.1. Machine Learning Applications for Indoor Visible Light Communications

As the encoder/decoder, the paper [128] focused on the design of convolutional autoencoder (C-AE) to support the multi-carrier OFDM based OCC system. In the encoder network, one-hot vector that comes from the quantizer passes through the neural network, which includes two dense layers with the ReLU activation function, two convolution and max pooling layers, and another convolution layer with the sigmoid activation function. In the decoder network, the reverse order of the neural network structure is implemented. The results yield a 100% and 99% accuracy at the SNR of 10 dB for 16 QAM and 64 QAM transmitted signals, respectively.

The ML technique is more widely used as the signal processor in indoor VLC systems. A memory-controlled deep LSTM neural network post equalizer to mitigate both linear and nonlinear impairments in a high-speed PAM-based VLC system has been studied [129]. The proposed post equalizer has a hidden layer consisting of a memory controlled deep LSTM neural network (32 layers), and a softmax function-based output layer corresponding to the modulation order. Experiment results show that compared with the conventional FIR-based equalizer, the Q factor can be improved by 1.2 dB using the LSTM-based post-equalizer, and the transmission distance is increased by over 30%. When further compared with the Volterra-based equalization nonlinear scheme, the complexity can be significantly reduced.

In [130], the researchers proposed a Z-Score Averaging neural network (Z-NN) algorithm to reduce the ISI in rolling shutter based OCC system. The raw gray-scale values in R, G, B channels are processed by the Z-score averaging algorithm [131], and then fed into the FFNN, which has a hidden layer with four neurons and a sigmoid activation function output layer. The cross-entropy loss function is used to update the weight parameters. Compared to the results of two classic methods, at a transmission distance of 2.5 m, the traditional Extreme-Value-Averaging (EVA) scheme cannot decode the received pattern, whereas the logistic regression scheme can satisfy the FEC threshold at 960 bit/s, and the Z-NN algorithm can further improve the bit rate to 1200 bit/s. The ISI in rolling shutter based OCC systems has also been suppressed using a more complicated two-dimensional CNN (2D-CNN) to further improve the data transmission performance [132]. This 2D-CNN takes extracted signals in R, G, B channels as the input, followed by two cascaded convolution layers with different numbers of kernels and kernel sizes. Each CNN layer also adopts the ReLU activation function and batch normalization to improve the performance. A fully connected layer with eight neurons is then used for classification, and the softmax function produces the probability distribution of the symbols. The experiment compares the BER performance obtained by the ANN, 1D-CNN and double-equalization [133] at different data rates, and results show the 2D-CNN has the lowest BER.

4.1.2. Machine Learning Applications for Indoor Near-Infrared Communication

In the studies reviewed above, they all use the visible light range for data transmission. In addition to VLC systems, the ML technique has also been explored in OWC systems using the near-infrared wavelength band. In high-speed near-infrared OWC systems, the spatial diversity scheme has been considered to overcome the channel blockage problem and to avoid possible communication interruptions. However, the delay between different channels may cause a severe ISI problem. The ML technique has been investigated to solve this issue, where the study [134] demonstrated a RNN-based symbol decision scheme to solve fractional symbol-period delays (i.e., delays less than one symbol period) induced ISI. The proposed RNN algorithm has three layers: the input layer (IL), the bi-directional LSTM layer (BL) and the output layer (OL). At the IL, each symbol sequence is divided into multiple groups, which are fed into the BL sequentially. BL consists of two LSTM sub-layers: a forward layer and a backward layer for learning the dependency among the data. Finally, the outputs from the BL are cascaded and passed through a softmax layer in the OL to obtain the symbol decision. The capability of RNN has been experimentally demonstrated, where more than one-order-of-magnitude BER improvement has been achieved when compared with traditional method (i.e., selecting the optimum sampling points), and the RNN also performs better than the previously studied FC-NN method.

To handle longer channel delays (i.e., delays more than one symbol period), an attention-augmented long-short term memory (ALSTM) RNN symbol decision scheme has been further developed [135], where an additional attention layer is added before the output layer. The attention layer keeps the states of LSTM cells at all processing steps, and hence, the dependencies among the received symbols can be better captured, especially among non-neighboring symbols. The ALSTM based method shows over one-order-of magnitude BER improvement compared with the RNN without attention layer.

In addition, previous studies mainly implement the ML algorithms using high-end GPUs, which are typically expensive and power hungry, limiting practice applications (particularly the short-range indoor applications). To solve this problem, a FPGA-based RNN hardware accelerator has been proposed for indoor near-infrared OWC systems [136]. Results show that compared with GPU-based implementations, similar BER performance can be obtained, while the latency and power consumption can be reduced by 61% and 58.1%, respectively.

4.1.3. Other Machine Learning Based Applications in Indoor OWC System

In addition to the applications mentioned above, which focus on signal processing, the ML technique has also been applied in other aspects of indoor OWC systems. In [137], a ML-based feature separation network for transmitter fingerprinting (TF-FSN) to distinguish illegal transmitters and to improve the physical layer security in the indoor VLC system has been proposed.

The proposed neural network structure is depicted in Figure 15. The feature separation network consists of three parts: the feature extractor (FE), the label predictor (LP) and the feature classifier (FC). The FE includes three residual stacks which contain two residual units, and each residual unit contains one convolutional layer, one batch normalization layer and one ReLU activation layer. Then, the outputs are fed into a fully connected layer for dimension reduction, whose output are further evenly divided as the inputs of the LP and FC, respectively. The LP works as a fully connected layer, which classifies the high-dimensional private feature into different labels. At the FC, a loss function with the soft subspace orthogonality constraint is used to optimize the training process, so that the interference of channel effects can be eliminated from the extracted private features. This algorithm is verified by experiment, showing 92.65% and 98% accuracy of identification and verification, respectively. The system also shows strong robustness at different distances over a wide range of SNR.

The ML technique has also been applied to the handover between LiFi access points (APs) and WiFi APs in a heterogeneous indoor wireless system, where a fully-connected neural network with one hidden layer is utilized [138]. Four features including the user’s speed, the separation between LiFi APs, the height of LiFi APs, and the number ratio between LiFi APs and WiFi APs are selected as the input parameters. The tanh activation function is used at the output layer to capture nonlinear effects in the system. Simulation results show that compared to the received signal strength (RSS)-based and trajectory-based handover methods, using the ML based approach can improve the user throughput by up to 260% and 50%, respectively. The

Table 3. A summary of current ML based applications in indoor OWC system.

Wavelengths	Applications	ML Algorithms	References
Visible light	Encoder/decoder	CNN, DBN, AdaBoost ANN	[128,139,140]
	Equalization	K-means, LSTM, MLP, CNN, DNN-LMS	[129,130,132,141]
	Physics layer security	CNN, K-means	[137,142]
	Heterogeneous handover	ANN	[138,143,144]
Near-infrared	Delay tolerance	LSTM-RNN, ALSTM-RNN	[134,135]
	FPGA-based hardware accelerator	RNN	[136]

DBN: deep belief network; AdaBoost: adaptive boosting; DNN-LMS: deep neural network-least mean square.

provides a summary of current ML based applications in indoor OWC system.

4.2. Machine Learning Applications for OWC Based Indoor Positioning

Nowadays, indoor localization is an attractive research topic due to its application in the location-based services, such as indoor navigation, opportunistic marketing and smart healthcare [145]. Since the Global Positioning System (GPS) in indoor applications suffers from severe attenuation and multipath dispersion, it is difficult to provide accurate position information. Therefore, alternative solutions have risen including RF-based and OWC-based methods [146]. For indoor localization, the RF-based methods such as Wi-Fi, Bluetooth, RFID and Zigbee have been widely investigated and applied [147]. However, due to the signal attenuation and multipath dispersion, the accuracy for RF-based indoor localization schemes is typically limited to several tens of centimeters. Compared to RF based techniques, the OWC based approaches can provide better accuracy in the centimeter scale and offer immunity to electromagnetic interference in sensitive environment, like hospitals and airplanes [148]. Similar with RF-based positioning systems, the OWC-based indoor positioning can be classified into three categories: the RSS, angle-of-arrival (AOA) and time-of-arrival (TOA)/time difference of arrival (TDOA) based approaches. For RSS-based methods, the received signal intensity reduces as the distance between transmitter and receiver increases, thus providing location estimation at the receiver side. TOA/TDOA explores the absolute or different time of signal arrival from the transmitter to the receiver to calculate the distance for localization. AOA refers to the propagation direction of light incident on the receiver for localization, which can typically be obtained from the image transformation and channel model [149].

However, the aforementioned approaches require precise channel models, accurate time synchronization and low background noise interference to realize accurate indoor localization. Compared to traditional OWC localization methods, the promising ML algorithms can bring better performance and efficiency to indoor OWC positioning applications. The

Table 4. A summary of current ML based indoor localization performance.

System Setup	ML Algorithm	Positioning Algorithm	Cell Size	Accuracy	Transmitter	Receiver
OOK Sim. [150]	MLP	TDOA	$5\times 5\times 3$ m	1.662 cm 2D	4 LEDs	PD
OOK FDM 50 kbps Exp. [79]	MLP	RSS	$1\times 1\times 2.5$ m	3.65 cm 2D	3 LEDs	PD
QPSK FDM 50 kbps Exp. [151]	MLP	RSS	$0.9\times 1\times 0.4$ m	<1 cm 3D	3 LEDs	PD
OOK 5 W LED Exp. [152]	MLP	Transfor-mation	$0.85\times 0.85\times 1.85$ m	1.49 cm 3D	4 LEDs	Camera
OOK 10 MHz LED Sim. [153]	MLP CNN KNN	RSS based finger-printing	$5\times 5\times 3$ m	21.4 cm 17.15 cm 29.5 cm 3D	multiple IR-LEDs	PD
OOK CDMA 9 W LED Sim. [80]	ELM	RSS based finger-printing	$0.8\times 0.8\times 1.5$ m	3.65 cm 3D	multiple LEDs	PD
OOK TDM 2 Mbps 25 W 3 MHz LED Sim. [154]	SVM, RF, DT, KNN	RSS	$5\times 5\times 2.5$ m	8.6–13 cm 2D	4 LEDs	PD
OOK Exp. [155]	TLFM	Fingerprints (RSS, DIFF, HLF)	$3\times 2\times 2.5$ m	5 cm 3D	4 LEDs	PD

DT: decision tree, RF: random forest, TLFM: two-layer fusion network, DIFF: difference between pairs of signal strength, HLF: hyperbolic location fingerprint.

provides a summary of current ML based indoor localization performance.

The study [150] proposed a TDOA based ANN method for two-dimensional indoor localization. The obtained time difference of the received signal from different LEDs works as the input feature to the ML model. A simple MLP algorithm is used to train the datasets, and a sigmoid activation function is used to map the actual position at the output layer. Simulation results show that 1.662 cm positioning error can be realized in a $5\times 5\times 3$ m room. In [151], the researchers demonstrated a two-layer ANN to achieve RSS-based three-dimensional indoor positioning. The RSS information is collected from three LEDs as the input of the ANN. The sigmoid activation function is applied in the two hidden layers, with 8 nodes and 5 nodes, respectively, and there is no activation function used in the output layer. The Adam optimization algorithm is adopted for training the proposed deep learning model. The measured results show that the average positioning error is less than 1cm within a $0.9\times 1\times 0.4$ m unit, which has been significantly improved when compared with the one hidden layer algorithm.

Currently, OWC-based localization schemes mostly assume the receiver and transmitter planes are parallel, which ignore the critical tilt angle of the receiver in practical scenarios. To address this problem, some researchers applied the inertial measurement unit (IMU) to measure the user equipment (UE) tilt angle for positioning [156]. However, the accuracy of IMU in indoor environment may not be guaranteed, and the extra device can increase the cost of the system. From this point of view, ML algorithms can provide a promising solution to estimate the orientation of receiver and to achieve better positioning performance. In [153], the authors employed two deep ANN models, one based on the MLP and the other based on CNN, to map the instantaneous received SNR to the user 3D position and the UE orientation efficiently.

Figure 16 illustrates the proposed simultaneous position and orientation estimation algorithm. For the proposed MLP, the pre-processed SNR of the received signals work as input features. There are d hidden layers where each hidden layer consists of the dense layer, ReLU layer, dropout layer and normalization layer. The proposed CNN has a similar structure except replacing the dense layer by the convolution layer. The output layer consists of six neurons, where each output neuron is related to one of the three position parameters and the three angle parameters ($x,y,z,\alpha ,\beta ,\gamma$). Results show that for an indoor room with a size of $5\times 5\times 3$ m and considering both LOS and NLOS channels, when compared with the popular KNN algorithm, which achieves estimation accuracies of 29.5 cm, 17.7° and 3.925° for the position, $\alpha$, and $\beta /\gamma$, respectively, and the proposed MLP and CNN can realized better performance: the MLP algorithm can achieve 21.4 cm, 15.5° and 2.17°, respectively, and the CNN can achieve 17.15 cm, 12.5° and 2.136°, respectively. Besides, ref. [157] proposed a MLP algorithm to correct the position error caused by the tilt of the receiver in an OCC-based indoor positioning system, since when the tilt angle is different, the features like positions and shapes of the LEDs on the captured images are also different. At the input layer, five features of the projected ellipse image of the LEDs, including two position coordinates ($x,y$), two axis parameters (major axis and minor axis) and the area, are fed into the MLP. The sigmoid activation function is used to map the nonlinear relationship between the input and the hidden nodes. The output feature is the distance between the transmitter and the receiver. Results show that with different numbers of hidden nodes 10, 30, and 50, the average positioning error is 3.6 cm, 2.1 cm and 1.9 cm, respectively, which significantly reduces the impact caused by the tilt camera.

So far, most papers of ML-based indoor OWC localization applications only focus on improving the accuracy of position, while ignoring the importance of LED-ID detection and recognition. In [158], the authors design a ML-based LED identity (LED-ID) detection and recognition method for visible light-based positioning systems. It adopts the pulse width modulation (PWM) method with different frequencies and duty cycles to create unique features for each LED, so that the unique pattern of the image can be captured by camera. Two typical linear classifiers, named fisher classifier and SVM, are applied to classify the captured bright and dark stripes patterns from different LEDs. Experimental results show that the proposed scheme can improve the ability of LED-ID detection to classify a large number of LEDs for large scale indoor positioning systems.

5. Conclusions

In this article, we have provided a comprehensive survey for the recent ML related research in short-reach optical communications (<100 km). This survey covers three fields of the state-of-the-art ML-based applications: OPM and MFI, signal processing and in-building/indoor OWC systems. The structure, advantages and limitations of these ML techniques have been discussed in detail. The emergence and development of ML techniques can save the cost of hardware and integration compared to non-ML techniques. In addition, ML-based OPM techniques are flexible and reconfigurable. Hence, it can adapt to dynamic and increasingly complex network structures and modulation formats. Furthermore, ML-based techniques have been developed to be no longer limited to a single function. In terms of OPM and MFI, some multi-functional approaches have been proposed to undertake the estimation of multiple OPM parameters and perform modulation formats classification simultaneously. In the field of signal processing, ML-based techniques are not limited to simple nonlinear equalization either. Auto-encoders that can optimize the system in an end-to-end manner are under study, and soft-demappers are under reach. In indoor OWC systems, the ML has shown excellent performance to solve the limitations such as pass loss, ambient noise and multipath propagation. For indoor OWC positioning systems, ML has been widely used to improve the position accuracy, realize simultaneous position and orientation estimations, and achieve the recognition of LED-IDs.

However, the capability of many ML-based applications to handle feature extraction and complicated analysis is restricted, which normally require manual feature engineering. As a result, the ML techniques based on multi-layer neural network architectures, such as CNN, DNN, and LSTM, have recently been proposed to automatically learn features from raw data. Nonetheless, these approaches require a large amount of data for training, and some of them need pre-processing the raw data to improve accuracy. The combination of ML technologies and domain expertise offers predictive potential in the absence of big data sets or incompletely representative data, and it may become the critical study direction in the future. On the other hand, ML-based techniques are sensitive to network environment changes. System instability or reconfiguration can result in the requirement of retraining the ML model. How to perform real-time efficient retraining is also an important research topic for future exploration.