Error Probability of a Coherent M-ary PSK FSO System Influenced by Phase Noise

Abstract

In this paper, we aim to develop an analytical framework for design and analysis of new generation mobile networks fronthaul/backhaul links based on the application of free-space optical (FSO) technology. Taking the receiver hardware imperfections into account, we present an efficient analytical approach in analyzing average symbol error probability (SEP) of the coherent FSO system employing M-ary phase-shift keying (PSK). Optical signal transmission is influenced by pointing errors and atmospheric turbulence. The signal intensity fluctuations caused by atmospheric turbulence are modeled by general Málaga ($\mathcal{M}$) distribution, which takes into account the effect of multiple scattered components. We estimate the range of the signal-to-noise ratio at which the SEP floor appears, as well as the value of this non-removable error floor. The results illustrate that the effect of imperfect phase error compensation on the SEP is more critical under weaker turbulence conditions and for higher order modulation formats. Based on the analytical tools presented here, it is possible to estimate tolerable value of standard deviation of phase noise for the given value of SEP. This value of standard deviation is an important parameter in designing the phase-locked loop filter in the receiver.

1. Introduction

1.1. Background and Related Work

Mobile network data traffic grew 40 percent between the first quarters of 2021 and 2022, respectively. According to [1], at the end of 2021, there were around 8.2 billion mobile subscriptions, and this number is expected to increase to around 9.1 billion by the end of 2027. During the same time, the share of mobile broadband subscriptions will increase from 84 to 93 percent. Implementation of the fifth generation (5G) mobile systems has been enabling broadband data transmission. These systems should support high user data rates (up to 10 Gbps) and high reliability (about 99.999), low energy consumption and low latency (less than 1 ms), as well as high security levels and density of user terminals. These demands will be achieved by using a small-cell concept with a combination of millimeter wave (mmWave) technology. Besides designing antenna systems for mmWave 5G wireless communication services [2], special attention should also be paid to designing links between core network and access nodes (backhaul), as well as links between central unit and multiple remote distributed units (fronthaul). The designing of backhaul and fronthaul parts of 5G and beyond networks becomes an important challenge [3,4]. Besides optical fiber and millimeter waves, free-space optics (FSO) technology has been recognized to be able to fulfill the requirements of 5G and beyond backhaul and fronthaul networks. Generally, FSO is a promising technology for first-mile and last-mile links because of larger bandwidth, greater immunity to electromagnetic interference, better security and greater flexibility compared with traditional radio-frequency systems [5].

The signal propagation over an FSO link is influenced by several impairments. The intensity of optical signal propagating over an FSO channel is a random process due to turbulence-induced scintillation. Actually, inhomogeneity in atmospheric temperature and pressure cause variations of refractive index along the optical beam propagation path. The resulting atmospheric turbulence influences the signal phase and intensity fluctuations [6]. The stochastic variations of optical signal irradiance can be statistically characterized by different distributions, such as K, Gamma–Gamma, Fisher–Snedecor and exponential Weibull distribution [5,6,7,8]. In addition, Málaga ($\mathcal{M}$) distribution was proposed for modeling the atmospheric turbulence channel when there is a light line-of-sight component (LOS), as well as scattered components by the eddies on the propagation axis and off-axis eddies. Some other distributions are particular cases of Málaga ($\mathcal{M}$) distribution for specific numerical values of its parameters [9,10]. In [11], the authors used the so-called mixture Gamma distribution to study the outage and error performance of FSO systems. The mixture Gamma distribution was initially proposed in [12] with the aim to approximate more complex Málaga ($\mathcal{M}$) distributions in estimating FSO system performance.

Besides turbulence-induced scintillation, misalignment fading (also known as pointing errors) has the strong influence on FSO system performance as well. Pointing errors are caused by building sway, thermal expansion and wind. This phenomenon consists of boresight and jitter. Boresight exists when there is the fixed displacement between transmitter laser and receiver detector. Jitter corresponds to random offset of received beam center at the detector surface. Both effects were considered in [12], while the effect of jitter was emphasized in [13,14].

Nowadays, FSO systems are mostly based on intensity modulation with direct detection (IM/DD) and on-off keying [5]. To achieve the best performance of these systems, it is necessary to adapt the decision threshold under varying turbulence conditions. In order to avoid adapting a threshold, subcarrier intensity modulation (SIM) technology was proposed for FSO links and has been extensively studied [5]. In the receiver of SIM-based systems, the optical signal is firstly converted to an electrical one. After that, the demodulation and detection are performed in the electrical domain. It is necessary to estimate the received signal phase variation in the process of demodulation. This estimation is often performed by the phase-locked loop (PLL). The received signal phase recovery is not perfect. Actually, there is a difference between received signal phase and estimated phase. This difference is a random process, and it is known under the name of phase noise. The effect of phase noise on SIM-based systems was considered in [15,16,17,18,19,20]. In [15], we studied the deleterious effect of imperfect reference signal recovery on detection of SIM-based multilevel phase-shift keying (MPSK) signals over strong turbulence conditions. Song et al. [16] studied the effect of phase noise on detection of SIM-based binary PSK and quaternary PSK signals over the FSO channel with lognormal scintillation, while this analysis is extended to M-ary PSK signal detection in [17]. In [18], a similar analysis was performed for the case of Gamma–Gamma turbulence induced scintillation together with pointing errors, where the approximate expressions for error probability were derived. In [19], the authors derived approximate formulas for error probability during diversity reception of SIM-based MPSK signals transmitted over an FSO channel in the presence of phase noise. Varotsos et al. [20] analyzed the effect of phase noise on error performance of the SIM-based PSK system over the FSO channel with Málaga ($\mathcal{M}$) turbulence induced scintillation. They derived approximate expressions for SEP for single-hop and multi-hop FSO decode-and-forward relaying configurations. In our previous work [21], we analyzed the performance of the SIM-based M-ary differential PSK FSO system over Gamma–Gamma turbulence channel and pointing errors. In that case, our aim was to estimate the effect of phase noise generated by the local oscillator used for down-conversion in the differential PSK receiver. In [22], we gave a review of papers considering the effect of phase noise on the performance of SIM-based FSO systems. Based on the results of those contributions, it is evident that phase noise has a strong effect on error performance of SIM-based PSK FSO systems.

1.2. Motivation

Results from [23] proved that coherent FSO systems have improvements of 24–30 dB compared with SIM-based FSO systems over different turbulence conditions. Such high-performance FSO systems can be used in backhaul as an integral part of 5G beyond networks to provide a seamless connection with fiber optic counterparts [24,25]. Recently, with the advent in coherent detection technology that allows for background noise rejection, high sensitivity, and improved spectral efficiency, there has been a resurgence of interest in coherent FSO communications with coherent homodyne/heterodyne receivers [26], and references therein. Some deep tech companies have been developing 100 Gbs coherent FSO systems capable of transmitting data over distances up to 10 km [27]. It is obvious that coherent FSO systems, regardless of the fact that they are more complex to implement compared with IM/DD FSO systems, will play a significant role in development of modern mobile networks in the near future. This research is exactly along this line and aims to provide a theoretical basis not only for evaluating the performance of these systems, but also for their design.

At the transmitter of a coherent FSO system, an information bearing signal modulates a lightwave signal by applying amplitude, frequency or phase modulation. At the receiver, a received optical signal is combined with a signal of local oscillator (LO), and the resulting signal is converted by a photodetector to electrical signal. The difference between a received optical signal phase and an LO output signal phase is known as a phase error. This phase error is compensated by PLL in an electrical part of the receiver. In the case when the frequency of LO differs for several gigahertzes from the frequency of the received optical signal, heterodyne detection is performed [28,29,30]. The imperfect phase error compensation in the electrical part of the coherent receiver is a significant deleterious effect in coherent FSO systems. This imperfect phase error compensation is also known as phase noise. The error probability degradation caused by the phase noise in coherent FSO systems was examined in [28,29]. The analysis related with the effect of phase noise in both works was carried out for binary phase-shift keying (BPSK) modulation format, and the results showed that the standard deviation of the phase noise larger than 20 degrees causes undesirable error probability degradations. The error rate analysis of FSO communication system with coherent detection in the presence of phase noise was further analyzed in [30] where pre-detection and post-detection equal-gain receivers of BPSK signals were considered.

Higher order modulation formats are implemented in order to increase the spectral efficiency. In that case, the effect of phase noise on the error probability is expected to be stronger because the decision areas become narrower as the modulation order increases. There is a practical need to develop an efficient analytical method for estimating the effect of phase noise on the coherent FSO system performance when higher order modulated signals are transmitted over a turbulence channel.

1.3. Contribution

Motivated by the possibility of implementing coherent FSO technology into 5G beyond backhaul and fronthaul networks, in this work, we study the effect of imperfect phase error compensation on SEP performance of the coherent FSO system. The primary aim of this work is to present the analytical framework for analyzing the error probability degradation caused by the phase noise for higher order modulation formats. In addition, in contrast to [28,29], where strong turbulence conditions were examined, and in contrast to [30], where the Gamma–Gamma model was utilized for turbulence modeling, we consider the case when atmospheric turbulence channel is modeled by Málaga ($\mathcal{M}$) distribution [9,10,31]. In addition, the random misalignment exists between the transmitter laser and receiver photodetector, which is modeled such as in [13,14].

In [28,29,30], in order to estimate the influence of phase noise on FSO system performance when binary PSK signals are transmitted, the authors applied the method using the characteristic function. In order to obtain the numerical values of the error probability, it is necessary to perform a two-fold numerical integration, whereby the sub-integral function contains the Gaussian hypergeometric function (see, for example, [26,27,28]). In contrast to these works, we provide a procedure for determining the error probability in detecting multilevel PSK signals, in the case when turbulence induced scintillation is modeled by a more general distribution. In doing so, we use the Fourier Series Method (FSM), which is suitable for asymptotic analysis that will be illustrated in the following text.

In our analysis, we use FSM for representing the probability density function (PDF) describing the received composite signal phase statistic [32]. Firstly, we derive the novel analytical expressions for coefficients in Fourier series expansion. Secondly, under assumption that phase error compensation is performed by PLL in a coherent receiver, such as in [28,29,30], we present the analytical expressions for symbol error probability (SEP) in the form of convergent series. In order to evaluate the numerical value of error probability, it is necessary to truncate the series. In order to estimate system performance as quickly as possible, it is necessary to sum the least number of terms that ensure the fixed value of the truncation error. Because of that, our analysis contains a strong mathematical background. The convergence of this series is proved, and the upper bound for truncation error is estimated. This analytical expression for SEP enables us to determine the numerical value of the error floor appearing in the system, and to identify the signal-to-noise ratio (SNR) range when this error floor appears. These two magnitudes are very important system parameters from the point of view of energy efficiency. The analytical results are verified by Monte Carlo simulations.

1.4. Structure

Section 2 presents the system, channel, and receiver model. Section 3 contains four subsections. In the first part of Section 3, we derive novel expressions for Fourier coefficients in representing the PDF of the received signal phase. These coefficients are further incorporated into the formula for SEP presented in the second part of Section 3. The third part of Section 3 presents convergence analysis of the formula for SEP. In the fourth part of Section 3, we illustrate that our approach is convenient for asymptotic analysis and derives a simple formula for estimating the SEP floor. Numerical results with appropriate comments are presented in Section 4, while some concluding remarks are given in Section 5.

2. System and Channel Model

After optical signal transmission over an FSO channel influenced by atmospheric turbulence and pointing errors, coherent detection is performed by mixing received optical signals with the LO beam. Next, the PIN photodetector (PD) is used to converting an optical signal into an electrical one, which is further recovered by the PSK demodulator in the electrical domain. The block-scheme of the system considered here is presented in Figure 1. Based on the quite similar analysis presented in [28,29], the electrical signal at the MPSK receiver input can be presented as

$$i\left(t\right)=2R\sqrt{(}{P}_s{P}_{\mathrm{LO}})cos({\omega }_{\mathrm{IF}}t+{\varphi }_m+\varphi )+n\left(t\right),$$

(1)

where R denotes the PD responsivity, optical signal power is equal to $P_s=A_{r}I$, where $A_r$ is detector area, and I represents the optical signal irradiance. The LO signal power is denoted by $P_{\mathrm{LO}}$, and ${\omega }_{\mathrm{IF}}={\omega }_{\mathrm{OS}}-{\omega }_{\mathrm{LO}}$ is the difference between the optical and LO signal frequencies, respectively. The encoded signal phase information is denoted by ${\varphi }_m$, and $\varphi$ is the random signal phase generated due to transmission over an FSO channel. The system is affected by zero-mean additive white Gaussian noise (AWGN) with variance ${\sigma }_n^2=qR{P}_{\mathrm{LO}}T/2$, where q is an electronic charge, and T is symbol interval [28]. The instantaneous SNR per symbol is defined as [28]

$$\gamma =\frac{4R^2{P}_s{P}_{\mathrm{LO}}}{2{\sigma }_n^2}=\frac{R{P}_{s}T}{q}=\frac{R{A}_{r}T}{q}I=\frac{{\eta }_e{A}_{r}T}{h\nu }I=C_{c}I.$$

(2)

The responsivity is equal to $R={\eta }_{e}q/\left(h\nu \right)$, where ${\eta }_e$ is the detector quantum efficiency, h is Planck’s constant, and $\nu$ is the frequency of the received optical signal. The multiplicative constant for coherent FSO system is defined as $C_c={\eta }_e{A}_{r}T/\left(h\nu \right)$.

The optical signal transmission via free space is affected by $\mathcal{M}$-distributed atmospheric turbulence and pointing errors. Since it takes into account the effect of multiple scattered components, the $\mathcal{M}$ distribution represents a more general model compared to mostly adopted Gamma–Gamma distribution [9,10]. Besides the component $C_L$ occurring due to LOS contribution, there is the component $C_{CS}$ scattered by the eddies on the propagation axis, and it is coupled to the component $C_L$. The $\mathcal{M}$ distribution also joins the component $C_{GS}$ scattered to the receiver by the off-axis eddies. This component is statistically independent from both components $C_L$ and $C_{CS}$ [10]. The PDF of irradiance I accounted for both $\mathcal{M}$-distributed atmospheric turbulence and pointing errors is ([10], (21))

$$f_I\left(I\right)=\frac{{\xi }^2\mathit{A}}{2}{I}^{-1}\sum _{k=1}^{\beta }{a}_k{G}_{1,3}^{3,0}\left(\frac{\alpha \beta I}{(g\beta +{\Omega }^{\prime })A_0}{\displaystyle |}\begin{array}{c}{\xi }^2+1\\ {\xi }^2,\alpha ,k\end{array}\right),$$

(3)

where $G_{p,q}^{m,n}\left(·\right)$ is Meijer’s G-function ([33], (9.301)), while constants $\mathit{A}$ and $a_k$ are defined as ([10], (8))

$$\begin{array}{cc}\hfill & \mathit{A}\stackrel{\Delta }{=}\frac{2{\alpha }^{\frac{\alpha }{2}}}{g^{1+\frac{\alpha }{2}}\Gamma \left(\alpha \right)}{\left(\frac{g\beta }{g\beta +{\Omega }^{\prime }}\right)}^{\beta +\frac{\alpha }{2}},\hfill \\ \hfill & a_k\stackrel{\Delta }{=}\left(\genfrac{}{}{0pt}{}{\beta -1}{k-1}\right)\frac{{\left(g\beta +{\Omega }^{\prime }\right)}^{1-\frac{k}{2}}}{\left(k-1\right)!}{\left(\frac{{\Omega }^{\prime }}{g}\right)}^{k-1}{\left(\frac{\alpha }{\beta }\right)}^{\frac{k}{2}}.\hfill \end{array}$$

(4)

The natural number $\beta$ defines the amount of fading parameter, while a positive parameter $\alpha$ is related to the effective number of large-scale cells of the scattering process. Furthermore, the parameter g represents the average power of the component $C_{GS}$ scattered by off-axis eddies, defined as $g=\mathrm{E}\left[{\left|C_{GS}\right|}^2\right]=2b_0\left(1-\rho \right)$, where $2b_0=\mathrm{E}\left[{\left|C_{CS}\right|}^2+{\left|C_{GS}\right|}^2\right]$ is the average total power of the scatter components. The parameter $\rho$$(0\le \rho \le 1)$ represents the contribution of $C_{GS}$ into the total power of scattered components. Next, ${\Omega }^{\prime }=\Omega +2b_0\rho +2\sqrt{2b_0\rho \Omega }cos\left({\varphi }_A-{\varphi }_B\right)$ is the average power from the coherent contributions, with the average power of the LOS defined as $\Omega =\mathrm{E}\left[{\left|C_L\right|}^2\right]$. The deterministic phases of the LOS and the coupled to LOS scatter term are represented as ${\varphi }_A$ and ${\varphi }_B$, respectively [10].

Related to the pointing errors, the parameter $\xi$ is defined as $\xi =a_e/\left(2{\sigma }_s\right)$, where $a_e$ is the equivalent beam radius at the receiver, and ${\sigma }_s$ is jitter standard deviation. Parameter $a_e$ is related to the parameter $a_L$ being the optical beam radius at the distance L, as $a_e^2=a_L^2\sqrt{\pi }\mathrm{erf}\left(v\right)/\left(2vexp\left(-v^2\right)\right)$, with $A_0={\mathrm{erf}}^2\left(v\right)$, $v=\sqrt{\pi }a/\left(\sqrt{2}{a}_d\right)$, where a denotes the radius of circular detector [13,14] and $\mathrm{erf}(·)$ is the error function ([33], (8.250.1)). The parameter $a_L$ is related with the beam radius at the waist, $a_0$, and the radius of curvature, $F_0$, by $a_L=a_0{(\left({\Theta }_0+{\Lambda }_0\right)(1+1.63{\sigma }_R^{12/5}{\Lambda }_1))}^{1/2}$, where ${\Theta }_0=1-L/F_0$, ${\Lambda }_0=2L/\left(k{a}_0^2\right)$, ${\Lambda }_1={\Lambda }_0/\left({\Theta }_0^2+{\Lambda }_0^2\right)$ [13,14]. The Rytov variance is defined as ${\sigma }_R^2=1.23C_n^2{\left(2\pi /\lambda \right)}^{7/6}{d}^{11/6}$, The Rytov variance determines intensity of atmospheric turbulence being defined as ${\sigma }_R^2=1.23C_n^2{\left(2\pi /\lambda \right)}^{7/6}{L}^{11/6}$, where $\lambda$ is the wavelength, and $C_n^2$ is the refractive index structure parameter.

Based on the instantaneous SNR definition in (2) and the PDF in (3), the PDF of $\gamma$ is derived as [31]

$$f_{\gamma }\left(\gamma \right)=\frac{{\xi }^2\mathit{A}}{2\gamma }\sum _{k=1}^{\beta }{a}_k{G}_{1,3}^{3,0}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)\gamma }{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}{\xi }^2+1\\ {\xi }^2,\alpha ,k\end{array}\right),$$

(5)

where $\kappa ={\xi }^2/\phantom{{\xi }^2\left({\xi }^2+1\right)}\left({\xi }^2+1\right)$, and the average SNR per symbol for the coherent detection is $\overline{\gamma }=\mathrm{E}\left[\gamma \right]=\mathrm{E}\left[I\right]C_c$ [31]. The average SNR per bit, ${\mu }_b$, is defined as ${\mu }_b=\overline{\gamma }/{log}_{2}M$.

At the receiver, coherent detection is performed by combining received optical signal with the LO beam. There is a difference between received signal random phase (${\varphi }_{\mathrm{OS}}$) and LO output signal random phase (${\varphi }_{\mathrm{LO}}$). This difference, denoted by $\varphi ={\varphi }_{\mathrm{OS}}-{\varphi }_{\mathrm{LO}}$, is also a random process. This random process is compensated by PLL in the receiver in the electrical domain. However, the compensation is not perfect, i.e., there is a difference between phase $\varphi$ and voltage-controlled local oscillator output signal phase $\widehat{\varphi }$. This difference, $\phi =\varphi -\widehat{\varphi }$, is a random process fluctuating over time, and it is known as a phase noise. The phase noise is described by a Tikhonov PDF given by [21]

$$\begin{array}{c}\hfill f_{\phi }\left(\phi \right)=\frac{exp\left(bcos\left(\phi \right)\right)}{2\pi I_0\left(b\right)},\left|\phi \right|\le \pi ,\end{array}$$

(6)

where $I_0(·)$ is the zero order modified Bessel function of the first kind, while the n-th order modified Bessel function of the first kind is defied by ([33], (8.431)), $b=1/{\sigma }_{\phi }^2$, and ${\sigma }_{\phi }^2$ is the variance of the phase noise. Furthermore, the Tikhonov PDF can be expressed in the Fourier series as [21]

$$\begin{array}{c}\hfill f_{\phi }\left(\phi \right)=\frac{1}{2\pi }+\sum _{n=1}^{\infty }{c}_{n}cos\left(n\phi \right),\left|\phi \right|\le \pi ,\end{array}$$

(7)

where

$$\begin{array}{c}\hfill c_n=\frac{I_n\left(b\right)}{\pi I_0\left(b\right)}.\end{array}$$

(8)

3. Error Rate Analysis

In this section, we present the PDF of the composite received signal phase in Fourier series form and derive the novel expression for Fourier coefficients. After that, by applying FSM, we present the formula for SEP in convergent series form, and give the proof of convergence of this series, as well as the estimation of the upper bound for truncation error. In addition, we present asymptotic analysis when average SNR tends to infinity.

3.1. Representation of PDF of Signal Phase in Fourier Series Form

In order to evaluate the SEP performance of the coherent FSO system, the PDF of the composite received signal phase $\psi$ is represented in the Fourier series form as it was suggested in [32].

The conditional PDF of the composite signal phase in a Fourier series form due to additive noise can be presented as [32]

$$\begin{array}{c}\hfill f\left(\left.\psi \right|\gamma \right)=\frac{1}{2\pi }+\sum _{n=1}^{\infty }{a}_n\left(\gamma \right)cos\left(n\psi \right),\left|\psi \right|<\pi ,\end{array}$$

(9)

where $a_n\left(\gamma \right)$ is the Fourier coefficient for the AWGN channel defined as [32]

$$\begin{array}{c}\hfill a_n\left(\gamma \right)=\frac{\Gamma \left(\frac{n}{2}+1\right)}{n!\pi }{\gamma }^{\frac{n}{2}}exp(-\gamma ){}_1{F}_1\left(\frac{n}{2}+1;n+1;\gamma \right),\end{array}$$

(10)

where ${}_1{F}_1\left(·;·;·\right)$ denotes the confluent hypergeometric function ([33], (9.21)).

The average PDF of the resulting received signal phase can be determined as

$$\begin{array}{c}\hfill f_{\psi }\left(\psi \right)=\underset{0}{\overset{\infty }{\int }}f\left(\left.\psi \right|\gamma \right)f_{\gamma }\left(\gamma \right)\mathrm{d}\gamma ,\end{array}$$

(11)

where $f_{\gamma }\left(\gamma \right)$ is the PDF of the instantaneous SNR given by (5).

After substituting (5), (9) and (10) into (11), the PDF of phase $\psi$ is given as

$$\begin{array}{cc}\hfill f_{\psi }\left(\psi \right)& =\frac{1}{2\pi }+\frac{{\xi }^2\mathit{A}}{2\pi }\sum _{n=1}^{\infty }\sum _{k=1}^{\beta }\frac{a_k\Gamma \left(\frac{n}{2}+1\right)}{n!}cos\left(n\psi \right)\hfill \\ \hfill & \times \underset{0}{\overset{\infty }{\int }}{\gamma }^{\frac{n}{2}-1}exp\left(-\gamma \right){}_1{F}_1\left(\frac{n}{2}+1;n+1;\gamma \right)G_{1,3}^{3,0}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)\gamma }{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}{\xi }^2+1\\ {\xi }^2,\alpha ,k\end{array}\right)\mathrm{d}\gamma .\hfill \end{array}$$

(12)

The product of exponential and confluent hypergeometric functions in (12) is represented in terms of Meijer’s G-function based on ([34], (07.20.26.0015.01)) as

$$\begin{array}{c}\hfill exp\left(-\gamma \right){}_1{F}_1\left(\frac{n}{2}+1;n+1;\gamma \right)=\frac{\Gamma \left(n+1\right)}{\Gamma \left(\frac{n}{2}\right)}{G}_{1,2}^{1,1}\left(\gamma {\displaystyle |}\begin{array}{c}1-\frac{n}{2}\\ 0,-n\end{array}\right).\end{array}$$

(13)

Furthermore, by inserting (13) into (12), and after applying ([34], (06.05.16.0002.01) and (06.05.03. 0001.01)), the PDF of phase $\psi$ is re-written as

$$\begin{array}{cc}\hfill f_{\psi }\left(\psi \right)& =\frac{1}{2\pi }+\frac{{\xi }^2\mathit{A}}{4\pi }\sum _{n=1}^{\infty }\sum _{k=1}^{\beta }{a}_{k}ncos\left(n\psi \right)\hfill \\ \hfill & \times \underset{0}{\overset{\infty }{\int }}{\gamma }^{\frac{n}{2}-1}{G}_{1,2}^{1,1}\left(\gamma {\displaystyle |}\begin{array}{c}1-\frac{n}{2}\\ 0,-n\end{array}\right)G_{1,3}^{3,0}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)\gamma }{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}{\xi }^2+1\\ {\xi }^2,\alpha ,k\end{array}\right)\mathrm{d}\gamma .\hfill \end{array}$$

(14)

The integral in (14) is solved by utilizing ([34], (07.34.21.0013.01)) as

$$\begin{array}{c}\hfill \underset{0}{\overset{\infty }{\int }}{\gamma }^{\frac{n}{2}-1}{G}_{1,2}^{1,1}\left(\gamma {\displaystyle |}\begin{array}{c}1-\frac{n}{2}\\ 0,-n\end{array}\right)G_{1,3}^{3,0}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)\gamma }{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}{\xi }^2+1\\ {\xi }^2,\alpha ,k\end{array}\right)\mathrm{d}\gamma =G_{3,4}^{4,1}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)}{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}1-\frac{n}{2},1+\frac{n}{2},{\xi }^2+1\\ {\xi }^2,\alpha ,k,0\end{array}\right).\end{array}$$

(15)

The final PDF of phase $\psi$ is derived as

$$\begin{array}{c}\hfill f_{\psi }\left(\psi \right)=\frac{1}{2\pi }+\frac{{\xi }^2\mathit{A}}{4\pi }\sum _{n=1}^{\infty }\sum _{k=1}^{\beta }{a}_{k}n{G}_{3,4}^{4,1}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)}{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}1-\frac{n}{2},1+\frac{n}{2},{\xi }^2+1\\ {\xi }^2,\alpha ,k,0\end{array}\right)cos\left(n\psi \right).\end{array}$$

(16)

The PDF of phase $\psi$ of the composite received signal is represented in the Fourier series form as

$$\begin{array}{c}\hfill f_{\psi }\left(\psi \right)=\frac{1}{2\pi }+\sum _{n=1}^{\infty }{b}_{n}cos\left(n\psi \right),\left|\psi \right|\le \pi ,\end{array}$$

(17)

where the Fourier coefficient $b_n$ depends on specific channel conditions. In our case, based on previous derivation, for the FSO channel influenced by the $\mathcal{M}$ distributed atmospheric turbulence and pointing errors, the Fourier coefficient $b_n$ is derived as

$$b_n=\frac{{\xi }^2\mathit{A}n}{4\pi }\sum _{k=1}^{\beta }{a}_k{G}_{3,4}^{4,1}\left(\frac{\alpha \beta \kappa \left(g+{\Omega }^{\prime }\right)}{\left(g\beta +{\Omega }^{\prime }\right)\overline{\gamma }}{\displaystyle |}\begin{array}{c}1-\frac{n}{2},1+\frac{n}{2},{\xi }^2+1\\ {\xi }^2,\alpha ,k,0\end{array}\right).$$

(18)

For channel propagation conditions observed in [28,29,30], by applying quite the similar procedure, these Fourier series coefficients can be derived in the form

$$b_n^{\mathrm{GG}}={\displaystyle \frac{n}{2\pi \Gamma \left(\alpha \right)\Gamma \left(\beta \right)}}{G}_{2,3}^{3,1}\left({\displaystyle \frac{\alpha \beta }{\overline{\gamma }}}{\displaystyle |}\begin{array}{c}1-\frac{n}{2},1+\frac{n}{2}\\ \alpha ,\beta ,0\end{array}\right).$$

(19)

3.2. Symbol Error Probability

Based on FSM, the average SEP of the coherent M-ary PSK FSO system with ideal coherent detection (the phase noise is absent) is

$$P_s^{\phi =0}=1-\underset{-\pi /M}{\overset{\pi /M}{\int }}{f}_{\psi }\left(\psi \right)\mathrm{d}\psi =1-\frac{1}{M}-\sum _{n=1}^{\infty }\frac{2b_n}{n}sin\left(\frac{n\pi }{M}\right).$$

(20)

In the case of the coherent FSO system in the presence of the phase noise, a conditional SEP is found as

$$\begin{array}{c}\hfill P_s\left(\phi \right)=1-\underset{\phi -\pi /M}{\overset{\phi +\pi /M}{\int }}{f}_{\psi }\left(\psi \right)\mathrm{d}\psi =1-\frac{1}{M}-\sum _{n=1}^{\infty }\frac{2b_n}{n}sin\left(\frac{n\pi }{M}\right)cos\left(n\phi \right),\end{array}$$

(21)

After averaging (21) over the PDF of the phase noise given in (7), the SEP of the coherent PSK FSO system is derived as

$$\begin{array}{c}\hfill P_s=\underset{-\pi }{\overset{\pi }{\int }}{P}_s\left(\phi \right)f_{\phi }\left(\phi \right)\mathrm{d}\phi =1-\frac{1}{M}-\sum _{n=1}^{\infty }\frac{2\pi b_n{c}_n}{n}sin\left(\frac{n\pi }{M}\right),\end{array}$$

(22)

where the Fourier coefficients $b_n$ and $c_n$ are previously defined in (18) and (8), respectively. The expression for the Fourier coefficients in (18) was derived for the first time in this paper. This expression allows us to evaluate the numerical value of SEP for any values of the channel parameters after application in (22).

Those numerical results related to the effect of imperfect compensation of phase error on error probability presented in [28,29] can be checked by using Fourier coefficients given by (19) in combination with formula for SEP (22).

If symbols are mapped into bits by Gray rule, then the bit error rate (BER) can be simply approximated as $\mathrm{BER}\approx \mathrm{SEP}/{log}_2\mathit{M}$ ([35], p. 271).

The proof of convergence of the series in (22) is presented in the following subsection, where estimation of the truncation error in summation (22) is also given.

3.3. Convergence Analysis

The SEP expression in (22) is re-written as

$$P_S=1-\frac{1}{M}-2\pi B,$$

(23)

where $B={\displaystyle \sum _{n=1}^{\infty }}{D}_{n}sin(n\pi /M)$ and $D_n=b_n{c}_n/n$. It holds $b_n>b_{n+1}>0$ and $c_n>c_{n+1}>0$, thus

$$D_n=\frac{b_n{c}_n}{n}>\frac{b_{n+1}{c}_{n+1}}{n+1}=D_{n+1}.$$

(24)

In addition, it is true that

$$\underset{n\to \infty }{lim}{D}_n=\underset{n\to \infty }{lim}\frac{b_n{c}_n}{n}=0.$$

(25)

Furthermore, according to ([33], (1.342.1)), it holds that

$$\sum _{k=1}^{n}sin\left(\frac{k\pi }{M}\right)=\frac{sin\left(\frac{n\pi }{2M}\right)sin\left(\frac{\left(n+1\right)\pi }{2M}\right)}{sin\left(\frac{\pi }{2M}\right)}\le \frac{1}{sin\left(\frac{\pi }{2M}\right)}=\mathrm{const}.$$

(26)

Based on (23), (24) and (25), by applying Dirichlet criterion ([33], (0.228.2) and (0.229)), it is proved that a series B in (23) is convergent.

In order to estimate the truncation error in evaluating SEP, (22), i.e., (23), is presented in the form

$$P_S=1-\frac{1}{M}-2\pi \sum _{n=1}^{\infty }{D}_{n}sin\frac{n\pi }{M},$$

(27)

where $D_n$ is a decreasing zero-sequence defined before. Replacing n by $M(n-1)+\nu$, $\nu =1,\dots ,M$, and using the fact that

$$sin\frac{\pi }{M}{[(M(n-1)+\nu ]=(-1)}^{n-1}sin\frac{\nu \pi }{M},\nu =1,\dots ,M,$$

(28)

the previous formula for $P_S$ becomes

$$P_S=1-\frac{1}{M}+2\pi \sum _{n=1}^{\infty }\sum _{\nu =1}^{M-1}{D}_{M(n-1)+\nu }{(-1)}^{n}sin\frac{\nu \pi }{M},$$

(29)

because the last term in the inner sum vanishes (for $\nu =M$). We point out here that we do not change the order of the series terms, but only group them so that they are first negative, then positive, etc.

Thus, $P_S$ reduces to

$$P_S=1-\frac{1}{M}+\sum _{n=1}^{\infty }{(-1)}^n{q}_n,$$

(30)

where

$$q_n=2\pi \sum _{\nu =1}^{M-1}{D}_{M(n-1)+\nu }sin\frac{\nu \pi }{M}>0$$

(31)

and it is easy to see that $\underset{n\to \infty }{lim}{q}_n=0$.

With this procedure, we obtain an alternative series for which we have a remainder estimate if the series is approximated by a partial sum

$$P_S=1-\frac{1}{M}+\sum _{n=1}^N{(-1)}^n{q}_n+E_N,$$

(32)

where $|E_N|\le q_{N+1}$.

The convergence acceleration can be achieved by applying the Euler–Abel transformation ([36], §1.3.2) as it was used in the recent article [37] (see also ([36], §1.3.2)).

3.4. Asymptotic Analysis

For finding a limit when $\overline{\gamma }\to +\infty$, we need to re-examine the derivation of (17). Namely, to obtain this equation, it is necessary to perform an integration step in (14), or more precisely, in (11), under the assumption that the average SNR tends to infinity. Strictly mathematically speaking, the integral (11) itself is an improper integral of the first kind that is in Riemann sense defined as the limit of proper integral when the upper bound tends to infinity. This in turn presents a problem, as we need to find a double limit

$$f_{\psi }\left(\psi \right)=\underset{h,\overline{\gamma }\to +\infty }{lim}\underset{0}{\overset{h}{\int }}f\left(\psi \right|\gamma )f_{\gamma }(\gamma ,\overline{\gamma })\mathrm{d}\gamma .$$

(33)

In the previous equation, instead of $f_{\gamma }\left(\gamma \right)$, we use notation $f_{\gamma }(\gamma ,\overline{\gamma })$ in order to emphasize that PDF depends on average SNR. On the other hand, when average SNR tends to infinity, according to (5), the probability of any other value being realized becomes infinitesimally small, as

$$\underset{\overline{\gamma }\to +\infty }{lim}{G}_{1,3}^{3,0}(\alpha \beta \kappa ·{\displaystyle \frac{g+{\Omega }^{\prime }}{g\beta +{\Omega }^{\prime }}}·{\displaystyle \frac{\gamma }{\overline{\gamma }}}{\displaystyle |}\begin{array}{c}{\xi }^2+1\\ {\xi }^2,\alpha ,k\end{array})=0,0\le \gamma <+\infty ,$$

(34)

while, for $\gamma \to +\infty$, it is indeterminate. Therefore, probability density function $f(\gamma ,\overline{\gamma })$ has the property of

$$\underset{\overline{\gamma }\to +\infty }{lim}f(\gamma ,\overline{\gamma })=\left\{\begin{array}{cc}0,\hfill & \gamma <+\infty \hfill \\ \Phi ,\hfill & \gamma \to +\infty \hfill \end{array}\right.,$$

(35)

where $\Phi \to +\infty$, in order to satisfy a probability normalization condition. A specified value for $\Phi$ stems from the indeterminate nature of a corresponding limit in (34), which permits this value to be arbitrary, including an infinite value. The property is equivalent to the following statement:

$$\underset{\overline{\gamma }\to +\infty }{lim}{f}_{\gamma }(\gamma ,\overline{\gamma })=\underset{\overline{\gamma }\to +\infty }{lim}\delta (\gamma -\overline{\gamma }),$$

(36)

where $\delta (·)$ represents Dirac’s delta function. Then, we can write (33) as

$$f_{\psi }\left(\psi \right)=\underset{\overline{\gamma }\to +\infty }{lim}\underset{0}{\overset{+\infty }{\int }}f\left(\psi \right|\gamma )\delta (\gamma -\overline{\gamma })\mathrm{d}\gamma =\underset{\overline{\gamma }\to +\infty }{lim}f\left(\psi \right|\overline{\gamma })$$

(37)

Therefore, when $\overline{\gamma }\to +\infty$, Equation (11) tends to

$$f_{\psi }\left(\psi \right)={\displaystyle \frac{1}{2\pi }}+\sum _{n=1}^{+\infty }cos\left(n\psi \right)\underset{\gamma \to +\infty }{lim}{a}_n\left(\gamma \right)$$

(38)

According to definition in (10), and ([34], (07.20.06.0007.01)), we have

$$b_n=\underset{\gamma \to +\infty }{lim}{a}_n\left(\gamma \right)={\displaystyle \frac{1}{\pi }},$$

(39)

which proves the proposition when SNR tends to infinity. Then, $f_{\psi }\left(\psi \right)=\delta \left(\psi \right)$. In that case, when SNR tends to infinity, based on (22), SEP can be estimated as

$$\begin{array}{c}\hfill P_s^{\mathrm{floor}}=1-\frac{1}{M}-\sum _{n=1}^{\infty }\frac{2c_n}{n}sin\left(\frac{n\pi }{M}\right).\end{array}$$

(40)

4. Numerical Results

Based on the convergence analysis presented in Section 3.3 ((31) and (32)) for a given value of the upper limit of the truncation error, we determine the number of terms N ($|E_N|\le q_{N+1}$ ) in the sum for the symbol error probability given by (22) (or (20) in the case of perfect compensation of phase error). Furthermore, the value of SEP is estimated by using (22), i.e., (20), in combination with (18), i.e., (8). The basic parameters influencing the error rate system performance are: turbulence strength parameters ($C_n^2$, $\alpha$, $\beta$, $\rho$) and standard deviation (${\sigma }_s$) of pointing error (related to misalignment between transmitter laser and receiver photodiode), signal-to-noise ratio per bit (${\mu }_b$), wavelength of light ($\lambda$) and modulation order (M), as well as the standard deviation of phase noise (${\sigma }_{\phi }$). For obtaining numerical results, we use values of parameters from experimental measurements [9]. The Rytov variance determines the intensity of atmospheric turbulence, with the wavelength $\lambda =785$ nm and the FSO link distance $L=1$ km. The pointing errors effect is described by the jitter standard deviation, ${\sigma }_s$. The radius of a circular detector aperture is $a=5$ cm, the beam radius at the waist is $a_0=5$ cm, and the radius of curvature takes a value $F_0=-10$. Normalized average optical power of the FSO hop is $\Omega +2b_0=1$. All numerical results are confirmed by independent Monte Carlo simulations.

Figure 2 represents the average SEP dependence on the average SNR per bit assuming different values of phase noise standard deviation. SEP decreases with increasing ${\mu }_b$, but only in the range of medium values of ${\mu }_b$. In the region of large values of ${\mu }_b$, SEP tends to a constant value called the SEP floor. This SEP floor cannot be reduced by neither increasing the signal power, nor by improving the channel conditions, but depends only on the phase noise standard deviation. Furthermore, we observe the SEP dependence on a different amount of scattering power coupled to the LOS. A higher value of $\rho$ means that the power of scattered component by off-axis eddies is smaller. Since the presented results are obtained for the same value of the Rytov variance [9], the scattering component by off-axis eddies will have a crucial impact on the system performance. The results show that a larger value of the parameter $\rho$ improves the system performance. It is expected because, when the value of parameter $\rho$ tends to one, the whole amount of scattering components power belongs to that scattering component coupled to an LOS component, and there is not a component scattered by offline eddies. In the case of $\rho =1$, the SEP performance is the best possible, since the power of $C_{GS}$ equals zero (the total received optical power is equal to the sum of the power of an LOS component and power of a scattered component coupled to an LOS component, while the component $C_{GS}$ scattered by the off-axis does not exist). Additionally, it can be observed that the amount of the scattering components has a stronger impact on the average SEP when the FSO receiver is affected by weaker phase noise, i.e., for lower values of ${\sigma }_{\phi }$. When hardware imperfections are more pronounced, the effect of the FSO channel conditions on the SEP is minor.

Figure 3 shows the dependence of the bit error probability on the signal-to-noise ratio per bit when detecting 16PSK signals. It should be noted that BER decreases with increasing SNR in the area of low and medium SNR values. However, for high SNR values, i.e., when the signal power is higher, the BER dependencies enter saturation. When BER curve reaches saturation, i.e., when the floor is reached, it no longer makes sense to increase the signal power because the performance is not improved. The value of this floor is affected only by the standard deviation of the phase noise. The higher it is, the lower the value of the floor. It should be noted that even very small changes in the standard deviation can significantly change the value of the floor. For example, if the standard deviation is increased from 2.5 degrees to 3 degrees, the floor increases by more than one order of magnitude. It is very important to notice the SNR area when the floor is reached, and it is even more important to estimate this floor. The red lines show the floor values, which are estimated based on the derived formula (40). The same figure also shows the dependence of BER on SNR when there is no phase noise, i.e., when the phase error compensation in the receiver is perfect. This dependence is shown in blue. It should be noted that this dependence is essentially different compared with a case when there is a phase noise in the receiver. Namely, in this case, BER decreases monotonically for all SNR values, i.e., there is no occurrence of floor. Based on this, it can be concluded that the influence of the phase noise is of essential importance in coherent FSO systems and that this phenomenon should be given special attention when designing the receiver.

Figure 4 aims to show the influence of the pointing errors on the SEP for different degrees of hardware imperfections. The influence of the phase noise is actually defined by the value of the standard deviation (${\sigma }_{\phi }$) of the phase noise. The higher this value is, the stronger is the influence of the phase noise. Similarly, the impact of misalignment is defined by the value of the standard deviation (${\sigma }_s$) of pointing errors. The higher its value is, the more pronounced is the misalignment. When ${\sigma }_{\phi }=8^0$, then the SEP is insensitive to the increase in the ${\sigma }_s$ up to 0.125 m. In the case when ${\sigma }_{\phi }={12}^0$, the SEP is insensitive to pointing errors as long as the ${\sigma }_s$ is less than 0.2 m. Therefore, the influence of pointing errors is stronger in conditions when the phase noise is weaker.

The average SEP dependence on the phase noise standard deviation for different values of modulation order is shown in Figure 5. Numerical results are obtained for different atmospheric turbulence conditions based on the values of the parameters given in ([9], Table I). The results show that the BPSK format is insensitive to the phase noise as long as the ${\sigma }_{\phi }$ is less than 16 degrees, while the QPSK becomes sensitive already when the ${\sigma }_{\phi }$ is 8 degrees, while in the case of applying the 8PSK format, the SEP begins to increase significantly already for a ${\sigma }_{\phi }$ value of 2.5 degrees. In addition, the effect of phase noise on SEP values is stronger when atmospheric turbulence is weaker ($C_n^2$ is lower). In other words, when $C_n^2$ is lower, the SEP begins increasing at a lower value of ${\sigma }_{\phi }$.

The numerical results illustrate that developed mathematical framework allows us to connect the required error rate performance of the system with the modulation order and parameters of the receiver under given channel conditions. Our work should not be understood only as performance analysis, but we also contribute to the design of such systems. For given conditions over the FSO channel, from the condition that a certain SEP value should be reached, it is possible to estimate the maximum value of the standard deviation of the phase noise for which this will be achieved. This value of the standard deviation of the phase noise plays a very important role in receiver design. More precisely, this information is important in the design of the phase-locked loop circuit whose role is to compensate the phase error that occurs in the coherent receiver when the incoming optical signal and the local laser signal are mixed.

5. Conclusions

We have analyzed the performance of a coherent FSO system when turbulence is modeled by general Málaga ($\mathcal{M}$) distribution including misalignment fading. Taking into account the imperfect compensation of the phase error that occurs in the coherent receiver when mixing the optical signal and the signal of the local laser, we have derived an analytical expression for the error probability in detecting multilevel PSK signals.

The results have shown that turbulence conditions significantly affect the SEP in the range of mean SNR values. The influence of pointing errors is stronger when the phase noise is weaker, i.e., when the standard deviation of the phase noise is smaller. In the case of applying higher-order modulation, the receiver becomes significantly less tolerant to the influence of the phase noise. For example, in the case of 8PSK, there is already a significant deterioration of the SEP when standard deviation of phase noise is around 2.5 degrees, while BPSK modulation format is insensitive for values of phase noise standard deviation even up to 16 degrees. The results have also illustrated that the effect of phase noise on SEP values is stronger when atmospheric turbulence is weaker.

The results show that the phase noise causes the appearance of a floor, which only depends on the standard deviation of the phase noise, and not on the signal power and turbulence conditions over the channel. The higher the standard deviation of phase noise is, the lower the value of the floor. For example, in the case of 16PSK transmission, increasing of standard deviation of phase noise from 2.5 degrees to 3 degrees will cause the BER floor to increase by more than one order of magnitude.

The results presented here show that SEP floor cannot be decreased by increasing the signal power. In our further work, we will design quasi-cyclic low-density parity-check codes to downscale this SEP floor. Our aim will be to examine if those powerful error correction codes have error floors under given scenarios.