Theoretical and Experimental Investigation of Differential Modulation and Detection in FSO Systems

Abstract

In free-space optical (FSO) communication systems, on–off keying (OOK) modulation is widely used due to its simplicity. However, systems applying OOK suffer from the BER floor in atmospheric turbulence channels, leading to persistently high BER even at high SNR. To mitigate this limitation in atmospheric turbulence channels, differential modulation and detection (DMD) can be adopted. An in-depth theoretical and experimental investigation of DMD in FSO systems is conducted in this paper, considering the effects of turbulence. A comprehensive derivation of the system performance for DMD under atmospheric turbulence channels is also provided, with the results of research revealing that DMD outperforms OOK in high-SNR regions. To validate the theoretical analysis, an experimental platform is set up to sample the fluctuation of light intensity. Furthermore, the system performance of DMD is analyzed under varying scintillation indices, modulation depths, and transmission rates in this paper. Based on the data acquired from experiments, the results corroborate the analytical findings, confirming the great advantages of DMD in turbulent environments. The insights provided in this study establish a foundation for practical FSO system design, enabling the development of simpler and more reliable communication systems.

1. Introduction

In recent years, due to its high transmission rate, strong security, license-free spectrum, and broad bandwidth, FSO technology has seen widespread adoption and rapid development in the field of communications [1,2,3,4,5,6,7,8,9,10]. As a technology which utilizes optical carriers to transmit data through unguided media [11], FSO system performance is susceptible to factors such as atmospheric turbulence and pointing errors, leading to random fluctuations in received optical power and consequently degrading the effective SNR of the link [12,13,14]. Among these, atmospheric turbulence—induced by variations in atmospheric pressure and temperature—has a particularly significant impact on system performance [3,15].

In FSO systems employing intensity modulation/direct detection (IM/DD) and OOK modulation, turbulence-induced intensity scintillation causes substantial detection errors at the receiver, severely limiting system performance [16]. Notably, OOK systems often exhibit a BER floor phenomenon, where the BER fails to drop below a certain threshold even at high SNR [17]. To address this issue, prior studies have proposed maximum likelihood (ML) detection and maximum likelihood sequence detection (MLSD) methods based on channel models and temporal correlation information, with their performance advantages validated through simulations [16,18]. However, these schemes require precise channel estimation and face challenges such as high computational complexity and implementation difficulties, which become increasingly critical under the growing demand for high data rates [19,20,21,22,23,24,25]. In addition, its practical implementation in engineering is very difficult.

Based on the characteristics of adjacent symbol variations for modulation and decision, DMD can be applied to address the BER floor in OOK at high SNR and the engineering challenges of complex modulation [26]. This paper analyzes the performance of the DMD method under various channel scenarios. Various advantages of DMD are analyzed in our research for practical applications. Our research reveals the intersection point (performance advantage point) in the BER curves of DMD and OOK. Although DMD performs modestly under low-SNR conditions, its performance surpasses the BER floor limitation in high-SNR regions, demonstrating a clear advantage over OOK. Importantly, as the scintillation index increases, this performance advantage point shifts toward lower SNR values. To explain this phenomenon, we carry out theoretical research into SNR variation with scintillation index through analytical and numerical methods. The results validate its effectiveness through experiments and analysis. Additionally, in FSO systems, modulation depth fluctuations significantly impact communication performance [27,28,29,30], degrading BER in amplitude-modulated systems. The results demonstrate that the proposed DMD exhibits unique advantages, considering the impact of the modulation depth. Even under low-modulation-depth conditions, DMD maintains a significant BER advantage over OOK at high SNR. Our further investigation also shows that in the same environmental conditions, the system performance of DMD improves with increasing symbol rate, especially at high SNR. This characteristic enables DMD to enhance system performance without requiring complex channel state estimation or other adaptive adjustments.

This paper systematically investigates the application of differential modulation and detection techniques in an FSO system. It also identifies the performance advantages of the DMD method compared to the conventional OOK method and provides both theoretical analysis and experimental validation for this phenomenon. The research focuses on three key mechanisms, including the performance advantage point of DMD compared to OOK, the impact of modulation depth on DMD communication performance, and the influence of symbol rate on the system performance of DMD. Also, an experimental investigation is conducted to validate the theoretical analysis.

2. Theoretical Investigation of Differential Modulation and Detection

Figure 1 demonstrates the basic working principle of DMD. The original received signal at the receiver can be expressed as

$$r\left(i\right)=I\left(i\right)\ast t\left(i\right)+n\left(i\right)$$

(1)

Here, $I\left(i\right)$ represents the intensity fluctuation and $t\left(i\right)\in \{0,1\}$ denotes the signal after differential modulation. $n\left(i\right)$ represents the additive white Gaussian noise (AWGN), which follows a normal distribution with a mean of 0 and variance of ${\sigma }_N^2$. To implement differential detection, we need the current received signal value $r\left(i\right)$ and its preceding symbol value $r(i-1)$.

In this study, we assume that the channel is quasi-static. This assumption is derived from our analysis of actual atmospheric turbulence data. From our research, we can observe that atmospheric turbulence has a significant number of low-frequency components, primarily concentrated below 1000 Hz. In the system, the communication rate of the DMD method remains above 20K Hz, which is far greater than 1000 Hz. Therefore, this assumption is reasonably valid. So it stands that $I\left(i\right)\approx I(i-1)$. Therefore, the differential signal can be derived as

$$\begin{array}{c}r\left(i\right)-r(i-1)=\left[I\right(i)\ast t(i)+n(i\left)\right]-\left[I\right(i-1)\ast t(i-1)+n(i-1\left)\right]\hfill \\ \\ \approx I\left(i\right)\ast \left[t\right(i)-t(i-1\left)\right]+\left[n\right(i)-n(i-1\left)\right]\hfill \end{array}$$

(2)

At the detection end, the received signal is identified by comparing the difference between two consecutive symbols. If the difference exceeds the decision threshold, it is decided as bit “1”; otherwise, it is decided as bit “0”. At the receiver, the optimal decision threshold $Opt\_Th$ is derived by leveraging channel characteristics. The received signal $R\left(i\right)$ is then obtained by comparing the absolute difference between consecutive symbols $r\left(i\right)-r(i-1)$ with $Opt\_Th$, as follows:

$$R\left(i\right)=\left\{\begin{array}{c}1,\left|r\left(i\right)-r(i-1)\right|\ge Opt\_Th\hfill \\ \\ 0,\left|r\left(i\right)-r(i-1)\right|<Opt\_Th\hfill \end{array}\right.$$

(3)

At the receiver, two types of detection errors may occur:

False “1”: When bit 0 is transmitted but incorrectly decoded as 1.

False “0”: When bit 1 is transmitted but incorrectly decoded as 0.

The corresponding error probabilities are, respectively, expressed as

$$\begin{array}{c}P\left(1\right|0)=Pr\{\left|r\left(i\right)-r(i-1)\right|\ge Opt\_Th\}{|}_{\left|t\left(i\right)=t(i-1)\right|}\hfill \\ \\ =Pr\{\left|n\left(i\right)-n(i-1)\right|\ge Opt\_Th\}\hfill \end{array}$$

(4)

$$\begin{array}{c}P\left(0\right|1)=Pr\{\left|r\left(i\right)-r(i-1)\right|\le Opt\_Th\}{|}_{t\left(i\right)\ne t(i-1)}\hfill \\ \\ =Pr\{\left|I\left(i\right)+n\left(i\right)-n(i-1)\right|\le Opt\_Th\}\hfill \end{array}$$

(5)

However, the existence of threshold currents in lasers leads to variations in modulation depth for the DMD in practical experiments. Modulation depth refers to the extent to which the optical intensity of a light signal changes after modulation, compared to its original state. It is generally used to describe the proportion of variation in light intensity. Thus, the expression for modulation depth is $m={\displaystyle \frac{1-I_{min}}{1+I_{min}}}$. $I_{min}$ represents the minimum optical intensity during modulation (the low level or ‘0’ of the signal) and can be expressed as $I_{min}={\displaystyle \frac{1-m}{1+m}}$. While this has no effect on the transmission of bit “1”, it causes bit “0” to be transformed into “${\displaystyle \frac{1-m}{1+m}}$”, which presents the depth of the modulation.

Equation (7) represents the probability of misjudgment where the transmitted symbol is ‘1’, but it is judged as ‘0’ at the decision stage. Therefore, at the decision instant, the modulation depth has an impact on the symbol ‘0’. Consequently, the aforementioned equation requires modification as follows:

$$\begin{array}{c}P\left(1\right|0)=Pr\{\left|r\left(i\right)-r(i-1)\right|\ge Opt\_Th\}{|}_{\left|t\left(i\right)=t(i-1)\right|}\hfill \\ \\ =Pr\{\left|n\left(i\right)-n(i-1)\right|\ge Opt\_Th\}\hfill \end{array}$$

(6)

$$\begin{array}{c}P\left(0\right|1)=Pr\{\left|r\left(i\right)-r(i-1)\right|\le Opt\_Th\}{|}_{t\left(i\right)\ne t(i-1)}\hfill \\ \\ =Pr\{\left|I\left(i\right)-{\displaystyle \frac{1-m}{1+m}}I(i-1)+n\left(i\right)-n(i-1)\right|\le Opt\_Th\}{|}_{I\left(i\right)\approx I(i-1)}\hfill \\ \\ =Pr\{\left|{\displaystyle \frac{2m}{1+m}}I\left(i\right)+n\left(i\right)-n(i-1)\right|\le Opt\_Th\}\hfill \end{array}$$

(7)

For the analysis of error probability, the following treatments are applied in our research. In this paper, the logarithm of the intensity fluctuation follows a normal distribution with a mean of 1 and variance of ${\sigma }_I^2$. The intensity fluctuations (due to turbulence) are modeled as a Gamma–Gamma distribution. The atmospheric turbulence channel is modeled as [31]

$$f_I\left(i\right)={\displaystyle \frac{2{\left(\alpha \beta \right)}^{(\alpha +\beta )/2}}{\Gamma \left(\alpha \right)\Gamma \left(\beta \right)}}{i}^{(\alpha +\beta )/2-1}{K}_{\alpha -\beta }\left(2\sqrt{\alpha \beta i}\right)$$

(8)

$\Gamma$ is the Gamma function. $K_{\alpha -\beta }$ is the modified Bessel function of the second kind. $\alpha$ and $\beta$ are two parameters that characterize the intensity of the scintillation index, defined as [32].

$$\alpha ={(e^{0.49{{\sigma }_I}^2/{(1+1.11{{\sigma }_I}^{12/5})}^{7/6}}-1)}^{-1}$$

(9)

$$\beta ={(e^{0.51{{\sigma }_I}^2/{(1+0.69{{\sigma }_I}^{12/5})}^{5/6}}-1)}^{-1}$$

(10)

Here, ${{\sigma }_I}^2$ represents the scintillation index of atmospheric turbulence. The additive white Gaussian noise (AWGN) is characterized by a zero-mean Gaussian distribution with variance of ${{\sigma }_N}^2$. For $\left|n\left(i\right)-n(i-1)\right|$, the differential signal $\left|n\left(i\right)-n(i-1)\right|$ (under the null hypothesis) follows a zero-mean Gaussian distribution with variance $2{{\sigma }_N}^2$. To express $\left|{\displaystyle \frac{2m}{1+m}}I\left(i\right)+n\left(i\right)-n(i-1)\right|$ in Equation (7) more accurately, it can be rewritten as $H\left(i\right)=\left|{\displaystyle \frac{2m}{1+m}}I\left(i\right)+N\left(i\right)\right|$. Here, $n\left(i\right)-n(i-1)$ can be expressed as $N\left(i\right)$.

Therefore, $P\left(1\right|0)$ and $P\left(0\right|1)$ can be further expressed as

$$\begin{array}{c}P\left(1\right|0)={\int }_{Opt\_Th}^{\infty }{f}_N\left(i\right)di+{\int }_{-\infty }^{-Opt\_Th}{f}_N\left(i\right)di=erfc\left({\displaystyle \frac{Opt\_Th}{2{\sigma }_N}}\right)\hfill \end{array}$$

(11)

$$\begin{array}{c}P\left(0\right|1)={\int }_{Opt\_Th}^{Opt\_Th}{f}_H\left(i\right)di={\int }_{Opt\_Th}^{Opt\_Th}(f_I\left(i\right)\otimes f_N\left(i\right))di\hfill \\ \\ ={\int }_{-\infty }^{\infty }{\displaystyle \frac{1}{2}}(erfc\left({\displaystyle \frac{-Opt\_Th-i}{2{\sigma }_N}}\right)-erfc\left({\displaystyle \frac{Opt\_Th-i}{2{\sigma }_N}}\right))f_I\left(i\right)di\hfill \end{array}$$

(12)

When the number of transmitted and received signal symbols is sufficiently large, the probability of erroneous judgment can be equivalently treated as the bit error rate (BER). Therefore, the total BER (probability of erroneous judgment) P is

$$P=P\left(1\right|0\left)P\right(1)+P(0\left|1\right)P\left(0\right)$$

(13)

Here, $P\left(1\right)$ and $P\left(0\right)$ represent the probabilities of transmitting the symbol “1” and the symbol “0” in the entire signal, respectively. Since the number of transmitted symbols is sufficiently large, we can assume that $P\left(1\right)=P\left(0\right)={\displaystyle \frac{1}{2}}$. Therefore, the total bit error rate (or probability of erroneous judgment) P can be expressed as [26]

$$\begin{array}{c}P={\displaystyle \frac{1}{2}}erfc\left({\displaystyle \frac{Opt\_Th}{2{\sigma }_N}}\right)+{\int }_{-\infty }^{\infty }\begin{array}{c}{\displaystyle \frac{1}{4}}erfc\left({\displaystyle \frac{-Opt\_Th-\frac{2m}{1+m}i}{2{\sigma }_N}}\right)-{\displaystyle \frac{1}{4}}erfc\left({\displaystyle \frac{Opt\_Th-\frac{2m}{1+m}i}{2{\sigma }_N}}\right)f_I\left(i\right)di\hfill \end{array}\hfill \end{array}$$

(14)

The total bit error rate P can be regarded as a function of $Opt\_Th$ [26]. Thus, taking the partial derivative of P with respect to $Opt\_Th$, if the partial derivative function ${\displaystyle \frac{dP}{dOpt\_Th}}$ has a zero point (${\displaystyle \frac{dP}{dOpt\_Th}}=0$), this indicates that $P(Opt\_Th)$ has a critical point. Since $P(Opt\_Th)$ is a continuous function, the minimum BER ($\min \left(P\right)$) must satisfy

$$\begin{array}{c}{\displaystyle \frac{dP}{dOpt\_Th}}={\displaystyle \frac{1}{4}}(erfc\left({\displaystyle \frac{-Opt\_Th-\frac{2m}{1+m}i}{2{\sigma }_N}}\right)-{\displaystyle \frac{1}{2}}erfc\left({\displaystyle \frac{Opt\_Th-\frac{2m}{1+m}i}{2{\sigma }_N}}\right)f_I\left(i\right)-{\displaystyle \frac{1}{2}}{\displaystyle \frac{exp(-{\left(\frac{Opt\_Th}{2{\sigma }_N}\right)}^2)}{{\sigma }_N\sqrt{\pi }}}=0\hfill \end{array}$$

(15)

Here, by solving for the zero point of the partial derivative function ${\displaystyle \frac{dP}{dOpt\_Th}}$, the optimal decision threshold can be determined, thereby minimizing the bit error rate.

3. Experimental Setup and Analysis

The experimental platform setup is illustrated in Figure 2. The experiment was conducted in an outdoor environment with a transmission distance of 500 m between the transmitter and the receiver.

The system primarily consists of four components, including the transmitter, the channel, the receiver, and the PC terminal. The transmitter incorporates a power supply and an 850 nm infrared laser source, and the laser source has a transmission power of 250 mW with a beam divergence of 1 mrad. The receiver assembly includes a lens of 10 mm in diameter, an optical filter, a PIN PD with a responsivity of 0.59 mA/mW, an amplifier circuit, an analog-to-digital converter (ADC) module, and an interface circuit. The optical filter is an interference-coated optical filter with a full width at half maximum (FWHM) of 10nm. In addition, the ADC has a resolution of 16 bits.

At the transmitter, the driver provides a current to power the LD and the temperature controller controls the temperature of the laser emission. The optical beam is then transmitted into the atmospheric turbulence channel via the transmitting antenna. At the receiver, the beam is collected and focused onto the PIN by the receiving antenna. The received signal goes through an optical filter and amplifier before being converted into digital electrical signals by the ADC. It is then processed for diversity reception and stored internally. Finally, the signal is transmitted via the interface circuit to a PC-connected laptop.

At the PC terminal, a pseudorandom sequence is transmitted for offline processing and BER statistical analysis. The experimental platform established in this paper can be used to sample the fluctuation of light intensity. Subsequently, the pseudorandom binary sequence (PRBS) is transmitted from the PC side, followed by performance analysis.

This method, using a semi-experimental and semi-simulation approach, allows the flicker index of the environment in which the communication system is located to remain constant, with performance evaluated at equal SNR intervals. At the receiving terminal, the collected data represents the fluctuations of intensity in different weather conditions. In this study, a total of 200 s of intensity fluctuation data were sampled during the experiment. For the purpose of clear presentation in the paper, a two-second segment is extracted and shown. A segment of the normalized data obtained is shown in Figure 3. On 4 June 2025 and 11 June 2025, data showing the fluctuation of intensity was collected between 14:00 and 20:00 in the afternoon. The experiment was conducted in Longpao Subdistrict, Luhe District, Nanjing City, Jiangsu Province, China (Jiangbei New Area).

In addition, the electrical signals received at the receiver end in this paper can indirectly reflect the power level of the optical signals at the receiver. By measuring the magnitude of the electrical signals at the receiver end, we are able to monitor and collect data in real time, which is beneficial for subsequent analysis.

As illustrated in Figure 4, the scintillation index of 0.2259 yields distinct probability density (PD) and power spectral density (PSD) curves. The turbulence model assumed in the theoretical model of this paper is the Gamma–Gamma distribution model. The experimentally obtained turbulence data in Figure 4a exhibit conformity with the Gamma–Gamma distribution, which is adopted in our research. Figure 4b shows the power spectral density of the received power. From Figure 4b, we can observe that the low-frequency components of the turbulent channel are mainly concentrated below 1000 Hz and are relatively stable, which also indirectly reflects the stability of the turbulent channel.

The PSD estimation method used in this paper is the Welch’s averaged periodogram method. The original signal is divided into multiple segments, with each segment typically having the same length. Then, a window function (such as Hamming or Hanning) is applied to each segment to reduce spectral leakage. Fast Fourier Transform (FFT) is performed on each windowed segment to compute the periodogram of each segment. Finally, the periodograms of all segments are averaged to obtain the final PSD estimate. The advantage of this PSD estimation method lies in the introduction of data segmentation and overlap. By averaging the periodograms of multiple segments, the variance of a single periodogram is significantly reduced, resulting in a smoother PSD estimation. Moreover, it supports the selection of window functions, allowing for flexible trade-offs between spectral resolution and spectral leakage to accommodate different signal characteristics. For research purposes, this method is computationally efficient. The segmented processing reduces the length of individual data segments, thereby lowering the computational complexity of the Fourier transform, making it well-suited to long data sequences.

Figure 4b indicates that the impact of turbulence is primarily concentrated in the low-frequency region. Consequently, the turbulence effect exhibits more pronounced impacts on low-frequency signals.

Based on the experimental results in this paper, it can be observed that atmospheric turbulence contains a significant amount of low-frequency components. Meanwhile, the communication system operates at a symbol rate that is much higher than the frequency of these low-frequency components during turbulence. This difference has relatively little impact on the OOK method. However, the core principle of the DMD method is to detect signal variations based on the difference between the values of consecutive symbols. As a result, the DMD method is better able to capture the changes in atmospheric turbulence at high SNR levels.

4. System Performance and Discussion

4.1. Impact of Atmospheric Turbulence on the System Performance

At the symbol rate of 10Mbps, the BER performance variation corresponding to scintillation index variances of 0.0207 in Figure 5a, 0.2259 in Figure 5b, and 0.3730 in Figure 5c is shown. The theoretical curves in Figure 5 demonstrate that the theoretical results are in good agreement with the experimental data. As the SNR increases, both DMD as well as OOK reduce the BER of the system. In low-SNR regions, OOK exhibits greater reliability compared to DMD. But DMD exhibits superior BER performance compared to the OOK method in high-SNR regions. In high-SNR regions, DMD can surpass the BER limit of OOK and improve system performance by one to two orders of magnitude.

According to Figure 5a, we consider a scintillation index of 0.0207 to represent an extremely weak turbulence environment. Under such conditions, when the SNR is below 20 dB, the DMD method shows inferior performance compared to OOK. However, when the SNR is above 20 dB, the DMD method not only surpasses the BER floor of the OOK method but also achieves a BER advantage of two orders of magnitude over the OOK method. The core principle of the DMD method is to detect signal variations based on adjacent symbols, making it more sensitive to turbulence effects, whereas OOK cannot adapt to symbol variations. As a result, the DMD method is better able to capture the changes in atmospheric turbulence in high-SNR regions. In our research, the experiments were conducted repeatedly under the same experimental conditions. Furthermore, the performance comparison between the DMD method and the OOK method was analyzed by scanning at equal SNR intervals to evaluate the BER performance. Therefore, the experimental setup and analysis demonstrate strong rationality and scientific validity.

Although increased atmospheric turbulence intensity reduces system reliability, DMD still demonstrates significant performance advantages. From Figure 5 it can be observed that as the SNR increases, the BER curves of DMD and OOK intersect (performance advantage point). Moreover, as the scintillation index of atmospheric turbulence increases, this performance advantage point shifts toward lower SNR values. Due to this characteristic, DMD holds strong practical significance for engineering applications.

This paper assumes that the turbulence obeys the Gamma–Gamma distribution model. Therefore, the atmospheric turbulence scintillation index satisfies the following relationship [33]

$${{\sigma }_I}^2={\displaystyle \frac{〈I^2〉}{{〈I〉}^2}}-1={\displaystyle \frac{1}{\alpha }}+{\displaystyle \frac{1}{\beta }}+{\displaystyle \frac{1}{\alpha \beta }}$$

(16)

At the receiver side, the distribution of the differential detection statistic $Z=\left|r\left[k\right]-r[k-1]\right|$ can be expressed as

$$f_Z\left(z\right)=\underset{0}{\overset{\infty }{\int }}{f}_I\left(i\right)\ast {\displaystyle \frac{1}{\sqrt{4\pi {{\sigma }_n}^2}}}{e}^{-{\displaystyle \frac{{(z-i)}^2}{4{{\sigma }_n}^2}}}di$$

(17)

The expansion for BER detection using finite differences, with Hermite high-order expansion correction, is carried out up to the 20th term:

$$P\left(0\right|1)=\underset{-\infty }{\overset{+\infty }{\int }}{\displaystyle \frac{1}{2}}(erf\left({\displaystyle \frac{Th-i}{2{\sigma }_n}}\right)-erf\left({\displaystyle \frac{-Th-i}{2{\sigma }_n}}\right))f_I\left(i\right)di$$

(18)

$$P\left(0\right|1)\approx {\displaystyle \frac{1}{\sqrt{\pi }}}\sum _{t=1}^{20}{w}_t(Q\left({\displaystyle \frac{-Th-{\mu }_t}{\sqrt{2}{\sigma }_n}}\right)-Q\left({\displaystyle \frac{Th-{\mu }_t}{\sqrt{2}{\sigma }_n}}\right)))$$

(19)

Furthermore, ${\mu }_t$ can be calculated as follows:

$${\mu }_t=e^{-2{{\sigma }_x}^2+z_t\sqrt{8{{\sigma }_x}^2}}$$

(20)

The performance advantage point of the DMD and OOK method satisfies the following system of equations:

$$\left\{\begin{array}{c}{P}_{diff}=P_{\mathrm{ook}}\hfill \\ \\ {\displaystyle \frac{d}{dTh}}P\_diff={\displaystyle \frac{1}{2}}(f_z\left(Th\right)+f_z(-Th)-{\displaystyle \frac{1}{{\sigma }_n\sqrt{\pi }}}{e}^{-{\left({\displaystyle \frac{Th}{2{\sigma }_n}}\right)}^2})=0\hfill \end{array}\right.$$

(21)

For convenience in subsequent derivations, the influence of higher-order terms is neglected. High-order Taylor expansion expands $erf\left(x\right)$ and $Q\left(x\right)$ at $x=0$ up to $o\left(x^9\right)$.

$$erf\left(x\right)\approx {\displaystyle \frac{2}{\sqrt{\pi }}}(x-{\displaystyle \frac{x^3}{3}}+{\displaystyle \frac{x^5}{10}}-{\displaystyle \frac{x^7}{42}})+o\left(x^9\right)$$

(22)

$$Q\left(x\right)\approx {\displaystyle \frac{1}{2}}-{\displaystyle \frac{x}{\sqrt{2\pi }}}+{\displaystyle \frac{x^3}{6\sqrt{2\pi }}}-{\displaystyle \frac{x^5}{40\sqrt{2\pi }}}+{\displaystyle \frac{x^7}{336\sqrt{2\pi }}}+o\left(x^9\right)$$

(23)

During the derivation process in this paper, the expansion of Equations (22) and (23) was carried out to facilitate the subsequent calculation of the BER. The expansions in Equations (22) and (23) were then used to ultimately derive the relationship between the scintillation index and the shift in the performance advantage point, as expressed in Equation (26). Moreover, the chosen number of expansion terms facilitates the numerical derivation of the equations.

From our research, the theoretical and experimental curves in Figure 6 demonstrate the rationality of the derivations and also indirectly confirm that the number of expansion terms used in Equations (22) and (23) meets the required accuracy.

As a result, the SNR can be expressed comprehensively as

$$SNR={\displaystyle \frac{〈I^2〉}{2{{\sigma }_n}^2}}$$

(24)

The SNR can then be substituted into the aforementioned system of equations and expanding yields as follows:

$$\left\{\begin{array}{c}\underset{0}{\overset{\infty }{\int }}{f}_I\left(i\right)\ast \sqrt{{\displaystyle \frac{SNR}{8\pi E〈I^2〉}}}{e}^{-{\displaystyle \frac{{(Th-i)}^{2}SNR}{2E〈I^2〉}}}di+\underset{0}{\overset{\infty }{\int }}{f}_I\left(i\right)\ast \sqrt{{\displaystyle \frac{SNR}{8\pi E〈I^2〉}}}{e}^{-{\displaystyle \frac{{(Th+i)}^{2}SNR}{2E〈I^2〉}}}di\hfill \\ \\ =\sqrt{{\displaystyle \frac{SNR}{2\pi E〈I^2〉}}}{e}^{-{\displaystyle \frac{T{h}^{2}SNR}{2E〈I^2〉}}}\hfill \\ \\ erf\left(\sqrt{{\displaystyle \frac{SNR}{2E〈I^2〉}}}Th\right)-Q({\displaystyle \frac{Th-{\mu }_t}{\sqrt{2}}}\sqrt{{\displaystyle \frac{SNR}{2E〈I^2〉}}}-{\displaystyle \frac{1}{\sqrt{\pi }}}{\displaystyle \sum _{t=1}^{20}}{w}_t\left(Q\left({\displaystyle \frac{-Th-{\mu }_t}{\sqrt{2}}}\sqrt{{\displaystyle \frac{SNR}{2E〈I^2〉}}}\right)\right)))\hfill \\ \\ =1-2Q\left(\sqrt{{\displaystyle \frac{SNR}{2E〈I^2〉}}}\right)\hfill \end{array}\right.$$

(25)

By numerical methods, it is solved as

$$\begin{array}{c}SNR\approx {\displaystyle \frac{-ln\left({\sigma }^2\right)}{{\sigma }^2}}\ast (1-{\displaystyle \frac{1}{1+e^{-({\sigma }^2-0.15)/0.05}}})\hfill \\ \\ +{\displaystyle \frac{4e^{\frac{1}{4}ln(1+{\sigma }^2)}}{\frac{1}{2}ln(1+{\sigma }^2)}}(1+{\displaystyle \frac{5}{4}}{\sigma }^2+{\displaystyle \frac{5}{16}}{\sigma }^4-{\displaystyle \frac{7}{64}}{\sigma }^6)\ast {\displaystyle \frac{1}{1+e^{-({\sigma }^2-0.15)/0.05}}}+10\hfill \end{array}$$

(26)

Through our research, we obtained the theoretical curve of the intersection-point SNR versus atmospheric turbulence scintillation index, and compared it with experimentally measured curves as follows:

As shown in Figure 6, the intersection points of the BER curves between DMD and OOK vary under different conditions. With the increasing intensity of atmospheric turbulence, the BER curve intersection shifts towards low SNR, demonstrating a trend where a minimum SNR is required for DMD to surpass the BER limit in the OOK system. Figure 6 reveals good agreement between theoretical derivations and experimental data. This phenomenon arises because DMD employs a differential modulation and detection mechanism based on adjacent symbols, making it more sensitive to turbulence effects, whereas OOK cannot adapt to symbol variations. Consequently, DMD exhibits superior advantages in different turbulent environments.

4.2. Impact of Modulation Depth on System Performance

Modulation depth (MD) fundamentally reflects the control strength of the modulation signal over the carrier [31]. In the FSO system, since the minimum emission current of the laser is greater than zero at the transmitter side, the transmit power remains above zero even for bit "0", resulting in a modulation depth less than one. In the optical communication system, the influence of modulation depth cannot be ignored, especially in the application of Modulated Retro-Reflector (MRR) technology. Consequently, deviations in modulation depth degrade system performance and increase BER in communication systems.

As shown in Figure 7, the decreasing modulation depth leads to an increase in BER for both DMD and OOK in FSO systems. With higher SNR, the impact of modulation depth on OOK diminishes and eventually converges to consistent performance. For DMD, however, the influence of modulation depth becomes more pronounced as SNR increases. Nevertheless, variations in modulation depth cannot eliminate the performance superiority of DMD over OOK at high SNR levels. Moreover, larger modulation depth yields more significant performance gains for DMD. This implies that in practical engineering applications, as laser threshold currents continue to decrease and modulation depths increase, the performance advantages of the DMD approach will become even more prominent.

4.3. Impact of Symbol Rate on System Performance

In communication systems, the impact of symbol rate on system performance is also significant. Through our research, we observe that under identical conditions, the system performance of DMD improves as the symbol rate increases. At high SNR levels, its bit error rate exhibits a significant decrease. However, for the OOK detection scheme, the increase in symbol rate does not demonstrate any noticeable change in the system’s BER performance.

From the observations in Figure 8a–d, we find that as the symbol rate increases, the BER of DMD shows a significant decreasing trend. However, the impact of symbol rate variation on OOK is relatively minor, with almost no change, even at high SNR levels. As shown in Figure 8e,f, under weak turbulence conditions, when the SNR is 25 dB, the BER of DMD differs by an order of magnitude between symbol rates of 20 Kbps and 20 Mbps. Similarly, under strong turbulence conditions, when the SNR is 25 dB, the BER difference between 20 Kbps and 20 Mbps symbol rates also approaches one order of magnitude. Regardless of turbulence intensity, DMD demonstrates superior performance advantages over OOK at high SNR levels. In contrast, OOK remains insensitive to symbol rate variations, with minimal impact on FSO systems employing OOK, particularly noticeable under high-SNR conditions.

The underlying reason for this phenomenon is the fact that for DMD, the key factor significantly affecting BER is the selection of the optimal fixed detection threshold. The derived optimal threshold is highly sensitive to the difference between adjacent symbols. This means that as the symbol rate increases, the turbulence effects on consecutive symbols within shorter time intervals become smoother and more similar, thereby enhancing the temporal correlation between symbols. The stronger the temporal correlation, the more accurate the determination of the optimal threshold, resulting in lower BER.

Furthermore, according to the PSD curve in Figure 4, we observe that turbulence effects are primarily concentrated in the low-frequency region and begin to decay rapidly beyond 1 kHz. For FSO systems using OOK, where the symbol rate typically exceeds 20 Kbps, variations in symbol rate have a relatively minor impact on BER. However, since DMD makes a decision on the signal detected at the receiver based on partial channel state information under turbulence, lower symbol rates lead to insufficient channel perception by DMD, consequently resulting in less accurate threshold calculations.

5. Conclusions

In summary, the theoretical and experimental investigation of differential modulation and detection in FSO systems proposed in this paper effectively addresses the reliability issues in optical signal transmission through atmospheric turbulence channels. The results demonstrate that under relatively high-BER conditions, the differential modulation and detection method significantly enhances communication system reliability. Furthermore, through theoretical derivation and experimental analysis, we established a model for performance advantage points and validated it with experiments. To further optimize the DMD model and meet engineering requirements, we introduced the concept of modulation depth and analyzed the performance of DMD under its influence. Finally, to address the impact of signal transmission rate on BER in FSO systems, we experimentally demonstrated that increasing the symbol rate under certain conditions can effectively reduce the BER when using DMD. The results and analysis establish the foundation of the DMD method’s application in engineering work.