Research on Mitigating Atmosphere Turbulence Fading by Relay Selections in Free-Space Optical Communication Systems with Multi-Transceivers

Abstract

In free-space optical communication (FSOC) systems, atmospheric turbulence can bring about power fluctuations in receiver ends, restricting channel capacity. Relay techniques can divide a long FSOC link into several short links to mitigate the fading events caused by atmospheric turbulence. This paper proposes a Reinforcement Learning-based Relay Selection (RLRS) method based on Deep Q-Network (DQN) in a FSOC system with multiple transceivers, whose aim is to enhance the average channel capacity of the system. Malaga turbulence is studied in this paper. The presence of handover loss is also considered. The relay nodes serve in decode-and-forward (DF). Simulation results demonstrate that the RLRS algorithm outperforms the conventional greedy algorithm, which implies that the RLRS algorithm may be utilized in practical FSOC systems.

1. Introduction

Due to the advantages of high bandwidth, large capacity, and no frequency authorization, free-space optical communication (FSOC) technology is highly favored by various communication transmission systems [1,2,3]. However, in ground transmission scenarios, it suffers from atmospheric turbulence fading. The atmospheric turbulence can introduce severe events including wavefront aberration, power scintillation, and beam wander, which degrade the channel capacity of FSOC systems. To address this limitation, relay technology can be applied in FSOC systems, enhancing system performance by dividing long links into several short links [4,5,6], where short links have gentler conditions. In addition, relay technology can also eliminate the requirement for unobstructed transmission between the transmitting and receiving nodes [7,8,9].

Ref. [10] proposes a relay selection algorithm that minimizes the bit error rate (BER) of the system and analyzes the performance of various strategies under different atmospheric turbulence conditions, pointing errors, relay numbers, and distance configurations. Additionally, Ref. [11] presents a link scheduling algorithm to improve bandwidth utilization and reduce the number of idle FSOC links. Ref. [12] proposes a distributed relay selection algorithm, where each relay node transmits data only if its signal-to-noise ratio (SNR) exceeds a threshold value, thereby minimizing the system’s BER. In Ref. [13], a relay selection scheme is proposed to improve diversity gain when the buffer data of the relay node is limited. Reference [14] indicates that by optimally configuring the number and spatial positioning of relay nodes, the bit error rate (BER) can be significantly reduced, thereby enhancing communication efficiency and reliability in turbid water environments. However, as the number of relays increases, traditional model-based relay selection methods often lead to a significant increase in selection complexity and implementation delay. In recent years, with the advancement of machine learning technologies, more researchers have sought to optimize relay selection through machine learning approaches. For instance, references [15,16] applied deep learning to relay resource allocation in FSO systems, successfully overcoming the reliance on system models inherent in traditional algorithms, significantly reducing the complexity of channel capacity analysis, and maximizing capacity. This demonstrates the substantial potential of deep learning in handling unknown and complex models. Furthermore, references [17,18] proposed supervised learning-based methods for relay selection in complex network environments, offering new solutions suitable for dense networks and latency-sensitive scenarios. However, the performance of supervised learning models is highly dependent on the quality and quantity of training data; if the data quality is poor, the model’s performance may degrade significantly. Reference [19] introduced a deep reinforcement learning (DRL)-based method that does not rely on any prior knowledge of channel statistical information, allowing for optimized power allocation under unknown channel models and fading distributions. Similarly, references [20,21] also employed reinforcement learning, modeling the relay selection process as a Markov Decision Process (MDP), targeting outage probability and mutual information as objective functions, and proposed a relay selection scheme based on deep Q networks (DQNs). Ref. [22] considered the handover loss between different nodes and proposed a DQN-based relay selection algorithm to maximize the average channel capacity.

In summary, reinforcement learning maximizes cumulative rewards by optimizing long-term returns, whereas supervised learning focuses only on the optimal choice in the current time slot, without considering the impact on future slots. In the context of relay selection in this paper, the presence of switching losses means that the choice made in each time slot will affect the channel capacity in subsequent slots. Therefore, we propose a Reinforcement Learning-based Relay Selection (RLRS) algorithm based on a Dueling DQN structure, which aims to mitigate the harm on the channel capacity caused by turbulence (the turbulence is modeled as Malaga turbulence). The handover loss of switching between different relay nodes is considered. Unlike Ref. [21], which models the channel as an MDP process, meaning that the channel information at each moment is related to the previous, this paper assumes that the channels in different time slots are independent, and the relay node selected for each time slot is modeled as an MDP process. Different from Ref. [22], which considers the scenario of a single transmitting node, this paper assumes multiple transmitting nodes, which means that different transmitting nodes need to select different relay nodes, thereby increasing the complexity of the relay selection algorithm. Therefore, the main innovations of this paper are as follows:

• Based on the Dueling DQN structure, the RLRS algorithm is proposed to maximize the average channel capacity, thus mitigating the degradation caused by atmospheric turbulence;
• The average channel capacity expression of the decode-and-forward (DF) mode in an FSOC relay system with multiple transceiver nodes is considered and the handover loss is derived;
• In the implementation of the RLRS algorithm, the actions are encoded in a multi-digit format, and a reward function with a penalty term is designed based on whether the multi-digit actions are repeated.

The structure of this paper is as follows. Section 2 describes the system model and problem presentation. Section 3 details the specific process of the RLRS algorithm. In Section 4, the simulation results of the RLRS algorithm are given and compared with the conventional greedy algorithm. Conclusions are drawn in Section 5.

2. System Model and Problem Formulation

2.1. System Model

The system model of this paper is shown in Figure 1. In a multi-transceiver FSOC system with relays, there are P transmitting terminals (TX-1, …, TX-P), P receiving terminals (RX-1, …, RX-P), and N relay terminals (RLY-1, …, RLY-N). The red line represents the laser link. At any time, all the transmitting terminals will choose a different relay terminal as the relay node, which means that there are P parallel independent links in each time slot. In this paper, it is assumed that the relay terminals employ the DF method, and that all transmitting terminals and relay terminals adopt On–Off Keying (OOK) modulation in the NRZ format, chosen for its simple transceiver structure and balanced 0/1 signal characteristic, which simplify clock data recovery at the receiver. The Gaussian noise power between the transmit-relay and relay-receive terminals is ${\sigma }_r^2$ and ${\sigma }_s^2$, respectively.

2.2. Malaga Turbulence and Pointing Errors

Before analyzing the probability density function (PDF) of channel fluctuations, considering pointing errors, it is essential to select an appropriate turbulence model. The log-normal, lognormal-Rician, and gamma-gamma models are among the most prominent [23]. However, the log-normal model is limited to scenarios with weak irradiance fluctuations. Although the lognormal-Rician model aligns well with experimental data, it lacks a closed-form solution for its integral and has inherent convergence issues. Alternatively, the gamma-gamma model is proposed as a practical substitute for the lognormal-Rician model, owing to its more manageable mathematical formulation. It is worth noting that Ref. [24] introduces the Malaga turbulence (M distribution), applicable to both plane and spherical waves across all turbulence conditions, ranging from weak to extremely strong in the saturation regime. This distribution unifies the log-normal and gamma-gamma models into a single closed-form expression, encompassing various models proposed for atmospheric optical communications. Consequently, the gamma-gamma and log-normal distributions are special cases of the Malaga model under different turbulence conditions [25]. The Mdistribution PDF of irradiance is expressed as follows:

$$f_h\left(h\right)=A\sum _{k=1}^{\beta }{a}_k{h}^{{\displaystyle \frac{\alpha +k}{2}}-1}{K}_{\alpha -k}\left(2\sqrt{{\displaystyle \frac{\alpha \beta h}{\gamma \beta +{\mathsf{\Omega }}^{\prime }}}}\right)$$

(1)

$$\left\{\begin{array}{l}A\triangleq {\displaystyle \frac{2{\alpha }^{\alpha /2}}{{\gamma }^{1+\alpha /2}\Gamma (\alpha )}}{\left({\displaystyle \frac{\gamma \beta }{\gamma \beta +{\mathsf{\Omega }}^{\prime }}}\right)}^{\beta +\alpha /2}\\ a_k\triangleq \left(\begin{array}{c}\beta -1\\ k-1\end{array}\right){\displaystyle \frac{{(\gamma \beta +{\mathsf{\Omega }}^{\prime })}^{1-k/2}}{(k-1)!}}{\left({\displaystyle \frac{{\mathsf{\Omega }}^{\prime }}{\gamma }}\right)}^{k-1}{\left({\displaystyle \frac{\alpha }{\beta }}\right)}^{k/2}\end{array}\right.$$

(2)

where $\alpha ,\beta ,\gamma ,{\Omega }^{\prime }$ are the Malaga channel parameters. In Equation (1), K_ν(·) is the modified Bessel function of the second kind and order ν.

Let $h_k^{i,j}$ represent the channel gain between the i-th ($1\le i\le P$) transmitting terminal and the j-th ($1\le j\le N$) relay terminal in the k-th time slot. Similarly, let $g_k^{j,i}$ represent the channel gain between the j-th ($1\le j\le N$) relay terminal and the i-th ($1\le i\le P$) receiving terminal in the k-th time slot. In this paper, atmospheric attenuation, turbulence fluctuation, and pointing errors are considered. Then, using the Malaga model, the PDF of the channel fluctuation with pointing errors can be expressed as follows:

$$f_h(h)={\displaystyle \frac{{\xi }^{2}A}{2h}}{{\displaystyle \sum _{k=1}^{\beta }{a}_k\left(\frac{\alpha \beta }{\gamma \beta +{\mathsf{\Omega }}^{\prime }}\right)}}^{-{\displaystyle \frac{\alpha +k}{2}}}\cdot G_{1,3}^{3,0}\left[{\displaystyle \frac{\alpha \beta }{\gamma \beta +{\mathsf{\Omega }}^{\prime }}}{\displaystyle \frac{h\cdot h_k^{i,j}{g}_k^{j,i}}{h_l{A}_0}}\left|\begin{array}{l} {\xi }^2+1\hfill \\ {\xi }^2,\alpha ,k\hfill \end{array}\right.\right]$$

(3)

where $G_{1,3}^{3,0}\left[\cdot |\right]$ represents the Meijer’G function, $h_l$ is the path loss which obtains turbulence attenuation, geometric loss, and both transmission and reception losses. ${\xi }^2$ and $A_0$ represent the pointing error parameter.

2.3. Effect of Handoff Loss

Similarly to Ref. [21], $\tau$ is defined as the handoff loss during the handover of adjacent nodes. However, Ref. [21] only considers the case of a single transmitting terminal, and all relay nodes always point to the transmitting terminal and the receiving terminal, thus eliminating the problem of relay node-handover direction. In this paper, the issue of relay node-pointing switches needs to be additionally considered. As mentioned above, in the kth time slot, it is assumed that the relay node $R_{R_i}^k$ is selected for the i-th transmitting terminal to the i-th receiving terminal. Similarly, in the (k + 1)-th time slot, the relay node RX-i is selected for the i-th transmitting terminal to the i-th receiving terminal. Then, in the (k + 1)-th time slot, the handover loss ${\tau }_i^{k+1}$ from TX-i to $R_{R_i}^{k+1}$ and then to RX-i link should be calculated as follows:

$${\tau }_i^{k+1}=\max \left\{\left|R_{R_i}^{k+1}-R_{R_i}^k\right|\tau ,\left|R_{R_i}^{k+1}-R_{R_i}^{\text{'}k+1}\right|\tau \right\}$$

(4)

where $R_{R_i}^{\text{'}k+1}$ represents the direction of the $R_{R_i}^{k+1}$ relay node before the (k + 1)-th time. Therefore, considering the handover loss, the channel capacity in the k-th time slot should be expressed as follows:

$$C_k={\displaystyle {\sum }_{i=1}^P{C}_k^i}={\displaystyle {\sum }_{i=1}^P\left(1-{\tau }_i^k\right){C^{\prime }}_k^i}$$

(5)

where $C_k^i$ is the channel capacity of the link from TX-i to RX-i through the relay node in the k-th time slot under the premise of considering the handover loss. ${C^{\prime }}_k^i$ is the channel capacity of the link from TX-i to RX-i through the relay node in the k-th time slot without considering the handover loss. The relationship between the two is $C_k^i=\left(1-{\tau }_i^k\right){C^{\prime }}_k^i$. Consider that in the k-th time slot, if the relay node $R_{R_i}^k$ is selected for the link from the i-th transmitting terminal to the i-th receiving terminal, the expression of ${C^{\prime }}_k^i$ is

$${C^{\prime }}_k^i=\min \left\{{\log }_2\left(1+{\left(\eta P{h}_k^{i,{\mathsf{R}}_{R_i}^k}\right)}^2/{\sigma }_r^2\right)\right.\left.,{\log }_2\left(1+{\left(\eta P{g}_k^{i,{\mathsf{R}}_{R_i}^k}\right)}^2/{\sigma }_d^2\right)\right\}$$

(6)

The objective of this study is to maximize the average channel capacity in M time slots, namely

$$\underset{\left\{{\mathsf{\Omega }}_M\right\}}{\max }{\displaystyle \frac{1}{M}}{\displaystyle \sum _{k=0}^{M-1}{C}_k} s.t.1\le {\left\{R_{R_i}^k\right\}}_{\begin{array}{l}i=1,\dots P\\ k=0,\dots ,M-1\end{array}}\le N$$

(7)

where ${\mathsf{\Omega }}_M$ represents the set of relay nodes selected in M time slots, that is, ${\mathsf{\Omega }}_M={\left\{R_{R_i}^k\right\}}_{i=1,\dots P;k=0,\dots ,M-1}$. From Equation (7), the system needs to carefully select relay nodes in each time slot because each selection will affect the handover loss in subsequent time slots, further impacting the final channel capacity. In the actual systems, the study cannot predict the channel state of subsequent time slots, but fortunately, the study can leverage reinforcement learning techniques to optimize the relay selection in each time slot by maximizing the characteristics of long-term cumulative reward functions.

As mentioned above, the primary objective of this paper is to maximize the average channel capacity of M time slots. Therefore, the cumulative channel capacity with a discount factor starting from time k needs to be defined:

$$G\left(k\right)={\displaystyle \sum _{t=k}^{\infty }{\gamma }^{t-k}{C}_k}$$

(8)

where γ represents the discount factor, which ranges between 0 and 1. Therefore, the problem of maximizing the average channel capacity in M time slots can be transformed into the form of Equation (9)

$$\underset{\left\{{\Omega }_M\right\}}{\max }\mathbb{E}\left[G\left(k\right)\right] s.t.1\le {\left\{{ℛ}_{R_i}^k\right\}}_{\begin{array}{l}i=1,\dots P\\ k=0,\dots ,M-1\end{array}}\le N$$

(9)

3. The RLRS Algorithm

In reinforcement learning, the problem to be solved is typically modeled as an MDP. The MDP generally consists of four elements $\left(S,A,R,P\right)$: the state space $\mathcal{S}$, the action space $\mathcal{A}$, the reward function at the current time $\mathcal{R}$, and the transition probability $\mathcal{P}$. The next state is only determined by the current state and the current action and is independent of previous states. In the relay selection model of this paper, since the transition probability is unknown, it can be modeled as an incomplete MDP, consisting of three elements $\left(S,A,R\right)$. Correspondingly, $S_k$,$A_k$,$R_k$ are defined as the state, action, and immediate reward functions at the k-th time slot, respectively. In the model of this paper, the specific contents of $S_k$,$A_k$,$R_k$ are as follows:

■

State $S_k$: In any k-th time slot, state $S_k$ includes two parts: the pointing $I_k$ (including 2P + N elements) of all nodes in the (k-1)th time slot and the channel gain of the S-R link and R-D link in all current time slots (including 2PN elements).

■

Action $A_k$: In any k-th time slot, action $A_k$ represents the sequence number (including P elements) of the relay nodes selected for all transmitting nodes. There are $N!/\left(N-P\right)!$ possibilities in the action space. With $N!/\left(N-P\right)!\le N^P$, we can write any action $A_k$ into the form of a P-bit N-ary number, that is, $A_k=\left[A_k^1,A_k^2,\dots ,A_k^P\right],A_k^i\in \left[0,N-1\right]$.

■

Immediate reward function $R_k$: When action $A_k$ is performed under state $S_k$, an immediate reward function $R_k$ will be obtained. This reward function is utilized to indicate whether action $A_k$ performed in the current state is beneficial.

Since action $A_k$ is written in the form of a P-bit N-ary number, the action space is expanded from $N!/\left(N-P\right)!$ dimension to $N^P$ dimension. In this case, there will be situations where multiple transmitting nodes select the same relay. To avoid this, we define $R_k$ as follows:

$$R_k=\left\{\begin{array}{l}-1,if there is the same element in A_k\\ C_k, else\end{array}\right.$$

(10)

In reinforcement learning, the agent maximizes the cumulative reward function with a discount factor by learning the optimal strategy. In order to evaluate the performance of the strategy, the Q-value is typically introduced as a metric, defined as follows: $Q(s,a)=\mathbb{E}\left[G_t\mid s=S_t,a=A_t\right]$.When the state and action spaces are small enough, the common reinforcement learning approach is the Q-table method. By listing all actions for each state, the Q-value table of each action-–state pair is obtained, and the action with the largest Q-value is selected as the optimal strategy. However, in this paper, the state space is continuous, and the action space is also very large, making it impossible to list all Q-tables. Therefore, a DQN is adopted. In a DQN, a neural network is used to simulate the action value function, allowing it to handle each action value function in high-dimensional state space and output the maximum value function.

Inspired by Ref. [16], this paper also adopts a Dueling-DQN network structure and proposes an RLRS algorithm. However, the action space, state space, and reward function in this paper differ from those in Ref. [16]. The overall structure of this paper’s RLRS algorithm is shown in Figure 2. The RLRS algorithm includes two neural networks—one is an online neural network, the other a target neural network, both with the same network structure. The input layer, hidden layer, and output layer structures of the RLRS algorithm differ slightly from those of conventional DQNs. In the RLRS structure, the layer before the output layer is divided into two parts: the state value part $V\left(s\right)$ and the action advantage part $A\left(s,a\right)$. They are expressed in Equation (11).

$$V(s)=V(s\mid \omega ,\vartheta ), A(s,a)=A(s,a\mid \omega ,\zeta )$$

(11)

where $\omega$ represents the weight threshold of the other layers except the last layer in the online network. $\vartheta$ and $\zeta$, respectively, represent the weight threshold of the state-value part and the action-advantage-value part in the online network. Since the target network has the same structure as the online network, the corresponding parameters ${\omega }^{\prime }$, ${\zeta }^{\prime }$, and ${\vartheta }^{\prime }$ in the target network can be defined. Similarly, the parameters of the online network can be completely determined by $\omega$, $\zeta$, and $\vartheta$.

The input layer of the RLRS algorithm has 2PN + 2P + N neurons, corresponding to the dimensions of the state space, and the output layer has N elements, corresponding to the dimensions of the expanded action space. As mentioned earlier, the output of the online network represents the Q-value of the optimal state–action pair in the current strategy, which can be expressed as follows:

$$Q(s,a)=Q(s,a\mid \omega ,\zeta ,\vartheta )=V(s\mid \omega ,\vartheta )+\left(A(s,a\mid \omega ,\zeta )-{\displaystyle \frac{1}{\left|A\right|}}{\displaystyle \sum _{a^{\prime }}A}\left(s,a^{\prime }\mid \omega ,\zeta \right)\right)$$

(12)

where $|•|$ represents the number of elements in $•$. Similarly, $Q^{\prime }(s,a\mid {\omega }^{\prime },{\zeta }^{\prime },{\vartheta }^{\prime })$ represents the output of the target network when the input is $s$ and the selection action is $a$. In any k-th time slot, in the process of selecting an action $A_k$, a $\epsilon$- greedy approach is adopted, that is, the network will arbitrarily select an action in the state space with the probability of $\epsilon$. Otherwise, it will select the action with the maximum Q-value, as shown in Equation (13).

$$A_k=\left\{\begin{array}{l}{\left.\underset{A_k}{\mathrm{argmax}}Q(s,a\mid \omega ,\zeta ,\vartheta )\right|}_{s=S_t,a=A_k},1-\epsilon probability\\ A_k\in \mathcal{S},\epsilon probability\end{array}\right.$$

(13)

In the k-th time slot, based on the state $S_k$, after selecting $A_k$, the immediate reward $R_k$ of k-th time slot will be obtained, and then it enters the next state $S_{k+1}$. At this point, it yields a set of empirical values $\left(S_k,A_k,R_k,S_{k+1}\right)$, which can be put into experience pool $\mathcal{D}$. The experience pool has a first-in, first-out structure and can store $\left|\mathcal{D}\right|$ elements. If the experience pool $\mathcal{D}$ is not full, the experience value $\left(S_k,A_k,R_k,S_{k+1}\right)$ will be directly stored in the experience pool; if the experience pool is already full, the earliest stored experience will be removed, and the new experience value will be stored in the experience pool $\mathcal{D}$.

$$\mathcal{L}(\omega ,\zeta ,\vartheta )\approx {\displaystyle \frac{1}{X}}{\displaystyle \sum _l\left(r_l+\gamma {\max }_{A_{l+1}}{Q}^{\prime }\left(S_{l+1},A_{l+1}\mid {\omega }^{\prime },{\zeta }^{\prime },{\vartheta }^{\prime }\right)\right.}{\left.-Q\left(S_l,A_l\mid \omega ,\zeta ,\vartheta \right)\right)}^2$$

(14)

The online network is updated first,

$$\begin{array}{l}\omega \leftarrow \omega -\varsigma \cdot {\nabla }_{\omega }\mathcal{L}\\ \zeta \leftarrow \zeta -\varsigma \cdot {\nabla }_{\zeta }\mathcal{L}\\ \vartheta \leftarrow \vartheta -\varsigma \cdot {\nabla }_{\vartheta }\mathcal{L}\end{array}$$

(15)

where ${\nabla }_{\omega }$,${\nabla }_{\zeta }$, and ${\nabla }_{\vartheta }$ represent the gradient values of the loss function $\mathcal{L}(\omega ,\zeta ,\vartheta )$ with respect to $\omega$、$\zeta$, and $\vartheta$, respectively. $\varsigma$ represents the learning rate of the online network. After updating the online network, the target network is subsequently updated. Assuming the learning rate of the target network is $\lambda$, the network parameters are updated as follows:

$$\begin{array}{l}{\omega }^{\prime }\leftarrow \lambda \omega +(1-\lambda ){\omega }^{\prime }\\ {\zeta }^{\prime }\leftarrow \lambda \zeta +(1-\lambda ){\zeta }^{\prime }\\ {\vartheta }^{\prime }\leftarrow \lambda \vartheta +(1-\lambda ){\vartheta }^{\prime }\end{array}$$

(16)

The pseudocode diagram of the proposed RLRS algorithm is depicted in Algorithm 1.

Algorithm 1. The pseudocode diagram of the proposed RLRS algorithm.
Input: The FSOC system simulator and its parameters. Output: Optimal action $A_k$ of each time slot.
1:	Initialize experience replay memory $\mathcal{D}$.
2:	Initialize $\omega$, $\zeta$ and $\vartheta$ with random weights and initialize ${\omega }^{\prime }$, ${\zeta }^{\prime }$ and ${\vartheta }^{\prime }$.
3:	Initialize the minibatch size with X.
4:	FOR episode in $\left\{1,2,3,\cdots \right\}$ DO
5:	Observe the environment initial state $S_0$.
6:	FOR $k$= 1 to K DO
7:	Select a relay selection action $A_k$ by Equation (11), and execute action $A_k$.
8:	IF there is same element in $A_k$
9:	Calculate immediate reward $R_k=-1$.
10:	ELSE
11:	Calculate immediate reward $R_k=C_k$.
12:	Obtain next state $S_{k+1}$ and store transition data $\left(S_k,A_k,R_k,S_{k+1}\right)$ in replay memory $\mathcal{D}$.
13:	IF $\mathcal{D}$ is full
14:	Sample a random minibatch of X sets of transition data from $\mathcal{D}$.
15:	Update the online network by (13).
16:	Update the target network by (14).
17:	END FOR
18:	END FOR

4. Simulation Results

In this section, the performance simulation of the RLRS algorithm is presented, and the simulation parameters are shown in

Table 1. Simulation parameters.

Parameter Name	Value
Number of relay nodes $N$	4
Number of transmitting nodes $P$	2
Number of time slot $M$	5, 10, 50
Unit handover loss $\tau$	0.05–0.25
Responsiveness of detector $\eta$	0.9
Channel parameters $\left(\alpha ,\beta ,\rho ,{\Omega }^{\prime },A_0,\rho \right)$	5.97, 4.39, 0.596, 1, 0.0032, 6.25
Normalized power $P_t/{\sigma }_r,P_t/{\sigma }_s$	10
Discount factor $\gamma$	0.9
Online learning rate $\varsigma$	0.001

. As a comparison, a greedy algorithm is utilized as a reference to demonstrate the advantages and effectiveness of the proposed RLRS algorithm.

According to the above parameters and pseudo-code, the following simulation is obtained. Figure 2a,b shows the cumulative reward curve and the loss function curve for a time slot number of 10. Figure 2a shows the training results across different episodes. The training process can be roughly divided into three parts: the experience pool storage part, the parameter update part, and the training completion part. Before the experience pool is full, the cumulative reward curve fluctuates without a clear upward trend. Once the experience pool is full, the system extracts experiences from the pool and utilizes the mini-batch method for parameter updates. At this point, the cumulative reward curve shows an overall upward trend. After the 400th episode, the cumulative reward value tends to stabilize, and the stable cumulative reward value is superior to the performance of the greedy algorithm. The loss function curve in Figure 2b shows a periodic trend of decline, indicating the convergence of the algorithm.

Figure 3a,b shows the cumulative reward curve and the loss function curve for a time slot number of 50. Overall, the curve trends in Figure 3 are almost identical to those in Figure 2. This also demonstrates that the proposed RLRS algorithm in this paper can converge under different time slot conditions and outperforms the greedy algorithm.

Figure 4 presents the average channel capacity curves under different handover loss conditions. Figure 4a,b describes the cases of 10 time slots and 50 time slots. As can be seen from Figure 4a,b, the proposed RLRS algorithm in this paper consistently outperforms the greedy algorithm under various handover loss conditions. Additionally, as the handover loss increases, the gap between the two algorithms gradually widens, with the RLRS algorithm providing at least a 2.47% improvement in channel capacity. This is because the conventional greedy algorithm only selects the relay node with the largest channel gain without considering handover loss. As a result, when the handover loss increases, its capacity decreases linearly. In contrast, the proposed RLRS algorithm in this paper takes both handover loss and channel gain into account, so as to maximize the long-term rewards of relay selection in each time slot.

Figure 5 shows the average capacity of the RLRS algorithm and the traditional greedy algorithm when fog’s influence is considered. According to the Kruise formula, the attenuation due to fog (dB/km) is defined as follows:

$${\alpha }_{\mathrm{atm}}(V)={\displaystyle \frac{3.912}{V}}{\left({\displaystyle \frac{\lambda }{550(\mathrm{nm})}}\right)}^{-q}$$

(17)

where $\lambda$ is the wavelength (in nm), V represents the visibility (in km). q is a parameter that depends on the particle size distribution of scattering particles, given as follows:

$$q=\left\{\begin{array}{ll}1.6\hfill & V\ge 50\mathrm{km}\hfill \\ 1.3\hfill & 6\mathrm{km}<V<50\mathrm{km}\hfill \\ 0.585V^{1/3}\hfill & V<6\mathrm{km};\hfill \end{array}\right.$$

(18)

This paper selected the wavelength as 1550 nm. The visibility is set to 1 km, 2 km, 5 km, 10 km, 50 km, respectively. As shown in Figure 5a, which illustrates Malaga turbulence, the average capacity increases with visibility. This is due to the fact that greater visibility results in reduced attenuation. Figure 5b illustrates Gamma-Gamma turbulence with $\alpha =8.43$, $\beta =6.92$, where the Malaga turbulence can reduce to Gamma-Gamma turbulence with $\rho \to 1$ and ${\Omega }^{\text{'}}=1$. As can be seen from Figure 5a,b, the RLRS algorithm always performs better than the traditional greedy algorithm. Besides the situation of weak turbulence, which has a larger average capacity in both the RLRS and greedy cases, Figure 5a,b also show that the RLRS algorithm performs more effectively under strong turbulence. This is because strong turbulence can cause significant variations in channel gains across multiple potential links. Consequently, during relay selection, the difference in system performance between high-quality and low-quality relays becomes more pronounced, leading to a more noticeable improvement in system performance with the RLRS algorithm.

5. Conclusions

This paper proposes a relay selection algorithm to mitigate the capacity degradation caused by atmospheric turbulence, where there are N transmitting nodes (and N receiving nodes) and P relay nodes. We also consider the handover loss caused by the handover process of different relay nodes. The RLRS algorithm is proposed based on a Dueling DQN structure to maximize the average channel capacity. In the implementation of the RLRS algorithm, the actions are encoded in a multi-digit format, and a reward function with a penalty term is designed based on whether the multi-digit actions are repeated. Finally, through simulation comparison, the RLRS algorithm proposed in this paper is shown to be superior to the greedy comparison algorithm, which can enhance the channel capacity by at least 2.47%, with the gap increasing as handover loss rises. Moreover, under stronger turbulence, the RLRS algorithm demonstrates a noticeable improvement in channel capacity, further validating its effectiveness in combating the capacity loss induced by turbulence. Therefore, this study not only provides a reference design scheme for the relay FSOC system to further fight against turbulence, but also demonstrates the potential of reinforcement learning in improving channel capacity, which offers new insights and directions for future high-performance communication system design.