Reinforcement Learning-Based Cloud-Aware HAPS Trajectory Optimization in Soft-Switching Hybrid FSO/RF Cooperative Transmission System

Abstract

Space–air–ground systems employing free-space optical (FSO) communication leverage high-altitude platform stations (HAPS) to deliver seamless and ubiquitous connectivity. Although FSO links offer high capacity, they are highly susceptible to cloud extinction, which severely degrades link availability. Hybrid FSO/radio-frequency (RF) transmission and cloud-aware HAPS trajectory optimization can enhance resilience. However, the conventional cloud-aware hybrid FSO/RF transmission system based on hard-switching (HS) between the FSO and RF links leads to frequent link transitions and unstable throughput. To address these challenges, we propose a joint optimization framework that integrates soft-switch between FSO and RF links with deep reinforcement learning (DRL) for HAPS trajectory optimization. Soft-switching based on rateless codes (RCs) enables simultaneous transmission over both links, where the receiver accumulates packets until successful decoding with a single feedback. The feedback frequency of RC is sparse, which avoids feedback storms but also poses challenges to HAPS trajectory optimization. The DRL agent proactively optimizes HAPS trajectories to avoid cloud cover and maintain link availability. To address the sparse feedback of RCs for DRL training, a reward-shaped proximal policy optimization (PPO)-based agent is developed to jointly optimize throughput and trajectory smoothness. Simulations using realistic ERA5 data show that RC-PPO achieves higher throughput and smoother trajectories compared to the HS-PPO baseline.

1. Introduction

The explosive growth of global data traffic has revealed the limitations of terrestrial networks in achieving ubiquitous connectivity [1]. Space–air–ground integrated networks (SAGINs) with free-space optical communication (FSO), using high-altitude platform stations (HAPS) to relay communications between satellites and ground stations, have emerged as a promising solution [2,3]. FSO links offer ultra-high bandwidth but suffer from low reliability due to atmospheric conditions such as clouds and atmospheric turbulence (AT) [4,5]. Hybrid FSO/RF transmission systems leverage RF backup to enhance reliability [6,7,8]. Such hybrid systems have been deployed in dual-hop relay links [9] and SAGIN scenarios to improve coverage and capacity [10]. Conventional hybrid FSO/RF systems employ hard-switching (HS) strategies that select links based on instantaneous SNR values [11], suffering from frequent transitions and unstable throughput. FSO/RF systems based on rateless code (RC), such as Raptor code [12], enable soft-switching by transmitting distinct encoded packets simultaneously over both the FSO and RF channels [13], thus eliminating switching overhead and improving reliability against burst disruptions. To avoid cloud cover, HAPS trajectory optimization is also an effective method to enhance link stability and throughput.

Deep reinforcement learning (DRL) offers a powerful framework for adaptive HAPS trajectory optimization in dynamic atmospheric environments [14]. For example, ref. [15] integrates 5G low-latency connectivity with a Deep Q-Network (DQN) for 3D trajectory optimization. Ref. [16] employs the deep deterministic policy gradient (DDPG) algorithm for coordinating HAPS coverage. However, these studies foucus on RF-only aerial communication networks. When extending DRL to hybrid FSO/RF systems, ref. [17] applies the A2C algorithm to optimize the HAPS trajectory in the HS FSO/RF transmission system, which relies on dense, per-step reward signals. This dense-reward strategy is fundamentally incompatible with the RC-based soft-switching FSO/RF system. For RC, a meaningful reward is available only upon successful block decoding, which introduces a critical challenge of reward sparsity. This sparse feedback (reward) severely complicates temporal credit assignment and hinders efficient exploration [18]. Ref. [19] successfully used Proximal Policy Optimization (PPO) with reward shaping to compensate for such reward sparsity when guiding UAVs to move to destinations. To the best of our knowledge, no existing work has investigated HAPS trajectory planning in an RC-based soft-switching hybrid FSO/RF system under stochastic cloud dynamics.

This paper proposes a joint optimization framework that integrates rateless-coded physical-layer transmission with PPO-based trajectory learning in a hybrid FSO/RF transmission system. Key contributions are as follows:

• The cloud-aware HAPS trajectory optimization problem in soft-switching hybrid FSO/RF systems is formulated and solved by a PPO-based DRL approach, under the stochastic moving occluding cloud (SMOC) model derived from the ERA5 dataset.
• A potential-based reward-shaping mechanism within the PPO framework is developed to mitigate sparse decoding feedback of RCs, delivering faster convergence and superior performance over threshold-based HS-PPO schemes.

The remainder of this paper is organized as follows. Section 2 presents the scheme of the trajectory-optimized HAPS–ground station (GS) hybrid FSO/RF link, FSO/RF channel models, and the throughput of HS and RC-based hybrid FSO/RF systems. Section 3 details the proposed RC-PPO algorithm and the HS-PPO algorithm as a reference and presents the corresponding Markov Decision Process (MDP) formulation and reward shaping strategy. Section 4 provides simulation results. Section 5 gives the conclusion.

2. System Model

This section presents the mathematical framework for the HAPS-assisted hybrid FSO/RF communication system. Firstly, a simplified three-tier SAGIN architecture is introduced. Subsequently, the channel model of FSO and RF links are presented. At last, the throughputs for hybrid FSO/RF systems with HS and RC are elaborated.

2.1. Space–Air–Ground Architecture

As illustrated in Figure 1, we consider a simplified three-tier Space–Air–Ground (SAG) system, which constitutes a fundamental building block of standard SAGINs. The LEO satellite operates at an orbital altitude of approximately 600 km and serves as the primary data source. The HAPS, positioned at $H_{haps}=20$ km in the stratosphere, functions as an aerial relay node to bridge the satellite-to-ground communication gap. The satellite-to-HAPS link employs FSO communication to exploit its high bandwidth capacity, while the HAPS-to-GS link utilizes hybrid FSO/RF transmission to provide complementary link diversity.

A discrete-time model is adopted with time step index $t\in \{1,2,\dots ,T\}$. The position of HAPS at time t is denoted as $p_h\left[t\right]=(x\left[t\right],y\left[t\right],H_{haps})$, where $\left(x\right[t],y[t\left]\right)$ represents the horizontal coordinates and $H_{haps}$ (the altitude of HAPS) remains constant. The GS is located at $p_g=(0,0,H_{gs})$ with $H_{gs}=0$. The instantaneous slant distance between HAPS and GS is given by:

$$L\left[t\right]=\parallel p_h\left[t\right]-p_g\parallel =\sqrt{x{\left[t\right]}^2+y{\left[t\right]}^2+H_{haps}^2}.$$

(1)

And the elevation angle (Figure 1) is $\theta \left[t\right]=\arctan (H_{haps}/\sqrt{x{\left[t\right]}^2+y{\left[t\right]}^2})$.

2.2. Channel Model

2.2.1. FSO Channel Model

The composite channel gain of the downlink FSO channel between HAPS and GS is expressed as:

$$H_{\mathrm{FSO}}\left[t\right]=h_g\left[t\right]·h_a\left[t\right]·h_l\left[t\right],$$

(2)

where $h_g\left[t\right]$ is the geometric loss due to the divergence of the beam on the slant path distance $L\left[t\right]$ [20], $h_a\left[t\right]$ follows the Gamma-Gamma distribution characterizing AT-induced fading [21], and $h_l\left[t\right]$ represents the loss of cloud-induced attenuation. Among these factors, $h_l\left[t\right]$ is critical for optimizing the HAPS trajectory due to its spatial heterogeneity. The cloud-induced attenuation follows the Beer–Lambert law [22]:

$$h_l\left[t\right]=\exp (-\sigma L\left[t\right]).$$

(3)

$\sigma$ (km⁻¹) is the attenuation coefficient [23], determined based on the visibility (V) (in km) and the wavelength (in nm) as

$$\sigma ={\displaystyle \frac{3.91}{V}}{\left({\displaystyle \frac{\lambda }{550}}\right)}^{-q}.$$

(4)

Here, $\lambda$ is the optical wavelength, V is the visibility, and q is the size-distribution parameter of the scattering particles depending on V. The expression of V is given by:

$$V=-{\displaystyle \frac{\ln \left(0.002\right)}{{\beta }_f\left[t\right]}},$$

(5)

where ${\beta }^f\left[t\right]$ is the cloud-induced attenuation of the FSO link obtained by integration over the slant path:

$${\beta }^f\left[t\right]={\displaystyle \frac{1}{\sin \left(\theta \right[t\left]\right)}}{\int }_{h_0}^{h_{\mathrm{top}}}{\beta }_{\mathrm{ext}}^f\left(h\right)\mathrm{d}h.$$

(6)

Here, $\theta \left[t\right]$ is the elevation angle and $h_0$ and $h_{\mathrm{top}}$ are the cloud base and top altitudes. The extinction coefficient ${\beta }_{\mathrm{ext}}^f\left(h\right)$ for the FSO link is given by [24]:

$${\beta }_{\mathrm{ext}}^f\left(h\right)={\displaystyle \frac{6.51\times {10}^3·c\left(h\right)}{{\rho }_w·r_e}},$$

(7)

where ${\rho }_w=1$ g/cm³ is the water density, $r_e$ ($\mathsf{\mu }$m) is the particle effective radius [24], and $c\left(h\right)$ is the vertical profile of liquid water content (LWC) in g/m³. The three-dimensional (3D) distribution of LWC is modeled as a spatially correlated stochastic field, as detailed in Section 4.1.

The FSO link employs intensity modulation with direct detection (IM/DD) using on–off keying (OOK). The received optical power is $P_{\mathrm{rx}}^f\left[t\right]=P_{\mathrm{tx}}^f{G}_{\mathrm{tx}}^f{G}_{\mathrm{rx}}^f{H}_{\mathrm{FSO}}\left[t\right]$, where $P_{\mathrm{tx}}^f$ denotes the transmit optical power and $G_{\mathrm{tx}}^f$ and $G_{\mathrm{rx}}^f$, respectively, denote the transmitter and receiver telescope gains. The instantaneous electrical SNR is given by:

$${\gamma }_{\mathrm{FSO}}\left[t\right]={\displaystyle \frac{{\left(R{P}_{\mathrm{rx}}^f\left[t\right]\right)}^2}{N_0^f{B}_f}},$$

(8)

where R (A/W) is the photodetector responsivity, $N_0^f$ (A²/Hz) is the noise power spectral density, and $B_f$ (Hz) is the receiver bandwidth of the FSO link.

2.2.2. RF Channel Model

The RF channel provides complementary connectivity with enhanced reliability under cloud-obscured conditions. The RF channel gain incorporates free-space path loss (FSPL) and cloud-induced attenuation. The expression of FSPL is given by:

$${\mathrm{FSPL}}_r\left[t\right]={\left({\displaystyle \frac{4\pi L\left[t\right]f_c}{c_0}}\right)}^2,$$

(9)

with RF carrier frequency $f_c$ and speed of light $c_0$. The cloud attenuation coefficient ${\beta }^r\left[t\right]$ (km⁻¹) is computed similarly to (6) for FSO links by path-integrating the specific attenuation coefficient. The specific RF attenuation coefficient ${\beta }_{\mathrm{ext}}^r\left(h\right)$ (km⁻¹) is derived from Mie scattering theory and modeled as [25]:

$${\beta }_{\mathrm{ext}}^r\left(h\right)=k_{\mathrm{ext}}^r·c\left(h\right),$$

(10)

where $k_{\mathrm{ext}}^r$ (km⁻¹/(g·m⁻³)) is the mass extinction coefficient at RF frequency $f_c$.

Similar to (8), the instantaneous SNR of RF link with QPSK modulation is given by:

$${\gamma }_{\mathrm{RF}}\left[t\right]={\displaystyle \frac{P_{\mathrm{tx}}^r{G}_{\mathrm{tx}}^r{G}_{\mathrm{rx}}^r{H}_{\mathrm{RF}}\left[t\right]}{N_0^r{B}_r}},$$

(11)

where $P_{\mathrm{tx}}^r$ is the transmitting power, $G_{\mathrm{tx}}^r$ and $G_{\mathrm{rx}}^r$ are the transmitting and receiving antenna gains, $N_0^r$ (W/Hz) is the spectral density of RF noise power, and $B_r$ (Hz) is the receiver bandwidth of the RF link.

2.3. Hybrid FSO/RF Systems

This section describes the throughput of hybrid FSO/RF communication systems based on HS and RC. These two schemes differ fundamentally in how they exploit link diversity, which affects trajectory optimization.

2.3.1. Hard Switching

The HS scheme selects either the FSO or RF link at each time step based on instantaneous channel quality. The instantaneous throughput is given by:

$$T_{\mathrm{HS}}\left[t\right]=\left\{\begin{array}{cc}{R}_{\mathrm{FSO}}{P}_{\mathrm{suc}}^{\mathrm{FSO}}\left[t\right],\hfill & \mathrm{if}{\gamma }_{\mathrm{FSO}}\left[t\right]\ge {\gamma }_{\mathrm{th}},\hfill \\ R_{\mathrm{RF}}{P}_{\mathrm{suc}}^{\mathrm{RF}}\left[t\right],\hfill & \mathrm{if}{\gamma }_{\mathrm{FSO}}\left[t\right]<{\gamma }_{\mathrm{th}},\hfill \end{array}\right.$$

(12)

where ${\gamma }_{\mathrm{th}}$ is the switching threshold, $R_{\mathrm{FSO}}$ and $R_{\mathrm{RF}}$ are the transmitted data rates, $P_{\mathrm{suc}}^{\mathrm{FSO}}\left[t\right]$ and $P_{\mathrm{suc}}^{\mathrm{RF}}\left[t\right]$ are the per-packet success reception probabilities accounting for forward error correction (FEC). The optimal threshold value maximizes the expected throughput:

$${\gamma }_{\mathrm{th}}^{★}=\arg \underset{{\gamma }_{\mathrm{th}}}{max}\mathbb{E}\left[T_{\mathrm{HS}}\left[t\right]\right].$$

(13)

For the hybrid FSO/RF system, the bit error rate (BER) for the FSO link with OOK modulation and the RF link with QPSK modulation depend on their respective signal-to-noise ratios (SNRs) at the receiver. The BER for each link can be uniformly expressed as:

$${\mathrm{BER}}_i\left[t\right]=\frac{1}{2}\mathrm{erfc}\left(\sqrt{{\kappa }_i·{\gamma }_i\left[t\right]}\right),$$

(14)

where $i\in \{f,r\}$ denotes the FSO or RF link, ${\gamma }_i\left[t\right]$ is the instantaneous SNR, and ${\kappa }_i$ is the modulation-dependent efficiency factor, with ${\kappa }_f=0.5$ for OOK and ${\kappa }_r=1$ for QPSK. For a packet of $L_p$ bits in length, assuming FEC capable of correcting up to $t_{\mathrm{corr}}$ bit errors, the packet success probability is computed via the binomial sum:

$$P_{\mathrm{suc}}\left[t\right]=\sum _{i=0}^{t_{\mathrm{corr}}}\left({\displaystyle \genfrac{}{}{0pt}{}{L_p}{i}}\right){\left(\mathrm{BER}\left[t\right]\right)}^i{\left(1-\mathrm{BER}\left[t\right]\right)}^{L_p-i}.$$

(15)

2.3.2. Rateless Coding

RCs enable the generation of an arbitrary number of encoded packets until the receiver accumulates sufficient packets to decode the original source block [13]. Unlike traditional channel codes with fixed code rates, RCs automatically adjust the number of encoded packets required to be sent for successful decoding based on channel conditions. Raptor code, which represents the state of the art in RCs, consists of a high-code-rate precode (typically Low-Density Parity-Check (LDPC) code [26]) followed by a Luby Transform (LT) [27] outer code and is utilized in this paper.

Let $s\in {\mathrm{GF}}_2^{N_s}$ denote the source symbol vector containing $N_s$ information symbols. The precode generates an intermediate symbol vector (Figure 1):

$$c={\mathbf{G}}_{\mathrm{pre}}s,$$

(16)

where ${\mathbf{G}}_{\mathrm{pre}}$ is the precode generator matrix producing $N_c$ intermediate symbols. The LT encoder then produces output symbols via:

$$x_i=\underset{j=1}{\overset{N_c}{⨁}}{\mathbf{G}}_{\mathrm{LT}}(i,j)c_j,$$

(17)

where ${\mathbf{G}}_{\mathrm{LT}}(i,·)$ is a random sparse row sampled according to a robust soliton degree distribution and ⊕ denotes XOR over ${\mathrm{GF}}_2$. Each output symbol $x_i$ is formed by XORing a small subset of intermediate symbols, with the subset size and selection determined by the degree distribution.

In the RC scheme, both FSO and RF links transmit different encoded symbols simultaneously. Crucially, any correctly received symbol contributes to decoding, regardless of which link delivered it. This superposition property eliminates the need for link selection decisions and enables full utilization of both links under all channel conditions. The receiver can successfully decode the source data block once it accumulates approximately $N_s(1+{\epsilon }_c)$ distinct encoded symbols, where ${\epsilon }_c$ is the average overhead required by the belief-propagation (BP) decoder. This overhead is typically ${\epsilon }_c\approx 0.05$–$0.1$ for well-designed Raptor codes [13]. The impact of dynamic SNR fluctuations on the hybrid link is captured through the packet success probabilities $P_{suc}^{FSO}$ and $P_{suc}^{RF}$ in Equation (15), which determine the rate of correctly received symbols. Therefore, the expected effective throughput for RC is given by:

$$\mathbb{E}\left[T_{\mathrm{RC}}\right]={\displaystyle \frac{R_{\mathrm{FSO}}{P}_{\mathrm{suc}}^{\mathrm{FSO}}+R_{\mathrm{RF}}{P}_{\mathrm{suc}}^{\mathrm{RF}}}{1+{\epsilon }_c}}.$$

(18)

3. Trajectory Optimization

This section presents the trajectory optimization framework for hybrid FSO/RF communication schemes. Firstly, the PPO algorithm and MDP formulation are introduced as the foundation of DRL. Subsequently, the trajectory optimization strategies for the HS and RC schemes using PPO-based DRL are elaborated.

3.1. Proximal Policy Optimization

PPO is an on-policy actor-critic algorithm that uses a clipped surrogate objective to constrain policy updates, thereby preventing large deviations from the current policy and ensuring stable training. The objective of PPO, which is maximized at each iteration, is given by:

$${\mathcal{L}}_t^{\mathrm{PPO}}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[L_t^{\mathrm{CLIP}}\left(\theta \right)-c_1{L}_t^{\mathrm{VF}}\left(\theta \right)+c_2\mathcal{H}\left[{\pi }_{\theta }\right]\left(s_t\right)\right],$$

(19)

where ${\widehat{\mathbb{E}}}_t[·]$ denotes the empirical average over a finite batch of samples, $\mathcal{H}\left[{\pi }_{\theta }\right]\left(s_t\right)$ denotes an entropy bonus, $L_t^{\mathrm{VF}}\left(\theta \right)$ is the value function loss, and $c_1,c_2>0$ are weighting coefficients.

The clipped surrogate objective $L_t^{\mathrm{CLIP}}\left(\theta \right)$ prevents excessively large policy updates by constraining the probability ratio between the current and old policies, given by:

$$L_t^{\mathrm{CLIP}}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min\left({\mu }_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left({\mu }_t\left(\theta \right),1-{ϵ}_{\mathrm{clip}},1+{ϵ}_{\mathrm{clip}}\right){\widehat{A}}_t\right)\right],$$

(20)

where the function clip $\left({\mu }_t\left(\theta \right),1-{\epsilon }_{\mathrm{clip}},1+{\epsilon }_{\mathrm{clip}}\right)$ acts as a clipping operation that constrains the probability ratio ${\mu }_t\left(\theta \right)$ to the interval $[1-{\epsilon }_{\mathrm{clip}},1+{\epsilon }_{\mathrm{clip}}]$. ${ϵ}_{\mathrm{clip}}$ (typically 0.1–0.3) is a hyperparameter. The probability ratio ${\mu }_t\left(\theta \right)$ at time step t is given by:

$${\mu }_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t\mid s_t)}}.$$

(21)

Moreover, the advantage estimator (${\widehat{A}}_t$) is formulated as a discounted sum of future temporal-difference (TD) residuals (${\delta }_t$), given by:

$${\widehat{A}}_t=\sum _{l=0}^{T-t}{\gamma }^l{\delta }_{t+l},$$

(22)

where $\gamma \in (0,1)$ is the discount factor, and the TD residual is defined as:

$${\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right).$$

(23)

Here, $r_t$ is the immediate reward received from the environment, and $V\left(s_t\right)$ is the state value, which can be used to evaluate the state $s_t$ under the current policy.

The value function loss ($L_t^{\mathrm{VF}}\left(\theta \right)$) minimizes the mean squared error (MSE) between predicted state values and empirical returns, given by:

$$L_t^{\mathrm{VF}}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[{\left(V_{\theta }\left(s_t\right)-R_t\right)}^2\right],$$

(24)

where $R_t={\sum }_{l=0}^{T-t}{\gamma }^l{r}_{t+l}$ is the discounted sum of future rewards. The entropy term ($\mathcal{H}\left[{\pi }_{\theta }\right]$) encourages exploration by preventing premature convergence to deterministic policies, given by

$$\mathcal{H}\left[{\pi }_{\theta }\right]\left(s_t\right)=-{\mathbb{E}}_{a\sim {\pi }_{\theta }(·|s_t)}\left[\log {\pi }_{\theta }\left(a\right|s_t)\right].$$

(25)

Network parameters are updated after collecting a trajectory of length T, using the collected batch to perform multiple epochs of minibatch gradient descent on the PPO objective. The pseudocode of the algorithm is shown in Algorithm 1.

Algorithm 1 PPO for HAPS Trajectory Optimization
Require: Policy and value network ${\pi }_{\theta },V_{\theta }$, hyperparameters $\gamma ,{ϵ}_{\mathrm{clip}},c_1,c_2,K$
1:	for each training iteration do
2:	Collect trajectory $\mathcal{D}={\left\{(s_t,a_t,r_t,s_{t+1})\right\}}_{t=0}^T$ using ${\pi }_{\theta }$
3:	Compute returns $R_t={\sum }_{l=0}^{T-t}{\gamma }^l{r}_{t+l}$
4:	Compute TD residuals ${\delta }_t=r_t+\gamma V_{\theta }\left(s_{t+1}\right)-V_{\theta }\left(s_t\right)$
5:	Compute advantages ${\widehat{A}}_t={\sum }_{l=0}^{T-t}{\gamma }^l{\delta }_{t+l}$
6:	Normalize advantages: ${\widehat{A}}_t\leftarrow ({\widehat{A}}_t-\mu )/\sigma$
7:	Store old policy probabilities ${\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)$
8:	for epoch $k=1,\dots ,K$ do
9:	for mini-batch $\mathcal{B}\subset \mathcal{D}$ do
10:	Compute ${\mathcal{L}}_t^{\mathrm{PPO}}\left(\theta \right)=L_t^{\mathrm{CLIP}}\left(\theta \right)-c_1{L}_t^{\mathrm{VF}}\left(\theta \right)+c_2\mathcal{H}\left[{\pi }_{\theta }\right]\left(s_t\right)$
11:	Update $\theta$ via gradient ascent on ${\mathcal{L}}_t^{\mathrm{PPO}}\left(\theta \right)$
12:	end for
13:	end for
14:	end for
15:	return Optimized policy ${\pi }_{{\theta }^{*}}$

3.2. Trajectory Optimization with PPO-Based DRL

The HAPS trajectory optimization problem is formalized as a finite-horizon MDP defined by the tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma )$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, P is the transition probability, and R is the reward function. The control objective for both RC and HS schemes is to find an optimal policy maximizing the expected discounted return:

$${\pi }^{*}=arg\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{s_0\sim \mu ,a_t\sim {\pi }_{\theta }(·|s_t),s_{t+1}\sim P(·|s_t,a_t)}\left[\sum _{t=0}^T{\gamma }^t{r}_t\right].$$

(26)

The state space, action space, and reward function differ significantly between the RC and HS schemes due to their distinct communication paradigms.

3.2.1. RC-PPO

The state space should contain all information needed for the RC-PPO agent to make decisions and preserve the Markovian property of the environment, given by:

$$s_t^{\mathrm{RC}}=\left(x_t,y_t,{\psi }_t,{\tau }_t,{\mathit{\rho }}_t\right)\in {\mathcal{S}}^{\mathrm{RC}},$$

(27)

where $(x_t,y_t)\in {\mathbb{R}}^2$ denote the current coordinates of the HAPS, ${\psi }_t\in [0,2\pi )$ is the heading angle measured clockwise from north, ${\tau }_t=T-t$ is the number of steps remaining, and ${\rho }_t=[\rho \left({\mathbf{p}}_{t,1}\right),\dots ,\rho \left({\mathbf{p}}_{t,N_p}\right)]\in {[0,1]}^{N_p}$ contains cloud extinction coefficients sampled at $N_p$ probe points $\left\{{\mathbf{p}}_{t,i}\right\}$ along prospective directions.

Since RC allows both links to transmit simultaneously without switching overhead, the action space contains only navigation decisions:

$$a_t^{\mathrm{RC}}=\Delta {\psi }_t\in {\mathcal{A}}^{\mathrm{RC}},$$

(28)

where $\Delta {\psi }_t\in [-{\psi }_{max},+{\psi }_{max}]$ is the discrete heading change with ${\psi }_{max}={30}^{\circ }$ representing the maximum single-step turn angle.

State transitions decompose into deterministic kinematic evolution and stochastic cloud field dynamics. For simplicity, the speed of the HAPS is assumed constant of $v_0$, and the effects of wind (atmospheric currents) are not considered in the kinematic model. The HAPS position evolves according to:

$$\begin{array}{cc}\hfill x_{t+1}& =x_t+v_0\Delta t\cos ({\psi }_t+\Delta {\psi }_t),\hfill \\ \hfill y_{t+1}& =y_t+v_0\Delta t\sin ({\psi }_t+\Delta {\psi }_t),\hfill \\ \hfill {\psi }_{t+1}& =({\psi }_t+\Delta {\psi }_t)\mathrm{mod}2\pi .\hfill \end{array}$$

(29)

The decoding progress ${\eta }_t\in [0,1]$, defined as the fraction of successfully received packets to the total required, evolves according to:

$${\eta }_{t+1}=min\left(1,{\eta }_t+{\displaystyle \frac{{\Lambda }_t^{\mathrm{RC}}\Delta t}{N_s(1+{\epsilon }_c)}}\right),$$

(30)

where $N_s$ is the total number of source packets, ${\epsilon }_c$ is the redundancy overhead in the RCs, and ${\Lambda }_t^{\mathrm{RC}}$ is the aggregate packet-arrival rate from both links. Noted, in real-world deployment, the receiver provides feedback only upon successful decoding of complete data blocks, making ${\eta }_t$ unavailable for real-time decision-making by the HAPS. The aggregate rate:

$${\Lambda }_t^{\mathrm{RC}}=\sum _{i=1}^{N_p}{w}_i\left(R_{\mathrm{FSO}}·P_{suc}^{\mathrm{FSO}}\left(\rho \left({\mathbf{p}}_{t,i}\right)\right)+R_{\mathrm{RF}}·P_{suc}^{\mathrm{RF}}\left(\rho \left({\mathbf{p}}_{t,i}\right)\right)\right),$$

(31)

where $R_{\mathrm{FSO}}$ and $R_{\mathrm{RF}}$ are the transmitted symbol rates of the FSO and RF links, $P_{suc}^{\mathrm{FSO}}\left(\rho \right)$ and $P_{suc}^{\mathrm{RF}}\left(\rho \right)$ denote the probability of successful symbol reception as a function of extinction loss of cloud, and $w_i=\exp (-d_i/d_0)$ are distance-dependent weight factors where $d_i$ is the distance from HAPS to the ith probe position and $d_0$ is a normalization constant, thereby assigning higher influence to nearer probes. At each time step, $N_p$ probe points ${\left\{{\mathbf{p}}_{t,i}\right\}}_{i=1}^{N_p}$ are placed along candidate directions at varying distances from the current HAPS position to estimate future channel conditions.

In order to maximize the average capacity over the allowed T time steps, the reward function balances multiple objectives and is defined as:

$$r_t^{\mathrm{RC}}=\alpha {\Lambda }_t^{\mathrm{RC}}+\beta \Delta {\eta }_t-{\kappa }_{\psi }|\Delta {\psi }_t|-{\kappa }_d{\left({\displaystyle \frac{d_t}{d_0}}\right)}^2+q{R}_{\mathrm{suc}},$$

(32)

where ${\Lambda }_t^{\mathrm{RC}}$ encourages high instantaneous throughput, $\Delta {\eta }_t={\eta }_{t+1}-{\eta }_t$ provides dense feedback on decoding progress through potential-based reward shaping, $|\Delta {\psi }_t|$ discourages sharp heading changes to maintain flight stability, and ${(d_t/d_0)}^2$ with $d_t=\sqrt{x_t^2+y_t^2}$ penalizes excessive distance from the GS to accelerate policy convergence speed and improve link quality. $\alpha$, $\beta$, ${\kappa }_{\psi }$, ${\kappa }_d$, and q are weighting factors to balance the physical and task-driven scales of their corresponding terms.

The terminal reward $R_{\mathrm{suc}}$ indicates mission completion:

$$R_{\mathrm{suc}}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\eta }_t=1\mathrm{and}t\le T,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(33)

An episode terminates when either the mission succeeds ($R_{\mathrm{suc}}=1$) or timeout occurs (${\tau }_t=0$).

The inclusion of $\Delta {\eta }_t$ in the reward function addresses the sparse feedback challenge inherent in RC. According to the potential-based shaping theorem [19], our shaping term $\beta \Delta {\eta }_t$ corresponds to the potential function $\Phi \left(s\right)=\beta {\eta }_t$, which preserves the optimal policy while providing dense training signals.

Importantly, although ${\eta }_t$ is used during training to accelerate policy learning, the trained policy ${\pi }_{\theta }(a_t^{\mathrm{RC}}\mid s_t^{\mathrm{RC}})$ depends only on the observable state $s_t^{\mathrm{RC}}=(x_t,y_t,{\psi }_t,{\tau }_t,{\mathit{\rho }}_t)$ and requires no real-time decoding feedback at deployment.

3.2.2. HS-PPO

For HS-PPO, the state space omits decoding progress but includes the currently selected link represented by the binary indicator $H_t$:

$$s_t^{\mathrm{HS}}=(x_t,y_t,{\psi }_t,{\tau }_t,H_t,{\mathit{\rho }}_t)\in {\mathcal{S}}^{\mathrm{HS}},$$

(34)

where $H_t\in \{\mathrm{FSO},\mathrm{RF}\}$ denotes the currently active link. The action space includes both navigation and link selection decisions:

$$a_t^{\mathrm{HS}}=(\Delta {\psi }_t,h_t)\in {\mathcal{A}}^{\mathrm{HS}},$$

(35)

where $h_t\in \{0,1\}$ denotes the currently active link (0 for FSO, 1 for RF). The instantaneous reward is defined as:

$$r_t^{\mathrm{HS}}=\alpha {\Lambda }_t^{\mathrm{HS}}-{\kappa }_{\psi }|\Delta {\psi }_t|-\xi \mathbb{I}[H_t\ne H_{t-1}]-{\kappa }_d{\left({\displaystyle \frac{d_t}{d_0}}\right)}^2+q{R}_{\mathrm{suc}},$$

(36)

where $\xi >0$ penalizes frequent link switch, and ${\Lambda }_t^{\mathrm{HS}}$ is the data rate from the active link:

$${\Lambda }_t^{\mathrm{HS}}=\left\{\begin{array}{cc}{R}_{\mathrm{FSO}}·P_{suc}^{\mathrm{FSO}}\left(\rho \left({\mathbf{p}}_t\right)\right),\hfill & \mathrm{if}{H}_t=\mathrm{FSO},\hfill \\ R_{\mathrm{RF}}·P_{suc}^{\mathrm{RF}}\left(\rho \left({\mathbf{p}}_t\right)\right),\hfill & \mathrm{if}{H}_t=\mathrm{RF}.\hfill \end{array}\right.$$

(37)

The terminal reward $R_{\mathrm{suc}}$ for HS-PPO is defined similar to that for RC-PPO, with episodes terminating upon mission completion or timeout.

4. Simulation and Results

This section presents the cloud field generation, training setup, and evaluation of the proposed HS-PPO and RC-PPO agents. Key parameters are summarized in

Table 1. Simulation parameters.

Parameter	Value
HAPS altitude ($H_{haps}$)	20 km
Clouds base altitude ($h_0$)	1 km
Clouds max altitude ($h_{max}$)	10 km
Receiver aperture diameter (D)	1 m
Responsivity of PD (R)	0.8
Variance of background noise (${\sigma }^f$)	250 $\mathsf{\mu }$W
Noise power spectral density (${\sigma }^r$)	−100 dB/MHz
Optical transmit power ($P_t^f$)	1 W
RF transmit power ($P_t^r$)	1 W
Telescope gain of transmitter, receiver ($G_{tx}^f,G_{rx}^f$)	70 dB
Antenna gain of transmitter, receiver ($G_{tx}^r,G_{rx}^r$)	50 dB
Optical wavelength (${\lambda }^f$)	1550 nm
RF frequency ($f^r$)	30 GHz
Optical bandwidth ($B_f$)	10 GHz
RF bandwidth ($B_r$)	500 MHz

.

4.1. Cloud Field Generation

The 3D LWC field is generated using the SMOC model [28], leveraging statistics of cloud cover and average integrated liquid water content (ILWC) from the ERA5. A log-normal distribution of the content of liquid cloud water (CLWC) over the given site is obtained based on the extracted information from the dataset. Spatially correlated 3D LWC fields are then synthesized by generating random Gaussian fields based on empirical spatial correlations from ERA5, ensuring realistic cloud structures. Finally, an analytical vertical profile $c\left(h\right)$ is applied to modulate the altitude-dependent LWC distribution within each cloud column:

$$c\left(h\right)=\left\{\begin{array}{cc}{\displaystyle \frac{W}{b_c^{a_c}\Gamma \left(a_c\right)}}{(h-h_0)}^{a_c-1}{e}^{-(h-h_0)/b_c},\hfill & h\ge h_0\hfill \\ 0,\hfill & h<h_0\hfill \end{array}\right.$$

(38)

where W is the columnar ILWC (in kg/m²), $h_0$ is the cloud base altitude, $\Gamma (·)$ denotes the gamma function, and $a_c$, $b_c$ determine the shape of the clouds’ vertical profile, given by:

$$\begin{array}{cc}\hfill a_c& =4.27\exp [-4.93(C+0.06\left)\right]+54.12\exp [-61.25(C+0.06\left)\right]+1.71,\hfill \\ \hfill b_c& =3.17a_c^{-3.04}+0.074,\hfill \end{array}$$

(39)

where C is the CLWC. The resulting 3D LWC field is used to compute the specific extinction coefficients of the cloud for the FSO and RF channels according to (7) and (10), respectively.

4.2. PPO Training Configuration

Both RC-PPO and HS-PPO agents employ identical neural network architectures, consisting of a three-layer MLP feature extractor with hidden dimensions [128, 64, 32] and ReLU activations. The feature extractor feeds into separate policy and value heads. The policy head was initialized using orthogonal initialization with a gain of 0.01 to reduce initial entropy. Each cycle collects 200 episodes to form a batch. This batch is reused for 10 update epochs with a mini-batch size of 64. Evaluation is performed with 200 deterministic test episodes employing the static cloud fields generated in Section 4.1. PPO training hyperparameters are detailed in

Table 2. PPO training hyperparameters.

Parameters	Values
Feature layers	[128, 64, 32] (ReLU)
Policy head init gain	0.01
Learning rate ($\alpha$)	$3\times {10}^{-4}$
Discount ($\gamma$)	0.99
Clip ratio (${ϵ}_{clip}$)	0.2
Entropy coeff./Value coeff.	0.01/0.5
Gradient clip	0.5
Mini-batch size	64
Update epochs per batch (K)	10
Total environment steps	${10}^6$

.

Moreover, for the weighting parameter settings of the reward function (Equations (32) and (35)), an iterative behavior-driven tuning methodology is employed. Specifically, for the RC-PPO agent, $\alpha$ is fixed as the baseline to reflect the primary objective of maximizing throughput; $\beta$ is selected via grid search to balance the decoding progress signal and throughput exploration; ${\kappa }_{\psi }$ and ${\kappa }_d$ serve as regularization terms to encourage smooth trajectories and movement toward the ground station, respectively. q is set to a large scalar to clearly signify task completion. The specific parameter values are summarized in

Table 3. Final tuned reward weights and their tuning rationale.

Weight	Value	Role and Tuning Rationale
Throughput ($\alpha$)	1	Fixed to establish reward scale, directly reflecting the objective of maximizing data rate.
Decoding Progress ($\beta$)	5	Provides dense gradients for decoding progress. Value chosen via grid search.
Heading Penalty (${\kappa }_{\psi }$)	0.35	Penalizes abrupt heading changes. Initially small; increased if trajectories exhibited excessive jitter.
Distance Penalty (${\kappa }_d$)	0.1	Encourages the agent to maintain proximity to the GS, accelerating learning convergence. Initially small; increased if the agent strayed too far.
Terminal Reward (q)	100	Task completion signal. Large scalar reward that reinforces successful mission completion.

.

4.3. Results and Discussion

Figure 2 shows representative HAPS trajectories and their corresponding instantaneous throughput achieved by the HS-PPO and RC-PPO agents under varied cloud conditions. To systematically assess the performance of both schemes, 100 random cloud distributions were first generated using the SMOC model. For each cloud distribution, the ILWC along the slant path was averaged over the starting point to the GS (indicated by the red dashed line in Figure 2). The mean values of ILWC were then ranked across 100 cloud distribution cases to approximately characterize the difficulty of trajectory optimization. For each typical HAPS mission duration (approximately 0.8–1.3 h for a 100 km flight at 75–120 km/h [29]), the cloud field can be regarded as slowly varying (with a temporal decorrelation scale of 15–29 h [30]). Hence, the use of a static cloud map in each episode is justified.

Representative cloud scenarios were selected at the 25th, 50th, and 75th among 100 cloud distributions of this ranked distribution, corresponding to light, moderate, and heavy cloud-density distributions. The horizontal position of the starting point is uniformly fixed at (25 km, 25 km), and the position of the GS is set at (100 km, 100 km).

The overall objective of HAPS trajectory optimization is to approach the GS while actively avoiding dense cloud regions, thereby minimizing signal attenuation and maximizing both throughput and communication coverage of the HAPS. Under light cloud-density conditions (Figure 2a), both HS-PPO and RC-PPO agents tend to advance directly toward the GS, achieving favorable throughput performance due to minimal cloud-induced extinction. As cloud density increases (Figure 2b,c), HS-PPO exhibits substantial throughput degradation attributed to severe FSO link attenuation in dense cloud regions, necessitating RF channel backup. In contrast, RC-PPO achieves higher effective hybrid throughput by dynamically leveraging transient FSO transmission windows while maintaining a reliable RF backup link. This strategy enables RC-PPO to reduce signal interruption durations and feedback latency compared to HS-PPO.

In terms of quantified throughput, under light cloud, the average throughput of RC-PPO is 9.34 Gbps, which is improved by 12.8% compared with the throughput of 8.28 Gbps for HS-PPO. As cloud density increases to a moderate level, RC-PPO reaches 9.68 Gbps, representing a significant 89.4% improvement compared to the throughput (5.11 Gbps) of HS-PPO. Under heavy cloud, the throughput of RC-PPO maintains 7.06 Gbps, yielding a 51.8% improvement over the 4.65 Gbps of HS-PPO. The results demonstrate that the RC-PPO system exhibits substantial throughput improvement under light, moderate, and heavy cloud conditions. RC-PPO demonstrates the most pronounced throughput advantage over HS-PPO under moderate cloud-density conditions. In addition, under heavy cloud-density conditions (Figure 2c), a non-RL scheme is also evaluated, which follows a straight-line path from the starting point directly to the GS without any adaptive cloud avoidance. Results show that RC-PPO achieves the highest average throughput, outperforming both the non-RL RC and HS-PPO schemes. It confirms that the performance gain stems not only from the use of RCs but from the predictive trajectory optimization enabled by DRL.

Figure 3 illustrates the training performance of the RC-PPO and HS-PPO agents. Although a direct quantitative comparison of their absolute return values is complicated for the differences in reward design, the resulting HAPS trajectories suggest that both schemes achieve effective policy learning. Notably, despite the inherently sparser feedback of the RCs (informative feedback is available only upon successful decoding of entire data blocks), the RC-PPO agent exhibits slightly more stable convergence performance. This improved stability is attributed to a shaped-reward mechanism specifically designed to provide incremental feedback on packet decoding progress. By incorporating denser reward signals, RC-PPO effectively mitigates variance in policy gradient estimates, thereby improving sample efficiency throughout training.

5. Conclusions

This paper presents an integrated framework that combines RCs with a PPO-based DRL algorithm for HAPS trajectory optimization in a hybrid FSO/RF SAG system. By embedding potential-based reward shaping around decoding progress, the proposed RC-PPO method addresses key limitations of conventional HS schemes, enabling more reliable exploitation of both FSO and RF channels under stochastic cloud obstruction. Simulations demonstrate that RC-PPO improves achievable capacity and transmission reliability. Under moderate cloud scenarios, the average throughput of the RC-PPO agent is 9.68 Gbps, achieving 89.4% improvement compared to that of HS-PPO (5.11 Gbps). Moreover, RC-PPO exhibits more stable training convergence compared to HS-PPO.

The proposed framework in this paper can be further extended to multi-HAPS/multi-GS systems by adopting multi-agent reinforcement learning (MARL) strategies, such as Multi-Agent PPO (MA-PPO), where multiple HAPSs coordinate their trajectories and transmission policies to optimize coverage and load balancing.