Joint Traffic Prediction and Handover Design for LEO Satellite Networks with LSTM and Attention-Enhanced Rainbow DQN

Abstract

With the increasing scale of low Earth orbit (LEO) satellite networks, leveraging non−terrestrial networks (NTNs) to complement terrestrial networks (TNs) has become a critical issue. In this paper, we investigate the issue of handover satellite selection between multiple terrestrial terminal groups (TTGs). To support effective handover decision-making, we propose a long short-term memory (LSTM)-network-based traffic prediction mechanism based on historical traffic data. Building on these predictions, we formulate the handover strategy as a Markov Decision Process (MDP) and propose an attention-enhanced rainbow-DQN-based joint traffic prediction and handover design framework (ARTHF) by jointly considering the satellite switching frequency, communication quality, and satellite load. Simulation results demonstrate that our approach significantly outperforms existing methods in terms of the handover efficiency, service quality, and load balancing across satellites.

1. Introduction

Non−terrestrial network (NTN) communication is pivotal for overcoming the coverage limitations of terrestrial networks (TNs), ensuring reliable and seamless connectivity under challenging conditions and enabling global communication in sixth−generation (6G) wireless systems [1,2]. Low Earth orbit (LEO) satellites, with their extensive coverage, reduced latency, and cost−effective deployment [3], have emerged as a cornerstone for addressing the constraints of traditional infrastructure, facilitating efficient global connectivity [4]. However, the rapid movement of LEO satellites results in limited coverage time for terrestrial terminal groups (TTGs), presenting challenges in providing continuous service. Thus, designing effective handover policies is crucial to maintain high−quality service in dynamic satellite networks.

In recent years, numerous studies have investigated handover strategies. In [5], the authors proposed a strategy based on the distance between satellites and TTGs, selecting the nearest satellite for handover. By comprehensively evaluating the signal−to− interference−plus−noise ratio (SINR), bandwidth utilization, transmission delay, packet loss rate, and monetary cost, the authors of [6] proposed a service−adaptive multi−criteria vertical handover algorithm that reduces the dropping probability and enhances the overall system throughput. Furthermore, the authors of [7] proposed a multi−metric handover method that uses the Floyd algorithm to identify the shortest path in a dynamic graph, effectively reducing the total handover delay and avoiding unnecessary handovers. Nevertheless, these methods primarily considered network conditions at the moment of decision−making.

As an emerging deep learning technique, deep reinforcement learning (DRL) facilitates the maximization of cumulative rewards through repeated trial−and−error interactions, making it especially suitable for long−term optimization problems. In [8], the authors introduced a satellite handover strategy based on multi−agent Q−learning, aiming to minimize the average satellite handovers while satisfying the load constraints of each satellite. In [9], a centralized−training−and−distributed−execution scheme based on deep Q−networks (DQNs) was proposed, where environmental parameters were trained by a training node in the backhaul network and are distributed to TTGs for independent handover decisions. Additionally, the authors of [10] addressed the freshness of information and proposed an age−oriented satellite handover strategy based on dueling−double−deep−Q−learning (D3QN) to minimize the long−term peak age of information. However, these approaches failed to accurately model the communication traffic demands of TTGs, which could result in a large number of TTGs simultaneously accessing a previously underloaded satellite, causing network load imbalances and degrading quality of service (QoS).

Traffic prediction utilizes historical traffic data and advanced algorithms to forecast communication demands, establishing a robust foundation for subsequent decision −making [11]. Early studies relied on simple learning methods, such as linear regression [12] and support vector regression (SVR) [13], for traffic prediction. With the advent of neural networks, their superior generalization ability and predictive accuracy have made deep learning the dominant choice for modern traffic prediction methods. The authors of [14] addressed wireless traffic forecasting with an attention−enhanced deep residual convolutional neural network (CNN). On the other hand, the authors of [15] reformulated traffic prediction as a classification task, using CNNs to assign traffic in fixed time intervals to predefined categories, and thus, improve the accuracy. As a step further, to capture temporal dependencies, the authors of [16] employed gated recurrent units (GRUs) to achieve precise traffic forecasts. Nevertheless, the aforementioned methods often fail to adequately capture long−range temporal dependencies in complex, highly variable traffic patterns, particularly in satellite communication systems characterized by fluctuating user activity, recurring usage patterns, and periodic application demands [17].

In this paper, we address the critical challenge of generating effective handover policies for TTGs in a satellite−to−ground communication system, where TTGs remain stationary while LEO satellites continuously orbit the Earth, necessitating robust solutions for seamless connectivity and load balancing in NTNs. We propose the Attention−Enhanced Rainbow DQN−based Joint Traffic Prediction and Handover Design Framework (ARTHF), which combines advanced traffic forecasting with intelligent handover decision−making to enhance the performance of LEO satellite networks. The main contributions in this paper are summarized as follows:

• We propose a traffic prediction module based on long short−term memory (LSTM) networks, which accurately forecasts TTG traffic demands by leveraging historical traffic data, thereby enabling proactive and reliable handover decisions.
• We model the handover decision problem as a Markov Decision Process (MDP) and solve it using an attention−enhanced rainbow DQN, which optimizes handover policies by jointly considering satellite switching frequency, communication quality, and load distribution.
• We validate the proposed framework through extensive simulations using a realistic LEO satellite constellation model. The results show that the proposed ARTHF effectively reduces handover frequency, enhances service quality, and achieves a fair load distribution among the satellites.

2. System Model and Problem Formulation

2.1. System Model

We investigate the issue of handover selection for numerous TTGs in satellite−to−ground communication, as illustrated in Figure 1. As depicted, the system comprises K static TTGs on the ground, each requiring continuous communication services. Above them, a constellation of LEO satellites continuously orbits. For each TTG, there is always one ‘serving satellite’ providing active communication, while several other satellites are visible and available for a potential handover, forming a dynamic set of choices.

All satellites are assumed to operate at the same altitude, denoted as $h_s$, and maintain a constant speed $v_s$. Each TTG is simultaneously covered by multiple satellites at any given time. However, due to the relative motion between the satellites and the TTGs, the visibility time of each satellite to a specific TTG is limited. When a satellite can no longer cover TTG k due to its movement, the satellite network broadcasts system information to configure a set of visible satellites for the TTG, enabling a seamless handover and sustained communication. For TTG k, the number of visible satellites at time slot t is denoted as $M_k^t$, with each satellite indexed by $m\in {\mathcal{M}}_k^t=\{1,2,\dots ,M_k^t\}$. The service status for TTG k at time slot t is represented by the vector ${\mathbf{s}}_k^t=(s_{k,1}^t,s_{k,2}^t,\dots ,s_{k,M_k^t}^t)$, where $s_{k,m}^t$ is a binary variable indicating whether satellite m is serving TTG k at time slot t, defined as

$$s_{k,m}^t=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{TTG}k\mathrm{is}\mathrm{currently}\mathrm{served}\mathrm{by}\mathrm{satellite}m\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(1)

Note that different TTGs may have distinct resource requirements, such as throughput and latency. To effectively model these varying demands, we introduce a normalized resource demand weight $w_k^t$ for each TTG, ranging from 0 to 1. The sum of the weights of all TTGs accessing a particular satellite indicates the satellite’s load state. We assume each TTG periodically retrieves load information for its visible satellites from the System Information Block (SIB) broadcast. The load of the $M_k^t$ visible satellites for TTG k is represented as

$${\mathbf{I}}^t=\{l_1^t,l_2^t,\dots ,l_{M_k^t}^t\},$$

(2)

where $l_m^t$ denotes the sum of the normalized demand weights of all the TTGs accessing satellite m.

We assume that TTG k accurately determines its location via Global Navigation Satellite System (GNSS) services, denoted as ${\mathbf{p}}_k=[{\overline{x}}_k,{\overline{y}}_k,0]\in {\mathbb{R}}_{+}^3$, where ${\overline{x}}_k$ and ${\overline{y}}_k$ denote its fixed horizontal coordinates. Additionally, TTG k can obtain the position and velocity of its visible satellites at time slot t from the broadcast ephemeris [18], denoted as ${\mathbf{q}}_m^t=[x_m^t,y_m^t,h_s]\in {\mathbb{R}}_{+}^3$, where $m\in {\mathcal{M}}_k^t$, and $x_m^t$ and $y_m^t$ are the satellite’s time−varying horizontal coordinates. Using this information, TTG k can calculate the distance to satellite m, expressed as

$$d_{k,m}^t=\sqrt{{(x_m^t-{\overline{x}}_k)}^2+{(y_m^t-{\overline{y}}_k)}^2+h_s^2}.$$

(3)

With this distance, the elevation angle ${\theta }_{k,m}^t$, defined as the angle between the horizontal plane and the line connecting satellite m and TTG k, can be calculated using

$${\theta }_{k,m}^t=arcsin\left({\displaystyle \frac{h_s}{d_{k,m}^t}}\right).$$

(4)

Subsequently, satellite m’s remaining cover time for TTG k can be derived from a geometric relationship denoted as

$$T_{k,m}^t={\displaystyle \frac{R_E+h_s}{v_s}}\left(arcsin\left({\displaystyle \frac{R_E}{R_E+h_s}}cos{\theta }_{\min }\right)-arcsin\left({\displaystyle \frac{R_E}{R_E+h_s}}cos{\theta }_{k,m}^t\right)\right),$$

(5)

where $R_E$ denotes the radius of the Earth, and ${\theta }_{min}$ represents the minimum visible elevation angle.

2.2. Communication Model

The channel quality between LEO satellites and TTGs varies rapidly due to their relative movement, making it a crucial factor in determining handover policies. Given the significant distance between them, which leads to relatively slow spatial signal variations, we primarily focus on large−scale fading (LSF). According to document [19] proposed by the International Telecommunication Union Radio Communications Sector (ITU−R), the LSF in satellite−to−ground communication mainly comprises four parts, i.e., free−space path loss, rain attenuation, ionospheric attenuation, and shadowing, expressed as

$${\mathrm{LSF}}_{k,m}^t\left[\mathrm{dB}\right]={\mathrm{PL}}_{k,m}^{t,fre}+{\mathrm{PL}}_{k,m}^{t,rai}+{\mathrm{PL}}_{k,m}^{t,ion}+{\gamma }_{k,m}^{t,sha},$$

(6)

where ${\mathrm{PL}}_{k,m}^{t,fre}$, ${\mathrm{PL}}_{k,m}^{t,rai}$, ${\mathrm{PL}}_{k,m}^{t,ion}$, and ${\gamma }^{sha}$ represent the free−space path loss, rain attenuation, ionospheric attenuation, and shadowing from satellite m to TTG k at time slot t, respectively.

The free−space path loss is calculated by

$${\mathrm{PL}}_{k,m}^{t,fre}\left[\mathrm{dB}\right]=32.45+20{log}_{10}\left(f_c\right)+20{log}_{10}\left(d_{k,m}^t\right),$$

(7)

where $f_c$ denotes the carrier frequency.

The rain attenuation, accounting for the rainfall rate, elevation angle, frequency, and polarization, is computed based on the ITU−R P.837 [20] using

$${\mathrm{PL}}_{k,m}^{t,rai}\left[\mathrm{dB}\right]={\gamma }_R^t·d_{k,m}^{t,eff},$$

(8)

where ${\gamma }_R^t$ represents the rain absorption coefficient, calculated by

$${\gamma }_R^t=k·R_{\alpha }^t,$$

(9)

where k and $\alpha$ are coefficients dependent on the frequency and polarization, respectively, and $R_{\alpha }^t$ represents the polarization−adjusted rainfall rate. Additionally, $d_{k,m}^{t,eff}$ represents the effective path length of the signal through the rain region, expressed as

$$d_{k,m}^{t,eff}={\displaystyle \frac{d_R^t}{cos{\theta }_{k,m}^t}},$$

(10)

where $d_R^t$ denotes the projected length of the rain region along the signal propagation direction, calculated by

$$d_R^t={\displaystyle \frac{h_0}{sin{\theta }_{k,m}^t}},$$

(11)

where $h_0$ represents the height of the $0 °\mathrm{C}$ isotherm.

Ionospheric attenuation primarily arises from two components: absorption loss and scintillation loss, expressed as

$${\mathrm{PL}}_{k,m}^{t,ion}\left[\mathrm{dB}\right]={\mathrm{PL}}_{k,m}^{t,abs}+{\mathrm{PL}}_{k,m}^{t,sci},$$

(12)

where ${\mathrm{PL}}_{k,m}^{t,abs}$ and ${\mathrm{PL}}_{k,m}^{t,sci}$ represent the ionospheric absorption loss and ionospheric scintillation loss, respectively, from satellite m to TTG k at time slot t. The absorption loss primarily results from the interaction between the signal and electrons in the ionospheric D and E layers, expressed as

$${\mathrm{PL}}_{k,m}^{t,abs}={\int }_{h_{start}}^{h_{end}}\kappa \left(h\right)·N_e\left(h\right)dh,m\in {\mathcal{M}}_k^t,$$

(13)

where $h_{\mathrm{start}}$ and $h_{\mathrm{end}}$ represent the starting and ending heights of the ionospheric D and E layers, respectively, while $N_e\left(h\right)$ and $\kappa \left(h\right)$ denote the height−dependent electron density and ionospheric absorption coefficient, respectively. Ionospheric scintillation loss arises from the irregular structure of the electron density in the ionospheric F layer. According to ITU−RP.618 [19], its probability density function can be expressed as

$$p\left({\mathrm{PL}}_{k,m}^{t,sci}\right)={\displaystyle \frac{1}{\sqrt{2\pi }·{\sigma }_{sci}}}·exp\left(-{\displaystyle \frac{{({\mathrm{PL}}_{k,m}^{t,sci}-{\mu }_{sci})}^2}{2·{\sigma }_{sci}^2}}\right),m\in {\mathcal{M}}_k^t,$$

(14)

where ${\mu }_{sci}$ and ${\sigma }_{sci}$ represent the mean and standard deviation of the scintillation attenuation, respectively.

As for the shadowing, we model it as a log−normal distribution with a mean of 0 and a variance of ${\alpha }_{k,m}^{t,sha}$, which is related to the satellite elevation angle ${\theta }_{k,m}^t$ and can be found in TR38.811 [21].

Therefore, the expected power of the signal received by TTG k from satellite m is denoted as

$$P_k^t=P_m^t·{10}^{-{\displaystyle \frac{{\mathrm{LSF}}_{k,m}^t}{10}}},$$

(15)

where $P_k^t$ and $P_m^t$ represent the received and transmitted signal power at TTG k and satellite m, respectively. Consequently, the signal−to−noise ratio (SNR) at the receiver of TTG k is

$${\mathrm{SNR}}_{k,m}^t\left[\mathrm{dB}\right]=10{log}_{10}\left({\displaystyle \frac{P_k^t}{N_0}}\right),$$

(16)

where $N_0$ denotes the power of additive white Gaussian noise (AWGN).

2.3. Problem Formulation

In this paper, we focus on selecting optimal handover satellites for TTGs. Since each handover of TTG k to satellite m introduces a signaling overhead and increases the risk of transmission interruptions, we propose a cost function to quantify the handover cost defined as

$${\Lambda }_k^t=\left\{\begin{array}{cc}-\mu ,\hfill & \mathrm{if}{\mathbf{s}}_k^t\ne {\mathbf{s}}_k^{t-1},\hfill \\ 0,\hfill & \mathrm{if}{\mathbf{s}}_k^t={\mathbf{s}}_k^{t-1},\hfill \end{array}\right.$$

(17)

where $\mu$ is a positive constant.

Additionally, to enhance the QoS for TTGs and encourage connections to satellites with a better link quality, we consider the SNR between TTGs and satellites. Furthermore, to achieve load balancing between satellites while comprehensively considering the current and future traffic demands of terminals, we define a load cost as

$${\Psi }_{k,m}^t=-\left(I_m^t+(1-w_p)·d_{k,cur}^t+w_p·d_{k,pre}^t\right),$$

(18)

where $d_{k,cur}^t$ represents the current load of the TTG, $d_{k,pre}^t$ denotes the predicted load of the TTG, and $w_p\in [0,1]$ is the prediction weight.

By integrating these factors, the problem of handover selection can be formulated as

$$\begin{array}{cc}\hfill & \underset{\mathcal{A}}{max}\sum _{t=0}^T\sum _{k=1}^K\left({\Lambda }_k^t+{\beta }_p·{\mathrm{SNR}}_{k,m}^t+{\gamma }_p·{\Psi }_{k,m}^t\right),\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill \\ \hfill & \mathrm{C}1:s_{k,m}^t\in \{0,1\},\forall k\in \mathcal{K},\forall m\in {\mathcal{M}}_k^t,\hfill \\ \hfill & \mathrm{C}2:a_k^t\in \{1,\dots ,M_k^t\},\forall k\in \mathcal{K},\forall t\in \mathcal{T},\hfill \\ \hfill & \mathrm{C}3:{\theta }_{k,m}^t\ge {\theta }_{min},\hfill \end{array}$$

(19)

where $\mathcal{A}=\{a_1^1,a_2^1,\dots ,a_K^1,\dots ,a_1^T,a_2^T,\dots a_K^T\}$ denotes the set of handover strategies across all TTGs and slots, and ${\beta }_p$ and ${\gamma }_p$ are weighting coefficients that balance the contributions of the SNR and load cost, respectively.

3. Proposed Joint Traffic Prediction and Handover Design Framework

We propose a unified framework that combines traffic prediction and a handover policy design for LEO satellite networks. As shown in Figure 2, the workflow begins with traffic forecasting using an LSTM network, followed by satellite load computation based on the predicted TTG traffic. This information, along with the satellite status, forms the MDP state space, which is then used by an attention−enhanced rainbow DQN to derive optimal handover policies.

3.1. LSTM−Enabled Traffic Prediction

LSTM networks, with their ability to model sequential data and capture long−term dependencies, are well−suited for predicting TTG traffic demand, enabling optimized satellite communication performance. To leverage LSTM for traffic prediction, we model the historical traffic data of each TTG as a time series with dimensions $[T_l,F_l]$, where $T_l$ denotes the number of historical time slots used for prediction, and $F_l$ represents the number of channels for each TTG’s traffic data. As illustrated in Figure 3, data from each TTG is processed independently by a two−layer stacked LSTM architecture. Within this architecture, the first LSTM layer sequentially processes the input feature vector ${\mathbf{x}}^t\in {\mathbb{R}}^{F_l}$ for each time slot t, leveraging its previous hidden state ${\mathbf{h}}_{\left(1\right)}^{t-1}$ to generate current hidden states ${\mathbf{h}}_{\left(1\right)}^t$. These outputs are then fed into the second LSTM layer, which further refines them to produce ${\mathbf{h}}_{\left(2\right)}^t$. Once all inputs have been processed by the stacked LSTM layers, the final hidden states ${\mathbf{h}}_{\left(2\right)}^t$ from the second LSTM layer are stacked and passed through a series of fully connected layers with ReLU activation to generate traffic predictions $d_{k,pre}^t$ for each TTG.

Each LSTM layer employs forget, input, and output gates to regulate the information flow, maintaining a memory cell that selectively retains or discards information to capture temporal dependencies. For each time slot t in the first layer, the LSTM processes the input feature vector ${\mathbf{x}}^t\in {\mathbb{R}}^{F_l}$ and the previous hidden state ${\mathbf{h}}_{\left(1\right)}^{t-1}$. First, the forget gate discards irrelevant information from the previous cell state ${\mathbf{c}}_{\left(1\right)}^{t-1}$, expressed as

$${\mathbf{f}}^t=\sigma \left({\mathbf{W}}_{f,\left(1\right)}·\left[{\mathbf{h}}_{\left(1\right)}^{t-1},{\mathbf{x}}^t\right]+{\mathbf{b}}_{f,\left(1\right)}\right),$$

(20)

where ${\mathbf{W}}_{f,\left(1\right)}$ is the forget gate’s weight matrix, ${\mathbf{b}}_{f,\left(1\right)}$ is its bias vector, and $\sigma$ denotes the sigmoid function. The input gate then decides what new information to store, which is calculated by

$${\mathbf{i}}^t=\sigma \left({\mathbf{W}}_{i,\left(1\right)}·\left[{\mathbf{h}}_{\left(1\right)}^{t-1},{\mathbf{x}}^t\right]+{\mathbf{b}}_{i,\left(1\right)}\right),$$

(21)

where ${\mathbf{W}}_{i,\left(1\right)}$ is the input gate’s weight matrix, and ${\mathbf{b}}_{i,\left(1\right)}$ is the bias vector. Concurrently, a candidate cell state is computed using

$${\stackrel{\sim }{\mathbf{c}}}^t=tanh\left({\mathbf{W}}_{c,\left(1\right)}·\left[{\mathbf{h}}_{\left(1\right)}^{t-1},{\mathbf{x}}^t\right]+{\mathbf{b}}_{c,\left(1\right)}\right),$$

(22)

where ${\mathbf{W}}_{c,\left(1\right)}$ is the candidate state’s weight matrix, and ${\mathbf{b}}_{c,\left(1\right)}$ is its bias vector. Then, the cell state is updated using

$${\mathbf{c}}_{\left(1\right)}^t={\mathbf{f}}^t\odot {\mathbf{c}}_{\left(1\right)}^{t-1}+{\mathbf{i}}^t\odot {\stackrel{\sim }{\mathbf{c}}}^t,$$

(23)

combining retained and new information. Finally, the output gate produces the hidden state

$${\mathbf{h}}_{\left(1\right)}^t={\mathbf{o}}^t\odot tanh\left({\mathbf{c}}_{\left(1\right)}^t\right),$$

(24)

where ${\mathbf{o}}^t=\sigma ({\mathbf{W}}_{o,\left(1\right)}·[{\mathbf{h}}_{\left(1\right)}^{t-1},{\mathbf{x}}^t]+{\mathbf{b}}_{o,\left(1\right)})$, with ${\mathbf{W}}_{o,\left(1\right)}$ as the output gate’s weight matrix and ${\mathbf{b}}_{o,\left(1\right)}$ as its bias vector. The second LSTM layer applies identical gated operations with its own weight matrices $({\mathbf{W}}_{f,\left(2\right)},{\mathbf{W}}_{i,\left(2\right)},{\mathbf{W}}_{c,\left(2\right)},{\mathbf{W}}_{o,\left(2\right)})$ and bias vectors $({\mathbf{b}}_{f,\left(2\right)},{\mathbf{b}}_{i,\left(2\right)},{\mathbf{b}}_{c,\left(2\right)},{\mathbf{b}}_{o,\left(2\right)})$ to produce its hidden state ${\mathbf{h}}_{\left(2\right)}^t$.

3.2. MDP Framework for Satellite Handover

Based on the system model established in Section 2, we formulate the problem as an MDP, where each TTG is considered an agent that makes decisions based on the state information and selects optimal actions. An MDP is typically defined as a tuple $(\mathcal{S},\mathcal{A},\mathcal{R},\gamma )$ that describes the agent’s sequential decision−making process. The following discussion provides a detailed explanation of these components:

• State space $\mathcal{S}$: The state space for each TTG is constructed based on the current link status and the load conditions derived from the predicted traffic. For TTG k, each state $s^t\in \mathcal{S}$ is represented as a matrix $\mathbf{S}\in {\mathbb{R}}^{M_k^t\times 7}$, where each row corresponds to one of the $M_k^t$ visible satellites. The seven columns capture the satellite’s service status $s_{k,m}^t$, elevation angle ${\theta }_{k,m}^t$, distance $d_{k,m}^t$, remaining service time $T_{k,m}^t$, SNR ${\mathrm{SNR}}_{k,m}^t$, normalized demand weight $l_m^t$, and computed load information ${\Psi }_{k,m}^t$. Notably, the load information is obtained by combining the predicted traffic demand from Section 3.1 with the real−time satellite state through Equation (18), enabling each TTG to make decisions that are both adaptive and anticipatory.
• Action space $\mathcal{A}$: The action space defines the handover decisions available to the TTG in each time slot. Each action $a^t\in \mathcal{A}$ specifies the satellite selected by the TTG for handover.
• Reward function $\mathcal{R}$: The reward function quantifies the immediate reward given by the environment after the TTGs take action $a^t$ in state $s^t$. This reward comprehensively considers the handover cost, the coverage quality of the TTG, and the load balancing between satellites, which is expressed as

  $$r^t=\sum _{k=1}^K\left({\Lambda }_k^t+\beta ·{\mathrm{SNR}}_{k,m}^t+\gamma ·{\Psi }_{k,m}^t\right).$$

  (25)
• Discount factor $\gamma$: This determines the weight of future rewards relative to immediate rewards, with $\gamma \in [0,1]$. A higher $\gamma$ value places greater emphasis on long−term rewards.

3.3. Attention−Enhanced Rainbow DQN

To address the satellite handover problem formulated as an MDP, we propose the attention−enhanced rainbow DQN, a reinforcement learning framework that integrates a self−attention mechanism with the dueling architecture of a rainbow DQN, which dynamically selects the optimal satellite for handover from a variable set of visible satellites.

The model input is obtained by zero−padding $\mathbf{S}\in {\mathbb{R}}^{M_k^t\times 7}$ to form ${\mathbf{S}}_e\in {\mathbb{R}}^{M_k^{max}\times 7}$, where $M_k^{max}$ represents the maximum number of visible satellites for TTG k. As shown in Figure 4, the proposed attention−enhanced rainbow DQN architecture integrates a self−attention mechanism with a dueling network structure to process satellite data and estimate Q−values. The input ${\mathbf{S}}_e$ is passed through a series of layers, including a self−attention module, fully connected layers, and the dueling architecture, ultimately producing selection probabilities for handover decisions. This structure effectively captures inter−satellite dependencies and adapts to varying numbers of visible satellites. In particular, the self−attention layer captures contextual relationships between satellites and adapts dynamically to changes in $M_k^t$. For satellite m, the feature vector ${\mathbf{s}}_m={\mathbf{S}}_e(m,:)$ is projected into query, key, and value vectors, which are expressed as

$$\begin{array}{cc}\hfill & {\mathbf{q}}_m={\mathbf{s}}_m{\mathbf{W}}_Q,\hfill \\ \hfill & {\mathbf{k}}_m={\mathbf{s}}_m{\mathbf{W}}_K,\hfill \\ \hfill & {\mathbf{v}}_m={\mathbf{s}}_m{\mathbf{W}}_V,\hfill \end{array}$$

(26)

where ${\mathbf{W}}_Q$, ${\mathbf{W}}_K$, and ${\mathbf{W}}_V$ are learnable matrices, and the output dimension is denoted as D. Then, the attention score is calculated by

$${\alpha }_{m,n}={\displaystyle \frac{exp({\mathbf{q}}_m·{\mathbf{k}}_n/\sqrt{D})}{{\sum }_{n=1}^{M_k^t}exp({\mathbf{q}}_m·{\mathbf{k}}_n/\sqrt{D})}},$$

(27)

where $\sqrt{D}$ is a scaling factor to stabilize the gradients. The output vector for satellite m is

$${\mathbf{z}}_m=\sum _{n=1}^{M_k^t}{\alpha }_{m,n}{\mathbf{v}}_n,$$

(28)

constituting the m-th column of matrix $\mathbf{Z}$, where each row captures the inter−satellite dependencies and global contextual information.

The matrix $\mathbf{Z}$ then passes through several fully connected layers with ReLU activations, each processing rows independently to preserve the matrix structure, thereby producing $\mathbf{H}$ and extracting higher−level features via nonlinear transformations. A mask $\mathbf{o}$ identifies valid satellites, defined as

$${\mathbf{o}}_m=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathbf{S}}_e(m,:)\ne 0,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(29)

yielding ${\mathbf{H}}^{\prime }=\mathbf{H}\odot \mathbf{o}$ and retaining valid satellite features.

The dueling architecture, a core component of a rainbow DQN, then estimates the Q−value distributions across two branches. For the state−value branch, the valid features in ${\mathbf{H}}^{\prime }$ are averaged, denoted as

$${\mathbf{h}}_{\mathrm{avg}}={\displaystyle \frac{1}{M_k^t}}\sum _{m=1}^{M_k^t}{\mathbf{H}}^{\prime }(m,:),$$

(30)

and processed through a fully connected layer to obtain the state value $\mathbf{v}$. In parallel, the advantage branch processes $\mathbf{H}$ with a similar fully connected structure to generate the advantage values $\mathbf{A}$, where each row $\mathbf{A}(m,:)$ represents the advantage distribution corresponding to satellite m.

Finally, the network combines the state value $\mathbf{v}$ and the output of the advantage value $\mathbf{A}$ to obtain the Q−value estimation matrix $\mathbf{Q}$. The Q−value distribution for satellite m is computed as

$$\mathbf{Q}(m,:)={\mathbf{v}}^{\top }+\left(\mathbf{A}(m,:)-{\displaystyle \frac{1}{M_k^{max}}}\sum _{n=1}^{{\mathcal{M}}_k^{max}}\mathbf{A}(n,:)\right).$$

(31)

By applying the softmax function to each row of $\mathbf{Q}$, the network derives the selection probability distribution $\mathbf{P}$, where each row $\mathbf{P}(m,:)$ indicates the likelihood of selecting satellite m for each TTG.

4. Numerical Results

4.1. Simulation Settings

To evaluate the effectiveness of the proposed ARTHF in addressing the handover problem, we conducted simulations in an idle−mode TTG handover scenario within a large−scale satellite network. The simulations were performed using a custom−built platform developed with Satellite Tool Kit (STK) 12.2 and Python 3.8. To ensure realism and reliability, satellite radio frequency (RF) and TTG parameters were configured in accordance with the NTN RF system specifications outlined in 3GPP TR 38.821. Specifically, the Ka−band (20 GHz) was adopted in the simulation, as is commonly used in LEO satellite systems, such as Starlink, to support high−bandwidth data transmission. For the satellite RF parameters, a system bandwidth of 400 MHz was configured, along with a wave Effective Isotropic Radiated Power (EIRP) density of 4 dBW/MHz. The satellites were equipped with directional antennas, featuring a maximum gain of 38.5 dBi and a 3 dB Half−power Beamwidth (HPBW) of 1.7647°. The satellite beam diameter was set as 20 km, with an antenna equivalent aperture of 0.25 m. A minimum communication elevation angle of 40° was maintained. For the user RF parameters, the frequency was also set to 20 GHz. The TTGs were likewise equipped with directional antennas, which featured a receiver gain of 39.7 dBi. The antenna noise temperature for the TTGs was 150 K, and the noise factor was 1.2 dB.

The simulation modeled a typical large−scale LEO satellite constellation, based on the design of SpaceX’s Starlink, consisting of multiple sub−constellations, as outlined in

Table 1. Constellation structure parameters.

Sub−Constellation	Orbital Altitude	Number of Satellites	Inclination	Number of Orbital Planes
1	540 km	1584	53°	72
2	550 km	1584	53°	72
3	560 km	348	97.6°	6
4	560 km	172	97.6°	4
5	570 km	720	70°	36

. The TTGs were arranged in a $20\times 20$ grid, scaled to a 25 km grid size, with the traffic distribution derived from the Milan traffic dataset [22] due to its realistic representation of temporal traffic variations, which are critical for assessing handover strategies in dynamic satellite networks, enabling the simulation of diverse traffic conditions, including high- and low−demand periods. Within each 15 s interval (three time slots), the dataset captured dynamic traffic patterns that ranged from peak loads to lower loads that reflected realistic user demand fluctuations. The simulation platform generated link relationship data at each time slot, where each slot lasted 5 s. Traffic data for each TTG, sourced from the Milan dataset, was updated every three slots (15 s). To align with this granularity, the handover trigger interval was also set as three slots, and traffic conditions were adjusted accordingly.

The ARTHF integrates two neural network modules. The LSTM−based traffic predictor adopts a two−layer stacked architecture that processes historical data with dimensions [$T_l=3$, $F_l=3$], representing three time steps and three feature channels per TTG. Each LSTM layer comprises three units, followed by fully connected layers (128, 64, 32, 1) with ReLU activations to generate traffic predictions. The attention−enhanced rainbow DQN for handover decision−making accepts a zero−padded input state matrix ${\mathbf{S}}_e\in {\mathbb{R}}^{M_k^{max}=8\times 7}$. The model includes a self−attention layer, followed by three fully connected layers (256, 128, 64), and adopts a dueling architecture that separates the state−value and advantage streams. The resulting Q−values guide the action selection process. Both networks are trained with a learning rate of 0.001 and a discount factor $\gamma =0.95$ for 1000 episodes.

4.2. Performance Discussion

To assess the performance of the proposed ARTHF, we compared it with the following methods: (1) a classic rainbow DQN algorithm incorporating traffic prediction, denoted as ‘rainbow DQN + traffic prediction’; (2) an attention−enhanced rainbow DQN algorithm operating under fixed traffic conditions (i.e., without traffic prediction), denoted as ‘attention−enhanced rainbow DQN’; and (3) a classic rainbow DQN algorithm under fixed traffic conditions, denoted as ‘rainbow DQN’. These methods were selected to isolate and evaluate the individual contributions of the traffic prediction and attention mechanisms, while also providing fair comparisons against both traditional and advanced DRL−based handover strategies. For the algorithms without traffic prediction, the lack of insight into TTG traffic variations leads to the adoption of a greedy approach for load−balancing rewards. Specifically, these algorithms encourage all TTGs to select the satellite with the currently lowest load, without considering the real traffic demands of the TTGs in the reward design.

Figure 5 shows the average number of handovers per slot for each algorithm. The different algorithms evaluated and presented in the figure’s legend are the ’attention−enhanced rainbow DQN’; the ’rainbow DQN’; the ’ARTHF (attention−enhanced rainbow DQN + traffic prediction)’, which is our proposed framework; and the ’rainbow DQN + traffic prediction’. The attention−enhanced rainbow DQN significantly reduces the handover frequency compared with the classic rainbow DQN, which struggles with varying numbers of candidate satellites due to fixed input structures. This leads to positional bias and unbalanced training, where satellites later in the input are undervalued. By introducing a self−attention layer, the improved architecture captures inter−satellite relationships and eliminates positional bias, enabling more accurate and stable handover decisions. Notably, the ARTHF performs similarly to its non−predictive counterpart in terms of handover frequency, indicating that traffic prediction does not compromise handover stability and instead improves other performance aspects. Thanks to the Milan dataset’s inclusion of both peak and off−peak traffic scenarios, the framework’s robustness across fluctuating demand levels is well validated, consistently achieving lower handover frequency compared with the baseline methods and demonstrating its efficiency and stability in diverse network loads.

To evaluate the load−balancing performance, we used Jain’s index [23] and the satellite peak load as key metrics. Jain’s index is a widely used measure of fairness, ranging from 0 (worst) to 1 (perfect fairness), indicating how evenly traffic is distributed across satellites. As shown in Figure 6, the attention−enhanced algorithms consistently achieved significantly higher Jain’s fairness indices compared with the classic DQN architecture, highlighting the limitations of the traditional approach in managing constrained candidate satellites. By incorporating the traffic prediction, the improved architecture further increased the Jain’s fairness index, though the improvement was modest, demonstrating that anticipating future demand promotes a more equitable distribution of network load. Figure 7 illustrates the peak load experienced by each satellite at each time slot, providing a clear visualization of the congestion risk. The prediction−driven algorithm significantly reduced the satellite peak loads compared with its non−predictive counterpart, indicating a lower likelihood of system congestion. In particular, under high−load conditions, the ARTHF leverages traffic forecasts to proactively assign TTGs with high expected demand to underutilized satellites. This predictive load−balancing strategy mitigates congestion and enhances the overall fairness, as evidenced by the improved Jain’s index and reduced peak loads.

The communication quality was evaluated using the average SNR across all TTGs, as shown in Figure 8. The prediction−driven algorithms demonstrated a significant increase in the average SNR compared with both its non−predictive counterparts, highlighting a core advantage of traffic forecasting. By anticipating future traffic demands, the algorithm enables differentiated handover strategies. TTGs with predicted high future demand can be assigned to satellites prioritized for load capacity, even if the instantaneous SNR is slightly compromised. Conversely, TTGs with a lower predicted demand can be allocated to satellites offering optimal signal quality without significantly impacting the overall network load. This flexibility allows the system to better optimize the overall communication quality. Notably, in low−load scenarios, the ARTHF framework naturally prioritizes the signal strength, selecting satellites with an optimal SNR and thereby ensuring a robust performance under varying traffic conditions.

In summary, the simulation results clearly demonstrate the superior performance of the ARTHF in optimizing handover decisions for LEO satellite networks. By integrating LSTM−based traffic prediction with an attention−enhanced rainbow DQN, the framework enables proactive load balancing and alleviates congestion under high−load conditions, as reflected in the improved Jain’s fairness index and reduced peak loads shown in Figure 6 and Figure 7. It also preserves high communication quality in low−load scenarios, as indicated by the enhanced SNR in Figure 8, while consistently achieving lower handover frequencies across diverse traffic scenarios, as illustrated in Figure 5. These findings, supported by the realistic and dynamic traffic patterns of the Milan dataset, highlight the robustness and adaptability of ARTHF for practical use in dynamic satellite−to−ground communication systems.

5. Conclusions

In this study, we addressed the critical challenge of handover selection in large−scale LEO satellite−to−ground networks involving numerous TTGs. To adapt to dynamic traffic patterns, we incorporated an LSTM−based traffic prediction model using historical data, enabling proactive and informed decision−making. The handover problem was formulated as an MDP, and we propose an attention−enhanced rainbow DQN algorithm that jointly considers the satellite switching frequency, communication quality, and satellite load. The proposed ARTHF features a flexible architecture applicable to various constellation configurations and traffic scenarios, as reflected in the Milan dataset’s diverse traffic patterns. Extensive simulations consistently demonstrated ARTHF’s significant superiority over the baseline methods, where it achieved substantial improvements in load balancing, improved communication quality, and reduced handover frequency. These compelling results affirm ARTHF’s robustness, efficiency, and practical effectiveness in ensuring seamless and optimized connectivity within dynamic satellite network systems.