Computation Offloading in Space–Air–Ground Integrated Networks for Diverse Task Requirements with Integrated Reliability Mechanisms

Abstract

The sixth-generation (6G) system has been attracting increasing attention from both industry and academia, with the space–air–ground integrated network (SAGIN) identified as one of its key applications. This study investigates a SAGIN framework tailored for deployment in remote areas. To address the differing needs of users with emergency and routine tasks, an offloading strategy is proposed that enables direct offloading for emergency tasks and optimized UAV-assisted offloading for routine tasks. Additionally, considering the limited satellite coverage duration, a reliability mechanism for task offloading is designed. The study formulates a task offloading optimization problem aimed at maximizing the completion rate of routine tasks—while reducing their energy consumption and latency—under the premise of guaranteeing the completion of emergency task offloading. The problem is modeled as a Markov Decision Process (MDP). To solve it, a D-MAPPO reinforcement learning algorithm is proposed, which integrates the Dirichlet distribution with the Multi-Agent Proximal Policy Optimization (MAPPO) framework. Simulation results show that, compared with the MAPPO and PPO algorithms, the delay is reduced by 38% and 31%, respectively, while the energy consumption is reduced by 7% and 48%, respectively.

1. Introduction

With the ongoing development of sixth-generation (6G) wireless communication systems, aerial access networks and space–air–ground integrated networks (SAGINs) have become major focal points for both industry and academia [1,2]. In recent years, significant research efforts have been dedicated to exploring the application potential of SAGIN in domains such as the Internet of Things (IoT) [3], cognitive communication [4], and edge computing [5], particularly in expanding its use cases in remote areas. By integrating satellites in space, unmanned aerial vehicles (UAVs) in the air, and terrestrial base stations and data centers, SAGIN constructs a cross-domain collaborative network architecture, offering critical support for achieving seamless global connectivity [6]. In particular, in remote areas such as deserts, oceans, and sparsely populated regions—where terrestrial base stations are scarce and costly to deploy—traditional communication networks often fail to provide adequate coverage. Meanwhile, data collected from environmental monitoring in these regions hold significant value for global information systems. Owing to its extensive coverage, high flexibility, and reliability, SAGIN is considered a promising solution to enable real-time data collection and transmission for the Internet of Remote Things (IoRT), and to address the connectivity challenges in remote environments [7].

Companies such as SpaceX and OneWeb are accelerating the deployment of large-scale low Earth orbit (LEO) satellite constellations, redefining network architecture to enable low latency, high capacity, and global services [8,9,10]. This trend offers new opportunities for building global connectivity. Compared to medium Earth orbit (MEO) and geostationary Earth orbit (GEO) satellites, LEO satellites are closer to the Earth and thus better suited for latency-sensitive communications. LEO satellites are rapidly advancing in terms of capacity and coverage [11,12,13,14]. Their low-latency characteristics make them ideal for IoT applications. The rise of large-scale LEO constellations is fundamentally reshaping network structures to achieve the goals of low latency, high capacity, and global coverage. This lays a solid foundation for realizing a truly interconnected world, provides reliable communication for remote areas, and helps bridge the digital divide. Moreover, Mobile Edge Computing (MEC), a revolutionary concept aimed at enhancing the low-latency and high-bandwidth capabilities of communication systems, has also been integrated into the SAGIN architecture to enable proximal deployment of computing resources and satisfy real-time processing and high-speed transmission requirements [15].

Despite SAGIN’s vast development potential, its practical implementation faces multiple challenges. Due to the highly dynamic nature of satellite communications and the inherent resource limitations, the network is prone to interruptions and data loss. Therefore, there is an urgent need to build highly reliable and stable systems to ensure communication quality [16]. At the same time, latency and energy consumption remain key optimization targets. In remote scenarios, user requirements differ significantly across applications. For instance, emergency rescue and telemedicine demand extremely low latency, while environmental monitoring and resource exploration focus more on energy efficiency and coverage. Balancing these heterogeneous demands while improving overall latency and energy performance has become a central research issue.

In edge computing environments, end devices often have limited computing capabilities, and edge servers themselves face constraints in both number and resources. When the volume of offloading requests exceeds the processing threshold of edge nodes, service quality inevitably degrades—leading to increased task failure rates and service latency [17]. To address this issue, researchers have proposed various resource allocation and task scheduling optimization strategies, aiming to alleviate edge overload through the design of effective offloading decision models [18]. Furthermore, UAV-based auxiliary computing platforms have been introduced to expand the service scope and capacity of edge computing, forming multi-level air–ground collaborative offloading systems. However, due to limitations in spectrum resources and node capacity—particularly in extreme environments such as disaster zones, oceans, and deserts—edge computing still faces significant challenges in providing adequate coverage and computational support for dense IoT terminals in remote areas [19,20].

In conclusion, SAGIN research aimed at remote regions must not only address fundamental connectivity issues but also optimize latency and energy performance across diverse application scenarios. It is equally critical to take into account the limited computing resources of edge nodes when designing efficient and intelligent task offloading and resource management mechanisms. In the future, by enabling intelligent coordination and scheduling of space–air–ground resources, SAGIN’s integrated performance in remote applications is expected to improve significantly, promoting a more balanced development of global communication networks.

In this study, we investigate a SAGIN that combines hybrid cloud services with mobile edge computing (MEC). In our design, both cloud servers and satellites are regarded as components of the MEC infrastructure, while UAVs are designated as the primary decision-making agents. The objective is to enhance the overall quality of service (QoS) by enabling UAVs to select optimal task offloading strategies. Beyond the conventional network parameters such as bandwidth and computational capacity, we further consider the limitations on the number of tasks that can be accepted by cloud servers and satellites. This allows for a more comprehensive analysis of how these constraints affect offloading decisions and overall network performance. To address the diverse requirements of different task types, we classify tasks into two categories: urgent tasks and normal tasks. Correspondingly, we design two distinct offloading strategies to ensure multi-tier QoS guarantees. Moreover, while conventional deep reinforcement learning (DRL) algorithms have demonstrated adaptability in dynamic environments, most rely on discrete action spaces, thereby limiting optimization efficiency. To overcome this, we propose a novel multi-agent reinforcement learning algorithm—D-MAPPO—by integrating the Dirichlet distribution with Multi-Agent Proximal Policy Optimization (MAPPO). This approach leverages continuous action spaces, enabling more efficient optimization in dynamic environments. This approach leverages continuous action spaces, enabling more efficient optimization in dynamic environments. The main contributions of this work are summarized as follows:

• A SAGIN suitable for remote areas is proposed. Considering the unique application requirements of remote regions, computational tasks are classified into normal and urgent tasks to address diverse demands under different scenarios.
• To address the limited satellite coverage time and improve communication quality, a reliability mechanism for task offloading from ground sensors and UAVs to satellites is proposed. In addition, the communication process, satellite coverage duration, and overall network cost are modeled within the proposed framework.
• Based on the distinct characteristics of urgent and normal tasks, a computation offloading problem is formulated to jointly optimize network energy consumption and latency. The optimization problem is modeled as a Markov Decision Process (MDP). Considering the multidimensional offloading decision space and specific constraints, a Dirichlet-based Multi-Agent Proximal Policy Optimization (D-MAPPO) algorithm is proposed to enable the learning of optimal task offloading strategies.
• Extensive simulation results demonstrate that the D-MAPPO algorithm achieves faster and more stable convergence. Moreover, it consistently outperforms benchmark methods—including Beta-MAPPO, PPO, Local, Offloading, and Random—in terms of latency reduction, energy efficiency, and offloading success rate.

The remainder of this paper is organized as follows: Section 2 reviews related work in the field of SAGIN. Section 3 presents the proposed system model. Section 4 formulates and analyzes the offloading optimization problem. Section 5 introduces and implements the MAPPO algorithm. In Section 6, we analyze the experimental results. Finally, Section 7 concludes the paper.

2. Related Work

As a core architecture to support future 6G communications and edge computing, SAGINs demonstrate great potential in applications such as emergency communications, large-scale IoT, and vehicular networks. Existing studies have extensively explored the design of network architectures, task offloading mechanisms, and algorithmic solutions to balance critical performance metrics such as energy consumption, latency, and robustness. This chapter systematically reviews relevant progress from two perspectives: computation offloading and algorithm design.

2.1. Task Offloading in SAGIN

Reference [21] proposes a task offloading model based on deep reinforcement learning and user experience. After a user generates a blockchain task, a Proof-of-Work (PoW) consensus mechanism is introduced to package transaction information into blocks, ensuring the authenticity and reliability of interactions within the system. By incorporating total system delay and user rewards, the concept of user experience is defined, and an optimal user experience objective is formulated.

A previous study [22] investigates data collection for IoT sensors using UAVs assisted by low Earth orbit (LEO) satellites. Specifically, UAVs collect data from IoT sensors and transmit it back to Earth via two modes: (1) a delay-tolerant mode using UAV-carried storage and (2) a delay-sensitive mode through the UAV–satellite communication network.

Another study [23] addresses communication challenges in remote areas lacking cellular infrastructure. It explores the role of UAVs in SAGIN, where UAVs act as control-layer nodes due to their flexible deployment and relay capabilities. Equipped with millimeter-wave radars and visual sensors, UAVs gather multisource data to reduce uncertainty and improve decision accuracy. Meanwhile, UAVs collect computation tasks from mobile devices within their coverage areas and offload them to other processing units to enhance computational efficiency.

Considering the unreliable aerial communication environment in remote regions, the offloading process for power-constrained IoT devices remains challenging. A previous study [24] proposes an energy-efficient EC-SAGIN architecture, where IoT devices select the most suitable LEO satellite or UAV for task offloading based on their energy levels, communication conditions, and computing capabilities. Similarly, Ref. [25] highlights the critical role of computation offloading and resource allocation under limited satellite communication and computing resources. By jointly optimizing offloading decisions and allocating wireless and computational resources, the approach minimizes system energy consumption while meeting latency constraints.

The authors of [26] study computation offloading in large-scale MIMO-enabled MEC systems, where mobile devices with limited resources offload tasks to high-performance edge servers. The objective is to minimize both the power consumption and offloading delay of mobile devices under stochastic network conditions.

Overall, the above studies are either conducted under general remote-area scenarios without considering emergency offloading situations or lack reliability mechanisms to ensure stable transmission of control information. Consequently, there remains a need for a network architecture and computation offloading mechanism that can operate efficiently under normal conditions while providing rapid responsiveness during emergencies in remote regions.

2.2. Optimization Algorithms

A previous study [27] proposes a hybrid algorithm combining Particle Swarm Optimization (PSO) and a Greedy Strategy (GS), referred to as PSO&GS, to minimize the system’s average response delay and obtain a near-optimal solution. Extensive simulations verify the convergence of the proposed algorithm, and numerical results show that it achieves excellent convergence performance, reducing the average response delay to about 0.65–0.85 times that of the baseline algorithms.

Another study [28] aims to minimize the overall energy consumption and delay during data aggregation and computation processes. A multi-agent deep reinforcement learning algorithm is employed, integrating value decomposition and double deep Q-network (DQN) techniques to optimize data aggregation and enable a cost-effective cooperative offloading process. Experimental results demonstrate that, compared with traditional approaches, the proposed method reduces training time, data processing volume, energy consumption, and task duration by 20%, 11.4%, 5.6%, and 11.2%, respectively, while serving up to 98% of IoT devices.

The authors of [29] develop a novel deep risk-sensitive reinforcement learning algorithm. Specifically, it evaluates the risk associated with each state and jointly learns optimal parameters to balance delay minimization and risk control. Simulation results show that, compared with probabilistic configuration methods, the proposed approach reduces task processing delay by 30% while satisfying UAV energy constraints.

In [30], the authors model the ground power facility (GPF) association and power control subproblems as a multi-agent, time-varying K-armed bandit problem. A joint optimization algorithm based on multi-agent temporal difference (TD) reinforcement learning is proposed, alternately optimizing the two subproblems. Results demonstrate that, under various conditions (including different noise power levels, GPF bandwidths, and GPF quantities), the proposed method improves overall system energy efficiency by 16.23%, 86.29%, and 5.11% compared with three baseline algorithms (random path, average transmission power, and random device association).

Another study [31] introduces an asynchronous federated learning-based adaptive collaborative aggregation algorithm (AFLS), enabling participating nodes to update model parameters dynamically during aggregation. Experiments show that with effective collaboration among nodes, the proposed framework achieves outstanding performance in image classification tasks, reaching 95% accuracy with fewer communication rounds.

Ref. [32] presents a reinforcement learning-based control framework for cascade formation of perturbed surface vehicles. The framework aims to achieve precise trajectory tracking while minimizing a predefined cost function to enhance system performance. Theoretical analysis ensures that both individual agents and the overall closed-loop control system satisfy the uniformly ultimately bounded (UUB) condition for tracking and learning errors. Results confirm that the proposed approach exhibits excellent robustness and feasibility in formation control under disturbed environments.

However, traditional intelligent algorithms and single-agent reinforcement learning methods can no longer meet the optimization requirements of large-scale network environments. In existing multi-agent reinforcement learning research, various constraints in offloading decisions are typically learned implicitly through network training. Nevertheless, there is still a lack of algorithms explicitly designed to adapt to action space constraints and structurally integrate these limitations within the learning process.

3. System Model

This section describes the overall network architecture and its mathematical models.

Table 1. Summary of Notations in Our System.

Notation	Description	Notation	Description
$I,U,S$	Sets of sensors, UAVs, and satellites	$N_0$	Spectral density of added white Gaussian noise (AWGN)
$i,u,s$	Numbers of sensors, UAVs, and satellites	${\theta }_G$	Communication between ground
C	Cloud server	$d_E$	Earth’s radius
G	Ground equipment (including ground sensors, UAVs, and cloud servers)	$d_o$	Satellite orbit altitude
M	Set of tasks	$d_{GS}$	Distance between GD and satellite
m	Number of tasks	${\theta }_c$	Coverage angle of low Earth orbit (LEO) satellites
$\phi$	Data amount of a task	$L_S$	Coverage arc length of the satellite
$\rho$	Complexity of a task	$T_S$	Coverage time of the satellite
n	Normal task	$V_S$	Satellite speed
$ur$	Urgent task	${\mu }_{(u,u)}$	Proportion of local computation done by UAV u
$p^{loss}$	Average path loss of the air-to-ground channel	${\mu }_{(u,C)}$	Proportion of task offloaded to the cloud server by UAV u
$p^{los}$	Line-of-sight (LOS) between ground equipment and UAV	${\mu }_{(u,s)}$	Proportion of task offloaded to the satellite by UAV u
r	Horizontal distance between UAV and ground equipment	$\tau$	Energy coefficient
h	Flight altitude of the UAV	$N_i^n$	Total number of normal tasks generated by sensor i
${\eta }^{LoS}$,${\eta }^{NLoS}$	Additional losses in LOS and non-LOS links based on free-space path loss	$N_i^{ur}$	Total number of urgent tasks generated by sensor i
$f_c$	Carrier frequency	$N_u$,$N_C$,$N_s$	Total number of normal tasks received by UAV u, cloud server, and satellite s
c	Speed of light	$f_u$,$f_C$,$f_s$	Computational capacities of UAV u, cloud server, and satellite s
$R_I,R_U,R_S,R_C$	Maximum transmission rates of ground sensors, UAVs, satellites, and cloud servers	$T_u^{tran},T_C^{tran},T_s^{tran}$	Transmission delays for task collection by UAV u, cloud server, and satellite s
$B_I,B_U,B_S,B_C$	Bandwidth of ground sensors, UAVs, satellites, and cloud servers	$E_u^{tran},E_C^{tran},E_s^{tran}$	Transmission energy consumption for task collection by UAV u, cloud server, and satellite s
$P_I,P_U,P_S,P_C$	Transmission powers of ground sensors, UAVs, satellites, and cloud servers	$T_u^{comp},T_C^{comp},T_s^{comp}$	Computation delays for UAV u, cloud server, and satellite s
${\sigma }^2$	Gaussian noise power	$E_u^{comp},E_C^{comp},E_s^{comp}$	Computation energy consumption for UAV u, cloud server, and satellite s
$\xi$	Channel coefficient	$T_u^{back},T_C^{back},T_s^{back}$	Backhaul delays for UAV u, cloud server, and satellite s
F	Rician factor	$E_u^{back},E_C^{back},E_s^{back}$	Backhaul energy consumption for UAV u, cloud server, and satellite s
$\alpha$	Distance attenuation factor	b	Overhead coefficient
${\xi }^{LoS},{\xi }^{NLoS}$	LOS and non-LOS channel gains between satellites and communication devices	$T_S^{tran}$	Total time for transmitting tasks to the satellite
$R_{I,S},R_{U,S}$	Maximum transmission rates of ground sensors and UAVs to satellites	${\phi }_S^{tran}$	Total data transmitted to the satellite
$G_0$	Fixed antenna gain

provides explanations for all parameters in this section.

3.1. Network Architecture

As shown in Figure 1, for urgent tasks, the sensor directly offloads them to the satellite for processing, and the satellite returns the computation results directly to the sensor. For normal tasks, they are first collected by UAVs. The sensors upload the tasks to the UAVs, which, based on the task offloading strategy, may then offload the tasks to either satellites or cloud servers. Each UAV supports bidirectional communication with both sensors and cloud servers. Note that since a return link between the satellite and the sensor has already been established, the satellite does not need to send the computation results back to the UAV. Therefore, the communication between the UAV and the satellite is unidirectional.

Let I denote the set of sensors, where $i\in I$; U denote the set of UAVs, where $u\in U$; S denote the set of satellites, where $s\in S$; and C denote the cloud server, which is assumed to be a single entity. Let M represent the set of tasks, where $m\in M$, with $M^n$ denoting normal tasks and $M^{ur}$ representing urgent tasks.

3.2. Task Model

In practical environmental monitoring IoT systems, the urgency levels of tasks vary across different scenarios. Based on their urgency, tasks are categorized into two types:

(1)

Urgent Tasks: Tasks triggered by emergency events such as wildfires, earthquakes, or other sudden incidents are defined as urgent tasks. Among all tasks, urgent tasks represent a small proportion but must be processed with the highest priority. When there are too many tasks to unload, priority should be given to unloading the completion rate of urgent tasks.

(2)

Normal Tasks: Most computational tasks generated by ground sensors are classified as normal tasks. Hence, optimizing normal tasks is the central focus of computation offloading in the system. Given that normal tasks are generally delay-tolerant, a joint optimization of energy consumption and latency is desirable.

3.3. Communication Model

(1)

Communication Between UAVs and Ground Sensors: According to [33], the air-to-ground communication channel depends on the UAV’s altitude, the elevation angle, and the propagation environment [34]. As described in [35], the average path loss for the air-to-ground channel can be expressed as follows:

$$\begin{array}{c}\hfill P^{\mathrm{loss}}\left(r,h\right)=20log\left({\displaystyle \frac{4\pi f_c{\left(h^2+r^2\right)}^{\frac{1}{2}}}{c}}\right)+P^{\mathrm{Los}}{\eta }^{\mathrm{Los}}+\left(1-P^{\mathrm{Los}}\right){\eta }^{\mathrm{NLos}},\end{array}$$

(1)

where $P^{Los}$ denotes the probability of a line-of-sight (LoS) link between the UAV and the ground devices(GDs); h is the UAV altitude and r is the horizontal distance to the sensor; ${\eta }^{Los}$ and ${\eta }^{NLos}$ are the additional losses associated with LoS and non-LoS (NLoS) conditions [36]. $f_c$ is the carrier frequency, and ccc is the speed of light. According to [37], the values in remote areas are $\left({\eta }^{Los},{\eta }^{NLos}\right)=0.1,2.1$.

We assume that the communication link between the sensor and the UAV operates in the C-band spectrum. The maximum transmission rate for the ground sensor $\left(R_I\right)$ and the UAV $\left(R_U\right)$ is given by

$$\left\{\begin{array}{c}{R}_I=B_I{log}_2\left(1+{\displaystyle \frac{P_I·{10}^{\frac{P^{\mathrm{loss}}}{10}}}{{\sigma }^2}}\right)\hfill \\ R_U=B_U{log}_2\left(1+{\displaystyle \frac{P_U·{10}^{\frac{P^{\mathrm{loss}}}{10}}}{{\sigma }^2}}\right),\hfill \end{array}\right.$$

(2)

where $B_I$ and $B_U$ are the bandwidths of the ground sensor and the UAV, respectively. It is worth noting that device bandwidths are not fixed and may vary dynamically [38]. $P_I$ and $P_U$ represent the transmit power of the sensor and the UAV, respectively, while ${\sigma }^2$ denotes Gaussian noise power.

(2)

Communication Between UAVs and the Cloud Server: The cloud server is also deployed on the ground, so its communication model with UAVs is similar to that between UAVs and sensors. The main difference is that the cloud server not only receives task data but also sends the processed results back to the UAV, which then delivers them to GD. Therefore, the maximum transmission rate $R_c$ between the UAV and the cloud server is given by

$$R_c=B_c{log}_2\left(1+{\displaystyle \frac{P_c{10}^{\frac{P^{\mathrm{loss}}}{10}}}{{\sigma }^2}}\right),$$

(3)

where $B_c$ is the bandwidth of the cloud server, and $P_c$ is its transmit power.

(3)

Communication between UAV and ground sensors and satellites: The communication links between UAV and satellites are mainly based on clear line-of-sight (LoS) links, supplemented by a small number of NLoS links. We model the communication channel between UAV and GD and satellites as a Rician channel [39,40]; therefore, the channel gains of LoS and NLoS are integrated, so the channel coefficients between UAV and ground sensors and satellites $\xi =\sqrt{{\displaystyle \frac{F}{1+F}}}{\xi }^{LoS}+\sqrt{{\displaystyle \frac{{\left(d_{s,u}\right)}^{-\alpha }}{1+F}}}{\xi }^{NLoS}$, where F is the Rician factor, $\alpha$ denotes the distance attenuation factor, and ${\xi }^{LoS}$ and ${\xi }^{NLoS}$ denote the LoS and NLoS channel gains between satellites and communication devices, respectively. Finally, the maximum transmission rates $R_{U,S}$ and $R_{I,S}$ from UAVs and ground sensors to the satellite, as well as the satellite’s maximum transmission rate $R_S$, can be determined as follows:

$$\left\{\begin{array}{cc}{R}_{U,S}\hfill & =B_{U}lo{g}_2(1+{\displaystyle \frac{P_U{G}_0{\left|\xi \right|}^2}{N_0{B}_U}}),\hfill \\ R_{I,S}\hfill & =B_{I}lo{g}_2(1+{\displaystyle \frac{P_I{G}_0{\left|\xi \right|}^2}{N_0{B}_U}}),\hfill \\ R_S\hfill & =B_{S}lo{g}_2(1+{\displaystyle \frac{P_S{G}_0{\left|\xi \right|}^2}{N_0{B}_S}}),\hfill \end{array}\right.$$

(4)

where $B_S$ is the available bandwidth of the satellite, $G_0$ is the fixed antenna gain. $P_s$ denotes the transmit power of the satellite, and $N_0$ is the spectral density of the additional white Gaussian noise (AWGN).

3.4. Satellite Coverage Model

Considering the dynamic characteristics of low Earth orbit (LEO) satellites, sensors and UAVs can communicate with satellites only when they are within the satellite’s coverage area. Therefore, it is essential to model the coverage time of these satellites as a fundamental reference for making informed task offloading decisions. Figure 2 illustrates the geometric relationship between LEO satellites and GD, based on which the LEO satellite coverage time model is derived. It is worth noting that the satellites are positioned at an altitude of 300 km above the Earth’s surface, while UAVs operate at altitudes below 100 m. Given that the UAVs’ altitude is negligible compared to that of the satellites, the UAV height is disregarded in the satellite coverage model. Thus, UAVs and sensors share the same satellite coverage model, and in this context, both UAVs and sensors are collectively referred to as GD.

In Figure 2, $d_E$ represents the Earth’s radius, $d_o$ denotes the satellite’s orbital altitude, and $d_{GS}$ is the distance between the GD and the satellite. According to [41], the elevation angle ${\theta }_c$ of communication between ground equipment and the satellite can be expressed as follows:

$${\theta }_G=arccos\left({\displaystyle \frac{d_E+d_o}{d_{GS}}}sin{\theta }_c\right),$$

(5)

where ${\theta }_c$ is the coverage angle of the LEO, denoted as follows:

$${\theta }_c=arccos\left({\displaystyle \frac{d_E}{d_E+d_O}}cos{\theta }_G\right)-{\theta }_G.$$

(6)

From this, we can calculate the coverage arc length of the satellite $L_S$ as

$$L_S=2\left(d_E+d_o\right){\theta }_c.$$

(7)

Therefore, the coverage arc length of the satellite can be calculated, and then the coverage time of the satellite $T_S$ can be calculated:

$$T_S={\displaystyle \frac{L_S}{V_S}},$$

(8)

where $V_S$ is the satellite’s flight speed.

3.5. Computational Model

In the operational environment considered in this study, tasks can be processed by UAVs, satellites, or cloud servers, while sensors are not responsible for task processing. For a given task mmm generated by a sensor, we define $m=\left\{{\phi }_m,{\rho }_m\right\}$, where ${\phi }_m$ represents the data size of the task, and ${\rho }_m$ denotes the computational complexity, i.e., the number of CPU cycles required to process one bit of data. Normal tasks are first collected by UAVs and then processed based on the UAVs’ offloading decisions. The offloading decision of UAV u is defined as $({\mu }_{(u,u)},{\mu }_{(u,C)},{\mu }_{(u,s)}),{\mu }_{(u,u)}+{\mu }_{(u,C)}+{\mu }_{(u,s)}=1$. Here, ${\mu }_{(u,u)}$ represents the proportion of tasks computed locally by UAV u, ${\mu }_{(u,C)}$ denotes the proportion of tasks offloaded to the cloud server, and ${\mu }_{(u,s)}$ represents the proportion of tasks offloaded to the satellite. In contrast, urgent tasks are directly offloaded by the sensors to the satellites for processing, with the results subsequently transmitted back to the sensors by the satellites.

The following section presents an analysis of the computational models for the processing devices within the network, including UAVs, satellites, and cloud servers.

(1)

UAV computation: Each UAV collects tasks offloaded from multiple sensors. However, since UAVs have limited computational resources, offloading a portion of tasks is necessary. Let $N_i^n$ denote the total number of normal tasks generated by sensor iii; thus, the total number of tasks collected by UAV u is $N_u={\sum }_i^I{N}_i^n$. The transmission time for UAV u to collect all tasks is given by

$$T_u^{tran}={\displaystyle \frac{b{\sum }_x^{N_u}{\phi }_x^n}{R_I}}$$

(9)

where $b>1$ is the transmission overhead factor [42]. The total transmission energy consumption for UAV u to collect all normal tasks is as follows:

$$E_u^{tran}=P_I{T}_u^{tran}.$$

(10)

According to the above equations, the number of tasks computed locally by UAV u can be represented as $⌊{\mu }_{(u,u)}{N}_u⌋$, the total local computational latency of UAV u is

$$T_u^{comp}={\displaystyle \frac{{\sum }_x^{\lfloor {\mu }_{(u,u)}{N}_u\rfloor }{\phi }_x^n{\rho }_x^n}{f_u}},$$

(11)

where ${\phi }_x^n$ and ${\rho }_x^n$ represent the data size and computational complexity of the x-th task, respectively. $f_u$ denotes the computational capability of UAV u. The total local computational energy consumption of UAV u is:

$$E_u^{comp}=\tau f_u^2{\sum }_x^{\lfloor {\mu }^{(u,u)}{N}_u\rfloor }{\phi }_x^n{\rho }_x^n$$

(12)

where $\tau$ is the energy coefficient and depends on the CPU structure of the UAV u. When the calculation is complete, the UAV u sends the resultant data back to the sensor with a return time of

$$T_{u,i}^{back}={\displaystyle \frac{b{\phi }_u^{re}}{R_{u,i}}},$$

(13)

where ${\phi }_u^{re}$ denotes the total data volume of the UAV u result data. The energy consumption of the UAV u return data is as follows:

$$E_u^{back}=P_U{T}_u^{back}.$$

(14)

(2)

Cloud server computation: The cloud server processes tasks offloaded from multiple UAVs. Since the cloud server’s computational resources are limited, the total number of tasks received by the cloud server is $N_C={\sum }_u^U{\mu }_{(u,C)}{N}_u$. The transmission time and the energy consumption for the cloud server to receive UAV-offloaded tasks are denoted as $T_c^T$ and $E_c^T$, respectively:

$$T_c^{tran}={\displaystyle \frac{b{\sum }_x^{N_C}{\phi }_x^n}{R_U}},$$

(15)

$$E_C^{tran}=P_U{T}_C^{tran}.$$

(16)

The computational latency $T_C^C$ and computational energy consumption $E_C^C$ of the cloud server are

$$T_c^{comp}={\displaystyle \frac{{\sum }_x^{N_C}{\phi }_x^n{\rho }_x^n}{f_c}},$$

(17)

$$E_u^{comp}=\tau f_C^2{\sum }_x^{N_C}{\phi }_x^n{\rho }_x^n,$$

(18)

where $f_C$ is the computational capability of the cloud server. The cloud server needs to return the computation result of the task to the UAV, and the return delay $T_C^{Back}$ and the energy consumption $E_C^{Back}$ of the cloud server will be

$$T_C^{Back}={\displaystyle \frac{b{\phi }_C^{re}}{R_C}},$$

(19)

$$E_C^{Back}=P_C{T}_C^{Back}.$$

(20)

(3)

Satellite computation: The satellite not only has to deal with the tasks offloaded by the UAV locally, but also has to deal with the urgent tasks uploaded by the sensors. When satellites receive offloaded tasks, urgent tasks are prioritized to ensure timely processing. The total number of normal tasks received by satellite s is $N_s={\sum }_u^U{\mu }_{(u,s)}{N}_u$, where $N_i^{ur}$ represents the total number of urgent tasks generated by sensor iii. The transmission time and energy consumption for satellite s to receive both urgent and normal tasks are represented by $T_s^{tran}$ and $E_s^{tran}$, respectively.

$$\begin{array}{c}\hfill T_s^{tran}=T_s^{tran,ur}+T_s^{tran,n}={\displaystyle \frac{b{\sum }_i^I{\sum }_x^{N_i^{ur}}{\phi }_x^{ur}}{R_{I,S}}}+{\displaystyle \frac{b{\sum }_x^{N_s}{\phi }_x^n}{R_{U,S}}}\end{array}$$

(21)

$$E_{i,s,m}^{tran,ur}=P_{i,s}{T}_s^{tran,ur}+P_{u,s}{T}_s^{tran,n}.$$

(22)

The computation delay and energy consumption of satellite s for processing urgent and normal tasks are denoted as $T_s^{comp}$ and $E_s^{comp}$, respectively:

$$T_s^{comp}={\displaystyle \frac{{\sum }_l^I{\sum }_x^{N_l^{ur}}{\phi }_x^{ur}{\rho }_x^{ur}+{\sum }_x^{N_s}{\phi }_x^n{\rho }_x^n}{f_s}}$$

(23)

$$E_s^{comp}=\tau f_s^2\left(\sum _i^I\sum _x^{N_i^{ur}}{\phi }_x^{ur}{\rho }_x^{ur}+\sum _x^{N_s}{\phi }_x^n{\rho }_x^n\right)$$

(24)

where $f_s$ represents the computational capability of satellite s. The satellite directly returns the computation results to the sensors, and the backhaul delay and energy consumption of satellite s are expressed as $T_s^{back}$ and $E_s^{back}$, respectively:

$$T_s^{back}={\displaystyle \frac{b{\phi }_s^{re}}{R_s}}$$

(25)

$$E_s^{back}=P_s{T}_s^{back}$$

(26)

where ${\phi }_s^{re}$ denotes the amount of result data returned by satellite s.

3.6. Reliability Mechanisms

A task is deemed to have been offloaded successfully only when every single bit has been uploaded without fail. However, given that the coverage time of a satellite is limited, there is a possibility that task offloading to a satellite might end in failure. Should this happen to an urgent task, the consequences would be nothing short of catastrophic.

To address this, a two-pronged approach is proposed. First, a lower bound is established for the maximum transmission rate of the device, which can be determined through the satellite coverage model. Second, when each device proceeds to offload tasks to the satellite, it first calculates the maximum number of tasks that can be offloaded. In this process, priority is given to offloading urgent tasks.

From Section 3.4, we can determine the satellite coverage time $T_S$. The total transmission time for satellite mission data should be less than the satellite coverage time, i.e.,

$$T_s^{tran}={\displaystyle \frac{{\phi }_s^{tran}}{R_{I,S}}}<T_s$$

(27)

where ${\phi }_S^{tran}$ is the total data volume transmitted to the satellite; thus, it can be concluded that the limiting condition for the maximum amount of data transmitted by the UE is

$${\phi }_s^{tran}<R_{I,S}{T}_s.$$

(28)

4. Problem Analysis

This study considers both delay and energy consumption for normal and urgent tasks in SAGIN. For urgent tasks, the primary concerns are whether the task can be completed and how quickly it can be completed. In contrast, normal tasks focus on minimizing energy consumption and delay, provided that the task is successfully completed. According to the proposed task offloading mechanism, urgent tasks are directly offloaded to satellites for processing, where energy consumption and delay are disregarded, and only the task completion is considered. Moreover, for both urgent and normal tasks, successful offloading to the designated processing node is deemed sufficient for task completion. Therefore, the analysis focuses on the offloading success rate of all tasks. The offloading success rate $\delta$ can be expressed as follows:

$$\delta ={\displaystyle \frac{N^{su}}{N^{up}}},$$

(29)

Here, $N^{su}$ and $N^{up}$ represent the total number of successfully offloaded tasks and the total number of offloaded tasks, respectively. Based on the above, the task offloading optimization problem can be formulated as follows: under the constraint of ensuring a high completion rate for urgent task offloading, the goal is to improve the completion rate of normal tasks while minimizing their energy consumption and delay. The optimization problem is formulated as Problem P1 :

$$\begin{array}{cc}\hfill \left(P1\right):& \underset{\mu }{min}\left(E+T\right) andmax\left({\delta }^n\right)\hfill \\ \hfill s.t.& C1:B_{min}\le B\le B_{max}\hfill \\ \hfill & C2:\phi <R_{u,s}{T}_s\hfill \\ \hfill & C3:f_{min}\le f\le f_{max}\hfill \\ \hfill & C4:0\le {\mu }_{(u,u)}\le 1\hfill \\ \hfill & C5:0\le {\mu }_{(u,C)}\le 1\hfill \\ \hfill & C6:0\le {\mu }_{(u,s)}\le 1\hfill \\ \hfill & C7:{\mu }_{(u,u)}+{\mu }_{(u,C)}+{\mu }_{(u,s)}=1\hfill \\ \hfill & C8:{\delta }^{ur}=1\hfill \\ \hfill & C9:{\phi }_{min}^n\le {\phi }^n\le {\phi }_{max}^n\hfill \\ \hfill & C10:{\rho }_{min}^n\le {\rho }^n\le {\rho }_{max}^n\hfill \end{array}$$

(30)

where ${\phi }^n$ indicates the success rate of offloading common tasks. $C1$ denotes that the bandwidth resources of all the devices in the network are uncertain, but are within the limits. $C2$ denotes that the maximum amount of mission data from the UAV to the satellite should satisfy the reliability constraint. Constraint $C3$ specifies that the computational capacities of all computing devices in the network are uncertain but within a limited range. Constraints $C4$–$C6$ ensure that the offloading ratios in the offloading strategy remain within (0, 1), while $C7$ requires that the sum of each UAV’s offloading decisions equals 1, noting that UAVs may offload computation tasks to multiple satellites. Constraint $C8$ guarantees a 100% success rate for urgent task offloading, and Constraints $C9$–$C10$ ensure that the task size and complexity of normal tasks remain within their respective minimum and maximum bounds.

5. D-MAPPO Algorithm Design

To address the computation offloading optimization problem described above, we propose a reinforcement learning algorithm based on the Dirichlet probability distribution and Multi-Agent Proximal Policy Optimization (MAPPO). This algorithm demonstrates strong adaptability to continuous and dynamic environments, making it particularly well-suited for the SAGIN scenario considered in this study.

5.1. MDP Design

The Markov Decision Process (MDP) is a mathematical framework commonly used to model sequential decision-making problems and is widely applied in reinforcement learning (RL). An MDP characterizes the interaction between an agent and its environment through states, actions, state transitions, and a reward function, enabling the agent to learn an optimal decision-making policy [Reinforcement Learning: An Introduction]. In the context of computation offloading in space–air–ground integrated networks (SAGINs), the problem can be formulated as an MDP to optimize offloading strategies, thereby improving computational efficiency while reducing energy consumption and delay. In this study, we define the total operation time of the SAGIN system as T, which is further divided into t discrete time slots.

(1)

State Space: The state space defines all possible states of the environment and serves to describe the current condition of the system, thereby providing a basis for the agent’s decision-making. In the context of the computation offloading problem, the state space should adequately represent the availability of computational resources and the execution status of tasks, enabling the agent to select an appropriate offloading strategy accordingly. In this study, each UAV is modeled as an individual agent. The state $s_t$ of each agent at time slot t is defined as follows:

$$\begin{array}{c}\hfill s_t=\left(f_C,B_C,\psi ,f_s,B_s,{\theta }_{c,s},\dots ,f_S,\right.\left.B_S,{\theta }_{c,S},f_u,B_u,N^n,N^{ur},\rho ,\phi \right),\end{array}$$

(31)

where $f_C$ is the computational resource of the cloud server, $B_C$ is the bandwidth resource of the cloud server, and $\psi$ is the number of tasks that the cloud server can handle. $f_s$ is the computational resource of the satellite s, $B_s$ is the bandwidth resource of the satellite s, and ${\theta }_{(c,s)}$ is the coverage angle of the satellites, $s\in S$; note that there are multiple satellites in the network frame in the paper, and the status $s_t$ records the status of all satellites. $f_u$ is the computational resource of UAV u and $B_u$ is the bandwidth resource of UAV u. $N^n$ is the number of common tasks, $N^{ur}$ is the number of urgent tasks, $\rho$ is the computational complexity of the tasks, and $\phi$ is the task size of a single task.

(2)

Action Space: the action set defines the actions that an intelligent can perform in each state. In the computational offloading problem, the action determines the allocation ratio of tasks on different computational nodes. So, in this paper, the unloading ratio ${\mu }_{(u,u)},{\mu }_{(u,C)},{\mu }_{(u,s)},\dots ,{\mu }_{(u,S)}$ is taken as the action set, i.e., the action $a_t=({\mu }_{(u,u)},{\mu }_{(u,C)},{\mu }_{(u,s)},\dots ,{\mu }_{(u,S)})$ of each intelligent body in the time slot t, where the unloading ratio in the action set $a_t$ has to satisfy the constraint $d\sim g$ in the problem $P1$.

(3)

State Transition: State transition describes the process by which the system evolves from one state to another after an action is taken. In the computation offloading problem, state transitions depend on how offloading decisions affect resource availability and task status. In this study, state transitions are influenced by the following factors: Changes in the available computational resources of each device after task execution. The arrival of new tasks follows the completion of previous ones. The state transition process can be described by the probability model $\mathcal{P}\left(s_{t+1}\right|s_t,a_t)$, which captures the impact of offloading decisions on the future state of the system.

(4)

Reward Function: The reward function evaluates the quality of offloading decisions and serves as the foundation for policy optimization in reinforcement learning. The objective of this study is to ensure a high completion rate for urgent task offloading while improving the completion rate of normal tasks and minimizing their energy consumption and delay. Accordingly, the reward function $r{s}_t,a_t$ is defined as follows:

$$r\left(s_t,a_t\right)=\left\{\begin{array}{cc}-\left[\left(1-k\right)E+kT\right]\hfill & ,{\delta }^{ur}=1 and {\delta }^n=1\hfill \\ -\left[\left(1-k\right)E+kT\right]-\iota \left(1-{\delta }^n\right)N^n\hfill & ,{\delta }^{ur}=1 and {\delta }^n\ne 1\hfill \\ -\left[\left(1-k\right)E+kT\right]-\iota \left(1-{\delta }^n\right)N^n-\iota \left(1-{\delta }^{ur}\right)N^{ur}\hfill & ,{\delta }^{ur}\ne 1 and {\delta }^n\ne 1\hfill \end{array}\right.$$

(32)

Among them, k is the weight coefficient between energy consumption and delay, which can be adjusted according to different user requirements, $\iota$ indicates the penalty coefficient, which is penalized for every uninstalled task, and the total penalty will be reduced with the improvement of task uninstallation success rate, $\left(1-{\delta }^n\right)N^n$ indicates the number of uninstalled tasks in common tasks, and $\left(1-{\delta }^{ur}\right)N^{ur}$ indicates the number of uninstalled tasks in urgent tasks. In this paper, we set $k=0.5,\iota =7$.

(5)

Policy: denoted by $\pi$, is a static mapping from a state $s_t$ to an action $a_t$. In other words, when the system is in state $s_t$, it selects the corresponding action $a_t$ as prescribed by the policy.

Accordingly, the optimization problem of improving the success rate of general task offloading, while jointly minimizing latency and energy consumption based on an MDP framework, can be formulated as Problem $P2$:

$$\begin{array}{cc}\hfill \left(P2\right):& \underset{\pi }{min}\underset{T\to \infty }{lim}\mathbb{E}\left[{\displaystyle \frac{1}{T}}\sum _t^{T}r\left(s_t,a_t\right)\pi \right]\hfill \\ \hfill s.t.& C1\sim C1\hfill \end{array}$$

(33)

Here, $\left(33\right)$ denotes the expected average reward. The original Problem $P1$ is thereby transformed into Problem $P2$, which aims to identify an optimal policy $\pi$ associated with the reward $r(s_t,a_t)$ that chooses an action a in state s to minimize the expected average cost. This formulation of Problem $P2$ is a constrained Markov Decision Process (CMDP), representing a classic MDP with additional constraints. Solving such a CMDP in the presence of uncertainty is highly challenging.

On the one hand, standard MDPs have been extensively studied, and deterministic policies can be derived using iterative methods such as policy iteration or value iteration [43]. However, due to the simultaneous consideration of constraints and objectives, these methods are not applicable to CMDPs. On the other hand, while CMDPs with known transition probabilities can be solved via linear programming, this approach is not suitable for CMDPs with uncertainty, as the number of arriving tasks is not deterministic.

5.2. Dirichlet Probability Distribution and Representation of Offloading Decisions

In the computation offloading environment described in this paper, UAVs act as intelligent agents that allocate computation tasks among local processing, cloud servers, and satellites. The constraints on the action set $a_t$ are defined in Formula (30) and $C4$–$C7$, where each variable lies between 0 and 1, and their sum equals 1.

Since the action space is continuous and constrained by a simplex, traditional Gaussian or Beta distributions cannot satisfy these conditions. Using such distributions would require additional normalization or regularization, increasing computational complexity, wasting resources, and slowing optimization.

To address this, the Dirichlet distribution is adopted for policy modeling, as it naturally satisfies the above constraints. The Dirichlet distribution is defined as follows:

$$P(x_1,\dots ,x_J|{\alpha }_1,\dots ,{\alpha }_J)={\displaystyle \frac{1}{B\left(\alpha \right)}}\prod _{j=1}^J{x}_j^{{\alpha }_j-1}$$

(34)

where $x_J$ represents the offloading ratios satisfying ${\sum }_j^J{x}_j=1$, and ${\alpha }_J$ are the parameters of the Dirichlet distribution. $B\left(\alpha \right)$ denotes the normalization term of the Beta function. When all ${\alpha }_J$ are equal, the distribution is approximately uniform; when some ${\alpha }_J$ are larger, the distribution skews toward the corresponding dimensions.

In the Actor network, the state set $s_t$ is input into a multilayer perceptron (MLP) to generate the Dirichlet distribution parameters:

$${\alpha }_t=\mathrm{softplus}\left({\mathrm{MLP}}_{\theta }\left(s_t\right)\right)+ϵ$$

(35)

where $\mathrm{softplus}\left(x\right)=ln(1+e^x)$ ensures ${\alpha }_j>0$, and $ϵ>0$ is a small constant for numerical stability. The offloading ratios are obtained by sampling

$${\mu }_t\sim \mathrm{Dirichlet}\left({\alpha }_t\right).$$

(36)

5.3. MAPPO Algorithm Design

D-MAPPO is an extension of MAPPO [44] that incorporates a Dirichlet-based policy. It follows the centralized training and decentralized execution (CTDE) architecture, where each UAV maintains an independent Actor network for decision-making, while the Critic network shares parameters and uses global state information for value estimation.

In the multi-UAV computation offloading scenario considered in this paper, task offloading ratios must be dynamically allocated among local computation, cloud servers, and satellites. The decisions of different UAVs are interdependent, exhibiting strong multi-agent cooperation characteristics. To achieve efficient joint optimization, a multi-agent MAPPO framework is employed. This CTDE-based method enables global coordination while preserving distributed decision-making flexibility. Additionally, PPO’s clipping mechanism enhances policy convergence stability [45]. Combined with Dirichlet-based continuous action modeling, MAPPO effectively learns the optimal offloading ratio policy for UAVs, minimizing overall system latency and improving task success rates.

The key steps of the MAPPO algorithm are as follows:

(1)

Network Structure Design: Both the Actor and Critic networks adopt three fully connected layers with tanh activation functions to enhance nonlinearity and expressive capacity. The Actor network includes an additional fully connected layer to output the Dirichlet distribution parameters, providing a probabilistic representation for continuous action selection. The Critic network outputs a scalar value representing the state value. Both networks are trained using the Adam optimizer.

(2)

Advantage Estimation: Advantage estimation measures how much better a specific action is compared to the average. It is a critical component of MAPPO, helping reduce variance and improve learning efficiency. MAPPO employs Generalized Advantage Estimation (GAE), defined as follows:

$${\delta }_t=r_t+\gamma V_{\varphi }\left(s_{t+1}\right)-V_{\varphi }\left(s_t\right)$$

(37)

$${\widehat{A}}_t=\sum _{l=0}^{T-t-1}{\left(\gamma \lambda \right)}^l{\delta }_{t+1}$$

(38)

where ${\delta }_t$ is the Temporal Difference (TD) error, $r_t$ is the immediate reward after UAV action at time slot t, $V_{\varphi }\left(s_t\right)$ is the Critic’s estimated value of state $s_t$, $\varphi$ denotes Critic parameters, $\gamma$ is the discount factor, and $\lambda$ controls the bias–variance trade-off.

(3)

Policy Update: The log-probability of the Dirichlet policy is given by

$$log{\pi }_{\theta }\left({\mu }_t\right|s_t)=\sum _{j=1}^3({\alpha }_{t,j}-1)log{\mu }_{u,j}-logB\left({\alpha }_t\right)$$

(39)

where ${\pi }_{\theta }$ denotes the current policy. The PPO clipping loss is defined as follows:

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left({\mu }_t\right|s_t)}{{\pi }_{{\theta }_{old}}\left({\mu }_t\right|s_t)}}$$

(40)

$$L^{clip}\left(\theta \right)={\mathbb{E}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t\right)\right]$$

(41)

where $r_t\left(\theta \right)$ is the probability ratio used to control the update step size, ${\pi }_{{\theta }_{old}}$ represents the previous policy, and $clip\left(\right)$ limits update magnitude to prevent gradient explosion. $ϵ$ is a hyperparameter. Incorporating Dirichlet entropy regularization enhances exploration, and the Actor loss is defined as follows:

$$L^{\pi }\left(\theta \right)=-L^{clip}\left(\theta \right)+c_H{\mathbb{E}}_t\left[\mathcal{H}\left(\mathrm{Dir}\left({\alpha }_t\right)\right)\right]$$

(42)

where $c_H$ is the entropy coefficient encouraging exploration, and $\mathcal{H}(Dir\left({\alpha }_t\right)$ denotes the entropy of the Dirichlet distribution.

(4)

Value Function Update: The Critic network is optimized using the mean squared error (MSE) loss:

$$L^{VF}\left(\varphi \right)={\mathbb{E}}_t\left[{\left(V_{\varphi }\left(s_t\right)-R_t\right)}^2\right],R_t={\widehat{A}}_t+V_{\varphi }\left(s_t\right)$$

(43)

where $R_t$ is the immediate discounted return serving as the target for value function learning.

(5)

Joint Parameter Optimization: The total loss function is expressed as

$$L_{\mathrm{total}}=L^{\pi }\left(\theta \right)+c_V{L}^{VF}\left(\varphi \right)$$

(44)

where $c_V$ is the weighting coefficient for the value loss. The Actor and Critic parameters are updated separately using the Adam optimizer.

In this study, the D-MAPPO framework is adopted to optimize UAV offloading strategies. Specifically, each UAV’s offloading decision is represented as $a_t=({\mu }_{(u,u)},{\mu }_{(u,C)},{\mu }_{(u,s)},\dots ,{\mu }_{(u,S)})$, corresponding to the proportions of local computation, offloading to the cloud server, and offloading to multiple satellites. During training, the Actor network, based on the current observation $s_t$, outputs the Dirichlet parameters ${\alpha }_t$ to sample feasible offloading ratios ${\mu }_t$, ensuring actions always satisfy the constraints. The Critic network evaluates the global state value for advantage estimation and policy updating. Following the CTDE framework, agents interact with the environment, receive rewards and next states, compute advantages ${\widehat{A}}_t$ via GAE, update the Actor using PPO clipping loss, and refine the Critic with MSE loss. Through iterative training, the joint optimization of Actor and Critic parameters progressively enhances policy performance until convergence to a near-optimal offloading strategy capable of efficiently allocating computation tasks in dynamic multi-agent environments.

The pseudo-code of the MAPPO algorithm is shown in Algorithm 1:

Algorithm 1 Multi-Agent Proximal Policy Optimization (MAPPO)

1: Initialize policy network ${\pi }_{\theta }$ and value network $V_{\varphi }$ 2: Initialize replay buffer $\mathcal{D}$ 3: for episode $=1$ to Max_Episodes do 4:     Reset environment and get initial state $s_0$ 5:     for timestep $=1$ to Max_Timesteps do 6:         for each agent i do 7:            Get observation $o_i$ from state s 8:            Sample action $a_i\sim {\pi }_{\theta }\left(o_i\right)$ 9:         end for 10:         Execute joint action $(a_1,\dots ,a_N)$ in environment 11:         Receive next state $s^{\prime }$, rewards $(r_1,\dots ,r_N)$, and done flag 12:         Store $(s,o_1,a_1,r_1,\dots ,o_N,a_N,r_N,s^{\prime },\mathrm{done})$ in $\mathcal{D}$ 13:         Update state: $s\leftarrow s^{\prime }$ 14:     end for 15:     for each agent i do 16:         Compute TD-error ${\delta }_t=r_t+\gamma V_{\varphi }\left(o_i^{\prime }\right)-V_{\varphi }\left(o_i\right)$ 17:         Compute advantage estimate ${\widehat{A}}_t$ using GAE 18:     end for 19:     for each optimization step do 20:         Sample mini-batch from $\mathcal{D}$ 21:         Compute PPO policy loss: $$L^{\mathrm{clip}}\left(\theta \right)=\mathbb{E}[min(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t]$$ 22:         Compute value function loss: $$L^{\mathrm{VF}}\left(\varphi \right)={(V_{\varphi }\left(o_i\right)-R_t)}^2$$ 23:         Update $\theta$ using gradient ascent on $L^{\mathrm{clip}}\left(\theta \right)$ 24:         Update $\varphi$ using gradient descent on $L^{\mathrm{VF}}\left(\varphi \right)$ 25:     end for 26:     Clear replay buffer $\mathcal{D}$ 27: end for

6. Experiment Analysis

This subsection presents the results of our simulation experiments. We begin by detailing the parameter settings used in the experiments. Then, we compare the performance of the D-MAPPO algorithm under different hyperparameter configurations to select the optimal ones. Finally, we compare our proposed algorithm with several baseline algorithms to demonstrate its superiority.

6.1. Parameter Settings

In the simulations, we implemented the simulation environment and optimization algorithms using Python 3.8.10 and PyTorch 1.9.1. The environment was configured with (U = 5) UAVs, (S = 3) low Earth orbit (LEO) satellites, and (C = 1) cloud server. The total number of normal tasks was approximately 236, and the total number of urgent tasks was approximately 36, accounting for about 13% of all tasks. Unless otherwise noted, all parameter values are taken from the references [41,46,47], and the parameter settings are listed in

Table 2. Parameter Settings.

Parameter	Value	Parameter	Value
$P_i$	$0.1\mathrm{W}$	${\theta }_G$	${40}^{\circ }$
$P_u,P_c,P_s$	$1\mathrm{W}$	$V_s$	$7.8\mathrm{km}/\mathrm{s}$
$B_s,B_c$	≈$10\mathrm{MHz}$	$\tau$	${10}^{-25}$
$B_u$	≈$5\mathrm{MHz}$	${\phi }_m$	$[0.6,1.2]\mathrm{M}$
$B_i$	≈$1\mathrm{MHz}$	${\rho }_m$	≈$500\mathrm{cycles}/\mathrm{bit}$
$f_u$	≈$0.5\mathrm{GHz}$	$f_s,f_C$	≈$1\mathrm{GHz}$

.

Table 3. Hyperparameter Settings of D-MAPPO.

Parameter	Value	Parameter	Value
Number of agents	5	Actor network dim	(128, 128, 64, 5)
entropy coefficient	0.01	Critic network dim	(128, 128, 64, 1)
Clipping parameter	0.2	Discount factor	0.95
GAE parameter	0.95	batch size	128
rollout length	75	number of training epochs per update	128

lists the hyperparameter values used for the D-MAPPO algorithm.

Based on the above parameter settings, we optimized the task offloading strategies using our proposed algorithm along with five baseline algorithms under identical environmental conditions. The six algorithms compared are as follows:

Beta-MAPPO [48]: A MAPPO algorithm that models the action space using a Beta distribution. It requires training to adapt to the continuous action space constraints in this environment and is suitable for multi-agent offloading problems.

PPO [49]: A single-agent reinforcement learning algorithm. Given the 20-dimensional joint action space of the five UAVs, Dirichlet or Beta distributions are not applicable; hence, a discrete action space is used.

Local: Each UAV processes its collected tasks locally without offloading.

Offloading: All tasks are offloaded to cloud servers and satellites for execution.

Random: Actions are selected completely at random, without considering environmental states or policies, serving as a lower-bound baseline.

6.2. D-MAPPO Algorithm Convergence

To verify the convergence of the algorithm and to select optimal hyperparameters for achieving the best performance of the D-MAPPO algorithm, we first compare the convergence behavior under different hyperparameter settings, including the learning rate (lr), discount factor (gamma), and the number of policy update epochs (n_epochs). The comparison results are shown in Figure 3.

Figure 3 shows the performance of the D-MAPPO algorithm during training under different hyperparameter settings. In Figure 3a, with a learning rate of 0.0003 (red line), the algorithm converges the fastest, stabilizing and reaching the highest average reward within about 200 episodes, demonstrating the best learning efficiency. With a learning rate of 0.003 (blue line), convergence is slower, but the algorithm still achieves high performance after 400 episodes and maintains good stability. When the learning rate is set to 0.00003 (green line), the algorithm exhibits significant fluctuations, with the average reward significantly lower than the first two cases, and it fails to effectively converge. This indicates that a too-small learning rate limits policy updates, negatively affecting learning performance.

Figure 3b shows the results for different discount factors: 0.8, 0.95, and 0.99. With a discount factor of 0.95 (red line), the algorithm converges quickly and stabilizes at a high reward, indicating that long-term rewards are beneficial for policy learning. With a discount factor of 0.99 (blue line), although the optimization performance is poor before 700 episodes, there is an improvement later, though it still does not outperform the 0.95 case. With a discount factor of 0.8 (green line), the algorithm converges faster initially, but the rewards decrease later, suggesting that short-sighted reward estimates are detrimental to long-term optimization.

Figure 3c illustrates the effects of different numbers of policy update epochs (5, 7, and 10). When $n_{epochsis}$ is set to 5 or 7 (red and green lines), the algorithm performs steadily and converges quickly, avoiding overfitting. When $n_{epochs}$ is set to 10 (blue line), the algorithm performs well initially, but the rewards decrease later, suggesting that excessive updates may lead to overfitting and reduce generalization performance.

Based on the comparison of the three hyperparameters, we find that the D-MAPPO algorithm achieves better optimization performance when lr = 0.0003, gamma = 0.95, and n_epochs = 5. Next, we will compare the above values with other benchmark algorithms.

After selecting appropriate hyperparameters, we compare the optimization performance of the proposed algorithm with five other baseline algorithms. During the optimization process, tasks that are not successfully offloaded are regarded as timeout tasks, and fixed delay and energy consumption values are assigned to these timeout cases.

6.3. Comparison of Reward Values Among Different Algorithms

As shown in Figure 4, the figure presents the training performance of six algorithms under the same environment: D-MAPPO (red line), Beta-MAPPO (green line), PPO (blue line), Local (black line), Offloading (orange line), and Random (sky blue line). The horizontal axis represents training episodes, and the vertical axis denotes average reward. D-MAPPO performs the best, with the average reward rising rapidly in a short time and stabilizing within approximately 200 episodes. The final converged reward is −73, significantly higher than the other algorithms, indicating strong convergence speed and policy quality. This demonstrates its notable advantages in handling dynamic action spaces and multi-agent coordination. Beta-MAPPO shows very low rewards in the initial stages, even below the Random strategy, indicating significant instability. However, it rebounds after about 150 episodes and continues to improve, eventually stabilizing at −105. Its performance surpasses PPO and Local but remains inferior to D-MAPPO. This “lagging” behavior stems from the unstable convergence of action sampling using the Beta distribution. Although its long-term performance is acceptable, its training efficiency is lower than that of D-MAPPO. The PPO algorithm shows a relatively smooth convergence process and eventually converges around −90. While it performs better than the Local, Offloading, Random, and Beta-MAPPO strategies, it is clearly outperformed by D-MAPPO. This suggests that single-agent strategies struggle to capture coordination in multi-agent environments, limiting adaptability. The Local strategy stabilizes around −126. Since all normal tasks are computed locally without involving offloading, no penalty is incurred. In contrast, the Offloading strategy offloads all tasks. Given the limited capacity of the cloud server and satellite, many tasks cannot be successfully offloaded, resulting in accumulated penalties and consistently low reward values around −500. The reward values of the Random strategy fluctuate greatly due to the highly stochastic nature of its offloading decisions, resulting in large performance variations.

6.4. Delay Optimization Comparison

Figure 5 compares the ordinary task delay optimization performance of the local, offloading (full), and random algorithms with D-MAPPO and Beta-MAPPO. D-MAPPO shows remarkable delay reduction from the early training stages, stably converging to the minimum delay (<0.6 s) after about 200 rounds, surpassing other algorithms. Beta-MAPPO, despite high initial delays, rapidly optimizes to stabilize between 0.8 s and 0.9 s, underperforming compared to D-MAPPO and PPO. PPO converges more slowly, with a final delay of approximately 0.95 s, second only to D-MAPPO. The local and random strategies stabilize at around 1.0 s and 1.1 s, respectively, highlighting the limitations of lacking dynamic scheduling capabilities. The offloading strategy remains at high delay (>3 s) without significant optimization. In summary, D-MAPPO demonstrates exceptional learning ability and decision-making efficiency in delay optimization, confirming its superiority in complex multi-agent offloading scenarios.

Figure 6 presents a performance comparison of the D-MAPPO algorithm with Beta-MAPPO, PPO, local, offloading, and random algorithms in terms of the minimum delay for normal tasks and the relative minimum delay change rate compared to the local algorithm. The analysis shows that D-MAPPO achieves the best performance in minimizing delay, with a value of 0.52 s, representing improvements of 38% and 31% over MAPPO and PPO, 47% over the local algorithm, and 50% over the fully random algorithm. Regarding the minimum delay change rate, the offloading algorithm exhibits the greatest change relative to the local algorithm, approximately 1.25, while D-MAPPO shows the least change at −0.47, demonstrating its significant advantage in delay optimization.

6.5. Energy Consumption Optimization Comparison

Figure 7 compares the performance of several algorithms in optimizing energy consumption for normal tasks. The D-MAPPO algorithm shows a rapid drop in energy consumption during the early training phase and stabilizes after approximately 100 episodes, eventually reaching around 0.07 KJ, achieving the best optimization performance. The energy consumption of the Beta-MAPPO algorithm decreases at a slower pace, stabilizing around 0.072 KJ after about 200 episodes. The PPO algorithm’s energy consumption gradually drops from 0.7 KJ to about 0.2 KJ, with considerable fluctuations, indicating unstable optimization performance. The energy consumption of the Local algorithm consistently remains at 0.025 KJ. This is because wireless data transmission requires significant transmission power, especially for long distances such as satellite communication. The Local algorithm avoids such communication, resulting in the lowest and most stable energy consumption, though at the cost of flexibility. The Offloading algorithm consumes as much as 2.7 KJ and shows no change throughout the training, indicating the worst performance. The Random algorithm exhibits significant fluctuations in energy consumption, but it remains generally stable between 0.5 and 0.6 KJ. Overall, both D-MAPPO and Beta-MAPPO have clear advantages in energy optimization and effectively reduce energy consumption, with D-MAPPO outperforming Beta-MAPPO in terms of optimization performance.

Figure 8 illustrates, in a bar-line chart, the comparison of different algorithms in terms of minimum energy consumption for normal tasks and the relative minimum energy change rate compared to the local algorithm. The x-axis represents the algorithm, the left y-axis indicates average energy consumption (KJ), and the right y-axis indicates the minimum energy change rate. D-MAPPO achieves an energy consumption of 0.072 KJ with a change rate close to 1.68, performing the best. Beta-MAPPO and PPO show slightly higher energy consumption and higher change rates of 1.87 and 4.14, respectively. The local algorithm consumes approximately 0.025 KJ, as it does not perform task offloading. The offloading algorithm has the highest energy consumption at 1.83 KJ and a change rate of 71.8%, performing the worst. Random achieves 0.48 KJ with a change rate of 18.27. Overall, D-MAPPO demonstrates the lowest and most stable energy consumption, with improvements of 7% and 48% over MAPPO and PPO, respectively.

6.6. Optimization Comparison of Task Offloading Success Rate

Figure 9 compares the optimization performance of D-MAPPO and five other algorithms in terms of the task offloading success rate for normal tasks. The D-MAPPO algorithm rapidly improves the offloading success rate in the early training phase, reaching 97% after around 100 episodes and continuing to rise slowly until it stabilizes at 100%, demonstrating excellent convergence and efficiency. The Beta-MAPPO algorithm shows a low success rate at the beginning of training but rises significantly after about 200 episodes, eventually stabilizing between 98% and 99%, performing slightly worse than D-MAPPO. The PPO algorithm’s offloading success rate increases slowly from 85–90% to 93–95%, reflecting a moderate level of performance. The Local algorithm consistently maintains a 100% success rate, representing an extreme strategy that always processes tasks locally. The Random algorithm’s success rate fluctuates between 90% and 93%, showing unstable performance. Due to the limited capacity of cloud servers and satellites to receive tasks, the Offloading algorithm exhibits very poor success rates. Overall, the D-MAPPO algorithm achieves the best performance in optimizing task offloading success rates, followed by Beta-MAPPO, while PPO and Random perform relatively worse. The Local and Offloading strategies lack flexibility. This indicates that reinforcement learning-based algorithms such as D-MAPPO and Beta-MAPPO have significant advantages in optimizing task offloading success rates and can effectively enhance system performance.

Figure 10 compares the offloading success rates for urgent tasks, normal tasks, and all tasks under each algorithm at their respective peak reward performance. The figure shows that the D-MAPPO algorithm achieves a 100% offloading success rate across all task types, indicating its ability to efficiently handle offloading under varying task conditions. Beta-MAPPO and Local algorithms also achieve 100% success rates for both urgent and normal tasks. The PPO algorithm performs slightly worse but still maintains a relatively high success rate. In contrast, the Offloading algorithm yields significantly lower success rates across all task types, ranging from approximately 0.7 to 0.75. The Random algorithm achieves a more consistent success rate of around 0.85, but still underperforms compared to reinforcement learning-based algorithms. This highlights the significant advantage of the D-MAPPO algorithm in handling various task types and effectively improving offloading success rates.

7. Conclusions

This paper aims to enhance the quality of service in network communications for environmental monitoring in remote areas by proposing a network framework based on SAGIN. A systematic introduction of the framework is provided, along with precise modeling of the communication model, task model, satellite coverage model, computation model, and reliability model. To handle different types of computational tasks, they are categorized into urgent and normal tasks. Urgent tasks are prioritized for offloading, with the success rate as the sole optimization objective, while normal tasks are optimized based on a combination of success rate, delay, and energy consumption. Based on the above models, an optimization problem is formulated to ensure the offloading success rate of urgent tasks while improving the success rate and reducing the delay and energy consumption of normal tasks. To solve the optimization problem, this paper proposes the D-MAPPO algorithm, which integrates the Dirichlet distribution with the MAPPO method. The feasibility of the proposed algorithm is validated through simulation experiments, and comparative results with benchmark algorithms further demonstrate the superiority of D-MAPPO in optimization performance.