Learning-Based Collaborative Computation Offloading in UAV-Assisted Multi-Access Edge Computing

Abstract

Unmanned aerial vehicles (UAVs) have gained considerable attention in the research community due to their exceptional agility, maneuverability, and potential applications in fields like surveillance, multi-access edge computing (MEC), and various other domains. However, efficiently providing computation offloading services for concurrent Internet of Things devices (IOTDs) remains a significant challenge for UAVs due to their limited computing and communication capabilities. Consequently, optimizing and managing the constrained computing, communication, and energy resources of UAVs are essential for establishing an efficient aerial network infrastructure. To address this challenge, we investigate the collaborative computation offloading optimization problem in a UAV-assisted MEC environment comprising multiple UAVs and multiple IODTs. Our primary objective is to obtain efficient offloading strategies within a multi-heterogeneous UAV environment characterized by limited computing and communication capabilities. In this context, we model the problem as a multi-agent markov decision process (MAMDP) to account for environmental dynamics. We employ a multi-agent deep deterministic policy gradient (MADDPG) approach for task offloading. Subsequently, we conduct simulations to evaluate the efficiency of our proposed offloading scheme. The results highlight significant improvements achieved by the proposed offloading strategy, including a notable increase in the system completion rate and a significant reduction in the average energy consumption of the system.

1. Introduction

A variety of computationally and communication-intensive applications, such as virtual reality, facial recognition, and autonomous driving, are anticipated to operate on resource-constrained Internet of Things devices (IOTDs) [1,2]. Simultaneously, IOTDs introduce a spectrum of requirements encompassing diverse service quality levels and computational resources. Nevertheless, the limited computational resources pose a significant impediment to the widespread adoption of these innovative applications. Multi-access edge computing (MEC) emerges as an effective solution to tackle this challenge. By offloading computational tasks from IOTDs with restricted computing capabilities and finite battery power to edge servers located in close proximity, MEC has the potential to significantly reduce both latency and energy consumption.

However, deploying edge servers in remote or underdeveloped terrestrial areas presents significant challenges. For instance, establishing edge servers in sparsely populated regions like isolated mountains and islands may not be a cost-effective proposition. In contrast, the UAV-assisted MEC environment offers a promising solution to meet the needs of underserved regions where terrestrial base stations are either absent or insufficient [3,4].

The UAV-assisted MEC environment harbors substantial potential in future wireless systems, especially for maritime applications and emergency scenarios. This technology can extend coverage to IOTDs in remote locations, hotspots, and emergency zones that may have limited access to terrestrial base stations or are underserved by conventional means [5,6,7]. Researchers are increasingly drawn to the UAV-assisted MEC environment due to its adaptability and cost-effectiveness. A significant challenge in deploying the UAV-assisted MEC environment lies in efficiently offloading a variety of tasks from diverse IOTDs. This offloading should be based on considerations such as the computational resources and energy consumption of the UAVs.

Numerous researchers have made significant contributions to the field of UAV-assisted MEC environments, addressing various aspects such as energy efficiency [8], trajectory planning [9], and the joint optimization of communication and computation [10]. However, these existing research findings come with certain limitations. Firstly, a majority of current scenarios center around a single UAV, necessitating the formation of UAV swarms to distribute the computational workload due to the constrained processing capacity of individual UAVs. Secondly, many approaches tackle computation offloading problems using traditional optimization techniques, often as one-time optimizations that consider a specific system state. Given the dynamic nature of UAV-assisted MEC environments, effectively applying these methods proves to be challenging.

In response to this challenge, this paper conducts a comprehensive investigation into the optimization problem of cooperative computing offloading strategies within UAV-assisted MEC environments. The principal contributions of this study can be succinctly summarized as follows:

•

We investigate the problem of collaborative computation offloading involving multiple UAVs under the constraints of limited communication range and latency. Our goal is to collectively minimize the average energy consumption while increasing the overall efficiency.

•

We transform the optimization problem of cooperative computation offloading into a multi-agent markov decision process (MAMDP). In this framework, each UAV serves as an individual agent, and these agents make decisions by considering local real-time observations and the current policy. These decisions involve selecting the task offloading target and allocation ratio.

•

We design an algorithm based on multi-agent deep deterministic policy gradient (MADDPG). With offline training, each UAV can make real-time offloading decisions. We conduct a series of simulation experiments to prove the effectiveness of the algorithm in dynamic environments.

This paper employs a multi-agent reinforcement learning approach to address the optimization of cooperative user computation offloading policies in a UAV-assisted MEC environment characterized by constrained communication ranges and computing resources. The remainder of the paper is arranged as follows. Section 2 reviews current literature relevant to our study. Section 3 introduces the mathematical model for the considered UAV-assisted MEC environment. Section 4 presents the solution based on the MADDPG algorithm. Section 5 conducts a series of simulation experiments to demonstrate the effectiveness of our proposed algorithm. Section 6 provides a summary of this paper.

2. Related Work

The computation offloading problem has consistently been a research focal point in the context of UAV-assisted MEC environments, with numerous research contributions of significant importance. To this end, we have conducted a comprehensive review of existing research and categorized it into three key domains: optimization-based offloading methods, game-theory-based offloading methods, and reinforcement-learning-based offloading methods.

Optimization-based offloading methods. Optimization-based offloading methods have found widespread application in task offloading within the UAV-assisted MEC environment. These algorithms primarily focus on single or multi-objective optimization to enhance performance. Diao et al. [11] introduced an alternating optimization algorithm that combines a greedy approach, successive convex approximation, and functional analysis methods. They applied this algorithm to jointly optimize offloading parameters and UAV trajectories with the goal of minimizing the average peak age of information, average energy consumption of IOTDs, and average energy consumption of UAVs. Lv et al. [12] employed Bayesian optimization to ascertain the optimal offloading decision, aiming to minimize overall energy consumption while maintaining adherence to mission quality of service requirements. Chen et al. [13] devised a parallel offloading strategy using Liapunov optimization. This strategy efficiently identifies optimal decisions for latency-sensitive and computation-intensive tasks, achieving joint optimization encompassing green energy utilization, task division factors, CPU frequencies, and transmission power. Hu et al. [14] introduced a three-step block coordinate descent algorithm. This algorithm concurrently optimizes the reflection coefficient of reconfigurable intelligent surfaces, the received beam formation vector at the access point, and the local energy allocation strategy for user equipment. The aim is to maximize the total completed task input bits for all users. Guo et al. [15] employed a Lyapunov-based approach to optimize multi-user partial computational offloading. Their goal was to minimize the energy consumption of all users while ensuring compliance with time delay constraints. Luo et al. [16] introduced a particle swarm optimization (PSO) algorithm for the joint optimization of offloading decisions, communication allocation, and computational resources. Their objective was to minimize both latency and cost. Abbas et al. [17] introduced a real-time computing task offloading and resource allocation strategy with a user-centric approach. They employed the Lyapunov drift plus penalty optimization method to create a dynamic partial offloading solution aimed at minimizing energy consumption and monetary costs while maximizing task completion rates. Sheng et al. [18] employed a joint optimization approach for offloading decisions and resource allocation, utilizing a convex-concave iterative process. This method effectively mitigates co-channel interference while addressing differentiated upload delays among users.

Game theory-based offloading methods. Game theory-based offloading methods have been successfully applied to formulate, design, or optimize operations in various representative environments [19]. Zhou et al. [20] introduced a novel optimization model designed to maximize the expected offloading rate of multiple agents. They achieved this by optimizing the offloading threshold and then transformed the problem into a game-theoretic framework. Their analysis included an examination of the existence of Nash equilibrium strategies within this game-theoretic model. Pham et al. [21] formulated the offloading decision problem as a potential game and devised a distributed offloading scheme with low complexity using a game-theoretic approach. Xiao et al. [22] addressed MEC grouping for task offloading in MEC cooperation by framing it as a utility maximization problem. They accomplished this by creating a non-cooperative game strategy choice using regret matching. Chen et al. [23] framed the multiuser task offloading problem as a game and derived a Nash equilibrium strategy for task offloading. Teng et al. [24] converted the multi-server multi-task allocation and scheduling problem into a non-cooperative game. They demonstrated the existence of a Nash equilibrium and introduced a low-complexity response updating algorithm that converges to the Nash equilibrium. Fang et al. [25] framed the problem of minimizing the overall latency for all users in the network as a matching game considering heterogeneous demands. They employed a multi-round cooperative matching algorithm to discover a stable match for this game. You et al. [26] introduced a novel game-theoretic algorithm for multilayer computational and network resource allocation in the context of computational offloading within the MEC environment. Zhang et al. [27] introduced a game-theory-based approach to determine the optimal offloading strategy aimed at minimizing the weighted cost associated with time delay and energy consumption.

Reinforcement-learning-based offloading methods. Reinforcement learning methods have garnered substantial attention for resolving intricate decision problems in dynamic and uncertain environments [28]. Wang et al. [29] introduced a task offloading method based on meta-reinforcement learning. This approach facilitates rapid and adaptive task offloading decisions in MEC environments by learning the initial parameters of the task offloading policy. Liu et al. [30] presented a deep reinforcement learning-based approach for offloading and resource allocation to enhance system performance and efficiency. Zhou et al. [31] proposed a combined strategy for computational offloading and resource allocation utilizing deep reinforcement learning to decrease the energy consumption of the entire MEC system. Tang et al. [32] introduced a distributed algorithm, leveraging model-free deep reinforcement learning, to optimize offloading decisions with the goal of minimizing expected long-term costs. Dai et al. [33] presented an asynchronous actor–critic algorithm aimed at identifying the optimal stochastic computational offloading strategy, with the objective of minimizing long-term energy efficiency in industrial networks. Gao et al. [34] proposed a framework based on the proximal policy optimization algorithm. This framework addresses location privacy protection and facilitates effective task offloading decisions, achieving a balance between computational task performance requirements and user location privacy preservation. Wang et al. [35] achieved efficient computational offloading by employing intelligent decision-making strategies within a reinforcement learning framework. This approach is aimed at reducing the energy consumption and latency of mobile devices. Lu et al. [36] introduced a deep reinforcement learning-driven computation offloading approach that efficiently transfers computational tasks to edge servers. This approach relies on intelligent decision-making strategies to enhance computational performance and user experience.

To clearly distinguish our research and underscore its unique characteristics, we conducted a comparative analysis between our study and the previously mentioned works.

•

First, we address the problem of optimizing cooperative offloading involving multiple UAVs. Considering the constraints of limited computing resource and communication range, our primary goal is to minimize the average energy consumption of these UAVs while ensuring latency constraints are met.

•

Second, the predominant approaches in the current landscape heavily rely on game-theory-based methods and optimization-based techniques, which are deterministic optimization methods well-suited for static environments. However, their adaptability diminishes when faced with dynamic environmental changes and the necessity for continuous offloading. To address this dynamism and the continuous nature of offloading decisions, we incorporate deep reinforcement learning methods.

•

Third, the majority of existing research predominantly relies on single-agent reinforcement learning algorithms, which makes it challenging to address cooperative issues involving multiple agents. To overcome this limitation, we enhance our approach by adopting the MADDPG framework, which transforms the multi-user cooperative offloading game into a multi-agent model.

3. Models and Problem Definition

In this section, we first formulate mathematical models to describe the UAV-assisted MEC environment. We then provide a formal definition of the optimization problem under consideration. The mathematical notations used in this paper are summarized in Abbreviations, where the symbols are listed in the order introduced in the paper.

3.1. The UAV-Assisted MEC Environment

In this paper, we consider a computing environment comprising n UAVs (denoted as ${\mathrm{UAV}}_1,{\mathrm{UAV}}_2,\dots ,{\mathrm{UAV}}_n$) in addition to multiple IOTDs. Each UAV is equipped with limited computational and communication capabilities, serving as a task offloading provider for the IOTDs  [8,37], as illustrated in Figure 1.

Figure 1. A UAV-assisted MEC environment with multiple UAVs and multiple IOTDs.

We assume that each UAV adheres to a predetermined flight path while operating independently. Positioned at fixed coordinates, a UAV can provide computation offloading services to IOTDs within its coverage area. Periodically, UAVs receive computationally intensive tasks of varying sizes from IOTDs. To ensure compliance with the latency constraint due to limited computational and communication capabilities, each UAV offloads tasks to other UAVs within its communication range. The main objective of this work is to minimize the average energy consumption while maintaining the average response latency constraint.

In this framework, we discretize the runtime into discrete time slots $t\in \left\{1,2,3,\dots ,T\right\}$, where T represents the finite time horizon. UAVs receive computationally intensive tasks from IOTDs at the initiation of each time slot. The UAVs tailor their offloading schemes based on the state information of other UAVs observed in the environment. ${\mathrm{UAV}}_i$ gauges the communication circumstances with other UAVs in the prevailing environment through multi-sensor information fusion techniques, denoted as $U_i=({\mathrm{UAV}}_i^1,{\mathrm{UAV}}_i^2,\dots ,{\mathrm{UAV}}_i^n)$. ${\mathrm{UAV}}_i^j=1$ signifies that ${\mathrm{UAV}}_j$ falls within the communication range of ${\mathrm{UAV}}_i$; conversely, ${\mathrm{UAV}}_i^j=0$, with $1\le j\ne i\le n$ indicates the opposite. Considering the energy constraints of UAVs, it is essential that the energy consumed for computation and communication remains within the limits of the UAV’s battery power. Addressing this, potential solutions, such as current charging methods [38,39], could prevent battery depletion.

3.2. Offloading Schemes of UAVs

Typically, in UAV-assisted MEC environments, tasks can be executed locally or offloaded to a ground base station or a cloud center for processing. However, we introduce an innovative concept where tasks can also be offloaded to nearby UAVs for collaborative computation. The aim of this innovative approach is to maximize computational resource utilization across multiple UAVs, considering the limitations imposed by UAVs’ limited communication range, with the objective of improving task execution efficiency and performance.

Let $d_i^t$ (measured in units of bits, $1\le i\le n$) denote the size of the offloaded computational tasks received by ${\mathrm{UAV}}_i$ from IOTDs at the beginning of the time slot t. To ensure the completion of these computational tasks within the predefined latency constraints, each UAV is required to formulate its own offloading strategy based on the information it gathers. That is, each UAV must determine the percentage of computational tasks to be offloaded to other UAVs within its communication range.

We use vector $\left({\lambda }_{i,1}^t,{\lambda }_{i,2}^t,\dots ,{\lambda }_{i,n}^t\right)$ to denote the offloading scheme of ${\mathrm{UAV}}_i$ at time slot t. In this representation, ${\lambda }_{i,i}^t$ signifies the percentage of computational tasks that ${\mathrm{UAV}}_i$ handles locally, while ${\lambda }_{i,j}^t$ ($i\ne j$) denotes the percentage of computational tasks that ${\mathrm{UAV}}_i$ offloads to $\mathrm{UA}{\mathrm{V}}_j$ for collaborative execution. Then, we have ${\lambda }_{i,1}^t+{\lambda }_{i,2}^t+\cdots +{\lambda }_{i,n}^t=1$. Note that the UAVs that are not in the communication range of ${\mathrm{UAV}}_i$ do not participate in the offloading decision process, i.e., if ${\mathrm{UAV}}_i^j=0$, then ${\lambda }_{i,j}^t=0$.

Let us consider that the number of CPU cycles needed to execute 1-bit data on ${\mathrm{UAV}}_i$ is denoted as $c_i$ (measured in cycles/bit). Since the size of data not offloaded from ${\mathrm{UAV}}_i$ to other UAVs for processing at time slot t is ${\lambda }_{i,i}^t{d}_i^t$ bits, the number of CPU cycles required to process this data is given by

$$\begin{array}{c}\hfill C_i^{loc,t}=c_i{\lambda }_{i,i}^t{d}_i^t.\end{array}$$

(1)

Furthermore, ${\mathrm{UAV}}_i$ also receives computational tasks from other UAVs at time slot t, resulting in a total data size of the received computational tasks of ${\sum }_{j=1,j\ne i}^n{\lambda }_{j,i}^t{d}_j^t$ bits. Consequently, the CPU cycles required to process these computational tasks from other UAVs on ${\mathrm{UAV}}_i$ can be expressed as

$$\begin{array}{c}\hfill C_i^{acc,t}={\displaystyle \sum _{j=1,j\ne i}^n}{c}_i·{\lambda }_{j,i}^t{d}_j^t.\end{array}$$

(2)

Therefore, the total number of CPU cycles required for all computational tasks to be executed by ${\mathrm{UAV}}_i$ at time slot t can be expressed as

$$\begin{array}{c}\hfill C_i^t=C_i^{loc,t}+C_i^{acc,t}=c_i{\lambda }_{i,i}^t{d}_i^t+{\displaystyle \sum _{j=1,j\ne i}^n}{c}_i·{\lambda }_{j,i}^t{d}_j^t=c_i\left({\lambda }_{i,i}^t{d}_i^t+{\displaystyle \sum _{j=1,j\ne i}^n}{\lambda }_{j,i}^t{d}_j^t\right).\end{array}$$

(3)

3.3. Energy Consumption Models

In this section, we discuss the energy consumption models of UAVs, which include computational energy consumption and communication energy consumption.

3.3.1. Computational Energy Consumption

The computational energy consumption of ${\mathrm{UAV}}_i$ refers to the energy consumption generated by executing all computational tasks on ${\mathrm{UAV}}_i$.

In general, the computational power consumption of ${\mathrm{UAV}}_i$ (measured in watts) can be expressed as $P_i^{comp}=k·f_i^3$, where k is a constant factor related to the CPU chip architecture, and $f_i$ (measured in cycles/second) is the execution speed of ${\mathrm{UAV}}_i$ [40]. Based on the previous description, it can be observed that the overall time required to perform all computational tasks on ${\mathrm{UAV}}_i$ during time slot t is

$$\begin{array}{c}\hfill T_i^{comp,t}=\frac{C_i^t}{f_i}=\frac{c_i\left({\lambda }_{i,i}^t{d}_i^t+{\displaystyle \sum _{j=1,j\ne i}^n}{\lambda }_{j,i}^t{d}_j^t\right)}{f_i}.\end{array}$$

(4)

Then, the computational energy consumption of ${\mathrm{UAV}}_i$ during time slot t can be expressed as

$$\begin{array}{c}\hfill E_i^{comp,t}=P_i^{comp}·T_i^{comp,t}=k{c}_i{f_i}^2\left({\lambda }_{i,i}^t{d}_i^t+{\displaystyle \sum _{j=1,j\ne i}^n}{\lambda }_{j,i}^t{d}_j^t\right).\end{array}$$

(5)

3.3.2. Communication Energy Consumption

The communication energy consumption of ${\mathrm{UAV}}_i$ refers to the energy consumption generated by ${\mathrm{UAV}}_i$ during the process of offloading computational tasks to other UAVs.

In a general context, UAVs establish communication with each other using wireless channels within the airspace. According to Shannon’s theorem [41], the data transmission rate $R_{i,j}$ (measured in Mbps) for offloading tasks from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$ can be calculated as

$$\begin{array}{c}\hfill R_{i,j}=B_{i,j}{log}_2(1+\frac{q_{i,j}{P}_{i,j}^{comm}}{B_{i,j}{N}_{i,j}}).\end{array}$$

(6)

Here, $B_{i,j}$ represents the channel bandwidth from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$ (measured in MHz), $P_{i,j}^{comm}$ stands for the transmission power from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$, $q_{i,j}$ indicates the channel gain from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$ (measured in dBm), and $N_{i,j}$ signifies the noise power spectral density from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$ (measured in dBm/Hz).

The equation can be rearranged to determine $P_{i,j}^{comm}$ as follows:

$$\begin{array}{c}\hfill P_{i,j}^{comm}=\frac{B_{i,j}{N}_{i,j}(2^{R_{i,j}/B_{i,j}}-1)}{q_{i,j}}.\end{array}$$

(7)

It can be observed that the transmission time required of ${\mathrm{UAV}}_i$ to offload computational tasks to ${\mathrm{UAV}}_j$ during time slot t is

$$\begin{array}{c}\hfill T_{i,j}^{comm,t}=\frac{{\lambda }_{i,j}^t{d}_i^t}{R_{i,j}·{10}^6}.\end{array}$$

(8)

Hence, the communication energy consumption resulting from offloading computational tasks from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$ during time slot t can be expressed as

$$\begin{array}{c}\hfill E_{i,j}^{comm,t}=P_{i,j}^{comm}·T_{i,j}^{comm,t}=\frac{{\lambda }_{i,j}^t{d}_i^t}{R_{i,j}·{10}^6}·\frac{B_{i,j}{N}_{i,j}\left(2^{R_{i,j}/B_{i,j}}-1\right)}{q_{i,j}}.\end{array}$$

(9)

Then, we can represent the communication energy consumption of ${\mathrm{UAV}}_i$ during time slot t as

$$\begin{array}{c}\hfill E_i^{comm,t}={\displaystyle \sum _{j=1,j\ne i}^n}{E}_{i,j}^{comm,t}={\displaystyle \sum _{j=1,j\ne i}^n}\left(\frac{{\lambda }_{i,j}^t{d}_i^t}{R_{i,j}·{10}^6}·\frac{B_{i,j}{N}_{i,j}\left(2^{R_{i,j}/B_{i,j}}-1\right)}{q_{i,j}}\right).\end{array}$$

(10)

3.3.3. Average Energy Consumption

Based on the models of computational energy consumption and communication energy consumption, the total energy consumption generated by ${\mathrm{UAV}}_i$ during time slot t can be expressed as

$$\begin{array}{c}\hfill E_i^t=E_i^{comp,t}+E_i^{comm,t}.\end{array}$$

(11)

Let $d^t$ denote the total size of offloaded computational tasks received by all UAVs during time slot t, i.e., $d^t={\sum }_{i=1}^n{d}_i^t$. Next, let $d_i^{t^{\prime }}$ denote the total size of all computational tasks to be executed by ${\mathrm{UAV}}_i$ at time slot t, i.e.,

$$\begin{array}{c}\hfill d_i^{t^{\prime }}={\lambda }_{i,i}^t{d}_i^t+{\displaystyle \sum _{j=1,j\ne i}^n}{\lambda }_{j,i}^t{d}_j^t.\end{array}$$

(12)

Then, we can obtain the average energy consumption at time slot t as

$$\begin{array}{c}\hfill E^t=\frac{d_1^{t^{\prime }}}{d^t}{E}_1^t+\frac{d_2^{t^{\prime }}}{d^t}{E}_2^t+\cdots +\frac{d_n^{t^{\prime }}}{d^t}{E}_i^t=\frac{1}{d^t}{\displaystyle \sum _{i=1}^n}{d}_i^{t^{\prime }}{E}_i^t.\end{array}$$

(13)

3.4. Average Response Latency

In this section, we analyze the average response latency of all computational tasks in the computing environment.

In our computing environment, tasks are either executed locally on a UAV or offloaded to other UAVs. This distinction leads to different response latency calculations. Local execution introduces computation latency, while offloading introduces both computation and transmission latency.

Recall that we defined $d_i^t$ as the size of the offload computational tasks received by ${\mathrm{UAV}}_i$ from IOTDs at the beginning of the time slot t. These computational tasks were proportionally distributed by ${\mathrm{UAV}}_i$, where data of size ${\lambda }_{i,i}^t{d}_i^t$ was executed locally on ${\mathrm{UAV}}_i$, while data of size ${\sum }_{j=1,j\ne i}^n{\lambda }_{i,j}^t{d}_i^t$ was offloaded to other UAVs for collaborative execution. It is clear that the response latency of the task assigned to itself by ${\mathrm{UAV}}_i$ in time slot t can be expressed as

$$\begin{array}{c}\hfill T_i^{loc,t}=\frac{C_i^{loc,t}}{f_i}=\frac{c_i{\lambda }_{i,i}^t{d}_i^t}{f_i}.\end{array}$$

(14)

Since the response latency of the computational tasks that ${\mathrm{UAV}}_i$ offloads to ${\mathrm{UAV}}_j$ for remote execution in time slot t (including computation and transmission latency) can be expressed as

$$\begin{array}{c}\hfill T_{i,j}^{\mathrm{off},t}=\frac{c_j·{\lambda }_{i,j}^t{d}_i^t}{f_j}+\frac{{\lambda }_{i,j}^t{d}_i^t}{R_{i,j}·{10}^6},\end{array}$$

(15)

the average response latency of the computational tasks offloaded by ${\mathrm{UAV}}_i$ for remote execution can be expressed as

$$\begin{array}{c}\hfill T_i^{\mathrm{off},t}={\displaystyle \sum _{j=1,j\ne i}^n}\left(\frac{{\lambda }_{i,j}^t}{1-{\lambda }_{i,i}^t}{T}_{i,j}^{\mathrm{off},t}\right)=\frac{1}{1-{\lambda }_{i,i}^t}{\displaystyle \sum _{j=1,j\ne i}^n}\left({\lambda }_{i,j}^t{T}_{i,j}^{\mathrm{off},t}\right)\end{array}$$

(16)

Then, the average response latency of computational tasks received by ${\mathrm{UAV}}_i$ from IOTDs during time slot t can be expressed as

$$\begin{array}{c}\hfill T_i^t={\lambda }_{i,i}^t·T_i^{loc,t}+(1-{\lambda }_{i,i}^t)T_i^{\mathrm{off},t}={\lambda }_{i,i}^t·T_i^{loc,t}+{\displaystyle \sum _{j=1,j\ne i}^n}{\lambda }_{i,j}^t·T_{i,j}^{\mathrm{off},t}.\end{array}$$

(17)

Therefore, the average response latency of all computational tasks in the computing environment during time slot t can be calculated as

$$\begin{array}{cc}\hfill T^t=& \frac{d_1^t}{d^t}{T}_1^t+\frac{d_2^t}{d^t}{T}_2^t+\cdots +\frac{d_n^t}{d^t}{T}_n^t=\frac{1}{d^t}{\displaystyle \sum _{i=1}^n}{d}_i^t{T}_i^t.\hfill \end{array}$$

(18)

3.5. Problem Definition

In this section, we formally describe the optimization problem to be solved in this paper.

Given n UAVs, the communication circumstances $U_i$ for all $1\le i\le n$; the execution speed $f_i$ for all $1\le i\le n$; the task related parameters, including $d_i^t$, $c_i$ for all $1\le i\le n$; the communication-related parameters $R_{i,j}$, $B_{i,j}$, $q_{i,j}$, $N_{i,j}$ for all $1\le j\ne i\le n$; the constant factor related to the CPU chip architecture k; and the latency constraint $T^{*}$, obtain the optimal offloading scheme $\left({\lambda }_{i,1}^t,{\lambda }_{i,2}^t,\dots ,{\lambda }_{i,n}^t\right)$ for each UAV in every time slot, aiming to minimize average energy consumption while ensuring compliance with the average response latency constraint. The optimization problem can be formulated as follows:

$$\begin{array}{cc}\hfill {\mathrm{P}}_1:& min{E}^t,\hfill \\ \hfill \mathrm{s}.\mathrm{t}.C1:& \left\{\begin{array}{cc}{\lambda }_{i,j}^t=0,\hfill & \mathrm{if}{\mathrm{UAV}}_i^j=0;\hfill \\ {\lambda }_{i,j}^t\in [0,1],\hfill & \mathrm{if}{\mathrm{UAV}}_i^j=1.\hfill \end{array}\right.\hfill \\ \hfill C2:& {\lambda }_{i,1}^t+{\lambda }_{i,2}^t+\cdots +{\lambda }_{i,n}^t=1, \mathrm{for}\mathrm{all}1\le i\le n,\hfill \\ \hfill C3:& T^t\le T^{*}, \mathrm{for}\mathrm{all}t\in \left\{1,2,3,\dots ,T\right\}.\hfill \end{array}$$

(19)

In this formulation, constraint $C1$ denotes the proportion of tasks that each UAV offloads to other UAVs within its communication range as a value in the range [0, 1]. Constraint $C2$ ensures that each UAV’s computational task has an appropriate executor and that all tasks are executed. Constraint $C3$ specifies that the average response latency of the UAVs in completing tasks within each time slot must be less than the defined latency constraint, represented as $T^{*}$.

4. Our Solutions

4.1. Problem Transformation

In a dynamic UAV-assisted MEC environment, system state transitions at each time slot t are contingent upon the real-time system state and the current task offloading policy. Consequently, the offloading process within the UAV-assisted MEC environment constitutes an MAMDP [42]. To address this optimization problem, we naturally consider the application of deep reinforcement learning to train intelligent agents. These agents can be deployed on edge service controllers to generate task offloading strategies based on the observed system state. Subsequently, UAVs execute these task offloading actions, and rewards are allocated to the agent upon successful execution. Meanwhile, the system proceeds to the next state, and the agent refines its policy by leveraging the experiences gained from these states, actions, and rewards. In this paper, we define the terms state, observation, action, and reward as follows.

State: The environment state is the information used to determine subsequent actions, observations, and rewards, and it is a function of historical data. In the context of UAV-assisted MEC environment, we define the environmental state for time slot t as follows:

$$\begin{array}{c}\hfill s^t=\left\{d_1^t,d_2^t,\dots ,d_n^t,G_1,G_2,\dots G_n\right\}.\end{array}$$

(20)

We denote the set $G_i$ to encompass various attributes of ${\mathrm{UAV}}_i$ and communication-related information as follows:

$$\begin{array}{cc}\hfill G_i=& \{f_i,c_i,k,B_{i,1},\dots ,B_{i,n},R_{i,1},\dots ,R_{i,n},q_{i,1},\dots q_{i,n},N_{i,1},\dots ,N_{i,n},U_i\}.\hfill \end{array}$$

(21)

Observation: In our framework, we consider the data size of tasks received by UAVs as our observation. The observation for time slot t can be expressed as

$$\begin{array}{c}\hfill O^t=\left\{d_1^t,d_2^t,\dots ,d_n^t\right\}.\end{array}$$

(22)

Action: In the context of a UAV-assisted MEC environment, agent interactions are facilitated through a predefined set of actions that encompass all possible choices. At time slot t, each agent’s action involves determining the proportions ${\lambda }_{i,j}^t$ by which ${\mathrm{UAV}}_i$ offloads tasks to ${\mathrm{UAV}}_j$ within its communication range. Consequently, the action of ${\mathrm{UAV}}_i$ at time slot t is represented as

$$\begin{array}{c}\hfill a_i^t=\left({\lambda }_{i,1}^t,{\lambda }_{i,2}^t,\dots ,{\lambda }_{i,n}^t\right).\end{array}$$

(23)

Given the presence of n agents in the UAV-assisted MEC environment, we can derive the set of actions for all n agents at time slot t as follows

$$\begin{array}{c}\hfill a^t=\left\{a_1^t,a_2^t,\dots a_n^t\right\}.\end{array}$$

(24)

Reward: The reward is a function that assesses the impact of an action taken by an entity in a specific state. In the context of the UAV-assisted MEC environment, our objective is to ensure that the agent’s task offloading policy complies with the latency constraint $T^{*}$ while minimizing the average energy consumption. To achieve this, we define the following reward function

$$\begin{array}{c}\hfill r^t=\left\{\begin{array}{cc}\frac{{\omega }_e}{E^t},\hfill & \mathrm{if}{T}^t\le T^{*};\hfill \\ {\omega }_t\left(T^{*}-T^t\right),\hfill & \mathrm{if}{T}^t>T^{*}.\hfill \end{array}\right.\end{array}$$

(25)

In this context, ${\omega }_e$ and ${\omega }_t$ represent the coefficients of the reward function. Given that agents in the UAV-assisted MEC environment collaborate to achieve optimization objectives, this reward function is shared among all agents in this environment.

4.2. Preliminaries of the MADDPG

Reinforcement learning aims to find the optimal decision in uncertain environments on the basis of qualitative and noisy on-line performance feedback provided by the environments [43]. The agent corrects its policy in the environment by continuously interacting iteratively, allowing the agent to learn the best or near-optimal solution by maximizing the expected cumulative reward [44,45].

We employ the multi-agent deep reinforcement learning algorithm MADDPG to collaboratively optimize the objective function defined in Equation (19). As depicted in Figure 2, this algorithm integrates the deep deterministic policy gradient (DDPG) within a cooperative multi-agent learning framework, where each agent receives the same immediate reward. The MADDPG is built upon the actor–critic framework, leveraging both centralized training and decentralized execution.

Figure 2. The MADDPG framework in UAV-assisted MEC environment.

The MADDPG algorithm introduces several enhancements to both the actor–critic and DDPG algorithms, addressing the limitations of traditional reinforcement learning in coping with complex multi-agent environments while preserving DDPG’s efficiency and applicability to continuous action spaces. In traditional reinforcement learning, each agent is continually learning to enhance its policy, resulting in a dynamically unstable environment from the perspective of each agent. This dynamic instability makes the network environment non-stationary, thereby violating the markov assumptions necessary for stable and convergent learning.

Unlike traditional single-agent algorithms, the MADDPG involves n agents. Let $\pi =\left\{{\pi }_1,{\pi }_2,\dots ,{\pi }_n\right\}$ represent the set of strategies adopted by all agents for selecting their actions based on their observations, and let $\theta =\left\{{\theta }_1,{\theta }_2,\dots ,{\theta }_n\right\}$ denote the parameter set of networks. Each agent updates its policy independently with the aim of maximizing the expected cumulative reward. The expected cumulative reward of agent i is as follows:

$$\begin{array}{c}\hfill J\left({\theta }_i\right)=E_{s\sim p^{\pi },a_i\sim {\pi }_i}\left[{\displaystyle \sum _{t=1}^T}{\gamma }^{t-1}{r}^t\right].\end{array}$$

(26)

In this context, $r^t$ represents the reward obtained by agent i at time slot t. $p^{\pi }$ signifies the state distribution under the policy $\pi$. $\gamma$ stands for the discount factor, utilized to discount future rewards. The strategy gradient, a parameter that directly adjusts the strategy gradient, aims to maximize $J\left({\theta }_i\right)$ by taking steps in the direction of ${\nabla }_{{\theta }_i}J\left({\theta }_i\right)$. To maintain clarity, we will exclude the time index in the forthcoming equations. In the context of a stochastic strategy, the resulting strategy gradient formula is as follows:

$$\begin{array}{c}\hfill {\nabla }_{{\theta }_i}J\left({\theta }_i\right)=E_{s\sim {\rho }^{\pi },a_i\sim {\pi }_i}\left[{\nabla }_{{\theta }_i}log{\pi }_i\left(a_i\mid o_i\right)Q_i^{\pi }\left(x,a_1,a_2\cdots ,a_n\right)\right].\end{array}$$

(27)

Here, $Q_i^{\pi }\left(x,a_1,\cdots ,a_n\right)$ is a centralized action–value function that takes as inputs the actions of all agents, $a_1,a_2\cdots ,a_n$ in addition to some state information x, and outputs the Q-value for agent i. x represents the observations made by all agents, referred to as $x=\left[o_1,o_2,\dots ,o_n\right]$.

In contrast to random strategies, deterministic strategies dictate specific actions in particular states. Extending the preceding stochastic strategy to a deterministic strategy, denoted as ${\mu }_{{\theta }_i}$ (abbreviated as ${\mu }_i$), allows us to derive the gradient of the deterministic strategy as follows:

$$\begin{array}{c}\hfill {\nabla }_{{\theta }_i}J\left({\mu }_i\right)=E_{x,a\sim D}\left[{\left.{\nabla }_{{\theta }_i}{\mu }_i\left(o_i\right){\nabla }_{a_i}{Q}_i^{\mu }(x,a_1,a_2,\cdots ,a_n)\right|}_{a_i={\mu }_i\left(o_i\right)}\right].\end{array}$$

(28)

D is the experience replay buffer, retaining all the data generated through the agent’s interactions with the environment, each composed of $\left(x,x^{\prime },a_1,a_2,\dots ,a_n,r\right)$, where $x^{\prime }$ is all agent observations at the next moment.

During the training phase, the parameters of both the actor and critic networks are updated by randomly sampling a mini-batch of size $\kappa$ from the experience replay buffer. The gradient for the actor network of agent i can be computed using the following equation:

$$\begin{array}{c}\hfill {\nabla }_{{\theta }_i}J\left({\mu }_i\right)\approx \frac{1}{\kappa }\sum _j{\left.{\nabla }_{{\theta }_i}{\mu }_i\left(o_i^j\right){\nabla }_{a_i}{Q}_i^{\mu }(x^j,a_1^j,a_2^j,\dots ,a_n^j)\right|}_{a_i={\mu }_i\left(o_i^j\right)}{}^{\prime }\end{array}$$

(29)

where j is the index of the sample. The critic network of agent i can be updated by minimizing the loss function as follows:

$$\begin{array}{c}\hfill L\left({\theta }_i\right)\approx \frac{1}{\kappa }\sum _j{\left(y^j-Q_i^{\mu }\left(x^j,a_1^j,a_2^j\dots ,a_n^j\right)\right)}^2,\end{array}$$

(30)

where $y^j$ is the sum of the agent’s immediate reward and discount target Q-value. $y^j$ is obtained from the following equation:

$$\begin{array}{c}\hfill y^j=r^j+{\left.\gamma Q_i^{{\mu }^{\prime }}\left(x^{{}^{\prime }j},a_1^{{}^{\prime }j},a_2^{{}^{\prime }j},\cdots ,a_n^{{}^{\prime }j}\right)\right|}_{a_k^{{}^{\prime }}={\mu }_k^{{}^{\prime }}\left(o_k^{{}^{\prime }j}\right)}{}^{\prime }\end{array}$$

(31)

where ${\mu }^{\prime }=\left\{{\mu }_{{\theta }_1^{\prime }},{\mu }_{{\theta }_2^{{}^{\prime }}},\dots ,{\mu }_{{\theta }_n^{{}^{\prime }}}\right\}$ is the set of target policies with delayed parameters ${\theta }_i^{{}^{\prime }}$. To enhance the stability of the network training process, we will implement a soft update strategy in which the target network parameters ${\theta }_i^{{}^{\prime }}$ will be updated based on the current network parameters ${\theta }_i$ according to the following equation:

$$\begin{array}{c}\hfill {\theta }_i^{{}^{\prime }}\leftarrow \left(1-\epsilon \right){\theta }_i^{{}^{\prime }}+\epsilon {\theta }_i,\end{array}$$

(32)

where $0<\epsilon \ll 1$, which represents the smoothing factor. Upon completion of training, each agent can independently choose its actions based on local observations without knowledge of other agents’ information.

Algorithm 1 summarizes the detailed flow of the MADDPG algorithm for UAV-assisted MEC environment proposed in this paper. First, we randomly initialize the agent’s networks (lines 1–2) and initialize our replay buffer memory (line 3). The agents in the environment obtain the initial state of the observation, and the agent selects the task offloading percentage of UAVs within the communication range according to its policy and exploration noise (line 8). We employ Gaussian noise to enhance the action detection strategy, where $\psi$ represents the Gaussian noise. After the agent performs the selected action, the agent obtains a step reward, the next observation (lines 9–10). Then, the transition experiences of each agent are stored in the replay buffer memory, and the scenario state is updated to the next state (lines 11–12). The critic and actor network parameters of each agent are updated based on Equations (30) and (29), respectively (lines 16–17). Then, the target network parameters for each agent are updated according to Equation (32). The algorithm is terminated when it reaches the defined maximum number of sets.

Algorithm 1 MADDPG-Based Task Offloading Optimization in UAV-assisted MEC Environment

Input:     $\left\{{\mathrm{UAV}}_1,{\mathrm{UAV}}_2,\dots ,{\mathrm{UAV}}_n\right\}$,$U=\left\{U_1,U_2,\dots ,U_n\right\}$ Output:     $\left[({\lambda }_{1,1}^t,{\lambda }_{1,2}^t,\dots ,{\lambda }_{1,n}^t),\dots ,({\lambda }_{n,1}^t,{\lambda }_{n,2}^t,\dots ,{\lambda }_{n,n}^t)\right]$      1: Randomly initialize the weights of actor and critic networks      2: Randomly initialize the weights of target actor and critic networks      3: Initialize the size of replay buffer memory      4: for episode $e=1:EP$ do      5:       Initialize a random process $\psi$ for action exploration      6:       Receive initial observations      7:       for step $t=1:ST$ do      8:             For each agent i, select action $a_i^t={\mu }_i\left(O_i^t\right)+{\psi }^t$ w.r.t the exploration and current policy      9:             Execute the actions $a^t=\left\{a_1^t,a_2^t,\dots a_n^t\right\}$    10:             Obtain global reward $r^t$, new observations $O_i^{t+1}$    11:             Store the experience transitions in the replay buffer memory    12:             $x\leftarrow x^{\prime }$    13:             for agent $i=1$ to n do    14:                   Randomly select a mini-batch of experience transitions $\kappa$ from the replay buffer memory    15:                   Set $y^j$ according to Equation (31)    16:                   Update the critic network through minimizing the loss based on Equation (30)    17:                   Update actor network utilizing the sampled policy gradient based on Equation (29)    18:             End for    19:             Update target network parameters via Equation (32)    20:       End for    21: End for

5. Performance Evaluation

We performed a series of simulations to assess the performance of the MADDPG-based task offloading algorithm. Initially, we examined the impact of various parameters on the MADDPG scheme’s performance. Subsequently, we compared the MADDPG-based task offloading algorithm with four other schemes in diverse environments.

5.1. Simulation Settings

To assess the performance of our algorithm, we conducted simulations in two distinct scenarios involving six UAVs and eight UAVs, respectively. In these scenarios, each UAV follows a predefined flight path during independent operations. Positioned at fixed coordinates, the UAVs offer computational offloading services to IOTDs within their coverage zones. These UAVs receive computational tasks from IOTDs, with task sizes ranging from 10 Mb to 25 Mb. The parameters used in our experiments are as follows: constant factor related to the CPU chip architecture $k={10}^{-27}$, number of CPU cycles needed to execute 1-bit data $c_i\in \left\{400,500,600,800\right\}$ cycles/bit, execution speed $f_i=2.0+0.4(i-1)$ GHz, channel bandwidth $B_{i,j}=3.0+0.1(j-1)$ MHz, data transmission rate $R_{i,j}=10+0.5(j-1)$ Mbps, noise power spectral density $N_{i,j}=-174-0.2(j-1)$ dBm/Hz; all of the above i and j take a range of values $1\le j\ne i\le n$, where n is the number of UAVs in the current environment.

Our simulations were implemented in the PyCharm development environment using PyTorch and Python 3.7.3. In our algorithm, each agent is equipped with both a critic network and a actor network. these networks are four-layer fully connected neural networks, commonly referred to as multilayer perceptrons. Specifically, the hidden layers in these fully connected neural networks were set to $[512,256]$ and $[256,128]$, respectively, with ReLU serving as the activation function in the hidden layers. The important hyperparameters used in the experiments are shown in Table 1.

Table 1. Experimental parameters.

Parameter	Definition	Setting
$EP$	Episode	1,000,000
D	Replay buffer size	1,000,000
${\omega }_e$	Coefficient for reward function	1000
${\omega }_t$	Coefficient for reward function	10
$\sigma$	Gaussian noise variance	0.05∼0.5
$ST$	Steps	1
$\gamma$	Discount factor	0.1
$\alpha$	Critic network learning rate	0.01
$\beta$	Actor network learning rate	0.01
$\epsilon$	Softupdate factor	0.01
$p_i$	Number of iterations	200
$p_n$	Number of particles	100
$c_1$	Acceleration coefficients	1.5
$c_2$	Acceleration coefficients	1.5
$p_w$	Inertia weight	0.5

5.2. Parameter Analysis

The training regimen for our scheme unfolds as follows: The initiation process encompasses the initialization of the agent network and the replay buffer memory. The agents within the environment receive initial observations. Each agent independently selects the task offloading ratios for UAVs within their communication range, guided by a combination of their individual strategies and exploration noise. Following the execution of their chosen actions, agents receive step rewards and the subsequent set of observations. The experiences of each agent are recorded within the replay buffer memory. Concurrently, the environment’s state transitions to the subsequent state. Agents update their network parameters based on their interactions with the environment. The algorithm will terminate when the predefined maximum number of episodes is reached.

During the training process, we investigate the impact of learning rates and discount factors on the convergence of the MADDPG-based scheme proposed in this study.

5.2.1. Impact of Learning Rate on Convergence

Figure 3a depicts the convergence performance of the Adam optimizer in the MADDPG algorithm for three different learning rates. As seen in Figure 3a, the total reward for all learning rates increases with increasing learning time and converges within a specific range. This is because, as the learning episodes increase, the actor and critic networks adjust their parameters to fit the optimal strategy and action values. Furthermore, we can find that setting the learning rate to 0.01 obtains rewards higher than 0.1 and 0.001. This is because the learning rate determines the step size of the change in the weights with the direction of the gradient. The larger the learning rate, the more significant the change in weights. Therefore, a large or small learning rate can make the network fall into a local optimum solution.

Figure 3. Convergence performances of MADDPG with different parameters. (a) Different learning rates. (b) Different discount factors.

5.2.2. Impact of Discount Factor on Convergence

Figure 3b depicts the convergence performance of three different discount factors in the MADDPG algorithm. In this experiment, the discount factor $\gamma$ is set to 0.1, 0.5, and 0.9, respectively. It can be seen from the figure that the algorithm converges with the highest reward when the discount factor is set to 0.1. This is because the discount factor impacts the balance between current and future rewards. Given our optimization objective, which prioritizes immediate rewards, setting the discount factor to 0.1 results in higher rewards.

5.3. Comparison Experiments

To evaluate the efficacy of our solution in the context of the UAV-assisted MEC environment, we conduct comparisons with the following benchmark solutions in scenarios involving six and eight UAVs, respectively.

•

PSO scheme: We treat the set of offloading decisions made by all UAVs in the current environment as a particle, utilizing Equation (25) as the fitness function for the PSO algorithm. The parameters for the PSO algorithm are presented in Table 1.

•

URM scheme: In the URM scheme, each UAV first determines the number of UAVs within its communication range (denoted as m). It then distributes the tasks equally between itself and the other UAVs within its communication range. Thus, the size of the task that ${\mathrm{UAV}}_i$ shares with other UAVs in its communication range is $\frac{d_i^t}{m+1}$.

•

Proximal policy optimization (PPO) scheme: Each UAV has an independent PPO [46] model and each UAV makes independent offloading decisions.

•

DDPG scheme: Each UAV operates autonomously with its independent DDPG model, devoid of a centralized critic network and local information sharing. Each UAV autonomously makes offloading decisions. We employ the neural network structure consistent with the MADDPG algorithm for both the actor and critic networks of the DDPG.

We employ the MADDPG algorithm alongside PPO and DDPG algorithms to iteratively interact with the environment and update network parameters within a specified environment. This allows us to conduct a comparative analysis of their convergence performance by recording the average reward per 100 episodes.

As depicted in Figure 4, we have compared the convergence performance of our approach with that of the PPO and DDPG algorithms across various environments. In the initial phase, the average reward is relatively low due to the random initialization of network parameters. During the initial 20,000 training episodes, both the MADDPG and DDPG algorithms interacted with the environment to accumulate experiences for each agent, which were subsequently stored in the replay buffer memory. After completing these initial 20,000 episodes, network parameters are updated based on the data stored in the replay buffer memory. It is essential to note that the PPO algorithm operates as an online algorithm, necessitating the generation of data using the current policy and updating the policy based on this data. As depicted in Figure 4, MADDPG outperforms DDPG and PPO in terms of average reward. MADDPG achieves these outstanding results because during training, the critic of each agent in MADDPG has access to information regarding the actions of other agents.

Figure 4. Convergence performance of our approach, DDPG, and PPO in various environments. (a) Six-UAV environment. (b) Eight-UAV environment.

Based on the environmental parameters in Section 5.1, we randomly initialized five states to assess the performance gap between our scheme and other benchmark algorithms. These states consist of sets of task sizes received by UAVs from the IOTDs. These tasks vary in size from 10 Mb to 25 Mb and are generated following a uniform distribution. Within each state, the offloading policies prescribed by our scheme and the other benchmark algorithms are evaluated, and the average energy consumption and average task execution latency for various offloading policies are computed.

Figure 5 illustrates the average latency and average energy consumption of offloading policies provided by five different algorithms in environments with six and eight UAVs across five distinct states. The latency constraints in the environments with six and eight UAVs were 1.45 s and 0.95 s, respectively. Figure 5a,b show the simulation environment with six UAVs, and Figure 5c,d show the simulation environment with eight UAVs. We observe a significant reduction in the average response latency for performing computational tasks received from IOTDs within each time slot as we increase the computational resources, specifically the number of UAVs. It is also relatively easy to find that the average energy consumption required for the computation is much higher because the increase in the number of UAVs means more computational tasks from IOTDs are accepted. From Figure 5a,c, we can easily find that the offloading decisions made by the MADDPG algorithm satisfy the latency constraint for the random five states. Figure 5b,d demonstrate that the average energy consumption for processing the computational tasks using the MADDPG algorithm consistently outperforms the other baseline algorithms.

Figure 5. Average latency and average energy consumption of different schemes. (a) Six-UAV environment. (b) Six-UAV environment. (c) Eight-UAV environment. (d) Eight-UAV environment.

Figure 5 shows the comparative effect of our algorithm and the baseline algorithm for only five random states. To evaluate the real effectiveness of the algorithms in practical environments, we expanded the initial five random states to a total of 1000 random states. Employing the same methodology as described above, we collected offloading strategies from various schemes across different states. Subsequently, we computed the average energy consumption and average latency associated with these offloading strategies for task execution. It is worth noting that we only calculate the sum of the average energy consumption when all algorithms meet the latency constraints, as there may be cases of violation.

In environments featuring six and eight UAVs, the DDPG baseline algorithm breached latency constraints 187 and 3 times, respectively. The PSO baseline algorithm exceeded these constraints 233 and 90 times in the corresponding scenarios, while the PPO baseline algorithm did so 3 and 16 times in the same contexts. Notably, neither the MADDPG algorithm nor the URM algorithm violated the latency constraint. Figure 6 displays the total average energy consumption of the five algorithms across 1000 random states, where all algorithms successfully adhere to the latency constraints. The results indicate that our scheme, employing the MADDPG algorithm, achieves lower average energy consumption in comparison to the benchmark algorithm. For instance, with $n=6$, the offloading scheme offered by MADDPG exhibits an average energy consumption that is 21.06% lower than that of DDPG, 16.72% lower than that of PPO, 8.93% lower than that of URM, and 6.99% lower than that of PSO.

Figure 6. The total average energy consumption of various schemes meeting the latency constraint in 1000 random states.

In conclusion, our scheme, employing the MADDPG algorithm, effectively reduces average energy consumption while adhering to latency constraints in diverse environments. This affirms the efficacy of our approach across various scenarios.

6. Conclusions

In this paper, we investigate the problem of collaborative computation offloading involving multiple UAVs under the constraints of limited communication range and latency. Our primary objectiveis is to minimize the average energy consumption of these UAVs while ensuring latency constraints are met. To address the high dynamism and complexity of this problem, we formulate the cooperative offloading problem as an MAMDP and use a cooperative algorithm based on the MADDPG to tackle it effectively. The MADDPG algorithm significantly reduces energy consumption in UAV-assisted MEC environment by facilitating cooperative computational offloading while meeting the latency constraints. Simulation results demonstrate the superiority of the MADDPG algorithm over existing methods. It is worth noting that in this article we simplified the process through which UAVs receive tasks from IOTDs. In our upcoming research, we will delve into the optimization of computational offloading for IOTDs.