Efficient Delay-Sensitive Task Offloading to Fog Computing with Multi-Agent Twin Delayed Deep Deterministic Policy Gradient

Abstract

Fog computing presents a significant paradigm for extending the computational capabilities of resource-constrained devices executing increasingly complex applications. However, effectively leveraging this potential critically depends on the implementation of efficient task offloading mechanisms to proximal fog nodes, particularly under conditions of high resource contention. To address this challenge, we introduce MAFCPTORA (multi-agent fully cooperative partial task offloading and resource allocation), a decentralized multi-agent deep reinforcement learning algorithm for cooperative task offloading and resource allocation. We evaluated the performance of MAFCPTORA and compared it against recent approaches. MAFCPTORA demonstrated superior performance compared to recent methods, achieving a significantly higher average reward (0.36 ± 0.01), substantially lower average latency (0.08 ± 0.01), and reduced energy consumption (0.76 ± 0.14).

1. Introduction

The rapid proliferation of Internet of Things (IoT) and consumer electronics (CE) devices, projected to reach a staggering scale by the end of the decade [1], has spurred the development of real-time, low-latency applications in domains such as industrial IoT [2], voice recognition [3], and medical IoT [4]. The inherent resource limitations of these devices prevent them from executing such demanding applications, necessitating efficient task offloading to other computing resources with minimal communication latency. Consequently, fog computing has received significant attention as a promising platform for processing these IoT applications [5,6,7,8,9].

Effectively allocating resources in the distributed, dynamic, and resource-constrained nature of fog devices presents a fundamental challenge for diverse and delay-sensitive IoT tasks. This is further complicated by the dynamic network conditions and the potential lack of effective Fog-to-Fog (F2F) cooperation. To tackle these complexities, prior research has explored various techniques, including machine learning algorithms [7,10] for adaptive decision-making, agent-based systems [9,11,12,13] for distributed control, metaheuristic algorithms [8] for near-optimal solutions, deep reinforcement learning [14], and traditional optimization techniques [15].

While existing approaches commonly focus on offloading tasks to fog nodes, simply enabling this is insufficient. Effectively balancing resource allocation across heterogeneous fog nodes, addressing the need for distributed execution of intensive tasks, and enabling efficient F2F cooperation remain critical. Furthermore, optimizing for a single scheduling parameter often fails to simultaneously ensure high QoS for users and low energy costs for providers [15]. Therefore, finding an optimal solution for balancing task offloading and resource allocation within the dynamic and resource-constrained fog computing environment, particularly for demanding applications, remains a fundamental challenge.

To address this challenge, this paper aims to answer the following fundamental question: How can we minimize system-wide communication and computation latency while simultaneously reducing the energy consumption of fog nodes in the context of executing demanding IoT applications? To this end, we propose a novel approach leveraging multi-agent deep reinforcement learning (MADRL) for efficient task offloading and scheduling on heterogeneous and resource-limited fog computing resources. Integral to this approach is Fog-to-Fog (F2F) cooperation, allowing fog nodes to communicate and coordinate their actions to optimize both local and global performance. Our key contributions include:

• Formulate the task offloading and resource allocation problem as a Markov decision process (MDP) with a stochastic continuous action space to address computational and transmission delays in a multi-task, multi-node environment.
• Develop a Multi-Agent Fully Cooperative Partial Task Offloading and Resource Allocation (MAFCPTORA) algorithm that enables parallel task execution across adjacent fog nodes in an F2F architecture, leading to faster execution, reduced latency and energy consumption, and balanced workload distribution.
• Evaluate the performance of our algorithm, demonstrating its effectiveness in improving the deadline fulfillment rate for IoT tasks while minimizing the total energy consumption and makespan of the system, thereby benefiting fog service providers.

The rest of this paper is organized as follows: Related research works are presented in Section 2, followed by Section 3, which describes the system architecture, including the task model, dynamic task offloading, and resource allocation models. Section 3.6 presents the problem statements. The proposed TD3-based task offloading and resource allocation algorithms and key parameters are detailed in Section 4 and Section 5. A discussion of simulation results and performance analysis is presented in Section 6. Finally, the conclusion and future recommendations are provided in Section 7.

2. Related Work

In this section, we review related work.

Table 1. Description of abbreviations.

Symbol	Description
MAFCPTORA	Multi-Agent Fully Cooperative Task Offloading and Resource Allocation
MADRL	Multi-Agent Deep Reinforcement Learning
MADDPG	Multi-agent deep deterministic policy gradient
MAPPO	Multi-agent proximal policy optimization
TD3	Twin Delayed Deep Deterministic
MATD3	Multi-agent twin delayed DDPG
DRL	Deep Reinforcement Learning
MDP	Markov Decision Process

is the description of abbreviations used throughout the paper. Table 2 provides a summary of research that focuses primarily on task offloading across IoT, edge, fog, and cloud environments. Machine learning (ML) techniques, particularly deep reinforcement learning (DRL), are increasingly being explored to tackle the complexities of task offloading and resource allocation in fog computing, aiming to improve quality of service (QoS). These approaches include a multi-agent soft actor critic (MASAC) [16], multi-agent deep deterministic policy gradient (MADDPG) [13], multi-agent proximal policy optimization (MAPPO) [17], and multi-agent twin delayed DDPG (MATD3) [18].

2.1. Machine Learning-Based Approaches

Zhu et al. [19] employed deep learning to balance task completion time and energy consumption by minimizing a weighted sum, leveraging network information like resource status and channel conditions. However, their approach relies on a pre-trained model with a fixed dataset, potentially limiting its adaptability to dynamic network conditions. Ghanavati et al. [8] introduced a novel task offloading and scheduling algorithm with emphasis on enhancing the efficiency of fog computing while simultaneously minimizing energy consumption. Their approach leverages the Ant Colony Optimization for Mining Outliers (AMO) algorithm, a biologically inspired optimization technique. A key aspect of their proposed method involves strategically distributing tasks among geographically proximate fog nodes. This localized task allocation aims to reduce communication overhead and latency, contributing to both improved performance and energy savings. Furthermore, their work addresses the critical trade-off between the system’s operational lifetime and the energy consumed by fog computing services, allowing end-user devices to influence this balance based on their specific requirements and priorities. The authors conducted a comprehensive empirical performance evaluation, comparing their AMO-based scheduling algorithm against established optimization techniques such as Particle Swarm Optimization (PSO), the Bug Life Algorithm (BLA), and the Genetic Algorithm (GA). The results of this evaluation reportedly demonstrate the superiority of their proposed approach in terms of both energy efficiency and processing speed, suggesting a significant advancement in task scheduling for fog computing environments.

Table 2. Related Research Work.

Domain	Problem	RL Method	QoS Objectives	Agent Type	Network	Papers
VFC	V2V Partial Computation Offloading	SAC	Minimize task execution delay and energy consumption	Multi-Agent (UAV)	Dynamic	[16]
IoT-edge-UAV	Task offloading and Resource allocation	MADDPG	Minimize computation cost	Multi-Agent (UAV)	Dynamic	[20]
Vehicular Fog	Optimize Ratio-Based Offloading	DQN	Minimize average system latency, and lower decision and convergence time	Single-Agent	Dynamic	[21]
VFC	Minimize waiting time and end-to-end delay of tasks	PPO	Minimize waiting time, end-to-end delay, packet loss; Maximize percentage of in-time completed tasks	Single-Agent	Dynamic	[22]
Multi-Fog Networks	Joint Task Offloading and Resource Allocation	DRQN	Maximize processing tasks and minimize task drops from buffer overflows	Multi-Agent	Dynamic	[23]
Edge-UAV	Task offloading	MADDPG	Minimize energy consumption and delay	Multi-Agent UAV	Unstable	[11]
IoMT	Computation offloading and resource allocation	MADDPG	Latency Reduction	Multi-Agent	unstable	[13]
Multi-cloud (Fog)	Dynamic Computation Offloading	CMATD3	Minimize long-term average total system cost	Multi-Agent (Cooperative)	Dynamic	[24]
MEC	Multi-channel access and task offloading	DPPG	Minimize task computation delay	MTA	Stable	[25]
Vehicular Fog	Load Balancing problem	TPDMAAC	Minimize system cost	MV	hybrid	[14]
Vehicular Fog	Task offloading and computation load	AC	Maximize resource trading	MA-GAC	hybrid	[9]
Fog-RAN	Computation Offloading and Resource Allocation	DDPG	Minimize task computation delay and energy consumption	Multi-agent (Federated)	Dynamic	[26]
MEC and Vehicular Fog	Trafic congestion and MEC overload	TD3	Minimize latency and energy consumption	Multi-agent	hybrid	[18]

Table 2. Related Research Work.

Domain	Problem	RL Method	QoS Objectives	Agent Type	Network	Papers
VFC	V2V Partial Computation Offloading	SAC	Minimize task execution delay and energy consumption	Multi-Agent (UAV)	Dynamic	[16]
IoT-edge-UAV	Task offloading and Resource allocation	MADDPG	Minimize computation cost	Multi-Agent (UAV)	Dynamic	[20]
Vehicular Fog	Optimize Ratio-Based Offloading	DQN	Minimize average system latency, and lower decision and convergence time	Single-Agent	Dynamic	[21]
VFC	Minimize waiting time and end-to-end delay of tasks	PPO	Minimize waiting time, end-to-end delay, packet loss; Maximize percentage of in-time completed tasks	Single-Agent	Dynamic	[22]
Multi-Fog Networks	Joint Task Offloading and Resource Allocation	DRQN	Maximize processing tasks and minimize task drops from buffer overflows	Multi-Agent	Dynamic	[23]
Edge-UAV	Task offloading	MADDPG	Minimize energy consumption and delay	Multi-Agent UAV	Unstable	[11]
IoMT	Computation offloading and resource allocation	MADDPG	Latency Reduction	Multi-Agent	unstable	[13]
Multi-cloud (Fog)	Dynamic Computation Offloading	CMATD3	Minimize long-term average total system cost	Multi-Agent (Cooperative)	Dynamic	[24]
MEC	Multi-channel access and task offloading	DPPG	Minimize task computation delay	MTA	Stable	[25]
Vehicular Fog	Load Balancing problem	TPDMAAC	Minimize system cost	MV	hybrid	[14]
Vehicular Fog	Task offloading and computation load	AC	Maximize resource trading	MA-GAC	hybrid	[9]
Fog-RAN	Computation Offloading and Resource Allocation	DDPG	Minimize task computation delay and energy consumption	Multi-agent (Federated)	Dynamic	[26]
MEC and Vehicular Fog	Trafic congestion and MEC overload	TD3	Minimize latency and energy consumption	Multi-agent	hybrid	[18]

2.2. Reinforcement Learning

The user device computation request and the fog resources scenario are dynamic, where the node resource in the network changes continuously. This network topology with variable computing and network resources explored in [27,28] emphasizes the challenge of designing an optimal model for task offloading and resource allocation decisions to improve response time. To deal with this instability context, [29] designed a distributed task priority (TPRA) and maximal utilization (MaxRU) resource allocation scheme. Furthermore, the application of RL to address the changes in workload status and resource availability of the fog node is explored in [28,30]. The resource allocation mechanisms are designed based on the actual response time measured in terms of the average delay of unit task offloading changes over time in the system. This primarily affects the decisions for task offloading and allocation of resources among the $F_{m,j}$s and their cooperation approaches. However, offloading tasks to a specific $F_j$ results in a long response time due to limited resources.

2.3. Deep Reinforcement Learning (DRL)

Recognizing that task offloading in dynamic fog environments, characterized by stochastic computing tasks, varying resources, and fluctuating network channels [31], aligns well with DRL’s capabilities, researchers have explored its application. Zhang et al. [32] proposed a DRL-based approach for multi-user, multi-fog environments with dynamic workloads and server capacities, aiming to minimize task execution delay and energy consumption, albeit requiring coordination for their hybrid action space. Qiu et al. [33] addressed the limitations of traditional long-term performance optimization by introducing a distributed and collective training method to maximize cumulative rewards in multi-edge, multi-user offloading scenarios, claiming improved convergence and reduced system cost. Furthermore, Baek et al. [23] presented a deep recurrent Q-network (DRQN) for partially observable, dynamic fog environments with heterogeneous resources, focusing on CPU and memory. Their approach emphasizes local cooperation to maximize aggregated local rewards, with nodes relying solely on local observations. However, the search for a suitable fog node can lead to an exponentially growing search space as task size increases. More recently, Wu et al. [17] proposed a task-offloading algorithm using an improved multi-agent proximal policy optimization (MAPPO) to minimize delay and energy consumption, while Bai et al. [24] applied a DQN-based solution to address user request delays and energy consumption.

From an architectural and management standpoint, a key challenge lies in deciding the optimal placement and management strategy for learning-based offloading algorithms within distributed fog environments, whether centrally managed through the fog gateway or distributed across individual fog nodes. Ghanavati et al. [34] emphasized the importance of fault-aware task scheduling in fog computing platforms for ensuring reliability, efficiency, and resource utilization. Thus, Dynamic Fault Tolerant Learning Automata (DFTLA) is proposed as an approach to fault-tolerant learning automata task scheduling. An active variable learning automaton is embedded in every fog node of a DFTLA-based system, which continuously monitors the process and makes appropriate decisions in response to inputs from the environment. Comparing the proposed DFTLA scheduler with three baseline methods, they evaluated its performance. Using the proposed algorithm, reliable task execution can be accomplished while responding quickly and consuming little energy. All performance evaluation criteria showed that the proposed approach outperformed the baseline algorithms. Cao et al. [25] investigated a distributed deep reinforcement learning (DRL) approach, asserting that multi-agent deep deterministic policy gradient (MADDPG) and multi-agent proximal policy optimization (MAPPO) offer more accurate and faster decision-making compared to single-agent systems in real-time scenarios. Similarly, Hussein et al. [7] considered latency constraints arising from the distributed nature of fog nodes, aiming to identify nodes that meet quality of service (QoS) requirements.

Chen et al. [35] proposed a cooperative multi-agent deep reinforcement learning (CMADRL) framework where IoT devices learn independently but are centrally trained to minimize system-wide costs associated with energy consumption and server rentals. To address these dynamic contexts, multi-agent DRL (MADRL) has emerged as a prominent approach, with agents deployed in both decentralized [11,14] and centralized [13] architectures. Furthermore, Zhang et al. [36] categorized multi-agent settings in stochastic task offloading games based on their reward structures: fully cooperative, fully competitive, and mixed. In this vein, Suzuki et al. [37] and Zhu et al. [38] implemented MADRL techniques employing central training with decentralized execution strategies.

2.4. Multi-Agent Deep Reinforcement Learning

Multi-agent deep reinforcement learning (MADRL) has emerged as a powerful paradigm for tackling complex resource management challenges in distributed systems. Its applicability spans diverse domains, including UAV-based task offloading and resource allocation in air-to-ground (A2G) networks [20], where it optimizes for average local and global maxima in continuous search spaces. MADRL has also been instrumental in enhancing vehicular fog resource utilization for many-to-many task offloading in VFC networks [9] and in managing task offloading within space-aerial-ground integrated networks (SAGIN) [11], while ensuring user device quality of service (QoS). Furthermore, Suzuki et al. [37] reinforce the utility of cooperative MADRL in addressing reinforcement learning problems within multi-cloud edge networks. Building upon this foundation and drawing inspiration from [13], which explores multi-UAV environments with unreliable computational resources, this work formulates the joint task offloading and resource allocation problem as a stochastic game. Multi-agent TD3 (MADRL-based TD3) [39] is proposed to address challenges in large systems by reducing the input and action space for individual agents, leading to faster convergence and more efficient processing than single-agent TD3. A key distinction of our proposed fog network lies in its composition of both reliable and unreliable computational sources and communication networks, enabling adaptation to a wider range of situational scenarios.

In summary, existing research predominantly focuses on vertical task offloading (to edge, fog, or cloud), with less attention given to horizontal offloading among peers or neighbors. Furthermore, simultaneously considering factors like mobility and task dependencies remains a significant challenge. Notably, few prior works in fog computing address task priority, neighbor service availability, or resource contribution incentives. The complexity and robustness of decision-making algorithms, especially their decision time, are also often overlooked.

3. System Model

This section details our proposed two-layer network model, task model, communication model, computation model, and formulation of the problem. For the purpose of this study, we assume reliable fog nodes and communication links.

Table 3. List of key parameters.

Notation	Description
$F_n$	Number of Fog nodes
$U_n$	Number of User devices
$M_{f,n}$	Available Memory at each Fog node
$C_f$	Available CPU at each Fog node
$T_{m,j}$	Task processing delay
${\tau }_{m,j}^o$	Task offloading delay
$\beta$	Transmission Bandwidth
$E_k$	Energy consumption of task k
$\zeta$	Battery life time
$H_c$	Channel condition
$\phi$	Number of channel available
$K_n$	Task to be executed
$V_k$	Total size of the task $K_n$
$H_i$	Channel transmission power
${\tau }_{max}$	Maximum tolerable delay
Q	Waiting tasks in each Fog node
$S_i$	State of each agent
$O\left(t\right)$	Observation of environment at time t
$a_i$	Individual agent action
$X_n$	Offloading decision
$R_{f,k}\left(t\right)$	Resource allocation decision at time t
$\mu$ and $\mu \widehat{\ast }$	Optimal and target policy
$r_t$ and ${\tilde{r}}_t$	Individual and collective average reward

shows the lists of notation used throughout this article.

3.1. Network Model

We model a multi-user, multi-fog environment (as illustrated in Figure 2) to handle diverse service demands. The network features multiple heterogeneous and resource-constraint user devices $\left(U_n\right)$, where $n=1,2,\dots ,N$) connected wirelessly to a hierarchical fog layer (F). There is also a set of $F=F_1,F_2,\dots ,F_n$ heterogeneous fog nodes, where $M\le N$. Each fog node maintains its resource status and shares updates with its neighbors, enabling collaborative task offloading and resource management. Moreover, their processing capacity increases from the bottom to the top layer, whereas energy consumption and latency also increase.

Our system model incorporates a hybrid two-stage task execution process: User-to-Fog (U2F) and Fog-to-Fog (F2F). In this model, task execution proceeds in two phases: In the U2F phase, a task originating from a user device ($U_n$) is offloaded to a fog node $F_m$ (the task-owning node, $m\in 1,2,\dots ,N$). $F_m$ is considered to minimize task transmission delay and response time. We refer to the task execution on $F_m$ as local computation. The second phase, the F2F phase, is invoked in the event that the execution of a task locally at $F_m$ may not guarantee QoS requirements. In this case, we employ a horizontal distributed F2F cooperation mechanism following the approach in [28] depicted in Figure 1. In this case, tasks are redistributed from $F_m$ to an adjacent node $F_j$ ($j\in 1,2,\dots ,N$) for further processing or collaborative execution. Figure 2 illustrates the task offloading and execution scenario.

In this fog layer, task allocation to adjacent fog nodes ($F_j$) is governed by an offloading probability ratio. This ratio is dynamically determined by factors such as the available resources of each $F_j$ and the requirements of the user’s task, including its computation deadline and priority. Within this cooperative peer-to-peer (F2F) framework, each fog node ($F_m$ and its neighbors $F_j$) can make independent offloading decisions. Consequently, the set of tasks to be offloaded to a specific $F_j$ is denoted as $\left(K_{n,j}=1,2,\dots ,n\right)$. To analyze the offloading decision, the time step (decision episodes) is presented as a discrete time $(T=1,2,3,\dots ,T)$.

3.2. Task Model

We consider multiple tasks initiated from $U_n$s denoted as $K_n$, where $\{n=1,2,\dots ,N\}$. The tasks can be represented as a directed acyclic graph (DAG) shown in Figure 3.

The task profile of the nth user application can be defined in Equation (1):

$$K_n=\{V_k,C_n,{\tau }_{max}\}$$

(1)

where $V_k$, $C_n$, and ${\tau }_{max}$ denote the task data size, the CPU cycles required to compute the task, and the maximum tolerable deadline (expired time) of the task $K_n$, respectively. The task computation $\left(K_n\right)$ at time $\left(t\right)$ is defined by the total size of the task $\left(V_k\right)$ and the intensity of the device computation $b_u\left(t\right)$ at time $\left(t_i\right)$.

The DAG task in Figure 3 demonstrates parallel task execution enabled by task dependencies. Taking task $K_n$, composed of sub-tasks $k_1,k_2,k_3,k_4,k_5,k_6,k_7$, as an example, $k_1$, $k_2$, and  $k_3$ can run in parallel, $k_4$ wait for execution of $k_2$, then $k_5$ and $k_6$ waits for execution of $k_4$, while $k_7$ is dependent on $k_5$ and $k_6$ and executes only after both are finished.

3.3. Dynamic Task Partitioning

In our F2F task offloading mechanism, dynamic task partitioning is employed. A DRL agent learns to partition tasks effectively by observing real-time system conditions such as resource availability, task requirements, network bandwidth, and computational cost. For partial offloading, where tasks are distributed across multiple adjacent fog nodes, each fog node $F_j$ is assigned an offloading probability ratio $p_j$, where $(0<p_j<1)$. The complete probability distribution for a task $\left(K_n\right)$ being offloaded from $F_m$ to its neighbors is represented by the vector $P=(p_1,p_2,\dots ,p_n+1)$. Where $p_n\left(t\right)=p_n$ and $p_n\left(t\right)=0$ if node $f_m$ processes all the arrived tasks $k_n$ locally, while $F_m\left(t\right)=p_n+1$ and $p_n\left(t\right)\#0$ implies that the node $F_m$ offloads the number $p_n\left(t\right)$ of $k_n$ to adjacent $f_j$. This probability vector guides the decision-making process for distributing portions of the task to available $F_j$ nodes, considering their computational capabilities, the network latency between $F_m$ and each $F_j$, and their current resource availability. Equation (2), presents the task offloading probability ratio for each $F_j$:

$$p_j={\displaystyle \frac{k_{n,j}}{{\sum }_{j=1}^n{k}_{n,j}}}$$

(2)

where ${\sum }_{j=1}^n{k}_{n,j}$ defines the sum of tasks offloaded to adjacent $F_j$ and Equation  (3), for task execution by $f_j$.

$$k_{n,j}=p_j\times K_n$$

(3)

For the local task execution, the remaining partition to be executed at the source $\left(f_m\right)$ is defined as in Equation (4):

$$P_m=1-\sum _{j=1}^n{p}_j$$

(4)

where $K_n$ indicates the total number of tasks K being offloaded and defined as $(K_n={\sum }_{j=1}^n{k}_{n,j})$). With this, the total probability ratio of all $F_j$ must ensure Equation (5):

$$\sum _{j=1}^n{p}_m+p_j=1$$

(5)

This ensures that the sum of all partial offloading ratios forms a valid probability distribution in a continuous action space, where each $x_{m,j}\in (0,1)$.

Following this partitioning scheme, task offloading proceeds dynamically among horizontally connected fog nodes. To meet task execution requirements under fluctuating conditions, an overloaded fog node ($F_m$) transfers tasks to less utilized or idle nodes ($F_j$). Figure 2 illustrates this dynamic task partitioning and offloading process.

3.4. Communication Model

The communication model in our proposed network architecture encompasses two types of communication, which are User-to-Fog (U2F), which involves continuous communication between user devices ($U2F$) and fog nodes ($F_m$), and Fog-to-Fog (F2F), specifically between fog nodes $F_m$ and $F_j$. This section details these communication aspects.

3.4.1. U2F Communication

Following the model in [40], wireless communication between user devices ($U_n$) and a fog node ($F_m$) is subject to multi-path fading and path loss. The probability of path loss ($PL\left(d_{uf}\right)$) considers both line-of-sight (LoS) and non-line-of-sight (NLoS) conditions. The path loss $PL$ between $U_n$ and the neighboring fog node $F_m$ is adopted from [28,41]. To aid offloading decisions, the channel status $\left(H_c\right)$ is classified as busy (1) or idle (0). Therefore, the path loss $\left(\eta \right)$ is presented as follows at Equations (6) and  (7).

$$P{L}_{\left(d_{uf}\right)}^t=\sigma \left(d_{uf}\right)\times P{L}_{LoS}\left(d_{uf}\right)(1-\sigma \left(d_{uf}\right))\times P{L}_{NLoS}\left(d_{uf}\right)$$

(6)

where $\sigma \left(d_{uf}\right)$, $P{L}_{LoS}\left(d_{uf}\right)$, and $P{L}_{NLoS}\left(d_{uf}\right)$ represent LoS probability, LoS path loss $\eta 1$, and NLoS path loss $\eta 2$, respectively.

$$P_L=20log\left(d_{\mathrm{km}}\right)+20log\left(B_{\mathrm{kHz}}\right)+32.45\left(\mathrm{dB}\right)$$

(7)

The task transmission delay ${\tau }_{tr}^t$ is measured based on the bandwidth of the channel, the size of the offloaded task $K_n$, the transmission power of the user device’s channel, and the distance to the $F_m$.

$$\begin{array}{c}{T}_{of{f}_k}^t={\displaystyle \frac{K_n^t{d}_{m,f}}{{\tau }_{tr}^t}};\hfill \\ {\tau }_{tr}^t={\beta }_{u,f}^t·log\left(1+{\displaystyle \frac{h_{u,f}^t·{\zeta }_{u,f}^t}{\sigma }}\right)\hfill \end{array}$$

(8)

We model the channel transmission rate ${\tau }_{tr}^t$ from node $f_m$ to node $f_j$ according to [42], where ${\beta }_{u,f}^t$, $h_{u,f}^t$, ${\zeta }_{u,f}^t$, and $\sigma$ are the bandwidth, the channel power gain, the transmission power, and the standard deviation of the noise power gain, respectively. Therefore, the sum of tasks offloaded to adjacent $F_j$, at the transmission time, is guided by Equation (8). We assume that the maximum available transmission power at a fog node corresponds to the power required for offloading the maximum number of tasks as presented in [23]. Accordingly, the node is capable of transmitting $p_j^t$ tasks within a single time slot, such that the transmission energy consumption does not exceed the maximum permissible value ${\zeta }_{f,\left(tr\right)}^t\le {\zeta }_{f,\left(tr\right)}^{\left(max\right)}$. The signal-to-noise ratio $\left(SNR\right)$ is described as $SNR={\displaystyle \frac{P_{signal}}{P_{noise}}}$. The noise power ratio and channel capacity correlations are designed according to Shannon’s Law by Equation (9) to present the capacity-dependent relationship in the wireless network.

$$C=\beta lo{g}_2(1+(S/N))$$

(9)

where $C,\beta ,S,N$ are the channel capacity, channel bandwidth, average received signal power, and average noise power, respectively.

3.4.2. F2F Communication

Fog-to-Fog (F2F) communication is triggered when the originating fog node $F_m$ cannot satisfy the quality of service (QoS) demands of a task from user $U_n$. Subsequently, a direct offloading process occurs between $F_m$ and neighboring fog nodes $F_j$ to enable parallel execution of the partitioned sub-task on $F_j$. Consequently, the transmission and computation delays within these fog layers are considered in our analysis. The communication network connecting fog nodes $F_m$ and $F_j$ is modeled as a flat, wired, fiber, optic Ethernet-based infrastructure. Therefore, the communication cost is evaluated based on transmission delay and propagation delay, both of which depend on the distance between the nodes and the speed of light. The propagation delay between $F_m$ and $F_j$ is calculated as $P_{m,j}={\displaystyle \frac{d_{m,j}}{C_l}}$, where $d_{m,j}$ represents the distance between $F_m$ and $F_j$, and $C_l$ is the speed of light. For task offloading between fog nodes, we assume a negligible noise ratio.

The energy consumed during the transmission of task portions $K_n$ between fog nodes ($F_m$ to $F_j$) in partial offloading accounts for both transmission and reception energy as indicated Equation (10). This consumption is influenced by the channel transmission power, the distance between the nodes, the size of the transferred data, and the energy model of the communication interface.

$$E_{m\to j}^{tr}\left(t\right)=E_{m\to j}^{trans}+E_{m\to j}^{rec}$$

(10)

where $E_{m\to j}^{trans}$ is ${\zeta }_{trans}\times T_{trans}$ and $E_{m\to j}^{rec}$ is ${\zeta }_{rec}\times T_{rec}$, where $\zeta$ and T are power and time consumed during transmission and receiving data, respectively. The transmission power utilized by fog node $f_n$ is calculated based on Equations (11) and (12) as follows

$${\zeta }_{n,\left(tr\right)}^t={\displaystyle \frac{{\beta }_{m,j_i^t}^t·{\sigma }^2}{\eta 1d_{m,j_i^t}-\eta 2}}\left({\displaystyle \frac{k_n{p}_j^t}{2^{T\beta (m,f_n^t)}}}-1\right)$$

(11)

Moreover, the total data transmission rate $d_{k_n}$ for the task $K_n$ over a set of multiple orthogonal channels from $F_m$ is calculated with Shannon’s capacity formula [43] as follows.

$${\tau }_{tr}^{t}total=\sum _{t=1}^T\sum _{n=1}^N{\beta }_n\left(t\right){log}_2\left(1+{\displaystyle \frac{{\zeta }_{n,\left(tr\right)}^t{h}_n\left(t\right)}{\sigma }}\right)$$

(12)

where ${\beta }_n\left(t\right)$, $h_n\left(t\right)$, ${\zeta }_{n,\left(tr\right)}^t$, and $\sigma$ represent the nth channel bandwidth, channel gain, allocated transmission power, and noise power spectral density, respectively.

The transmission and communication delay between $\left(F_m\right)$ and $\left(F_j\right)$, for each individual sub-task, can be calculated Equation (13)

$${\tau }_{m,j}^o={\displaystyle \frac{k_{n,j}\times {\tau }_{tr}^{t}total}{{\beta }_{m,j}}}+P_d$$

(13)

where $P_d$ and ${\beta }_{m,j}$ are the propagation delay and the achievable data transmission rate between $F_m$ and $F_j$, respectively. Therefore, the total sub-task transmission latency can be computed using Equation (14) as the average of $T_{m,j}^o$ such that

$${\tau }_{total}^o=\sum _{t=1}^T\sum _{j=1}^N{\tau }_{m,j}^o$$

(14)

where, n is the number of sub-tasks $k_{n,j}$ offloaded to $f_j$.

3.4.3. Local Computations

Assume $T_{F_m}^l$, $V_k^l$, and $C_n$ denote the local computing delay to compute the task portion of $K_n$, the task size at $F_m$, and the number of CPU cores, respectively. So, the sum of all tasks executed locally at $f_m$ offloaded from $U_n$, the  delay can be expressed by Equation (15)

$${\tau }_{f_m}^l=\sum _{t=1}^T\sum _{m=1}^N{\displaystyle \frac{V_k^l}{C_n}}·p_m$$

(15)

where $C_n$ depends on the total task size $V_k$ and $b_u\left(t\right)$ to measure the required intensity of computational effect at the single node to complete its assigned portion. Energy is not a major concern in the fog layer; energy consumption is expressed in Equation (16) measured per CPU cycle ${\zeta }_n$, and computing tasks $K_n$ are considered to calculate energy as follows.

$$E_{F_m}^l=\sum _{t=1}^T\sum _{m=1}^N({\zeta }_n\times K_n^e)$$

(16)

3.4.4. Offloading Computation Delay

As highlighted before, a fog node $\left(F_m\right)$ accepts full tasks from $U_n$ until its maximum tolerable capacity and stops accepting when its workload queue is full. Then, $F_m$ offloads the task to the cooperating adjacent $\left(F_j\right)$ node when it is unable to achieve the objective of the task’s maximum tolerable thresholds. The computation delay for each $F_j$ is computed as follows in Equation (17):

$${\tau }_{c,j}^o={\displaystyle \frac{k_{n,j}\times V_{k_n}^o}{C_j}}$$

(17)

where $V_{k_n}$ represents the total size of task $k_n$ and $C_j$ denotes the maximum number of CPU cores available on fog node $F_j$. Following the parallel task execution model presented in [28], we consider scenarios where multiple sub-tasks are executed concurrently across several fog nodes. Therefore, the overall task computation delay is determined by the longest execution time among all its sub-tasks computed by Equation (18).

$${\tau }_{C,max}^o=max({\tau }_{C,j_1}^o,{\tau }_{C,j_2}^o,\dots ,{\tau }_{C,j_n}^o)$$

(18)

In this case, Equation (19), the total delay ${\tau }_{m,j}^t$in $F_m$ and $F_j$ is the total transmission latency ${\tau }_{total}^o$ plus task execution latency ${\tau }_{c,max}^o$ and local computation delay ${\tau }_{f_m}^l$.

$${\tau }_{m,j}^t={\tau }_{total}^o+{\tau }_{c,max}^o+{\tau }_{f_m}^l$$

(19)

We note that our work primarily focuses on task offloading between fog nodes, assuming that the user device has already offloaded the task to an initial fog node $F_m$. Therefore, we assume that tasks in the fog node queue are handled on a first-come, first-served (FCFS) manner without accounting for contention dynamics or task prioritization.

3.5. Energy Consumption

The energy consumption for each $F_j$ can be calculated as follows using Equation (20).

$$E_{f_j}^o={\zeta }_n\times k_{n,j}^e$$

(20)

Whereas, the total task execution energy consumption when tasks are executed in multiple $F_j$s is given by Equation (21):

$$E_{j,\mathrm{total}}=\sum _{t=1}^T\sum _{j=1}^N{E}_{f_j}^o$$

(21)

Since the partial offloading model is applied in this work, the total system-wise task computation delay $T_{total}$ of the task considers both $F_m$ and $F_j$ as indicated in Equation (22).

$${\tau }_{total}={\tau }_{F_m}^l+{\tau }_{F_j}^o$$

(22)

Moreover, in the partial task offloading approach, the total energy consumption $E_{total}$ considers both task execution at local $F_m$ and adjacent $F_j$ as well as energy consumption at the transmission of task $K_n$. Therefore, $E_{total}$ is defined at Equation (23) as follows:

$$E_{m,j}^t=E_{F_m}^l+E_{j,\mathrm{total}}+E_{m,j}^{tr}$$

(23)

3.6. Problem Formulation

We now formulate the optimization problem with the aim of minimizing the overall system latency and energy consumption in Equation (24a), thereby improving response time and optimizing energy efficiency as follows:

$$\underset{p_n}{min}\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}\sum _{t=1}^T\sum _{j=1}^N\left({\omega }_t{\tau }_{m,j}^t+{\omega }_e{E}_{m,j}^t\right)$$

(24a)

subject to

$$C1:p_m+\sum _{j=1}^n{p}_j=1$$

(24b)

$$C2:0\le p_m,\left\{\sum _{j=1}^n{p}_{j,1},\dots p_{j,n}\right\}\le 1,\forall m,j\in F$$

(24c)

$$C3:\sum _{t=1}^T\sum _{n=1}^N{p}_j\times k_{n,j}\le C_j,\forall j\in F,\forall k\in K$$

(24d)

$$C4:{\tau }_{total}\le {\tau }_{max}{K}_n,\forall n\in N$$

(24e)

$$C5:E_{m,j}^{exc}\times K_n\le E_{threshold},\forall n\in N$$

(24f)

$$C6:\sum _{k=1}^N\sum _{p=0}^1{K}_n\le K_{n,max},\forall m\in M$$

(24g)

where N, T, $\left({\omega }_t\right)$, and $\left({\omega }_e\right)$ are the number of fog nodes, time steps, and weight coefficients of computation delay and energy consumption for both local and offloading cases. The values of $\left({\omega }_t\right)$ and $\left({\omega }_e\right)$ are within the interval $[0,1]$ and satisfy the constraint $\left({\omega }_t\right)+\left({\omega }_e\right)=1$. The objective function is formulated to ensure that a user device offloads a task to a specific fog node in a particular time step t. In this optimization problem the constraints are presents:

• $C1$ (24b) defines the total task execution ratio between $f_m$ and multiple $f_j$s should not exceed the maximum (i.e., ≤1).
• $C2$ (24c) defines the probability of offloading ratio ranges from $\left(0\right)$ up to 1
• $C3$ (24d) ensures that the size of the offloaded task to a fog node should not exceed the node’s maximum capacity.
• $C4$ (24e) shows the latency constraints concerning the maximum tolerable delay threshold $\left({\tau }_{max}\right)$ of offloaded tasks in terms of transmission and execution deadlines,
• $C5$ (24f) verifies that the available energy in the fog node remains above a specified threshold.
• $C6$ (24g) the sum of the offloaded tasks should not exceed the maximum task.

This problem is in the class of NP-hard problems [34]. Therefore, we will discuss our proposed approach in Section 4.

From network historical data to train the DL-based model and replay buffer $\left(\mathcal{D}\right)$ data for the actor-critic RL network to predict resources, such as bandwidth, channel condition, task queue, memory, and others, for the future state prediction. This MADRL model recognizes the availability and utilization of resources as a complex optimization problem with dependencies and interactions among multiple variables. The dynamic network and resource context create a problem for the agent to converge under optimal conditions. According to [44], in F2F cooperation, they share their updated state and maintain a dynamically updated list of the best nodes, as framed in

Table 4. Cooperative $F2F$ sample extracted knowledge frame.

Fog ID	Status	CPU Core	CPU Usage (%)	Memory Usage (%)	Bandwidth	Queue	Idle Time (%)	Task Processed	Next Hope
$F_1$	Busy	8	70%	50%	100 Mbps	$[K_1,K_2,K_3]$	30%	10	$F_2$

.

The agent partially observes local states such as workload, available resources, energy status, latency to other fog nodes, QoS metrics, task nature, node status (active, idle, or sleep), and performance metrics from historical data from each fog node. According to [28], this time-varying context is reflected in the topology designed in Figure 2 for interactions of the user device, and the status of the fog node can be characterized in terms of distance, resources, and task computation ability. Therefore, MAFCPTORA is designed to maximize the overall return accumulated over time, rather than the immediate reward associated with a particular decision.

3.7. TD3 Algorithm Overview

TD3 is a DRL algorithm designed for tasks with continuous action spaces. It is based on the Deep Deterministic Policy Gradient (DDPG) algorithm and incorporates improvements like Clipped Double-Q Learning and Target Policy Smoothing to enhance stability, increase learning efficiency, and minimize overestimation bias compared to DDPG [18]. Two value functions estimate state-action pair values in a clipped Double-Q Learning, and the minimum estimate is used as the goal value during updates. The target value clips a specific range, which helps to mitigate overestimation problems that are frequently seen with single Q-value estimators. The offloading action used the target policy smoothing from the Q-learning target based on target policy $\left(\mu {\theta }_{targ}\right)$ with clipped noise, and then the action satisfied the probability of $({\alpha }_{low}\le \alpha \le {\alpha }_{high})$. Therefore, the target offloading actions are defined as

$${\alpha }^{\prime }\left(s^{\prime }\right)=clip(\mu {\theta }_{targ}\left(s^{\prime }\right)+clip(ϵ,-c,c),{\alpha }_{low},{\alpha }_{high})$$

(25)

Equation (25) adds exploration noise to adapt the deterministic policy into a stochastic behavior during the critic update, which helps stabilize learning. A TD3 agent is an actor-critic DRL agent that aims to find the optimal policy to maximize the expected cumulative rewards over the long term. The actor network determines the action, such as the offloading decision or ratio distribution. In contrast, the critic network evaluates the value (expected reward) of taking that action in a given state.

4. A Decentralized Mult-Agent TD3 Based Task Offloading Architecture

The task offloading problem in Equation (24a) is fundamentally modeled as a Markov decision process (MDP) as presented in [41], implying that the next state depends only on the current state and the chosen action. In more realistic scenarios where the agent only has a limited view, the problem is framed as a Partially Observable MDP (POMDP). This context affects the model’s decisions and long-term performance because it focuses on the immediate maximum value. To address this scenario, the decentralized partially observable MDPs (Dec-POMDP) approach was applied to a multi-agent setting as depicted in Figure 4. For this, a multi-agent system task offloading decision-making process involves independent TD3 agents deployed in each fog node. These agents observe the environment and make their decisions based on their observations. The environment’s state includes dynamic factors relevant to decision-making.

Each agent typically only partially observes local states, which include information like workload, available resources, energy status, latency to other fog nodes, QoS metrics, task nature, node status (active, idle, or sleep), performance metrics from historical data, channel condition, location, and distance between fog nodes.

In cooperative multi-agent environments, policies aim to optimize a joint objective, often framed under Dec-POMDPs. These systems pose challenges such as an exponentially growing joint state-action space as the number of agents increases and non-stationarity due to dynamic network and resource availability [45]. These dynamics are formally framed with the MDP tuples $(s,a,s^{\prime },r,d)$ where:

• State Space $\left(s\right)$: In this task offloading problem, referring to the state of the fog $\left(F\right)$ at time step $\left(t\right)$ observed by the agent from the environment. It includes the task node $\left(F_m\right)$, resource status, and channel condition, modeled as a set of $s_t=\{F_t,R{S}_t\}$, $\forall s_t\in S$, where $F_t$ represents a set of fog nodes and $R{S}_t$ a set of resources at time step t.
• Action Space $\left(a\right)$: the MATD3 agents observe the state and make a decision of action $\left(a_t\right)$ based on partial offloading. The agent’s decisions are based on partial observability of the system state. We consider a continuous action space where some tasks are executed at $F_m$, and the remaining are offloaded to an adjacent fog node $F_j$, $(F_m\to F_j)$ as a set of actions $a_t\in A$ is the action in time step t. The set of actions at each time step t is defined as $(p_0^0,p_n^1,\dots ,p_0^R),\dots (p_n^0,p_n^1,\dots ,p_n^R)$, where $p_n^R$ represents the ratio of the task n allocated to available resource R.
• Reward $\left(r\right)$: $r(s,a,s^{\prime })$ is the reward when action a is taken in state s, and the next state is $s^{\prime }$. Where the reward is determined with the state transition function $\left(s^{\prime }\right|s,a)$ with the probability that state $\left(s^{\prime }\right)$ happens after action $\left(a\right)$ is taken in state $\left(s\right)$.
• Done (d): Whether the TD3 agent in the fog environment reaches the terminal state or not is communicated by this element. In this environment, the agent knows that the termination occurs after a hundred offloading decisions or if an offloading action causes $F_n$ resources to enter a busy state.

A multi-user ($U_n$) and multi-fog ($F_{m,j}$) network environment is considered to simulate decentralized training, and the agent computes tasks in a distributed manner. In the MADRL setting, the agent explores the environment and stores the replay buffer ($\mathcal{D}$) experience as an MDP table $(s_t,a_t,r_t,s_{t+1})$ to train the target network. The current state of the agent is updated with real-time data observed from the environment. After that, the individual agent $F_{m,j}$ can make independent task offloading decisions for the collective result. Depending on the reward design, policies can be trained using either local or collective feedback. Practically, the agent learns the task offloading policy from the network and observes the real-time task and resource information to make an online decision. A decentralized training and execution are used so that each agent updates its actor based on observation for joint actions of agents to enhance collective reward.

The agent’s primary goal is to maximize the overall return accumulated over time, rather than focusing solely on immediate rewards. The collective reward function is often structured to penalize (use negative values for) metrics like latency and energy consumption. In this case, to deal with the instability of the data set, the Min-Max normalization is applied to stabilize reward variation as illustrated by Equation (26).

$$x_{norm}={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}}$$

(26)

Among them, $x_{norm}$ is the normalized data, x is the original data, and $x_{max}$ and $x_{min}$ are the maximum and the minimum values in the replay buffer data set $\left(\mathcal{D}\right)$, respectively. The TD3 agent is utilized as an off-policy algorithm with a deterministic policy $\mu (s;\theta )$ and twin Q-value functions $Q_1(s,a;w_1)$ and $Q_2(s,a;w_2)$ to estimate the expected return. The goal of the reward design is to guide agents toward minimizing normalized latency and energy consumption during task offloading. This is represented using Equation (27) as follows

$$r_{m,j}=-({\omega }_t·T_{normal}+{\omega }_e·E_{\mathrm{normal}})+r_{\mathrm{penalty}}$$

(27)

In a Markov decision process (MDP) setting, Equation (28) is used to calculate the cumulative reward at time $\left(t\right)$ as:

$$r_t=\sum _{k=0}^N{\gamma }^k{r}_{t+k+1}$$

(28)

where $(\gamma \in [0,1\left]\right)$ is the discount factor that determines the influence of future rewards. Due to the stochastic nature of the environment, a shaped reward is computed in Equation (29) using an exponential moving average to ensure stable learning and consistent offloading decisions:

$${\tilde{r}}_t=(1-\alpha )·{\tilde{r}}_{t-1}+\alpha ·r_t$$

(29)

In decentralized multi-agent settings, this shaping is done separately for each fog agent i, allowing agents to adapt independently to local dynamics. We adapted MADRL scenarios from [23], where agents are designed to cooperate to achieve a common goal, typically by sharing a single reward function that reflects the performance of the entire system. This means all agents work towards maximizing the same objective of minimizing total system delay and energy consumption as a set common objective for multiple agents. Agents may share their local rewards as feedback [35]. This approach reduces communication overhead compared to sharing observations and actions. The learning mechanism in such environments relies on DRL principles. The Bellman Equation (30) helps approximate the action-value function $Q(s,a)$ during value iteration as follows:

$$Q(s,a)\leftarrow r+\gamma ·\underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime })$$

(30)

In our case, we utilize the TD3-based MAFCPTORA algorithm to identify the optimal deterministic policy from stochastic environments. The TD3 method is particularly suited for continuous action spaces [18], where the actor network learns a policy $\mu (s;\theta )$ and the critic evaluates its quality using twin Q-value networks:

$$\mu :\mathcal{S}\to \mathcal{A}$$

5. Algorithm Design

To address the aforementioned task offloading problem, we employ a decentralized, Multi-agent, Fully Cooperative Partial Task Offloading and Resource Allocation (MAFCPTORA) algorithm based on the actor-critic network. The MAFCPTORA algorithm is designed in two steps: task offloading decision and resource allocation.

Proposed MAFCPTORA-Algorithm

We designed a novel Multi-agent, Fully Cooperative Partial Task Offloading and Resource Allocation (MAFCPTORA) algorithm based on probability using the TD3 approach. A decentralized TD3 agent is deployed at each $F_n$ and is responsible for their own independent decision to find the optimal offloading policy. The TD3 agent is utilized as an off-policy algorithm with a deterministic policy $\mu (s;\theta )$ and twin Q-value functions $Q_1(s,a;w_1)$ and $Q_2(s,a;w_2)$ to estimate the expected return. For this task offloading problem, DDPG was used in [13]; however, overestimating Q-values poses a challenge for policy breaking due to exploiting errors in the Q-function. We employed TD3, which uses two Q-value functions, $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$, trained by minimizing the Mean Squared Bellman Error (MSBE) to ensure stable value estimation, Equation (31). The clipped double-Q learning is defined where both of the Q-functions use the single target to find a smaller target value using the two Q functions:

$$y(r,s^{\prime },d)=r+\gamma (1-d)\underset{i=1,2}{min}{Q}_{{\varphi }_{i,\mathrm{targ}}}\left(s^{\prime },a^{\prime }\left(s^{\prime }\right)+ϵ\right)$$

(31)

In this case, both Q-functions learn by regressing to the target as Equations (32) and (33):

$$L({\varphi }_1,\mathcal{D})={\mathbb{E}}_{(s,a,r,s^{\prime },d)\sim \mathcal{D}}\left[{\left(Q_{{\varphi }_1}(s,a)-y(r,s^{\prime },d)\right)}^2\right]$$

(32)

$$L({\varphi }_2,\mathcal{D})={\mathbb{E}}_{(s,a,r,s^{\prime },d)\sim \mathcal{D}}\left[{\left(Q_{{\varphi }_2}(s,a)-y(r,s^{\prime },d)\right)}^2\right]$$

(33)

The parameter $\left(\mu \theta \right(s\left)\right)$ is then updated by carrying out the gradient descent of the performance measure $Q\left(\varphi 1\right)$. This MAPFCTORA algorithm is decomposed into two sequential stages, as outlined in Algorithm 1 and Algorithm 2, which correspond to task offloading and resource allocation $R_{f,k}\left(t\right)$, respectively.

Algorithm 1 Multi-agent cooperative Partial Task Offloading

1: Input: initial policy parameter $\theta$, Q-function parameter ${\varphi }_1$, ${\varphi }_2$, number of fog nodes $F_n$, learning rate $\alpha$, discount factor $\gamma$, number of training episode e, agent id i, time $\tau$, replay buffer size $\mathcal{D}$, exploration rate $ϵ$ 2: Initializ: ${\theta }_{\mathrm{targ}}\leftarrow \theta$, ${\varphi }_{\mathrm{targ},1}\leftarrow {\varphi }_1$, ${\varphi }_{\mathrm{targ},2}\leftarrow {\varphi }_2$ 3: Output: Optimal policy $\pi$ 4: for each $F_m\in \{F_1,\dots ,F_n\}$ do 5:     Initialize $F_m$, ${\mu }_{{\theta }_m}$, $Q_{{\varphi }_{1,m}},Q_{{\varphi }_{2,m}}$, and targets ${\widehat{\mu }}_{{\theta }_m},{\widehat{Q}}_{{\varphi }_{1,m}},{\widehat{Q}}_{{\varphi }_{2,m}}$ 6:     Observe initial state at $F_m$$S_m\left(0\right)$ 7: end for 8: for each decision step $t=0,\dots ,\tau -1$ do 9:     for all fog nodes $F_m$ in parallel do 10:         Generate exploration noise: $a_m\left(t\right)={\mu }_{{\theta }_m}\left(S_m\left(t\right)\right)+\mathcal{N}(0,\sigma )$ 11:         Clip action: $a_m\left(t\right)\leftarrow \mathrm{clip}(a_m\left(t\right),a_{low},a_{high})$ 12:         With probability $P_m$, select random $a_m\left(t\right)$ (exploration) 13:         Compute offloading ratio: $x_{m,j}\sim \pi \left(x_{m,j}\right|s_m\left(t\right))$ 14:         Allocate task portions $K_{x_{m,j}}$ to neighbor fog nodes $F_j$ 15:         Execute offloaded tasks at $F_j$ and remaining $x_m=1-{\sum }_j{x}_{m,j}$ locally 16:         Observe $s_{m,j}(t+1)$ and compute reward using Equation (29) 17:         Store transition $(s_{m,j}\left(t\right),a_{m,j}\left(t\right),r_{m,j}\left(t\right),s_{m,j}(t+1),d)$ in $\mathcal{D}$ 18:     end for 19:     for all fog nodes $F_j$ in parallel do 20:         if enough samples in $\mathcal{D}$ then 21:            Sample batch $B=\left\{(s,a,r,s^{\prime },d)\right\}$ from $\mathcal{D}$ 22:            Compute target actions: $a^{\prime }={\mu }_{{\theta }_{targ}}\left(s^{\prime }\right)+\mathcal{N}(0,\sigma )$ 23:            Compute target Q-value using Equation (31) 24:            Update critics ${\varphi }_1,{\varphi }_2$ via gradient descent: $${\varphi }_i\leftarrow {\varphi }_i-\alpha {\nabla }_{{\varphi }_i}{\mathbb{E}}_B\left[{(Q_{{\varphi }_i}(s,a)-y)}^2\right]$$ 25:            Update actor $\theta$ via deterministic policy gradient ascent using Equation (34) 26:            Update target networks using Equation (35) 27:         end if 28:     end for 29: end for

A deterministic policy gradient to update the two critics ${\varphi }_1,{\varphi }_2$, Equation (34) and target networks ${\varphi }_{targ,i}$ with Equation (35).

$$\theta \leftarrow \theta +\alpha {\nabla }_{\theta }{\mathbb{E}}_{s\in B}\left[Q_{{\varphi }_1}(s,{\mu }_{\theta }\left(s\right))\right]$$

(34)

$${\varphi }_{targ,i}\leftarrow \rho {\varphi }_{targ,i}+(1-\rho ){\varphi }_i{\theta }_{targ}\leftarrow \rho {\theta }_{targ}+(1-\rho )\theta$$

(35)

In the TD3 algorithm, a deterministic policy $\mu (s;\theta )$ is trained to map the observed state $s\left(t\right)$ to an action $a_i\left(t\right)$. The policy is learned to maximize the Q-value estimated by the critic network $Q_{{\varphi }_1}$ guided by Equation (36):

$$\underset{\theta }{max}{\mathbb{E}}_{s\sim D}\left[Q_{{\varphi }_1}(s,{\mu }_{\theta }\left(s\right))\right]$$

(36)

In this work, maximizing the cumulative reward is the objective of each agent in this decentralized multi-agent system. Furthermore, the cooperation of each TD3 agent (fog node) with the others follows its respective trained policy, denoted as $\mu =({\mu }_1,{\mu }_2,\dots ,{\mu }_n)$. In addition, the expectation $\left(\mathbb{E}\right)$ is defined with the sum over time and the discounted reward by $\gamma$ at each time step t. Then, the TD3 agent $F_{m,j}$’s value function under policy ${\mu }_i$ at state s is defined by Equation (37):

$$V^{{\mu }_i}\left(s\right)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{R}_i(s_t,a_t,s_{t+1})\right],$$

(37)

where $R_i(s_t,a_t,s_{t+1})$ is the reward function at time step t.

The value of agent $F_{m,j}$ in this multi-agent system depends on the collective policy $\mu$ rather than just its own policy ${\mu }_i$. The relationship is given by Equations (38) and (39):

$$V_i^{{\mu }_i^{*},{\mu }_{-i}^{*}}(s,w)\ge V_i^{{\mu }_i,{\mu }_{-i}^{*}}\left(s\right),\forall s\in \mathcal{S}\mathrm{and}{\mu }_i,$$

(38)

where ${\mu }_{-i}^{*}$ represents the set of policies of all other agents except ${\pi }_i$.

Additionally, the extended $ϵ$-Nash equilibrium is defined by the following general formula:

$$V_i^{{\mu }_i^{*},{\mu }_{-i}^{*}}\left(s\right)\ge V_i^{{\mu }_i,{\mu }_{-i}^{*}}\left(s\right)-ϵ,\forall s\in \mathcal{S}\mathrm{and}{\mu }_i.$$

(39)

Here, ${\mu }_i^{*}$ is the best response of agent $F_{m,j}$ to the policies ${\mu }_{-i}^{*}$ of other agents.

Algorithm 2 Multi-agent Fully Cooperative Resource Allocation

1: Input: Trained actor networks ${\mu }_{{\theta }_i}$ for all agents $F_{m,j}$ 2: Initialize task resource requirements: $V_k$, ${\theta }_t$, H, and $F_{m,j}$ resources ($C_f$, memory, $\beta$) 3: Initialize ${\mu }_{{\theta }_i}$ and $Q_{{\varphi }_i}$ for each agent $F_{m,j}$ with random weights ${\theta }_i,{\varphi }_i$ 4: Initialize target networks ${\pi }_{{\theta }_i}^{\prime }$ and $Q_{{\varphi }_i}^{\prime }$ with weights ${\theta }_i^{\prime }\leftarrow {\theta }_i,{\varphi }_i^{\prime }\leftarrow {\varphi }_i$ 5: for epoch $e=0,1,\dots ,E-1$ do 6:     for each decision step $t=0,1,\dots ,\tau -1$ do 7:         Parallelize the following for each agent $F_{m,j}$: 8:         for each $F_{m,j}$ in parallel do 9:            Use trained actor network to decide resource allocation by Equation (40) 10:            Allocate $C_f$, memory, $\beta$ for task processing based on $a_i\left(t\right)$ 11:            Monitor latency, energy consumption, and resource utilization by Equations (24d)–(24f) 12:         end for 13:     end for 14:     Compute feedback metrics using Equation (28) 15:     Update task states for the next decision step Equation (31) 16: end for

In this TD3-based task offloading, Gaussian noise is added to the stochastic policy to encourage exploration. The action selection formula uses policy noise for exploration:

$$a_i\left(t\right)={\mu }_{{\theta }_i}\left(S_i\left(t\right)\right)+\mathcal{N}(0,\sigma )$$

(40)

• $a_i\left(t\right)$: The action chosen by agent $F_{m,j}$ at time t.
• ${\mu }_{{\theta }_i}\left(S_i\left(t\right)\right)$: The output of the actor network parameterized by ${\theta }_i$, given the state $S_i\left(t\right)$.
• $\mathcal{N}(0,\sigma )$: illustrates the Gaussian noise in terms of mean $\left(0\right)$ and standard deviation $\sigma$ Equation (41).

$$\mathcal{N}(0,\sigma )\sim {\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp\left(-{\displaystyle \frac{x^2}{2{\sigma }^2}}\right)$$

(41)

• x: A random variable sampled from the distribution.
• $\sigma$: The standard deviation controls the magnitude of the exploration.

The addition of Gaussian noise ensures that the agent explores actions around the deterministic policy ${\mu }_{{\theta }_i}$ and avoids premature convergence to suboptimal solutions. Moreover, the update critic $Q_{{\varphi }_i}$ is done by minimizing the loss function.

$$L\left({\varphi }_i\right)=\mathbb{E}\left[{(Q_{{\varphi }_i}(S,A)-y)}^2\right]$$

(42)

where $y=r_i+\gamma Q_{{\widehat{\varphi }}_i}(S^{\prime },A^{\prime })$

In this case, $L\left({\varphi }_i\right)$ presents the loss function for the critic network $\left(Q_{{\varphi }_i}\right)$ parameterized by $\varphi$. The loss measures the mean squared error (MSE) between the predicted Q value and the target Q value y. The target Q-value y combines the immediate reward $r_i$ and the discounted future value of the next state-action pair, estimated using the target critic network $Q_{{\widehat{\varphi }}_i}$. We use this MSE update to train the critic to better predict the expected cumulative reward for each pair of state actions. The smallest mean square error (MSE) between the predicted reward of the critic model and the target reward is then calculated, and this error is used to update the twin critic models using delayed backpropagation. Minimizing this loss improves the accuracy of value estimates, which aids the actor in policy optimization.

Algorithms 1 and 2 We update the actor policy ${\varphi }_i$ at each time step $\left(t\right)$, corresponding to the delayed policy update by estimating the Q-value of state and action using Equations (34) and (35). The computational complexity of the TD3-based MAFCPTORA algorithm is influenced by the number of agents (N), state and action dimensions ($d_s$ and $d_a$), batch size (B), and hidden layer width (H). For each agent, the actor and critic networks contribute a forward pass complexity of $O(H(d_s+d_a)+H^2)$. During training, the critic updates dominate the time complexity with $O(2B·(H(d_s+d_a)+H^2))$ per agent. Including the delayed actor updates, the total training step per agent scales as $O\left(B(d_s+d_a+H)H\right)$. Hence, for N agents:

$$O(N·B·(d_s+d_a+H)H)$$

Dynamic task partitioning involves computing and normalizing task distribution vectors across n neighbors, leading to a per-agent cost of $O\left(n\right)$ and a total of $O(N·n)$ per time step. In decentralized environments, communication overhead arises due to state sharing among neighbors, costing $O(N·n·d_s)$ per step. MATD3 demonstrates faster convergence compared to benchmark solutions, including MAIDDPG, MASAC, and MAPPO. Therefore, our MAFCPTORA algorithm maintains scalable performance with linear growth in agents and manageable quadratic growth in network size.

6. Performance Evaluation

In this section, we evaluate the performance of the proposed approach. We also compared it with several existing approaches. Specifically, we used the MAIDDPG, MATD3, MASAC, and MAPPO DRL baseline algorithms. These algorithms work for decentralized training and execution in a continuous action space based on probability. This implementation represents the state as the task state (task size), resource status, channel conditions of connected nodes (allocated or free), and neighboring nodes. These algorithms follow the actor-critic network, where the actor-critic network trains using exploration and exploitation, updating the network based on the rewards received. Meanwhile, the critic observes the global states and actions of all agents $F_{m,j}$ and evaluates the combined actions of the agents. The assumption is that each user device may connect to multiple fog nodes and select multiple nodes to offload its sub-task at a particular time. This selection process is guided by the collective reward function presented in Equation (24a) as the task completion time against the delay threshold.

6.1. Simulation Setting

In this section, the experimental setting adopted for simulation is described. We consider a scenario of a multi-user ($U_n$s) and multi-Fog nodes ($F_n$s) environment. Each ($U_n$s) is equipped with a single-core CPU with a frequency between 1 GHz and 2 GHz, while ($F_{m,j}$s) features multi-core CPUs with 2 to 8 cores and frequencies ranging from 1.6 GHz to 3 GHz [46]. The computing and communication capabilities of $F_{m,j}$s are assumed to be heterogeneous. The bandwidth between $U_n$ and $F_m$ is 250 Kbps up to 54 Mbps, depending on the weight of the task [44]. The data transmission rate between $F_{m,j}$ also ranges to 100 Mbps [44]. The offloading decision is affected by the task arrival rate $(\lambda$) packets per second, as indicated by [18] to measure the workload intensity of $F_{m,j}$. The intensity of the task execution rate $\left(\mu \right)$ at $F_{m,j}$ is measured in terms of the total task executed per episode duration. A custom fog environment is designed based on OpenAI Gym [47] standards.

6.2. Result and Performance Analysis

The simulation results were structured to address key research objectives: improving offloading decisions and minimizing overall system latency and energy consumption in stochastic horizontal fog environments. In comparing these algorithms, we conduct the training based on

Table 5. Simulation parameters.

Parameters	Quantity
Number of User devices	100
Number of Fog nodes	6
Distance between fog nodes $\left(Dis{t}_{m,j}\right)$	1 km to 5 km
Task arrival rate $\lambda$ at $F_{m,j}$	3 × 102 packet per second
Available CPU core ${\nu }_{f_{m,j}}$	[2–4]
Bandwidth between fog nodes $\beta$	50 MHz
Channel condition H	$[0\le x\le 1]$
Computational capacity $U_i^{min}$	[100–1000] MIPS
Computational capacity $F_{m,j_n^{max}}$	50 MIPS
CPU frequency range $F_{m,j_{\mathit{min}}^{max}}$	$[200\times {10}^6$–$15\times {10}^8]$ Hz
Path loss exponent $PL\left(d_{uf}\right)$	2.7
$U_i^{min}$ Channel transmission power ${\tau }_{t,r}^u\left(t\right)$	10 dBm
$U_i^{max}$ Channel transmission power ${\tau }_{t,r}^u\left(t\right)$	30 dBm
Number of episode	10,000
Observation step per episode	100
Learning rate $\alpha$	0.001
Discount Factor $\gamma$	0.99
Weight coefficient ${\omega }_t$ and ${\omega }_e$	0.7 and 0.3

for 10,000 episodes. Each model is able to deal with this stochastic network and shows good learning progress, as depicted in Figure 5. From these perspectives, MATD3 outperforms the rest to return a stable progressive, long-term cumulative reward formulated in Equation (24a). The overall system-wide evaluation metrics are defined based on the average reward, latency, and energy per 100 episodes to get a more stable and reliable performance estimation. Therefore, the average reward, the average latency, and the average energy consumption per episode are recorded every 100 steps.

Figure 5a presents the evaluation reward comparison for the MAPPO, MASAC, MAIDDPG, and MATD3 models. Figure 5b presents the latency evaluation, where TD3 quickly converges and maintains a consistently low response time. In contrast, MASAC struggles with higher latency, exhibiting a significantly longer response time compared to the other algorithms. Figure 5c presents the evaluation of energy consumption per episode, highlighting TD3’s efficiency with notably low energy usage. All evaluation results demonstrate progressive learning for each model competitively. According to Figure 5, the MATD3 algorithm delivers a consistent performance across several metrics and manages the trade-off between offloading decisions, response time, and energy consumption quite well.

The ablation study is conducted in MATD3, our main evaluation target, to examine the sensitivity performance with a change in the weighting parameter $\left(\omega \right)$. Comparative methods are used as baselines without extensive parameter tuning, consistent with their original configurations. Figure 5d shows the ablation study for the effects of the weight coefficient on average reward, latency, and energy consumption. We begin with the fact that the base weight coefficients ${\omega }_t$ and ${\omega }_e$ are set to 0.5, thereby assigning equal priority to latency and energy consumption in the joint optimization objective. The result of the base weight coefficients shows a balanced trade-off. Hence, we sacrificed ${\omega }_e$ to give more priority to minimizing task execution latency over energy consumption. The ablation study indicates that prioritizing latency significantly reduces average latency while energy consumption increases, but it maintains an acceptable range. The best trade-off was observed at ${\omega }_t=0.7$, achieving both low latency and minimum energy consumption. Moreover, the average reward exhibits an overall increasing trend; however, a slight decline is observed at the final stage, likely due to the influence of minimized latency.

Figure 6 illustrates the performance of MATD3 based on individual metrics: average reward, latency, and energy consumption evaluated per episode. The average rewards, latency, and energy performance of MATD3 are stable in convergence with progressive learning.

Figure 6a illustrates the average reward evaluation result, which stays stable in a range between 0.32 and 0.36 across the episodes. Figure 6b shows promising latency, mostly ranging between 0.08 and 0.18s, suggesting relatively best response times. Figure 6c shows the energy consumption laid between 0.6 and 1.4, indicating a tolerable balance between performance and efficiency.

The evaluation findings of MAPPO, MASAC, MAIDDPG, and MATD3 are presented in

Table 6. Summary evaluation result for the four Models.

Model	Ave Reward	Ave Latency	Ave Energy Consumption
MAPPO	0.34 ± 0.01	0.17 ± 0.02	0.96 ± 0.15
MASAC	0.35 ± 0.01	0.20 ± 0.03	0.92 ± 0.14
MAIDDPG	0.32 ± 0.01	0.14 ± 0.0143	0.9998 ± 0.1873
MATD3	0.36 ± 0.01	0.08 ± 0.01	0.76 ± 0.14

Note: Bold values indicate the best performance achieved by the MATD3 algorithm for each metric.

according to the average reward, latency, and energy consumption. Among the models compared, MATD3 shows superior performance in all three metrics. It achieves the highest average reward of (0.36 ± 0.01), indicating more effective task offloading decisions. Furthermore, it significantly reduces the average latency to (0.08 ± 0.01), which is considerably lower than the other three models, suggesting faster task execution. Energy consumption is also minimized at (0.76 ± 0.14), showcasing the model’s efficiency in resource usage. In contrast, MAPPO and MASAC show relatively competitive rewards despite having higher latency and energy consumption. They achieve a nearly similar reward level (0.34 ± 0.01) and (0.35 ± 0.01), respectively. The highest average latency is observed in MASAC, with a score of (0.2 ± 0.03), indicating a lower response time relative to the other models. However, MAIDDPG exhibits the highest average energy consumption at (0.999 ± 0.187) among all four models. The convergence of these models is observed to be stable during the training period, with low standard deviations across metrics, affirming the reliability of the learned policies. Our experimental results further validate the effectiveness of this approach, where parallel F2F task execution enabled the best-performing model, demonstrating its suitability for real-time, distributed applications.

7. Conclusions and Future Direction

In this paper, we addressed the challenges of task offloading in dynamic fog computing environments, particularly under scenarios with high computational demands. We proposed a MAFCPTORA, a decentralized multi-agent deep reinforcement learning approach for partial task offloading in a horizontal fog-to-fog (F2F) architecture. This approach allows fog nodes to make autonomous and coordinated decisions, optimizing sub-task offloading based on real-time resource status, workload, and network conditions. The proposed MAFCPTORA algorithm enhances task execution efficiency by reducing latency and energy consumption while maintaining the quality of service (QoS) for end-user applications. We evaluated the performance of our method against various state-of-the-art MADRL algorithms, including MATD3, MASAC, MAIDDPG, and MAPPO. MATD3 demonstrated superior performance, achieving the highest average reward of 0.36 ± 0.01, the lowest average latency of 0.08 ± 0.01, and the lowest energy consumption of 0.76 ± 0.14. These results validate the effectiveness of our decentralized approach for scalable and efficient task offloading in fog computing environments.

In future work, we plan to consider the task success rate for dynamic resource requirements while optimizing response time. We design the tasks to be split into sub-tasks so that dependency-heavy tasks are planned to be addressed in the extension of this work. Implementing the proposed algorithm in practical application scenarios for deployment is also important, as it provides valuable insights. The absence of a multi-objective optimization framework to jointly balance latency, energy usage, bandwidth constraints, and battery health under dynamic and uncertain network conditions is a major gap. This is another gap we plan to address. Also, we plan to extend our work with a fault tolerance approach when a fog node fails. A fallback mechanism is important in mission-critical tasks.