Research on Offloading and Resource Allocation for MEC with Energy Harvesting Based on Deep Reinforcement Learning

Abstract

Mobile edge computing (MEC) systems empowered by energy harvesting (EH) significantly enhance sustainable computing capabilities for mobile devices (MDs). This paper investigates a multi-user multi-server MEC network, in which energy-constrained users dynamically harvest ambient energy to flexibly allocate resources among local computation, task offloading, or intentional task discarding. We formulate a stochastic optimization problem aiming to minimize the time-averaged weighted sum of execution delay, energy consumption, and task discard penalty. To address the energy causality constraints and temporal coupling effects, we develop a Lyapunov optimization-based drift-plus-penalty framework that decomposes the long-term optimization into sequential per-time-slot subproblems. Furthermore, to overcome the curse of dimensionality in high-dimensional action, we propose hierarchical deep reinforcement learning (DRL) solutions incorporating both Q-learning with experience replay and asynchronous advantage actor–critic (A3C) architectures. Extensive simulations demonstrate that our DRL-driven approach achieves lower costs compared with conventional model predictive control methods, while maintaining robust performance under stochastic energy arrivals and channel variations.

1. Introduction

The increasing number of smart devices and mobile terminals generates a large amount of data to be processed, which also poses a significant challenge to the operation of mobile devices and network systems [1]. The limited computing power and data storage capacity of mobile devices in the processing of application data can lead to high energy consumption and delay in the completion of computing tasks [2]. For computationally intensive real-time applications, computing tasks often cannot be completed within strict time constraints due to the limited computing resources and battery capacity of mobile devices. This not only fails to meet the requirements of ultra-low latency applications but also leads to task abandonment when continuous execution is infeasible, thereby degrading device performance [3].

To address these challenges, MEC emerges as a promising paradigm that deploys computational and storage resources in close proximity to mobile devices and network systems [4]. MEC offloading enables mobile devices to achieve computational agility, low latency, and energy efficiency [5]. Compared with cloud computing, MEC servers reduce task offloading latency and minimize both task transmission and computational energy consumption by being physically near end-users during task execution [6]. Massive tasks generated by mobile devices can be offloaded to nearby MEC servers for processing, alleviating the computational burden on devices and addressing resource limitations [7].

MEC offloading is a novel solution for resource-constrained mobile devices using MEC server computing resources [8]. However, mobile devices face strict battery capacity limitations. They cannot provide continuous energy supply for computational tasks. This energy constraint frequently leads to insufficient power for task processing. Even after offloading, these tasks may still be discarded due to resource shortages. Such interruptions significantly degrade processing efficiency and ultimately reduce service quality [9,10]. Although the power supply problem can be solved using high-capacity batteries and uninterrupted charging, this approach leads to increased hardware costs and the unavailability of mobile devices in certain scenarios [11]. To overcome these bottlenecks, energy harvesting technology has emerged as a promising green MEC candidate that can extend the task processing of battery capacity-constrained mobile devices in a sustainable manner [12] to ensure that tasks can be completed within the allowed time. Mobile devices can harvest energy from solar energy, wind energy, human motion, and many other sources [9]. The collected energy can be used by the mobile device for local task processing or for offloading computing tasks to MEC servers and computing on the cloud. However, due to the dynamic changes and diversity of EH, computational offloading, and channel state information of users in EH-enabled MEC network systems, existing methods struggle to handle these complexities, motivating the use of deep reinforcement learning for adaptive decision making. In addition, the demand of different users’ computational task applications for MEC network system makes the resource allocation problem in MEC network systems more complex [13].

The resource allocation problem in the time-varying EH-enabled MEC application scenario with multiple users and multiple MEC servers is characterized by high parameter dimensionality, making it difficult to obtain accurate a priori statistical information, which leads to an increasingly sophisticated resource allocation problem [14] and poses a challenge for the energy management of users in MEC [3]. Machine learning methods such as deep learning and deep reinforcement learning provide a novel solution to problems that occur within time-varying environments like dynamic offloading and unstable energy harvesting in wireless network communication systems [15,16]. The complexity of the solution increases significantly alongside the growing number of users and computing servers in the MEC network, making it difficult to apply on a large scale [17]. Deep reinforcement learning achieves optimal long-term goals through the enhancement of network edge intelligence in the absence of a priori knowledge, based on the dynamically changing environment at the edge of the MEC network, thereby continuously learning, alternating states, and interacting continuously with the network in the environment [18]. Therefore, existing research leverages artificial intelligence approaches in MEC to empower network edge intelligence techniques with unique advantages in solving MEC computational offloading and resource allocation strategies for dynamic systems.

This paper addresses the joint optimization of computation offloading decisions, resource allocation, and energy harvesting in MEC systems. The goal is to ensure quality of service (QoS) for users while balancing energy consumption, latency, and task discard costs. The computational tasks generated by users are discarded when they are deemed intractable or do not satisfy the constraints, thus yielding task discard costs. The goal is to minimize the overall weighted sum of the overall delay, energy consumption, and discard cost for the users in the MEC network system. We consider the corresponding formulated problem as a mixed-integer nonconvex optimization problem and first propose a value-based iterative reinforcement learning method, namely Q-learning, to obtain an optimal joint optimization scheme for resource allocation and energy harvesting. However, as the number of users increases, the state and action in the system grows exponentially, as well as the data dimensions, which leads to a huge challenge for Q-table maintenance [16]. To address this issue, we further propose a deep reinforcement learning-based method, A3C reinforcement learning, which can effectively handle dynamic and complex problems. In summary, Q-learning and A3C form a hierarchical solution framework. The former validates the RL approach in simplified settings, while the latter addresses the scalability and complexity challenges of real-world MEC systems. This combination is not redundant but strategically complementary, ensuring both theoretical rigor and practical utility. It demonstrates a systematic progression from foundational RL techniques to advanced deep RL methodologies, tailored to the problem’s dual needs of clarity in small-scale modeling and efficiency in large-scale deployment.

The main contributions of this paper can be summarized as follows:

• We design an MEC network system based on energy harvesting, adopting nonlinear energy harvesting techniques. This system focuses on the study of wireless and computing resource allocation as well as offloading decision optimization in multi-user and multiple MEC server computing scenarios. Users can choose among local computing, offloading computing, and task discarding modes. The main problem is formulated as minimizing the total cost composed of the energy consumption, delay, and task discarding cost. Meanwhile, we jointly optimize bandwidth allocation, power allocation, computing resources, local CPU frequency, and energy harvesting.
• Due to the time-varying nature of the user’s energy collection in this EH-enabled MEC system, a Lyapunov-based architecture is employed to design a Lyapunov penalty plus offset function. This leads to the formation of a problem that aims to minimize the total cost by weighting the delay, energy consumption, and task discard cost.
• Subsequently, we propose MDP decisions, design action and state, determine reward functions, and propose a Q-learning approach to solve the problem of resource and offload allocation optimization.
• To solve the problem of large Q-matrix data, the A3C method is further proposed, and the A3C-based algorithm is designed.

2. Related Work

Research on computational offloading techniques in MEC has attracted extensive attention from industry and academia. Related studies mainly focus on improving energy efficiency, optimizing resource allocation, and reducing latency. The main related works are listed in

Table 1. Main comparison of related work.

Existing Works	Methodology	Optimization Objectives	Experimental Scenarios	Consider Constraints
[19]	Multiobjective optimization problem	Achieve maximum revenue and maximum service utilization	For mobile devices with latency and resource constraints	Task delay
[20]	Nested algorithm	Minimize the total energy consumption of the system	Compute offload wireless charging	Task delay
[21]	Genetic algorithm, heuristic algorithm	Minimize task delay	Mobile perception scene	Delay constraint, power constraint
[22]	Lyapunov optimization theory	Minimize the long-term average energy efficiency of the network	A multi-user and multi-server MEC Internet of Things system	Computing resources, power resource constraint
[23,24,25,26]	Lyapunov optimization theory	Minimize system cost	Multi-server MEC system	Energy constraint, task queue stability constraint
[27]	Deep reinforcement learning framework	Minimize the energy time cost (ETC) of MD	Multi-task MEC system	Delay constraints
[28]	A3C Deep Reinforcement Learning algorithm	Minimize the energy consumption of the MEC system and maximize the throughput of the blockchain system	Data sharing in MEC	delay constraints
[29]	DDPG	Minimize the long-term average delay of all computing tasks	Dynamic resource management in MEC	Power constraints, energy consumption constraints, computer resource constraints
[30]	The ACDQN algorithm of deep reinforcement learning	Minimize the weighted sum of energy consumption and delay	NOMA assists the MEC system	Channel resource constraints
[31]	Lyapunov-guided deep reinforcement learning	Maximize the weighted sum calculation rate	Multi-user MEC	offloading decision constraints, resource allocation constraints

.

2.1. Computational Offloading and Resource Allocation Study

For latent and resource-constrained mobile devices, an adaptive service offloading scheme is conceived for MEC with maximum revenue and maximum service utilization [32]. The authors in [19] studied the optimization problem of the resource management of offloading the transmission of the MMWave MEC system from the data transmission requirements of communication-oriented users and computing-oriented users. They transformed the MOP problem into a single-objective optimization problem (SOP) and proposed a three-stage iterative resource allocation algorithm under the premise of ensuring Pareto optimality. The authors in [20] designed a computationally offloading wireless charging scheme. This scheme optimizes data division, time allocation, transmission power, and energy beam formation using a new nested algorithm to minimize the total system energy consumption under power and time delay constraints. The authors in [21] designed a device-to-device collaborative MEC-based system and proposed a user mobility-aware task offloading architecture. This architecture jointly addresses task allocation and power optimization under user mobility, distributed resources, task attributes, and user device energy constraints, with the goal of minimizing task latency. The authors in [22], in order to solve the trade-off between delay and energy efficiency, studied an online offloading and resource allocation algorithm based on Lyapunov optimization theory to achieve the trade-off between EE and service delay $\left[O\right(1/V),O(V\left)\right]$.

However, these studies lack a comprehensive view on integrating energy harvesting with MEC in multi-user multi-server scenarios while handling multiple costs. Our paper fills this gap by designing an EH-based MEC system, using a Lyapunov architecture; we apply Q-learning and the A3C algorithm to optimize resource allocation and offloading decisions more effectively.

2.2. Energy Harvesting-Driven MEC Systems

Energy harvesting has been extensively studied as a technology that can continuously harvest various types of energy to support the continuous operation of user devices. EH has been integrated into MEC network systems to support user devices for long-term computational offloading. In contrast to the previous study, in [23], the authors deploy energy harvesting devices on edge servers to form a hybrid green energy–grid supply model. In [9], the authors combine social network sensing and energy harvesting techniques and propose a dynamic computational offloading scheme that takes a queuing theory approach to model energy consumption and delay performance in the system, and they utilize game theory to address the interactions within the design community. In [24], the authors furthered proposed a task allocation strategy for an EH-based MEC system, constructing a task queue and using Lyapunov optimization to design an online dynamic task allocation algorithm for effectively balancing device energy consumption and execution latency and enhancing system performance. The researchers in [25] proposed an energy harvesting scheme in which devices first collect energy from the base station, and then utilize the harvested energy for partial or full data transmission back to the same base station. By jointly optimizing the stopping rule for energy harvesting and the allocation of fading blocks during data transmission, they maximize the average achievable data rate. To solve the coupling problem among the causal energy constraint, the offloading ratio and the allocation of resources in the computational unloading process, local and remote computational models and energy harvesting models are established in [26], and the Lyapunov method was used to find the optimal solution for multi-user partial offloading.

Prior work on EH in MEC includes diverse approaches like hybrid energy models and Lyapunov-based optimizations, but lacks comprehensive handling of multi-user multi-server scenarios with nonlinear EH while optimizing resources and minimizing total costs. Our paper bridges this gap, designing an EH-based MEC system with nonlinear EH and using Lyapunov architecture and Q-learning via MDP decisions to optimize resource and offload allocation and minimize total costs more effectively.

2.3. Reinforcement Learning in MEC

Although the above research work considers some resource allocation and optimization problems in terms of MEC computation offloading supported by EH, when the number of users and servers in the MEC network system is large in scale, it enhances the complexity of the problem, so it is worthwhile to study the work that combines artificial intelligence methods applied with the MEC network system. In [27], a deep reinforcement learning framework based on an actor–critic learning structure was proposed due to the difficulty for traditional numerical optimization methods in addressing the strong coupling between combinatorial offloading decisions and task execution. In [28], in an MEC system with blockchain support, the asynchronous learning method was used to solve the optimization problem of secure data sharing. In [29], the dynamic resource management problem of power control and computational resource allocation in industrial IoT MEC using deep reinforcement learning is investigated. The researchers in [30] proposed a deep reinforcement learning algorithm to jointly optimize the partial computation offloading strategy and channel resource allocation problem in a dynamic network environment with time-varying channels. In [33], a time-attentive deterministic policy gradient-based algorithm is proposed. This algorithm is different from traditional reinforcement learning methods and is used to jointly optimize the computational offloading and resource allocation problems in MEC. The researchers in [31] proposed a new framework for joint Lyapunov and deep reinforcement learning to solve the coupled problem of binary offloading and system resource allocation decisions in different time frames in a MEC network system with time-varying channels and randomly arriving user tasks.

Previous MEC research with EH used AI methods for optimization. Yet, with large-scale users and servers, it fails to comprehensively integrate nonlinear EH, multi-user multi-server scenarios, and total cost minimization. Our study bridges this gap by designing an EH-based MEC system with nonlinear EH. Utilizing a Lyapunov-based architecture and Q-learning via MDP decisions, we optimize resource and offload allocation while effectively cutting the total cost, an area previously unaddressed.

Current work mainly focuses on static resource allocation or single-objective optimization but lacks joint optimization for the problem of dynamic energy harvesting coupled with high-dimensional decision making in multi-user multi-server scenarios. In addition, traditional model predictive control methods struggle to guarantee robustness in the face of time-varying channels and stochastic energy arrivals, while existing deep reinforcement learning methods do not effectively incorporate the Lyapunov optimization framework in order to address energy causality constraints [34,35]. In contrast, this study proposes, for the first time, a joint optimization framework of hierarchical deep reinforcement learning and Lyapunov drift penalty, which significantly reduces the complexity of the high-dimensional action by decomposing the long-term optimization into temporal subproblems, while ensuring energy queue stability.

3. System Model

Based on the above study, we propose a novel and more comprehensive study of resource allocation and strategies based on the dynamic offloading of user tasks using energy harvesting techniques. The system model is shown in Figure 1, which consists of mobile devices, Macro Base Stations (MBSs) and MEC servers. In this system model, we assume that there are a number of MDs, denoted as $\mathfrak{n}=\{1,2,\dots ,i,\dots ,N\}$, $i\in \mathfrak{n}$. Each MD is equipped with an EH component, which harvests energy from wireless power supply devices, including solar, wind, and other green energy sources. Meanwhile, in this system model, MEC servers with powerful processing capabilities are integrated into the MBSs of the edge network. The set of MEC servers is defined as $\mathfrak{m}=\{1,2,\dots ,m,\dots ,M\}$, $m\in \mathfrak{m}$. Each mobile device is able to select the associated MBS based on distance and channel status information. Computational tasks can be offloaded to the MEC server for processing. The generation of computational tasks follows the Bernoulli process [26]. The arrival events of computational tasks obey a Bernoulli process [26], i.e., within each time interval t, the task of user i arrives with probability $p_i=1$ and does not arrive otherwise, $p_i=0$. The computational tasks of MDs can be processed locally. However, since the execution of intensive computation applications is constrained by the computing capacity of MDs, it is feasible to offload the computational tasks to MEC servers for computing. As a result, the quality of services and user experience can be further improved [36]. A computational task can also be discarded when the channel state is unstable or affected by the task processing delay.

In this paper, we adopt a time slot model where the length of each time slot is $\tau$ and $t\in \{0,1,\dots ,T\}$ denotes the set of time slots. We presume that the channel states in each time interval are independent of each other and remain in a stable state but vary over different time slots [36]. In time interval t, each mobile device has a task request, and we represent this computational task using a tuple, i.e., $A=<{\lambda }_i\left(t\right),{\tau }_{i,d}\left(t\right)>$, consisting of two parameters. Here, ${\lambda }_i\left(t\right)$ denotes the amount value of the arrived computation task at time slot t. Without loss of generality, ${\lambda }_i\left(t\right)<{\lambda }_i^{max}\left(t\right)$, ${\lambda }_i^{max}\left(t\right)$ is the maximum amount of computation at time slot t, and ${\tau }_{i,d}\left(t\right)$ denotes the maximum delay in computing the task completion. We focus on delay-sensitive applications whose execution duration is not greater than the time slot length, i.e., ${\tau }_{i,d}\left(t\right)⩽\tau$. For ease of reference, we give the key notations of the proposed system in

Table 2. Summary of key notations.

Notations	Meanings
$\mathbb{U}$	The set of users in the system
$\mathfrak{n}$	The set of mobile devices
$\mathfrak{m}$	The set of MBSs
t	The time slot
$\tau$	The length of each time slot
${\lambda }_i\left(t\right)$	The data size of the arrived computation task
${\tau }_{i,d}\left(t\right)$	The maximum delay in computing the task completion
$D_i\left(t\right)$	The set of computation task offloading decision strategies
$x_{i,l}\left(t\right)$	Local computing decision
$x_{i,m}\left(t\right)$	The decision of offloading from MD i to MBS m server process
$x_{i,d}\left(t\right)$	The decision of computation task discarding
$f_{i,l}\left(t\right)$	The computation resource allocation of the local computation of MD i
$f_{i,m}\left(t\right)$	The computation resource allocation of MBS server m for MD i
${\omega }_{i_m}\left(t\right)$	The bandwidth allocation between MD i and MBS m
$p_{i,m}\left(t\right)$	The power allocation of MD i for task offloading
$e_i\left(t\right)$	The harvested energy of MD i in time slot t
$B_i\left(t\right)$	The battery capacity of MD i in time slot t
${\alpha }_i$	The weighting parameter of energy consumption
$\beta$	The weighting parameter of the delay of task completion
${\gamma }_i$	The weighting parameter of the cost of task discarding
$g\left(t\right)$	Channel gains between MDs and MBSs
F	Available MEC server computing resources
${\lambda }_i\left(t\right)$	The data size of the arrived computation task
$H\left(t\right)$	Channel interference between MDs and MBSs

.

3.1. Computing Model

Each computation task can be performed locally on the mobile device or offloaded to an MEC server at the remote network edge for computing. Meanwhile, both computation methods may be infeasible when the mobile device has inadequate energy or the task processing latency cannot be satisfied. Each MD independently makes a decision on how to process its computational tasks. However, these decisions are not entirely independent. MDs can obtain certain information from the MEC servers via a control channel. This information includes, but is not limited to, the current computational load of the MEC server, the available bandwidth for offloading, and the expected delay in processing tasks on the server. Based on this information, MDs dynamically choose between local computation, offloading to the MEC server, or discarding the task. Hence, the computation task may be discarded. We define the computation task offloading decision strategy vector of the proposed MEC EH-enabled system in time slot t as $D_i\left(t\right)=\{x_{i,l}\left(t\right),x_{i,m}\left(t\right),x_{i,d}\left(t\right)|0,1\}$, where $x_{i,l}\left(t\right)=1$ indicates that the computation task of user i performs the computation locally, $x_{i,m}\left(t\right)=1$ represents the computation will be offloaded to the MEC server for executing, and $x_{i,d}\left(t\right)=1$ indicates the computation task is discarded in time slot t. The computation offloading decision schemes should meet the following constraints:

$$x_{i,l}\left(t\right)+x_{i,m}\left(t\right)+x_{i,d}\left(t\right)=1$$

(1)

3.1.1. Local Computation

In this study, we employ a dynamic voltage scaling technique to allocate computational resources when the user chooses to perform computational task processing locally. In a real situation, each MD has varying computational processing power and energy consumption per CPU cycle. Based on this case, we denote $f_{i,l}\left(t\right)$ as the computing capacity of MD i at time slot t, and $f_{i,l}\left(t\right)⩽f_i^{max}$. The time delay for the computing task can be expressed as

$$T_{i,l}\left(t\right)=c_i{\lambda }_i\left(t\right)/\phantom{c_i{\lambda }_i\left(t\right)f_{i,l}\left(t\right)}{f}_{i,l}\left(t\right)$$

(2)

where $c_i$ refers to the computation intensity in CPU cycles per bit [22].

Correspondingly, the energy consumption of locally calculating during time slot t is

$$E_{i,l}\left(t\right)=\tau \omega f_{i,l}^3\left(t\right)$$

(3)

where $\omega$ represents the effective switched capacitance depending on the chip architecture of the device itself.

3.1.2. MEC Server Computation

When the computational tasks of MD i are offloaded to the associated MEC server m for processing, we assume that the computational capacity of the MEC server is limited and the maximum computational capacity is set to be $F_m$.

$$T_{i,m}\left(t\right)=c{\lambda }_i\left(t\right)/\phantom{c{\lambda }_i\left(t\right)f_{i,m}\left(t\right)}{f}_{i,m}\left(t\right)$$

(4)

The energy consumption of computing using MEC server m can be given as [16]

$$E_{i,m}\left(t\right)=p_m{c}_i{\lambda }_i\left(t\right)/\phantom{c_i{\lambda }_i\left(t\right)f_{i,m}\left(t\right)}{f}_{i,m}\left(t\right)$$

(5)

where $p_m$ denotes the power of the MEC server when it processes the task.

In addition, when the computational task of MD i is processed on the MEC server, MD i is also waiting for the results. We assume that MD i is in standby mode during the waiting period, which will also consume a certain amount of energy. We set the power in the standby state as $p_{i,id}\left(t\right)$. Thus, the energy consumption in the standby mode can be expressed as

$$E_{i,id}\left(t\right)=p_{i,id}\left(t\right)c_i{\lambda }_i\left(t\right)/\phantom{c_i{\lambda }_i\left(t\right)f_{i,m}\left(t\right)}{f}_{i,m}\left(t\right)$$

(6)

3.2. Communication Model

Under the coverage of MBSs, the computation tasks of MDs are constrained by the local computational capacity, the surplus battery energy, and the maximum tolerated delay, which makes local computation difficult to execute. In this case, the task request can be offloaded to the associated MEC server for processing. During the process of offloading computation tasks, the coverages of MBSs will overlap, which will lead to signal interference. When the computation tasks of multiple MDs are offloaded to the same MBS simultaneously, we consider using dynamic channel allocation technology to allocate channel bandwidth according to the demands of the MDs. Then, the computation task offload rate between MD i and MBS m can be expressed as

$$r_{i,m}\left(t\right)={\omega }_{i_m}\left(t\right)B_{i,m}{log}_2(1+{\displaystyle \frac{p_{i,m}\left(t\right)g_{i,m}\left(t\right)}{I+{\omega }_{i_m}\left(t\right)B_{i,m}{H}_{j,m}\left(t\right)}}),$$

(7)

where $H_{j,m}\left(t\right)={\displaystyle \sum _{j\in \mathfrak{n},j\ne i}}{p}_{j,m}\left(t\right)g_{j,m}\left(t\right)$ represents the interference from other users j to MBS m, $g_{i,m}\left(t\right)=d_{i,m}^{\sigma }\left(t\right)$, $d_{i,m}\left(t\right)$ denotes the distance between MD i and MBS m, and the path loss factor is set as $\sigma =-4$.

The transmission delay and energy consumption of computation task offloading from MD i to MBS m can be shown, respectively, as

$$T_{i,m}^r\left(t\right)={\lambda }_i\left(t\right)/\phantom{{\lambda }_i\left(t\right)r_{i,m}\left(t\right)}{r}_{i,m}\left(t\right)$$

(8)

$$E_{i,m}^r\left(t\right)=p_{i,m}\left(t\right){\lambda }_i\left(t\right)/\phantom{p_{i,m}\left(t\right){\lambda }_i\left(t\right)r_{i,m}\left(t\right)}{r}_{i,m}\left(t\right)$$

(9)

We express the overall computation delay and energy consumption of MD i in time slot t, respectively, as follows:

$$T_i\left(t\right)=x_{i,l}\left(t\right)T_{i,l}\left(t\right)+x_{i,m}\left(t\right)[T_{i,m}^c\left(t\right)+T_{i,m}^r\left(t\right)],$$

(10)

$$E_i\left(t\right)=x_{i,l}\left(t\right)E_{i,l}\left(t\right)+x_{i,m}\left(t\right)[E_{i,m}^r\left(t\right)+E_{i,m}^c\left(t\right)+E_{i,id}\left(t\right)].$$

(11)

3.3. Task Drop Model

When the remaining energy of the MDs is insufficient to support the tasks generated in the current time interval t for local computation or offloading to the edge servers of the MBS for processing, and when the channel state information from the MDs to the MBS is unstable during the offloading process of the computation tasks, resulting in the phenomenon of channel depth fading, which makes it difficult to offload the tasks successfully, the computation tasks generated in the current time interval will be discarded. Since the discarded tasks will affect the MDs’ task processing, we will impose a penalty on each discarded task, and the penalty cost of the MD for the computation of a discarded task in time interval t is shown as follows:

$$C_i\left(t\right)={\psi }_i{x}_{i,d}\left(t\right){\lambda }_i\left(t\right)\tau$$

(12)

where ${\psi }_i$ denotes the penalty cost of each discarded task of MD i.

3.4. Energy Harvesting Model

Within the proposed MEC offloading network system model based on energy harvesting technology, MDs are able to capture the renewable energy sources to support themselves in task execution. To better describe the energy harvesting situation, we model the energy harvesting process as a serial energy packet arrival process that follows a Poisson process with an average arrival rate $e_i\left(t\right)$. Furthermore, $e_i\left(t\right)⩽e_i^m\left(t\right)$, where $e_i^m\left(t\right)$ is referred to as the rate of maximum energy arrival. The renewable energy processes can characterize stochastic and intermittent natures. The energy packets are independently and identically distributed in different time slots. The harvested energy will be stored in the battery of each MD, which can be used for local computation or computation task offloading in the next time slot. We denote $B_i\left(t\right)$ as the battery energy level at the beginning of time slot t. Without loss of generality, $B_i\left(t\right)<\infty$, $\forall t\in T$.

The total energy consumption should be no more than the battery level at time slot t and satisfy the following condition:

$$E_i\left(t\right)⩽B_i\left(t\right)<\infty ,t\in \mathcal{T}.$$

(13)

Thus, the battery energy level of MD i evolves as below:

$$B_i\left(t+1\right)=B_i\left(t\right)-E_i\left(t\right)+e_i\left(t\right),t\in \mathcal{T}.$$

(14)

4. Problem Formulation

In this subsection, we consider the constraints of energy consumption, execution delay, queue stability, and quality of service. Based on these constraints, we formulate a weighted sum. This weighted sum aims to minimize the long-term average energy consumption, execution latency, and penalty costs for task dropping. Each MD i in each starting time slot t needs to make a decision as to which computation method to employ for the arrived computation tasks, i.e., local computation, MEC server computation, or discarding. In order to minimize the overall weighted sum, we will optimize the offloading decision $\mathbf{x}\left(t\right)=\{x_{i,l}\left(t\right),x_{i,m}\left(t\right),x_{i,d}\left(t\right)\}$, the allocation of MD i’s local computing resources $f_{i,l}\left(t\right)$, the channel bandwidth allocation ${\omega }_{i_m}\left(t\right)$ between MD i and MBS m, the transmission power allocation $p_{i,m}\left(t\right)$, the computing resources $f_{i,m}\left(t\right)$ allocated by the MEC server m for MD i, and the acquired energy $e_i\left(t\right)$. To ensure the QoS of the proposed system, the minimum rate of task transfer between MDs and MBSs is set as $R_m$. For simplicity, the parameter vector used for optimization can be defined as $\mathbf{\Omega }\left(t\right)\triangleq \{\mathbf{x}\left(t\right),f_{i,l}\left(t\right),f_{i,m}\left(t\right),p_i\left(t\right),{\omega }_{i_m}\left(t\right),e_i\left(t\right)\}$. Therefore, the formulated problem is shown below:

$$\mathrm{P}1:\underset{\mathbf{\Omega }\left(t\right)}{min}{\displaystyle \frac{1}{T}}\sum _{t=1}^T[{\alpha }_i{E}_i\left(t\right)+{\beta }_i{T}_i\left(t\right)+{\gamma }_i{C}_i\left(t\right)]$$

$$x_{i,l}\left(t\right)+x_{i,m}\left(t\right)+x_{i,d}\left(t\right)=1,$$

(15)

$$\sum _{i=1}^N{x}_{i,m}\left(t\right)r_{i,m}\left(t\right)⩾R_m,$$

(16)

$$f_{i,l}\left(t\right)⩽f_i^{max},$$

(17)

$$p_{i,m}\left(t\right)⩽p_i$$

(18)

$$\sum _{m=1}^M{x}_{i,m}\left(t\right)f_{i,m}\left(t\right)⩽F_m,$$

(19)

$$\sum _{m=1}^M{x}_{i,m}\left(t\right){\omega }_{i_m}\left(t\right)⩽1,$$

(20)

$$x_{i,l}\left(t\right)T_{i,l}\left(t\right)+x_{i,m}\left(t\right)[T_{i,m}^c\left(t\right)+T_{i,m}^r\left(t\right)]⩽T_i^{max}$$

(21)

$$x_{i,l}\left(t\right)E_{i,l}\left(t\right)+x_{i,m}\left(t\right)[E_{i,m}^r\left(t\right)+E_{i,id}\left(t\right)]<B_i\left(t\right)$$

(22)

$$B_i\left(t+1\right)=B_i\left(t\right)-E_i\left(t\right)+e_i\left(t\right),i\in \mathfrak{n},t\in \mathcal{T}.$$

(23)

where $\{{\alpha }_i,{\beta }_i,{\gamma }_i\}$ represents the weight parameters of energy consumption, delay, and discarding cost in the whole system, respectively. Constraint (15) represents the decision condition of the computing task, and each MD can only choose one task processing method at the beginning time slot. Constraint (16) aims to guarantee the QoS during computational task offloading. The total task offloading rate of all MDs associated with MBS m requires that the lower bound should be greater than $R_m$. Constraints (17) and (18), respectively, represent the constraints on computational resource allocation for the local computation of MDs and the constraints on power optimization for the offloaded computation of MDs. Constraint (19) indicates that the computational resources allocated by MBS m for the associated MDs cannot be greater than $F_m$. Constraint (20) indicates that the sum of the bandwidth allocation ratio of MBS m to MDs is less than 1. Constraint (21) signifies that the computational task for MD i is completed within the maximum allowable delay $T_i^{max}$. Moreover, constraint (22) implies that in time slot t, the energy consumed by MD i for local computation or computation offloading is less than the battery capacity. Constraint (23) represents the formula for the battery capacity in time slot $t+1$.

However, energy constraint (22) leads to a coupled offloading decision of MDs among various time slots, which means the offloading decisions in different time slots are interrelated and thus pose a challenge for the optimization of the decision strategies. Similarly to [3,9], we are able to introduce a nonzero lower bound $E_i^{min}\left(t\right)$ and an valid upper bound $E_i^{max}\left(t\right)$ as the minimum and maximum battery-charging energy in each time slot, respectively. Furthermore, in order to deal with the situation caused by the energy constraint and the coupled offloading decisions, we can eliminate the coupling effect generated by the battery in each time slot and optimize the system performance by neglecting the energy constraint. Thus, problem P1, with the following formulation, can be modified as

$$\mathrm{P}2:\underset{\mathbf{\Omega }\left(t\right)}{min}{\displaystyle \frac{1}{T}}\sum _{t=1}^T[{\alpha }_i{E}_i\left(t\right)+{\beta }_i{T}_i\left(t\right)+{\gamma }_i{C}_i\left(t\right)]$$

s.t.(15) − (21), (23)        

(24a)

$$\begin{array}{cc}\hfill & E_i\left(t\right)\in \left\{0\right\}\cup [E_i^{min}\left(t\right),E_i^{max}\left(t\right)].\hfill \end{array}$$

(24b)

Based on the investigation of [26], we find that due to the energy causality constraint, i.e., when the residual energy of the battery of the MD is insufficient, the computation offloading decisions in the current time slot will affect the offloading decision of the next time interval. As a result, the existence of coupled-energy causal constraints between different time slots, which will lead to complicated offloading decisions and resource allocation. Similarly to [3,36], important parameters such as virtual energy queues and perturbation parameters were introduced for the battery of each MD.

Definition 1.

To handle the energy causality constraint leading to inter-slot decision coupling, we introduced the virtual energy queue as ${\tilde{B}}_i\left(t\right)=B_i\left(t\right)-{\theta }_i$, where ${\tilde{B}}_i\left(t\right)$ indicates the actual battery energy that the MD is able to consume.

Lemma 1.

${\theta }_i$ is the perturbation parameter of the introduced MD cell, and the value of ${\theta }_i$ satisfies the following constant term boundary condition:

$${\theta }_i⩾{\tilde{E}}_i^{max}+V{\psi }_i{\tilde{E}}_i^{{min}^{-1}}$$

(25)

We consider a model where ($0<V<\infty$) is a nonnegative control parameter (in joules, J) used for weighing the importance of the energy consumption for local and offloading computation. ${\tilde{E}}_i^{max}$ represents the maximum energy consumption.

$${\tilde{E}}_i^{max}=min\{max\{E_{i,l}p\left(t\right),E_{i,m}^r\left(t\right)+E_{i,m}^c\left(t\right)\},E_i^{max}\left(t\right)\}$$

(26)

In time slot t, we define the virtual energy queue vector of the system as $\tilde{\mathbf{B}}\left(t\right)\triangleq [{\tilde{B}}_1\left(t\right),{\tilde{B}}_2\left(t\right),\dots ,{\tilde{B}}_N\left(t\right)]$, and it is necessary to ensure the stability of the virtual queue. So, we define a quadratic Lyapunov function on virtual queues as shown below:

$$L\left(\tilde{\mathbf{B}}\left(t\right)\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^N{\tilde{B}}_i{\left(t\right)}^2,L\left(\tilde{\mathbf{B}}\left(t+1\right)\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^N{\tilde{B}}_i{\left(t+1\right)}^2$$

(27)

Then, we can introduce a one-step conditional Lyapunov cost function that pushes the quadratic Lyapunov function to a bounded level and obtains a stable virtual queue:

$$\Delta \left(\tilde{\mathbf{B}}\left(t\right)\right)\triangleq \mathbb{E}\left[L\left[\tilde{\mathbf{B}}\left(t+1\right)\right]-L\left(\tilde{\mathbf{B}}\left(t\right)\right)|\tilde{\mathbf{B}}\left(t\right)\right].$$

(28)

The drift-plus-penalty function can be defined as

$$\Delta v\left(\tilde{\mathbf{B}}\left(t\right)\right)=\Delta \left(\tilde{\mathbf{B}}\left(t\right)\right)+V\mathbb{E}[G\left(t\right)|\tilde{\mathbf{B}}\left(t\right)],$$

(29)

where $V(V>0)$ is a control parameter.

Parameter V is used in the Lyapunov framework to balance the trade-off between queue stability and optimization objectives. Specifically, a larger value of V will be more inclined to minimize the long-term average cost but may sacrifice queue stability, while a smaller value of V prioritizes the stability of the virtual energy queue but may reduce the convergence efficiency of the optimization objective. Adjusting V, the system can achieve a flexible trade-off between performance and dynamic resource management.

Lemma 2.

For any feasible decision $\Omega \left(t\right)$ to be applicable at any time, the offset-plus-penalty function satisfies the following condition:

$$\begin{array}{cc}\hfill \Delta v\left(\tilde{\mathbf{B}}\left(t\right)\right)& ⩽V\mathbb{E}\left[\sum _{i=1}^N[{\alpha }_i{E}_i\left(t\right)+{\beta }_i{T}_i\left(t\right)+{\gamma }_i{C}_i\left(t\right)]|\tilde{\mathbf{B}}\left(t\right)\right]+C+\mathbb{E}\left[\sum _{i=1}^N\left[{\tilde{B}}_i\left(t\right)(e_i\left(t\right)-E_i\left(t\right))\right]|\tilde{\mathbf{B}}\left(t\right)\right]\hfill \end{array}$$

(30)

where $C={\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^N}{[E_i^{max}\left(t\right)}^2+({\tilde{E}}_i^{max}{)}^2]$.

Proof. 

From the equations of (27), we can obtain that

$$L\left(\tilde{\mathbf{B}}\left(t+1\right)\right)-L\left(\tilde{\mathbf{B}}\left(t\right)\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^N\left[{\left({\tilde{B}}_i\left(t\right)-E_i\left(t\right)+e_i\left(t\right)\right)}^2-{\tilde{B}}_i{\left(t\right)}^2\right]⩽{\displaystyle \frac{1}{2}}\sum _{i=1}^N\left[{\left(E_i^{max}\right)}^2+{\left({\tilde{E}}_i^{max}\right)}^2\right]+{\tilde{B}}_i\left(t\right)\left(e_i\left(t\right)-E_i\left(t\right)\right),$$

(31)

and let $C={\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^N}{[E_i^{max}\left(t\right)}^2+({\tilde{E}}_i^{max}{)}^2]$.    □

According to the Lyapunov optimization method, we can convert the objective problem into an upper bound function optimized mainly by Equation (30). Based on this, problem P2 can be transformed into a series of optimization problems for each time interval. In these intervals, the battery capacity level of MD i can be continuously maintained stable under the influence of perturbation parameter ${\theta }_i$. Here, we neglect the constant term in the upper bound of the drift-plus-penalty function. By doing so, we can transform long-term average optimization problem P2 into a series of optimization problems per time slot, which simplifies the problem-solving process and makes it more tractable. The new problem is shown as problem P3 below:

$$\mathrm{P}3:\underset{\mathbf{\Omega }\left(t\right)}{min}VG\left(t\right)+\sum _{i=1}^N{\tilde{B}}_i\left(t\right)[e_i\left(t\right)-E_i\left(t\right)]$$

(32)

s.t.(15) − (21), (23), (24b)

(33)

5. Problem Solution

In this section, we will investigate how to solve problem P3. This problem involves the optimization of energy harvesting, the offloading decision strategy, and resource allocation. Our goal is to obtain the optimal offloading decision, the allocation of transmission power, the allocation of bandwidth between MD and MBS, and the allocation of computing resources for both local computation and the MEC servers. Since problem P3 is a mixed-integer optimization problem, because the computation offloading decision is an integer variable and the other optimization variables are continuous variables, we employ a deep reinforcement learning-based approach to address it. We formulate the system as a discrete MDP problem. An MDP is a mathematical framework. It is used to model decision making. This modeling is for situations where outcomes are partly random. Also, outcomes are partly under the control of a decision-maker. Here, we aim to maximize the system payoff through this MDP model. Since the transfer probability of system states and system rewards cannot be predicted in advance, we propose a model-free scheme based on DRL to solve the MDP problem. We adopt a four-tuple $<\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}>$ to represent it, where $\mathcal{S}$ represents the state, $\mathcal{A}$ denotes the action, $\mathcal{P}$ indicates the state transfer probability, and $\mathcal{R}$ refers to the system reward.

5.1. MDP Framework

Nearly all programming problems controlled by decisions can be characterized in terms of MDP [37]. In general, MDP is solvable using linear programming or dynamic programming methods. Within the MDP framework, it is important to note the information interaction between MDs and MEC servers. MDs, as agents, can obtain state information from the environment, which includes not only local parameters such as battery level and computation capacity but also remote parameters provided by MEC servers via a control channel. This control channel facilitates the sharing of critical information needed for making informed offloading and resource allocation decisions. Herein, we will attempt to use the RL-based methods to deal with these challenging problems. Therefore, above all, we enumerate the related critical elements in MDP as shown below:

• State: In current time slot t, we define the state of the system as $s\left(t\right)\in \mathcal{S}$, including the channel gains $\mathbf{g}\left(t\right)=\{g_{1,1}\left(t\right),\dots ,g_{i,m}\left(t\right),\dots ,g_{N,M}\left(t\right)\}$ between MDs and MBSs. We formulate a four-tuple $<\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}>$ to represent the system. The available MEC server computing resources are denoted as $\mathbf{F}=\{F_1,\dots ,F_m,\dots ,F_M\}$; the data size of each arrival task, $\mathit{\lambda }\left(t\right)=\{{\lambda }_1\left(t\right),\dots {\lambda }_i\left(t\right),\dots ,{\lambda }_N\left(t\right)\}$; channel interference received between the MD and MBS, $\mathbf{H}\left(t\right)=\{H_{1,1}\left(t\right),\dots ,H_{i,m}\left(t\right),\dots ,H_{N,m}\left(t\right)\}$; and the harvested energy, $\mathbf{E}\left(t\right)=\{e_1\left(t\right),e_2\left(t\right),\dots ,e_i\left(t\right),\dots ,e_N\left(t\right)\}$. Thus, we describe the vector of state system at time slot t as

  $$\mathbf{s}\left(t\right)=\left(\mathbf{g}\right(t),\mathbf{F},\lambda (t),\mathbf{H}(t),\mathbf{E}(t\left)\right)$$

  (34)
• Action: Each agent needs to independently adopt actions based on the current state $s\left(t\right)$ and time slot t, the action is defined as $a_i\left(t\right)\in \mathcal{A}$, which includes computational resource allocation, wireless resource allocation, offloading decision schemes, and energy harvesting, i.e., the available computing resource capacity $f_{i,l}\left(t\right)$ of the MD, the computational resource allocation of MEC server $f_{i,m}\left(t\right)$, the transmission power of MD $p_{i,m}\left(t\right)$, the bandwidth allocation between MD i and MBS m, ${\omega }_{i,m}\left(t\right)$, and computation offloading decision schemes $\mathbf{x}\left(t\right)=\{x_{i,l}\left(t\right),x_{i,m}\left(t\right),x_{i,d}\left(t\right)\}$. Hence, we define the action as a set of tuples:

  $$\mathcal{A}=(f_{i,l}\left(t\right),f_{i,m}\left(t\right),p_{i,m}\left(t\right),{\omega }_{i_m}\left(t\right),e_i\left(t\right),x\left(t\right))$$

  (35)
• Reward: After taking a feasible action, MDs can obtain a corresponding reward $r\left(t\right)\in \mathcal{R}$ for the action $a_i\left(t\right)$ taken from the environment based on the current state $s\left(t\right)$. In this investigation, we focus on formulating a multiobjective optimization problem between the task completion delay, overall energy consumption, and the cost of task discard. Then, we transform the primal problem into an optimization problem based on the Lyapunov framework by optimizing the upper bound of the Lyapunov drift-plus-penalty function to solve the corresponding resource allocation, energy harvesting, and offloading decision schemes. It is notable that the reward value of each agent should fulfill the constraints so as to guarantee the validity of the computing results. The reward function is shown below:

  $$r\left(t\right)=\left\{\begin{array}{cc}-\Phi \left(t\right),& \mathrm{if}\mathrm{constraints}\mathrm{are}\mathrm{satisified}\\ 0,& \mathrm{otherwise}\end{array}\right\}$$

  (36)

  where $\Phi \left(t\right)=VG\left(t\right)+{\displaystyle \sum _{i=1}^N}{\tilde{B}}_i\left(t\right)[e_i\left(t\right)-E_i\left(t\right)]$.

$\Phi \left(t\right)=VG\left(t\right)+{\displaystyle \sum _{i=1}^N}{\tilde{B}}_i\left(t\right)[e_i\left(t\right)-E_i\left(t\right)]$, where $VG\left(t\right)$ denotes the instantaneous cost weighed through the Lyapunov framework, which includes factors such as the task completion delay and overall energy consumption. This is used to guide the algorithm to focus on the immediate performance of the system during the optimization process; ${\displaystyle \sum _{i=1}^N}{\tilde{B}}_i\left(t\right)[e_i\left(t\right)-E_i\left(t\right)]$ is then related to the virtual energy queue, which is used to balance the energy consumption between different time gaps to ensure the long-term stable operation of the system. In MEC and energy harvesting environments, such a rewarding design can motivate the intelligent agent to satisfy the task requirements while rationally utilizing energy to minimize the total system cost.

In accordance with the MDP model given above, such an optimization problem can be transformed into maximizing the long-term cumulative reward obtained by MDs through the optimal control strategy. We define the control policy of the proposed system as $\pi :\mathcal{S}\to \mathcal{A}$, which indicates a mapping relationship from state $s\left(t\right)$ to action $a\left(t\right)$. Each agent determines an action $a\left(t\right)=\{f_{i,l}\left(t\right),f_{i,m}\left(t\right),p_{i,m}\left(t\right),{\omega }_{i_m}\left(t\right),e_i\left(t\right)\}\in \mathcal{A}$ by formulating a strategy $\pi$ based on the observed environmental state $s\left(t\right)=\left\{\mathbf{g}\right(t),\mathbf{F},\lambda (t),\mathbf{H}(t\left)\right\}\in \mathcal{S}$.

In this study, the action contains discrete decisions (offloading patterns) and continuous decisions (resource allocation). For this hybrid action, we use the following approach to deal with it.

For the discrete offloading mode decisions, the output of the policy network goes through the Softmax function to obtain the probability distribution $\mathsf{\pi }\mathsf{\theta }$(x|s) for each offloading mode, where x denotes the offloading model, s denotes the current state, and $\mathsf{\theta }$ is a parameter of the policy network. At each time step, the specific offloading mode decision is obtained by sampling according to this probability distribution.

For continuous resource allocation actions $(f_{i,l}\left(t\right),f_{i,m}\left(t\right),p_{i,m}\left(t\right),{\omega }_{i_m}\left(t\right),e_i\left(t\right))$, the policy network outputs the mean $\mu$ and standard deviation $\sigma$ of the continuous actions, and then samples from a Gaussian distribution $N(\mu ,{\sigma }^2)$ to obtain specific continuous action values. In order to ensure that the action values are within a reasonable range, we trim the sampled action values, i.e., $a=clip(a,a_{min},a_{max})$, where a is the sampled action value, and $a_{min}$ and $a_{max}$ are the minimum and maximum values of the action, respectively.

5.2. Q-Learning Algorithm

Q-learning is considered a model-free form of reinforcement learning and can be seen as an asynchronous dynamic programming methodology [38]. The Q-learning algorithm employs a value iteration method to replace the Q-value, and then the optimal action policy is obtained based on the computed Q-value. The Q-learning algorithm contains an agent, action, state and reward. Specifically, in the implementation of Q-learning, the agent first selects an action $a\left(t\right)$, constructs a state–action pair $\left(s\right(t),a(t\left)\right)$ based on state $s\left(t\right)$, and calculates the Q-value under the current state–action pair based on the Q-function. Subsequently, the agent will receive a reward from the current environment and then the agent transfers the state from $s\left(t\right)$ to $s(t+1)$. The agent computes the Q-value and updates the Q-table continuously, based on the maximum Q-value in each state–action pair so as to determine the optimal policy.

Herein, we design a Q-learning-based approach to solve the MDP decision problem. The Q-value computed from the state–action pairs in each time slot, which is estimated using the Q-learning method, will be stored in the Q-table, and then the Q-table is updated with the value iteration method. The Q-value can be expressed as

$$\begin{array}{c}Q(s\left(t\right),a\left(t\right))=(1-\alpha )Q\left(s\left(t\right),a\left(t\right)\right)+\alpha \left[R\left(s\left(t\right),a\left(t\right)\right)+\gamma maxQ\left(s(t+1),a(t+1)\right)\right]\hfill \end{array}$$

(37)

$Q\left(s\right(t),a(t\left)\right)$ denotes the Q-value calculated from the state–action pair in time slot t, where $0<\alpha <1$ indicates the learning rate, $R\left(s\left(t\right),a\left(t\right)\right)$ denotes the reward obtained by the agent for the state–action pair, and $\gamma$ is a discount factor.

Each agent gradually learns to update individual actions using the appropriate functions to obtain optimal offloading decisions and resource allocation through continuous observation of the system environment, and this process minimizes the overall cost of the system in the absence of other user information. Each agent develops a corresponding strategy through continuous interactive learning, and takes the calculated Q-value as a long-term cumulative reward. Initially, in the implementation of the Q-learning Algorithm 1, Q-values with respect to each state–action pair in the Q-matrix are sparsely and randomly set, and the values in the Q-matrix accumulate as the learning time slot increases. In addition, we use $\chi$-greedy for action decisions in order to achieve the exploration–exploitation trade-off during the Q-learning process. We utilize $\chi$-greedy for action decision making. The agent selects actions with probability $1-\chi$ to maximize the Q-value and random actions with probability $\chi$. The expression of $\chi$-greedy is shown below:

$$a\left(t\right)=\left\{\begin{array}{cc}\mathrm{random}\mathrm{action}& \chi \\ argmaxQ\left(s\right(t),a)& 1-\chi \end{array}\right\}.$$

(38)

Each agent receives the corresponding reward $r(s,a)$ after performing action $a\left(t\right)$ and transitions from state $s(t+1)$. The Q-value matrix among the state–action pairs is updated according to Equation (37).

The time complexity of the Q-learning Algorithm 1 is jointly determined by the state, the action, and the number of training rounds. The time complexity is $O(E\ast T^{max}\ast |S|\ast |A|)$, where $|S|$ is the number of states, $|A|$ is the number of actions, E is the number of training rounds, and $T^{max}$ is the maximal time step of each round. This complexity grows exponentially with dimension D.

Algorithm 1 Computational offloading and resource allocation algorithms based on Q-learning

1: Input: state $\mathcal{S}$, action $\mathcal{A}$, learning rate $\alpha$, discount factor $\gamma$. 2: Output: Q-value for each state-action team. 3: Initialization: set arbitrarily for $\forall s\in \mathcal{S}$, $a\in \mathcal{A}$. 4: for each episode do 5:     Initialization state $s\left(t\right)$; 6:     while $T^{episode}\le T^{max}$ do 7:         Select an action with random probability in the current state $s\left(t\right)$; 8:         if $\Phi \le \epsilon$ then 9:            Randomly select an action $a\left(t\right)$; 10:         else 11:            Set $a\left(t\right)=argmaxQ\left(s\right(t),a)$; 12:            Perform the action $a\left(t\right)$, get the reward $r\left(t\right)$ and start the next state $s(t+1)$; 13:            Update $Q\left(s\right(t),a(t\left)\right)=Q\left(s\right(t),a(t\left)\right)+\alpha \left[r\right(s\left(t\right),a\left(t\right))+\gamma maxQ(s(t+1),$ $a(t+1))-Q(s\left(t\right),a\left(t\right)\left)\right]$ ; 14:            Set $s\left(t\right)=s(t+1)$; 15:            $T^{episode}=T^{episode}+1$; 16:         end if 17:     end while 18: end for

5.3. A3C Algorithm

Although the best policy can be obtained by constantly updating and recording the Q-values in the Q-table, it is difficult to effectively maintain the Q-table when the space of the feasible action states of the agent becomes larger and larger. This will increase the dimensionality of the Q-table data matrix and pose a great challenge to Q-value data storage, search, and renewal.

The A3C algorithm is one of the DRL algorithms that are simpler, faster, and more robust than other DRL algorithms such as actor–critic learning algorithms and deep deterministic policy gradient (DDPG) algorithms [39]. A3C leverages the value-based and policy-based methods for both discrete and continuous actions [40,41]. Using multiple CPU threads on a single machine, A3C can efficiently obtain optimal policies using an asynchronous actor learner.

As depicted in Algorithm 2, the A3C algorithm consists of multiple actor learners and a global network. In particular, asynchronous multithreading is employed to implement multiple actors. All training weights obtained during the process are stored in the global network. At the beginning of training, the global network transmits the relevant parameters to these actor learners simultaneously. The parameters trained by the actor learners are uploaded to the global network. After a period of time, the global network sends updated parameters to the actor–critic to ensure that they can share a common strategy.

The A3C algorithm requires maintaining a policy $\pi (a_t|s_t;\eta )$ that consists of a set of action probabilities $a_t$ and a value function $V(s_t;{\eta }_v)$. The value function is used to evaluate how good a certain state is. The policy and the value function are determined by parameters $\eta$ and ${\eta }_v$, respectively. Policy $\pi (a_t|s_t;\eta )$ and value functions $V(s_t;{\eta }_v)$ are approximated using a single convolutional neural network. Specifically, the estimates of the policy function and the value function are output by the softmax layer and the linear layer, respectively. The agents in the A3C algorithm update the strategy and the rules using the estimated value function and the advantage function, respectively.

The current environment shifts from state $s_t$ to state $s_{t+1}$ with a certain probability, and each intelligence point receives a reward $r_t$. The state value function of A3C can be defined as

$$V(s_t;{\theta }_v)=E[G_t|s=s_t,\pi ]=E[{\displaystyle \sum _{k=0}^{\infty }}{\gamma }^k{r}_{(t+k)}|s=s_t,\pi ]$$

(39)

where $G_t={\displaystyle \sum _{k=0}^{\infty }}{\gamma }^k{r}_{(t+k)}$ denotes the discounted reward, i.e., the cumulative discounted reward of state $s_t$. $\gamma \in [0,1]$ is the discount factor, which indicates how future rewards affect the current state value.

In using the K-step reward, the discounted reporting is given by

$$R_t\left({\theta }_v\right)={\displaystyle \sum _{i=0}^{k-1}}{\gamma }^i{r}_{t+i}+{\gamma }^{k}V(s_{t+k};{\theta }_v),$$

(40)

where k is the upper bound on the maximum value $t_{max}$, $r_{t+i}$ is the immediate reward, and $\gamma =(0,1]$ is the discount factor.

Based on Equation (40), the advantage function can be written as

$$A(a_t,s_t;\theta ,{\theta }_v)=R_t\left({\theta }_v\right)-V(s_t;{\theta }_v)$$

(41)

The policy loss function is defined as

$$f_{\pi }\left(\theta \right)=log\pi (a_t|s_t;\theta )(R_t-V(s_t;{\theta }_v)+\beta H\left(\pi (s_t;\theta )\right)$$

(42)

$H\left(\pi (s_t;\theta )\right)$ is an entropy term utilized to encourage exploration during the training process so as to prevent possible premature convergence. Parameter $\beta$ is used to control the strength of the regularization of the entropy, which facilitates the trade-off between exploration and exploitation.

In the context of our analysis related to the relevant functions and parameters, when we aim to explore the characteristics and behaviors of the function $f_v\left({\theta }_v\right)$, solving for the partial derivative $f_v\left({\theta }_v\right)$ with respect to ${\theta }_v$ yields the following equation:

$${\nabla }_{\theta }{f}_{\pi }\left(\theta \right)={\nabla }_{\theta }log\pi (a_t|s_t;\theta )(R_t-V(s_t;{\theta }_v))+\beta {\nabla }_{\theta }H\left(\pi (s_t;\theta )\right)$$

(43)

The actor can be updated using the following equation:

$$d\theta \leftarrow d\theta +{\nabla }_{{\theta }^{\prime }}log\pi (a_t|s_t;{\theta }^{\prime })(R_t-V(s_t;{\theta }_v))+\delta {\nabla }_{{\theta }^{\prime }}H\left(\pi (s_t;{\theta }^{\prime })\right)$$

(44)

Solving for the partial derivative of $f_{\pi }\left({\theta }_v\right)$ with respect to ${\theta }_v$ yields the following equation:

$${\nabla }_{{\theta }_u}{f}_v\left({\theta }_v\right)=2(R_t-V(s_t;{\theta }_v)){\nabla }_{{\theta }_v}V(s_t;{\theta }_v)$$

(45)

The critics are updated on the basis of the following cumulative gradient:

$$d{\theta }_v\leftarrow d{\theta }_v+2(R_t-V(s_t;{\theta }_v)){\nabla }_{{\theta }_v}V(s_t;{\theta }_v)$$

(46)

We use the RMSProp algorithm to minimize the loss function, and the gradient estimate is calculated as

$$g=\alpha g+(1-a)\Delta {\theta }^2,$$

(47)

where a denotes the momentum, and $\theta$ denotes the cumulative gradient of the loss function.

We can then update the RMSProp algorithm based on the following estimated gradient descents:

$$\vartheta \leftarrow \theta -\eta {\displaystyle \frac{\Delta \theta }{\sqrt{g+\epsilon }}}$$

(48)

where $\Delta$ denotes the learning rate, and $\epsilon$ denotes a small positive value.

Algorithm 2 presents the A3C-based joint optimization of the computation offloading decision, energy harvesting time allocation, and mode selection. The time complexity of A3C depends on three key factors: the number of parallel threads N, the time step of a single update T, and the computational complexity of the neural network. The overall time complexity can be expressed as $O(N\ast T\ast d)$.

Algorithm 2 Joint optimization of the computation offloading decision, energy harvesting time allocation, and mode selection

1: Initialize: Set the global network parameters of actor network and critic network as $\theta$ and ${\theta }_v$ respectively. 2: Set the local network parameters of actor network and critic network as ${\theta }^{\prime }$ and ${\theta }_v^{\prime }$ respectively 3: Set the $T=0$, $t=1$, $\eta$, $T_{max}$, $t_{max}$. 4: Iteration: 5: while  $T<T_{max}$  do 6:     for $w=1$ to W do 7:         Set ${\theta }^{\prime }$=$\theta$, ${\theta }_v^{\prime }$=${\theta }_v$ 8:         Obtain the system state $S\left(t\right)$ 9:         for $t<t_{max}$ do 10:            Perform the action $a\left(t\right)$ under the policy $\pi (a_t|s_t;{\theta }^{\prime })$; 11:            Get the reward $r\left(t\right)$ and the new state $s(t+1)$; 12:         end for 13:         $r\left(t\right)\left\{\begin{array}{cc}0& \mathrm{forterminalstate}\\ V(s_t,{{\theta }^{\prime }}_v)& \mathrm{fornon}-\mathrm{terminalstate}\end{array}\right\}$ 14:         for $t=t_{max}$ do 15:            $R=r\left(t\right)+\gamma R$ 16:            Obtain accumulate gradient wrt ${\theta }^{\prime }$ based on (44) 17:            Obtain accumulate gradient wrt ${\theta }_v^{\prime }$ based on (46) 18:         end for 19:         Update ${\theta }^{\prime }$ and ${\theta }_v^{\prime }$ based on (47) 20:         $T=T+1$ 21:     end for 22: end while

5.4. A3C Algorithm Core Parameter Settings and the Training Process

In this study, we designed a neural network architecture with specific configurations for the actor and critic networks. The actor network takes an input layer corresponding to the state-space dimension ($S=|g,F,\lambda ,H|=512$). It consists of two hidden layers, each containing 256 neurons with the ReLU activation function. The output layer represents the action-space dimension, where continuous actions pertain to resource allocation parameters, and discrete actions refer to offloading decisions. The probabilities of discrete actions are processed using the Softmax function.

The critic network shares the first two hidden layers with the actor network. Its output layer is a scalar value function that evaluates the state value. Regarding hyperparameters, we set the learning rate $\eta =1\times {10}^{-4}$, utilizing the RMSProp optimizer with a momentum parameter $\alpha =0.9$. The discount factor $\gamma =0.99$ is employed to balance short-term and long-term rewards. An entropy regularization parameter $\beta =0.01$ is introduced to encourage policy exploration and prevent premature convergence. To avoid gradient explosion and stabilize the training process, a gradient clipping parameter of $cli{p}_{n}orm=0.5$ is used. We use asynchronous threads $W=8$ to explore different environmental samples in parallel, thereby accelerating the training process. The training consists of $T_{max}=5000$ epochs, with each epoch containing 1000 time slot iterations.

Our implementation framework is based on PyTorch 2.0, and we refrain from relying on third-party libraries such as Stable Baselines to ensure that the algorithm logic is fully consistent with the research description. The key modules include environmental interaction, where each asynchronous thread independently simulates the dynamic behavior of the MEC system, including energy harvesting, channel variations, and task arrivals, to generate state–action–reward samples. Additionally, for parameter synchronization, the global network aggregates the gradient updates from each thread every 50 steps to avoid parameter divergence during asynchronous training. The selection of parameters is well founded. The neural network structure draws inspiration from the successful configurations of the classic A3C algorithm in continuous control tasks. We performed ablation experiments to verify the impact of the dimensions of the hidden layer on the convergence speed. The gradient clipping and entropy regularization parameters were determined through cross-validation to strike a balance between exploration efficiency and policy stability. The number of asynchronous threads was set to 8, which effectively balances computational resource utilization and training stability, as an excessive number of threads may lead to parameter update conflicts.

In the training process of the A3C algorithm, we adopted a dynamic learning rate strategy. The initial learning rate was set to 0.001, and as the training progressed, the learning rate was adjusted to decay by a factor of 0.95 every 100 training rounds. This learning rate setting helped the algorithm to quickly explore the environment in the early stages of training, and more accurately converge to the optimal policy in the later stages. We set the batch size to 64. The choice of batch size has a significant impact on the training effectiveness of the algorithm, as larger batch sizes allow the algorithm to be more stable when updating parameters, but increase memory requirements and training time, while smaller batch sizes speed up training, but may lead to larger fluctuations in the training process. The A3C algorithm uses an asynchronous training mechanism to improve the efficiency of the training, and we set up eight worker threads, each of which was a thread of a worker that learns in an independent instance of the environment. These worker threads simultaneously interact with the environment to collect experience samples and update the global neural network parameters. In our experiments, we used NVIDIA GeForce RTX 3090 GPUs with Intel Core i9-12900K CPUs to run the A3C algorithms and PyTorch as the deep learning framework for algorithm implementation. This hardware configuration can meet the computational requirements of the A3C algorithm in the large-scale training process to ensure efficient training. At the same time, we also reasonably allocated and optimized the hardware resources to avoid resource bottlenecks and ensure the accuracy and reliability of the experimental results.

6. Results

In this study, simulation experiments were conducted to verify the feasibility and superiority of the proposed method. And we list the simulation parameters in

Table 3. Simulation parameters.

Notations	Value
$\omega$	${10}^{-27}$
$\tau$	1 ms
$p_i^{max}$	2 W [33]
$f_i^{max}$	1.5 GHz [42]
$d_{i,m}$	[10, 400] m
$B_{i,m}$	[2, 12] MHz
$E_{i,max}$	20 mJ
$p_m$	4 W
${\psi }_i$	0.0001
$F^{max}$	10 GHz
$c_i$	5900 cycle/bit
$\sigma$	−4
$T_{i,max}$	2 S
${\lambda }_i$	$[1,10]\times 2^{10}$ bit
N	10
M	4
$p_{i,id}$	0.002 W

.

Figure 2 depicts the average task computation cost with respect to the distance between the user and the MBS, under local and edge server computing approaches, and under the influence of different task arrival rates and weighting factors. As can be seen in Figure 2, the cost of task processing for the user device remains constant with increasing distance, but the total computational processing cost increases continuously with increasing distance between the user and the MBS when offloading computational tasks to the edge server, which generates task transfer energy consumption, server computation energy consumption, and user device energy consumption in the idle state while the tasks are being processed. At the same time, when the distance between the user and the base station is close, the total cost of choosing to offload the computational tasks to the edge service computation is smaller than that of local computation because the edge server in the base station has powerful computational resources. In contrast, as the distance increases, the computing tasks of the user consume more energy during the offloading process, which results in a higher computational cost compared to local processing. In addition, it can be seen from the figure that as the computational task arrival rate increases, it leads to an increasing amount of computational tasks to be processed, thus increasing the computational cost for the user. With the same computational task arrival rate, as the weight factor of latency decreases and the weight factor of energy consumption increases, the total cost of user computational tasks decreases in both modes of local and offloaded computation, which requires continuous optimization to reduce costs for application scenarios with high latency requirements.

Figure 3 represents the analysis of the task discard ratio under different computation offloading modes with respect to the distance between the user and the base station. From Figure 3, it can be concluded that the task discard ratio of local computing does not change with the distance between the user and the base station. The task discard ratio in the local computing mode is related to the computing capability of the user device, task delay, energy consumption, and battery energy. Moreover, when the distance between the user and the base station is short, the proportion of tasks discarded in the local computing mode is higher than that in other computing offload modes. This is because at shorter distances, the transmission energy consumption and time delay required for other computing offload modes are lower, making them more efficient and cost-effective compared to the local computing mode. However, as the distance increases, due to changes in channel state information during offloading, path loss and the resulting interference become greater. The delay and energy consumption of task offloading also increase, which may lead to difficulties in completing the user’s computational tasks, and thus the user’s computational tasks will be discarded. Compared with edge computing and the dynamic offloading mode of computing, the Q-learning-based strategy proposed in this study can select different states and actions according to factors such as the offloading process, the transmission distance of user computing tasks, and channel state information to maximize the reward for user devices and reduce the proportion of tasks discarded due to long-distance transmission.

Figure 4 and Figure 5, respectively, depict the relationships between energy harvesting power and the task discard percentage of user devices using energy harvesting techniques.

As shown in Figure 4, an increase in energy harvesting power leads to a decrease in the cost of processing user computational tasks. This is because a higher power enables the user device to gather more energy per unit time, and renewable energy consumption incurs no execution costs. When the energy harvesting power is low, it is hard to fully utilize the limited energy, resulting in a high task discard rate. As the energy increases, the discard rate declines, but slowly. This is due to the Q-Learning algorithm’s inability to quickly adapt to the complex state changes brought about by energy variations.

In contrast, the A3C algorithm, with its well-crafted neural network architecture, can rapidly respond to energy harvesting power fluctuations. Its input layer takes in state information like energy harvesting power and task data volume. The hidden layer then deeply extracts features, and the output layer generates appropriate action strategies. When energy is scarce, A3C prioritizes critical tasks and allocates energy rationally, maintaining a relatively low discard rate. When energy is abundant, it can flexibly use the surplus energy to further reduce the discard rate, demonstrating excellent adaptability and decision-making capabilities.

Figure 5 reveals that, similarly to the task execution cost, the proportion of discarded tasks for different user-task computation offloading modes drops as the energy harvesting power rises. A higher power means the user device can obtain more renewable energy, providing more energy for task computation and offloading, effectively reducing the discard proportion caused by energy shortages.

The A3C algorithm’s asynchronous multi-threaded architecture allows it to efficiently process high-dimensional state information. It can dynamically adjust policies and allocate resources based on the task arrival rate, keeping the average latency low even at high arrival rates. Overall, A3C has a lower average latency than Q-learning across all task arrival rates, with a more pronounced advantage at high rates, highlighting its superiority in handling dynamic loads.

Figure 6 shows the relationship between the user’s computational task arrival rate and the task discard ratio. From Figure 6, it can be seen that as the user’s computational task arrival rate increases, the user’s task discard ratio also increases. Since the user tasks have limited computing resources and energy in the user’s device when local computing is selected, local computing is limited by computing resources and energy as the task arrival rate increases, which results in a higher percentage of discarded computational tasks. Compared to local computing, the percentage of discarded computational tasks in the offloading mode is lower because the offloading mode has more powerful task communication and computational resources. The strategy proposed in this study, however, reduces the task discard ratio as the task arrival rate increases because Q-learning is able to dynamically adjust the resource allocation and offloading strategy according to the surrounding environment.

The A3C algorithm can dynamically adjust the computational resource allocation and task offloading strategy according to the number of users in the experiment. When the number of users is small, it makes reasonable use of local resources to reduce energy consumption, and when the number of users increases, it finds a better balance between local computation and offloading, so that the system energy consumption gradually increases.

From Figure 6, it can be seen that the A3C algorithm consumes less energy than the Q-learning algorithm for different numbers of users. Especially after the number of users reaches a certain scale, the A3C algorithm’s advantage in controlling energy consumption is obvious, which verifies its effectiveness and superiority in dealing with large-scale user scenarios, and it can control energy consumption more effectively while guaranteeing system performance.

Figure 7 and Figure 8 validate the cumulative rewards and the number of steps for the proposed A3C algorithm for different training rounds, respectively. As shown in Figure 7, the cumulative reward for each round is presented. The A3C algorithm in this section has a positive effect as the rounds increase and the final results converge to a relatively stable value, indicating a good convergence of the algorithm. It can be seen from Figure 8 that as the training rounds increase, the A3C algorithm decreases the number of steps to reach the target and finally converges to a stable step value to achieve the optimal strategy.

7. Conclusions

In this paper, we consider an MEC offloading network system with energy harvesting support. We divide the time into multiple time intervals during which the mobile device can perform dynamic energy harvesting and store it in the battery. The user’s computational tasks can be computed locally or at the edge server, and we take into account the situation where the tasks will be discarded if they cannot be completed. In the context of this paper, the main problem lies in the weighted sum of the processing delay, energy consumption, and the cost of discarding the computational tasks. And the Lyapunov framework is used to construct the offset-plus-penalty function because of the energy collection coupling in adjacent time intervals. The rational setting of parameter V is crucial to the system performance. Future research could incorporate online adaptive methods to dynamically adjust V to cope with complex demands in time-varying environments. Finally, we employ Q-learning and A3C methods from reinforcement learning to solve the problem. Then, we propose Q-learning and A3C-based algorithms to jointly optimize computational offloading decisions, resource allocation, and energy harvesting.