Task Offloading Based on Virtual Network Embedding in Software-Defined Edge Networks: A Deep Reinforcement Learning Approach

Abstract

The advent of 5G/6G technologies and the pervasive deployment of IoT devices are driving the emergence of demanding applications that necessitate ultra-low latency, high bandwidth, and significant computational power. Traditional cloud computing models fall short in meeting these stringent requirements. To address this, Software-Defined Edge Networks (SDENs) have emerged as a promising architecture, yet efficiently managing their heterogeneous and geographically distributed resources poses substantial challenges for optimal application provisioning. In response, this paper proposes a novel framework for intelligent task offloading, which reframes the intricate multi-component application task offloading problem as a Virtual Network Embedding (VNE) challenge within a SDEN environment. We introduce a comprehensive model where complex applications are represented as Virtual Network Requests (VNRs). In this model, each VNR consists of virtual nodes that demand specific computing and storage resources, as well as virtual links that demand specific bandwidth and must adhere to maximum tolerable delay constraints. To dynamically solve this NP-hard VNE problem in the face of stochastic VNR arrivals and dynamic network conditions, we leverage Deep Reinforcement Learning (DRL). Specifically, a Soft Actor-Critic (SAC) agent is employed at the SDN controller. This agent learns a sequential decision-making policy for mapping virtual nodes to physical edge servers and virtual links to network paths. To guide the agent towards efficient resource utilization, we define the reward for each successful embedding as the long-term revenue-to-cost ratio. By learning to maximize this reward, the agent is naturally driven to find economically viable allocation strategies. Comprehensive simulation experiments demonstrate that our SAC-based VNE approach significantly outperforms other baselines across key metrics, affirming its efficacy in dynamic SDEN environments.

1. Introduction

The confluence of next-generation wireless technologies, notably 5G and beyond, and the proliferation of the Internet of Things (IoT) has catalyzed a paradigm shift toward a hyper-connected society [1,2]. This transformation is fostering a new wave of revolutionary applications, including immersive augmented and virtual reality (AR/VR), autonomous vehicles, industrial automation, and the nascent metaverse [3]. A common thread uniting these applications is their stringent demand for ultra-low latency, high bandwidth, and substantial computational power [4]. The conventional cloud computing model, which relies on centralized data centers, has proven increasingly inadequate for these latency-sensitive services due to the significant propagation delays incurred by routing data over long-haul networks and the potential for core network congestion. To bridge this gap, the paradigm of Mobile Edge Computing (MEC) has emerged, relocating computational and storage resources from the distant cloud to the network edge, in close proximity to end-users [5,6,7]. Concurrently, Software-Defined Networking (SDN) offers a programmable and centralized control plane, which integrated with MEC, forms a Software-Defined Edge Computing (SDEC) architecture [8,9,10]. This architecture provides the foundational agility and global network visibility necessary for dynamically managing distributed edge resources and orchestrating complex service delivery in real-time [11].

Despite the promise of the SDEC architecture, effectively managing its resources presents formidable challenges. The edge infrastructure is inherently heterogeneous and geographically dispersed, comprising a diverse array of physical assets such as powerful edge servers, computationally capable base stations, and interconnected network devices [12]. This distributed and varied nature complicates the core problem of resource allocation: how to efficiently and flexibly orchestrate these scattered resources to satisfy the diverse Quality of Service (QoS) requirements of myriad coexisting applications [13]. A brute-force or static allocation strategy is often suboptimal, leading to resource fragmentation, underutilization, and an inability to adapt to the dynamic fluctuations in service demands and network conditions. To address this fundamental challenge, network virtualization has been recognized as a pivotal enabling technology [14,15]. By decoupling services from the underlying physical hardware, it allows for the creation of multiple isolated, customized, and on-demand logical networks, commonly known as network slices, atop a shared physical infrastructure. The core technical problem in operationalizing network virtualization and network slicing is Virtual Network Embedding (VNE) [16]. Formally, VNE is the process of mapping a Virtual Network Request (VNR), an abstraction of a service’s resource demands, composed of virtual nodes (representing computational and storage requirements) and virtual links (representing communication and bandwidth requirements), onto the substrate physical network of edge servers and their interconnections [17,18]. This abstraction elegantly transforms the complex task of application-specific resource provisioning into a structured mapping problem, laying the groundwork for systematic and automated resource management.

The research landscape for VNE has evolved significantly over the years, progressing from early rule-based heuristics to more sophisticated, learning-based methodologies. Initial research on the VNE problem was predominantly focused on heuristic algorithms, which guide the search process using pre-defined, experience-driven rules. A classic and influential work in this area was proposed by Cheng et al. [19], who introduced the NodeRank algorithm. The core idea of NodeRank is to first compute a resource weight for all physical nodes, which holistically considers both their available resources and their topological significance within the network. Based on this calculated priority, physical nodes are then selected for mapping in a greedy fashion. Building on this, and recognizing that the efficient utilization of node resources can cascade to promote the efficient use of physical link resources, Zhang et al. [20] proposed two VNE algorithms based on multi-dimensional resource constraints, namely NRM-VNE and RCR-VNE. Other contributions in this domain have utilized meta-heuristic approaches, such as genetic algorithms and swarm intelligence, which employ population-based search strategies to explore the solution space more broadly than simple greedy methods [21]. Despite their simplicity and relatively low computational overhead, heuristic and meta-heuristic approaches suffer from several fundamental limitations that curtail their effectiveness, especially in large-scale and dynamic network environments. Their primary drawback is their myopic and greedy nature, which often leads them to converge on locally optimal solutions that are globally suboptimal. This shortsighted decision-making frequently results in severe resource fragmentation across the substrate network, where residual resources are scattered in small, unusable blocks. Consequently, the network’s ability to accept future, potentially more valuable, VNRs is diminished. Furthermore, these algorithms are based on static, pre-defined rules and lack the ability to adapt to the highly dynamic conditions of modern edge networks, such as stochastic VNR arrivals and fluctuating resource loads [22]. As the network scale increases, the performance of these static strategies degrades sharply, making them unsuitable for the demands of SDEN environments. To overcome these limitations, the mainstream approach for solving the VNE problem has increasingly shifted towards reinforcement learning (RL) [23,24]. The RL paradigm is exceptionally well-suited for this challenge as it reframes the VNE problem as a Markov Decision Process (MDP) [25]. In this formulation, the SDN controller, with its global network view, naturally assumes the role of an intelligent agent [26]. The current resource availability of the substrate network constitutes the state, the mapping decision for a virtual node or link represents the action, and a performance metric (e.g., revenue, acceptance rate) serves as the reward signal. By interacting with the network environment over time, the agent learns a sophisticated policy that maximizes the long-term cumulative reward, thereby transcending the limitations of static, rule-based methods. Several notable works have demonstrated the efficacy of RL in this domain. For instance, Yao et al. [27] proposed a continuous-decision VNE algorithm based on RL (CDRL), which pioneered the modeling of the node mapping stage as a sequential decision process, allowing the agent to make one mapping decision at a time. Pushing the boundaries of state representation further, Zhang et al. [28] introduced an innovative VNE algorithm that applies Graph Convolutional Networks (GCNs) within an RL framework (GRL). The GCN is leveraged to effectively extract spatial features and topological information from the substrate network, enabling the agent to make more informed decisions that account for the network’s structure. Furthermore, they designed a specific dynamic objective function based on a fitness matrix to guide the learning process, which was shown to significantly enhance the overall algorithm performance by aligning the reward signal more closely with the goal of sustainable resource allocation. Building upon these foundational DRL-based methods, more recent research has explored further specializations and architectural enhancements. For instance, He et al. [29] tackled a similar Virtual Network Function (VNF) placement problem by introducing an attention mechanism into a Deep Deterministic Policy Gradient (DDPG) framework. Their proposed A-DDPG algorithm leverages this mechanism to enable the DRL agent to focus on more relevant neighboring nodes with sufficient resources, thereby improving the quality of placement decisions. Concurrently, Xiao [30] proposed DVNE-DRL, a DRL-based framework notable for its detailed network feature extraction process and a multi-faceted reward function. The reward function in this work is explicitly shaped to jointly optimize for VNR acceptance rate, revenue-to-cost efficiency, and network load balancing, aiming to learn a more holistic embedding policy. Addressing the challenges of privacy and scalability in multi-domain networks, Zhang et al. [31] proposed an innovative framework combining Horizontal Federated Learning (HFL) with DRL. In their HFL-VNE approach, local DRL agents are deployed in each domain to make autonomous decisions, while a central server aggregates the model parameters, which avoids sharing raw local data and is designed to reduce cross-domain resource fragmentation while preserving privacy.

While the aforementioned approaches have advanced the field, they still face significant challenges in the context of dynamic SDENs. Firstly, heuristic algorithms, with their static and predefined rules, lack the adaptability to cope with stochastic VNR arrivals and fluctuating network resource loads. Secondly, their inherent myopic and greedy nature often leads to severe resource fragmentation, diminishing the network’s capacity to accept future, potentially more valuable, requests and thus undermining long-term performance. Thirdly, there is a need for a unified framework that can address the joint allocation of heterogeneous resources, holistically considering computing, storage, and network constraints.

Therefore, the problem we address is to develop an online and adaptive decision-making policy that can intelligently map incoming task offloading requests, framed as VNE problems, onto the physical SDEN infrastructure. The objective is to learn a policy that transcends short-term gains, aiming instead to optimize the long-term revenue-to-cost ratio and VNR acceptance rate, thereby ensuring both economic viability and high service availability. To tackle these challenges, this paper proposes a novel deep reinforcement learning framework. The primary contributions of our work are threefold:

(1)

A novel VNE-based formulation for task offloading in SDEC: We pioneer a formal model that transforms the intricate problem of application task offloading into a VNE problem. An application, composed of multiple interdependent subtasks, is abstracted as a Virtual Network Request (VNR), where virtual nodes represent the computational and storage demands of subtasks, and virtual links represent the data dependencies and communication bandwidth requirements between them. This modeling provides a unified and structured approach to the joint allocation of computing and network resources, which is a significant departure from conventional offloading models that often treat these resources disjointly.

(2)

A DRL framework for dynamic VNE: We design an intelligent and adaptive VNE algorithm based on the Soft Actor-Critic (SAC) model, a state-of-the-art DRL algorithm known for its sample efficiency and stability in continuous and large state spaces. The SDN controller acts as the learning agent, observing the global state of the edge network and making sequential embedding decisions. This framework is specifically tailored to learn a policy that maximizes long-term system performance in a highly dynamic environment where VNRs arrive stochastically.

(3)

DRL-driven optimization of key performance metrics: Our framework is meticulously designed to optimize critical long-term performance metrics for task offloading. This is achieved by engineering the reward function to directly reflect the revenue-to-cost ratio. By training the agent to maximize the cumulative reward, the learning process inherently steers the VNE policy towards solutions that not only increase revenue but also improve resource efficiency. As demonstrated in our experiments, this core optimization of the revenue-to-cost ratio consequently leads to significant improvements in the VNR acceptance rate and overall system revenue.

The remainder of this paper is structured as follows: Section 2 details the system architecture and problem formulation, where we formally model the SDEN and conceptualize the task offloading problem as a VNE formulation. Section 3 presents the VNE constraints that must be satisfied for a valid resource allocation, and defines the key performance metrics used for evaluating our proposed algorithm. In Section 4, we elaborate on the core contribution of this work, presenting the proposed task offloading algorithm based on VNE for SDEN, including its MDP formulation and the design of the SAC agent. Section 5 provides a comprehensive experimental evaluation, where we validate the performance of our approach through extensive simulations and analyze the results against several baseline algorithms. Finally, Section 6 concludes the paper.

2. System Model and Problem Formulation

This section introduces the architectural framework and formal models for our proposed task offloading scheme. We begin by defining the physical infrastructure of the SDEN. Subsequently, we abstract the application task requirements into a virtualized representation. Finally, we formulate the core task offloading challenge as a VNE problem. To clarify the process, a schematic diagram is shown in Figure 1. The symbols used in this paper are defined as shown in

Table 1. Summary of Notations.

Symbol	Description
Substrate Network (SDEN) Parameters
${\mathcal{G}}_{\mathcal{S}}$	The undirected graph representing the substrate network.
${\mathcal{N}}_{\mathcal{S}}$	The set of physical (substrate) nodes in the SDEN.
${\mathcal{L}}_{\mathcal{S}}$	The set of physical (substrate) links in the SDEN.
$n_s$	An individual physical node, $n_s\in {\mathcal{N}}_{\mathcal{S}}$.
$l_s$	An individual physical link, $l_s\in {\mathcal{L}}_{\mathcal{S}}$.
$C\left(n_s\right)$	Total computing capacity of a physical node $n_s$.
$M\left(n_s\right)$	Total memory/storage capacity of a physical node $n_s$.
$BW\left(l_s\right)$	Total bandwidth capacity of a physical link $l_s$.
$D\left(l_s\right)$	Communication latency of a physical link $l_s$.
Virtual Network Request (VNR) Parameters
${\mathcal{G}}_{\mathcal{V}}$	The directed graph representing a Virtual Network Request.
${\mathcal{N}}_{\mathcal{V}}$	The set of virtual nodes (subtasks) in a VNR.
${\mathcal{L}}_{\mathcal{V}}$	The set of virtual links (data dependencies) in a VNR.
$n_v$	An individual virtual node, $n_v\in {\mathcal{N}}_{\mathcal{V}}$.
$l_v$	An individual virtual link, $l_v\in {\mathcal{L}}_{\mathcal{V}}$.
$C\left(n_v\right)$	Computing capacity required by a virtual node $n_v$.
$M\left(n_v\right)$	Memory/storage capacity required by a virtual node $n_v$.
$BW\left(l_v\right)$	Bandwidth required by a virtual link $l_v$.
$D_{max}\left(l_v\right)$	Maximum tolerable delay for a virtual link $l_v$.
Mapping and State Variables
$\mathcal{F}$	The VNE mapping function, $\mathcal{F}:{\mathcal{G}}_{\mathcal{V}}\to {\mathcal{G}}_{\mathcal{S}}^{\prime }$.
$x_{n_s}^{n_v}$	Binary variable; 1 if $n_v$ is mapped to $n_s$, 0 otherwise.
$y_{l_s}^{l_v}$	Binary variable; 1 if $l_s$ is used to map $l_v$, 0 otherwise.
${\mathcal{P}}_{\mathcal{S}}$	A physical path in the substrate network, ${\mathcal{P}}_{\mathcal{S}}\subseteq {\mathcal{L}}_{\mathcal{S}}$.
$A_C\left(n_s\right)$	Available computing capacity on physical node $n_s$.
$A_M\left(n_s\right)$	Available memory capacity on physical node $n_s$.
$A_{BW}\left(l_s\right)$	Available bandwidth on physical link $l_s$.
Performance Metrics and Other Symbols
$t_{arr}^{\left(i\right)}$	Arrival time of the i-th VNR.
$t_{dep}^{\left(i\right)}$	Departure time of the i-th VNR.
$\mathrm{\Delta }{t}^{\left(i\right)}$	Lifetime of the i-th VNR.
$\mathrm{Rev}\left({\mathrm{VNR}}_i\right)$	Revenue generated by the i-th VNR.
$\mathrm{Cost}\left({\mathrm{VNR}}_i\right)$	Cost incurred by embedding the i-th VNR.
${\alpha }_C,{\alpha }_M,{\alpha }_{BW}$	Unit price coefficients for resources.
$\mathrm{hops}\left({\mathcal{P}}_{\mathcal{S}}\right)$	The number of links (hops) in a physical path ${\mathcal{P}}_{\mathcal{S}}$.
T	A long-term time horizon for evaluation.
${\mathcal{R}}_{avg}$	Long-term average revenue.
${\rho }_{RC}$	Long-term revenue-to-cost ratio.
${\rho }_{accept}$	VNR acceptance ratio.

.

2.1. Software-Defined Edge Network Architecture

The underlying physical infrastructure in our study is an SDEN, which is formally modeled as an undirected graph, denoted by ${\mathcal{G}}_{\mathcal{S}}=({\mathcal{N}}_{\mathcal{S}},{\mathcal{L}}_{\mathcal{S}})$. This graph represents the complete set of physical resources available at the network edge for service provisioning.

• Substrate Nodes (${\mathcal{N}}_{\mathcal{S}}$): The set ${\mathcal{N}}_{\mathcal{S}}$ comprises all physical nodes within the edge network. Each substrate node, $n_s\in {\mathcal{N}}_{\mathcal{S}}$, represents a physical entity such as an edge server or a base station equipped with computational capabilities. Each node is characterized by a vector of resource attributes, primarily its total computing capacity, denoted as $C\left(n_s\right)$, and its total storage or memory capacity, denoted as $M\left(n_s\right)$.
• Substrate Links (${\mathcal{L}}_{\mathcal{S}}$): The set ${\mathcal{L}}_{\mathcal{S}}$ represents the physical communication links interconnecting the substrate nodes. Each substrate link, $l_s\in {\mathcal{L}}_{\mathcal{S}}$, that connects two nodes $n_{s,i}$ and $n_{s,j}$ is defined by its total available bandwidth, $BW\left(l_s\right)$, and its inherent communication latency, $D\left(l_s\right)$, which aggregates propagation, transmission, and processing delays.

The entire SDEN is orchestrated by a logically centralized SDN controller [32]. This controller maintains a global and near real-time perspective of the network’s topology and the dynamic state of all substrate resources (e.g., available CPU, memory, and link bandwidth). This centralized intelligence is fundamental for executing globally aware and optimized embedding decisions.

2.2. Application Task Model as a Virtual Network Request

We model a complex, multi-component application that is a candidate for offloading as a Virtual Network Request (VNR). Each VNR, which arrives at the system at a specific time and has a defined lifetime, is represented as a directed graph ${\mathcal{G}}_{\mathcal{V}}=({\mathcal{N}}_{\mathcal{V}},{\mathcal{L}}_{\mathcal{V}})$. This abstraction encapsulates the application’s comprehensive, end-to-end resource requirements.

• Virtual Nodes (${\mathcal{N}}_{\mathcal{V}}$): The set ${\mathcal{N}}_{\mathcal{V}}$ corresponds to the individual subtasks or functional components of an application. Each virtual node, $n_v\in {\mathcal{N}}_{\mathcal{V}}$, specifies the resources necessary for its execution. These requirements are defined by the requested computational capacity, $C\left(n_v\right)$, and the requested memory capacity, $M\left(n_v\right)$.
• Virtual Links (${\mathcal{L}}_{\mathcal{V}}$): The set ${\mathcal{L}}_{\mathcal{V}}$ represents the data dependencies and required communication flows between the application’s subtasks. Each virtual link, $l_v\in {\mathcal{L}}_{\mathcal{V}}$, from $n_{v,i}$ to $n_{v,j}$ signifies that the output of subtask $n_{v,i}$ serves as input for subtask $n_{v,j}$. Each virtual link is characterized by its requested data transfer bandwidth, $BW\left(l_v\right)$, and a maximum tolerable delay constraint for the data transfer, $D_{max}\left(l_v\right)$.

2.3. Problem Formulation

Given the substrate SDEN model ${\mathcal{G}}_{\mathcal{S}}$ and a stochastic stream of incoming VNRs, each represented by a ${\mathcal{G}}_{\mathcal{V}}$, the task offloading problem is formally defined as a VNE process. This process seeks a mapping function, $\mathcal{F}:{\mathcal{G}}_{\mathcal{V}}\to {\mathcal{G}}_{\mathcal{S}}^{\prime }$, where ${\mathcal{G}}_{\mathcal{S}}^{\prime }$ is a valid subgraph of the substrate network ${\mathcal{G}}_{\mathcal{S}}$. A valid mapping must satisfy all resource demands of the VNR while strictly adhering to the capacity constraints of the substrate network. This overarching challenge is decomposed into two constituent sub-problems:

• Node Mapping: Each virtual node $n_v\in {\mathcal{N}}_{\mathcal{V}}$ must be exclusively mapped onto a single physical node $n_s\in {\mathcal{N}}_{\mathcal{S}}$ capable of satisfying its computational and memory requirements. Let a binary variable $x_{n_s}^{n_v}\in \{0,1\}$ indicate if $n_v$ is mapped to $n_s$. The node mapping function is constrained by:

  $$\sum _{n_s\in {\mathcal{N}}_{\mathcal{S}}}{x}_{n_s}^{n_v}=1,\forall n_v\in {\mathcal{N}}_{\mathcal{V}}.$$

  (1)
• Link Mapping: Each virtual link $l_v\in {\mathcal{L}}_{\mathcal{V}}$ connecting virtual nodes $n_{v,i}$ and $n_{v,j}$ must be mapped onto a physical path, denoted as ${\mathcal{P}}_{\mathcal{S}}(n_{s,i},n_{s,j})$, in the substrate network. This path connects the physical nodes $n_{s,i}$ and $n_{s,j}$ that host $n_{v,i}$ and $n_{v,j}$, respectively. The mapped path must collectively provide the required bandwidth and satisfy the end-to-end delay constraint. Let a binary variable $y_{l_s}^{l_v}\in \{0,1\}$ indicate if substrate link $l_s$ is part of the path mapping virtual link $l_v$.

The VNE problem is computationally equivalent to the multi-way separator problem and is thus NP-hard [33]. In the dynamic context of an SDEN, where VNRs arrive and depart stochastically, the primary objective is to develop a policy that can make intelligent embedding decisions online. The overarching goal transcends satisfying individual requests, aiming instead to optimize the long-term operational performance and economic efficiency of the entire system.

3. VNE Constraint and Performance Indicators

This Section details the specific mathematical constraints that must be upheld during the VNE process to ensure a valid resource allocation. Furthermore, it defines the key performance indicators (KPIs) utilized to evaluate the efficacy of our proposed DRL-based strategy. These metrics are essential for quantitatively assessing the long-term system performance and economic viability from the perspective of a network service provider.

3.1. VNE  Constraints

For a VNR to be successfully embedded onto the substrate network, a series of rigorous constraints governing both node and link resource allocation must be met. Let $A_C\left(n_s\right)$, $A_M\left(n_s\right)$, and $A_{BW}\left(l_s\right)$ represent the available computational, memory, and bandwidth resources on substrate node $n_s$ and link $l_s$, respectively.

• Node Resource Constraints: The aggregate resources required by all virtual nodes mapped onto a single physical node must not exceed its physical capacity.

  $$\sum _{\forall {\mathcal{G}}_{\mathcal{V}}}\sum _{n_v\in {\mathcal{N}}_{\mathcal{V}}}{x}_{n_s}^{n_v}·C\left(n_v\right)\le A_C\left(n_s\right),\forall n_s\in {\mathcal{N}}_{\mathcal{S}}.$$

  (2)

  $$\sum _{\forall {\mathcal{G}}_{\mathcal{V}}}\sum _{n_v\in {\mathcal{N}}_{\mathcal{V}}}{x}_{n_s}^{n_v}·M\left(n_v\right)\le A_M\left(n_s\right),\forall n_s\in {\mathcal{N}}_{\mathcal{S}}.$$

  (3)
• Link Resource Constraints: The sum of bandwidth allocated to all virtual links passing through a single physical link cannot exceed its total bandwidth capacity.

  $$\sum _{\forall {\mathcal{G}}_{\mathcal{V}}}\sum _{l_v\in {\mathcal{L}}_{\mathcal{V}}}{y}_{l_s}^{l_v}·BW\left(l_v\right)\le A_{BW}\left(l_s\right),\forall l_s\in {\mathcal{L}}_{\mathcal{S}}.$$

  (4)
• Latency Constraint: The cumulative latency of the physical path ${\mathcal{P}}_{\mathcal{S}}$ selected for a virtual link $l_v$ must not surpass the VNR’s maximum tolerable delay.

  $$\sum _{l_s\in {\mathcal{P}}_{\mathcal{S}}}D\left(l_s\right)\le D_{max}\left(l_v\right),\forall l_v\in {\mathcal{L}}_{\mathcal{V}}.$$

  (5)

Upon a successful embedding, the available resources of the corresponding substrate nodes and links must be updated by decrementing the allocated amounts. Conversely, when a VNR’s lifetime expires, its allocated resources are released back into the available resource pool.

3.2. Performance Evaluation Metrics

To systematically evaluate the performance of our DRL-based VNE strategy, we adopt three widely recognized, long-term metrics. These indicators reflect both the operational success and the economic efficiency of the system. For a given ${\mathrm{VNR}}_i$, let its arrival time be $t_{arr}^{\left(i\right)}$ and its departure time be $t_{dep}^{\left(i\right)}$, defining its lifetime as $\mathrm{\Delta }{t}^{\left(i\right)}=t_{dep}^{\left(i\right)}-t_{arr}^{\left(i\right)}$.

• Long-Term Average Revenue: This metric quantifies the total economic value generated by successfully accommodating VNRs over a sustained period. The revenue of a single VNR is a weighted sum of its requested resources, scaled by its lifetime.

  $$\mathrm{Rev}\left({\mathrm{VNR}}_i\right)=\mathrm{\Delta }{t}^{\left(i\right)}\left(\sum _{n_v\in {\mathcal{N}}_{\mathcal{V}}^{\left(i\right)}}\left({\alpha }_{C}C\left(n_v\right)+{\alpha }_{M}M\left(n_v\right)\right)+\sum _{l_v\in {\mathcal{L}}_{\mathcal{V}}^{\left(i\right)}}{\alpha }_{BW}BW\left(l_v\right)\right),$$

  (6)

  where ${\alpha }_C$, ${\alpha }_M$, and ${\alpha }_{BW}$ are unit price coefficients. The long-term average revenue over a time horizon T is:

  $${\mathcal{R}}_{avg}=\underset{T\to \infty }{lim}{\displaystyle \frac{1}{T}}\sum _{\forall i\mathrm{accepted}\mathrm{in}T}\mathrm{Rev}\left({\mathrm{VNR}}_i\right).$$

  (7)
• Long-Term Average Revenue-to-Cost Ratio (R/C Ratio): This metric assesses the economic efficiency by comparing the revenue generated against the cost of the physical resources consumed. The cost of embedding a VNR includes its consumption of node resources and the network resources, where the latter is proportional to the number of hops in the mapped paths.

  $$\begin{array}{c}\hfill \mathrm{Cos}\mathrm{t}\left({\mathrm{VNR}}_i\right)=\mathrm{\Delta }{t}^{\left(i\right)}\left(\sum _{n_v\in {\mathcal{N}}_{\mathcal{V}}^{\left(i\right)}}\left(C\left(n_v\right)+M\left(n_v\right)\right)+\sum _{l_v\in {\mathcal{L}}_{\mathcal{V}}^{\left(i\right)}}\mathrm{hops}\left({\mathcal{P}}_{\mathcal{S}}\right)·BW\left(l_v\right)\right),\end{array}$$

  (8)

  where $\mathrm{hops}\left({\mathcal{P}}_{\mathcal{S}}\right)$ is the number of physical links in the path ${\mathcal{P}}_{\mathcal{S}}$. The long-term R/C ratio is:

  $${\rho }_{RC}=\underset{T\to \infty }{lim}{\displaystyle \frac{{\sum }_{\forall i\mathrm{accepted}\mathrm{in}T}\mathrm{Rev}\left({\mathrm{VNR}}_i\right)}{{\sum }_{\forall i\mathrm{accepted}\mathrm{in}T}\mathrm{Cos}\mathrm{t}\left({\mathrm{VNR}}_i\right)}}.$$

  (9)
• VNR Acceptance Ratio: This fundamental KPI measures service availability and the system’s capacity to handle incoming demand. It is defined as the ratio of successfully embedded VNRs to the total number of VNRs that arrived over a time window.

  $${\rho }_{accept}=\underset{T\to \infty }{lim}{\displaystyle \frac{\left|\right\{{\mathrm{VNR}}_i\mid {\mathrm{VNR}}_i\mathrm{is}\mathrm{accepted}\mathrm{in}T\left\}\right|}{\left|\right\{{\mathrm{VNR}}_i\mid {\mathrm{VNR}}_i\mathrm{arrives}\mathrm{in}T\left\}\right|}}.$$

  (10)

  A high acceptance ratio signifies a robust and efficient embedding strategy that effectively mitigates resource fragmentation and minimizes service rejection.

4. The Proposed Task Offloading Algorithm Based VNE for SDEN

To address the dynamic and computationally complex VNE problem within the SDEN framework, we propose a novel resource allocation solution based on DRL. Our methodology formulates the sequential VNE process as a Markov Decision Process (MDP) [34] and employs a SAC agent, as shown in Figure 2. This agent, logically centralized at the SDN controller, is designed to learn a near-optimal embedding policy that maximizes long-term system objectives. This section provides a detailed exposition of the MDP formulation, the architectural design of the SAC agent, and the complete algorithm optimization process.

4.1. Markov Decision Process (MDP) Formulation

The sequential decision-making process inherent to VNE is formally modeled as an MDP, defined by the tuple $\langle \mathcal{S},\mathcal{A},\mathcal{P},R,\gamma \rangle$ [35]. Each component of this tuple is specifically tailored to the VNE problem as follows:

(1)

State Space ($\mathcal{S}$): The state ${\mathit{s}}_t\in \mathcal{S}$ at a given time step t must provide a comprehensive snapshot of the substrate network’s current resource availability. Building upon the models in Section 2, we define the state representation based on multi-dimensional features extracted for each physical node $n_s\in {\mathcal{N}}_{\mathcal{S}}$. For each node $n_s$, the feature vector consists of:

• Available Computing Resources ($A_C\left(n_s\right)$): The residual computing capacity on node $n_s$. This is defined as its total capacity minus the sum of capacities of all virtual nodes currently mapped to it:

  $$A_C\left(n_s\right)=C\left(n_s\right)-\sum _{\forall {\mathcal{G}}_{\mathcal{V}}}\sum _{n_v\in {\mathcal{N}}_{\mathcal{V}}}{x}_{n_s}^{n_v}·C\left(n_v\right).$$

  (11)
• Available Memory Resources ($A_M\left(n_s\right)$): The residual memory capacity, defined similarly:

  $$A_M\left(n_s\right)=M\left(n_s\right)-\sum _{\forall {\mathcal{G}}_{\mathcal{V}}}\sum _{n_v\in {\mathcal{N}}_{\mathcal{V}}}{x}_{n_s}^{n_v}·M\left(n_v\right).$$

  (12)
• Aggregated Available Bandwidth ($A_{BW}^{agg}\left(n_s\right)$): A measure of the network connectivity of node $n_s$, defined as the sum of available bandwidth on all links incident to it:

  $$A_{BW}^{agg}\left(n_s\right)=\sum _{l_s\in \mathrm{adj}\left(n_s\right)}{A}_{BW}\left(l_s\right),$$

  (13)

  where $\mathrm{adj}\left(n_s\right)$ is the set of links connected to $n_s$ and $A_{BW}\left(l_s\right)$ is the available bandwidth on link $l_s$.

The complete state is thus represented as a matrix ${\mathit{s}}_t\in {\mathbb{R}}^{|{\mathcal{N}}_{\mathcal{S}}|\times F}$, where $|{\mathcal{N}}_{\mathcal{S}}|$ is the number of physical nodes and F is the number of features per node. This structured representation allows the agent to perceive both nodal capacities and the local network topology context.

(2)

Action Space ($\mathcal{A}$): The embedding of a single VNR with $|{\mathcal{N}}_{\mathcal{V}}|$ virtual nodes is decomposed into a sequence of $|{\mathcal{N}}_{\mathcal{V}}|$ decision steps. At each step $k\in \{1,\dots ,|{\mathcal{N}}_{\mathcal{V}}\left|\right\}$, the agent considers the k-th virtual node, $n_{v,k}$, and performs an action, $a_k$. This action is the selection of a single physical node $n_s\in {\mathcal{N}}_{\mathcal{S}}$ for mapping. The action space at each step is therefore discrete, with $\left|\mathcal{A}\right|=|{\mathcal{N}}_{\mathcal{S}}|$. A composite action for an entire VNR is the ordered sequence of these individual node-mapping actions:

$${\mathit{a}}_{\mathrm{VNR}}=(a_1,a_2,\dots ,a_{|{\mathcal{N}}_{\mathcal{V}}|}),$$

(14)

where each $a_k\in {\mathcal{N}}_{\mathcal{S}}$. This sequential approach makes the complex combinatorial problem tractable for RL.

(3)

Reward Function (R): The reward function is designed to guide the agent towards learning a policy that maximizes resource efficiency. The primary optimization objective is the long-term revenue-to-cost ratio. Therefore, a reward is granted to the agent only upon the completion of a VNR embedding attempt. A successful embedding is rewarded with the calculated R/C ratio of that specific VNR. By maximizing the long-term cumulative reward, the agent is trained not just to accept VNRs, but to find economically efficient embeddings. This single-objective reward function implicitly encourages behaviors that also improve the other key metrics, such as leaving sufficient resources for future VNRs (enhancing the acceptance ratio) and prioritizing profitable requests (boosting revenue). The reward, $r_t$, is formally defined as:

$$r_t=\left\{\begin{array}{cc}{\rho }_{RC},\hfill & \mathrm{if}\mathrm{VNR}\mathrm{is}\mathrm{accepted},\hfill \\ 0,\hfill & \mathrm{if}\mathrm{VNR}\mathrm{is}\mathrm{rejected}.\hfill \end{array}\right.$$

(15)

(4)

State Transition Probability ($\mathcal{P}$) and Discount Factor ($\gamma$): The state transition probability, $\mathcal{P}\left({\mathit{s}}_{t+1}\right|{\mathit{s}}_t,{\mathit{a}}_{\mathrm{VNR}})$, is implicitly defined by the environment’s dynamics. The discount factor, $\gamma$, is set to 0.99, a standard value in DRL that encourages the agent to prioritize long-term cumulative rewards while giving immediate rewards significant weight.

4.2. The Soft Actor-Critic (SAC) Agent Architecture

Our framework employs a single-agent SAC model to effectively navigate the high-dimensional state space and learn a robust embedding policy. The SAC algorithm is an off-policy, actor-critic model designed for maximum entropy RL [36]. This objective encourages broad exploration by training a stochastic policy to maximize both the expected cumulative reward and the policy’s entropy. Within our single-agent context, the agent is composed of the following interconnected neural network components:

(1)

Actor Network (${\pi }_{{\mathit{\theta }}^A}$): The actor, also known as the policy network, is parameterized by ${\mathit{\theta }}^A$. It maps a given state ${\mathit{s}}_t$ to a probability distribution over the discrete action space, yielding ${\pi }_{{\mathit{\theta }}^A}\left({\mathit{a}}_t\right|{\mathit{s}}_t)$. During an embedding, the agent samples an action (i.e., selects a physical node) from this distribution.

(2)

Critic Networks (Q-Networks and V-Network): To ensure stable training and mitigate the overestimation bias common in value-based methods, our SAC agent utilizes a set of critic networks:

• Two Q-Networks ($Q_{{\mathit{\theta }}^{Q1}},Q_{{\mathit{\theta }}^{Q2}}$): Following the principle of clipped double-Q learning, two distinct Q-networks, parameterized by ${\mathit{\theta }}^{Q1}$ and ${\mathit{\theta }}^{Q2}$, are maintained. Each network takes a state-action pair $({\mathit{s}}_t,{\mathit{a}}_t)$ as input and outputs its estimated value, $Q_j({\mathit{s}}_t,{\mathit{a}}_t)$. The minimum of these two Q-values is used during the Bellman updates to provide a conservative, lower-bound estimate of the action-value, which helps prevent overestimation.
• A Value Network ($V_{{\mathit{\theta }}^V}$): This network, parameterized by ${\mathit{\theta }}^V$, estimates the value of a state, $V\left({\mathit{s}}_t\right)$. Its role is to simplify the update rules for the other networks.
• A Target Value Network ($V_{{\mathit{\theta }}^{\overline{V}}}$): A separate target V-network, with parameters ${\mathit{\theta }}^{\overline{V}}$, is maintained. Its parameters are not trained via gradient descent directly but are instead updated slowly as an exponential moving average of the main V-network’s parameters. This use of a “soft” target network provides a stable learning target for the Q-networks, a crucial element for convergence in temporal-difference learning.

The detailed network structures are specified in

Table 2. Network Structure Configuration for the SAC Agent.

Network	Layer	Input → Output Dimension	Activation
Actor (${\pi }_{{\mathit{\theta }}^A}$)	Input	$|{\mathcal{N}}_{\mathcal{S}}|\times F\to 128$	ReLU
	Hidden	$128\to 128$	ReLU
	Output	$128\to |{\mathcal{N}}_{\mathcal{S}}|$	Softmax
Value ($V_{{\mathit{\theta }}^V}$)	Input	$|{\mathcal{N}}_{\mathcal{S}}|\times F\to 128$	ReLU
	Hidden	$128\to 128$	ReLU
	Output	$128\to 1$	Linear
Q-Critics ($Q_{{\mathit{\theta }}^{Qj}}$)	Input	$\left(\right|{\mathcal{N}}_{\mathcal{S}}|\times F)\to 128$	ReLU
	Hidden	$128\to 128$	ReLU
	Output	$128\to \left|\mathcal{A}\right|$	Linear

. The input state matrix ${\mathit{s}}_t$ is first processed by convolutional layers to extract spatial features representing the network topology, followed by fully connected layers to produce the final outputs.

4.3. Algorithm Optimization Process

The SAC agent is trained using an off-policy approach, leveraging an experience replay buffer and optimizing for a maximum entropy objective.

(1)

Experience Replay Buffer ($\mathcal{D}$): The agent continuously interacts with the SDEN environment. Each complete interaction, from the start of a VNR embedding to its final outcome (success or failure), results in a trajectory of experience tuples. A summary tuple, $({\mathit{s}}_t,{\mathit{a}}_t,r_t,{\mathit{s}}_{t+1})$, representing the initial state, the composite action sequence, the final reward, and the resulting network state, is stored in a large replay buffer $\mathcal{D}$. For training, mini-batches of these experience tuples are randomly sampled from $\mathcal{D}$. This practice breaks the temporal correlation between consecutive samples and leads to more stable and efficient learning.

(2)

Maximum Entropy Objective: In contrast to traditional RL algorithms that solely seek to maximize the cumulative reward, SAC enhances the objective with a policy entropy term. For a discrete action space, the entropy of the policy $\pi$ at state ${\mathit{s}}_t$ is defined as:

$$\mathcal{H}\left(\pi (·|{\mathit{s}}_t)\right)=-\sum _{{\mathit{a}}_t\in \mathcal{A}}\pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)log\left(\pi \left({\mathit{a}}_t\right|{\mathit{s}}_t)\right).$$

(16)

The unified optimization objective is to find a policy ${\pi }^{*}$ that maximizes the sum of the expected reward and the policy’s entropy:

$${\pi }^{*}=arg\underset{\pi }{max}\sum _t{\mathbb{E}}_{({\mathit{s}}_t,{\mathit{a}}_t)\sim {\rho }_{\pi }}\left[r_t+\alpha \mathcal{H}\left(\pi (·|{\mathit{s}}_t)\right)\right],$$

(17)

where ${\rho }_{\pi }$ denotes the state-action marginals of the trajectory distribution induced by policy $\pi$. The temperature parameter, $\alpha >0$, is a crucial hyperparameter that controls the relative importance of the entropy term versus the reward, thereby balancing the exploration-exploitation trade-off.

(3)

Network Parameter Updates: The neural networks are updated iteratively by minimizing their respective loss functions, calculated using mini-batches of experiences sampled from $\mathcal{D}$. Let a mini-batch experience be denoted by $(\mathit{s},\mathit{a},r,{\mathit{s}}^{\prime })$.

• Value Network Update: The V-network, parameterized by ${\mathit{\theta }}^V$, is trained to minimize the soft Bellman residual. Its loss function, $L_V\left({\mathit{\theta }}^V\right)$, is given by:

  $$L_V\left({\mathit{\theta }}^V\right)={\mathbb{E}}_{\mathit{s}\sim \mathcal{D}}\left[{\displaystyle \frac{1}{2}}{\left(V_{{\mathit{\theta }}^V}\left(\mathit{s}\right)-{\mathbb{E}}_{\mathit{a}\sim {\pi }_{{\mathit{\theta }}^A}}\left[\underset{j=1,2}{min}{Q}_{{\mathit{\theta }}^{Qj}}(\mathit{s},\mathit{a})-\alpha log{\pi }_{{\mathit{\theta }}^A}\left(\mathit{a}\right|\mathit{s})\right]\right)}^2\right].$$

  (18)
• Q-Network Update: The parameters ${\mathit{\theta }}^{Qj}$ of each Q-network are trained to minimize the mean squared Bellman error (MSBE). The learning target is computed using the reward r and the value of the next state ${\mathit{s}}^{\prime }$ from the target V-network:

  $$L_Q\left({\mathit{\theta }}^{Qj}\right)={\mathbb{E}}_{(\mathit{s},\mathit{a},r,{\mathit{s}}^{\prime })\sim \mathcal{D}}\left[{\displaystyle \frac{1}{2}}{\left(Q_{{\mathit{\theta }}^{Qj}}(\mathit{s},\mathit{a})-\left(r+\gamma V_{{\mathit{\theta }}^{\overline{V}}}\left({\mathit{s}}^{\prime }\right)\right)\right)}^2\right].$$

  (19)
• Actor Network Update: The actor (policy network) is updated by minimizing the Kullback–Leibler divergence between the policy distribution and the softmax of the Q-function values. This encourages the policy to favor actions with higher Q-values. The policy loss, $L_{\pi }\left({\mathit{\theta }}^A\right)$, is:

  $$L_{\pi }\left({\mathit{\theta }}^A\right)={\mathbb{E}}_{\begin{array}{c}\mathit{s}\sim \mathcal{D}\\ \mathit{a}\sim {\pi }_{{\mathit{\theta }}^A}\end{array}}\left[\alpha log\left({\pi }_{{\mathit{\theta }}^A}\left(\mathit{a}\right|\mathit{s})\right)-\underset{j=1,2}{min}{Q}_{{\mathit{\theta }}^{Qj}}(\mathit{s},\mathit{a})\right].$$

  (20)
• Target Network Soft Update: At each training step, the parameters of the target value network, ${\mathit{\theta }}^{\overline{V}}$, are updated via a soft update, which is an exponential moving average of the main value network’s parameters. This ensures smoother and more stable updates.

  $${\mathit{\theta }}^{\overline{V}}\leftarrow \tau {\mathit{\theta }}^V+(1-\tau ){\mathit{\theta }}^{\overline{V}},$$

  (21)

  where $\tau \in [0,1]$ is the smoothing factor for the target update, typically a small value like $0.005$.

4.4. VNE Decision and Learning Algorithm

The complete online VNE decision and training procedure is outlined in Algorithm 1. The algorithm operates in a continuous loop. The SDN controller, acting as the agent, receives incoming VNRs. For each VNR, it uses its current policy (actor network) to make a sequence of embedding decisions. After the final outcome is determined, the agent receives a reward and stores the experience in the replay buffer. Periodically, the agent samples from this buffer to perform a gradient descent step on all its networks, thereby progressively refining its VNE strategy. This closed-loop process allows the agent to continuously adapt and improve its performance in a dynamic SDEN environment.

To evaluate the agent’s learning progress, a validation process is implicitly performed. Periodically during training (e.g., after each epoch), the current policy’s performance is measured, and the results are averaged to produce the convergence curves. Once the training converges, the testing phase begins. In this phase, the learned actor network ${\pi }_{{\mathit{\theta }}^A}$ is deployed and its parameters are frozen (i.e., no further training updates are performed). The agent then processes a new, unseen stream of VNRs. The performance metrics are collected throughout this phase to fairly benchmark our proposed algorithm against the baselines.

Algorithm 1: The Proposed Task Offloading Algorithm Based VNE for SDEN

Input: SDEN environment simulator generating ${\mathcal{G}}_S$ and a stream of VNRs; Uninitialized SAC networks; Hyperparameters. Output: A trained actor network ${\pi }_{{\mathit{\theta }}^A}$ capable of making proficient VNE decisions. Initialization: actor network ${\pi }_{{\mathit{\theta }}^A}$ and critic networks ($Q_{{\mathit{\theta }}^{Q1}},Q_{{\mathit{\theta }}^{Q2}},V_{{\mathit{\theta }}^V}$) with random parameters; target value network parameters by cloning: ${\mathit{\theta }}^{\overline{V}}\leftarrow {\mathit{\theta }}^V$; an empty replay buffer $\mathcal{D}$; state ${\mathit{s}}_0$. Main Training Loop:   1: for each epoch $m=1,2,\dots ,M$ do   2:     Receive a new VNR, ${\mathcal{G}}_{\mathcal{V}}$, from the environment;   3:     Initialize an empty mapping for the current VNR, $\mathcal{M}\leftarrow \varnothing$;   4:     Set node embedding success flag, $succes{s}_{node}\leftarrow \mathrm{true}$;   5:     for each virtual node $n_{v,k}\in {\mathcal{N}}_{\mathcal{V}}$ do   6:         The agent selects a physical node (action) based on its policy: ${\mathit{a}}_k\sim {\pi }_{{\mathit{\theta }}^A}$;   7:         if the chosen physical node ${\mathit{a}}_k$ has sufficient resources for $n_{v,k}$ then   8:            Add the mapping $(n_{v,k}\to a_k)$ to $\mathcal{M}$;   9:         else 10:            $succes{s}_{node}\leftarrow \mathrm{false}$; 11:            break the inner loop. $▹$ Node mapping failed; 12:         end if 13:     end for 14:     if $succes{s}_{node}$ is true then 15:         Attempt to map all virtual links $l_v\in {\mathcal{L}}_{\mathcal{V}}$ based on $\mathcal{M}$ using a k-shortest path algorithm; 16:         if all virtual links are successfully mapped (satisfying bandwidth and delay) then 17:            Calculate the reward: $r_t$; 18:         else 19:            Set the reward: $r_t\leftarrow 0$; 20:         end if 21:     else 22:         Set the reward: $r_t\leftarrow 0$; 23:     end if 24:     The environment transitions to the next state ${\mathit{s}}^{\prime }$; 25:     Store the experience tuple $\mathit{s},{\mathit{a}}_{\mathrm{VNR}},r_t,{\mathit{s}}^{\prime })$ in the replay buffer $\mathcal{D}$; 26:     Update the current state: $\mathit{s}\leftarrow {\mathit{s}}^{\prime }$; 27:     if $\left|\mathcal{D}\right|>\mathrm{BatchSize}$ then 28:         Sample a random mini-batch of transitions from $\mathcal{D}$. 29:         Update the Q-critic networks $Q_{{\mathit{\theta }}^{Q1}}$ and $Q_{{\mathit{\theta }}^{Q2}}$ by minimizing $L_Q\left({\mathit{\theta }}^{Qj}\right)$. 30:         Update the value network $V_{{\mathit{\theta }}^V}$ by minimizing $L_V\left({\mathit{\theta }}^V\right)$. 31:         Update the actor network ${\pi }_{{\mathit{\theta }}^A}$ by minimizing $L_{\pi }\left({\mathit{\theta }}^A\right)$. 32:         Perform a soft update on the target value network parameters: ${\mathit{\theta }}^{\overline{V}}\leftarrow \tau {\mathit{\theta }}^V+(1-\tau ){\mathit{\theta }}^{\overline{V}}$. 33:     end if 34: end for

5. Simulation Experiment

In this section, we present a comprehensive empirical evaluation of our proposed SAC-based VNE algorithm for task offloading in a SDEN environment. The primary objectives of these experiments are twofold: first, to demonstrate the learning convergence and stability of our DRL agent, and second, to rigorously benchmark its performance against several established VNE algorithms. We begin by detailing the simulation environment, including the generation of substrate network topologies and dynamic VNR arrivals, followed by a description of the baseline algorithms and performance metrics. Finally, we will analyze the training process and the comparative performance results.

5.1. Simulation Environment Setup

To create a realistic yet controlled experimental setting, we developed a discrete-time event simulator in Python 3.6.12, leveraging libraries such as NetworkX 1.11 for graph modeling and PyTorch 1.4.0 for implementing the DRL agent. The simulation environment is meticulously configured to mirror the key characteristics of a dynamic SDEN.

• Substrate Network (SDEN) Generation: The physical SDEN topology is generated using the GT-ITM topology generator, a standard tool for creating realistic network graphs. We simulate an edge network consisting of 100 physical nodes ($\parallel {\mathcal{N}}_{\mathcal{S}}\parallel =100$), representing a moderately sized metropolitan area network. These nodes are interconnected by 500 physical links ($\parallel {\mathcal{L}}_{\mathcal{S}}\parallel =500$). The computational and storage resources of the physical nodes, as well as the bandwidth and latency attributes of the physical links, are uniformly distributed within predefined ranges, reflecting the inherent heterogeneity of edge infrastructure. The specific resource distributions are detailed in

  Table 3. Configuration Details for the Simulation Environment.

  Parameter Category	Configuration
  Substrate Network (SDEN) Configuration
  Number of Physical Nodes ($\parallel {\mathcal{N}}_{\mathcal{S}}\parallel$)	100
  Number of Physical Links ($\parallel {\mathcal{L}}_{\mathcal{S}}\parallel$)	500
  Computing Capacity of Nodes ($C\left(n_s\right)$)	U [50, 100]
  Memory Capacity of Nodes ($M\left(n_s\right)$)	U [50, 100]
  Bandwidth of Links ($BW\left(l_s\right)$)	U [50, 100]
  Latency of Links ($D\left(l_s\right)$)	U [1, 20]
  Virtual Network Request (VNR) Configuration
  Total Number of VNRs (training & testing)	2000 (1000 + 1000)
  Number of Virtual Nodes ($|{\mathcal{N}}_{\mathcal{V}}|$)	U [2, 10]
  Virtual Link Connectivity Probability	0.5
  Requested Computing ($C\left(n_v\right)$)	U [1, 20]
  Requested Memory ($M\left(n_v\right)$)	U [1, 20]
  Requested Bandwidth ($BW\left(l_v\right)$)	U [1, 20]
  Max. Tolerable Delay ($D_{max}\left(l_v\right)$)	U [1, 20]
  SAC Agent Hyperparameter Configuration
  Actor Network Learning Rate (${\lambda }_A$)	0.0001
  Critic Networks Learning Rate (${\lambda }_C$)	0.0001
  Discount Factor ($\gamma$)	0.99
  Replay Buffer Size ($\left|\mathcal{D}\right|$)	50,000
  Batch Size	128
  Target Network Update Factor ($\tau$)	0.01
  Temperature Parameter ($\alpha$)	0.1

  .
• Virtual Network Request (VNR) Generation: To simulate a dynamic workload, VNRs arrive at the system according to a Poisson process, with a configurable average arrival rate ($\lambda$). The lifetime of each VNR follows an exponential distribution. The structural characteristics of each VNR, such as the number of virtual nodes and their connectivity, are also randomly generated. The number of virtual nodes per VNR is drawn from a uniform distribution between 2 and 10, and any two virtual nodes are connected with a probability of 0.5. The resource demands (computing, memory, bandwidth, and delay constraints) for each VNR are uniformly sampled from the ranges specified in Table 3.
• Experimental Parameters: Table 3 provides a comprehensive summary of all simulation parameters, including the physical network configuration, VNR characteristics, and the crucial hyperparameters used for training our SAC agent. These DRL parameters were determined through empirical tuning to ensure stable and efficient convergence.

5.2. Baseline Algorithms

To rigorously evaluate the performance of our proposed SAC-VNE algorithm, we compare it against a set of well-established and representative baseline algorithms from the literature, covering both heuristic and other learning-based approaches:

• NodeRank: A classic greedy heuristic algorithm. It prioritizes the mapping of virtual nodes onto physical nodes with the highest “rank”, a metric that combines both available resources and topological importance. It is a widely used benchmark for its simplicity and effectiveness in static scenarios.
• RCR: This heuristic algorithm is designed to optimize the Revenue-to-Cost Ratio directly. It follows a greedy approach by always selecting the node and link mappings that yield the best immediate R/C ratio for the current VNR, without considering long-term consequences.
• CDRL: A pioneering RL-based VNE algorithm that models the node mapping as a sequential decision process. This provides a relevant DRL-based comparison, although its core algorithm may differ from our SAC implementation.
• GRL: A more advanced DRL algorithm that integrates a GCN to extract topological features from the substrate network. This serves as a state-of-the-art DRL baseline, allowing us to assess the specific benefits of our SAC formulation against other powerful network representation techniques.

• All baseline algorithms are implemented and executed within the identical simulation environment to ensure a fair and rigorous comparison.

5.3. Performance Analysis

Before presenting the detailed performance results, it is important to clarify how our simulation environment addresses the variability of system load. Instead of testing under separate, static scenarios, our experimental setup is designed to simulate a continuous and dynamic operational environment where the system load inherently fluctuates over time. This is achieved through three key mechanisms:

• Stochastic VNR arrivals: VNRs arrive according to a Poisson process, a standard model for simulating random request arrivals. This naturally creates periods of high traffic intensity (high load) when many VNRs arrive in a short time, and periods of lower intensity (light/medium load).
• Finite VNR lifetimes: Each accepted VNR occupies resources for a finite, exponentially distributed lifetime before releasing them. The continuous cycle of resource allocation and deallocation ensures that the total occupied system resources (i.e., the system load) are constantly changing.
• Randomized resource demands: The resource requirements for each VNR are randomly generated from uniform distributions. This heterogeneity means that the impact of each VNR on the system load varies, further contributing to a dynamic load profile.

Therefore, the long-term performance metrics presented in our results are not reflective of a single load condition. Instead, they represent the average performance of each algorithm across a continuously evolving spectrum of system loads, which we argue is a more realistic and comprehensive method for evaluating its robustness and adaptability in a real-world dynamic SDEN environment.

The performance evaluation is conducted in two main phases. First, we analyze the agent’s learning behavior during the training phase to demonstrate convergence and stability. Second, we perform a rigorous comparative analysis of the final, converged policy against the selected baseline algorithms to validate its effectiveness and superiority.

5.3.1. Training Performance and Convergence Analysis

During the training phase, our SAC agent interacts with the simulated SDEN environment over a total of 600 epochs. To monitor the learning progress, we plot the evolution of the three key performance metrics as a function of the training epochs. These convergence curves, shown in Figure 3, Figure 4 and Figure 5, illustrate the agent’s ability to learn and progressively improve its embedding policy from a random initialization to a stable and effective strategy.

As depicted in all three figures, the initial performance of the agent is poor across all metrics. For instance, the long-term average revenue (Figure 3) starts at a low value, indicating that the randomly initialized policy is making inefficient or often unsuccessful embedding decisions. This is expected, as the agent has not yet learned any meaningful correlation between network states and optimal actions. However, as the training progresses, a clear upward trend is observable in all three metrics. The agent begins to explore the environment, and through the feedback provided by the meticulously designed reward function, it starts to favor actions that lead to higher revenue, better efficiency, and successful embeddings.

The curves exhibit a period of rapid improvement in the early stages (approximately the first 100–150 epochs), followed by a phase of diminishing returns where performance continues to improve but at a slower rate. After approximately 400 epochs, all three metrics, long-term average revenue, R/C ratio, and VNR acceptance ratio, begin to plateau and fluctuate within a stable, high-performance range. This behavior signifies that the SAC agent’s policy has converged. The agent has successfully learned to balance the exploration-exploitation trade-off, transitioning from exploring a wide range of actions to consistently exploiting its learned knowledge to make near-optimal embedding decisions. The stability in the later epochs confirms that the proposed training framework is robust and capable of producing a reliable policy for dynamic VNE.

5.3.2. Comparative Performance Evaluation

After the training process converges, the learned policy of our SAC-VNE agent is frozen and subjected to a comprehensive testing phase using a new, unseen set of VNRs. Its performance is benchmarked against the baseline algorithms across our three primary metrics over a simulated testing period of 24,000 time units. The results, presented in Figure 6, Figure 7 and Figure 8, demonstrate the sustained performance of each algorithm under a continuous stream of dynamic network loads.

Figure 6 illustrates the performance comparison in terms of long-term average revenue. Our proposed SAC-VNE algorithm consistently and significantly outperforms all baseline algorithms. This superiority stems from its ability to make farsighted decisions that optimize for long-term rewards, a stark contrast to the myopic nature of heuristic baselines like NodeRank and RCR, while the other DRL-based methods (GRL and CDRL) also show strong performance, SAC-VNE’s higher revenue indicates a more effective policy, likely due to the SAC algorithm’s enhanced exploration capabilities and stable learning dynamics, which allow it to discover more profitable embedding strategies.

Figure 7 presents the comparative analysis of the long-term R/C ratio, a direct measure of resource utilization efficiency. Once again, our SAC-VNE algorithm achieves the highest R/C ratio. This result is particularly significant as it demonstrates that our agent does not just chase high-revenue VNRs at any cost; it has learned to find resource-efficient mappings that minimize the consumption of substrate resources (e.g., by favoring shorter network paths). The greedy RCR baseline, while designed to optimize this metric, performs poorly in the long run, showcasing the pitfalls of local optimization, which can lead to severe resource fragmentation.

Finally, Figure 8 compares the VNR acceptance ratio of all algorithms. The SAC-VNE algorithm maintains the highest acceptance rate throughout the simulation. This outcome validates our hypothesis that a learning-based approach, optimized for long-term system health, can more effectively mitigate resource fragmentation than heuristic methods. By making intelligent, non-greedy choices, our agent preserves resource contiguity, thus leaving the network in a better state to accommodate future VNRs. The consistent performance gap between SAC-VNE and the other baselines across all three metrics provides strong evidence of its effectiveness and robustness for dynamic task offloading in SDENs.

Quantitatively, across the entire simulation period, our proposed SAC-VNE algorithm demonstrates a remarkable and consistent advantage. On average, it improves the long-term average revenue by 19.3%, the long-term revenue-to-cost ratio by 13.7%, and the VNR acceptance ratio by 9.1% when compared to the average performance of all baseline algorithms. These figures provide strong, quantitative evidence of our method’s effectiveness and robustness for dynamic task offloading in SDENs.

5.3.3. Result Discussion and Justification

The substantial performance enhancements achieved by the proposed SAC-VNE framework over the baselines can be justified by several key algorithmic properties:

• Overcoming myopic heuristics: Heuristic methods like NodeRank and RCR make myopic, greedy decisions to satisfy immediate VNR demands, while this provides short-term gains, it inevitably leads to bottleneck resource exhaustion and severe network fragmentation over time, causing future profitable VNRs to be rejected. Our SAC agent, conversely, optimizes for the long-term cumulative discounted reward. It learns to strategically sacrifice short-term suboptimal placements to preserve critical node and link contiguity for future, highly profitable requests.
• Maximum entropy exploration: Compared to traditional DRL methods like CDRL, our algorithm leverages the SAC framework. SAC’s unique maximum entropy objective intrinsically encourages broader exploration of the vast action space (mapping combinations). This prevents the agent from converging prematurely to local optima, ensuring a more robust and adaptable policy under dynamic traffic conditions.
• Stability of double-Q learning: The incorporation of clipped double Q-learning within the critic network design actively mitigates the overestimation bias that commonly plagues temporal-difference learning in complex state spaces. This allows the SAC agent to evaluate the true long-term value of a mapping action more accurately than simpler DRL variants, translating directly into superior resource allocation efficiency (highest R/C ratio) and maximized sustained revenue.

6. Conclusions

This paper addresses the critical challenge of efficient resource allocation in dynamic SDENs by proposing a novel framework that integrates SDN, VNE, and DRL. We innovatively reformulate the multi-component task offloading problem as a VNE challenge and employ a SAC agent to learn an optimal embedding policy. This agent is guided by a reward function designed to maximize the long-term revenue-to-cost ratio, thereby fostering both economically and resource-efficient decisions. Our comprehensive simulations validate the significant superiority of this approach. Specifically, the proposed SAC-VNE algorithm demonstrated an average improvement of 19.3% in long-term revenue, 13.7% in revenue-to-cost ratio, and 9.1% in VNR acceptance rate compared to established baselines. These results confirm that framing task offloading as a VNE problem and solving it with an advanced DRL technique is a highly effective and superior paradigm for intelligent and adaptive resource management in modern edge computing environments.