Offline Joint Network and Computational Resource Allocation for Energy-Efficient 5G and beyond Networks

Abstract

In order to cope with the ever-increasing traffic demands and stringent latency constraints, next generation, i.e., sixth generation (6G) networks, are expected to leverage Network Function Virtualization (NFV) as an enabler for enhanced network flexibility. In such a setup, in addition to the traditional problems of user association and traffic routing, Virtual Network Function (VNF) placement needs to be jointly considered. To that end, in this paper, we focus on the joint network and computational resource allocation, targeting low network power consumption while satisfying the Service Function Chain (SFC), throughput, and delay requirements. Unlike the State-of-the-Art (SoA), we also take into account the Access Network (AN), while formulating the problem as a general Mixed Integer Linear Program (MILP). Due to the high complexity of the proposed optimal solution, we also propose a low-complexity energy-efficient resource allocation algorithm, which was shown to significantly outperform the SoA, by achieving up to 78% of the optimal energy efficiency with up to 742 times lower complexity. Finally, we describe an Orchestration Framework for the automated orchestration of vertical-driven services in Network Slices and describe how it encompasses the proposed algorithm towards optimized provisioning of heterogeneous computation and network resources across multiple network segments.

1. Introduction

Beyond 5G (B5G) and 6G networks are envisioned to meet a plethora of service requirements supporting high resource and technology heterogeneity, while adopted architectural paradigms, such as Centralized-Radio Access Network (C-RAN) and Network Function Virtualization (NFV), offer the necessary high flexibility and configurability to the network. The increasing use of NFV, in particular, to “softwarize” functions and applications [1,2,3] (i.e., decouple the function logic from the underlying hardware running the actual code) in numerous Open System Interconnection (OSI) layers has transformed the majority of “traditional” network functions (e.g., Network Address Translation (NAT), firewall, load balancing, and Intrusion Detection/Prevention System (IDPS)) into Virtual Network Functions (VNFs), which can run on generic computational infrastructure that can be dynamically deployed. Although we use the generic term “Central Processing Unit (CPU) resources” to refer to this computational infrastructure, our model and ensuing analysis can capture any type of computational device, including Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs) and other computational acceleration platforms.

The NFV-supporting physical computational infrastructure contains a number of nodes, including traffic-forwarding switches and computational nodes of different capabilities that can host VNFs in the form of Virtual Machines (VMs), containers, or unikernels. Although traditional cellular architectures were mainly equipped with cloud servers, located at distant locations offering high computational power at low cost, the need for supporting ultra-low latency B5G services has motivated Multi-Access Edge Computing (MEC) [4]. To this end, MEC nodes offer computational capabilities very close to the user, often being attached to Base Stations (BSs), and are able to achieve ultra-low latency at the expense of high cost, thus leading to a nontrivial cost/distance tradeoff that needs to be quantitatively examined.

The above flexible deployment introduces a new exploitable “degree of freedom” regarding the potential location of the deployed VNFs (referred to as the “VNF placement problem” [5,6]) but also introduces new challenges to traditional user association and traffic routing problems, since these problems are strongly coupled with the location of the nodes running the VNFs. Furthermore, the VNFs themselves typically operate in synergy with each other to form Service Function Chains (SFCs), i.e., ordered sequences of VNFs which process packets in an End-to-End (E2E) manner, within the operator’s network, and according to specific rules matching a given service request. This ordering adds a crucial constraint to the routing problem, as the SFC is considered to successfully meet the service request only when the selected routed path passes through the VNFs comprising the specific SFC in exactly the order specified; for example, if an SFC is described by the ordered sequence NAT→Firewall→NAT, then all packets belonging to this SFC must be processed by only these three VNFs in the specified order.

The above discussion motivates the need to jointly study VNF placement alongside computational and communication resource allocation in a mobile network, while satisfying the SFC, throughput, and delay requirements [7,8,9]. In addition, B5G is expected to include a variety of different access and transport technologies with distinct characteristics that should be jointly studied. Specifically, apart from fiber links interconnecting the physical nodes, wireless links may also be deployed as an X-haul transport solution, to offer high flexibility close to the Access Network (AN) [10]. Millimeter wave (mmWave) constitutes a very promising candidate to serve this purpose, due to its high bandwidth availability and antenna gains that are able to compensate for the higher path loss in this band. Moreover, for the AN, comprised of gNodeBs (gNBs) densely overlaid with Small Cells (SCs), 5G-New Radio (5G-NR) proposes the use of multiple frequencies (including mmWave). Hence, a holistic network resource planning study should jointly consider: (1) all types of technologies, e.g., 5G-NR, mmWave, fiber, along with their benefits and constraints, and (2) the allocation of all different resource types (i.e., communication, computational, and storage).

It is also crucial to consider the whole network path from the traffic source to the destined User Equipment (UE), so as to satisfy the service latency constraint and offer true E2E optimality, within the operator’s network boundaries. Furthermore, achieving high energy efficiency is of utmost importance not only to limit the network operator’s operational costs (thus, increasing its revenue) but also to decrease the Information and Communications Technology (ICT) carbon footprint, leading to eco-friendly B5G networks. Hence, we focus on energy-efficient network resource planning solutions to jointly solve the user association, VNF placement (SFC chaining), and traffic routing problem in the highly heterogeneous B5G networks.

In addition, although the discussion so far has focused on (energy-optimal) establishment of the data plane to satisfy the requested services, there is a strong need for an accompanying flexible management and control plane that supports the actual dynamic provisioning and/or configuration of resources in B5G network infrastructures. Network Slicing is considered as a key enabling concept for achieving a high degree of automation in the provisioning of services in B5G networks, allowing, at the same time, resource sharing within the operator’s network and the fulfilment of performance requirements depending on the service type (i.e., enhanced Mobile Broadband – eMBB, ultra Reliable Low Latency Communication – uRLLC, and massive Machine Type Communication – mMTC). Network Slicing improves the way network and computational resources are allocated and also offers the possibility of performing runtime optimization that targets Quality of Service (QoS) preservation as well as energy-efficient deployments and the reduction of operational cost.

Combining Network Slicing, Software Defined Networking (SDN), and NFV/MEC orchestration techniques with advanced strategies for resource planning, VNFs of different types, potentially belonging to different network segments (radio, core, and transport network), can be placed and configured in an optimal manner, while the SFC connectivity is guaranteed through the establishment of optimized jointly computed network paths.

1.1. Related Work

The joint problem of VNF placement, SFC chaining, and routing in wired networks, mostly targeting cloud environments in the network core, has been widely studied with tools such as Dynamic Programming [11], knapsack algorithms [12], Monte Carlo Tree Search [13], and Benders decomposition [14]. The above works, including the recent ones in [7,15,16], formulate NP-hard Mixed Integer Linear (MILP) and Nonlinear Programs, for which heuristic algorithms are proposed and numerically evaluated. Hence, their essential differences lie in the considered constraints and objectives (i.e., minimum total network power consumption in [15], link utilization, overhead, and server power consumption in [16], and monetary profit in [7]). Models with a distinct MEC/cellular flavor appear in [17,18] (see also survey in [19]); however, [17] does not consider power consumption, while [18] models the link constraints more abstractly than in our paper and uses a different objective (i.e., minimize maximum link utilization).

Regarding Network Slicing, SDN, and NFV/MEC orchestration, many platforms, both commercial and open-source, provide functionalities that target the management and control of technology-specific resources. The Open Source Management and Orchestration (MANO) framework proposed by the European Telecommunications Standards Institute (ETSI), considered the most mature standardized solution for Network Service Lifecycle Management (LCM) [20], also supports Network Slicing but is not currently targeting the management of radio elements. On the other hand, O-RAN, being the most successful of the open Radio Access Network (RAN) initiatives, targets specifically the virtualization and management of radio functions [21]. During Horizon 2020 5G Infrastructure Public Private Partnership (5G-PPP) phase 2, some research initiatives focused on prototyping Network Slicing solutions considering (mostly) computational resources [22] and integrating, in some cases, mechanisms for joint management of the transport [23] and radio network segments [24].

1.2. Research Gap

Most of the work on resource allocation (i.e., VNF placement, SFC chaining and routing) mentioned in Section 1.1 does not consider mobile networks and, even when they do, they ignore the wireless AN segment. The last remark motivates our paper, which also studies joint VNF placement and routing in a MEC/cloud-enabled heterogeneous mobile network consisting of macro BSs and SCs. We explicitly include the AN, as well as potential wireless X-haul links, while accounting for the inherent wireless channel fluctuations. Hence, this paper extends the concrete and detailed communication model of [25] by adding all necessary controls for the computational resources and by modeling the associated delay and capacity constraints. In addition, the extensive technical documentation released by Standards Developing Organizations (SDOs) (such as ETSI) regarding the E2E network efficiency of mobile networks in conjunction with the NFV-supporting infrastructures [26,27], indicate that solutions targeting E2E network energy efficiency are of prime importance.

On the other hand, the multiplicity of technology-specific orchestration platforms presents interoperability challenges, which motivates current research in 5G and B5G networks towards delivery of E2E Network Slicing solutions that support the orchestration and management of lheterogeneous resources in distributed edge to cloud infrastructures and across the different network segments [28], i.e., radio, core, and transport. In addition, although there are solutions that optimize the hardware (optimized design of dedicated solutions) or the job allocation in the computing nodes, they provide local optimization benefits, while overlooking the holistic network optimization. To this end, so far, there has been no complete solution for a unified Orchestration Framework having a holistic view of the entire network (potentially by integrating existing technology-specific platforms such as ETSI MANO and O-RAN) and using it for the management and orchestration of Vertical-driven services within E2E Network Slices provisioned and configured over multi-technology components. The search for such a solution provides additional motivation for our paper.

1.3. Our Contributions

The paper’s contributions are summarized as follows:

• We formulate a concrete joint user association, traffic routing, and VNF placement optimization problem targeting at the overall E2E network performance optimization, with minimal assumptions, that accounts for both communication and computation resources in all segments of a mobile network (i.e., AN, edge, and core) and explicitly account for the AN segment, typically ignored in the literature.
• Due to the NP-hardness of the resulting problem, we also propose a heuristic algorithm, evaluate its performance via simulations and demonstrate its superior performance compared with other State-of-the-Art (SoA) algorithms. The proposed solutions can be applied for internode network optimization, in conjunction with approaches targeting intranode optimization for maximum performance.
• Expanding upon our previous work in [29], we also describe the proposed orchestration solution, which, integrated with the proposed algorithm, enables the automated and optimized provisioning and configuration of heterogeneous computational and network resources across all network segments, targeting the orchestration of virtualized services according to the expected performance requirements and the specified SFC.

Our formulation and algorithm can also be employed by a mobile network operator as an offline tool, during the network planning stage, to provide quantitative answers on the power expenditure and computational resources (both of them major components of Operational Expenditure (OPEX) and Capital Expenditure (CAPEX)) required to support a given set of services.

The rest of the paper is structured as follows: Section 2 describes the system model and problem formulation, while the proposed heuristic and the employed Orchestration Platform are presented in Section 3 and Section 4, respectively. Our performance evaluation methodology and comparison between the optimal solution and other SoA algorithms is described in Section 5 with the actual results being presented in Section 6. Section 7 concludes the paper. Notation-wise, sets are denoted with calligraphic symbols $\mathcal{V},\mathcal{F}$ etc. and $\stackrel{\Delta }{=}$ denotes equality by definition.

2. System Model and Problem Statement

We consider the RAN and Core segments of a mobile network and model them as a graph $\mathcal{G}(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of nodes (excluding mobile users) and $\mathcal{E}$ is the set of non-access edges/links among them, as illustrated in Figure 1 (which also shows access links, for completeness). The nodes in $\mathcal{V}$ comprise gNB and/or SCs (hereafter referred to as BSs) in the RAN, as well as switches/routers and other middlebox devices (e.g., load balancers, firewalls, etc.) in the Core. These devices typically operate as VNFs running in virtual instances (e.g., VMs, containers, etc.) utilizing computational resources collocated with network nodes.

Figure 1. Mobile network of heterogeneous communication and computation nodes along with middlebox functionality offered by deployed VNFs. Dashed color lines indicate 2 illustrative E2E paths selected for 2 highlighted UEs.

Each link $e=(u,v)\in \mathcal{E}$, where $u,v\in \mathcal{V}$, can be wired or wireless (the latter enables wireless X-hauling, typically mmWave), has a communication capacity $c_e$, and induces a delay ${\delta }_e$ (being the sum of transmission and propagation delays) to all packets traversing it. We partition $\mathcal{E}$ into the sets ${\mathcal{E}}_{fi}$ and ${\mathcal{E}}_{wl}$ of wired/fiber and wireless links, respectively, and denote with ${\mathcal{V}}_{sw}\subseteq \mathcal{V}$ the set of nodes in $\mathcal{V}$ that have at least one incident link in ${\mathcal{E}}_{fi}$ (i.e., $u\in {\mathcal{V}}_{sw}$ if there exists $w\in \mathcal{V}$ such that $(u,w)\in {\mathcal{E}}_{fi}$). We abstractly refer to the nodes in ${\mathcal{V}}_{sw}$ as “switches” since most fiber links in a mobile Core are typically Point-to-Point links among switches or routers. For modeling reasons, and although all physical links are inherently bidirectional, we explicitly distinguish between links $(u,v)$ and $(v,u)$ in $\mathcal{E}$ (and also for ${\mathcal{E}}_{fi},{\mathcal{E}}_{wl}$). We denote with $h\left(e\right)=u$ and $t\left(e\right)=v$ the head and tail, respectively, of directed link $(u,v)$.

The UEs are not included in $\mathcal{V}$ but rather in $\mathcal{J}$ (we also define $\tilde{\mathcal{V}}\stackrel{\Delta }{=}\mathcal{V}\cup \mathcal{J}$). Each UE $j\in \mathcal{J}$ connects to a BS $a\in$ ${\mathcal{A}}^{\left(j\right)}\subseteq$ $\mathcal{V}$, where the dependence on j captures the typical signal level-based association rules. We define $\mathcal{A}\stackrel{\Delta }{=}{\cup }_{j\in \mathcal{J}}{\mathcal{A}}^{\left(j\right)}$ as the set of all BSs and ${\mathcal{S}}^{\left(a\right)}\stackrel{\Delta }{=}\left\{j:a\in {\mathcal{A}}^{\left(j\right)}\right\}\subseteq \mathcal{J}$ as the set of UEs which may be served by BS $a\in \mathcal{A}$. We focus on Downlink (DL) and consider the set of DL AN links ${\mathcal{E}}_{AN}$ $\stackrel{\Delta }{=}$ $\left\{(a,j):a\in \mathcal{A},j\in {\mathcal{S}}^{\left(a\right)}\right\}$ with $\tilde{\mathcal{E}}$ $\stackrel{\Delta }{=}$ $\mathcal{E}\cup {\mathcal{E}}_{AN}$. To capture the underlying Physical layer constraints in the air interface, we also assume that each BS $a\in \mathcal{A}$ has a maximum number of ${\overline{N}}_a^{\left(RB\right)}$ Resource Blocks (RBs) to allocate to the UEs in ${\mathcal{S}}^{\left(a\right)}$.

Let ${\mathcal{V}}_c\subseteq \mathcal{V}$ be the set of network nodes which also have computational (i.e., CPU) resources able to host one or more VNFs. Additional computational resources such as memory and storage can be similarly handled and are omitted for simplicity and without loss of generality. For each node $y\in {\mathcal{V}}_c$, we denote with $c_y$ the amount of CPU resources, measured in Giga-Floating Point Operations per Second (GFLOPS). We denote with $\mathcal{F}$ the set of all available VNFs (viewed as complete software stacks) that can be deployed and allow for multiple instances of a given VNF in the same or different nodes depending on network traffic. Each VNF $f\in \mathcal{F}$ is described by the tuple $f\stackrel{\Delta }{=}(T_f,{\pi }_f,w_f,{\tau }_f)$, where $T_f$ is an identifier of the VNF’s functionality (e.g., NAT etc.), ${\pi }_f>0$ is the data processing capacity of the VNF (in Mbps), $w_f>0$ is the amount of CPU resources (in GFLOPS) required for the VNF’s operation, and ${\tau }_f>0$ is the data processing delay experienced by an individual data packet as it passes through the VNF. To ensure proper service endpoints, we also introduce the set ${\mathcal{F}}_{dum}\stackrel{\Delta }{=}\{f_{dum1},f_{dum2}\}$ of two “dummy” VNFs and include it into $\mathcal{F}$. The VNFs $f\in {\mathcal{F}}_{dum}\subseteq \mathcal{F}$ are characterized by ${\pi }_f=\infty$, $w_f=0$, ${\tau }_f=0$, which implies that the “dummy” VNFs work transparently, w.r.t., our model.

Let $\mathcal{C}$ be the set of SFCs, where SFC $\rho \in \mathcal{C}$ is described by the ordered sequence $\rho \stackrel{\Delta }{=}⟨f_{dum1},f_1^{\left(\rho \right)},\dots ,$ $f_{N_{\rho }}^{\left(\rho \right)},f_{dum2}⟩$, where $N_{\rho }$ is the number of non-dummy VNFs in $\rho$ and $f_i^{\left(\rho \right)}\in \mathcal{F}\setminus {\mathcal{F}}_{dum}$. The traffic of $\rho$ is properly served only if it passes through the VNFs in $\rho$ exactly matching the specified order. We also write $f⇝\rho$ to state that VNF f is contained in $\rho$ and define ${\mathcal{R}}_{\rho }\stackrel{\Delta }{=}\{f\in \mathcal{F}:f⇝\rho \}$. An equivalent description for $\rho$ is via a directed graph ${\mathcal{G}}^{\left(\rho \right)}({\mathcal{V}}^{\left(\rho \right)},{\mathcal{E}}^{(\rho })$ with the virtual node set ${\mathcal{V}}^{\left(\rho \right)}$ $\stackrel{\Delta }{=}\{f_{dum1},f_1^{\left(\rho \right)},f_2^{\left(\rho \right)},\dots ,f_{N_{\rho }}^{\left(\rho \right)},f_{dum2}\}$ (i.e., each VNF $f⇝\rho$ is a node of ${\mathcal{V}}^{\left(\rho \right)}$) and the virtual edge set ${\mathcal{E}}^{\left(\rho \right)}=\left\{\left(f_{dum1},f_1^{\left(\rho \right)}\right),{\left(f_i^{\left(\rho \right)},f_{i+1}^{\left(\rho \right)}\right)}_{1\le i\le N_{\rho }-1}\left(f_{N_{\rho }}^{\left(\rho \right)},f_{dum2}\right)\right\}$ describing the relative order of the VNFs (see bottom part of Figure 2 for an example of such a virtual graph). For any virtual edge $e^{\prime }\in {\mathcal{E}}^{\left(\rho \right)}$, we denote with $h\left(e^{\prime }\right)$, $t\left(e^{\prime }\right)\in \mathcal{F}$ the respective VNFs at the head and tail of $e^{\prime }$.

Figure 2. Embedding of virtual network (bottom part) into the physical network (top part). For illustration, 2 distinct SFCs are shown (in green and red) corresponding to 2 different UE requests, and each SFC is described by its own virtual network. ${\mathcal{G}}^{\left({\rho }_1\right)}$ (green color) contains 6 virtual nodes and 5 virtual edges, whereas ${\mathcal{G}}^{\left({\rho }_2\right)}$ contains 5 virtual nodes and 4 virtual edges. The virtual nodes will be mapped to actual physical nodes, while the virtual edges will be mapped to actual paths in the physical network.

Each UE $j\in \mathcal{J}$ requests a service type $q_j\in \mathcal{Q}$, with $q_j\stackrel{\Delta }{=}(s_{q_j},r_{q_j},{\delta }_{q_j},{\rho }_{q_j})$, where $s_{q_j}\in \mathcal{V}$ is the “source” node of the service (the “destination” node of service $q_j$ is UE j), $r_{q_j}>0$ is the E2E required throughput, ${\delta }_{q_j}>0$ is the E2E maximum allowed latency, and ${\rho }_{q_j}\in \mathcal{C}$ is the required SFC. We explicitly allow for sharing a VNF instance among two (or more) different SFCs, provided the VNF can meet the requirements imposed by the aggregate traffic of the services sharing this VNF. Furthermore, for each UE j, we use our knowledge (or estimates) of the Signal to Interference plus Noise Ratio (SINR) ${\sigma }_{a,j}$ of link $(a,j)\in {\mathcal{E}}_{AN}$ and the requested service rate $r_{q_j}$ to compute the number of RBs $N_{a,j}^{\left(RB\right)}$ needed to achieve this rate on link $(a,j)$. Assuming frequency-flat slow fading [25], we consider the simple case of uniform BS power allocation among the RBs, so that each RB is assigned a power of $p_a^{\left(RB\right)}$.

We can now succinctly state the problem to be solved as follows: for a set of service requests $\mathcal{Q}$ generated by a set of UEs $\mathcal{J}$, we seek to jointly determine the location of the VNFs that must be deployed to properly serve the requested SFCs, as well as the E2E routing path for each request, so that the total system power consumption is minimized (equivalently, the energy efficiency in bits/Joule over all requests is maximized). Computation of the selected routing path also includes determination of the BS to which each UE attaches to.

2.1. Power Consumption Model and Problem Formulation

Unless otherwise stated, we consistently use the following indices ranging over the respective sets: $j\in \mathcal{J}$, $a\in \mathcal{A}$, $q\in \mathcal{Q}$, $y\in {\mathcal{V}}_c$, $\tilde{y}\in {\mathcal{V}}_c\cup \mathcal{J}$, $f\in \mathcal{F}$, $q\in \mathcal{Q}$, $u,v,w\in \mathcal{V}$, $m,n\in {\mathcal{V}}_{sw}$. We introduce the decision variables $x_{j,a}$ as the Boolean indicator of whether UE j attaches to BS a, and ${\varphi }_{\tilde{y},f,q}$ as the Boolean indicator of whether VNF f requested by service q is deployed on node $\tilde{y}$. Furthermore, ${\theta }_e^{e^{\prime },q}$ (or its alias ${\theta }_{u,v}^{e^{\prime },q}$) is the Boolean indicator variable of whether the directed physical link $e=(u,v)\in \mathcal{E}$ belongs to the physical path in $\mathcal{G}$ onto which the virtual edge $e^{\prime }\in {\mathcal{E}}^{\left({\rho }_q\right)}$ is mapped for SFC ${\rho }_q$. Finally, $N_{f,\tilde{y}}$ is the number of instances of VNF f deployed on node $\tilde{y}$.

Towards superior energy-saving performance, we employ resources, devices, and links only when needed. Specifically, collocated CPU resources at node y are employed only when y actually runs deployed VNFs, as captured by the Boolean indicator variable ${\xi }_y$. Hence, the power consumed by CPU processing at node y is given by

$$P_y^{\left(CPU\right)}=P_y^{(CPU,i)}{\xi }_y+\left(P_y^{(CPU,m)}-P_y^{(CPU,i)}\right)·U_y,$$

(1)

where $P_y^{(CPU,m)}$, $P_y^{(CPU,i)}$ are the maximum and idle power of the CPU deployed at y and $U_y\stackrel{\Delta }{=}{\sum }_{f\in \mathcal{F}}\frac{N_{f,y}{w}_f}{c_v}$ is the CPU load factor at y [15].

Similarly, for a fiber link $(n,m)\in {\mathcal{E}}_{fi}$, the Boolean variable $z_{n,m}$ indicates whether the link actually carries traffic, in either link direction, for any request. To examine whether link $(n,m)\in {\mathcal{E}}_{fi}$ carries any traffic in the specific direction from n to m, we introduce the Boolean variable $w_{n,m}$, which implies that it must hold

$$\begin{array}{c}{z}_{m,n}\ge w_{m,n}\hfill \\ z_{m,n}\ge w_{n,m}\hfill \end{array}\hspace{1em}\forall (n,m)\in {\mathcal{E}}_{fi}.$$

(2)

We denote with ${\psi }_n$ the Boolean variable of whether the switch of node $n\in {\mathcal{V}}_{sw}$ is actually used. There exist certain consistency relations between these variables as shown in (3) below, where $C_1>0$ is a sufficiently large constant.

$$\begin{array}{c}\sum _{q\in \mathcal{Q}}\sum _{e^{\prime }\in {\mathcal{E}}^{\left({\rho }_q\right)}}{\theta }_e^{e^{\prime },q}\le C_1{w}_e,\hspace{1em}\forall e\in {\mathcal{E}}_{fi},\hfill \\ \sum _{e\in {\mathcal{E}}_{fi}:h\left(e\right)=n}{w}_e+\sum _{e\in {\mathcal{E}}_{fi}:t\left(e\right)=n}{w}_e\le C_1{\psi }_n,\hspace{1em}\forall n\in {\mathcal{V}}_{sw}.\hfill \end{array}$$

(3)

The above relations capture the fact that a node is active (i.e., ${\psi }_n=1$) only if it has active incident links carrying traffic and, similarly, a link is active only if it is carrying traffic for one of the requested SFCs.

The total power consumed by the switch in node n is

$$P_n^{\left(sw\right)}=P_{idle}^{\left(sw\right)}{\psi }_n+P_{port}\sum _{m\in {\mathcal{V}}_{sw}:(n,m)\in {\mathcal{E}}_{fi}}{z}_{n,m},$$

(4)

where $P_{idle}^{\left(sw\right)}$ denotes the switch idle power and the second term accounts for the active fiber links of the switch, with $P_{port}$ being the power consumed by each active port [15].

The power expenditure model for the mmWave links in ${\mathcal{E}}_{wl}$ follows [25] (see Equations (7)–(10) therein); specifically, the power consumed on link $e\in {\mathcal{E}}_{wl}$ is given by

$$P_e^{\left(mmW\right)}=N_{RF}^{\left(mmW\right)}\left({\chi }_e{P}_e^{(mmW,i)}+{\Delta }_e^{\left(mmW\right)}F\left({\ell }_e\right)\right),$$

(5)

where $N_{RF}^{\left(mmW\right)}$ is the number of Radio Frequency (RF) chains in the link, $P_e^{(mmW,i)}$ is the idle power of the link’s transmitter, ${\ell }_e\stackrel{\Delta }{=}{\sum }_{q\in \mathcal{Q}}{\sum }_{e^{\prime }\in {\mathcal{E}}^{\left({\rho }_q\right)}}{\theta }_e^{e^{\prime },q}/b_e$ is a load-dependent variable (where $b_e$ is the utilized bandwidth of link e), ${\Delta }_e^{\left(mmW\right)}$ is a slope parameter depending on the power electronics used in the link’s transmitter, and $F(·)$ is a piecewise-linear function accounting for the nonlinear dependence between achieved throughput and power expenditure. Finally, ${\chi }_e$ is a Boolean indicator variable for whether link e is actually used to serve any traffic; if not, the link’s transceiver is turned off.

The power expenditure for the AN links in ${\mathcal{E}}_{AN}$ is similarly modeled as follows: the power expended by a gNB BS $a\in \mathcal{A}$ is

$$P_a^{\left(gNB\right)}=N_{RF}^{\left(gNB\right)}·\left({\mu }_a{P}_a^{(gNB,i)}+{\Delta }_a^{\left(gNB\right)}\sum _{j\in {\mathcal{S}}^{\left(a\right)}}{x}_{j,a}{p}_a^{\left(RB\right)}{N}_{a,j}^{\left(RB\right)}\right),$$

(6)

where ${\mu }_a$ is the Boolean indicator for whether BS a actually serves any UEs, and $N_{RF}^{\left(gNB\right)}$, $P_a^{(gNB,i)}$, and ${\Delta }_a^{\left(gNB\right)}$ have the same semantics as in (5). For an SC BS, it similarly holds

$$P_a^{\left(SC\right)}=N_{RF}^{\left(SC\right)}·\left({\mu }_a{P}_a^{(SC,i)}+{\Delta }_a^{\left(SC\right)}\sum _{j\in {\mathcal{S}}^{\left(a\right)}}{x}_{j,a}{p}_a^{\left(RB\right)}{N}_{a,j}^{\left(RB\right)}\right).$$

(7)

We use the clever trick of [25] to convert the activation/power saving constraints into a set of linear constraints by introducing auxiliary Boolean variables ${\chi }_e$, for $e\in {\mathcal{E}}_{wl}$, and ${\nu }_a$, for $a\in \mathcal{A}$

$$\begin{array}{c}{\chi }_e+C_2{\sigma }_e\ge 1,\hspace{1em}\forall e\in {\mathcal{E}}_{wl},\hfill \\ 1-C_2{\chi }_e\le \sum _{q\in \mathcal{Q}}\sum _{e^{\prime }\in {\mathcal{E}}^{{\left(\rho \right)}_q}}{\theta }_e^{e^{\prime },q}\le C_2(1-{\sigma }_e),\hspace{1em}\forall e\in {\mathcal{E}}_{wl},\hfill \\ 1-C_1{\nu }_a\le \sum _{j\in \mathcal{J}}{x}_{j,a}\le C_2(1-{\nu }_a),\hspace{1em}{\mu }_a+C_1{\nu }_a\ge 1,\hspace{1em}\forall a\in \mathcal{A}.\hfill \end{array}$$

(8)

In addition to the basic self-consistency conditions

$$\begin{array}{c}{\varphi }_{s_{q_j},dum1,q_j}={\varphi }_{j,dum2,q_j}=1,\hspace{1em}\forall j\in \mathcal{J},\hfill \\ {\varphi }_{\tilde{y},f_{dum1,q_j}}=0,\hspace{1em}\forall j\in \mathcal{J},\forall \tilde{y}\in {\mathcal{V}}_c\setminus \left\{s_{q_j}\right\},\hfill \\ {\varphi }_{\tilde{y},f_{dum2},q_j}=0,\hspace{1em}\forall j\in \mathcal{J},\forall \tilde{y}\in {\mathcal{V}}_c\setminus \left\{j\right\},\hfill \end{array}$$

(9)

and

$$\begin{array}{c}{\varphi }_{j,f,q}=0,\hspace{1em}\forall j\in \mathcal{J},\forall q\in \mathcal{Q},\forall f\in \mathcal{F}\setminus \left\{f_{dum2}\right\},\hfill \\ {\varphi }_{\tilde{y},f,q}=0,\hspace{1em}\forall q\in \mathcal{Q},\forall f\in \mathcal{F}\setminus {\mathcal{R}}_{{\rho }_q},\forall \tilde{y}\in {\mathcal{V}}_c\cup \mathcal{J},\hfill \\ {\theta }_{a,j}^{{\dot{e}}_{q_j}^{\prime },q_j}=x_{j,a},\hspace{1em}\forall j\in \mathcal{J},\forall a\in \mathcal{A},\hfill \end{array}$$

(10)

following from the definitions, where ${\dot{e}}_{q_j}^{\prime }$ is the “last” virtual edge in ${\mathcal{E}}^{\left({\rho }_q\right)}$, i.e., $t\left({\dot{e}}_{q_j}^{\prime }\right)=1$, we also impose the constraints:

$$\begin{array}{cc}\hfill \sum _{y\in {\mathcal{V}}_c}{\varphi }_{y,f,q}& =1,\hspace{1em}\forall q\in \mathcal{Q},\forall f\in {\mathcal{R}}_q\setminus {\mathcal{F}}_{dum},\hfill \\ \hfill \sum _{a\in {\mathcal{A}}^{\left(j\right)}}{x}_{j,a}& =1,\hspace{1em}\forall j\in \mathcal{J},\hfill \\ \hfill x_{j,b}& =0,\hspace{1em}\forall b\notin {\mathcal{A}}^{\left(j\right)},\hfill \end{array}$$

(11)

which require that each requested VNF must be deployed at a node and that each UE must properly attach to exactly one of its allowable BSs (i.e., those BSs which can allocate a sufficient number of Resource Blocks to serve its requested traffic).

The constraint

$$\begin{array}{cc}\hfill \sum _{q\in \mathcal{Q}:f⇝{\rho }_q}{\varphi }_{y,f,q}{r}_q& \le N_{f,y}{\pi }_f,\hspace{1em}\forall f\in \mathcal{F},\forall y\in {\mathcal{V}}_c,\hfill \\ \hfill \sum _{f\in \mathcal{F}}{N}_{f,y}{w}_f& \le c_y,\hspace{1em}\forall y\in {\mathcal{V}}_c,\hfill \\ \hfill \sum _{f\in \mathcal{F}\setminus {\mathcal{F}}_{dum}}{N}_{f,y}& \le C_1{\xi }_y,\hspace{1em}\forall y\in {\mathcal{V}}_c,\hfill \end{array}$$

(12)

ensures that the number of deployed VNF instances on a node is sufficient to meet the data processing requirements for the incoming traffic and does not exceed the amount of available CPU resources, while ensuring that the computational node is deployed only when needed.

The link and BS capacity constraints are captured as follows:

$$\begin{array}{cc}\hfill \sum _{q\in \mathcal{Q}}\sum _{e^{\prime }\in {\mathcal{E}}^{\left({\rho }_q\right)}}{\theta }_e^{e^{\prime },q}{r}_q& \le c_e,\forall e\in \tilde{\mathcal{E}},\hfill \\ \hfill \sum _{j\in {\mathcal{S}}^{\left(a\right)}}{N}_{a,j}^{\left(RB\right)}& \le {\overline{N}}_a^{\left(RB\right)},\forall a\in \mathcal{A}\hfill \end{array}$$

(13)

which restricts the total amount of traffic flowing through any link and the total number of Resource Blocks allocated by any BS, while

$$\begin{array}{cc}\hfill \sum _{v:(u,v)\in \tilde{\mathcal{E}}}{\theta }_{u,v}^{e^{\prime },q}-& \sum _{w:(w,u)\in \tilde{\mathcal{E}}}{\theta }_{w,u}^{e^{\prime },q}={\varphi }_{u,h\left(e^{\prime }\right),q}-{\varphi }_{u,t\left(e^{\prime }\right),q},\hfill \\ \hfill \hspace{1em}& \forall q\in \mathcal{Q},\hspace{1em}\forall e^{\prime }\in {\mathcal{E}}^{\left({\rho }_q\right)},\hspace{1em}\forall u\in \tilde{\mathcal{V}},\hfill \end{array}$$

(14)

is a flow conservation and routing condition, which ensures that the packets of each requested SFC meet the corresponding VNFs in the correct order as they are routed through the selected path. The above equation essentially ensures the “proper” embedding of the virtual graph ${\mathcal{G}}^{\left({\rho }_q\right)}$, for each request $q\in \mathcal{Q}$, into the physical graph (see Figure 2). Finally, E2E delay constraint is captured by

$$\sum _{\begin{array}{c}y\in {\mathcal{V}}_c,\\ f\in {\mathcal{R}}_{q_j}\end{array}}{\varphi }_{y,f,q_j}{\tau }_f+\sum _{\begin{array}{c}e\in \mathcal{E},\\ e^{\prime }\in {\mathcal{E}}^{\left({\rho }_{q_j}\right)}\end{array}}{\theta }_e^{e^{\prime },q_j}{\delta }_e+\sum _{a\in {\mathcal{A}}^{\left(j\right)}}{x}_{j,a}{\delta }_{(a,j)}\le {\delta }_{q_j},\hspace{1em}\forall j\in \mathcal{J}.$$

(15)

Hence, the total power expenditure in the network is

$$P_{total}\stackrel{\Delta }{=}\sum _{n\in {\mathcal{S}}_{sw}}{P}_n^{\left(sw\right)}+\sum _{y\in {\mathcal{V}}_c}{P}_y^{\left(CPU\right)}+\sum _{e\in {\mathcal{E}}_{wl}}{P}_e^{\left(mmW\right)}+\sum _{a\in \mathcal{A}}{P}_a^{(gNB/SC)}$$

(16)

and we formulate our problem as the following NP-hard MILP

$$\begin{array}{cc}\hfill \hspace{1em}& \mathrm{minimize}\hspace{1em}{P}_{total},\hfill \\ \hfill \hspace{1em}& \mathrm{s}.\mathrm{t}.\hspace{1em}\left(2\right),\left(3\right),\left(8\right)–\left(15\right),\hfill \end{array}$$

(17)

where the control variables in (17) are $N_{f,v}$, ${\xi }_y$, $z_{m,n}$, $w_{m,n}$, ${\psi }_n$, ${\theta }_e^{e^{\prime },q}$, ${\chi }_e$, ${\sigma }_e$, ${\mu }_a$, ${\nu }_a$, ${\varphi }_{\tilde{y},f,q}$, $x_{j,a}$, $N_{a,j}^{\left(RB\right)}$ with index semantics as previously described. The MILP property of (17) follows from the simple observation (by visual inspection) that all of the above control variables appear as linear terms in the constraints (2), (3), (8)–(15) and as linear (in (1), (4), (6), (7)) or piecewise linear terms (in (5)) in the components of (16) comprising the objective function of (17), combined with the fact that the control variables take only integer or Boolean values. Note that a piecewise linear objective function can be readily converted into a purely linear form by introducing auxiliary variables (see Section 4.3.1 of [30]). Due to the high complexity of solving (17), we next propose a low-complexity heuristic algorithm and use (17) as a yardstick against which the heuristic’s performance is evaluated. For the reader’s convenience, we have collected all introduced notation into Table 1 at the end of the paper.

Table 1. List of Symbols and Notations.

Symbol	Interpretation	Notation for Element (In Case of Sets)	Symbol	Interpretation	Notation for Element (In Case of Sets)
Input parameters
$\mathcal{V}$	Set of network nodes (excluding mobile users)	$u,v$	${\mathcal{V}}_{sw}$	Set of nodes equipped with switch	$u,v$
$\mathcal{E}$	Set of network links (excluding access links)	$e=(u,v)$	$\mathcal{J}$	Set of UEs	j
${\mathcal{E}}_{fi}$, ${\mathcal{E}}_{wl}$	Set of fiber (resp. wireless) non-access links	$(u,v)$	${\mathcal{E}}_{AN}$	Set of access links	$(a,j)$
$c_e$	Communication capacity of link e	NA	${\delta }_e$	Delay (i.e., transmission + propagation) induced by link e	NA
${\mathcal{A}}^{\left(j\right)}$	Set of BSs that UE j can connect to	a	${\mathcal{S}}^{\left(a\right)}$	Set of UEs that can be served by BS a	j
${\mathcal{V}}_c$	Set of network nodes equipped with computational resources	y	$c_y$	Amount of computational resources available at node y	NA
$\mathcal{F}$	Set of available VNFs	$f=(T_f,{\pi }_f,w_f,{\tau }_f)$ where $T_f$: VNF id, ${\pi }_f$: VNF data processing capacity, $w_f$: amount of CPU resources required by VNF, ${\tau }_f$: delay induced by VNF processing
$\mathcal{C}$	Set of available SFCs	$\rho =⟨f_{dum1},f_1^{\left(\rho \right)},\dots ,f_{N_{\rho }}^{\left(\rho \right)},f_{dum2}⟩$
${\mathcal{R}}_{\rho }$	Set of VNFs comprising SFC $\rho$	f	${\mathcal{G}}^{\left(\rho \right)}$	Virtual directed graph describing SFC via virtual node set ${\mathcal{V}}^{\left(\rho \right)}$ and virtual edge set ${\mathcal{E}}^{\left(\rho \right)}$	NA
$\mathcal{Q}$	Set of requests by UEs	$q=(s_q,r_q,{\delta }_q,{\rho }_q)$ where s: source node of service request, $r_q$: E2E requested rate, ${\delta }_q$: E2E requested latency, ${\rho }_q$: requested SFC
$N_{a,j}^{\left(RB\right)}$	Number of RBs needed to be assigned by BS a to UE j to meet its requested rate
$P_{idle}^{\left(sw\right)}$	Switch idle power	NA	$P_e^{(mmW,i)}$	Idle power of transmitter in mmWave link e	NA
$P_a^{(gNB,i)}$	Idle power of gNB a	NA	$P_a^{(SC,i)}$	idle power of SC a	NA
$N_{RF}^{\left(mmW\right)}$	Number of RF chains used in transmitter of mmWave link	NA	$N_{RF}^{\left(gNB\right)}$, $N_{RF}^{\left(SC\right)}$	Number of RF chains used by gNB, SC	NA
${\Delta }_e^{\left(mmW\right)}$	Slope parameter of transmitter in mmWave link e	NA	${\Delta }_a^{\left(gNB\right)}$, ${\Delta }_a^{\left(SC\right)}$	Slope parameter of transmitter in gNB, SC a	NA
Decision variables
$x_{j,a}$	Boolean indicator of whether UE j actually connects to BS a		${\varphi }_{\tilde{y},f,q}$	Boolean indicator of whether VNF f requested by service q is deployed on node $\tilde{y}$
${\theta }_e^{e^{\prime },q}$, ${\theta }_{u,v}^{e^{\prime },q}$	Boolean indicator of whether physical link belongs to the physical path onto which the virtual link $e^{\prime }\in {\mathcal{E}}^{\left({\rho }_q\right)}$ is mapped for SFC $rh{o}_q$		$N_{f,\tilde{y}}$	Number of instances of VNF f deployed on node $\tilde{y}$
$z_{n,m}$	Boolean indicator of whether fiber link $(n,m)$ actually carries traffic in either of its directions		$w_e$, $w_{n,m}$	Boolean indicator of whether fiber link $e=(n,m)$ actually carries traffic from n to m
${\xi }_y$	Boolean indicator of whether the computational resources of node y are used		${\psi }_n$	Boolean indicator of whether the switch at node n is actually used to forward traffic
${\mu }_a$	Boolean indicator for whether BS a serves any UEs		${\chi }_e$	Boolean indicator of whether wireless link e is actually used to serve any traffic
${\nu }_a$	Auxiliary variable for ${\mu }_a$ [25]		${\sigma }_e$	Auxiliary variable for ${\chi }_e$ [25]

3. Proposed Energy-Efficient Vnf Placement, Traffic Routing, and User Association (Hero)

Our proposed Heuristic for Energy-efficient VNF placement, traffic Routing, and user assOciation (HERO) aims to maximize the network energy efficiency while ensuring low UE blocking probability. HERO consists of two stages, as shown in Figure 3. In the first stage, the traffic path is selected (i.e., user association and routing is performed), while in the second stage VNF placement takes place, ensuring correct VNF ordering.

Figure 3. Flowchart of the proposed energy-efficient VNF placement, traffic routing, and user association algorithm (HERO).

Initially, to ensure a high UE acceptance ratio, the UEs are sorted based on their service demands, giving priority to the UEs with the most delay-intolerant services. For UEs with the same delay requirements, priority is given to the UEs with higher rate demands. Then, for each UE, a weighted graph is constructed from the service traffic source to the UE with all feasible links and their respective power consumption acting as weights. HERO calculates the k shortest-weighted paths and starts with the first, as long as it satisfies the delay and link capacity constraints, taking into account the decisions for the already examined users. Otherwise, the next path is selected until either a path that satisfies all constraints is found or there are no other paths. In the latter case, the UE is blocked and HERO proceeds with the next as long as there is one.

After a valid path is found for the current UE, HERO proceeds to the second stage, where the UE VNFs are being placed. To that end, for each VNF, following the order of the UE SFC, a list is constructed with all the available computational nodes based on a parameter, denoted by H. This parameter is equal to the sum of the normalized node centrality (closeness), the normalized node computational capabilities ($c_y$), and the node CPU utilization. The latter is equal to (a) 1 when the studied VNF can be placed in the examined node without initiating a new VNF instance, (b) 0.1 when there is enough computational capacity to host the studied VNF in the examined node, but a new VNF instance is required, and (c) 0 otherwise. Subsequently, the node with the highest H for the selected VNF is selected, as long as it has sufficient computational resources to host it. Otherwise, the node with the next highest H is selected, until either the VNF is placed or there is no other node to examine in the selected path. In the latter case, the algorithm returns to stage 1 and the next path out of the k calculated is examined. The process is repeated for the new path until either all VNFs of the UE are placed or there is no other path to study and the UE is blocked. In the case where all UE VNFs are placed, the network conditions are updated, and the algorithm proceeds to the next UE. The aforementioned steps are repeated until all UEs are examined.

4. Interaction with Orchestration Framework

The proposed Orchestration Framework, whose high-level architecture is depicted in Figure 4, is based on the Vertical Slicer prototype in [31]. The Orchestration Framework aims at integrating and coordinating technology-specific platforms (e.g., ETSI Open Source MANO, SDN Controllers, O-RAN, etc.) to achieve the automated orchestration of E2E Network Slices, including the joint provisioning and configuration of heterogeneous resources, while considering the full chains of virtual functions associated with both mobile connectivity and application logic. The Vertical Slicer follows a service-driven approach: the high-level requirements provided by the Service Provider (e.g., number of expected UEs, Ultra-High Definition (UHD) streaming type) are dynamically mapped into performance requirements at the infrastructure layer in order to determine the characteristics of the E2E Network Slice in terms of mobile connectivity (e.g., downlink throughput) and the actual composition of the overall SFC (i.e., including network and application virtual functions) along with the dimension of its included components.

Figure 4. Orchestration Framework high-level architecture.

In terms of E2E Network Slice modelling, the Vertical Slicer implements a Network Slice Template where each SFC is split into three different Network Slice Subnets which specify, respectively, the deployment characteristics for the RAN, the core, and the Vertical service. The proposed modelling enhances the 3rd Generation Partnership Project (3GPP) Network Slice Network Resource Model [32] by integrating the Vertical services’ components. In addition, the Vertical Slicer also builds an overview of the needed connectivity services to be established over the transport network to fulfil the SFC.

In the orchestration and provisioning of the needed resources, the Orchestration Framework encompasses the resource allocation algorithm proposed in this paper. Specifically, when the Orchestration Framework receives a request for instantiating a service, the automated translation mechanism implemented at the Vertical Slicer determines the performance requirements and the SFC to be orchestrated. Then, the Vertical Slicer triggers a Placement Service, which calls the proposed algorithm, and provides as inputs both the service requirements and the SFC. The Placement Service takes decisions about network paths to be established at the data-plane and the placement of the different components in the SFC. The result of the computation is returned to the Vertical Slicer, which proceeds with the orchestration and configuration of the needed resources, in particular, by triggering and coordinating the target technology-specific platforms: NFV/MEC Orchestrators for VNFs’ LCM and SDN Controllers for the management and control of the data-plane. Once the service and the corresponding E2E Network Slice are properly instantiated and configured, the Vertical Slicer continues coordinating the overall LCM, interacting as needed with the technology-specific platforms for executing orchestration procedures.

5. Evaluation Methodology and Simulation Scenarios

In our work, the heuristics were developed in MATLAB, while IBM CPLEX [33] was used to compute the optimal solution of the MILP described in Section 2.1. The simulation scenarios considered a gNB sector area of 500 m radius overlaid with two SC clusters [25], as shown in Figure 5. Each cluster contained four possible SCs, which were randomly and uniformly placed in an 100-m-radius from the cluster centers. The minimum allowable distances are given in [25]. A subset of BSs, namely one SC (randomly selected) per cluster and the gNB, was assumed to have fiber access to the aggregation network. mmWave X-haul links could be deployed among BSs as long as their distance was lower than 200 m. The aggregation network consisted of two layers each one comprising four possible node positions, as shown in Figure 5. The aggregation layer nodes were connected among them and with the BSs via fiber so that there was no disconnected node. Hotspot UE traffic was also assumed [25].

Figure 5. Simulation scenario example topology.

We considered five different SFCs containing ordered combinations of the following VNFs: NAT (Network Address Translation), FW (Firewall), TM (Traffic Monitor), WOC (WAN Optimization Controller), IDPS (Intrusion Detection Prevention System), and VOC (Video Optimization Controller). For each SFC, the corresponding ordered VNF combination, data rate demand (uniformly distributed in the provided set), E2E delay requirement and share of the total SFC requests, are shown in Table 2. The data processing capacity as well as the GFLOPS requirement of each VNF type are given in Table 3. For a given number of UEs, we ran 10 different scenarios, with five different UE distribution snapshots each.

Table 2. SFC details [15].

Type	VNF Ordering	Throughput (Mbps)	Delay (ms)	Share (%)
Web	NAT→FW→TM→WOC→IDPS	[0.6–1]	500	20
VoIP	NAT→FW→TM→FW→NAT	[0.404–0.64]	100	20
Streaming	NAT→FW→TM→VOC→IDPS	[5–24]	100	39
Gaming	NAT→FW→VOC→WOC→IDPS	[0.24–0.5]	60	6
Ultra RT AI/ML	NAT→NAT	[15–25]	1	15

Table 3. VNF details [15].

Type	NAT	FW	TM	VOC	WOC	IDPS
Process Capacity (Mbps)	500	400	200	578	300	600
GFLOPS Requirement	110	440	55	110	110	440

We assumed 100 RBs allocated per gNB or SC (corresponding to $\mu$ = 0 in 5G numerology). The operating frequency of the gNB and SCs was 2 GHz, assuming orthogonal channels between the gNB and the SCs. However, the SCs of different clusters could interfere with each other. The mmWave X-haul links operated at 60 GHz, with 200 MHz channel bandwidth. For the AN and the mmWave links, we employed the link budget equation and related parameter values of [25]. The number of CPU cores was equal to 8, 24, and 48 for each node at the MEC, 1st, and 2nd Aggregation layers, respectively. Parameter $P_y^{(CPU,m)}$ was selected randomly from the set {55, 70} for the MEC nodes, from {150, 220} for the 1st Aggregation Layer and from {200, 278} for the 2nd Aggregation Layer nodes, while the idle power was assumed to be equal to 10% of the assigned maximum power values. Each fiber link had a capacity of 10 Gbps, while the rest of the simulation parameters are summarized in Table 4.

Table 4. Simulation Parameters [15,25,34].

Parameter	Value	Parameter	Value	Parameter	Value
$P_{idle}^{\left(sw\right)}$	315 W	$P_{port}$	7 W	Packet length	1.5 KB
$N_{RF}^{\left(gNB\right)}$	8	$N_{RF}^{\left(SC\right)}$	4	$N_{RF}^{\left(mmW\right)}$	64
${\Delta }_a^{\left(gNB\right)}$	4.7	${\Delta }_a^{\left(SC\right)}$	4	${\Delta }_e^{\left(mmW\right)}$	100
$P_a^{(gNB,i)}$	130 W	$P_a^{(SC,i)}$	6.8 W	$P_e^{(mmW,i)}$	3.9 W

The optimal solution and the proposed heuristic (HERO) were compared with the following SoA algorithms [15]:

• Holu: This algorithm first performed the VNF placement based on the node centrality (closeness) and the CPU utilization of computing nodes and then decided upon traffic routing targeting the minimization of power consumption, while satisfying the E2E service delay constraint.
• BCSP: It considered node centrality (betweeness) for the VNF placement and the shortest-path in terms of delay for routing, while meeting the E2E service delay constraint.

Given that both reference algorithms did not handle user association, we employed the default user association criterion for both, i.e., the UEs were connected to the BSs based on the highest received signal-to-interference-plus-noise ratio (SINR), and consequently, lowest number of required RBs. In addition, for a fair comparison, their UE examination order was selected to be the same as HERO.

6. Simulation Results and Discussion

In Figure 6 and Figure 7, we show the normalized w.r.t. the Optimal solution of (17) energy efficiency (in bits/Joule) and computational time (in logarithmic scale), respectively, for all algorithms and different number of UEs. As can be seen, HERO provided a very good tradeoff between energy efficiency and complexity compared to the other approaches, achieving up to 78% of the Optimal value, with up to 742 times lower complexity. All algorithms had a 100% user acceptance ratio in all cases, except for BCSP which achieved 96% for N = 20, 93% for N = 40, and 91% for N = 40. This is due to the fact that in BCSP the CPU utilization of the computing nodes was not taken into account, resulting in less efficient VNF placement, which under higher traffic load could lead to a few UEs being blocked. The inefficiency of BCSP’s VNF placement is also shown in Figure 8, where the power break down of all algorithms for different number of UEs per gNB sector area is presented. As shown, inefficient BCSP VNF placement led to a higher number of deployed computational nodes, and thus, higher power consumption.

Figure 6. Energy efficiency (bits/Joule) of all algorithms for different numbers of UEs per gNB area.

Figure 7. Execution time (s) in logarithmic scale of all algorithms for different numbers of UEs per gNB area.

Figure 8. Power (W) breakdown of all algorithms for low traffic (N = 10) and high traffic (N = 40).

Compared to Holu and BCSP, HERO achieved up to 60% and 86% higher energy efficiency, respectively, while keeping the complexity low, as shown in Figure 7. This is due to the fact that HERO additionally considered user association as part of the optimization problem leading to higher flexibility at the expense of a little higher complexity. On the other hand, in both Holu and BCSP, the serving BSs were already decided (based on the best SINR criterion) and then the optimal VNF placement and traffic routing from the UE traffic source to its serving BS were performed. This is also demonstrated in Figure 8, where the Optimal and HERO did not deploy the gNB in any case but used SCs as serving BSs (in contrast to Holu and BCSP), hence, leading to much lower power consumption. We also observed that the power consumption of the Optimal and HERO were scaling better than the SoA with increasing load (HERO still achieved 68% of the Optimal energy efficiency value when N = 40).

To further prove the improved performance of the proposed solutions (Optimal and HERO), we show in Figure 9, the percentage of active nodes and links for all algorithms for different number of UEs. As can be observed, Holu and BCSP, which did not optimize user association, kept the gNB always active, thus resulting in much higher power consumption, as shown in Figure 8. In addition, in Holu and BCSP, the number of active SCs, increased at a much higher rate with increasing load compared to the proposed approaches, which proves the scalability in terms of energy efficiency of the proposed solutions unlike the SoA. Compared to Holu, HERO activated fewer BH and FB links, and consequently fewer switches at the expense, however, of a higher number of active computational nodes. This is due to the fact that, in the considered scenario, which, however, was an accurate representation of actual networks of this type, the fiber links and switches had much higher energy impact ($P_{idle}^{\left(sw\right)}$ = 315 W) compared to the computational nodes, so that more efficient user association and routing had a higher impact on the overall performance compared to the VNF placement. As a result, Holu, which gave higher priority to VNF placement at the expense of traffic routing efficiency, while not taking user association into account at all, resulted, as already shown, in poorer overall performance. It is worth noting, however, that the proposed solutions are independent of the power consumption values, since the latter constitute the algorithms’s input, and thus, different values could lead to different results always targeting high energy efficiency. As a final remark, the performance gains of the proposed algorithms justify the motivation of our work that user association, VNF placement, and traffic routing should be jointly considered to guarantee true optimal E2E network performance.

Figure 9. Percentage of active nodes and links of all algorithms for different numbers of UEs per gNB area.

7. Conclusions

In this paper, we studied the joint VNF placement, user association, and traffic routing in B5G networks targeting energy efficiency maximization, while ensuring a high UE acceptance ratio. We modeled the aforementioned problem as a MILP with minimal assumptions, which captured all characteristics of the employed technologies, resource, and service types as well as their constraints and power consumption. To tackle the prohibitive complexity of the studied problem, we proposed HERO, an energy-efficient resource planning heuristic, which was shown to significantly outperform the SoA, while achieving up to 78% of the optimal value, with up to 742 times lower complexity. Finally, we described the Orchestration Framework that is responsible for the dynamic and automated orchestration of Vertical-driven services in tailored E2E Network Slices, and explained how such a framework is assisted by the proposed algorithms, which jointly compute the optimal allocation of both network and compute resources across the radio, core, and transport network segments.

Although the proposed formulation and heuristic algorithm can be extended to also handle service requests dynamically arriving in time (i.e., they can be used to implement an online joint computational and communications resource allocation policy), the computational complexity of the heuristic may still be too high to properly adapt to the small time scale of request arrivals. To this end, in the future, we plan to explore learning-based techniques (inspired from approximate reinforcement learning) to construct low-complexity online energy-efficient resource allocation algorithms.