VirtFogSim: A Parallel Toolbox for Dynamic Energy-Delay Performance Testing and Optimization of 5G Mobile-Fog-Cloud Virtualized Platforms

Abstract

It is expected that the pervasive deployment of multi-tier 5G-supported Mobile-Fog-Cloudtechnological computing platforms will constitute an effective means to support the real-time execution of future Internet applications by resource- and energy-limited mobile devices. Increasing interest in this emerging networking-computing technology demands the optimization and performance evaluation of several parts of the underlying infrastructures. However, field trials are challenging due to their operational costs, and in every case, the obtained results could be difficult to repeat and customize. These emerging Mobile-Fog-Cloud ecosystems still lack, indeed, customizable software tools for the performance simulation of their computing-networking building blocks. Motivated by these considerations, in this contribution, we present VirtFogSim. It is a MATLAB-supported software toolbox that allows the dynamic joint optimization and tracking of the energy and delay performance of Mobile-Fog-Cloud systems for the execution of applications described by general Directed Application Graphs (DAGs). In a nutshell, the main peculiar features of the proposed VirtFogSim toolbox are that: (i) it allows the joint dynamic energy-aware optimization of the placement of the application tasks and the allocation of the needed computing-networking resources under hard constraints on acceptable overall execution times; (ii) it allows the repeatable and customizable simulation of the resulting energy-delay performance of the overall system; (iii) it allows the dynamic tracking of the performed resource allocation under time-varying operational environments, as those typically featuring mobile applications; (iv) it is equipped with a user-friendly Graphic User Interface (GUI) that supports a number of graphic formats for data rendering; and (v) its MATLAB code is optimized for running atop multi-core parallel execution platforms. To check both the actual optimization and scalability capabilities of the VirtFogSim toolbox, a number of experimental setups featuring different use cases and operational environments are simulated, and their performances are compared.

1. Introduction

Modern mobile devices are equipped with a number of both heterogeneous Network Interface Cards (NICs) and multimedia sensors that allow them to host emerging perception-related applications, such as face/gesture detection/classification, visual text translation, fusion of sensed data, and video image processing, just to cite a few. Typically, these applications are delay-sensitive and computation-intensive, while the computing and battery capacities of current mobile devices are still limited. In order to cope with these limitations, the so-called Mobile Cloud Computing (MCC) paradigm proposes to offload computation-intensive tasks to the remote Cloud for the execution [1]. However, task offloading towards remote large-size cloud-based data centers undergoes large network latencies, mainly induced by the limited bandwidth still offered by multi-hop cellular Wide Area Networks (WANs) [1]. In principle, a more appealing approach could be, indeed, to allow mobile devices to exploit both the simultaneous utilization of the hosted NICs and the sub-millisecond network latencies featuring the forthcoming Fifth-Generation (5G) communication technology, in order to distribute task offloading suitably to both the remote Cloud and nearby small-sized virtualized data centers, generally referred to as Fog nodes [2].

The reason why the convergence of the three-pillar paradigm of Fog Computing (FC) [3], Cloud Computing (CC) [4] and 5G [5,6] is expected to improve both the energy and delay performance of task offloading of computing-intensive mobile applications relies on their complementary native features (see

Table 1. The expected Fog-Cloud-5G synergic interplay for the support of future computing-intensive and delay-critical Mobile applications.

FOG		CLOUD		5G
Distributed deployment and low-access latency	Fog nodes are expected to undergo a pervasive deployment. This reduces the access distance and, then, the resulting propagation delay	Centralized deployment and high-access latency	Cloud data centers typically serve large spatial areas and, then, incur high propagation delays	Real-time support for heterogeneous network technologies	A number of ultra-short-/short-/long- range UWB/WiFi/ 3G-4G Cellular network technologies are simultaneously supported and dynamically turned ON-OFF
Low-density resource virtualization	Clones of the served mobile devices may be allocated onto medium-/small-sized Fog servers in real time. In order to reduce the resulting bootstrapping delays, light low-density virtualization technologies are employed	High-density resource virtualization	Clones of the served devices are quasi-statically deployed by resorting to high-density virtualization of the computing resources of large servers	Dynamic provisioning, multiplexing, and isolation of the wireless bandwidth	Wireless bandwidth is dynamically provided on-demand to the requiring Mobile devices. Dynamic multiplexing is used, in order to provide inter-device isolation and per-device bandwidth guarantee.
Support for light delay-sensitive Mobile applications	Fog nodes may exploit low-latency WiFi links, in order to enable seamless light data/code offloading from Mobile devices	Support for computing-intensive delay-tolerant applications	By exploiting the their huge computing resources, Cloud data centers may support computing-intensive delay-tolerant applications offloaded by remote devices	Bandwidth pooling	The simultaneous utilization of multiple transmission technologies allows increasing the total available wireless bandwidth
Energy efficient	Resource-limited Mobile devices may save energy by exploiting nearby Fog nodes as computing clones	Energy hungry	Both network and server infrastructures hosted by Cloud data centers are resource-rich, but energy hungry	Ultra-low access latency	Sub-millisecond access delays are achieved by the synergic exploitation of multiple network technologies

for a synoptic comparison).

Overall, it is expected that the forthcoming 5G paradigm will be capable of simultaneously exploiting the benefits of centralized CC infrastructures, distributed FC infrastructures, and heterogeneous radio technologies, in order to provide seamless resource-augmentation to resource-limited mobile devices. The pivotal idea is to take full advantage of the cooperative radio resource management and distributed computing capability at nearby Fog nodes, in order to: (i) shorten the communication latencies for the execution of delay-sensitive light tasks; and (ii) resort to centralized remote cloud data centers only for the execution of delay-tolerant workload-intensive tasks.

1.1. The Multi-Tiered Networked Simulated Environment

According to these considerations, Figure 1 reports the general architecture of the virtualized 5G-enabled Mobile-Fog-Cloud technological platform that is considered in this paper.

The platform is composed of a virtualized Mobile device, a proximate virtualized Fog node, and a remote virtualized Cloud node. These nodes inter-communicate through Transport -layer TCP/IP connections that are supported, in turn, by the underlying 5G networking infrastructure. Specifically, in Figure 1, we have that:

• Mobile-Fog communication is supported by a two-way WiFi-based (possibly, mobile) single-hop TCP/IP connection. $R_U$ (resp., $R_D$) is the steady-state throughput of the Mobile-to-Fog (resp., Fog-to-Mobile) upstream (resp., downstream) TCP/IP one-way connection. Their peak values depend, in turn, on the adopted WiFi technology;
• Mobile-Cloud communication is supported by a two-way 3G/4G cellular (possibly, mobile and/or multi-hop) TCP/IP connection. $B_U$ (resp., $B_D$) is the steady-state throughput of the Mobile-to-Cloud (resp., Cloud-to-Mobile) upstream (resp., downstream) TCP/IP one-way connection;
• Cloud-Fog communication is supported by a two-way (possibly, wired and multi-hop) backhaul TCP/IP connection. $R_0$ is the corresponding steady-state throughput.

From a technological perspective, the key role of the 5G paradigm is to allow the computing nodes of Figure 1 to exploit simultaneously multiple heterogeneous radio technologies, in order to perform bandwidth pooling and/or trading network energy versus access latency (see the last column of Table 1). According to this consideration, the reference architecture of Figure 1 assumes that: (i) the mobile device is equipped with at least two Network Interface Cards (NICs), typically, a WiFi NIC and a 3G/4G Cellular NIC; (ii) the Fog node is equipped with a WiFi NIC and an Ethernet NIC; and (iii) the Cloud node is equipped with a Cellular NIC and an Ethernet NIC.

The mobile device must run a given application that is composed of multiple inter-connected tasks (i.e., subroutines or methods). Being energy limited and equipped with limited computing resources, the mobile device may decide to execute each task locally or offload it to the Fog node or the Cloud node. For this purpose, the Fog node (resp., the Cloud node) is virtualized, in order to host a Fog clone (resp., a Cloud clone) of the mobile device. These clones act as virtual processors and execute the offloaded tasks on behalf of the mobile device. At the Application layer, inter-task communications exploit the (aforementioned) Mobile-Fog, Mobile-Cloud, and Cloud-Fog 5G-supported TCP/IP connections.

In the considered reference scenario, the mobile clones at the Fog and Cloud nodes are implemented by resorting to the emerging CoNTainer (CNT)-based technology [7]. Its main peculiar feature is that it uses an Execution Engine, in order to dynamically carry out resource allocation (see Figure 2 of the VirtFogSim user guide for a description of the virtualized clone organization). As a consequence, the physical resources required by a container may be scaled up/down in real time by the corresponding Execution Engine. For this purpose, each server at the Mobile, Fog, and Cloud nodes hosts a number $nc\ge 1$ of containers. All the containers hosted by a same physical server share the pool of computing (e.g., CPU cycles) and networking (e.g., I/O bandwidth) physical resources made available by the physical CPU and NICs that equip the host server. The main task of the Execution Engine managing each container is to allocate dynamically the bandwidth and computing resources made available by the host server. In so doing, the containers at the Fog and Cloud nodes play the role of virtual clones for the associated mobile device and execute the tasks offloaded by the mobile device on behalf of it. For this purpose, each container acts as a Multi-core Virtual Processor, that comprises a number of (typically, homogeneous) Virtual Cores (VCs), whose processing frequencies are dynamically scaled up/down by the execution engine. The final goal of the Execution Engine is to allocate the pending application tasks over the set of available virtual cores according to the actually adopted task allocation strategy.

1.2. A Motivational Example

In order to gain a first intuitive insight about the potential energy-saving capability of the three-tiered networked computing platform of Figure 1, let us consider the toy example of Figure 2. It illustrates a “wise” strategy for allocating the tasks of an application DAG. The considered DAG is composed of eight tasks and nine (directed) edges. Each red-marked label $(a,b,c)$ on the nodes reports the (possibly, profiled) energies needed for the execution of the corresponding task at the Mobile, Fog, and Cloud, respectively. In the considered example, Nodes A and H are the input and output tasks of the overall DAG, respectively. Hence, by design, they are forced to be executed on the Mobile device (hence, the energies needed for their executions on the Fog and Cloud nodes are infinite; see the corresponding red-marked labels of Figure 2). Furthermore, in Figure 2, each blue-marked (resp., black-marked) label $(f,g)$ on the edges indicates the Mobile-to-Cloud and Cloud-to-Mobile (resp., Mobile-to-Fog and Fog-to-Mobile) energies needed for the inter-node transport of the data in the corresponding DAG edge. The computing energy ${\mathcal{E}}_{CMP}$ and the networking energy ${\mathcal{E}}_{NET}$ can be easily evaluated by summing the related elements in the red-marked, black marked, and blue-marked labels of Figure 2. These energies will be formally defined in Equations (5) and (6) of Section 3.2. The considered DAG being quite simple, it may be analyzed by hand. The analysis leads to the conclusion that the task allocation strategy that minimizes the total computing-plus-networking consumed energy ${\mathcal{E}}_{TOT}\triangleq {\mathcal{E}}_{CMP}+{\mathcal{E}}_{NET}$ allocates: (i) tasks $\left\{A,H\right\}$ to the Mobile device (see the green-colored tasks); (ii) tasks $\left\{C,G\right\}$ to the Fog (see the red-colored tasks); and (iii) tasks $\left\{B,D,E,F\right\}$ to the Cloud (see the blue-colored tasks). In so doing, the total energy consumed by the optimal task allocation strategy equates: ${\mathcal{E}}_{TOT}=32.0$ (Joule), with the computing energy ${\mathcal{E}}_{CMP}=20.5$ (Joule) and the networking energy ${\mathcal{E}}_{NET}=11.5$ (Joule). By contrast, the more direct (but sub-optimal) strategies that execute all tasks at the Mobile, Fog, and Cloud would require: ${\mathcal{E}}_{TOT}=57.0,35.0,$ and $43.0$ (Joule), respectively. This confirms that a “wise” task allocation strategy that exploits all the available Mobile-Fog-Cloud computing nodes may lead to energy saving, even in the presence of non-negligible inter-node network energy consumption.

1.3. Motivations, Main Contributions, and Organization of the Paper

Barring toy examples, it may be very challenging to plan online “wise” decisions about which tasks should be offloaded and towards which computing nodes, especially when the executions of the considered applications must be in real time, so that hard upper limits are present on the overall computing-plus-communication delays. Roughly speaking, the challenging issue stems from two main factors. First, since the combination of offloading decisions generally increases exponentially with the number of tasks of the considered DAG, exhaustive searches quickly become very time consuming, or even infeasible. Second, it is challenging to calculate analytically in closed-form the execution time of an application described by a DAG with a general (possibly, pseudo-random) topology, especially when the communication latencies must be also accounted for. In fact, the execution time generally depends on many factors like, for example, the parallelism of the Mobile-Cloud-Fog computing nodes, their (possibly, scalable) computing speeds, the number of virtual cores available at the Cloud and Fog nodes, as well as the available (and possibly scalable) inter-node network bandwidths, just to name a few.

Hence, to enable an energy-efficient real-time exploitation of the mobile technological platform of Figure 1, we need a flexible evaluation environment for the dynamic test of different task offloading and resource allocation strategies under programmable (e.g., settable by the user) models for the energy-delay profiles of the virtualized computing and network blocks composing the reference architecture of Figure 1. The hardware implementation of the 5G-supported Mobile-Fog-Cloud test-bed, although valuable, could be too cumbersome, and (moreover) it could not guarantee a repeatable and controllable performance environment.

Motivated by these considerations, in this paper, we present the general architecture and test the main functionalities of the VirtFogSim package. It is a new software toolbox that allows:

• simulation on parallel hardware machines;
• dynamic optimization;
• dynamic tracking;
• comparison;
• graphic rendering through an ad-hoc-designed Graphic Use Interface (GUI),

of the energy-delay performances of heuristic and meta-heuristic strategies for joint task offloading and dynamic resource allocation of application DAGs with general topologies over the three-tiered networked virtualized computing platform of Figure 1, under hard real-time constraints on the overall (i.e., computing-plus-communication) execution times. VirtFogSim allows the users to:

• test their desired application DAGs by customizing the simulation environment of Figure 1 through the setting of the 67 input parameters of the simulator package;
• track dynamically the energy-delay DAG performance against abrupt (possibly, unpredictable) changes of the simulated environment of Figure 1, like mobility-induced changes of the available up/down Cellular-WiFi bandwidths;
• optimize the obtained DAG performance against a number of metrics, like total consumed energy, network consumed energy, network bandwidth, computing frequency, and execution delays, just to name a few.

On the basis of an overview of the related work available in the open literature (see the next Section 2), we anticipate that the major peculiar features of the proposed VirtFogSim toolbox are the following ones:

• it allows the numerical evaluation of the delay-constrained minimization of the overall computing-plus-network energy consumed by the execution of the input DAG. The optimization is performed by task offloading and allocation of the per-core computing frequencies and up/down Cellular-WiFi bandwidths of the platform of Figure 1 in a joint and adaptive way;
• resource allocation is performed by explicitly accounting for the container-based virtualized nature of the reference platform of Figure 1. For this purpose, the VirtFogSim toolbox relies on a gradient-based primal-dual iterative procedure that implements a set of ad-hoc designed adaptive (e.g., time-varying) step-sizes: The goal is to speed up the convergence to the corresponding steady-states of the per-core computing frequencies and per-connection bandwidths to be dynamically allocated;
• it allows the user to test the energy-delay performances of six different task allocation strategies, namely the Genetic, Only-Mobile, Only-Fog, Only-Cloud, Only-Task Allocation, and Exhaustive-Search strategies (see Section 4), and to add new user-defined strategies;
• the code of the core engine of the simulator leverages the Parallel Toolbox of MATLAB, in order to exploit automatically the multi-core capability possibly retained by the execution environment of the simulator.

Furthermore, regarding the consideration of the formats of rendering data supported by the proposed VirtFogSim toolbox, we point out that it is equipped with a GUI that allows:

• the display of the dynamic time behavior of the performed resource allocation under the time-varying simulation environment set by the user and;
• the rendering of the data output by the simulator in tabular, bar-plot, and colored map graph formats (see Section 5).

Finally, a last contribution of the paper concerns the actual test of the numerical and scalability capabilities of the proposed VirtFogSim toolbox. In this regard, we point out that:

• the adaptive capability and the energy-delay performance of the (aforementioned) six task allocation strategies have been checked and compared under three benchmark DAGs. They refer to different use cases featuring some computing, multimedia, and scientific applications of practical interest and exhibit different symmetric/asymmetric/pseudo-random topologies of various sizes and in/out node degrees;
• the scaling capability of the simulator has been numerically profiled in terms of both simulation times and volumes of data that are exchanged among the running cores.

The rest of this contribution is organized as follows. After reviewing in Section 2 the related work on the current software tools for the simulation of Fog-Cloud computing platforms, in Section 3, we present the formal models describing the simulated technological platform of Figure 1. Section 4 is devoted to an in-depth description of the task offloading and dynamic resource allocation strategies supported by the current Version 4.0 of the VirtFogSim toolbox, as well as their implementation on multi-core parallel hardware execution platforms. Afterward, Section 5 describes the associated graphic formats for the display of the rendered data. Section 6 focuses on: (i) checking the numerical capability of VirtFogSim; and (ii) testing the scalability of the simulator in terms of both simulation time and memory consumption on multi-core parallel hardware execution platforms. Some hints for possible future developments and the availability of the VirtFogSim package are reported in the conclusive Section 7 and Section 8. The reader may refer to Appendix A for the presentation of the dual-mode user interfaces of the VirtFogSim toolbox and to the final Appendix B for the full list of the (settable) input parameters of the VirtFogSim simulator and the corresponding meaning, roles, and measuring units.

2. Related Work

The development of simulation tools for Fog computing is still in its infancy. Roughly speaking, the major part of the current contributions constitute the follow up of some (quite recent) toolkits designed for the simulation of the (somewhat more consolidated) environment of Cloud computing and, then, account for a limited number of specific aspects of the Fog computing paradigm. Under this perspective, an overview of the current open literature leads to the conclusion that the development of simulation tools for Cloud-Fog computing platforms has proceeded along three main research lines.

The first (somewhat more traditional) research line leads to the development of toolboxes for the software simulation of networked large-scale Cloud data centers [8,9,10].

In this regard, CloudSim [8] is a broad simulation toolkit that allows modeling and simulation of applications on a remote Cloud platform according to the Infrastructure-as-a-Service provisioning model. It allows the user to setup a customized modeling of the major building blocks of conventional Cloud infrastructures, like Virtual Machines (VMs), resource provisioning policies, resource consolidation policies, and VM migration policies. As a consequence, at the present time, CloudSim seems to be the most popular simulator for Cloud computing scenarios. However, although the Fog and Cloud computing paradigms share some virtualization features that, in principle, enable the re-use of some Cloud solutions even for Fog computing (see Table 1), the Fog environment mainly targets the support of delay-sensitive (possibly mobile) applications. From this point of view, the CloudSim toolkit presents three main deficiencies. First, it does not allow the customized setup of network-related parameters, like the per-link wireless access bandwidths and the round-trip-times of Transport-layer TCP/IP connections. Second, it relies on VM-based virtualization and does not allow the modeling of emerging CNT-based virtualization. Third, the implemented resource allocation policies are of the static-type, and no support for dynamic resource tracking is provided.

GreenCloud [9] is a follow up of the NS2 network simulator. Its main focus is on the modeling and simulation of the energy profiles of some main computing and network components of the Cloud ecosystem. Being an extension of the NS2 toolkit, GreenCloud allows the customized setting and simulation of the full TCP/IP protocol stacks equipping the switches of the intra-cloud network. However, it does not account for the Mobile-to-Cloud wireless access links and does not allow the characterization of the submitted application as DAGs.

The main goal of iCanCloud [10] is the simulation of Cloud-supported large ecosystems with thousands of served devices through a suitable extension of the native OMNeT++ platform. Hence, scalability being the main concern of this toolkit, it does not support dynamic resource tracking and per-device performance optimization.

A second more recent research line attempts to address the (aforementioned) deficiencies of Cloud-oriented toolkits by explicitly accounting for the specific features of real Fog platforms for the support of IoT-based sensor applications [11,12,13].

Under this perspective, SimIOT [11] equips the (previously-developed) SimIC [14] simulator by including a bottom IoT layer, in order to allow the user to model the request of Cloud resources of a settable number of sensor/actuator devices. This is obtained by including a communication broker module atop the native SimIC platform, in order to gather the sensed data and redirect them to the SimIC Cloud agents. As a consequence, SimIOT explicitly lacks support of an intermediate Fog layer.

IOTSim [12] is implemented as an extension of (aforementioned) CloudSim. It is targeted to the simulation of Fog computing environments in which sensor-acquired big data streams have to be quickly processed. For this purpose, the IOTSim platform equips the native CloudSim architecture with a storage layer and, then, provides the software primitives for the simulation and storage-induced delays. However, the core processing engine supported by the current IOTSim release is MapReduce-compliant, so that it implements batch processing models for the support of delay-tolerant big data applications.

iFogSim [13] is another toolbox whose implementation is an extension of CloudSim. It aims to simulate IoT-based Fog platforms by providing: (i) suitable software Java-based primitives for modeling the energy-delay performances of sensors, actuators, Fog, and Cloud nodes; (ii) two service models, e.g., the Sense-Process-Actuate and the Stream processing services; and (iii) two heuristic task allocation policies, namely the All-Cloud and Edge -ward task placement policies. However, in the current release of iFogSim, we have that: (i) all the supported devices’ profiling facilities and task allocation policies are inherently static; (ii) the network delays are assumed fixed; and (iii) support for the dynamic allocation and/or tracking of network bandwidths and computing frequencies is not provided.

Very recently, a third research line focused on the development of the software toolkit EdgeCloudSim [15], for the simulation and performance evaluation of general multi-tiered Cloud-Fog computing platforms. The main feature of EdgeCloudSim is the provisioning of a software environment for the setting and dynamic simulation of the profiles of WLAN/WAN networks, wireless network traffic, device mobility, Fog nodes, and Cloud nodes. For this purpose, EdgeCloudSim: (i) provides a customizable edge orchestration module for the performance evaluation of different task placement policies under different time scales; (ii) implements a customizable load generator that allows simulating the workload of the served mobile devices as streams of independent tasks with Poisson-distributed arrival times; and (iii) relies on the CloudSim support for the creation/management/migration/shut-down of VMs. Hence, like the proposed VirtFogSim, EdgeCloudSim aims at simulating the performance of multi-tiered networked Fog-Cloud platforms for the support of time-critical tasks. However, unlike VirtFogSim, EdgeCloudSim: (i) does not support the utilization of application DAGs and, then, does not allow the characterization of the inter-task dependency; (ii) being VM-oriented, does not support the characterization of container-based virtualization; (iii) does not provide software primitives for the tracking of the allocated network/computing resources; and (iv) being mainly focused on the delay-performance of the simulated platform, does not provide support for the modeling of the power/energy profiles of the Mobile, Fog, and Cloud nodes.

3. VirtFogSim: A View of the Simulated Formal Models

In this section, the basic definitions and formal assumptions about the constrained optimization problem tackled by the VirtFogSim engine are briefly explained. The reference framework is that already reported in Figure 1.

3.1. Profiling the Simulated Workflows

By definition, a task allocation vector:

$$\overrightarrow{x}\stackrel{\mathrm{def}}{=}\left[x\left(1\right)\equiv 1,x\left(2\right),\cdots ,x\left(V-1\right),x\left(V\right)\equiv 1\right],$$

(1)

is a dimensionless $\left(1\times V\right)$ ternary vector, whose components are defined as follows:

$$x\left(i\right)\stackrel{\mathrm{def}}{=}\left\{\begin{array}{c}0,\mathrm{if}\mathrm{the}{i}^{\mathrm{th}}\mathrm{task}\mathrm{is}\mathrm{executed}\mathrm{at}\mathrm{the}\mathrm{Fog}\mathrm{clone},\hfill \\ 1,\mathrm{if}\mathrm{the}{i}^{\mathrm{th}}\mathrm{task}\mathrm{is}\mathrm{executed}\mathrm{at}\mathrm{the}\mathrm{Mobile}\mathrm{Device},i=1,\cdots ,V.\hfill \\ 2,\mathrm{if}\mathrm{the}{i}^{\mathrm{th}}\mathrm{task}\mathrm{is}\mathrm{executed}\mathrm{at}\mathrm{the}\mathrm{Cloud}\mathrm{clone},\hfill \end{array}\right.$$

By design, the first and last tasks are always executed at the Mobile device, and therefore, $x\left(1\right)\equiv x\left(V\right)\equiv 1$ (see Equation (1)). Hence, the size of the set of the admissible task allocation vectors is $3^{V-2}$.

By definition, a resource allocation vector:

$$\overrightarrow{RS}\stackrel{\mathrm{def}}{=}\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]\left(\mathrm{bit}/\mathrm{s}\right),$$

(2)

is a $\left(1\times 7\right)$ non-negative row vector whose components are measured in bit/s. It reports the values assumed by (see Figure 1) in the order:

• the processing frequency $f_M$ at the Mobile device;
• the processing frequency $f_F$ at the Fog clone;
• the processing frequency $f_C$ at the Cloud clone;
• the throughput $R_U$ of the Mobile-to-Fog TCP/IP connection;
• the throughput $R_D$ of the Fog-to-Mobile TCP/IP connection;
• the throughput $B_U$ of the Mobile-to-Cloud TCP/IP connection;
• the throughput $B_D$ of the Cloud-to-Mobile TCP/IP connection.

3.2. The Considered Throughput-Constrained Joint Task and Dynamic Resource Allocation Problem

In this section, we give a glimpse of the constrained optimization problem whose solution is numerically evaluated by VirtFogSim. For this purpose, let ${\mathcal{E}}_M$, ${\mathcal{E}}_F$, and ${\mathcal{E}}_C$ (Joule) be the computing energies consumed by the Mobile device, Fog clone, and Cloud clone of Figure 1 in order to process the assigned tasks of the considered application DAG. Furthermore, let ${\mathcal{E}}_{WiFi}$, ${\mathcal{E}}_{CELL}$, and ${\mathcal{E}}_{WD}$ (Joule) be the energies consumed by the Mobile-Fog, Mobile-Cloud, and Cloud-Fog two-way TCP/IP connections, in order to transport the inter-task data among the Mobile, Cloud, and Fog computing nodes. Finally, let:

$${\mathcal{E}}_{TOT}\stackrel{\mathrm{def}}{=}{\vartheta }_M\times {\mathcal{E}}_M+{\vartheta }_F\times {\mathcal{E}}_F+{\vartheta }_C\times {\mathcal{E}}_C+{\mathcal{E}}_{WiFi}+{\mathcal{E}}_{CELL}+{\mathcal{E}}_{WD}\left(\mathrm{Joule}\right),$$

(3)

be the resulting computing-plus-communication total consumed energy, with ${\vartheta }_M$, ${\vartheta }_F$, and ${\vartheta }_C$ being binary $\left\{0,1\right\}$-valued constants.

Hence, the considered Joint Optimization Problem (JOP) is defined as follows:

$$\begin{array}{c}\underset{\overrightarrow{RS},\overrightarrow{x}}{min}{\mathcal{E}}_{TOT},\hfill \\ \mathrm{s}.\mathrm{t}.:\hfill \end{array}$$

(4a)

$$\begin{array}{c}\left(1/T_{DAG}\right)\ge T{H}_0^{MIN},\hfill \end{array}$$

(4b)

$$\begin{array}{c}\hfill 0\le f_M\le f_M^{MAX},\end{array}$$

(4c)

$$\begin{array}{c}0\le f_F\le f_F^{MAX},\hfill \end{array}$$

(4d)

$$\begin{array}{c}\hfill 0\le f_C\le f_C^{MAX},\end{array}$$

(4e)

$$\begin{array}{c}0\le R_U\le R_U^{MAX},\hfill \end{array}$$

(4f)

$$\begin{array}{c}\hfill 0\le R_D\le R_D^{MAX},\end{array}$$

(4g)

$$\begin{array}{c}0\le B_U\le B_U^{MAX},\hfill \end{array}$$

(4h)

$$\begin{array}{c}\hfill 0\le B_D\le B_D^{MAX},\end{array}$$

(4i)

$$\begin{array}{c}x\left(1\right)=x\left(V\right)=1,\hfill \end{array}$$

(4j)

$$\begin{array}{c}\hfill x\left(i\right)\in \left\{0,1,2\right\},i=2,\cdots ,\left(V-1\right).\end{array}$$

(4k)

In the above equations, we have that:

• $\overrightarrow{RS}$ is the resource allocation vector defined in Equation (2);
• $\overrightarrow{x}$ is the task allocation vector defined in Equation (1);
• $T_{DAG}$ (s) is the overall time needed to execute the assigned application DAG, in short, the DAG execution time. It generally depends on the optimization variables $\overrightarrow{RS}$ and $\overrightarrow{x}$;
• $T{H}_0^{MIN}$ (app/s) is the required minimum application throughput, i.e., the minimum number of application DAGs to be performed in the time interval of one second. It is a non-negative real number;
• Equations (4c)–(4g) account for the maximum available resources;
• Equations (4j)–(4k) account for the ternary nature of the considered task allocation vectors.

The considered JOP is solved in VirtFogSim by the RAP_p function described in Section 4.2. Before proceeding, four main remarks about the formulated JOP are in order.

First, it aims to jointly optimize the allocation of the tasks over the Mobile, Fog, and Cloud computing nodes of Figure 1 and the corresponding computing-plus-communication resources. Formally speaking, it is a mixed-integer non-convex optimization problem, that resists closed-form solution. In the sequel, we will denote as: ${\overrightarrow{RS}}^{\ast }$ and ${\overrightarrow{x}}^{\ast }$ the solution of the constrained JOP in Equations (4a)–(4k).

Second, for positive $T{H}_0^{MIN}$, the constraint in Equation (4b) enforces a hard (e.g., deterministic) QoS constraint on the minimum desired application throughput. As a consequence, too many larger values of $T{H}_0^{MIN}$ may give rise to infeasible JOP.

Third, the actual values of the binary constants: ${\vartheta }_M$, ${\vartheta }_F$, and ${\vartheta }_C$ depend on the application service model adopted by the Service Provider. Specifically, ${\vartheta }_M$ and/or ${\vartheta }_F$ and/or ${\vartheta }_C$ are unit valued (resp., vanish) when the energies consumed by the Mobile, Fog, and Cloud computing nodes are for free (resp., are taxed).

Fourth, the summation of the first three terms in the objective function in (3) represents the consumed computing energy, i.e.,

$${\mathcal{E}}_{CMP}\stackrel{\mathrm{def}}{=}{\vartheta }_M\times {\mathcal{E}}_M+{\vartheta }_F\times {\mathcal{E}}_F+{\vartheta }_C\times {\mathcal{E}}_C\left(\mathrm{Joule}\right),$$

(5)

while the summation of the remaining three terms is the corresponding wasted network energy ${\mathcal{E}}_{NET}$, formally defined as follows:

$${\mathcal{E}}_{NET}\stackrel{\mathrm{def}}{=}{\mathcal{E}}_{WiFi}+{\mathcal{E}}_{CELL}+{\mathcal{E}}_{WD}\left(\mathrm{Joule}\right).$$

(6)

3.3. Simulated Computing and Networking Energy Profiles

In this section, we (briefly) present the analytical expressions of the computing and network energies involving the objective function in (4a), together with the corresponding analytical expression of the (constrained) $T_{DAG}$ in (4b). The goal is to allow the VirtFogSim user to acquire basic insight about the roles played and the possible impacts of the input parameters in

Table A2. Input parameters of the VirtFogSim simulator and their simulated settings of Section 6.

Parameter	Meaning/Role	Measuring Units	Simulated Settings
V	Number of tasks of the application DAG	Dimensionless	$9\le V\le 45$
$\overrightarrow{s}$	V-tuple row vector of the task workloads	(bit)	Computing- and communi- cation-intensive DAG
A	$\left(V\times V\right)$ binary (i.e., $\left\{0,1\right\}$) DAG adjacency matrix	Dimensionless
$Da$	$\left(V\times V\right)$ weight matrix of the (directed) edges of the DAG	(bit)	Computing- and communi- cation-intensive DAG
$n_M$	Number of the (virtual) computing cores equipping the Mobile device	Dimensionless	$n_M=1$
$n_F$	Number of the (virtual) computing cores equipping the device clone at the Fog node	Dimensionless	$n_F=4$
$n_C$	Number of (virtual) computing cores equipping the device clone at the Cloud node	Dimensionless	$n_C=12$
$N{F}_{WiFi}$	Average per-connection number of failures of the one-way Mobile→Fog and Fog→Mobile WiFi TCP/IP connections	Dimensionless	$N{F}_{WiFi}=1.1$
$N{F}_{CELL}$	Average per-connection number of failures of the cellular one-way Mobile→Cloud and Cloud→Mobile TCP/IP connections	Dimensionless	$N{F}_{CELL}=0.1$
$N{F}_{WD}$	Average per-connection number of failures of the two-way Cloud↔Fog TCP/IP backhaul connection	Dimensionless	$N{F}_{WD}=0.01$
$f_M^{MAX}$	Per-core maximum computing frequency at the Mobile device	(bit/s)	$f_M^{MAX}=12\times {10}^6$
$f_F^{MAX}$	Per-core maximum computing frequency at the Fog clone	(bit/s)	$f_F^{MAX}=12\times {10}^6$
$f_C^{MAX}$	Per-core maximum computing frequency at the Cloud clone	(bit/s)	$f_F^{MAX}=12\times {10}^6$
$R_U^{MAX}$	Maximum bit rate of the WiFi-based one-way Mobile→Fog TCP/IP connection	(bit/s)	$R_U^{MAX}=8.0\times {10}^6$
$R_D^{MAX}$	Maximum bit rate of the WiFi-based one-way Fog→Mobile TCP/IP connection	(bit/s)	$R_D^{MAX}=9.0\times {10}^6$
$B_U^{MAX}$	Maximum bit rate of the cellular one-way Mobile→Cloud TCP/IP connection	(bit/s)	$B_U^{MAX}=6.5\times {10}^6$
$B_D^{MAX}$	Maximum bit rate of the cellular one-way Cloud→Mobile TCP/IP connection	(bit/s)	$B_D^{MAX}=7.0\times {10}^6$
$T{H}_0^{MIN}$	Scalar non-negative real parameter. It is the minimum required application throughput	(DAG/s)	$0.333\le T{H}_0^{MIN}\le 3.33$
${\theta }_M$	Binary $\left\{0,1\right\}$-parameter. It depends on the utilized application service model at the Mobile device	Dimensionless	${\theta }_M=0$ (resp., ${\theta }_M=1$) if computing-plus-networking energy consumed by the Mobile device is (resp., is not) for free
${\theta }_F$	Binary $\left\{0,1\right\}$-parameter. It depends on the utilized application service model at the Fog node	Dimensionless	${\theta }_F=0$ (resp., ${\theta }_M=1$) if the computing-plus-networking energy consumed by the Fog clone is (resp., is not) for free
${\theta }_C$	Binary $\left\{0,1\right\}$-parameter. It depends on the utilized application service model at the Cloud node	Dimensionless	${\theta }_C=0$ (resp., ${\theta }_C=1$) if computing-plus-networking energy consumed by the Cloud clone is (resp., is not) for free
$P_{CPU-M}^{\left(IDLE\right)}$	Power consumed in the idle state by the physical CPU at the Mobile device	(Watt)	$P_{CPU-M}^{\left(IDLE\right)}=1.2$
$P_{CPU-F}^{\left(IDLE\right)}$	Power consumed by a single physical server in the idle state at the Fog node	(Watt)	$P_{CPU-F}^{\left(IDLE\right)}=220$
$P_{CPU-C}^{\left(IDLE\right)}$	Power consumed by a single physical server in the idle state at the Cloud node	(Watt)	$P_{CPU-C}^{\left(IDLE\right)}=440$
$n{c}_M$	Number of containers simultaneously running on the Mobile CPU	Dimensionless	$n{c}_M=1$
$n{c}_F$	Number of containers simultaneously running on a single physical server at the Fog node	Dimensionless	$n{c}_F=16$
$n{c}_C$	Number of containers simultaneously running on a single physical server at the Cloud node	Dimensionless	$n{c}_C=24$
${\gamma }_M$	Positive exponent of the dynamic power consumption of the CPU at the Mobile device	Dimensionless	${\gamma }_M=3.2$
${\gamma }_F$	Positive exponent of the dynamic power consumption of a single physical server at the Fog node	Dimensionless	${\gamma }_F=3.1$
${\gamma }_C$	Positive exponent of the dynamic power consumption of a single physical server at the Cloud node	Dimensionless	${\gamma }_C=3.0$
$k_M$	Positive parameter. It profiles the dynamic power consumption of the CPU at the Mobile device	(Watt)/(bit/s)${\gamma }_M$	$k_M=7.5\times {10}^{-21}$
$k_F$	Positive parameter. It profiles the dynamic power consumption of a single physical server at the Fog node	(Watt)/(bit/s)${\gamma }_F$	$k_F=9.78\times {10}^{-20}$
$k_C$	Positive scalar parameter. It profiles the dynamic power consumption of a single physical server at the Cloud node	(Watt)/(bit/s)${\gamma }_C$	$k_F=1.14\times {10}^{-19}$
$r_M$	Scalar fraction of the overall computing power shared by the computing cores at the Mobile device	Dimensionless	$r_M=0$
$r_F$	Scalar fraction of the overall computing power shared by the computing cores of a single physical server at the Fog node	Dimensionless	$r_F=0.2$
$r_C$	Scalar fraction of the overall computing power shared by the computing cores of a single physical server at the Cloud node	Dimensionless	$r_C=0.1$
$P_{ETH}^{\left(IDLE\right)}$	Power consumed in the idle state by each physical Ethernet NIC at the Fog and Cloud nodes	(Watt)	$P_{ETH}^{\left(IDLE\right)}={10}^{-4}$
$P_{WiFi-M}^{\left(IDLE\right)}$	Power consumed in the idle state by each physical WiFi NIC at the Mobile device and Fog node	(Watt)	$P_{WiFi-M}^{\left(IDLE\right)}=1.3$
$P_{CELL-M}^{\left(IDLE\right)}$	Power consumed in the idle state by each physical cellular NIC at the Mobile device and Cloud node	(Watt)	$P_{CELL-M}^{\left(IDLE\right)}=0.82$
${\zeta }_{WiFi}^{\left(TX\right)}$	Positive scalar exponent of the dynamic power consumption of the WiFi NIC in the transmit mode	Dimensionless	${\zeta }_{WiFi}^{\left(TX\right)}=2.40$
${\zeta }_{WiFi}^{\left(RX\right)}$	Positive scalar exponent of the dynamic power consumption of the WiFi NIC in the receive mode	Dimensionless	${\zeta }_{WiFi}^{\left(RX\right)}=2.20$
${\zeta }_{CELL}^{\left(TX\right)}$	Positive scalar exponent of the dynamic power consumption of the cellular NIC in the transmit mode	Dimensionless	${\zeta }_{CELL}^{\left(TX\right)}=2.45$
${\zeta }_{CELL}^{\left(RX\right)}$	Positive exponent of the dynamic power consumption of the cellular NIC in the receive mode	Dimensionless	${\zeta }_{CELL}^{\left(RX\right)}=2.34$
$\eta$	Positive exponent of the Round Trip Times (RTTs) of the TCP/IP WiFi and Cellular connections	Dimensionless	$\eta =0.6$
$RT{T}_{WiFi}$	Average RTT of the TCP/IP WiFi up/down connections	(s)	$RT{T}_{WiFi}=1.0\times {10}^{-2}$
$RT{T}_{CELL}$	Average RTT of the TCP/IP cellular up/down connections	(s)	$RT{T}_{CELL}=1.0\times {10}^{-1}$
$RT{T}_{WD}$	Average RTT of the (possibly, multi-hop) two-way backhaul	(s)	$RT{T}_{WD}=1.0$
$MSS$	Maximum size of a TCP segment	(bit)	Typically $MSS=1.2\times {10}^4$
$Pro{b}_{LOSS}$	Loss probability of the Cloud↔Fog TCP/IP connection	Dimensionless	$Pro{b}_{LOSS}=1.56\times {10}^{-5}$
${\chi }_{WiFi}^{\left(TX\right)}$	Power profile of the WiFi NIC in the transmit mode	$\frac{\left(\mathrm{Watt}\right)}{{\left(\mathrm{bit}/\mathrm{s}\right)}^{{\zeta }_{WiFi}^{\left(TX\right)}}\times {\left(\mathrm{s}\right)}^{\eta }}$	${\chi }_{WiFi}^{\left(TX\right)}=5.0\times {10}^{-14}$
${\chi }_{WiFi}^{\left(RX\right)}$	Power profile of the WiFi NIC in the receive mode	$\frac{\left(\mathrm{Watt}\right)}{{\left(\mathrm{bit}/\mathrm{s}\right)}^{{\zeta }_{WiFi}^{\left(RX\right)}}\times {\left(\mathrm{s}\right)}^{\eta }}$	${\chi }_{WiFi}^{\left(RX\right)}=1.4\times {10}^{-14}$
${\chi }_{CELL}^{\left(TX\right)}$	Power profile of the cellular NIC in the transmit mode	$\frac{\left(\mathrm{Watt}\right)}{{\left(\mathrm{bit}/\mathrm{s}\right)}^{{\zeta }_{CELL}^{\left(TX\right)}}\times {\left(\mathrm{s}\right)}^{\eta }}$	${\chi }_{CELL}^{\left(TX\right)}=2.31\times {10}^{-13}$
${\chi }_{CELL}^{\left(RX\right)}$	Power profile of the cellular NIC in the receive mode	$\frac{\left(\mathrm{Watt}\right)}{{\left(\mathrm{bit}/\mathrm{s}\right)}^{{\zeta }_{CELL}^{\left(RX\right)}}\times {\left(\mathrm{s}\right)}^{\eta }}$	${\chi }_{CELL}^{\left(RX\right)}=8.1\times {10}^{-15}$
$I_{MAX}$	Maximum number of primal-dual iterations performed by the RAP_p function	Dimensionless	$500\le I_{MAX}\le 700$
$a_{MAX}$	Speed-factor of the iterations performed by the RAP_p function	Dimensionless	${10}^{-7}\le a_{MAX}\le 8\times {10}^{-7}$
$n{o}_{HOP}$	Number of hops of the two-way Cloud↔Fog backhaul connection	Dimensionless	$n{o}_{HOP}=4$
$P_{HOP}$	Per-hop average power consumed the two-way backhaul connection	(Watt)	$P_{HOP}=5.7\times {10}^{-1}$
$PS$	Population size of the simulated genetic functions. It must be even and no less than 4	Dimensionless	$120\le PS\le 400$
$CF$	Fraction of the population size that undergoes genetic crossover	Dimensionless	$CF=0.5$
$G_{MAX}$	Positive scalar integer parameter. It is the number of generations run by the genetic functions	Dimensionless	$20\le G_{MAX}\le 100$
$MN$	Number of elements of each task allocation vector that undergo genetic mutation	Dimensionless	$MN=\mathrm{round}\left(\left(V-2\right)/2\right)$
${\overrightarrow{a}}_{MAX-FT}$	3-tuple vector of positive speed-factors of the gradient-based iterations performed by FogTracker	Dimensionless	Each element of the vector ${\overrightarrow{a}}_{MAX-FT}$ is in the range: $\left[7.0\times {10}^{-8},9.0\times {10}^{-7}\right]$
$jum{p}_{WiFi}^1$	Non-negative parameter. It is the first multiply scaling applied by FogTracker to $R_U^{MAX}$ and $R_D^{MAX}$	Dimensionless	$jum{p}_{WiFi}^1=0.0$
$jum{p}_{CELL}^1$	Non-negative parameter. It is the first multiply scaling applied by FogTracker to $B_U^{MAX}$ and $B_D^{MAX}$	Dimensionless	$jum{p}_{CELL}^1=1.0$
$jum{p}_{WiFi}^2$	Non-negative parameter. It is the second multiply scaling applied by FogTracker to $R_U^{MAX}$ and $R_D^{MAX}$	Dimensionless	$jum{p}_{WiFi}^2=1.0$
$jum{p}_{CELL}^2$	Non-negative parameter. It specifies the second multiply scaling applied by FogTracker to $B_U^{MAX}$ and $B_D^{MAX}$	Dimensionless	$jum{p}_{CELL}^2=0.0$
$iteration\_number$	Number of iterations performed by FogTracker (must be in multiples of 5)	Dimensionless	$iteration\_number=5\times {10}^3$

of the simulator on the resulting solution $\left({\overrightarrow{RS}}^{\ast },{\overrightarrow{x}}^{\ast }\right)$ of the underlying JOP in Equations (4a)–(4k). In this regard, four introductory remarks are in order.

First, since the Mobile device of Figure 1 may both upload to and download from the connected Fog an Cloud clones, the previously-defined network energies: ${\mathcal{E}}_{WiFi}$ and ${\mathcal{E}}_{CELL}$ split in the summation of the corresponding up and down network energies plus the underlying idle energies, so that we can write:

$${\mathcal{E}}_{WiFi}={\mathcal{E}}_{WiFi}^{\left(IDLE\right)}+{\mathcal{E}}_{WiFi}^U+{\mathcal{E}}_{WiFi}^D\left(\mathrm{Joule}\right),$$

(7)

and:

$${\mathcal{E}}_{CELL}={\mathcal{E}}_{CELL}^{\left(IDLE\right)}+{\mathcal{E}}_{CELL}^U+{\mathcal{E}}_{CELL}^D\left(\mathrm{Joule}\right).$$

(8)

Second, in the considered framework of Figure 1, the Fog-Cloud (typically wired) backhaul is assumed sustained by a two-way TCP/IP transport connection that operates in the steady-state. Hence, according to the seminal analysis reported for example in [16], the corresponding transport throughput $R_0$ may be directly evaluated as in:

$$R_0=\frac{1.22\times MSS}{RT{T}_{WD}\times \sqrt{Pro{b}_{LOSS}}}\left(\mathrm{bit}/\mathrm{s}\right),$$

(9)

where (see Figure 1): (i) $MSS$ (bit) is the maximum size of a TCP segment; (ii) $RT{T}_{WD}$ (s) is the steady-state Round-Trip-Time of the two-way Fog-Cloud TCP/IP connection; and (iii) $Pro{b}_{LOSS}$ is the associated steady-state segment loss probability.

Third, according to a number of both analytical and experimental models (see, for example, [17,18] and the references therein), the computing and network energy present in the objective function in Equations (4a)–(4k) is the summation of static energy and dynamic energy. Specifically, we note that:

• the static energy accounts for the energy wasted by the device in the idle state (e.g., the device is turned ON, but it is not running). As a consequence, since the power consumed by the device in the idle state ${\mathcal{P}}_{IDLE}$ (Watt) does not depend on the optimization variables, the static energy depends on the optimization variables $\overrightarrow{RS}$ and $\overrightarrow{x}$ only through the corresponding DAG execution time $T_{DAG}$ (s);
• the dynamic energy accounts for the energy wasted by the device when it is in the running state. As a consequence, since the resulting dynamic power depends on the operating computing frequency or communication bit rate, the dynamic energy of each device depends on the optimization variables $\overrightarrow{RS}$ and $\overrightarrow{x}$ through both the DAG execution time $T_{DAG}$ (s) and the corresponding dynamic power ${\mathcal{P}}_{DYN}$ (Watt).

Fourth, a number of both analytical and experimental studies recently appeared in the open literature [18,19,20,21,22,23], supporting the conclusion that the dynamic power ${\mathcal{P}}_{DYN}\left(y\right)$ (Watt) consumed by a computing (resp., network) device operating at the computing frequency (resp., bit-rate) y may be accurately profiled by a power-like relationship, e.g., ${\mathcal{P}}_{DYN}=K{\left(y\right)}^{\gamma }$ (Watt), where both the coefficient K and exponent $\gamma$ depend on the specifically considered device and operating conditions. Hence, by definition, the corresponding dynamic energy ${\mathcal{P}}_{DYN}\left(y\right)$ (Joule) equates to:

$${\mathcal{E}}_{DYN}\left(y\right)\stackrel{\mathrm{def}}{=}{\mathcal{P}}_{DYN}\left(y\right)/y=K{\left(y\right)}^{\gamma -1}\left(\mathrm{Joule}\right).$$

(10)

Therefore, after indicating by:

• $\delta \left(x\right)$ Kronecker’s delta (e.g., $\delta \left(x\right)=1$ for $x=1$ and $\delta \left(x\right)=0$, otherwise); and by:
• $u_{-1}\left(x\right)$ the unit-step function (e.g., $u_{-1}\left(x\right)=1$ for $x>0$ and $u_{-1}\left(x\right)=0$, otherwise),

the above considerations lead to the following sum-power-like analytical models for the profiles of the (previously-introduced) computing energies:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{E}}_M=& \left(\frac{{\mathcal{P}}_{CPU-M}^{\left(IDLE\right)}}{n{c}_M}\right)\times T_{DAG}\times u_{-1}\left({\sum }_{i=1}^V\delta \left(x\left(i\right)-1\right)\right)\hfill \\ & +\left({\sum }_{i=1}^{V}s\left(i\right)\delta \left(x\left(i\right)-1\right)\right)K_M\left(1-r_M\right){\left(f_M\right)}^{\gamma M-1},\hfill \end{array}\end{array}$$

(11a)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{E}}_F=& \left(\frac{{\mathcal{P}}_{CPU-C}^{\left(IDLE\right)}}{n{c}_C}\right)\times T_{DAG}\times u_{-1}\left({\sum }_{i=1}^V\delta \left(x\left(i\right)-2\right)\right)\hfill \\ & +\left({\sum }_{i=1}^{V}s\left(i\right)\delta \left(x\left(i\right)-2\right)\right)K_C\left(1-r_C\right){\left(f_C\right)}^{\gamma C-1},\hfill \end{array}\end{array}$$

(11b)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{E}}_C=& \left(\frac{{\mathcal{P}}_{CPU-C}^{\left(IDLE\right)}}{n{c}_C}\right)\times T_{DAG}\times u_{-1}\left({\sum }_{i=1}^V\delta \left(x\left(i\right)-2\right)\right)\hfill \\ & +\left({\sum }_{i=1}^{V}s\left(i\right)\delta \left(x\left(i\right)-2\right)\right)K_C\left(1-r_C\right){\left(f_C\right)}^{\gamma C-1},\hfill \end{array}\end{array}$$

(11c)

where all the device-depending parameters: ${\mathcal{P}}_{CPU}^{\left(IDLE\right)}$, $nc$, K, r, and $\gamma$ involved in the above energy-profiling relationships are detailed in Table A2, together with their meaning/role and proper measuring units.

In order to introduce the energy profiles of the five up/down WiFi, up/down Cellular, and two-way backhaul connections referenced by Equations (7)–(9), let: ${\mathcal{V}}_{M\to F}$, ${\mathcal{V}}_{F\to M}$, ${\mathcal{V}}_{M\to C}$, ${\mathcal{V}}_{C\to M}$, and ${\mathcal{V}}_{F\leftrightarrow C}$ indicate the volumes (in bit) of data transported (see Figure 1): (i) from Mobile to Fog; (ii) from Fog to Mobile; (iii) from Mobile to Cloud; (iv) from Cloud to Mobile; and (v) exchanged between Fog and Cloud, respectively. Hence, directly from the basic definitions of the adjacency (binary-valued) $A\triangleq \left[a_{ij}\right]$ and (real-valued) $D_a\triangleq \left[d_{ij}\right]$ connection matrices, featuring the assigned DAG, as well as the task allocation vector $\overrightarrow{x}$ in Equation (1), these volumes may be formally expressed as:

$$\begin{array}{c}{\mathcal{V}}_{M\to F}=\left(1+{\overline{NF}}_{WiFi}\right)\left({\sum }_{i,j=1}^V{a}_{ij}\delta \left(x\left(i\right)-1\right)\delta \left(x\left(j\right)\right)d_{ij}\right),\hfill \end{array}$$

(12a)

$$\begin{array}{c}{\mathcal{V}}_{F\to M}=\left(1+{\overline{NF}}_{WiFi}\right)\left({\sum }_{i,j=1}^V{a}_{ij}\delta \left(x\left(i\right)\right)\delta \left(x\left(j\right)-1\right)d_{ij}\right),\hfill \end{array}$$

(12b)

$$\begin{array}{c}{\mathcal{V}}_{M\to C}=\left(1+{\overline{NF}}_{CELL}\right)\left({\sum }_{i,j=1}^V{a}_{ij}\delta \left(x\left(i\right)-1\right)\delta \left(x\left(j\right)-2\right)d_{ij}\right),\hfill \end{array}$$

(12c)

$$\begin{array}{c}{\mathcal{V}}_{C\to M}=\left(1+{\overline{NF}}_{CELL}\right)\left({\sum }_{i,j=1}^V{a}_{ij}\delta \left(x\left(i\right)-2\right)\delta \left(x\left(j\right)-1\right)d_{ij}\right),\hfill \end{array}$$

(12d)

$$\begin{array}{c}{\mathcal{V}}_{F\leftrightarrow C}=\left(1+{\overline{NF}}_{WD}\right)\left({\sum }_{i,j=1}^V{a}_{ij}\left(\delta \left(x\left(i\right)-2\right)\delta \left(x\left(j\right)\right)+\delta \left(x\left(i\right)\right)\delta \left(x\left(j\right)-2\right)\right)d_{ij}\right),\hfill \end{array}$$

(12e)

where $\overline{NF}$ is the average number of failures of the considered connection [22,23] (see also Table A2) and V is the number of tasks of the considered DAG. Hence, the idle, up, and down energies of the WiFi and Cellular network connections in Equations (7) and (8) may be profiled as follows:

$$\begin{array}{c}{\mathcal{E}}_{WiFi}^{\left(IDLE\right)}=\left({\vartheta }_M+{\vartheta }_F\right)\times {\mathcal{P}}_{WiFi-M}^{\left(IDLE\right)}\times T_{DAG}\times u_{-1}\left({\sum }_{i=1}^V\delta \left(x\left(i\right)\right)\right),\hfill \end{array}$$

(13a)

$$\begin{array}{c}{\mathcal{E}}_{WiFi}^U={\mathcal{V}}_{M\to F}\times \left({\vartheta }_M{\mathsf{\Omega }}_{WiFi}^{\left(TX\right)}{\left(R_U\right)}^{{\xi }_{WiFi}^{\left(TX\right)}-1}+{\vartheta }_F{\mathsf{\Omega }}_{WiFi}^{\left(RX\right)}{\left(R_U\right)}^{{\xi }_{WiFi}^{\left(RX\right)}-1}\right),\hfill \end{array}$$

(13b)

$$\begin{array}{c}{\mathcal{E}}_{WiFi}^D={\mathcal{V}}_{F\to M}\times \left({\vartheta }_M{\mathsf{\Omega }}_{WiFi}^{\left(RX\right)}{\left(R_D\right)}^{{\xi }_{WiFi}^{\left(RX\right)}-1}+{\vartheta }_F{\mathsf{\Omega }}_{WiFi}^{\left(TX\right)}{\left(R_D\right)}^{{\xi }_{WiFi}^{\left(TX\right)}-1}\right),\hfill \end{array}$$

(13c)

and:

$$\begin{array}{c}{\mathcal{E}}_{CELL}^{\left(IDLE\right)}=\left({\vartheta }_M+{\vartheta }_C\right)\times {\mathcal{P}}_{CELL-M}^{\left(IDLE\right)}\times T_{DAG}\times u_{-1}\left({\sum }_{i=1}^V\delta \left(x\left(i\right)-2\right)\right),\hfill \end{array}$$

(14a)

$$\begin{array}{c}{\mathcal{E}}_{CELL}^U={\mathcal{V}}_{M\to C}\times \left({\vartheta }_M{\mathsf{\Omega }}_{CELL}^{\left(TX\right)}{\left(B_U\right)}^{{\xi }_{CELL}^{\left(TX\right)}-1}+{\vartheta }_C{\mathsf{\Omega }}_{CELL}^{\left(RX\right)}{\left(B_U\right)}^{{\xi }_{CELL}^{\left(RX\right)}-1}\right),\hfill \end{array}$$

(14b)

$$\begin{array}{c}{\mathcal{E}}_{CELL}^D={\mathcal{V}}_{C\to M}\times \left({\vartheta }_M{\mathsf{\Omega }}_{CELLX}^{\left(RX\right)}{\left(B_D\right)}^{{\xi }_{CELL}^{\left(RX\right)}-1}+{\vartheta }_C{\mathsf{\Omega }}_{CELL}^{\left(TX\right)}{\left(B_D\right)}^{{\xi }_{CELL}^{\left(TX\right)}-1}\right).\hfill \end{array}$$

(14c)

Finally, the energy consumed by the backhaul connection reads as:

$$\begin{array}{c}\begin{array}{cc}\hfill {\mathcal{E}}_{WD}=& \left({\vartheta }_C{\mathcal{P}}_{ETH-C}^{\left(IDLE\right)}+{\vartheta }_F{\mathcal{P}}_{ETH-F}^{\left(IDLE\right)}\right)\times T_{DAG}\times \hfill \\ & u_{-1}\left({\sum }_{i,j=1}^V{a}_{ij}\left(\delta \left(x\left(i\right)\right)\delta \left(x\left(j\right)-2\right)+\delta \left(x\left(i\right)-2\right)\delta \left(x\left(j\right)\right)\right)\right)+{\mathcal{V}}_{F\leftrightarrow C}\times \left(\frac{{\mathcal{P}}_{WD}}{R_0}\right),\hfill \end{array}\end{array}$$

(15)

where the corresponding wasted dynamic power ${\mathcal{P}}_{WD}$ (Watt) is given by the product:

$${\mathcal{P}}_{WD}=n{o}_{HOP}\times {\mathcal{P}}_{HOP},$$

(16)

of the number of hops $n{o}_{HOP}$ of the connection by the per-hop consumed power ${\mathcal{P}}_{HOP}$ (Watt) (see Table A2).

3.4. Simulated Profiles of the per-DAG and per-Task Execution Times

The analytical expression assumed by $T_{DAG}$ in Equation (4b) depends on the vectors $\overrightarrow{RS}$, $\overrightarrow{x}$, as well as the adopted inter-node Task Scheduling and intra-node Task Service disciplines. The current implementation of VirtFogSim assumes that the adopted Task Scheduling discipline over the computing nodes and the adopted Task Service discipline at each computing node are both of the sequential type. They are, indeed, the Task Scheduling and Task Service disciplines currently adopted by a number of middleware software tools for the support of mobile computing, such as MAUI, Volare, Cuckoo, and CloneCloud (see, for example, [24,25] and the references therein for an overview of state-of-the-art middleware solutions for the support of computing-intensive smartphone-based applications).

As direct consequence, the following sum expression holds for the analytical profile of $T_{DAG}$:

$$T_{DAG}={\sum }_{i=1}^V\left(\delta \left(x\left(i\right)-1\right)T_M\left(i\right)+\delta \left(x\left(i\right)\right)T_F\left(i\right)+\delta \left(x\left(i\right)-2\right)T_C\left(i\right)\right),$$

(17)

where $T_M\left(i\right)$, $T_F\left(i\right)$, and $T_C\left(i\right)$ are the total execution times of the $i^{\mathrm{th}}$ task of the considered DAG when it is executed at the Mobile, Fog, and Cloud node, respectively.

In this regard, two main remarks are in order. First, each total execution time $T_N\left(i\right),N\in \{M,F,C\}$, in Equation (17), is, by design, the summation of two terms. The first term is the computing time (e.g., the service time): $T_N^{SER}\left(i\right),N\in \{M,F,C\}$, wasted for the processing of the $i^{\mathrm{th}}$ task at the computing node N, while the second term is the communication delay: $T_N^{NET}\left(i\right),N\in \{M,F,C\}$, induced by the transport from the other two computing nodes of the input data needed for the processing of the $i^{\mathrm{th}}$ task at the computing node N. Second, when a set of tasks is processed by the same computing node, the resulting overall communication delay is, by design, zero [23,26,27].

Therefore, according to these two remarks, the following analytical expressions hold for the profiles of the per-node execution times in Equation (17):

$$\begin{array}{c}\begin{array}{cc}\hfill T_M\left(i\right)& =T_M^{SER}\left(i\right)+T_M^{NET}\left(i\right)=\left(\frac{{\sum }_{j=1}^{V}s\left(j\right)\delta \left(x\left(j\right)-1\right)}{n_M{f}_M}\right)\hfill \\ & +\left[\left(1+{\overline{NF}}_{WiFi}\right)\left(\frac{{\sum }_{j=1}^V{a}_{ji}{d}_{ji}\delta \left(x\left(j\right)\right)}{R_D}\right)+\left(1+{\overline{NF}}_{CELL}\right)\left(\frac{{\sum }_{j=1}^V{a}_{ji}{d}_{ji}\delta \left(x\left(j\right)-2\right)}{B_D}\right)\right],\hfill \end{array}\end{array}$$

(18a)

$$\begin{array}{c}\begin{array}{cc}\hfill T_F\left(i\right)& =T_F^{SER}\left(i\right)+T_F^{NET}\left(i\right)=\left(\frac{{\sum }_{j=1}^{V}s\left(j\right)\delta \left(x\left(j\right)\right)}{n_F{f}_F}\right)\hfill \\ & +\left[\left(1+{\overline{NF}}_{WiFi}\right)\left(\frac{{\sum }_{j=1}^V{a}_{ji}{d}_{ji}\delta \left(x\left(j\right)-1\right)}{R_U}\right)+\left(\frac{{\sum }_{j=1}^V{a}_{ji}{d}_{ji}\delta \left(x\left(j\right)-2\right)}{R_0}\right)\right],\hfill \end{array}\end{array}$$

(18b)

$$\begin{array}{c}\begin{array}{cc}\hfill T_C\left(i\right)& =T_C^{SER}\left(i\right)+T_C^{NET}\left(i\right)=\left(\frac{{\sum }_{j=1}^{V}s\left(j\right)\delta \left(x\left(j\right)-2\right)}{n_C{f}_C}\right)\hfill \\ & +\left[\left(1+{\overline{NF}}_{CELL}\right)\left(\frac{{\sum }_{j=1}^V{a}_{ji}{d}_{ji}\delta \left(x\left(j\right)-1\right)}{B_U}\right)+\left(\frac{{\sum }_{j=1}^V{a}_{ji}{d}_{ji}\delta \left(x\left(j\right)\right)}{R_0}\right)\right],\hfill \end{array}\end{array}$$

(18c)

where $i=1,\cdots ,V$ and $n_M$, $n_F$, and $n_C$ are the number of cores equipping the Mobile, Fog, and Cloud nodes, respectively (see Table A2).

3.5. Simulated Adaptive Resource Allocation Framework

In this subsection, we briefly present the general analytical framework implemented by the RAP_p and FogTracker functions of Section 4.2 and Section 4.4, in order to perform the adaptive updating of the computing frequencies: $f_M$, $f_F$, $f_C$, the wireless network bandwidths: $R_U$, $R_D$, $B_U$, $B_D$, and the Lagrange multiplier $\lambda$ associated with the throughput constraint in Equation (4b). The final goal is to allow the user to acquire insights about the role played by the input step-size $a_{MAX}$ and related input step-size vector ${\overrightarrow{a}}_{MAX-FT}$ on the adaptive capability of the resource allocation engine implemented by VirtFogSim.

For this purpose, after indicating by $\mathcal{L}$ the Lagrangian function of the JOP of Equation (4), let: $y\in \{f_M,f_F,f_C,R_U,R_D,B_U,B_D,\lambda \}$ indicate a (scalar and non-negative) variable to be optimized, and let $y_{MAX}$ be its allowed maximum value. Furthermore, after denoting by t an integer-valued iteration index, let ${\nabla }_y\mathcal{L}\left(t\right)$ be the derivative of the Lagrangian function $\mathcal{L}$ with respect to the considered optimization variable y at iteration t. Hence, according to the so-called primal-dual iteration-based approach recently customized in [28] for broadband networked application scenarios, the adaptive updating of $y\left(t+1\right)$ reads as follows:

$$y\left(t+1\right)=max\{0,min\{y_{MAX},y\left(t\right)-g_y\left(t\right){\nabla }_y\mathcal{L}\left(t\right)\}\},$$

(19)

where the outer $max\{.\}$ (resp., the inner $min\{.\}$) accounts for the non-negative (resp., maximum) value of y.

The peculiar feature of the updating relationship in (19) is that it resorts to a y-depending and time-varying gain function: $g_y\left(t\right)$, in order to guarantee quick adaptation of the optimization variable y in response to abrupt changes of the wireless environment of Figure 1. On the basis of the general results of [28] about the convergence to the steady-state of the iterations in (19), we planned to implement in the current version of VirtFogSim the following time-varying gain formula:

$$g_y\left(t+1\right)=max\{a_{MAX},min\{a_{MAX}\times y_{MAX},\frac{1}{2}{\left(y\left(t\right)\right)}^2\}\}.$$

(20)

An examination of the above formula unveils the role played by the (positive) step-size $a_{MAX}$, as well as its potential impact on the adaptive capability of the overall simulator. Specifically, on the basis of (20), we expect that large (resp., small) values of $a_{MAX}$ lead to quick (resp., slow) response to abrupt environmental changes, together with large (resp., small) possible oscillations in the steady-state. In this regard, we anticipate that the final goal of the FogTracker function of Section 4.4 is to guide the user towards the right trade-off among these two $a_{MAX}$-depending contrasting behaviors.

4. VirtFogSim: Supported Task Allocation Strategies and Their Parallel Execution

The engine of the VirtFogSim simulator is built of a main core of eight software routines that implement a number of strategies (e.g., optimization policies), in order to check the feasibility and, then, solve numerically the constrained optimization problem in (4). MATLAB is the native environment under which the optimization routines are developed and run.

In this regard, it must be remarked that the VirtFogSim package is capable of automatically turning ON and exploiting the multi-core capability retained by the supporting hardware platform, in order to run parallel programming and, then, speed up the execution of the implemented task allocation policies. This is done in a fully-transparent way, e.g., without any direct user involvement. For this purpose, some instructions for parallel programming on multi-core hardware platforms specifically supported by the Parallel Toolbox of MATLAB are utilized by the VirtFogSim package. However, VirtFogSim is capable of automatically switching to the sequential execution mode when the underlying hardware platform is single-core or the MATLAB Parallel Toolbox is not installed.

4.1. General Architecture of the Developed Simulation Platform

The main functionalities of VirtFogSim are implemented by the functions listed in

Table 2. Main functions implemented by VirtFogSim.

Function	Description
$\left[\tilde{\overrightarrow{y}},{\tilde{\mathcal{E}}}_{TOT},{\tilde{\mathcal{E}}}_{NET}\right]=\mathrm{RAP}\_\mathrm{p}\left(\overrightarrow{x},\overrightarrow{z},\overrightarrow{s},Da,S_{RAP}\right)$	It implements the Resource Allocation Problem described in Equation (4).
$\left[{\mathcal{E}}_{aux},{\mathcal{E}}_{aux}^{NET},{\lambda }_{aux}\right]=\mathrm{FogTracker}\left(x_1,x_2,x_3\right)$	It tests the convergence rate to the steady-state and the steady-state stability of the primal-dual iterations performed by the RAP_p function when abrupt changes happen.
$\left[{\overrightarrow{x}}_{best},{\overrightarrow{RS}}_{best},{\mathcal{E}}_{best},{\mathcal{E}}_{best}^{NET},R{B}_{best}\right]=\mathrm{GeneticTA}\_\mathrm{par}$	It runs the parallel Genetic Task Allocation strategy.
$\left[{\overrightarrow{RS}}_{OM},{\mathcal{E}}_{OM},{\mathcal{E}}_{OM}^{NET}\right]=\mathrm{OM}\_\mathrm{S}$	It runs the Only Mobile strategy.
$\left[{\overrightarrow{RS}}_{OF},{\mathcal{E}}_{OF},{\mathcal{E}}_{OF}^{NET}\right]=\mathrm{OF}\_\mathrm{S}$	It runs the Only Fog strategy.
$\left[{\overrightarrow{RS}}_{OC},{\mathcal{E}}_{OC},{\mathcal{E}}_{OC}^{NET}\right]=\mathrm{OC}\_\mathrm{S}$	It runs the Only Cloud strategy.
$\left[{\overrightarrow{x}}_{OTAS},{\mathcal{E}}_{OTAS},{\mathcal{E}}_{OTAS}^{NET}\right]=\mathrm{O}\_\mathrm{TAS}\_\mathrm{par}$	It runs the Only Task Allocation strategy.
$\left[{\overrightarrow{x}}_{ESS},{\overrightarrow{RS}}_{ESS},{\mathcal{E}}_{ESS},{\mathcal{E}}_{ESS}^{NET},R{B}_{ESS}\right]=\mathrm{ES}\_\mathrm{S}$	It runs the Exhaustive Search strategy.

. In the sequel of the paper, these functions are briefly described. A deep explanation about the usage of such functions can be found in the VirtFogSim User Guide, which can be downloaded along with the software package (see Section 8). The main functions in Table 2 use some auxiliary functionalities implemented in the set of routines listed in

Table 3. Auxiliary functions implemented by VirtFogSim.

Function	Description
$\left[Chil{d}_v^1,Chil{d}_v^2\right]=\mathrm{Crossover}\left(Paren{t}_v^1,Paren{t}_v^2\right)$	It implements the Crossover operation.
${\overrightarrow{out}}_v=\mathrm{Mutation}\left({\overrightarrow{x}}_v\right)$	It implements the Mutation operation.
$\left[{\mathcal{E}}_{aux},{\mathcal{E}}_{NET}\right]=\mathrm{evaluatestaticenergy}\_\mathrm{p}\left({\overrightarrow{x}}_v,\overrightarrow{s},A,Da,S_{RAP}\right)$	It evaluates the static energy evaluation in O_TAS.
$TA=\mathrm{find}\_\mathrm{allocations}\left(N,K\right)$	It returns all the $K^N$ patterns of N numbers that can assume the first K integer values: $0,1,2,\dots ,K-1$.

.

Regarding the description of the general software architecture of the simulator, VirtFogSim acts as the main program that:

• allows the user to setup 67 input parameters that characterize the scenario to be simulated by the user (see Figure 1);
• calls the GeneticTA_par function for parallel execution and returns the corresponding:

  (a)

  ${\overrightarrow{x}}_{best}\in {\left\{0,1,2\right\}}^V$, i.e., ternary V-tuple best allocation vector;

  (b)

  ${\overrightarrow{RS}}_{best}\stackrel{\mathrm{def}}{=}\left[f_M^{\ast },f_F^{\ast },f_C^{\ast },R_U^{\ast },R_D^{\ast },B_U^{\ast },B_D^{\ast }\right]$ (bit/s), i.e. the seven-tuple vector of the optimal resource allocation of the Mobile computing frequency, Fog computing frequency, Cloud computing frequency, Mobile-to-Fog transport throughput, Fog-to-Mobile transport throughput, Mobile-to-Cloud transport throughput, and Cloud-to-Mobile transport throughput (see Figure 1);

  (c)

  ${\mathcal{E}}_{best}$ (Joule), i.e., the total computing-plus-network energy consumed by the infrastructure of Figure 1 under the returned ${\overrightarrow{x}}_{best}$ task allocation vector. For this purpose, the GeneticTA function calls, in turn, the auxiliary functions RAP, Crossover, and Mutation;

• optionally, calls the OM_S function and returns:

  (a)

  ${\overrightarrow{RS}}_{OM}\stackrel{\mathrm{def}}{=}\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s), i.e., the seven-tuple vector of the optimal resource allocation under the $\left(1\times V\right)$ All-Mobile task allocation vector: ${\overrightarrow{x}}_{OM}\stackrel{\mathrm{def}}{=}\left[1,1,\cdots ,1,1\right]$;

  (b)

  ${\mathcal{E}}_{OM}$ (Joule), i.e., the total computing-plus-network energy consumed by the infrastructure of Figure 1 under ${\overrightarrow{x}}_{OM}$. If the returned ${\mathcal{E}}_{OM}$ is infinite, then the All-Mobile task allocation ${\overrightarrow{x}}_{OM}$ is infeasible. The OM_S function calls, in turn, the RAP function;

• optionally, calls the OF_S function and returns:

  (a)

  ${\overrightarrow{RS}}_{OF}\stackrel{\mathrm{def}}{=}\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s), i.e., the seven-tuple vector of the optimal resource allocation under the $\left(1\times V\right)$ All-Fog task allocation vector: ${\overrightarrow{x}}_{OF}\stackrel{\mathrm{def}}{=}\left[1,0,\cdots ,0,1\right]$;

  (b)

  ${\mathcal{E}}_{OF}$ (Joule), i.e., the total computing-plus-network energy consumed by the infrastructure of Figure 1 under ${\overrightarrow{x}}_{OF}$. If the returned ${\mathcal{E}}_{OF}$ is infinite, then the All-Fog task allocation ${\overrightarrow{x}}_{OF}$ is infeasible. The OF_S function calls, in turn, the RAP function;

• optionally, calls the OC_S function and returns:

  (a)

  ${\overrightarrow{RS}}_{OC}\stackrel{\mathrm{def}}{=}\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s), i.e., the seven-tuple vector of the optimal resource allocation under the $\left(1\times V\right)$ All-Cloud task allocation vector: ${\overrightarrow{x}}_{OC}\stackrel{\mathrm{def}}{=}\left[1,2,\cdots ,2,1\right]$;

  (b)

  ${\mathcal{E}}_{OC}$ (Joule), i.e., the total computing-plus-network energy consumed by the infrastructure of Figure 1 under ${\overrightarrow{x}}_{OC}$. If the returned ${\mathcal{E}}_{OC}$ is infinite, then, the All-Cloud task allocation ${\overrightarrow{x}}_{OC}$ is infeasible. The OC_S function calls, in turn, the RAP function;

• optionally, calls the O_TAS_par function for parallel execution and returns:

  (a)

  ${\overrightarrow{x}}_{OTAS}\in {\left\{0,1,2\right\}}^V$, i.e., the ternary V-tuple best allocation vector computed by O_TAS under the assumption that the optimization of the resource allocation is not performed;

  (a)

  ${\mathcal{E}}_{OTAS}$ (Joule), i.e., the total computing-plus-network energy consumed by the infrastructure of Figure 1 under ${\overrightarrow{x}}_{OTAS}$. If the returned ${\mathcal{E}}_{OTAS}$ is infinite, then the O_TAS task allocation ${\overrightarrow{x}}_{OTAS}$ is infeasible. The O_TAS function calls, in turn, the Crossover, Mutation, and evaluatestaticenergy_p functions;

• optionally, calls the ES_S_par function and returns:

  (a)

  ${\overrightarrow{x}}_{ESS}\in {\left\{0,1,2\right\}}^V$, i.e., the ternary V-tuple best task allocation vector computed by performing the exhaustive search over the full population of task allocation vectors of the size $3^{V-2}$;

  (b)

  ${\overrightarrow{RS}}_{ESS}\stackrel{\mathrm{def}}{=}\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s), i.e., the seven-tuple vector of the optimal resource allocation under ${\overrightarrow{x}}_{ESS}$;

  (c)

  ${\mathcal{E}}_{ESS}$ (Joule), i.e., the total computing-plus-network energy consumed by the infrastructure of Figure 1 under ${\overrightarrow{x}}_{ESS}$. If the returned ${\mathcal{E}}_{ESS}$ is infinite, then the overall afforded optimization problem is infeasible. The ES_S function calls, in turn, the RAP function and requires that the task allocation vectors ${\overrightarrow{x}}_{best}$ and ${\overrightarrow{x}}_{OTAS}$ are already available;

• optionally, calls the FogTracker function. It returns the time plots over the interval: $\left[1,iteration\_number\right]$ of the:

  (a)

  total energy ${\mathcal{E}}_{TOT}$ consumed by the overall Mobile-Fog-Cloud ecosystem;

  (b)

  the corresponding energy ${\mathcal{E}}_{NET}$ consumed by the up/down WiFi, Cellular, and Backhaul connections of Figure 1; and,

  (c)

  the behavior of the lambda multiplier associated with the hard throughput constraint of Equation (4b),

  when abrupt changes in the pattern of the allocated tasks and/or maximum bandwidths of the WiFi/Cellular connections occur. The user may set the magnitude of these changes, in order to test various (possibly mobility induced) time-fluctuations of the simulated environment of Figure 1 (see Section 4.4 in the sequel for a full description of the FogTracker function and supported options).

The current version of the VirtFogSim simulator is equipped with a (rich) Graphical User Interface (GUI) that displays:

• the numerical values of the up/down WiFi-Cellular-Backhaul bandwidths and Mobile-Fog-Cloud computing frequencies returned by the allocation policies GeneticTA_par, O_TAS_par, ES_S_par, OM_S, OF_S, and OC_S in the form of colored bar plots;
• the (generally different) task allocation patterns performed by GeneticTA_par, O_TAS_par, ES_S_par, OM_S, OF_S, and OC_S in the form of suitably-colored labeled graphs of the underlying application DAG (see Appendix A.2 in the sequel for a description of the main screen-shoots returned by the GUI of VirtFogSim).

Furthermore, VirtFogSim allows the user to:

• set 67 input parameters, in order to customize the desired computing and communication setup of the infrastructure of Figure 1;
• select any subset of the (aforementioned) GeneticTA_p, OM_S, OC_S, OF_S, O_TAS_ par, ES_S_par, and FogTracker functions, in order to test and compare various task and/or resource allocation strategies under the scenario dictated by the desired input parameters.

4.2. Supported Task Allocation Strategies and Adaptive Resource Allocation

In this subsection, we describe the supported task allocation strategies provided by the VirtFogSim and listed in Table 2.

We begin with the parallel Only-Task Allocation Strategy (O_TAS_par) function that acts as follows:

• it performs task allocation by implementing (in a parallel way) a genetic algorithm. For this purpose, O_TAS_par calls the Crossover and Mutation functions over a randomly-generated population of $\left\{0,1,2\right\}$-ternary task allocation vectors of size $PS$;
• it evaluates the energy of each tested ternary allocation vector by computing the corresponding energies: ${\mathcal{E}}_M$, ${\mathcal{E}}_F$, ${\mathcal{E}}_C$, ${\mathcal{E}}_{WiFi}$, ${\mathcal{E}}_{WD}$, and ${\mathcal{E}}_{CELL}$ under the static (e.g., not optimized) maximal resource allocation vector: ${\overrightarrow{RS}}^{MAX}=\left[f_M^{MAX},f_F^{MAX},f_C^{MAX},R_U^{MAX},R_D^{MAX},B_U^{MAX},B_D^{MAX},R_0\right]$ (bit/s). For this purpose, O_TAS_par not longer calls the RAP_p function, but calls the evaluatestaticenergy_ p function.

The O_TAS_ par outputs are:

• the V-tuple $(0,1,2)$-ternary minimum-energy task allocation vector: $x\_OTAS$, which is computed by applying the genetic algorithm run by O_TAS_par under ${\overrightarrow{RS}}^{MAX}$;
• the corresponding total energy: ${\mathcal{E}}_{OTAS}$ (Joule) and network energy: $E_{OTAS}^{NET}$ (Joule) consumed by ${\overrightarrow{x}}_{OTAS}$ under the (aforementioned) fixed maximal resource allocation vector ${\overrightarrow{RS}}^{MAX}$.

The function O_TAS_ par assumes that all the global variables of parallel VirtFogSim are already setup. It also utilizes the following main set of local variables: $Popmatrix$, $Childlist$, $Mutationlist$, and $Candidatelist$, all of dimensions $PS\times (V+1)$.

Algorithm 1 reports a pseudo-code of the O_TAS_ par function. Due to the utilization of the parfor-cycle, the resulting asymptotic computational complexity of the O_TAS_ par function scales up as: $\mathcal{O}\left(\left(PS\times G_{MAX}\right)/n_{core}\right)$, where:

• $PS$ is the population size of the checked task allocation vectors;
• $G_{MAX}$ is the number of iterations (that is, the number of generations) run by the genetic algorithm implemented by $O\_TAS\_par$; and,
• $n_{core}\ge 1$ is the number of available parallel cores that support the execution of VirtFogSim.

Algorithm 1O_TAS_ par function.
function: $\left[{\overrightarrow{x}}_{OTAS},{\mathcal{E}}_{OTAS},{\mathcal{E}}_{OTAS}^{NET}\right]=O\_TAS\_par\left(PS,CF,G_{MAX},MN\right)$. Output: $\left(1\times V\right)$ ternary vector of the best task allocation ${\overrightarrow{x}}_{OTAS}\in {\left\{0,1,2\right\}}^V$ under the fixed maximal resource allocation vector; scalar total energy ${\mathcal{E}}_{OTAS}$ (Joule) and network energy ${\mathcal{E}}_{OTAS}^{NET}$ (Joule) consumed by ${\overrightarrow{x}}_{OTAS}$ under the fixed maximal allocation vector.
		▷Begin O_TAS_ par function
		▷Initialization phase
1:	Randomly generate a list $\left\{{\overrightarrow{x}}_i\in {\left\{0,1,2\right\}}^V,i=1,\cdots ,PS\right\}$ of task allocation vectors and store it into $\left[Popmatrix\right]$;
2:	Compute and store into the ${\left(V+1\right)}^{\mathrm{th}}$ column of $\left[Popmatrix\right]$ the energy consumed by each $\overrightarrow{x}\in \left[Popmatrix\right]$ by calling $PS$ times the evaluatestaticenergy_p function;
3:	Sort the $PS$ elements of $\left[Popmatrix\right]$ for increasing values of their energies;
4:	Copy the first row of the sorted $\left[Popmatrix\right]$ into ${\overrightarrow{x}}_{OTAS}$, ${\mathcal{E}}_{OTAS}$, and ${\mathcal{E}}_{OTAS}^{NET}$;
5:	Set the number $Q=⌈CF\times PS⌉$ of the elements of $\left[Popmatrix\right]$ for crossover at each generation;
6:	for$j=1:G_{MAX}$do	▷Iterative phase
7:	Perform the pair-wise crossover of the first Q elements of $\left[Popmatrix\right]$ by calling $\left(Q/2\right)$ times the Crossover function and store the Q crossover elements in $\left[Childlist\right]$;
8:	Randomly mute $MN$ positions of the last $\left(PS-Q\right)$ elements of $\left[PopMatrix\right]$ by calling $\left(PS-Q\right)$ times the Mutation function and store the mutated elements into $\left[Mutationlist\right]$;
9:	Compute and store the consumed energy of each element of $\left[Childlist\right]$ and $\left[Mutationlist\right]$ by calling $PS$ times the function evaluatestaticenergy_p;
10:	Copy the first Q elements of $\left[Popmatrix\right]$ and the overall $\left[Childlist\right]$ and $\left[Mutationlist\right]$ into $\left[Candidatelist\right]$;
11:	Sort the $\left(PS+Q\right)$ elements of $\left[Candidatelist\right]$ for increasing values of their energies;
12:	Copy the first $PS$ elements of $\left[Candidatelist\right]$ into $\left[Popmatrix\right]$;
13:	if the total energy of the first element of $\left[Popmatrix\right]$ is lower than ${\mathcal{E}}_{OTAS}$ then
14:	Copy the first row of $\left[Popmatrix\right]$ into ${\overrightarrow{x}}_{OTAS}$, ${\mathcal{E}}_{OTAS}$ and ${\mathcal{E}}_{OTAS}^{NET}$;
15:	end if
16:	end for
17:	return$\left[{\overrightarrow{x}}_{OTAS},{\mathcal{E}}_{OTAS},{\mathcal{E}}_{OTAS}^{NET}\right]$.	▷End O_TAS_ par function

The RAP_p function implements primal-dual adaptive iterations on the cluster of available parallel workers that supports the execution of VirtFogSim. The goal is to perform the optimal resource allocation under the input task allocation vector $\overrightarrow{x}$. Its input formal variables are the vectors $\overrightarrow{x}$, $\overrightarrow{z}$, and $\overrightarrow{s}$ and the matrices A, $Da$, and $S_{RAP}$. The corresponding output variables are the vector $\tilde{\overrightarrow{y}}$ and the scalars ${\tilde{\mathcal{E}}}_{TOT}$ and ${\tilde{\mathcal{E}}}_{NET}$.

Specifically, we have that:

• $\overrightarrow{x}$ is a $\left(1\times V\right)$-dimensional $\left\{0,1,2\right\}$-ternary vector. It fixes the allocation to the Fog/Mobile/Cloud nodes of the V tasks that compose the considered application DAG. Specifically, $x\left(i\right)=0,1$, and 2 means that the $i^{\mathrm{th}}$ application task is executed at the Fog clone, Mobile device, and Cloud clone, respectively;
• $z\triangleq \left[f_M^0,f_F^0,f_C^0,R_U^0,R_D^0,B_U^0,B_D^0,{\lambda }^0\right]$ is a $\left(1\times 8\right)$ vector of real-valued non-negative scalars. It fixes the vector starting point of the primal-dual iterations to be performed. Its first seven components are measured in bit/s, while the starting Lagrange multiplier ${\lambda }_0$ is measured in Joule.
• $\tilde{\overrightarrow{y}}\triangleq \left[{\tilde{f}}_M,{\tilde{f}}_F,{\tilde{f}}_C,{\tilde{R}}_U,{\tilde{R}}_D,{\tilde{B}}_U,{\tilde{B}}_D\right]$ is a $\left(1\times 7\right)$ vector of real-valued non-negative scalars. It returns the vector of the optimal resource allocation attained by the performed primal-dual iterations. Its seven components are measured in bit/s;
• ${\tilde{\mathcal{E}}}_{TOT}$ is the total communication-plus-computing energy consumed by the computed optimal resource allocation $\tilde{y}$ under the given task allocation x. ${\tilde{\mathcal{E}}}_{TOT}$ is measured in Joule.
• ${\tilde{\mathcal{E}}}_{NET}$ is the overall network energy consumed by the computed optimal resource allocation $\tilde{y}$ under the given task allocation x. ${\tilde{\mathcal{E}}}_{NET}$ is measured in Joule;
• s, A and $Da$ are respectively the global workload vector, adjacency matrix, and edge matrix of the underlying DAG;
• $S_{RAP}$ is the $\left(1\times 54\right)$-dimensional vector of all other global variables used by $RAP\_p$ for its execution.

Since RAP_p is executed by the calling functions in the body of parfor-cycles, the global variables: s, A, $Da$, and $S_{RAP}$ must be passed to the RAP_ p function as input parameters, in order to guarantee the synchronization of the parfor-cycles performed by multiple parallel workers.

The RAP_ p function requires $\left(110+6\times V\right)$ scalar local variables and two $\left(1\times I_{MAX}\right)$ vector local variables. The $I_{MAX}$ parameter is stored by $S_{RAP}$ and fixes the maximum number of allowed primal-dual iterations. The asymptotic implementation complexity of the RAP_ p function scales as: $\mathcal{O}\left(8\times I_{MAX}\right)$.

The parallel Exhaustive Search-Strategy (ES_S_par) function returns:

• the V-tuple $(0,1,2)$-ternary ${\overrightarrow{x}}_{ESS}$. It is the globally best task allocation vector, which is generated by performing the exhaustive search over the full population of the $3^{(V-2)}$ ternary task allocation vectors. The first and last components of each ternary task allocation vector are unit valued, i.e., the first and last tasks of the application DAG are executed, by design, at the Mobile device;
• the seven-tuple resource allocation vector: ${\overrightarrow{RS}}_{ESS}=\left[f_M^{\ast },f_F^{\ast },f_C^{\ast },R_U^{\ast },R_D^{\ast },B_U^{\ast },B_D^{\ast }\right]$ (bit/s), corresponding to the returned global task allocation vector ${\overrightarrow{x}}_{ESS}$;
• the total communication-plus-computing energy ${\mathcal{E}}_{ESS}$ (Joule), network energy ${\mathcal{E}}_{NET}^{ESS}$ (Joule), and bandwidth $R{B}_{ESS}$ (bit/s) of the backhaul connection consumed under the returned searched global best task allocation vector ${\overrightarrow{x}}_{ESS}$.

Being obtained through an exhaustive search, ${\mathcal{E}}_{ESS}$ is, by design, the global minimum total energy consumed under the considered application scenario of Figure 1.

In order to guarantee that the full search space of the the task allocation vectors is actually explored, the ES_S_par function calls the auxiliary function: find_ allocations, which returns the: $\left(3^{\left(V-2\right)}\times V\right)$-dimensional $POP$ matrix of all possible $(0,1,2)$-ary task allocation patterns. Afterward, ES_S_par calls the RAP_p function under each ternary task allocation vector returned by find_allocations, in order to evaluate the resulting best resource allocation vector, the corresponding total and network consumed energies, and the utilized backhaul bandwidth. After checking all the $3^{(V-2)}$ task allocations, ES_S_par sorts them for increasing values of their total consumed energies and, then, picks out the first element of the attained sorted list. This last comprises the globally best task allocation $x_{ESS}$, its associated resource allocation vector ${\overrightarrow{RS}}_{ESS}$, the corresponding consumed energies ${\mathcal{E}}_{ESS}$ and ${\mathcal{E}}_{NET}^{ESS}$, and the utilized backhaul bandwidth $R{B}_{ESS}$.

A pseudo-code of the ES_E_ par function is reported in Algorithm 2. The resulting asymptotic computational complexity scales up as: $\mathcal{O}\left(\left(3^{\left(V-2\right)}\times 8\times G_{MAX}\right)/n_{core}\right)$.

Algorithm 2ES_S_par function.
function: $\left[{\overrightarrow{x}}_{ESS},{\overrightarrow{RS}}_{ESS},{\mathcal{E}}_{ESS}^{NET},R{B}_{ESS}\right]=ES\_S\_par$. Output: $\left(1\times V\right)$ ternary vector of the best task allocation over the full population ${\overrightarrow{x}}_{ESS}$; $(1\times 7)$ resource allocation vector: ${\overrightarrow{RS}}_{ESS}\triangleq \left[f_M^{\star },f_F^{\star },f_C^{\star },R_U^{\star },R_D^{\star },B_U^{\star },B_D^{\star }\right]$ (bit/s) under ${\overrightarrow{x}}_{ESS}$; total energy ${\mathcal{E}}_{ESS}$ (Joule) and network energy ${\mathcal{E}}_{ESS}^{NET}$ (Joule) and backhaul bandwidth $R{B}_{ESS}$ (bit/s) consumed by ${\overrightarrow{x}}_{ESS}$.
		▷Begin ES_S_par function
1:	Call the find-allocation function, in order to generate the full set list: $\left\{{\overrightarrow{x}}_i\in {\left\{0,1,2\right\}}^V,i=1,\cdots ,3^{V-2}\right\}$ of ternary V-tuple candidate task allocation vectors, with the first and last elements set to the unit;
2:	Store (by row) $\left\{{\overrightarrow{x}}_i\right\}$ into the first V columns of the $\left[POP\right]$ matrix;
3:	for$i=1:3^{V-2}$do
4:	Compute the resource allocation vector and the total consumed energy of each $\left\{{\overrightarrow{x}}_i\right\}$ by calling the RAP_p function, and store them (by row) into the last columns of the $\left[POP\right]$ matrix;
5:	end for
6:	Sort the $3^{V-2}$ elements of the $\left[POP\right]$ matrix for increasing values of their total energies;
7:	Store the first row of the sorted $\left[POP\right]$ matrix into ${\overrightarrow{x}}_{ESS}$, ${\overrightarrow{RS}}_{ESS}$, ${\mathcal{E}}_{ESS}$, ${\mathcal{E}}_{ESS}^{NET}$ and $R{B}_{ESS}$;
8:	return$\left[{\overrightarrow{x}}_{ESS},{\overrightarrow{RS}}_{ESS},{\mathcal{E}}_{ESS},{\mathcal{E}}_{ESS}^{NET},R{B}_{ESS}\right]$.	▷End ES_S_par function

The parallel Genetic Task Allocation (GeneticTA_par) function assumes that all the global variables of parallel VirtFogSim are already setup. The GeneticTA_ par function jointly optimizes the task allocation and the resource allocation in an adaptive way. It returns:

• the $\left(0,1,2\right)$-ternary V-long vector ${\overrightarrow{x}}_{best}$ of the best searched task allocation;
• the corresponding $\left(1\times 7\right)$ vector: ${\overrightarrow{RS}}_{best}=\left[f_M^{\ast },f_F^{\ast },f_C^{\ast },R_U^{\ast },R_D^{\ast },B_U^{\ast },B_D^{\ast }\right]$ (bit/s), of the optimal resource allocation computed by the RAP_p function under the best task allocation vector ${\overrightarrow{x}}_{best}$;
• the total and network energies: ${\mathcal{E}}_{best}$, ${\mathcal{E}}_{best}^{NET}$ (Joule) that are consumed under the returned ${\overrightarrow{RS}}_{best}$;
• the actual transmission rate: $R{B}_{best}$ (bit/s) of the two-way Fog ↔ Cloud backhaul connection.

Task allocation is performed by GeneticTA_ par by running a genetic algorithm. For this purpose, GeneticTA_ par calls the Crossover and Mutation functions.

Resource allocation is performed by GeneticTA_ par in an adaptive way. For this purpose, at each parallel iteration called by a parfor-cycle, GeneticTA_ par calls the RAP_ p function. Afterward, GeneticTA_ par picks up and stores:

• the best (that is, the minimum energy) task allocation vector ${\overrightarrow{x}}_{best}$;
• the corresponding resource allocation vector ${\overrightarrow{RS}}_{best}$;
• the total consumed energy ${\mathcal{E}}_{best}$ and the corresponding network energy ${\mathcal{E}}_{best}^{NET}$ computed up to the current generation.

$PS$ is the size of each generation of task allocation vectors; $G_{MAX}$ is the number of carried out generations; $CF$ is the fraction of crossover elements of the current generation; and $MN$ is the number of mutated locations on a per-task-allocation basis.

Let $n_{core}\ge 1$ be the number of parallel cores (e.g., parallel workers) managed by the Parallel Toolbox of MATLAB, in order to support the execution of the VirtFogSim package. Hence, the resulting asymptotic implementation complexity scales as in: $\mathcal{O}\left(\left(PS\times G_{MAX}\times 8\times I_{MAX}\right)/\left(n_{core}\right)\right)$.

The Only Mobile Strategy (OM_S) function assumes that all the V tasks of the assigned application DAG are executed at the Mobile device. Then, it returns the $\left(1\times 7\right)$ vector ${\overrightarrow{RS}}_{OM}=\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s) of the corresponding resource allocation, the corresponding total computing-plus-communication consumed energy ${\mathcal{E}}_{OM}$ (Joule), and the (vanishing) network energy: ${\mathcal{E}}_{NET}^{OM}$ (Joule). It is expected that the returned $f_M$ is not zero, while the returned $f_C$, $f_F$, $R_U$, $R_D$, $B_U$, $B_D$, and ${\mathcal{E}}_{NET}^{OM}$ vanish. The asymptotic complexity of the OM_S function scales up as in: $\mathcal{O}\left(8\times I_{MAX}\right)$.

The Only Fog Strategy (OF_S) function assumes that the first and last tasks of the assigned application DAG are executed at the Mobile device, while all the remaining inner $(V-2)$ tasks are executed at the Fog clone. Then, OF_S returns the $\left(1\times 7\right)$ vector ${\overrightarrow{RS}}_{OF}=\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s) of the corresponding resource allocation, the corresponding total computing-plus-networking consumed energy ${\mathcal{E}}_{OF}$, and the network energy ${\mathcal{E}}_{NET}^{OF}$ (Joule). It is expected that the returned $f_M$, $f_F$, $R_U$, and $R_D$ are not zero, while the returned $f_C$, $B_U$, and $B_D$ vanish.

The Only Cloud Strategy (OC_S) function assumes that the first and last tasks of the assigned application DAG are executed at the Mobile device, while all the remaining inner $(V-2)$ tasks are executed at the Cloud clone. Then, it returns the $\left(1\times 7\right)$ vector ${\overrightarrow{RS}}_{OC}=\left[f_M,f_F,f_C,R_U,R_D,B_U,B_D\right]$ (bit/s) of the corresponding resource allocation, the corresponding total computing-plus-networking consumed energy ${\mathcal{E}}_{OC}$ (Joule), and the network energy ${\mathcal{E}}_{NET}^{OC}$ (Joule).

It is expected that the returned $f_M$, $f_C$, $B_U$, and $B_D$ are not zero, while the returned $f_F$, $R_U$, and $R_D$ vanish. The functions OM_S, OF_S, and OC_S call a sequential version RAP of the RAP_p function, with the following initialization vector: $z=\left[0,0,0,0,0,0,0,1\right]$.

4.3. Implemented Auxiliary Functions

In this subsection, we provide the description of some auxiliary functions called from the previous allocation routines. These auxiliary functions are listed in Table 3.

In the Crossover function, $Paren{t}_v^1$ and $Paren{t}_v^2$ are two V-tuple parent vectors to crossover, while $Chil{d}_v^1$ and $Chil{d}_v^2$ are the resulting V-tuple vectors produced by the performed crossover operation. The crossover operation generates an integer index I over the set $\left\{2,3,\dots ,(V-1)\right\}$. It is the pointer to the crossover point. Afterward, $Chil{d}_v^1$ and $Chil{d}_v^2$ are obtained by swapping the first I components of $Paren{t}_v^1$ and $Paren{t}_v^2$ with the last $\left(V-I\right)$ components of $Paren{t}_v^2$ and $Paren{t}_v^1$, respectively.

In the Mutation function, the global input parameter $MN$ fixes the number of the components of the input vector ${\overrightarrow{x}}_v$ to be mutated. ${\overrightarrow{x}}_v$ is the V-tuple $\left(0,1,2\right)$-ternary input vector to be mutated. ${\overrightarrow{out}}_v$ is the V-tuple $\left(0,1,2\right)$-ternary mutated output vector. It is guaranteed that the first and last components of ${\overrightarrow{out}}_v$ are unit valued. The function Mutation generates $MN$ random positions (with $1\le MN\le \left(V-2\right)$) and $MN$ random $\left(0,1,2\right)$-ternary mutation numbers. Then, it mutates ${\overrightarrow{x}}_v$ at the generated $MN$ positions by replacing the generated $MN$ mutation numbers for the corresponding components of ${\overrightarrow{x}}_v$.

In the evaluatestaticenergy_p function for parallel execution, ${\overrightarrow{x}}_v$ is the $(1\times V)$-long input $(0,1,2)$-ternary task allocation vector to be checked. Since the function evaluatestaticenergy_p is called in the body of the parfor-cycle by the $O\_TAS\_par$ function, all the needed global variables: $\overrightarrow{s}$, A, $Da$, and $S_{RAP}$ must be passed to evaluatestaticenergy_p as input parameters. This guarantees the synchronization of the used global variables under parallel execution. If ${\overrightarrow{x}}_v$ is a feasible task allocation pattern, ${\mathcal{E}}_{aux}$ and ${\mathcal{E}}_{NET}$ are the total and network energies consumed by ${\overrightarrow{x}}_v$ under the static maximal resource allocation vector defined as: ${\overrightarrow{RS}}^{MAX}=\left[f_M^{MAX},f_F^{MAX},f_C^{MAX},R_U^{MAX},R_D^{MAX},B_U^{MAX},B_D^{MAX}\right]$ (bit/s). If ${\overrightarrow{x}}_v$ is an infeasible task allocation pattern, then ${\mathcal{E}}_{aux}={\mathcal{E}}_{NET}=\infty$ is returned. The asymptotic complexity of the evaluatestaticenergy_p function is: $\mathcal{O}\left(7\right)$, where seven is the number of the computed resource variables.

Finally, the find_allocations function returns in the matrix $TA$ all the possible: $K^N$ patterns of N numbers that can assume the first K integer values: $0,1,2,\dots ,K-1$. This function is used by the ES_S_par strategy function to obtain all the possible allocations to be tested in the exhaustive search.

4.4. Dynamic Performance Tracking Function

The numerical outputs of the FogTracker function are three matrices that store the time behaviors of: (i) the consumed total energy ${\mathcal{E}}_{TOT}$ (Joule); (ii) the corresponding consumed network energy ${\mathcal{E}}_{NET}$ (Joule); and (iii) the $lambda$ multiplier (Joule), for values of the iteration index going from one to the given $iteration\_number$. The goal of FogTracker is to test the convergence rate to the steady-state and the steady-state stability of the primal-dual iterations performed by the RAP_ p function when abrupt changes of the maximum allowed WiFi and Cellular up/down bandwidths and task allocation vectors simultaneously happen. For this purpose, at the time indexes:

• $\mathrm{round}\left((1/5)\times iteration\_number\right)+1$;
• $\mathrm{round}\left((2/5)\times iteration\_number\right)+1$;
• $\mathrm{round}\left((3/5)\times iteration\_number\right)+1$;
• $\mathrm{round}\left((4/5)\times iteration\_number\right)+1$,

the initial values of the global variables: $R_U^{MAX}$, $R_D^{MAX}$, $B_U^{MAX}$, $B_D^{MAX}$ undergo abrupt changes. These changes are obtained by multiplying them by the (non-negative) input factors: $jum{p}_{WiFi}^1$, $jum{p}_{CELL}^1$, $jum{p}_{WiFi}^2$, and $jum{p}_{CELL}^2$. These jump coefficients are declared as global variables and set by the user. At the same time, the task allocation vector passes from ${\overrightarrow{x}}_1$ to ${\overrightarrow{x}}_2$ and, then, from ${\overrightarrow{x}}_2$ to ${\overrightarrow{x}}_3$. Finally, it comes back to ${\overrightarrow{x}}_1$.

The resulting time behaviors of the total energies, network energies, and lambda multiplier values over the time window: $[1,iteration\_number]$ are plotted under the three user-specified values of $a_{MAX}$, which are stored by the $(1\times 3)$ global input vector ${\overrightarrow{a}}_{MAX-FT}$. In so doing, the FogTracker function tests the convergence rate of the RAP_ p function when the task allocation vector undergoes abrupt changes. The actual feasibility of the task allocation vectors: ${\overrightarrow{x}}_1$, ${\overrightarrow{x}}_2$, and ${\overrightarrow{x}}_3$ is explicitly checked in the body of the FogTracker function. If at least one of these feasibility checks fails, the FogTracker function generates a suitable error message and, then, halts.

In detail, the FogTracker function performs the following nine steps:

• it begins to run the RAP_ p function over the time interval: $[1,\mathrm{round}\left((1/5)\times iteration\_number\right)]$, under the original settings of: $R_U^{MAX}$, $R_D^{MAX}$, $B_U^{MAX}$, $B_D^{MAX}$ dictated by the main program VirtFogSim and the first input task allocation vector ${\overrightarrow{x}}_1$;
• at the end of the iteration number: $\mathrm{round}\left((1/5)\times iteration\_number\right)$, the WiFi maximum bandwidths: $R_U^{MAX}$ and $R_D^{MAX}$ are multiplied by the (non-negative) scaling factor: $jum{p}_{WiFi}^1$, while the maximum cellular bandwidths: $B_U^{MAX}$ and $B_D^{MAX}$ are multiplied by the corresponding (possibly, coincident) scaling factor: $jum{p}_{CELL}^1$. Furthermore, the task allocation vector is changed into ${\overrightarrow{x}}_2$. In so doing, both the maximum available bandwidths and the task allocation vector undergo abrupt (typically, user mobility induced) variations;
• the RAP_ p function runs over the time-interval:
  $\left[\mathrm{round}\left((1/5)\times iteration\_number\right)+1,\mathrm{round}\left((2/5)\times iteration\_number\right)\right]$ under the setting of Step 2. Its initial vector is the last returned vector of the RAP_p at the previous iteration: $\mathrm{round}\left((1/5)\times iteration\_number\right)$;
• at the end of iteration number: $\mathrm{round}\left((2/5)\times iteration\_number\right)$, all four maximum bandwidths: $R_U^{MAX}$, $R_D^{MAX}$, $B_U^{MAX}$, and $B_D^{MAX}$ are restored to their original values. Furthermore, even the task allocation vector is set back to its original value ${\overrightarrow{x}}_1$;
• the RAP_ p function runs over the time interval:
  $\left[\mathrm{round}\left((2/5)\times iteration\_number\right)+1,\mathrm{round}\left((3/5)\times iteration\_number\right)\right]$ under the setting of Step 4. Its initial vector is the last returned vector of the RAP_ p at the previous iteration number: $\mathrm{round}\left((2/5)\times iteration\_number\right)$;
• after the iteration number: $\mathrm{round}\left((3/5)\times iteration\_number\right)$, the WiFi maximum up/down bandwidths: $R_U^{MAX}$, and $R_D^{MAX}$ are multiplied by the non-negative scaling factor $jum{p}_{WiFi}^2$, while the cellular maximum up/down bandwidths: $B_U^{MAX}$ and $B_D^{MAX}$ are multiplied by the scaling factor: $jum{p}_{CELL}^2$. At the same time, even the task allocation vector is changed to the third value ${\overrightarrow{x}}_3$. In so doing, both the maximum available bandwidths and the task allocation vector undergo abrupt (typically, user mobility-induced) variations;
• the RAP_ p function runs over the time-interval:
  $\left[\mathrm{round}\left((3/5)\times iteration\_number\right)+1,\mathrm{round}\left((4/5)\times iteration\_number\right)\right]$ under the setting of Step 6. Its initial vector is the last returned vector of the $RAP\_p$ at the previous iteration number: $\mathrm{round}\left((3/5)\times iteration\_number\right)$;
• after the iteration number: $\mathrm{round}\left((4/5)\times iteration\_number\right)$, all four WiFi/CELLmaximum up/down bandwidths are restored to their original values. Furthermore, even the task allocation vector is set back to the first value ${\overrightarrow{x}}_1$;
• finally, the RAP_ p function runs over the time-interval:
  $\left[\mathrm{round}\left((4/5)\times iteration\_number\right)+1,iteration\_number\right]$ under the setting of Step 8. Its initial vector is the last returned vector of the RAP_ p at the previous iteration number: $\mathrm{round}\left((3/5)\times iteration\_number\right)$.

Graphic plots of the time behaviors of the returned ${\mathcal{E}}_{TOT}$, ${\mathcal{E}}_{NET}$, and $lambda$ are displayed at the end of each FogTracker run (see Section 5 in the sequel). From the outset, it follows that the asymptotic complexity of the FogTracker function scales up as in: $\mathcal{O}\left(8\times iteration\_number\times 3\right)$.

Overall,

Table 4. A synoptic overview of the computational complexities of the main functions supported by the VirtFogSim package.

Function	Asymptotic Computational Complexity
GeneticTA_par	$\mathcal{O}\left(\left(PS\times G_{MAX}\times 8\times I_{MAX}\right)/\left(n_{core}\right)\right)$
OM_S	$\mathcal{O}\left(8\times I_{MAX}\right)$
OF_S	$\mathcal{O}\left(8\times I_{MAX}\right)$
OC_S	$\mathcal{O}\left(8\times I_{MAX}\right)$
O_TAS_par	$\mathcal{O}\left(\left(PS\times G_{MAX}\right)/n_{core}\right)$
ES_S	$\mathcal{O}\left(\left(3^{\left(V-2\right)}\times 8\times G_{MAX}\right)/n_{core}\right)$
FogTracker	$\mathcal{O}\left(8\times iteration\_number\times 3\right)$

reports a synoptic view of the asymptotic computational complexities of the described GeneticTA_par, OM_S, OF_S, OC_S, O_TAS_par, ES_S_par, and FogTracker functions under their parallel implementations.

5. VirtFogSim: Supported Formats of the Rendered Data

Under the current version of the simulator, both the VirtFogSim and VirtFogSimGUI interfaces (see Appendix A) support four main formats, in order to render the results output by the seven optimization algorithms of Table 4. The functions used to obtain these formats are listed in

Table 5. Rendering functions for the supported formats implemented in VirtFogSim.

Function	Description
$\text{print\_solution}\left(\overrightarrow{RS},\overrightarrow{\mathcal{E}},RB,strategy\right)$	It implements the tabular format.
$\text{plot\_solution}\left(\overrightarrow{x},\overrightarrow{RS},\overrightarrow{\mathcal{E}},RB,fignumber,strategy\right)$	It implements the colored bar plot format.
$\text{plot\_DAG}\left(\overrightarrow{s},A,Da,\overrightarrow{x},strategy,type\right)$	It implements the colored labeled DAG map.
$\text{plot\_FogTracker}\left({\mathcal{E}}_m,{\mathcal{E}}_m^{NET},{\lambda }_m,fignumber,interpolation\right)$	It implements the colored time plot format.

and fully described in the VirtFogSim User Guide (see Section 8). These rendering formats are:

• the Tabular format. It is enabled by the print_solution graphic function;
• the Colored Bar Plot format. It is enabled by the plot_solution graphic function;
• the Colored Time Plot format. It is enabled by the plot_FogTracker graphic function;
• the Colored Labeled DAG Map format. It is enabled by the plot_ DAG graphic function.

Specifically, the print_solution function prints on the MATLAB prompt the result obtained by running the tested strategies under the selected DAG and given input parameters (see Table A2). Its input parameters are: (i) the vector $\overrightarrow{RS}$ collecting the seven allocated resources; (ii) the vector $\overrightarrow{\mathcal{E}}$ collecting the total computing–plus-networking consumed energy; (iii) the bandwidth $RB$ of the Fog↔Cloud two-way backhaul connection; and (iv) a string $strategy$ representing the name of the tested strategy. Then, the function prints the following results in a numerical form:

• the Computing/Communication Ratio (CCR);
• the required minimum DAG execution throughput ($T{H}_0^{MIN}$);
• the per-core computing frequency at the Mobile device ($f_M$);
• the per-core computing frequency at the Fog clone ($f_F$);
• the per-core computing frequency at the Cloud clone ($f_C$);
• the transport bit rate of the WiFi-based TCP/IP Mobile-to-Fog connection ($R_U$);
• the transport bit rate of the WiFi-based TCP/IP Fog-to-Mobile connection ($R_D$);
• the transport bit rate of the 3G/4G Cellular TCP/IP Mobile-to-Cloud connection ($B_U$);
• the transport bit rate of the 3G/4G Cellular TCP/IP Cloud-to-Mobile connection ($B_D$);
• the transport rate of the TCP/IP two-way Fog↔Cloud backhaul connection ($R_B$);
• the total computing-plus-networking consumed energy (${\mathcal{E}}_{TOT}$);
• the consumed computing energy (${\mathcal{E}}_{COM}$);
• the consumed networking energy (${\mathcal{E}}_{NET}$);
• the percent networking-to-total ratio of the consumed energies: $\%\left({\mathcal{E}}_{NET}/{\mathcal{E}}_{TOT}\right)$.

If the input vector $\overrightarrow{RS}$ is empty, only a reduced set of information is printed.

The plot_solution function displays the: (i) utilized computing/bandwidth resources; (ii) task allocation pattern; and (iii) consumed energies, returned by the carried out VirtFogSim run in terms of suitable three-colored bar plots. In addition to the input used by the previous function, the other parameters are: (i) the $(0,1,2)$-ary allocation vector $\overrightarrow{x}$ to be displayed; and (ii) the number of the fignumber to be displayed. The function renders a figure that is composed of three horizontal subplots. These subplots display:

• the returned task allocation in the form of a three-color bar plot;
• the allocated computing frequencies and network bandwidths in the form of a plot with eight bars;
• the total, computing and networking consumed energies in the form of a plot with three bars.

If the passed task allocation vector $\overrightarrow{x}$ and/or resource allocation vector $\overrightarrow{RS}$ are empty, then only one or two subplots are rendered. An illustrative screen-shoot of the bar-plots rendered by the execution of the GeneticTA_par function is shown in Figure 3.

The plot_DAG function returns a graphic representation of the application DAG passed as the input. Its input parameters are: (i) the vector $\overrightarrow{s}$ containing the nodes’ weights of the DAG; (ii) the adjacency matrix A of the DAG; (iii) the matrix $Da$ of the weights of the DAG edges; (iv) the vector $\overrightarrow{x}$ of task allocation to be visualized; (v) a string of characters with the name of the policy that will be shown; and (vi) an optional three-valued parameter $type$, which allows selecting the desired DAG visualization format as follows:

• all DAG nodes with the same color and unlabeled;
• all DAG nodes with the same color and labeled by increasing identification numbers;
• each DAG node is numbered and colored on the basis of the allocation actually stored by the input vector $\overrightarrow{x}$.

The plot_DAG function needs the Bioinformatics Toolbox to be installed in the MATLAB environment. By default, the graph returned by plot_DAG retains the following features:

• all the workload values labeling the DAG nodes and edges are expressed in kilo-bit;
• red-colored tasks are understood to be allocated to the Fog node;
• green-colored tasks are understood to be allocated to the Mobile node; and,
• blue-colored tasks are understood to be allocated to the Cloud node.

An illustrative screen-shoot of the colored task map DAG rendered by the execution of the GeneticTA_par function is shown in Figure 4.

Finally, the plot_fogTracker function provides the graphic capabilities needed for a proper plot of the time-traces of the total energies, networking energies, and lambda multipliers generated by the FogTracker function under the three values of the step-size that are stored by the vector ${\overrightarrow{a}}_{MAX-FT}$ (see Table A2). Its input parameters are: (i) the matrix ${\mathcal{E}}_m$ of the total consumed energies returned by FogTracker; (ii) the matrix ${\mathcal{E}}_m^{NET}$ of the consumed networking energies returned by FogTracker; (iii) the matrix ${\lambda }_m$ of the values of the lambda multipliers returned by FogTracker; and (iv) an optional parameter $interpolation$ allowing the plot of an interpolated version of the figures if set to one. The function renders a figure composed of three horizontal sub-plots that trace:

• the total consumed computing-plus-networking energies under the three values of the step-size stored by ${\overrightarrow{a}}_{MAX-FT}$;
• the corresponding consumed networking energies;
• the related values of the lambda multiplier.

Some illustrative screen-shoot of the dynamic plots rendered by the FogTracker function are shown in Figures 7–9 of Section 6.2.

Pre-Loaded Application DAGs

In the current version of the VirtFogSimGUI interface (see Figure A1 in Appendix A) is available an archive that stores fourteen test DAGs, together with the related sets of (suitably-tuned) input parameters. These DAGs are ready-for-use, e.g., they may be retrieved by the user and, then, run under both the (previously-described) interfaces of the simulator.

The archived DAGs were retrieved from the current literature [25,26,27,29,30,31,32,33] and feature a number of heterogeneous real-world applications of practical interest. Their topologies cover a large spectrum (e.g., tree, fork, parallel, mesh, and hybrid topologies, just to name a few), and their number of nodes ranges from $V=9$ to $V=45$.

6. VirtFogSim in Action: Testing Its Numerical Capabilities

The aim of this section is to provide insights about the actual capability of the developed VirtFogSim package by: (i) numerically testing and comparing the energy-delay-tracking performance of its natively-supported optimization tools of Section 4.4 (see also Table 4) under three use cases of practical interest; and (ii) checking the performance of the underlying MATLAB code by numerically profiling its simulation times and volumes of inter-core exchanged data over a spectrum of multi-core execution environments.

All the carried out simulations have been done by exploiting the capabilities of a hardware execution platform equipped with: (i) an Intel 10-core i9-7900X processor; (ii) 32 GB of RAM DDR 4; (iii) an SSD with 512 GB plus an HDD with 2 TB; (iv) a GPU ZOTAC GeForce GTX 1070. The release R2018a of MATLAB provided the underlying software execution platform. It is equipped with the MATLAB Parallel Toolbox, in order to exploit the multi-core capability (possibly) offered by the host hardware platform.

We anticipate that, unless otherwise stated, all the simulations have been carried out under the parameter setting reported in the last column of the final Table A2 in the Appendix.

6.1. Use Cases and Related DAGs

Three real-world use cases have been selected, in order to test the performance of the proposed VirtFogSim package under different application environments featured by (very) heterogeneous workflows and DAG topologies. Figure 5 and Figure 6 report the graphs of the considered test DAGs.

DAG1 in Figure 5a describes the workflow of a (small-sized) parallel MAP/REDUCER computing application [34]. This DAG is composed of the parallel combination of two MAP/REDUCER nodes (e.g., MAP/REDUCER#1 on the left and MAP/REDUCER#2 on the right of Figure 5a, respectively). They share the same input and output nodes (e.g., Nodes 1 and 12 in Figure 5a, respectively), which are forced to be executed by the mobile device. Nodes 2 and 7 act as load balancers, while Nodes 3, 4, 5 and 8, 9 10 act as Mappers. Finally, Nodes 6 and 11 perform reduce operations. The resulting DAG topology of Figure 5a possesses the following features: (i) it is symmetric; (ii) all six input-output paths are equal-length; and (iii) MAP/REDUCER#1 (resp., MAP/REDUCER#2) is more computing-intensive (resp., more communication-intensive) than MAP/REDUCER#2 (resp., MAP/REDUCER#1).

DAG2 in Figure 5b details the workflow of a video navigation application [33], which involves the parallel execution of three sub-programs, namely a graphic sub-program (left section of Figure 5b), a subprogram for face detection (middle section of Figure 5b), and a video-processing subprogram (right section of Figure 5b). All these sub-programs share the same input and output nodes (e.g., Nodes 1 and 15 in Figure 5b), which implement data-rendering functionalities and, then, are forced to be executed at the Mobile device. DAG2 is medium-sized (e.g., it is composed of 15 tasks and 21 edges), and its topology possesses the following features: (i) it is asymmetric; (ii) it is composed of the parallel combination of three heterogeneous sub-DAGs, which exhibit fork, parallel, and tree-shaped sub-topologies, respectively; and (iii) the face detection and video processing subprograms are computing- and communication-intensive, respectively, while the graphic subprogram is of a mixed type.

Lastly, DAG3 in Figure 6 describes the workflow for the simulation of Newton’s equations describing pseudo-chaotic molecular dynamics [29]. It is a large-sized DAG (e.g., it is composed of 41 tasks and 70 edges), and its topology exhibits the following main features: (i) it is very irregular and mimics a random graph, which combines a number of heterogeneous sub-graphs with chain, parallel, fork, tree, and mesh-type sub-topologies; (ii) it is composed of a large number of pseudo-generated edges, which join randomly-selected task pairs. As a consequence, in Figure 6, there are a number of inter-crossing edges, as well as a number of variable-length paths that go from the input task to the output one. (iii) Task workloads and edge weights have been randomly generated.

Finally, we point out that the summations of task workloads and edge weights of all considered DAGs of Figure 5 and Figure 6 are the same and equate to 3.32 (Mbit) and 1.66 (Mbit), respectively. In so doing, we expect that the results of the performance comparisons carried out in the next sub-sections are fair.

6.2. Comparative Tracking Performance under Intermittent WiFi Connectivity

The goal of this section is to test the convergence speed to the steady-state and the steady-state stability of the primal-dual iterations implemented by the RAP_p function of Section 4.4 when abrupt changes of the available WiFi up/down bandwidths and task allocation patterns simultaneously happen. For this purpose, we run the FogTracker function of Section 4.4 at $T_{DAG}=0.3$ under the (previously-described) three test DAGs. The obtained dynamic behaviors of the total consumed energies ${\mathcal{E}}_{TOT}$, network energies ${\mathcal{E}}_{NET}$, and lambda multipliers are reported in Figure 7a–c (resp., Figure 8a–c and Figure 9a–c) for various test values of the speed-up factor $a_{MAX}$ of Section 3.5 under DAG1 (resp., DAG2 and DAG3).

All the reported time behaviors refer to a common application scenario under which both task allocation patterns and up/down WiFi bandwidths undergo simultaneous abrupt changes at the iteration indexes $t=1$, 1000, 2000, 3000, and 4000 (see the corresponding step-like jumps of the plots of Figure 7, Figure 8 and Figure 9). These changes are typically triggered by device mobility, which may give rise to an intermittent availability of the WiFi connectivity. More in detail, we have that: (i) at $t=1$, the up/down cellular (resp., WiFi) bandwidths are turned ON (resp., turned OFF), and all tasks are allocated to the Cloud node; (ii) at $t=1000$, the up/down WiFi bandwidths are turned ON, and all tasks are allocated to the Mobile node; (iii) at $t=2000$, the WiFi bandwidths are still turned OFF, and all tasks are re-allocated to the Cloud node; (iv) at $t=3000$, the up/down WiFi bandwidths are turned ON once time, and all tasks are allocated to the Fog node; and, finally, (v) at $t=4000$, the up/down WiFi bandwidths are definitively turned OFF, and all tasks are re-migrated to the Cloud node. After each change of the setup environment, the RAP_p function runs, in order to re-allocate both the per-clone computing frequencies at the Mobile-Fog-Cloud nodes and the corresponding up/down Cellular-WiFi bandwidths properly.

An examination of the time-plots reported in Figure 7, Figure 8 and Figure 9 leads to three main insights. First, even in the presence of the (aforementioned) abrupt changes of the operating setups, the corresponding lambda multipliers remain almost surely vanishing (see Figure 7c, Figure 8c, and Figure 9c), and this supports the conclusion that all the performed resource allocations computed by the RAP_p function are, indeed, feasible (e.g., they meet the constraint in Equation (4) on the allowed maximum DAG execution time). Second, we have numerically ascertained that the abrupt step-like jumps experienced by the plots of ${\mathcal{E}}_{NET}$ in Figure 7b, Figure 8b, and Figure 9b at the changing instants are the combined effects of both the ON-OFF availability of the WiFi connectivity and the associated re-allocation of the Cellular up/down bandwidths. Third, a comparative examination of the red-green-blue colored plots of Figure 7a, Figure 8a, and Figure 9a confirms that bigger values of the speed-up factor $a_{MAX}$ of Section 3.5 speed-up the convergence to the corresponding steady-states, but also tend to introduce larger oscillation phenomena.

Overall, two final lessons are learned by the carried out tracking analysis. First, at least under the considered application scenarios, values of $a_{MAX}$ ranging over the (quite broad) interval: $9.5\times {10}^{-8}$–$2.25\times {10}^{-7}$ exhibit good tradeoffs among the contrasting requirements of quick reaction to abrupt (possibly mobility-induced) changes of the operative scenarios and stable behavior in the steady-state. Second, values of the primal-dual iterations $I_{MAX}$ limited up to 500–700 suffice for attaining stable resource allocations, even in the presence of abrupt changes of the operation environment.

6.3. Comparative Task and Resource Allocation Performance

In this section, we compare the task placement, resource allocation, and energy performance of the (previously-described) GeneticTA_par strategy under the considered test DAGs. This is done for values of the allowed execution times $T_{DAG}$ of 0.3, 0.6, 0.9, 1.5, and 3.0 (s). The final goal is to acquire insights about the effects of the DAG topologies, their sizes, and corresponding distributions of the task workloads and edge weights on the allocation patterns returned by GeneticTA_par.

The numerical results obtained by running the VirtFogSim toolbox are reported in

Table 6. Task, resource allocation, and energy consumption returned by GeneticTA_par under DAG1. Resources are measured in Mbit/s and energies in Joule.

${\mathbf{T}}_{\mathbf{DAG}}$	Mobile Tasks	Fog Tasks	Cloud Tasks	${\mathbf{f}}_{\mathbf{M}}$	${\mathbf{f}}_{\mathbf{F}}$	${\mathbf{f}}_{\mathbf{C}}$	${\mathbf{R}}_{\mathbf{U}}$	${\mathbf{R}}_{\mathbf{D}}$	${\mathbf{B}}_{\mathbf{U}}$	${\mathbf{B}}_{\mathbf{D}}$	$\mathbf{BR}$	${\mathcal{E}}_{\mathbf{TOT}}$	${\mathcal{E}}_{\mathbf{NET}}$
0.3	$\left\{1,12\right\}$	$\left\{3,7-9,11\right\}$	$\left\{2,4-6,10\right\}$	11.81	3.00	2.50	7.42	8.25	3.96	1.16	0.0	31.81	16.72
0.6	$\left\{1,12\right\}$	$\left\{3,7-9,11\right\}$	$\left\{2,4-6,10\right\}$	11.75	2.38	2.47	6.97	7.61	2.31	0.95	0.0	28.76	13.70
0.9	$\left\{1,3,12\right\}$	$\left\{-\right\}$	$\left\{2,4-11\right\}$	11.60	0.00	2.00	0.00	0.00	0.75	0.76	0.0	27.59	11.38
1.5	$\left\{1,3,12\right\}$	$\left\{2,4-11\right\}$	$\left\{-\right\}$	11.50	1.82	0.00	6.75	7.15	0.00	0.00	0.0	26.11	9.48
3.0	$\left\{1,2,3,12\right\}$	$\left\{4-11\right\}$	$\left\{-\right\}$	11.35	1.79	0.00	6.72	7.10	0.00	0.00	0.0	25.75	9.45

,

Table 7. Task, resource allocation, and energy consumption returned by GeneticTA_par under DAG2. Resources are measured in Mbit/s and energies in Joule.

${\mathbf{T}}_{\mathbf{DAG}}$	Mobile Tasks	Fog Tasks	Cloud Tasks	${\mathbf{f}}_{\mathbf{M}}$	${\mathbf{f}}_{\mathbf{F}}$	${\mathbf{f}}_{\mathbf{C}}$	${\mathbf{R}}_{\mathbf{U}}$	${\mathbf{R}}_{\mathbf{D}}$	${\mathbf{B}}_{\mathbf{U}}$	${\mathbf{B}}_{\mathbf{D}}$	$\mathbf{BR}$	${\mathcal{E}}_{\mathbf{TOT}}$	${\mathcal{E}}_{\mathbf{NET}}$
0.3	$\left\{1,7,10-12,14,15\right\}$	$\left\{2,3,8,13\right\}$	$\left\{4-6,9\right\}$	11.90	2.47	2.48	7.50	8.51	0.95	6.64	0.0	27.09	8.45
0.6	$\left\{1,7,10-12,14,15\right\}$	$\left\{2,3,8,13\right\}$	$\left\{4-6,9\right\}$	11.80	2.35	2.37	7.35	8.39	0.93	6.52	0.0	26.78	8.35
0.9	$\left\{1,7,10-12,14,15\right\}$	$\left\{2-6,8,9,13\right\}$	$\left\{-\right\}$	11.68	1.83	0.00	6.85	8.21	0.00	0.00	0.0	23.63	7.33
1.5	$\left\{1,7,10-12,14,15\right\}$	$\left\{2-6,8,9,13\right\}$	$\left\{-\right\}$	11.58	1.73	0.00	6.65	8.11	0.00	0.00	0.0	22.81	7.30
3.0	$\left\{1-3,7,8,10-12,14,15\right\}$	$\left\{4-6,9\right\}$	$\left\{-\right\}$	10.98	1.64	0.00	6.55	8.01	0.00	0.00	0.0	22.50	7.22

and

Table 8. Task, resource allocation, and energy consumption returned by GeneticTA_par under DAG3. Resources are measured in Mbit/s and energies in Joule.

${\mathbf{T}}_{\mathbf{DAG}}$	Mobile Tasks	Fog Tasks	Cloud Tasks	${\mathbf{f}}_{\mathbf{M}}$	${\mathbf{f}}_{\mathbf{F}}$	${\mathbf{f}}_{\mathbf{C}}$	${\mathbf{R}}_{\mathbf{U}}$	${\mathbf{R}}_{\mathbf{D}}$	${\mathbf{B}}_{\mathbf{U}}$	${\mathbf{B}}_{\mathbf{D}}$	$\mathbf{BR}$	${\mathcal{E}}_{\mathbf{TOT}}$	${\mathcal{E}}_{\mathbf{NET}}$
0.3	$\left\{1,41\right\}$	$\left\{39,40\right\}$	$\left\{2-38\right\}$	11.92	4.37	2.74	0.00	6.27	0.95	0.00	3.70	18.37	6.16
0.6	$\left\{1,41\right\}$	$\left\{39,40\right\}$	$\left\{2-38\right\}$	11.90	4.35	2.48	0.00	6.26	0.94	0.00	3.70	17.85	6.02
0.9	$\left\{1,41\right\}$	$\left\{38-40\right\}$	$\left\{2-37\right\}$	11.77	4.25	2.47	0.00	6.25	0.95	0.00	3.70	17.60	6.01
1.5	$\left\{1,41\right\}$	$\left\{35-40\right\}$	$\left\{2-34\right\}$	11.67	5.01	2.45	0.00	6.23	0.92	0.00	3.70	17.17	5.81
3.0	$\left\{1,41\right\}$	$\left\{34-40\right\}$	$\left\{2-33\right\}$	11.50	4.05	2.37	0.00	6.09	0.90	0.00	3.70	17.17	5.78

under DAG1, DAG2, and DAG3, respectively.

A view of these results leads to the following four main sets of remarks.

Consumed total energies and allocated resources: A comparative examination of the profiled values reported in the 13^th column of Table 6, Table 7 and Table 8 points out that, in all simulated cases, the total consumed energy ${\mathcal{E}}_{TOT}$ remains limited up to 32 Joule. Since all the test DAGs share the same sum values of the task workloads and edge weights, this supports the conclusion that, at the first order, these factors play a major role in dictating the energy efficiency of the performed task and resource allocations. However, a more detailed examination of these results also unveils the following two trends: (i) at fixed DAG, the consumed energies tend to decrease for increasing values of $T_{DAG}$; and (ii) at fixed $T_{DAG}$, the energy consumption tends to decrease by passing from the (more symmetric and regular) DAG1 to the (more random and irregular) DAG3 (see the DAG graphs of Figure 5 and Figure 6). Roughly speaking, the first trend reflects the fact that larger values of $T_{DAG}$ make the underlying application environment more delay-tolerant. As a consequence, the RAP_p function lowers the steady-state computing frequencies and/or the network bandwidths (see the numerical values reported by the corresponding columns of Table 6, Table 7 and Table 8). This reduces, in turn, the dynamic (e.g., resource-depending) components of the total consumed energies. The second trend confirms the fact that, by design, the optimization capability of genetic-based strategies generally increases with the size and/or the pseudo-random irregular nature of the topology of the underlying DAGs.

Consumed network energies: The impact of the DAG topology and distribution of the edge weights on the resulting consumed network energy ${\mathcal{E}}_{NET}$ seems to be, indeed, more relevant. In fact, a comparative examination of the results reported in the last column of Table 6, Table 7 and Table 8 points out that: (i) under DAG1, the percent ratios of the network energies to the corresponding total ones quickly decrease for increasing values of $T_{DAG}$ and pass from 52% at $T_{DAG}=0.3$ s to: 36.5% at $T_{DAG}=3.0$ s; and (ii) under DAG2 and DAG3, the corresponding network-to-total energy ratios are not so sensitive to the values of $T_{DAG}$. They remain clipped, indeed, around: 31–32%, and around: 33.5–33.7% under DAG2 and DAG3, respectively. This supports the conclusions that: (i) DAG1 features a communication-intensive application, especially at low values of $T_{DAG}$; while (ii) DAG2 and DAG3 describe more computing-intensive applications.

Task allocation patterns: Columns 2, 3, and 4 of Table 6, Table 7 and Table 8 report the Identification Numbers (IDs) of the tasks allocated by GeneticTA_par to the Mobile, Fog, and Cloud nodes under DAG1, DAG2, and DAG3, respectively. Although the reported allocation patterns may strongly depend on the specifically considered DAGs, nevertheless, three main quite general trends seem to stand out. First, at low values of $T_{DAG}$, medium-sized and communication-intensive tasks are typically allocated at the Fog node, while large-sized and communication-light tasks are allotted at the Cloud node. The Mobile device only executes small-sized, but communication-intensive tasks. Second, more and more tasks are shifted from the Cloud node to the Fog node and/or to the Mobile device for decreasing values of $T_{DAG}$.

Utilization of the Fog-Cloud backhaul connection: A peculiar feature of the three-tiered technological platform of Figure 1 is the presence of a (possibly multi-hop and/or wired) two-way backhaul connection, which interconnects the Fog and Cloud nodes. Hence, it may be of interest to attain insights about its utilization for the support of the (previously-mentioned) task allocation patterns performed by GeneticTA_par. Intuitively, we expect that the backhaul connection is used when there are large-sized tasks to be allocated at the Cloud and the volumes of data output by the execution of these tasks are large. Therefore, since the Fog-to-Mobile WiFi connection is more energy efficient than the corresponding Cloud-to-Mobile cellular one, it may be energy-saving to transport the processed data from the Cloud to the Fog over the backhaul connection and, then, use the Fog as a relay node, in order to forward these data to the Mobile device by exploiting the WiFi downlink of Figure 1. This is, indeed, the strategy dictated by GeneticTA_par, in order to execute DAG3. In fact, an examination of the corresponding Table 8 points out that the winner strategy returned by GeneticTA_par utilizes: (i) the cellular bandwidth $B_U$ for the upload of the data to be processed by the Cloud; (ii) the backhaul bandwidth $BR$ for forwarding the processed data to the Fog node; and (iii) the WiFi bandwidth $R_D$ for the final download of the processed data to the Mobile device.

The final conclusion is that, in the considered operating scenario, the utilization of (single-hop) Cloud-to-Mobile and/or Mobile-to-Fog links is less energy efficient than the exploitation of the (multi-hop) Cloud-Fog-Mobile path.

6.4. The Performance Impact of Different Task and Resource Allocation Strategies

In this section, we compare the energy performances of the (previously-described) task and resource allocation strategies supported by the VirtFogSim toolbox under the considered test DAGs. The three-fold final goal is to acquire some insight about: (i) the energy reduction stemming from the dynamic optimization of the computing-networking resources versus the corresponding case of static resource usage; (ii) the performance gap between the genetic-based and the Exhaustive Search-based allocation strategies; and (iii) the energy-saving capability offered by the Mobile-Fog-Cloud three-tier computing platform of Figure 1 versus the corresponding Mobile only, Mobile-Cloud, and Mobile-Fog ones.

The profiled results are reported in Figure 10, Figure 11 and Figure 12 under DAG1, DAG2, and DAG3, respectively. Their examination gives arise to the following three main sets of remarks.

Dynamic-vs.-static resource allocation: A number of benchmark (even quite recent) contributions [26,30,31,33] afford the problem of the resource augmentation of mobile devices by developing various heuristic/meta-heuristic/optimal solutions for energy-efficient task offloading. However, they neglect considering, indeed, the companion problem of dynamic scaling of the computing and/or network resources. Hence, a key (still pending) question is: how much energy may be actually saved by jointly performing task and dynamic resource allocation? In this regard, we point out that both the GeneticTA_par and O_TAS_par strategies implemented by the VirtFogSim toolbox rely on the same genetic-based meta-heuristic for performing task placement. However, GeneticTA_par also performs dynamic resource allocation by invoking the auxiliary RAP_p function, while O_TAS_par does not scale up/down the involved computing frequencies and wireless bandwidths of Figure 1 and clips them at their corresponding maximum values (see Section 4.2). Hence, a direct comparison of the energies consumed by GeneticTA_par and O_TAS_par allows us to profile how much energy may be saved by performing dynamic resource allocation.

In this regard, an examination of the numerical results reported in the first and second bars of Figure 10, Figure 11 and Figure 12 leads to three main insights. First, the energy ratio: ${\mathcal{E}}_{O\_TAS\_par}/{\mathcal{E}}_{GeneticTA\_par}$ ranges over the intervals: 2.85–3.50, 3.46–4.10, and: 4.50–4.56 under DAG1, DAG2, and DAG3, respectively. Second, at fixed DAG, the energy savings stemming from performing dynamic resource allocation reaches their maxima at values of $T_{DAG}$ of the order of 0.6–0.9 s, while tending to somewhat decrease at smaller and higher execution delays. Third, the average energy-saving stemming from dynamic optimization is more relevant under DAG3.

Overall, the performed analysis unveils that the dynamic optimization of the allocated computing-networking resources plays, indeed, a pivotal role in reducing the energy consumption of the overall technological platform of Figure 1.

Genetic-vs.-Exhaustive Search performance comparison: The focus of this subsection is on the tradeoffs among energy performance and execution complexity that are attained by the (meta-heuristic) GeneticTA_par and the (optimal) exhaustive E_ESS_par strategies. In this regard, we point out that all the simulations of the GeneticTA_par function are carried out at population size $PS=120$ and per-population number of generations $G_{MAX}=100$. In so doing, the resulting computational complexity of the run GeneticTA_par code is of the order of (see Table 4) $\mathcal{O}\left(120\times 100\right)$ under all test DAGs. At the same time, since the implementation complexity of the E_ESS_par function scales up exponentially as $\mathcal{O}\left(3^{\left(V-2\right)}\right)$ with the size V of the considered DAG, the execution of E_ESS_par under DAG3 is out of discussion, while the resulting execution complexities are of the order of $\mathcal{O}\left(3^{10}\right)$ and $\mathcal{O}\left(3^{13}\right)$ under DAG1 and DAG2, respectively. As a consequence, the (profiled) execution times of E_ESS_par are about five- and 133-times larger than the corresponding ones of GeneticTA_par.

Regarding the analysis of the first bars of Figure 10 and Figure 11, the following three main insights may be drawn. First, under the (small-sized) DAG1, GeneticTA_par and E_ESS_par return the same optimal task and resource allocation patterns at $T_{DAG}=0.3,1.5,3.0$ s. Furthermore, the energy ratio: $E_{GeneticTA\_par}/E_{E\_ESS\_par}$ is limited up to $1.07$ (e.g., $7\%$) in the remaining two cases. Second, under the (medium-sized) DAG2, GeneticTA_par and E_ESS_par share the same energy consumption at $T_{DAG}=0.3$, while the corresponding energy ratio: $E_{GeneticTA\_par}/E_{E\_ESS\_par}$ is no larger than $1.26$ in the other cases.

Overall, the carried out analysis supports the conclusion that the tested implementation of GeneticTA_par is capable of attaining, indeed, good performance-vs.-complexity tradeoffs.

Three tiered-vs.-single and two-tiered execution platforms: A potential drawback of multi-tiered distributed computing platforms is that the number of involved communication links tends to grow with the number of inter-connected tiers, and this may lead, in turn, to an increment of the network component of the overall consumed energy. Hence, the goal of this subsection is to give insight into the following (rather basic) question: What is the net tradeoff among the reduction of the computing energy arising from the utilization of multi-tiered computing nodes and the corresponding increment of the network energy needed for their inter-connection? In order to address this question, the VirtFogSim toolbox makes available the OM_S, OF_S, and OC_S strategies. By design, they utilize only the Mobile device, the two-tiered Fog-Mobile platform, and the two-tier Cloud-Mobile platform for the execution of the application DAGs. Furthermore, all these strategies perform dynamic scaling of the utilized computing frequencies and wireless network bandwidths (see Section 4.2).

Hence, since GeneticTA_par exploits, by design, all the Mobile, Fog, and Cloud nodes of Figure 1 for task placement, a comparative analysis of the energy ratios in the last three bars of Figure 10, Figure 11 and Figure 12 provides a direct response to the above question by providing the following three main insights.

First, the energy ratio: $E_{OM\_S}/E_{GeneticTA\_par}$ takes values over the intervals: 2.84–3.84, 3.3–3.9, and 4.8–5.1, under DAG1, DAG2, and DAG3, respectively. The corresponding intervals of the energy ratios: $E_{OF\_S}/E_{GeneticTA\_par}$ and $E_{OC\_S}/E_{GeneticTA\_par}$ are: 1.53–1.88, 2.0–2.73, 1.23–1.32, and 1.35–1.52, 1.99–2.59, 1.35–1.63, respectively. Hence, in the carried out simulations, the minimum (e.g., worst-case) energy-savings guaranteed by the three-tier Mobile-Fog-Cloud platform over the Mobile, Mobile-Fog, and Mobile-Cloud ones are: 184%, 23% and 35%, respectively.

Second, at fixed $T_{DAG}$, the average energy savings offered by the Mobile-Fog-Cloud platform over the considered benchmark ones tend to be somewhat more substantial under DAG2. Roughly speaking, this is due to the fact that DAG2 is the parallel combination of three sub-DAGs, whose energy-saving executions “naturally” lead to the simultaneous utilization of all the available Mobile, Fog, and Cloud computing nodes (see Figure 5b).

Third, regarding the comparison of the relative energy performances of the OM_S, OF_S, and OC_S strategies, we may conclude that: (i) the OM_S strategy is the most energy-consuming under all the tested cases; and (ii) in general, the OC_S (resp., OF_S) strategy is more energy saving than the OF_S (resp., OC_S) one under the more regular (resp., more random) DAG1 and DAG2 (resp. DAG3).

As a concluding remark, we point out that the reported numerical results give practical evidence of the support offered by the proposed VirtFogSim toolbox in the analysis and optimization of the multiple performance aspects of the multi-tiered technological platform of Figure 1.

6.5. Scalability of the Simulation Time of the VirtFogSim Toolbox

The two-fold goal of this section is to test the scalability of the execution times of the VirtFogSim toolbox versus both the number of cores available for its execution and the computational complexity of the run task allocation strategies. For this purpose, DAG1 at $T_{DAG}=0.3$ has been considered, and then, both the execution times and the total input/output volumes (in MByte) of data transferred to/from the available pool of working cores have been numerically profiled through the tictoc and ticByte–tocByte commands available in the Parallel Toolbox of MATLAB. The attained results are reported by the bar-plots of Figure 13 and Figure 14. Each reported result has been obtained by averaging over 10 independent runs of the (complexity-tunable) GeneticTA_par task allocation strategy.

An examination of the execution times of Figure 13 points out that the speed-up factors attained at $n_{core}=2$, 4, 6, 8, and 10 over the benchmark case of $n_{core}=1$ are around 1.91, 3.51, 4.98, 5.94, and 6.60, respectively. Hence, these speed-factors scale in a (quasi-) linear way for values of $n_{core}$ limited up to 3–4, while the scaling becomes more and more sub-linear for larger values of $n_{core}$. The reason behind this scaling behavior of the execution times is provided by a comparative examination of the corresponding total input/output data traffic generated by the carried out parallel executions. In this regard, the traffic bars of Figure 13 unveil that both the generated volumes of data (which is the most) and the corresponding growing rate increase for increasing values of $n_{core}$. The resulting net effect is that the experienced execution times become more and more dominated by the corresponding inter-core communication times, so that the corresponding relative decrements in the execution times tend to decrease for large values of $n_{core}$.

This conclusion is further corroborated by the bar plots of Figure 14. They report the profiled execution times and generated inter-core traffic volumes obtained by running GeneticTA_par for increasing values of the population size $PS$ at fixed $n_{core}=10$. In fact, since the computational complexity of GeneticTA_par scales in a linear way with $PS$ (see Table 4), it could be expected that the same scaling behavior would be also exhibited by the corresponding profiled execution times. However, an examination of the execution-time bars of Figure 14 unveils that the slow-down factors attained at $PS=100$, 150, 200, 250, and 300 against the benchmark case of $PS=50$ do not scale, indeed, linearly, and are around 1.76, 2.38, 3.16, 3.40, and 4.53, respectively. Even in this case, the reason is the nonlinear growing behaviors of the corresponding input/output volumes of inter-core data (see the bars of data traffic in Figure 14).

In this regard, we point out that similar trends are also exhibited by the (profiled) execution times of the other parallel functions: ES_S_par and O_TAS_par implemented by the VirtFogSim package.

To conclude this section, we underline that the RAM used by VirtFogSim is essentially that allocated by the MATLAB environment. Additional memory space is used to store all the variables involved by the chosen strategies. This additional memory space depends mainly on the DAG size V and the population size $PS$, but anyway, it is limited up to the interval 50–150 MB for the tested cases.

7. Conclusions and Future Developments

The actual development of delay-sensitive 5G-supported mobile applications demands for the dynamic profiling and energy optimization of emerging multi-tiered Fog-Cloud virtualized ecosystems. Since field-trials and deployment of test-beds are expensive and could not guarantee repeatable results, the development of customizable simulation toolboxes is welcome. VirtFogSim is compliant with this expectation by developing a new software environment that accounts for the main system parameters featuring the computing and network aspects of Mobile-Fog-Cloud technological platforms. The core engine of the VirtFogSim toolbox allows the optimization, simulation, and tracking of a number of heuristic/meta-heuristic/exhaustive search-based policies for the energy-saving dynamic allocation of tasks and computing-networking resources needed for the delay-constrained execution of mobile applications described by general DAGs. The GUI equipping the VirtFogSim package allows a user-friendly rendering of the simulated data under a number of easy-to-understand graphic formats.

The current version of the VirtFogSim package being the first open-access release, it is amenable to further extensions along three main directions.

According to the underlying 5G paradigm, additional customizable primitives for the profiling and performance simulation of massive MIMO wireless access technologies could provide a first valuable extension [35,36,37]. Furthermore, the current version of the simulator relies on purely reactive formal methods for the dynamic optimization of the needed computing-plus-networking resources. Including pro-active optimization tools that are capable of exploiting mobility-triggered resource forecasting could be second extension of potential interest [38]. Finally, software functionalities for the support and performance evaluation of (possibly, mobility and/or failure-triggered) energy and delay-efficient task migration strategies could represent a third extension of practical interest.

8. Availability of the VirtFogSim Package

The complete software package of the VirtFogSim simulator and the corresponding User Guide may be downloaded for free on the GitHub repository site at: https://github.com/mscarpiniti/VirtFogSim. In addition, it can be downloaded from the authors’ web pages, specifically at: http://enzobaccarelli.site.uniroma1.it, by accessing the section: Downloadable packages.