Simulating Fog and Edge Computing Scenarios: An Overview and Research Challenges

Abstract

The fourth industrial revolution heralds a paradigm shift in how people, processes, things, data and networks communicate and connect with each other. Conventional computing infrastructures are struggling to satisfy dramatic growth in demand from a deluge of connected heterogeneous end points located at the edge of networks while, at the same time, meeting quality of service levels. The complexity of computing at the edge makes it increasingly difficult for infrastructure providers to plan for and provision resources to meet this demand. While simulation frameworks are used extensively in the modelling of cloud computing environments in order to test and validate technical solutions, they are at a nascent stage of development and adoption for fog and edge computing. This paper provides an overview of challenges posed by fog and edge computing in relation to simulation.

1. Introduction

It is commonly accepted that society is on the brink of what is termed the fourth industrial revolution (4IR), whereby cyber-physical systems will disrupt and transform entire industries and associated systems of production, management, and governance [1]. Schwab [1] posits that this revolution differs from the previous three revolutions because it is not based on breakthroughs in technology but rather communication and connectivity. At its core, 4IR is not a new concept and is couched in the concept of a networked society whose social structures and activities, to a greater or lesser extent, are organised around digital information networks that connect people, processes, things, data and networks [2,3]. As such, 4IR, Industry 4.0, and the Internet of Things (IoT) are often discussed in juxtaposition. 4IR is disruptive because it has the potential to radically redefine industrial performance trajectories and how, who, and where value is created, delivered and captured. Current exuberance about 4IR, Industry 4.0 and IoT is driven by both the advances and widespread adoption of several underlying technologies namely cloud computing, ubiquitous sensing, and mobile technologies, connected across a cloud-to-things (C2T) continuum. In conventional cloud computing, processing and storage typically takes place within the boundaries of a cloud and its underlying infrastructure. It is not designed to cater for the scale of geographically dispersed, heterogeneous end points and low latency required for many 4IR, Industry 4.0 and IoT use cases. As such, conventional paradigms of computing need to be rethought to cater for the scale of data processing and storage needed to support the requirements of 4IR, Industry 4.0 and IoT to function in a distributed, coordinated way at minimum latency [4].

Fog and edge computing are two relatively new paradigms of computing that have been proposed to address these challenges. NIST [4] recently defined fog computing as:

… a horizontal, physical or virtual resource paradigm that resides between smart end-devices and traditional cloud or data centers. This paradigm supports vertically-isolated, latency-sensitive applications by providing ubiquitous, scalable, layered, federated, and distributed computing, storage, and network connectivity.

Edge computing, in contrast, is local computing at the network layer encompassing the smart end-devices and their users [4]. As such, edge computing in its narrow definition excludes both fog and cloud computing [5], as shown in Figure 1. Fog and edge computing provides significant advantages for processing data closer to the source and thus mitigate latency issues, lower costs of data transmission, and reduce network congestion [6,7].

Fog and edge computing provide new and significant architecture design challenges for all of those involved in the 4IR/IoT chain of service provision. Fog and edge computing use cases vary by the extent and degree of (i) contextual location awareness and low latency, (ii) geographic distribution, (iii) scale and coordination of end-point networks, (iv) heterogeneity, interoperability and functionality of end points, (v) real-time vs batch processing, (vi) mobility of end points, and (vii) interplay between the edge, the fog and the cloud layers [4]. These factors determine the extent to which quality of service levels can be met, performance bottlenecks avoided, energy consumption optimised, under-use reduced, and costs managed efficiently and effectively [9]. However, the complexity and scale of these use cases are orders of magnitude greater than conventional enterprise and cloud computing scenarios. Successful deployment of 4IR/IoT use cases requires optimal placement of computation and storage along the C2T continuum on a case by case basis and new approaches to test resource placement and management strategies at scale.

Like cloud computing, researchers and developers seeking to test resource placement and management strategies for fog and edge computing face several significant challenges. Firstly, commercial service providers typically do not give the necessary infrastructure access or control to third parties [10]. Secondly, establishing a test bed with a high degree of verisimilitude is both complex, costly, resource and time-intensive. Thirdly, from a research perspective, the use of commercial third party services and proprietary test beds limit the extent to which experiments can be validated and results reproduced.

Cloud computing researchers have overcome these issues through a range of modelling techniques. Petri Nets [11], Markov Chains [12], Fault Tree (FT), Reliability Graphs (RG) and Reliability Block Diagrams (RBD) [13] have all been used as analytical modelling techniques in distributed system research. A Petri Net is a mathematical modelling language that is commonly used to describe dynamic and parallel system behaviour in order to analyse the performance and availability metrics of systems capable of being clearly described using mathematical definitions [14]. In Markov chains, system models are defined as a sequence of stochastic events that can be used for estimating system behaviour using complex probability distributions. The increasing scale, complexity and heterogeneity of C2T systems renders the use of traditional mathematical modelling techniques difficult to utilise, leading to adoption of other approaches such as simulation.

More recently, the use of simulation frameworks [15,16] has come to greater prominence in cloud computing research. Service configuration and resource placement and management strategies can be simulated prior to infrastructure deployment, performance can be optimised, technical and commercial hypotheses tested, and research results validated and reproduced in a low cost, low risk and often time-sensitive manner [10]. Research on fog and edge computing is still at a relatively early stage of development. Unsurprisingly, research on simulation frameworks to support fog and edge computing and the multitude of use cases that 4IR and IoT represent is lagging.

While there are several research surveys and articles exploring the challenges of fog computing (e.g., [17,18]), edge computing (e.g., [19,20]), simulation frameworks for cloud computing (e.g., [15,16]) and IoT data analytics (e.g., [21]), there is a paucity of publications addressing the challenges in modelling and simulating fog and edge computing specifically. This paper complements existing works on fog and edge computing by presenting the challenges and design considerations for simulation frameworks of fog and edge infrastructures. Our aim is to support both computing and simulation researchers entering this field in determining the requirements of the simulation platform that they would need to develop or employ to evaluate their technical proposals.

The remainder of this paper is organised as follows. The next section discusses challenges of fog and edge computing modelling and simulation. This is followed by a review of existing simulation frameworks for fog and edge computing and a gap analysis against support for four key criteria—(a) infrastructure level modelling, (b) application-level modelling, (c) resource management modelling, (d) mobility, and (e) scalability. Our analysis suggest significant gaps in relation to requirements to model mobility and scalability, after which the article concludes.

2. Fog and Edge Computing: Modelling and Simulation Challenges

The increasing ubiquity of mobile technologies and low-cost connected sensors has resulted in a deluge of computational and networking end points at several orders of magnitude than previous decades. Conventional computing infrastructures, including cloud computing, leverage geographical centralised data centres using relatively homogeneous commoditized hardware. Such infrastructures were not designed to cater for the processing, storage and data generated by billions of distributed end points operating in often dynamic environments with intermittent network connectivity. Unsurprisingly, service providers have faced unprecedented challenges planning and managing for greater demands while meeting minimum service levels. Fog computing has emerged to complement cloud computing. As discussed earlier, fog computing is located between smart end-devices at the edge of networks and traditional cloud or data centres [4]. It plays an important role in reducing network congestion and facilitating location awareness, mobility support, real-time interactions, scalability and interoperability [8,22]. In contrast, edge computing, in its purest sense, defined by the exclusion of cloud and fog, and is limited to a small number of peripheral network layers [4]. Both fog and edge computing present significant challenges for service providers and researchers including application architecture design and deployment, infrastructure and network management, mobility, resource management, and scalability.

2.1. Application Level Modelling

There is an infinite range of potential applications for fog and edge computing, ranging from simple-IoT based sensor monitoring to the complex data processing systems inherent in Industry 4.0, e-health, smart cities etc. Consequently, underlying applications vary in their needs based on the degree of (i) contextual location awareness and low latency, (ii) geographic distribution, (iii) scale and coordination of end-point networks, (iv) heterogeneity, interoperability and functionality of end points, (v) real-time vs batch processing, (vi) mobility of end points, and (vii) interplay between the edge, the fog and the cloud layers [4,22]. Provisioning for such heterogeneity requires significant planning upfront and ongoing optimisation throughout the C2T continuum including application design.

The majority of current fog and edge services that support applications can be further divided into three main categories - Content Distribution Networks (CDN), IoT and Virtual Network Functions (VNF). While all three use the same infrastructure, the functional aspects of each type of service are fundamentally different. CDN services focus mostly on static content replication and distribution across multiple locations. IoT services are used to offload data processing and storage from sensors to edge locations selectively pushing some of the data up the network stack to the cloud. VNFs are chains of network functions that handle mobile network protocol traffic (e.g., LTE stack) or provide network traffic filtering and routing functions, such as enterprise, firewall and VPN services. Law [23] suggests “a simulation model should always be developed for a particular set of objectives. In fact, a model that is valid for one objective may not be for another.” Modelling all of the applications deployed within a fog/edge network can be beneficial for infrastructure providers but constructing a simulation solution that can efficiently handle a set of such broad objectives is a challenge and needs careful consideration.

2.2. Infrastructure and Network Level Modelling

Due to cloud communication dependency and large volumes of data generated by fog and edge applications, network connectivity and capacity can be be a significant limitation, especially in the case of real-time delay-sensitive applications. This is particularly the case at the mobile edge. Mobility management is critical in mobile edge computing (MEC), especially in highly dynamic environments. To manage demand in MEC scenarios, massive numbers of small cells are deployed. In this scenario, the user range can be very limited and therefore handovers are more frequent, resulting in a heavy burden on the network [24].

Fog and edge devices make use of a wide variety of communication technologies, ranging from traditional low cost protocols, such as IEEE 802.11 to energy efficient protocols, such as IEEE 802.15.4 (ZigBee/6LoWPAN) and Bluetooth Low Energy (LE). Each of these technologies has an impact on the end point performance directly whether data processing, service time, data transfer delay etc. Arriving at the optimal network access technology, typically involves a trade-off between performance and cost and is often outside of the control of the service provider [17].

Fog and edge system models can extend to thousands of distributed site locations creating a network of resources spanning multiple countries. Each site can be comprised of computing and network equipment hosting multiple applications that can be accessed by edge service users. Creating such a model by hand, even at a higher abstraction level, is no longer practical from a time and effort perspective. To solve the problem an automated approach is required for model building. Integration with a monitoring data collection system can partially address the challenge by taking snap-shot of an existing infrastructure state [25]. However, in order to build meaningful system behaviour models the monitoring data has to undergo additional processing to extract behaviour trends of workload and application resource demands. Such a process brings big data management and processing challenges into play that require further development within the scope of the simulation domain.

2.3. Mobility

Recent studies focus on the emergence of 5G networks and the interaction between these networks and fog and edge computing. 5G networks offer network improvements through optimization of mobile resource usage, large data pre-processing, and context-aware services (using cell load, user location, and allocated bandwidth as information) [20]. Notwithstanding these improvements, as each fog and edge application may have different latency requirements and may generate different types of data and network traffic, a mechanism may be required to differentiate delay-sensitive flows such as network slicing [26].

Modelling user mobility aspects requires the implementation of geographic awareness logic, for example calculation of the nearest mobile access point based on user coordinates at each simulation timestep. Furthermore, availability and access to real-world data on end user mobility is problematic both legally and technically. Additional calculations further increase the complexity and computational resource demands of a given simulation platform. Intelligent model generators are one solution for creating fog and edge infrastructure workload models based on 3rd party socio-demographic and geographic data that can be used for simulation purposes [27].

2.4. Resource Management

The majority of 4IR, Industry 4.0 and IOT scenarios assume the generation, capture and analysis of data in volumes, variety and velocities orders of magnitude greater than before. This data may include useful information if such information can be identified [28]. For example, a basic connected vehicle system can generate tens of megabytes of data per second [5]. To provision infrastructure efficiently and effectively requires several key decisions, not least how the data will be collected, where and how data will be processed (edge, fog or cloud), and how often the data should be sent to the cloud for long-term storage or further analysis. There are two competing pressures informing these decisions - use of infrastructure and end user quality of service.

Complex Event Processing (CEP) systems are increasingly cited for processing and analysing high volumes of data and detecting events of interest when they occur [28,29]. However, the CEP task can be time-consuming, and commonly fog or edge devices present computational and storage capacity limitations, when compared to cloud capacity and capability. As such, caching is used widely to bring storage functionality to network edges with lower latency, less excessive bandwidth consumption, and reduced streaming times [19,30,31]. This is particularly the case for content distribution use cases such as IP video, forecast to account for a significant portion of all IP traffic in the coming years [32]. Wang et al. [19] summarise the main challenges in content caching in edge networks as caching placement, content popularity, caching policies and algorithms, and mobility awareness.

Understanding data load generation and its propagation through a given system is a worthwhile approach for deciding on (optimal) resource placements. Data load prediction has been presented as a solution for proactive system remediation [33]. In this case, historical (big) data stored at cloud and live data collected in the fog and edge devices are used to feed models and predict important metrics, such as resource usage and content popularity distribution [34]. Machine learning techniques have also been widely used to solve this problem. For instance, Zeydan et al. [35] use machine learning to predict the spatio-temporal user behaviour for proactive caching decisions with a goal of satisfying user demand by delivering low latency and higher QoE.

Noor et al. [36] identify energy efficiency as one of the significant challenges for mobile cloud resource management. For example, while offloading data processing to the cloud can reduce device battery consumption, it increases network bandwidth usage and power consumption, a significant contributor to rising energy consumption [37,38]. Unsurprisingly, energy efficiency is a major focus of cloud, fog and edge computing research including the optimisation of resource allocations under energy, performance, and QoS constraints [39,40,41]. Fog computing can introduce management flexibility by providing more options for data processing within the distributed network hierarchy. From the cloud perspective, deciding whether to cache or process the data offload is one way to alleviate network congestion and reduce data transfer costs; from an edge device perspective, offloading some computing tasks could enhance the service performance and be more energy efficient [42,43], since some edge devices are very energy constrained (and, at the same time, hungry energy consumers). As such, the performance profile of the edge device should be taken in to consideration, especially for real-time sensitive applications, such as vehicle-to-vehicle communications, vehicle-to-roadside communications and real-time financial trading applications that may require latencies below tens of milliseconds [5]. Intensive benchmark performance experiments are typically required to decide the best fog-to-edge configuration considering a range of factors including computational and storage capacity, battery life, mobility, communication interface etc. The importance of system availability for such devices can be an important consideration particularly in use cases where data loss or service outage can result in adverse outcomes for end users, for example, in health care monitoring systems.

The runtime, programmability, and interoperability of edge devices often differ due their heterogeneity resulting in data offloading issues [42]. The European Telecommunications Standards Institute (ETSI), the OpenFog Consortium, and others are seeking to address standardisation for multi-access edge computing [44] however such initiatives are at an early stage of development and face uphill challenges against the onslaught of new connected end points being introduced.

The simulation approach has proven to be a worthwhile endeavour in testing resource allocation and management in cloud. In their study, Stier et al. [45] presents direct integration between the simulation and optimisation framework which was implemented in order to test available resource management algorithms that can be directly used within a real system. In addition, simulation can also be used as part of resource management algorithms to narrow search space for the optimal solutions in an optimisation technique known as simulated annealing (SA). SA is an optimisation algorithm that uses local search approach of moving around the neighbouring values in a defined search space until the optimal solution is found [46]. Even though the simulated annealing technique goes outside the traditional scope of discrete event simulation (DES), the general annealing approach can be useful in testing different parameter variations within edge computing such as virtual Content Delivery Network (vCDN) deployments [47] or exploring infrastructure provisioning options [48]. Moving to the fog and edge domain the need for simulating resource management approaches remains one of the main simulation analysis features [33].

2.5. Scalability

The choice of a simulation tool depends significantly on the type of applications. This fact also dictates the granularity of the simulation. For example, macroscopic phenomena, such as routing strategies, can be studied by packet-level, using the DES approach. Notwithstanding this, a very accurate simulation might substantially hinder performance leading to similar results as other faster performing methods. Another key point when considering a simulation framework is the generality of the range of phenomena and applications that can be simulated. More general simulation frameworks are usually not focused on specific characteristics, but rather on a large number of parameters that may not be required by the user and may be very complex to setup and operate. These frameworks tend to cover a wide variety of applications and phenomena. On the other hand, dedicated simulation solutions are usually easier to use, tailored and optimised to specific applications and their complexity. However, dedicated solutions are not easily adaptable to other applications, without significant development effort. In our analysis of extant simulation frameworks for fog and edge computing, many are extensions of CloudSim and suffer from its limitations in terms of scale and focus. Others focus on specific use case scenarios. These present short term limitations for fog and cloud computing researchers but also opportunities for simulation research.

Experimenting with large scale systems requires compute resources to be available for a simulation framework to use. DES is the most popular approach used in cloud computing, as reflected in the above analysis. However, the sequential nature of the event queue is notoriously difficult to parallelize as each event can change the state of the system. Therefore, if an event is processed out of order the calculation may be incorrect. Having said that, where one can make clear divisions within the model, simulation events can be processed in independent clusters increasing the degree of parallelism. For example Varga and Sekercioglu [49] discuss a parallel discrete event simulation (PDES) approach that is capable of distribute simulation over multiple processors and machines, also avoiding memory bottlenecks by dividing the model across machines. Another example is Cloud2Sim, an extension of popular CloudSim [50] framework using Hazelcast and Infinispan in memory distributed data stores [51]. As seen in

Table 1. Fog and Edge Simulator Tools: Comparative Study.

Attributes	FogNetSim++	iFogSim	FogTorchII	EdgeCloudSim	IOTSim	EmuFog	Fogbed
Computing paradigm (target system)	Fog computing (general)	Fog computing (general)	Fog computing (general)	Edge computing (IoT)	Edge computing (IoT)	Fog computing (general)	Fog computing (general)
Infrastructure and network level modelling	Distributed data centres Sensors Fog nodes Broker Network links Delay Handovers Bandwidth	Cloud data centres Sensors Actuators Fog devices Network links Delay Network usage Energy consumption	Latency Bandwidth	Cloud data centres Network links Edge servers WLAN and LAN delay Bandwidth	Cloud data centre Latency Bandwidth	Network links Fog nodes Routers	Virtual nodes Switches Instance API Network links
Application level modelling	Fog network	Data stream Stream-processing	Fog applications	Mobile edge	IoT	Fog	Fog network
Resource management modelling	Resource consumption (RAM and CPU)	Resource consumption Power consumption Allocation policies	Resource consumption (RAM and storage)	Resource consumption (RAM and CPU) Failure due to mobility	Resource consumption (RAM, CPU and storage)	Workload	Resource consumption (RAM and CPU) Bandwidth Workload
Mobility	Yes	No	No	Yes	No	No	No
Scalability	Yes	No	No	No	Yes (MapReduce)	No	No

parallel DES execution is not widely adopted in newly released edge simulation tools limiting their application range.

The Discrete Time Simulation (DTS) approach can be used to attempt to combat such parallelisation difficulties associated with DES. DTS uses the concept of time-step to update the state of the system components, avoiding the need for pre-computation and storage of future events. This approach presents a significant reduction of the simulator’s memory requirements and enhances performance while enabling parallel processing [52], and, along with reduced memory requirements, provides a mechanism for the simulation of very large networks. The state of all components involved in a simulation e.g., sites, nodes, VMs, etc, can be updated in parallel, since, there are no dependencies between components thus enhancing scalability. The change of state of the constituent components is only affected by input requests. This approach substantially simplifies the design and incorporation of advanced power consumption models and strategies for path formation on networks. Furthermore, the granularity of the simulation is controlled by the choice of timestep - choosing a smaller timestep results in a very large number of timesteps, potentially increasing the accuracy of the simulation, while hindering performance. A large timestep typically results in the undersampling of the studied phenomenon neglecting transient phenomena that might substantially affect the result of the simulation. DTS simulation has been used in the context of very large scale simulation of traditional and self* based cloud environments in supercomputing environments [53,54].

DES and DTS approaches have both strong and weak points when compared to one another. DES is generally considered to be a suitable tool when applied to a problem requiring a more granular modelling approach which is more difficult to scale, whereas DTS typically enables the modelling of large scale systems with relatively less effort, but with the possibility of a higher degree of inaccuracy [55]. Both paradigms can be applied to the fog and edge domain depending on the experimentation objectives.

In terms of accuracy, there are challenges in relation to the validation of large scale systems as simulation represents an abstracted model of a real environment, hence it is imperative that the level of abstraction does not impede simulation result accuracy. The simulation validation process ensures that simulation experiments produce reliable estimations of system behaviour. Existing approaches suggest to validate simulation models with domain experts to ensure model face validity attributes and behaviour constraints [56]. Furthermore, simulation results should be validated by comparing simulated results with monitored real system data e.g., by visually comparing simulated and monitored results plotted side by side [56] or statistically comparing data distribution e.g., using a t-test approach [57]. Validating simulation models and results is not a new challenge, however applying validation techniques to fog and edge computing simulations can be challenging. Firstly, it is difficult to inspect the target environment due to size and complexity. Secondly, lack of access to real data impacts validation by comparison. Automatic or a semi-automatic validation methodologies capable of processing high volumes of data can potentially resolve or alleviate validation challenges by checking model data for consistency and result anomalies.

3. Fog and Edge Modelling and Simulation Tools

According to Dastjerdi and Buyya [58], in order to enable real-time analytics in fog and edge computing at the software-level, we must be concerned about different resource management and scheduling techniques including resource distribution, load balancing, migration, and consolidation. At the physical layer, fog and edge systems have many additional requirements that need to be addressed, such as network connectivity and capacity. This scale and complexity of C2T systems makes the use of realistic prototypes unfeasible. Similarly, commercial service providers typically do not give the necessary infrastructure access or control to third parties to test aforementioned techniques [10] and constructing a test bed with a high degree of verisimilitude is both complex, costly, resource and time-intensive. To overcome these issues, simulation frameworks provide a relatively low cost means to understand and evaluate fog and edge systems and eliminate ineffective policies and strategies [59].

Simulation has been used extensively to simulate traditional network infrastructures, such as the mainstream Wireless Sensor Networks (WSNs). Some examples of these simulators are NS-2, TOSSIM, EmStar, OMNeT++, J-Sim, ATEMU, and Avrora. These simulators are universally used to develop and test network protocols, especially in the initial design stage. They were not designed with fog and edge computing environments in mind; as such, they are outside the scope of this paper. We redirect the reader to a detailed survey by [60] for further information on these simulators.

While there are a wide range of simulators for cloud computing, there are relatively few that can be used to simulate fog and edge computing scenarios. Next, we briefly describe a selection of prominent simulators used for fog and edge modelling and compare them in qualitative terms.

FogNetSim++ [61] is a fog simulator tool that provides users with detailed configuration options to simulate a large fog network. It is designed on the top of OMNeT++ [62] which is an open source tool that provides an extensive library to simulate network characteristics using discrete event simulation. FogNetSim++ enables researchers to incorporate customised mobility models and fog node scheduling algorithms as well as managing handover mechanisms. A traffic management system is evaluated to demonstrate the scalability and effectiveness of the FogNetSim++ simulator in terms of CPU and memory usage. The authors provide a benchmark of network parameters, such as execution delay, packet error rate, handovers, and latency. However, FogNetSim++ does not yet support VM migration among fog nodes.

iFogSim [63] is a fog computing simulation toolkit that allows users to simulate fog computing infrastructures and execute simulated applications in order to measure performance in terms of latency, energy consumption and network usage. iFogSim is based and implemented on CloudSim [50]. iFogSim enables the modelling and simulation of fog computing environments for evaluating resource-management and scheduling policies. It measures performance metrics and simulates edge devices, cloud data-centres, sensors, network links, data streams, and stream-processing applications. In addition, iFogSim integrates simulated services for power monitoring and resource management at two separate levels i.e., application placement and the application scheduling. Two application module placement strategies are packaged to support multiple deployment scenarios, namely, (a) cloud-only placement, where all applications modules run in data centres and (b) edge-ward placement, where application modules run on fog nodes close to edge devices [59]. Furthermore, extensions are available to support the design of data placement strategies according to specific objectives such as minimisation of service latency, network congestion, and energy consumption [64]. It is also worth noting that as the fog computing paradigm has many similarities to cloud computing, CloudSim can also be used as a standalone application to implement many features of fog computing. iFogSim is not without its limitations. While it enables the definition of the location of devices getting service from the fog servers, this information is static and not updated by any mobility model. In addition, while being based on Cloudsim provides advantages, iFogSim is limited to DES and its scalability is limited.

Both EdgeCloudSim and IOTsim, like iFogSim, are also based on CloudSim. EdgeCloudSim is specifically design to evaluate the computational and networking needs of edge computing. Unlike iFogSim, EdgeCloudSim supports mobility. In fact, it provides the mobility model, network link model, and edge server model to evaluate the various facets of edge computing. In addition to its simulation capabilities, EdgeCloudSim is relatively user-friendly providing a mechanism to obtain the configuration of devices and applications from the XML files instead of defining them programmatically. IOTSim was designed to simulate edge computing environments where large data volumes are sent to a big data processing system by the IoT application [65]. As such, it adds a storage and the big data processing layer in to CloudSim. In the storage layer, the network and storage delays are simulated for IoT applications. The big data processing layer simulates MapReduce to support the batch-oriented data processing paradigm. Both EdgeCloudSim and IOTsim inherit the same scalability and DES limitations as iFogSim.

Brogi et al. [66] recently presented a prototype simulator, FogTorchII, that extends their previous work, FogTorch) [67]. Primarily designed to support application deployment in the fog, FogTorchII is an open source simulator developed in Java. It is capable of evaluating fog computing infrastructure deployments, it models software capabilities (operating system, programming languages, frameworks, etc.), hardware capabilities (CPU cores, RAM and storage), and QoS attributes including latency and bandwidth. FogTorchII uses Monte Carlo simulations to implement variations in communications links used as inputs. The final output consists of the aggregated results in terms of QoS-assurance and fog resource consumption through an indicator of the percentage of consumed RAM and storage. An acknowledged and major limitation of FogTorchII is scalability, an issue that Brogi et al. [66] hope to address by exploiting heuristics to reduce the search space [66].

Simulations make several simplifications that may not always hold true, especially with an infrastructure as dynamic as fog and edge computing. As such, several emulation frameworks were developed to address this limitation. EmuFog is an extensible emulation framework tailored for fog computing scenarios [68]. EmuFog enables the design of fog computing infrastructures ab initio and the emulation of real large scale applications and workloads which allows developers to implement and evaluate their behaviour as well as the induced workload in the network topology. The implementation process in EmuFog consists of four stages:

• A network topology is either generated or loaded from a file, supporting thus real-world topology datasets.
• The network topology is converted in an undirected graph, where nodes represent network devices (e.g., routers) and links correspond to the connections between them.
• The edge devices are determined and the fog nodes are placed according to a placement policy. Users are able to define the computational capabilities of fog nodes as well as the number of clients expected to be served by each node.
• Fog nodes are emulated from the network emulated environment, while the applications in any individual fog node are running under Docker containers.

Despite the usefulness of the EmuFog, the framework does not support mobility both for clients and fog nodes. Furthermore, EmuFog does not support hierarchical fog infrastructures.

Fogbed [69] is another emulator which extends the network emulator Mininet [70] framework to allow the use of Docker containers as virtual nodes. It provides capabilities to build cloud and fog testbeds. The Fogbed API enables adding, connecting and removing containers dynamically from the network topology. These features allow for the emulation of real-world cloud and fog infrastructures in which compute instances can be started and terminated at any point in time. Also, it is possible to change the run-time resource limitations for a container, such as CPU time and memory available. However, Fogbed does not yet support key aspects of fog computing including security, fault tolerance, scalability and reliability management.

Table 1 summarises the above simulator tools against six key qualitative attributes: (i) computing paradigm (target system), (ii) infrastructure-level modelling, (iii) application-level modelling, (iv) resource management modelling, (v) mobility, and (vi) scalability.

In summary, despite an increase interest on fog and edge computing, research on suitable simulation frameworks to support the requirements of this domain is lagging (see Table 1). Most of the existing simulation tools, albeit a small number, place a greater emphasis on fog computing. They have significant limitations in scalability and mobility support. All the existing simulators use DES at their core and the dependence on CloudSim for three of the simulators places an additional limitation to them particularly in terms of scalability. Therefore, there is an urgent need for simulation tools with greater coverage of characteristics of fog and edge computing.

Ficco et al. [71] argue that purely simulated environments and real testbeds are not sufficiently representative of real world scenarios and/or are unacceptably expensive. As such, they suggest that a hybrid pseudo-dynamic testing approach may increase verisimilitude by simulating a portion of the experimental scenario, while either emulating the edge and fog nodes under test or executing them in a real environment.

4. Conclusions

The emergence of the fourth industrial revolution and the Internet of Things is quickly becoming a reality. For the developer and research community, the availability of a means to test, validate, compare and reproduce technical proposals efficiently and cost-effectively is central to commercialisation and the scientific method. Like cloud computing, public clouds and test beds do not provide sufficient control of resources and infrastructure to validate technical solutions for fog and edge computing at the appropriate level of granularity. While modelling and simulation can address these issues, early attempts at simulation frameworks have significant gaps in their capability to model the complexity and specific requirements of fog and edge computing scenarios at the scale facing key stakeholders in the chain of service provision today, let alone the future.

Indeed, many existing fog and edge computing simulators derive from cloud computing simulation frameworks and may be inflected towards the cloud layer rather than the nuances of a multi-layered C2T continuum. This review of existing simulation frameworks and challenges in modelling and simulating fog and edge computing use cases provides a landscape of existing options but also a roadmap for future research in both fog and edge computing and the design of associated simulation frameworks.