Secure and Reliable IoT Networks Using Fog Computing with Software-Deﬁned Networking and Blockchain

: Designing Internet of Things (IoT) applications faces many challenges including security, massive trafﬁc, high availability, high reliability and energy constraints. Recent distributed computing paradigms, such as Fog and multi-access edge computing (MEC), software-deﬁned networking (SDN), network virtualization and blockchain can be exploited in IoT networks, either combined or individually, to overcome the aforementioned challenges while maintaining system performance. In this paper, we present a framework for IoT that employs an edge computing layer of Fog nodes controlled and managed by an SDN network to achieve high reliability and availability for latency-sensitive IoT applications. The SDN network is equipped with distributed controllers and distributed resource constrained OpenFlow switches. Blockchain is used to ensure decentralization in a trustful manner. Additionally, a data ofﬂoading algorithm is developed to allocate various processing and computing tasks to the OpenFlow switches based on their current workload. Moreover, a trafﬁc model is proposed to model and analyze the trafﬁc indifferent parts of the network. The proposed algorithm is evaluated in simulation and in a testbed. Experimental results show that the proposed framework achieves higher efﬁciency in terms of latency and resource utilization


Introduction
The Internet of Thing (IoT) is an adaptive self-configuring network that enables the communication and interaction between physical objects; transforming these objects from being blind to being smart [1,2].Recently, IoT gained significance because of the great impact it has had on all aspects of our life [3].IoT is expected to completely change our life by introducing a wide range of applications in various fields [4].These applications include smart home, smart cities, healthcare, smart vehicles and remote monitoring [5,6].IoT has a high market potential as it comes with big opportunities for various sectors, such as hardware manufacturers, service providers and software developers [7].
IoT represents the third generation of the Internet that is expected to connect billions of heterogeneous devices in a smart way [8].This large number of connected devices puts high constraints on the system structure and design in terms of [9][10][11]:

•
High system reliability,

•
Security and privacy,

•
Integration with other existing communication networks,

•
Traffic load, and Latency constraints for some applications.
To overcome these challenges and achieve higher system efficiency, capable of connecting this huge number of devices, new technologies and communication paradigms can be deployed to support IoT networks.These paradigms include distributed edge computing (e.g., Fog computing), software-defined networking (SDN), network virtualization and blockchain [12].
Edge computing is a new paradigm that aims to provide cloud services and computing capabilities, e.g., storage and processing, at the edge of the access network; one or two hops away from the end user [13].This introduces a way of moving from huge centralized data centres to the distributed cloud units with limited capabilities [14].Deploying edge computing for IoT networks achieves various benefits such as reducing the communication latency, providing a path for data offloading, increasing the spectral efficiency and the introduction of new services [15,16].
Fog computing is a form of edge computing that is suitable for IoT networks [17].It acts as an extension to the cloud computing paradigm to provide processing, computing and storage capabilities.It also introduces other cloud services to the communication nodes in the vicinity of the distributed Fog nodes.Fog computing supports various types of heterogeneous devices that can connect and communicate with the distributed Fog nodes, these devices include sensors, actuators and wireless gateways [18].Fog nodes refer to a computing unit powered by limited computational and storage resources that are deployed to serve connected devices.Fog computing IoT-enabled networks share various and significant advantages that include the improved system privacy and security, the reduction of end-to-end communication latency, higher system reliability and the reduction of traffic overhead and congestion [19,20].
The introduction of Fog computing to IoT presents new challenges.Managing and controlling Fog distributed nodes and synchronizing their operation with an IoT network that is located remotely is a challenge [21].However, deploying an orchestrator or a controller represents an efficient solution; this is the concept behind SDN.SDN physically separates the forwarding plane and the control plane to provide a dynamic network structure [22].The data plane represents the network part that is responsible for forwarding traffic, while the control plane is the part that makes the decision of the traffic.SDN networks generally consist of a centralized or distributed controller and distributed forwarding devices or switches.The controller connects and communicates with the network devices via an open standard interface protocol such as the OpenFlow (OF) protocol [23].SDN is known for its ability to achieve higher system flexibility and scalability.
Blockchain is another main paradigm that was recently deployed for the IoT networks to manage the distributed edge cloud units and work against heterogeneous cybersecurity attacks [24].Deploying the blockchain paradigm for IoT networks enables decentralization in a trustful manner.The introduction of blockchain technology to the IoT networks achieves various vital benefits that include the management of decentralized computing resources, increasing the overall flexibility of the system, achieving higher system security by preventing various cybersecurity threats and attacks, and reducing the cost of the system operation [25,26].Blockchain technology can be described as a peer-to-peer distributed ledger that is used to record all approved events and transactions.Recently, the blockchain paradigm was used to support applications and communication networks (e.g., IoT) beside the crypto-currency systems [27].
In this work, we provide a framework for an IoT-Fog system that integrates SDN and blockchain.This system introduces a distributed edge computing layer of Fog nodes that is deployed between the distributed heterogeneous IoT nodes and the IoT centralized cloud in order to make use of various benefits of the Fog computing.The network employs a distributed SDN controller scheme with the ability to introduce blockchain technology.The SDN network consists of distributed OF switches that are deployed with some limited computing capabilities and an SDN controller that can perform resource provisioning and orchestration in synchronization with Fog orchestration.The SDN network achieves higher system performance in terms of network management, flexibility and latency performances.Moreover, a data offloading algorithm is introduced to organize and manage the offloading scheme.The proposed algorithm makes use of the available resources of the OF switches and, thus, balances the load among the core network switches.Furthermore, a traffic model for modelling and managing IoT traffic among different network parts is introduced.
The main aim of the work is to provide IoT networks with high resource utilization efficiency, high flexibility and the reduction of end-to-end latency.The system is simulated and tested over a testbed to evaluate its performance.In Section 2, the related works are introduced.Section 3 provides the proposed IoT framework details and the data offloading algorithm and traffic model.In Section 4, the simulation and testing are presented and the experimental results are provided and analyzed.

Background and Related Works
There is no doubt that cloud computing and edge computing represent the main base of the fifth-generation cellular network (5G), IoT networks and future smart systems [28,29].There are many studies dedicated to the development and deployment of the edge computing units in communication networks, especially for cellular and IoT networks.Many researchers use the term cloudlet to refer to any secondary, small and limited-capability cloud units [30].There are many other forms of the edge cloud units that include Fog nodes and the micro-cloud units and other forms [15,31].
Fog computing is considered to be the most suitable edge computing platform for IoT networks and applications.Since it was first announced by Cisco as a form of edge computing and an extension of the cellular edge computing [32], researches and studies have been developed to analyse, define, improve and integrate Fog computing.Many works that consider Fog computing for IoT have been conducted; either without the deployment of SDN technology or with SDN.Most of these works are literature reviews; in the following section(s), we consider some of these efforts.
In Reference [33], the authors developed a framework for an IoT network with Fog computing deployment.This work was mainly developed for considering IoT applications from a Fog computing point of view.The authors introduced a distributed data flow mechanism, referred to as DDF, which is programmable.The dataflow programming model was used for building different IoT applications and services.The data algorithm was validated over the open-source flow-based run time and visual programming tool, Node-RED.The testing was introduced just to validate that the architecture and algorithm are suitable.However, no performance metrics were considered.
In Reference [34], the authors developed a hierarchical computing structure for medical applications over IoT networks.The hierarchical structure consists of a centralized cloud and distributed Fog units.The proposed paradigm was introduced to partition and accommodate the machine learning methods used for health care applications over IoT networks.The computation tasks and medical data have been distributed among two computing levels in a partitioning way that increases the system availability.Furthermore, a closed loop management technique was developed that is mainly dependent on the user's condition (e.g., medical parameters).The system was validated in terms of response time and availability.Our proposed work shares the similarity of using Fog paradigm with this work, while this work mainly considers medical applications over the IoT networks and also only considers availability as a performance metric.
In Reference [35], the authors proposed an internet of vehicles (IoV) Fog-based architecture, with SDN deployed.The work is the first study that considers such a structure and combines IoV with the Fog computing and SDN paradigms.The work mainly considers a specific problem, which is the SDN controller placement.The SDN network consists of two levels of controllers; the primary controller and secondary controller.The primary controller is a centralized one that takes the control and management task of the overall system.The secondary controller is a distributed controller dedicated to different regions of the covered area.The two controllers are physically connected.An optimization problem was solved to optimize the geographic placement of the distributed controllers.The work shares the similarity of deploying Fog computing and SDN with an IoT network with our proposed framework, while it considers only the IoV, which is a high mobility application.One main issue of this algorithm is that it has not been evaluated and that the performance was not checked.The authors only introduced a system structure.
In Reference [36], the authors developed a secure IoT system that deploys Fog computing, SDN and blockchain to enhance the security of IoT networks.The system uses SDN and blockchain to secure and control the distributed Fog architecture.Fog services have been allowed at the edge of the access network by the distributed Fog nodes.The system achieves higher latency and security efficiency since bringing computing resources at the edge of the IoT network could secure the core network traffic and minimize the end-to-end latency between IoT devices and the computing unit.The system introduces a novel security method that allows the system to adapt to the threat landscape automatically.This allows system administrators to run as many recommendations at the network edge as needed.The system was evaluated for different security scenarios and attacks.The main focus of this work is security issues, while our proposed framework is mainly concerned with the end-to-end latency performance and resources utilization.Furthermore, our developed SDN network completely differs from the SDN network used in this work since we use a distributed SDN controller with distributed resource powered OF switches.Feeding OF switches with ultra-small computing capabilities achieves various benefits to IoT networks in terms of latency and reliability.Moreover, we consider network traffic management by introducing a traffic model to control the data traffic among the network, which is also novel.
In Reference [37], the performance of the IoT networks with the Fog computing deployment is studied.A testbed of 50 IoT nodes, distributed Fog nodes and a controller was described.This testbed is used to validate the benefits of Fog computing.This work can be considered as an extension to this study, while in this work, we use powered OF switches with more capabilities and responsibilities.Furthermore, we introduce a structure of the system with the deployment of blockchain.Additionally, we introduce a data flow algorithm to manage the traffic among the proposed network.

IoT System Structure with Distributed Fog Computing and SDN
In this section, we introduce the proposed IoT system that comprises the distributed Fog computing with the SDN and blockchain paradigms.At first, the IoT system structure is introduced and the comprised paradigms and system components are defined.Then, a data offloading algorithm is described for the proposed structure.Finally, a traffic model for analyzing traffic among the proposed structure is introduced.

System Structure
The proposed system deploys Fog computing with blockchain and SDN to support IoT networks and applications.The system can be viewed as a three-layer system as illustrated in Figure 1.The first layer represents the device layer, which contains all IoT devices and sensor devices.These devices are used to measure and capture physical and environmental data.All devices deployed in this layer always have data to be transferred through the network.IoT devices are heterogeneous in terms of computing capabilities, i.e., storage and processing, and energy resources.These devices are battery operated and should be managed in an energy efficient way.
The second layer represents the Fog layer, which deploys Fog nodes to provide an offloading path for the captured data and enable other Fog computing benefits to the IoT network.This moves from the centralized computing scheme to the distributed computing scheme.Fog nodes are deployed at the edge of the access network and each Fog node can serve for a group of IoT devices associated with certain services and a dedicated location.The Fog node handles data forwarded from the dedicated IoT devices.Thus, the Fog layer enables data analyzing, classification and monitoring at the edge of the network.Computing results are forwarded to the higher cloud layer and a response is sent to the IoT devices in cases that required such response.
Adding distributed Fog to an IoT network provides an offloading path for the collected data and thus, reduces the data traffic at the core network.Additionally, Fog nodes provide the computing capabilities near to IoT devices and thus, reduce the end-to-end latency.Furthermore, the introduction of Fog computing increases the overall network flexibility and availability.
The top layer is the cloud layer that is represented by the remote cloud unit.The IoT cloud supports different IoT services and protocols.A service provider can integrate and connect the IoT cloud with other networks.Using the cloud layer, network clients are empowered to use, search and manage the computing resources and data.The cloud layer offers network users the ability to control and monitor the application.
the dedicated IoT devices.Thus, the Fog layer enables data analyzing, classification and monitoring at the edge of the network.Computing results are forwarded to the higher cloud layer and a response is sent to the IoT devices in cases that required such response.
Adding distributed Fog to an IoT network provides an offloading path for the collected data and thus, reduces the data traffic at the core network.Additionally, Fog nodes provide the computing capabilities near to IoT devices and thus, reduce the end-to-end latency.Furthermore, the introduction of Fog computing increases the overall network flexibility and availability.
The top layer is the cloud layer that is represented by the remote cloud unit.The IoT cloud supports different IoT services and protocols.A service provider can integrate and connect the IoT cloud with other networks.Using the cloud layer, network clients are empowered to use, search and manage the computing resources and data.The cloud layer offers network users the ability to control and monitor the application.The network also deploys two main communication paradigms, side by side with the three introduced levels.These paradigms are the SDN technology and the blockchain that are deployed to assist the system and provide control, management and security issues to the introduced system.The end-to-end system structure of the proposed IoT system is presented in Figure 2.

SDN Operation and Integration
The system deploys a single centralized physical SDN controller that controls and manages distributed Fog nodes and, hence, IoT devices.Figure 3 illustrates the three main layers of the deployed SDN model.The data plane of the SDN network contains all sensor nodes that could have additional recourses from the Fog nodes, while the control plane scheme is represented by the deployed SDN controller.
The SDN network also employs distributed OF switches that are powered by limited computing capabilities.These switches can provide some limited services in addition to the switching functions.The SDN controller is able to configure and manage all deployed OF switches via a proper interface, i.e., any supported version of OF protocol [38].The SDN controller employs a clustering algorithm introduced in Reference [39] so that each Fog node or a group of Fog nodes are associated with a distributed SDN controller.Distributed SDN controllers deploy packet migration function to provide security over the databases and work against saturation attacks [36].A distributed SDN network allows the network operator to program and manage Fog nodes and IoT devices via application programming interfaces (APIs).All distributed SDN controllers are connected by the blockchain to provide a high-security level to the proposed IoT network.The network also deploys two main communication paradigms, side by side with the three introduced levels.These paradigms are the SDN technology and the blockchain that are deployed to assist the system and provide control, management and security issues to the introduced system.The end-to-end system structure of the proposed IoT system is presented in Figure 2.

SDN Operation and Integration
The system deploys a single centralized physical SDN controller that controls and manages distributed Fog nodes and, hence, IoT devices.Figure 3 illustrates the three main layers of the deployed SDN model.The data plane of the SDN network contains all sensor nodes that could have additional recourses from the Fog nodes, while the control plane scheme is represented by the deployed SDN controller.
The SDN network also employs distributed OF switches that are powered by limited computing capabilities.These switches can provide some limited services in addition to the switching functions.The SDN controller is able to configure and manage all deployed OF switches via a proper interface, i.e., any supported version of OF protocol [38].The SDN controller employs a clustering algorithm introduced in Reference [39] so that each Fog node or a group of Fog nodes are associated with a distributed SDN controller.Distributed SDN controllers deploy packet migration function to provide security over the databases and work against saturation attacks [36].A distributed SDN network allows the network operator to program and manage Fog nodes and IoT devices via application programming interfaces (APIs).All distributed SDN controllers are connected by the blockchain to provide a high-security level to the proposed IoT network.

Blockchain Operation
Distributed Fog based SDN nodes are connected and managed via the blockchain technology that is used for updating flow table in a secure manner.Furthermore, the cloud layer is split into distributed clouds through the blockchain.
Introducing peer-to-peer paradigm (i.e., blockchain) to the distributed computing achieves various benefits to IoT network such as working against heterogeneous attacks and, thus, increases

Blockchain Operation
Distributed Fog based SDN nodes are connected and managed via the blockchain technology that is used for updating flow table in a secure manner.Furthermore, the cloud layer is split into distributed clouds through the blockchain.
Introducing peer-to-peer paradigm (i.e., blockchain) to the distributed computing achieves various benefits to IoT network such as working against heterogeneous attacks and, thus, increases

Blockchain Operation
Distributed Fog based SDN nodes are connected and managed via the blockchain technology that is used for updating flow table in a secure manner.Furthermore, the cloud layer is split into distributed clouds through the blockchain.
Introducing peer-to-peer paradigm (i.e., blockchain) to the distributed computing achieves various benefits to IoT network such as working against heterogeneous attacks and, thus, increases the overall system security, leading to increasing the flexibility of the system, achieving the required scalability of the IoT networks, and increasing the overall system availability.
In this paper, studying blockchain is limited to its functions as a structural component; other aspects will be studied in future publications.This is because the main objective of this work is the end-to-end latency, not the analysis of security issues.

Data Offloading Algorithm
The proposed system works based on the data flow algorithm illustrated in Figure 4.The network operation goes through various steps.The first step is authentication, as the IoT node should be authorized.The IoT node communicates directly with the IoT cloud to be authorized.Then, the IoT cloud performs the authentication process and identifies the device to be authorized.
the overall system security, leading to increasing the flexibility of the system, achieving the required scalability of the IoT networks, and increasing the overall system availability.
In this paper, studying blockchain is limited to its functions as a structural component; other aspects will be studied in future publications.This is because the main objective of this work is the end-to-end latency, not the analysis of security issues.

Data Offloading Algorithm
The proposed system works based on the data flow algorithm illustrated in Figure 4.The network operation goes through various steps.The first step is authentication, as the IoT node should be authorized.The IoT node communicates directly with the IoT cloud to be authorized.Then, the IoT cloud performs the authentication process and identifies the device to be authorized.
The next step is address detection, in which the cloud calls the service provider to determine the location of the IoT.For this purpose, the service provider refers to the SDN orchestrator, which makes an investment to locate the IoT device.Moreover, the SDN orchestrator populates the routing table with different routing paths between the IoT node and the cloud and locates all OF switches that are dedicated to this communication.The next step is address detection, in which the cloud calls the service provider to determine the location of the IoT.For this purpose, the service provider refers to the SDN orchestrator, which makes an investment to locate the IoT device.Moreover, the SDN orchestrator populates the routing table with different routing paths between the IoT node and the cloud and locates all OF switches that are dedicated to this communication.

•
As the system mainly considers resource utilization, it makes use of all the available resources.Consequently, the SDN controller allows the OF switches to handle some processing and computing tasks for the IoT forwarded data after the Fog level.The SDN controller estimates the OF switches with the available resources upon checking certain parameters.These parameters are the following: • Traffic access type, • Time delay constraints,

•
Processing power for servicing the IoT data, and • The current state of the OF switches in terms of traffic and resources.
The SDN controller decides the possibility of enabling the IoT data, passed to the core network through the Fog layer, to be a part of the available resources of the OF switches by optimizing the previous parameters and, thus, informs the selected switches.The orchestrator creates a virtual machine on the selected OF switches that are used for data processing.The next step is database migration.The IoT Cloud, through the service provider, migrates the database for servicing the IoT group over certain OF switches.The network continues working and OF switches aggregate and synchronize the IoT data with the cloud.
Handling computing tasks to OF switches achieves various benefits to our proposed IoT system structure, these benefits include the following: Useful for anti-persistence traffic in the core network, and • Efficient resource utilization.

Traffic Model
It is clear that reducing a part of the subscriber traffic in the local cloud reduces the total traffic, and, thus, increases the quality of service (QoS) of the traffic served by the network.Introducing Fog nodes with SDN to IoT networks has a great impact on the network traffic performance and efficiency.To enhance this performance, a Fog computing-based traffic model is introduced.This traffic model reflects the impact of introducing Fog computing on the traffic services over the network.
In order to estimate the efficiency of introducing Fog nodes (i.e., Fog computing) on the traffic performance and efficiency, the delivery time of the data offloaded is considered as the main metric, which reflects the impact of the Fog computing on the traffic service in the network.
The proposed traffic model considers the operation of the access network, the core network and the application server as queuing processes.The traffic model assumes a G/G/1 queuing system and also assumes that the main characteristic of the access network, core network and an application server is the delivery time T [40]. Figure 5 illustrates the proposed traffic model based on the G/G/1 queuing model.where, s is the service time and ε is the form factor [41].The efficiency of introducing Fog computing nodes on the traffic is E and can be calculated as the percentage decrease in the queuing delay of the ordinary IoT network (i.e., without the introduction of Fog computing nodes) and due to the existence of the Fog computing layer.
where, EF is the efficiency in the existence of Fog computing layer and Eo is the efficiency of the ordinary IoT system with no Fog layer.The maximum value of E corresponds to the maximal efficiency of using Fog computing nodes.Figure 6 shows the impact of the change of the probability of traffic forwarding to the Fog cloud layer on the efficiency E, for different values of ρ.As the probability increases, the Fog nodes can handle a higher amount of traffic and, thus, the efficiency The total traffic originated by a group of users (e.g., IoT nodes) in a cell or a base station has the intensity A. The user traffic may be forwarded to a nearby Fog node; the probability that this event happens is assumed to be P.This reduces the amount of traffic handled to the access network.Thus, the traffic served by the access network is equal to x, where x is calculated as follows: The intensity of the traffic handled by the Fog node is x , where x can be calculated as follows: As a result, the traffic service of the Fog computing node originates the traffic that is forwarded to the core network with intensity x , where x is calculated as follows: where, K is the probability constant with a value between zero and one.For K with any value below one, the amount of traffic forwarded to the core network is reduced and, thus, the Fog unit achieves traffic reduction and reduces the network congestion.The zero value of the constant K corresponds to the removal of the Fog computing layer.The total delivery time T can be calculated as follows [41]: where, s is the service time and ε is the form factor [41].The efficiency of introducing Fog computing nodes on the traffic is E and can be calculated as the percentage decrease in the queuing delay of the ordinary IoT network (i.e., without the introduction of Fog computing nodes) and due to the existence of the Fog computing layer.
where, E F is the efficiency in the existence of Fog computing layer and E o is the efficiency of the ordinary IoT system with no Fog layer.The maximum value of E corresponds to the maximal efficiency of using Fog computing nodes.Figure 6 shows the impact of the change of the probability of traffic forwarding to the Fog cloud layer on the efficiency E, for different values of ρ.As the probability increases, the Fog nodes can handle a higher amount of traffic and, thus, the efficiency increases.Furthermore, the dependence shows that the efficiency grows rapidly in the case of a high traffic value and grows slowly in the case of a small traffic value.Additionally, the efficiency varies from 0, when no traffic is directed to the Fog cloud, to 1, when all traffic is directed to the Fog cloud.Table 1.The experimental parameters and device specifications.

Device Specifications
Vendor Fujitsu

Performance Evaluation
In this section, the performance of the proposed IoT framework and all associated algorithms are evaluated.The proposed IoT-Fog system is experimentally tested in a testbed.Various parameters are considered as performance metrics.Moreover, the proposed offloading and traffic algorithms are simulated and the obtained results are analyzed.

Experiment Setup
In order to evaluate the performance of the proposed system structure and the data offloading algorithm, the following experiment is conducted.We construct the system shown in Figure 2, while the considered network components are presented in Table 1, with the introduction of the specifications of each component.Since the blockchain was considered as only a structural component in this work, it is not considered in the simulation part and the developed testbed doesn't deploy such technology.The x86 architecture is deployed to act as an OF switch, which is able to support processing and computing tasks [42].We employ 48 Raspberry nodes; each of them represents an IoT node.The 48 Raspberry nodes act as traffic generators that generate data traffic with an average of 6 per each node.The application layer supports the MQTT and CoAP protocols [43].The system is also simulated over the iFogSim simulator, which is a reliable Java-based simulation environment for simulating IoT networks with a distributed Fog computing structure [44].The iFogSim is built over the CloudSim environment and for the simulation process of the proposed system; CloudSim SDN is also involved in the SDN network [45].CloudSim SDN is also a reliable Java-based environment; built over the CloudSim [46].
The system is simulated over a machine with an Intel Core i5 processor, with a speed of 3.07 GHz and a memory of 16 GB.The considered simulation parameters are introduced in Table 2.
For the performance evaluation of the proposed system, the following performance metrics are considered for both the simulation and experimental works; resources utilization (e.g., storage, processing and energy) and the end-to-end latency.

Experimental Results
In order to evaluate the performance of deploying the distributed Fog computing and SDN paradigm, the system is simulated for the three considered cases.In the first case, the system is simulated without the deployment of the distributed Fog computing and an SDN network.In this case, distributed IoT devices had to communicate with the remote cloud and no nearby computing capabilities are provided.The second case represents the system with the distributed Fog computing layer and without the deployment of an SDN network.In this case, distributed IoT devices can use the nearby Fog computing capabilities.The final case represents the proposed IoT network with the deployment of distributed Fog computing controlled by an SDN network.Table 3 summarizes the considered case specifications.7 illustrates the amount of storage used by the system in the three considered cases.As the results indicate, the deployment of Fog computing achieves higher utilization performance of storage resources than the IoT system with only centralized cloud computing.Moreover, the proposed IoT system with distributed Fog computing and an SDN network achieves higher performance than the previously considered cases in terms of storage resources utilization.
Table 3.The considered simulation cases.

Case Deployed Communication Technology
Case   Figure 8 illustrates the utilization performances of the processing resources for each considered case.The proposed system utilizes the processing resources in an efficient way with higher performance than other considered systems.Figure 9 provides the total energy consumed for computing tasks by all network elements in each considered case based on the energy model introduced in Reference [47].The deployment of SDN with distributed Fog computing achieves a higher energy efficiency of the IoT network and thus, utilize the energy resources more efficiently.Figure 8 illustrates the utilization performances of the processing resources for each considered case.The proposed system utilizes the processing resources in an efficient way with higher performance than other considered systems.Figure 9 provides the total energy consumed for computing tasks by all network elements in each considered case based on the energy model introduced in Reference [47].The deployment of SDN with distributed Fog computing achieves a higher energy efficiency of the IoT network and thus, utilize the energy resources more efficiently.Figure 8 illustrates the utilization performances of the processing resources for each considered case.The proposed system utilizes the processing resources in an efficient way with higher performance than other considered systems.Figure 9 provides the total energy consumed for computing tasks by all network elements in each considered case based on the energy model introduced in Reference [47].The deployment of SDN with distributed Fog computing achieves a higher energy efficiency of the IoT network and thus, utilize the energy resources more efficiently.Figure 8 illustrates the utilization performances of the processing resources for each considered case.The proposed system utilizes the processing resources in an efficient way with higher performance than other considered systems.Figure 9 provides the total energy consumed for computing tasks by all network elements in each considered case based on the energy model introduced in Reference [47].The deployment of SDN with distributed Fog computing achieves a higher energy efficiency of the IoT network and thus, utilize the energy resources more efficiently.
Figure 10 provides the end-to-end system latency for each considered case.The results indicate that the proposed system achieves a higher latency efficiency.Thus, the proposed IoT system achieves higher efficiency in terms of computing resources utilization (e.g., processing, storage and energy) and latency.This is because of the deployment of the distributed edge computing paradigm brings the computing resources near IoT devices.Additionally, deploying an SDN for controlling and managing the IoT-Fog network is the key solution for performance enhancement.This is because of the previously mentioned benefits of SDN based networks.
switches.In the second case, the Fog nodes are deployed.The results indicate the high performance achieved in the case of Fog deployment.
Figure 12 illustrates the total latency of IoT traffic in the case of the network being operated without the Fog and SDN.In this case, the IoT nodes directly communicate with the IoT cloud.Figure 13 illustrates the latency for the proposed system where Fog nodes and an SDN network are deployed.By comparing the two figures, we can see vast variations in the latencies in both cases.Employing Fog nodes and an SDN network with the enabled processing capabilities of OF switches achieves a high reduction in the communication latency of IoT data and the better utilization of the computing resources, which can be considered the main benefit of the proposed system structure.

Experimental Results
Figure 11 illustrates the percentage of the average CPU load of the OF switches in two considered cases.In the first case, the network is operated without the Fog layer, this puts a great load on the OF switches.In the second case, the Fog nodes are deployed.The results indicate the high performance achieved in the case of Fog deployment.

Experimental Results
Figure 11 illustrates the percentage of the average CPU load of the OF switches in two considered cases.In the first case, the network is operated without the Fog layer, this puts a great load on the OF switches.In the second case, the Fog nodes are deployed.The results indicate the high performance achieved in the case of Fog deployment.
Figure 12 illustrates the total latency of IoT traffic in the case of the network being operated without the Fog and SDN.In this case, the IoT nodes directly communicate with the IoT cloud.Figure 13 illustrates the latency for the proposed system where Fog nodes and an SDN network are deployed.By comparing the two figures, we can see vast variations in the latencies in both cases.Employing Fog nodes and an SDN network with the enabled processing capabilities of OF switches achieves a high reduction in the communication latency of IoT data and the better utilization of the computing resources, which can be considered the main benefit of the proposed system structure.Figure 12 illustrates the total latency of IoT traffic in the case of the network being operated without the Fog and SDN.In this case, the IoT nodes directly communicate with the IoT cloud.Figure 13 illustrates the latency for the proposed system where Fog nodes and an SDN network are deployed.By comparing the two figures, we can see vast variations in the latencies in both cases.Employing Fog nodes and an SDN network with the enabled processing capabilities of OF switches achieves a high reduction in the communication latency of IoT data and the better utilization of the computing resources, which can be considered the main benefit of the proposed system structure.

Conclusions
Employing distributed Fog computing for IoT networks achieves various benefits since it brings the cloud computing capabilities (e.g., computing, storage and processing) near IoT nodes.This work has introduced a framework of the IoT system that deploys distributed Fog computing with the SDN and blockchain paradigms.The SDN employs a physical centralized/logical distributed controller with distributed OF switches to manage and control distributed Fog computing.The distributed OF switches have been empowered with limited resources that can be used for assisting forwarded traffic.The introduction of an SDN achieves higher flexibility and higher performance in utilizing computing resources.The work provides a novel offloading mechanism that handles certain processing and computing tasks to OF switches to reduce the data latency and achieve other benefits.The data offloading algorithm for controlling and managing data offloading over the proposed system was developed with the traffic model.The proposed system was simulated over a reliable environment and also experimentally evaluated via a developed testbed.Simulation and experimental results validate the system and ensure the efficiency claims.

Conclusions
Employing distributed Fog computing for IoT networks achieves various benefits since it brings the cloud computing capabilities (e.g., computing, storage and processing) near IoT nodes.This work has introduced a framework of the IoT system that deploys distributed Fog computing with the SDN and blockchain paradigms.The SDN employs a physical centralized/logical distributed controller with distributed OF switches to manage and control distributed Fog computing.The distributed OF switches have been empowered with limited resources that can be used for assisting forwarded traffic.The introduction of an SDN achieves higher flexibility and higher performance in utilizing computing resources.The work provides a novel offloading mechanism that handles certain processing and computing tasks to OF switches to reduce the data latency and achieve other benefits.The data offloading algorithm for controlling and managing data offloading over the proposed system was developed with the traffic model.The proposed system was simulated over a reliable environment and also experimentally evaluated via a developed testbed.Simulation and experimental results validate the system and ensure the efficiency claims.

Latency (ms)
Figure 13.The communication latency for the IoT-Fog system.

Conclusions
Employing distributed Fog computing for IoT networks achieves various benefits since it brings the cloud computing capabilities (e.g., computing, storage and processing) near IoT nodes.This work has introduced a framework of the IoT system that deploys distributed Fog computing with the SDN and blockchain paradigms.The SDN employs a physical centralized/logical distributed controller with distributed OF switches to manage and control distributed Fog computing.The distributed OF switches have been empowered with limited resources that can be used for assisting forwarded traffic.The introduction of an SDN achieves higher flexibility and higher performance in utilizing computing resources.The work provides a novel offloading mechanism that handles certain processing and computing tasks to OF switches to reduce the data latency and achieve other benefits.The data offloading algorithm for controlling and managing data offloading over the proposed system was developed with the traffic model.The proposed system was simulated over a reliable environment and also experimentally evaluated via a developed testbed.Simulation and experimental results validate the system and ensure the efficiency claims.

Figure 1 .
Figure 1.The main layers of the proposed Internet of Things (IoT)-Fog system.

Figure 1 .
Figure 1.The main layers of the proposed Internet of Things (IoT)-Fog system.

17 Figure 2 .
Figure 2. The system structure of the proposed IoT-Fog system with Software Designed Networking (SDN)/blockchain.

Figure 3 .
Figure 3.The layers of an SDN network.

Figure 2 . 17 Figure 2 .
Figure 2. The system structure of the proposed IoT-Fog system with Software Designed Networking (SDN)/blockchain.

Figure 3 .
Figure 3.The layers of an SDN network.

Figure 3 .
Figure 3.The layers of an SDN network.

Figure 4 .
Figure 4.The data flow algorithm.

Figure 4 .
Figure 4.The data flow algorithm.

Figure 5 .
Figure 5.The traffic service model.

Figure 5 .
Figure 5.The traffic service model.

Figure 6 .
Figure 6.The traffic efficiency for the IoT based Fog system.

Figure 6 .
Figure 6.The traffic efficiency for the IoT based Fog system.

Figure 7 .
Figure 7.The average resources utilization in terms of storage for the considered simulation cases.

Figure 8 .
Figure 8.The average resources utilization in terms of processing for the considered simulation cases.

Figure 9 .
Figure 9.The average resources utilization in terms of energy for the considered simulation cases.

Figure 7 . 17 Table 3 .Figure 7 .
Figure 7.The average resources utilization in terms of storage for the considered simulation cases.

Figure 8 .
Figure 8.The average resources utilization in terms of processing for the considered simulation cases.

Figure 9 .
Figure 9.The average resources utilization in terms of energy for the considered simulation cases.

Figure 8 .
Figure 8.The average resources utilization in terms of processing for the considered simulation cases.

Figure 7 .
Figure 7.The average resources utilization in terms of storage for the considered simulation cases.

Figure 8 .
Figure 8.The average resources utilization in terms of processing for the considered simulation cases.

Figure 9 .
Figure 9.The average resources utilization in terms of energy for the considered simulation cases.

Figure 9 .
Figure 9.The average resources utilization in terms of energy for the considered simulation cases.

Figure 10 .Figure 11 .Figure 10 .
Figure 10.The average end-to-end latency for the considered simulation cases.

Figure 10 .Figure 11 .Figure 11 .
Figure 10.The average end-to-end latency for the considered simulation cases.

Figure 12 .
Figure 12.The communication latency in case of direct access to the IoT cloud.

Figure 13 .
Figure 13.The communication latency for the IoT-Fog system.

Figure 12 .
Figure 12.The communication latency in case of direct access to the IoT cloud.

Figure 13 .
Figure 13.The communication latency for the IoT-Fog system.

Table 1 .
The experimental parameters and device specifications.

Table 2 .
The simulation parameters.

Table 3 .
The considered simulation cases.
Figures 7-9 illustrate the simulation results in terms of resources utilization.Figure

Table 3 .
The considered simulation cases.
Figure 12.The communication latency in case of direct access to the IoT cloud.