Enabling Heterogeneous IoT Networks over 5G Networks with Ultra-Dense Deployment—Using MEC/SDN

: The Internet of things (IoT) is the third evolution of the traditional Internet that enables interaction and communication among machines. Many IoT platforms and networks have been developed, and recently, market sectors have started to develop speciﬁc IoT applications and services. Integrating heterogeneous IoT networks with the existing ones, mainly with the cellular networks, is a great demand. IoT represents one of the main use cases of the ﬁfth-generation (5G) cellular system as announced by the 3rd Generation Partnership Project (3GPP) and the International Telecommunication Union (ITU). Integrating IoT networks with 5G networks face many challenges related to dense deployment and a massive number of expected connected devices. Thus, IoT network availability and scalability are the main requirements that should be achieved. To this end, this work provides a framework for integrating heterogeneous IoT networks with the 5G networks. The proposed system considers dense deployment and system scalability and availability requirements as announced by ITU and 3GPP. Our proposed structure deploys three main communication paradigms; mobile edge computing (MEC), device-to-device communications (D2D), and software-deﬁned networking (SDN). Our proposed system is evaluated over a reliable environment for various deployment scenarios, and the results validate the proposed structure. The proposed IoT/5G reduces the percentage of blocked tasks by an average of 30% than other traditional IoT networks. This increases the overall system availability and scalability since IoT networks can have more devices and tasks than existing IoT networks. Furthermore, our proposed structure reduces the overall consumed energy by an average of 20% than existing IoT networks, which is an effective metric for IoT networks.


Introduction
The Internet of things (IoT) is a recent communication paradigm considered to be the third evolution of the traditional Internet [1]. IoT is the communication system that considers connecting heterogeneous wireless devices and provides the communication medium for the interaction between machines [2]. As well as human-to-human (H2H) communication, IoT enables machine-to-machine (M2M) communication. M2M is the communication paradigm that has been introduced to enable the communications and interactions between machines, e.g., sensors and actuators [3]. Many market sectors consider developing IoT systems and applications, have been launched, and massive IoT applications in many fields have been developed [4]. However, there are many design Paper organization: The article is structured as follows: Section 2 introduces the current literature and compares the related works with the proposed IoT/5G system. The novelty of the proposed work is well defined in this section. Section 3 presents the proposed framework for the IoT/5G system. The system structure is introduced, and the main parts are clearly discussed. Section 4 introduces the developed energy-aware, and latency-aware offloading scheme for the proposed IoT/5G system Section 5 is introduced for performance evaluation and result in a discussion of the proposed system. Section 6 is introduced for conclusions and future directions.

Related Works
With the enormous increase in wireless and IoT devices, dense deployment becomes a major requirement of IoT networks. IoT is announced as the main use case of 5G systems as per 3GPP and ITU, and thus, the integration between existing IoT networks and cellular networks, mainly 5G systems, is a demand. These two challenges, i.e., scalability and integration with cellular systems, besides other existing network challenges of IoT networks, e.g., energy and availability, need to be considered. To overcome these challenges, recent communication paradigms have been introduced for heterogeneous IoT networks and applications that include MEC, SDN, D2D and Blockchain [16].
The existing literature considers a part of these challenges; however, this is for specific IoT networks. This work considers developing a framework for integrating IoT networks with 5G systems using MEC and SDN technologies. Our main contributions are clearly defined in the previous section. However, this section is introduced to illustrate the novelty of the proposed IoT/5G framework. Few studies have considered the integration of IoT networks with the 5G systems. In this section, we compare our proposed framework with the existing works by considering integrating IoT networks with cellular systems. Furthermore, since the dense deployment of IoT networks and energy efficiency are the main contributions of our proposed IoT framework, all recent existing works considering these objectives are included in the comparison. Table 1 presents a comparison between existing related works and our proposed IoT/5G system. The comparison introduces the used key enabling technology for the IoT networks and the considered IoT networks, e.g., low-power wide-area networks (LP-WAN), LoRaWAN, or the work suitable for any IoT network. Furthermore, the considered performance evaluation metrics are introduced for each work. Table 1. Internet of things (IoT)/5G system compared to existing related works.

Key Enabling Technology
IoT Network Application Performance Metrics Cellular Integration Fog/MEC D2D SDN [17] √ X √ Any General No evaluation √ [18] √ √ X Any General Latency X Scalability [15] X √ X Any General Energy X [19] √ X X Any General Offloading X [20] √ X X Any Vehicle-togrid Data classification √ [21] √ X X Any General Completion time X Energy [22] √ X X Any General Energy X [23] √ X X Any Smart home Temporal delay X  [24] √ X √ Any General Reliability X Latency [25] √ X X Any General Data analysis X [26] X √ X Any General Energy X Security [27] √ X X Any Autonomous vehicles Latency X [28] √ X X Any Time-critical apps. Latency X [29] √ X X Any Medical Energy X [30] √ X X Any General Latency X Energy [31] √ X √ Any General Security X [32] √ X X LoRaWAN Low latency apps Energy X [33] √ X X Any General Spectrum X Data management [34] √ X X LPWAN General Resource provisioning X [35] √ X X WAN-IoT Timesensitive Resources X IoT/5G √ √ √ Any General Scalability Energy √

IoT/5G System Structure
The proposed system deploys D2D communications, MEC and SDN technologies to integrate heterogeneous IoT networks with 5G networks. Figure 1 provides an end-to-end system structure of the proposed IoT/5G. The system can be viewed into two main parts; radio access networks, RAN, and core networks, CN. RAN comprises three subsystems; heterogeneous IoT networks, D2D-based networks, and RAN of the cellular networks. However, core networks deploy an SDN network with a multi-controller scheme.

Heterogeneous IoT Networks
Heterogeneous IoT networks, e.g., low-power wide-area networks (LPWAN), can be connected to the 5G cellular system via MEC servers, as presented in Figure 2. Each IoT gateway is connected to a MEC server via a fiber-wireless, FiWi, connection. The MEC server, deployed for IoT networks, is referred to as the I-MEC server. The introduction of MEC technology to the edge of the access networks of IoT networks achieves various benefits [16]. MEC servers, connected to IoT gateways, provide computing resources, e.g., processing and storage, to IoT devices; one communication hops away. This reduces the communication latency, increases the system availability and reduces the load on the core networks. Furthermore, the introduction of the I-MEC servers achieves higher efficiency in terms of energy, scalability and reliability. Since I-MEC servers provide computing resources near to end-devices, this reduces the load on the end devices, which reduces the consumed energy.

Heterogeneous IoT Networks
Heterogeneous IoT networks, e.g., low-power wide-area networks (LPWAN), can be connected to the 5G cellular system via MEC servers, as presented in Figure 2. Each IoT gateway is connected to a MEC server via a fiber-wireless, FiWi, connection. The MEC server, deployed for IoT networks, is referred to as the I-MEC server. The introduction of MEC technology to the edge of the access networks of IoT networks achieves various benefits [16]. MEC servers, connected to IoT gateways, provide computing resources, e.g., processing and storage, to IoT devices; one communication hops away. This reduces the communication latency, increases the system availability and reduces the load on the core networks. Furthermore, the introduction of the I-MEC servers achieves higher efficiency in terms of energy, scalability and reliability. Since I-MEC servers provide computing resources near to end-devices, this reduces the load on the end devices, which reduces the consumed energy.
MEC servers provide resources for existing and upcoming devices and thus increase the overall system scalability. Furthermore, the edge server can be scaled in terms of resources, e.g., storage and processing, to handle more traffic of new devices. Our main contribution is to integrate the heterogeneous IoT networks with the 5G cellular system, which is achieved efficiently by the proposed MEC-based structure introduced in Figure 2. The introduction of edge computing, i.e., I-MEC, to the access networks of the IoT networks provides an interface to the cellular networks since the I-MEC server is connected to nearby MEC servers of the cellular networks, which makes the IoT data  MEC servers provide resources for existing and upcoming devices and thus increase the overall system scalability. Furthermore, the edge server can be scaled in terms of resources, e.g., storage and processing, to handle more traffic of new devices.
Our main contribution is to integrate the heterogeneous IoT networks with the 5G cellular system, which is achieved efficiently by the proposed MEC-based structure introduced in Figure 2. The introduction of edge computing, i.e., I-MEC, to the access networks of the IoT networks provides an interface to the cellular networks since the I-MEC server is connected to nearby MEC servers of the cellular networks, which makes the IoT data available at the edge of the RAN of cellular networks.
For managing the data offloading process, we develop an energy-aware offloading scheme for the proposed MEC-based IoT structure. The proposed offloading scheme is based on our developed energy-aware offloading algorithm for multilevel MEC-based 5G systems, presented in [36]. The proposed offloading algorithm controls data offloading between IoT devices and I-MEC servers in a way that achieves higher energy efficiency and thus prolongs the lifetime of IoT devices. This comes with the announced 3GPP requirements of IoT as a use case of 5G [6]. The proposed offloading algorithm for IoT networks is presented in Section 4.1.

D2D Enabled Network
Heterogeneous sensors and other wireless devices may be deployed over 5G cellular networks. One way to enable these devices is to introduce an appropriate radio communication interface, besides the existing cellular interface, to unload the cellular networks. Furthermore, some of these devices, i.e., sensors and actuators, may not support the cellular interface, and thus, the need for another radio interface becomes a must. D2D is one of the best solutions to provide coverage for the distributed sensors and wireless devices [18]. This enables the dense deployment scenario required for 5G systems. Figure 3 illustrates the main structure of the D2D enabled networks. Distributed sensors and wireless devices that support D2D communications use the nearby devices that support cellular interface to forward their data to the cellular base station with the connected edge server. devices that support D2D communications use the nearby devices that support cellular interface to forward their data to the cellular base station with the connected edge server. Devices deployed in the proposed D2D networks are categorized into three main levels based on the device capabilities, e.g., processing and storage, and the radio communication interfaces supported by the device. The first level represents devices that support only D2D communication and does not support the cellular interface. These devices are referred to as low-level devices (LL). LL devices have limited resources, including energy, Devices deployed in the proposed D2D networks are categorized into three main levels based on the device capabilities, e.g., processing and storage, and the radio communication interfaces supported by the device. The first level represents devices that support only D2D communication and does not support the cellular interface. These devices are referred to as low-level devices (LL). LL devices have limited resources, including energy, processing and storage. These devices include sensors and actuators. The second level represents devices that support either D2D communication interfaces only or both cellular and D2D interfaces. These devices are wireless devices with acceptable computing resources that are higher than that of LL devices. These devices are referred to as gateway devices (GW). GW devices can handle some computing tasks to the nearby LL devices based on the currently available resources and the current level of energy of GW devices. Furthermore, if the available resources are not enough to handle the offloaded tasks, GW devices act as a relay and forward the received tasks from LL to the higher level.
The last level of devices is the high-level devices (HL), which represents devices with powerful computing resources and energy resources. These devices must support both cellular and D2D interfaces. Such examples of these devices include smartphones, tablets and notebooks. HL devices receive offloading requests from the distributed GW devices and nearby LL devices and provide the appropriate response upon the available resources. If the currently available resources of the HL device are enough to handle the received offloading requests, the HL device accepts the offloading request and handles the computing task for GW or LL device. When the current resources of the HL device are not enough to handle the offloading request, the HL device sends the request to the MEC server connected to the cellular base station. The previously developed network setup algorithm presented in [19], is used for the proposed D2D-based networks. The required offloading algorithm for managing task offloading between devices in D2D-based networks is presented in Section 4.2. The algorithm is energy and latency-aware to achieve a required QoS latency for computing tasks and to achieve high-energy efficiency.

Cellular Network
This part represents the heterogeneous 5G cellular cells with the connected MEC servers. Each eNB is connected to an edge server to provide computing resources to the distributed devices in the cell. The proposed system deploys homogeneous MEC servers with equal resources. The MEC server deployed for a cellular call is referred to as C-MEC. I-MEC servers are connected directly to nearby C-MEC servers via highspeed fiber connections. C-MEC servers handle the offloaded tasks from the connected I-MEC servers, cellular devices and corresponding D2D networks. Based on the currently available resources and the QoS latency of the computing task, the C-MEC server handles the computing task or offloads it to the core networks cloud. The proposed offloading algorithm, introduced in Section 4.3, provides the main steps required to perform the energy-aware and latency-aware offloading.

Core Network (CN)
The core network, CN, deploys the SDN technology, where the control plane and the data plane are separated. The proposed system deploys multiple SDN controllers since a physical SDN controller handles only a limited number of requests at a time, and as previously introduced, an SDN network with a single controller is efficient only for smallscale networks. SDN controllers communicate and interact via OpenFlow protocol [37]. Distributed OpenFlow switches connect to the appropriated SDN controller and receive their flow tables over the appropriate OpenFlow version deployed for the networks. A core networks cloud with powerful computing capabilities is deployed at the core networks and connected to the control plane of SDN networks. Our previously proposed controller placement algorithm, introduced in [38], is deployed to dynamically define the optimum number of SDN controllers required for the networks based on the current status of net-Electronics 2021, 10, 910 8 of 28 work traffic. Furthermore, the algorithm provides the optimum connections of OpenFlow switches with SDN controllers in a way that achieves the highest latency performance.

Energy-Aware and Latency-Aware Offloading Scheme
In this section, the proposed offloading algorithms for subnetworks are introduced. The proposed algorithms are built based on our previously developed energy-aware and latency-aware offloading algorithm developed for multilevel MEC structure, presented in [36]. Table 2 represents the notation of all used parameters and variables. This section is divided into three subsections. The first subsection introduces the developed offloading algorithm for IoT networks, while the second subsection presents the offloading scheme developed for D2D-based networks. The last subsection provides the offloading algorithm for the MEC-based cellular system. Energy threshold level of the I-MEC server E thr-C-MEC Energy threshold level of the C-MEC server E C-IoT Energy of the IoT device after local execution E C-LL Energy of the LL device after local execution E C-GW Energy of the GW device after handling the requested task E C-HL Energy of the HL device after handling the requested task E C-I-MEC Energy of the I-MEC server after handling the requested task Energy of the C-MEC server after handling the requested task E L-exc-IoT Total energy required for local execution of current computing task at the IoT device E l-exc-LL Total energy required for local execution of current computing task at the LL device E l-exc-GW Total energy required to execute a computing task at the GW device E l-exc-HL Total energy required to execute a computing task at the HL device E l-exc-I-MEC Total energy required to execute a computing task at the I-MEC server E l-exc-C-MEC Total energy required to execute a computing task at the C-MEC server E IoT Energy of the IoT device before local execution E LL Energy of the LL device before local execution Table 2. Cont.

E GW
Energy of the GW device before handling the requested task E HL Energy of the HL device before handling the requested task E I-MEC Energy of the I-MEC server before handling the requested task E C-MEC Energy of the C-MEC server before handling the requested task E h-GW Total energy required to handle a computing task at the GW device E h-HL Total energy required to handle a computing task at the HL device E h-I-MEC Total energy required to handle a computing task at the I-MEC server E h-C-MEC Total energy required to handle the computing task at the C-MEC server T exc-IoT Total execution time of computing task at the IoT device T exc-LL Total time required to execute a computing task at the LL device T exc-GW Total time required to execute a computing task at the GW device T exc-HL Total time required to execute a computing task at the HL device T exc-I-MEC Total time required to execute the requested task at the I-MEC server T exc-C-MEC Total time required to execute the requested task at the C-MEC server T h-GW Total time required to handle the computing task at the GW device T h-HL Total time required to handle the computing task at the HL device T h-I-MEC Total time required to handle the computing task at the I-MEC server T h-C-MEC Total time required to handle the computing task at the C-MEC server T tx Total transmission time of input data of the computing task Transmitting power of the IoT device P S Transmitting power of the IoT gateway connected to the I-MEC server P LL Transmitting power of the LL device P GW Transmitting power of the GW device P HL Transmitting power of the HL device η IoT Channel efficiency of the IoT network η S Channel efficiency of the IoT-cellular η D2D Channel efficiency of D2D network η C Channel efficiency of cellular interface

Energy-Aware and Latency-Aware Offloading Algorithm for IoT Networks
The proposed offloading algorithm consists of three offloading levels besides the local execution, as presented in Figure 4. The zero-level offloading, which corresponds to the local execution. This represents tasks that can be executed by the end devices without any need for the offloading process. This depends on the currently available resources of end-devices and the required QoS of the task. The first level of offloading represents the I-MEC servers. Tasks that cannot be executed by the end-devices are offloaded via the appropriate communication link to the I-MEC server connected to the IoT gateway. The I-MEC server accepts the offloading process or rejects it based on the currently available resources at the I-MEC server and the required QoS of the offloaded task. If the currently available resources of the I-MEC server are not enough to handle the offloaded task, the task is then offloaded to the higher level, i.e., the second level of offloading. The second offloading level is represented by the cellular MEC server, i.e., C-MEC. I-MEC server offloads the unhandled tasks to the connected C-MEC server. C-MEC server accepts the offloaded task or rejects it based on the currently available resources of enddevices and the required QoS of the task. Tasks that cannot be handled by the C-MEC server are offloaded to the final offloading level represented by the core network cloud. Algorithm 1 presents the developed energy-aware, a latency-aware offloading algorithm for MEC-based IoT networks.
End devices decide the main offloading decision based on the currently available resources, e.g., storage, processing and energy, at the device and the required QoS latency of the current task. Each device uses the program profile to calculate the size of the computing task, Z, and the required number of processing cycles, NCPU-CYC to handle the task. The decision engine of the end device is the hardware that implements our proposed offloading algorithm. It receives the information from the program profiler, i.e., Z and NCPU-CYC, and the information of the currently available resources from the resources schedular. Furthermore, the decision engine receives the required QoS latency, TQoS, of the current computing task, which represents the maximum allowable latency to handle the computing task. The decision engine uses these parameters to decide either the local execution or offloading. Calculate Z, NCPU-CYC-Bit 3: Calculate Texc-IoT 4: If (Texc-IoT ≤ TQoS) 5: DOff-T-IoT = 0 6: Calculate EL-exc-IoT, EC-IoT 7: If (EC-IoT > Ethr-IoT) 8: DOff-E-IoT = 0 The second offloading level is represented by the cellular MEC server, i.e., C-MEC. I-MEC server offloads the unhandled tasks to the connected C-MEC server. C-MEC server accepts the offloaded task or rejects it based on the currently available resources of enddevices and the required QoS of the task. Tasks that cannot be handled by the C-MEC server are offloaded to the final offloading level represented by the core network cloud. Algorithm 1 presents the developed energy-aware, a latency-aware offloading algorithm for MEC-based IoT networks.
End devices decide the main offloading decision based on the currently available resources, e.g., storage, processing and energy, at the device and the required QoS latency of the current task. Each device uses the program profile to calculate the size of the computing task, Z, and the required number of processing cycles, N CPU-CYC to handle the task. The decision engine of the end device is the hardware that implements our proposed offloading algorithm. It receives the information from the program profiler, i.e., Z and N CPU-CYC , and the information of the currently available resources from the resources schedular. Furthermore, the decision engine receives the required QoS latency, T QoS , of the current computing task, which represents the maximum allowable latency to handle the computing task. The decision engine uses these parameters to decide either the local execution or offloading. Algorithm 1 Latency-aware and energy-aware offloading algorithm for MEC-based IoT networks.

1:
Initialize T QoS , E thr-IoT , E thr-I-MEC 2: Calculate Z, N CPU-CYC-Bit 3: Calculate T exc-IoT 4: If (T exc-IoT ≤ T QoS ) 5: D Off-T-IoT = 0 6: Calculate E L-exc-IoT , E C-IoT 7: If (E C-IoT > E thr-IoT ) 8: D Off-E-IoT = 0 9: else 10: If (D off-IoT == 0) 17: Handle task locally 18: else 19: Offload Task to I-MEC server 20: end if 21: Calculate If I-MEC server accepts offloading request, and handles the computing task. 36: else 37: Offload Task to C-MEC server 38: end if At first, the decision engine calculates the total time required to execute the task locally, T exc-IoT , at the IoT device. This time is calculated based on the currently available resources at the IoT device according to (1). Then, the decision engine makes the energy and time decisions of offloading, D Off-E-IoT and D Off-T-IoT . The energy decision is calculated by comparing the residual energy of IoT device after task execution, E C-IoT , with the threshold energy, E thr-IoT , of IoT device. The threshold energy is predefined for IoT devices in a way to prolong the lifetime of IoT devices. If the residual energy of the IoT device after the local execution is higher than the threshold level of energy, the energy offloading decision is set to zero, and the local execution is recommended from the energy perspective. For a positive offloading decision, the task is directly offloaded to the I-MEC server. For a negative energy offloading decision, the time offloading decision is calculated. The time decision of offloading, D Off-T-IoT, is calculated by comparing the execution time, T exc-IoT , with the QoS latency, T QoS , as in (6).
If the decision engine decides to offload, i.e., D Off-IoT = 1, the computing task is offloaded to the I-MEC server. The I-MEC server deploys the same server structure developed in [36]. The decision engine of the I-MEC server receives the offloading request from the IoT device and decides either to handle the task at the I-MEC server or to offload the task to the higher offloading level, i.e., C-MEC server. The I-MEC server's decision engine calculates the execution time required to execute the requested task, T exc-I-MEC , and then the total time required to handle it, T h-I-MEC [36].
The decision engine of the I-MEC server calculates the total energy consumed to handle the requested task at the I-MEC server, E h-I-MEC .
Then, the I-MEC server's decision engine calculates the binary energy and time decision values of offloading, D Off-E-I-MEC and D Off-T-I-MEC . The energy decision is calculated by comparing the residual energy, after handling the requested task, with the threshold level of energy, E thr-I-MEC , as presented in (13). However, the time decision is calculated by comparing the total time required to handle the requested task with the required QoS latency, T QoS [36,39].
For a negative offloading decision, i.e., D off-I-MEC = 0, the task is handled at the I-MEC server. However, for a positive offloading decision, i.e., D off-I-MEC = 1, the task is offloaded to the higher level, i.e., C-MEC server [36,40].
Once the C-MEC server receives an offloading request, the decision engine of the server performs similar calculations as the I-MEC server. At first, the total time and energy required to handle the requested task, T h-C-MEC and E h-C-MEC , is calculated.
The C-MEC server's decision engine makes the energy and time decisions of offloading, D Off-E-C-MEC and D Off-T-C-MEC , according to (21,22).
For a negative offloading decision, the task is handled at the edge server of cellular networks, i.e., C-MEC. Otherwise, the task is offloaded to the core networks cloud.

Energy-Aware and Latency-Aware Offloading Algorithm for D2D Networks
Algorithm 2 presents the developed energy-aware and latency-aware offloading algorithm for D2D networks deployed over a 5G cellular systems. The offloading algorithm introduces four levels of offloading besides local execution. Figure 5 illustrates the main offloading levels for D2D-based networks. The zero levels represent the local execution, where computing tasks are executed locally at the task generated device, e.g., LL, GW and HL. If the task-generated device has enough resources to handle the generated computing task within the announced QoS latency, the task is executed locally without any need for task offloading.
The C-MEC server's decision engine makes the energy and time decisions of offload-ing, DOff-E-C-MEC and DOff-T-C-MEC, according to (21,22).
For a negative offloading decision, the task is handled at the edge server of cellular networks, i.e., C-MEC. Otherwise, the task is offloaded to the core networks cloud.

Energy-Aware and Latency-Aware Offloading Algorithm for D2D Networks
Algorithm 2 presents the developed energy-aware and latency-aware offloading algorithm for D2D networks deployed over a 5G cellular systems. The offloading algorithm introduces four levels of offloading besides local execution. Figure 5 illustrates the main offloading levels for D2D-based networks. The zero levels represent the local execution, where computing tasks are executed locally at the task generated device, e.g., LL, GW and HL. If the task-generated device has enough resources to handle the generated computing task within the announced QoS latency, the task is executed locally without any need for task offloading. However, if there are no available resources for local execution, the task is offloaded to the higher offloading level. There are four available offloading levels. Each of them corresponds to an executing device. The first level represents the offloading to the GW device, while the second offloading level represents the offloading to the HL device. The third offloading level is the edge server, i.e., C-MEC server, while the last level is the core networks cloud. Figure 6 illustrates heterogenous task handling and offloading over a D2D network, using the proposed offloading scheme [18]. However, if there are no available resources for local execution, the task is offloaded to the higher offloading level. There are four available offloading levels. Each of them corresponds to an executing device. The first level represents the offloading to the GW device, while the second offloading level represents the offloading to the HL device. The third offloading level is the edge server, i.e., C-MEC server, while the last level is the core networks cloud. Figure 6 illustrates heterogenous task handling and offloading over a D2D network, using the proposed offloading scheme [18]. Algorithm 2 Latency-aware and energy-aware offloading algorithm for D2D-based networks.
The offloading algorithm for D2D-based networks has similar steps as that of IoT networks. The end device, e.g., the LL device, decides either local execution or offloading. The decision engine of the end device calculates the required time and energy for handling the task locally, Texc-LL and EL-exc-LL, based on the currently available resources.
For a positive offloading decision, the LL device offloads the computing task to the first offloading level, i.e., GW device. The GW device receives the offloading request and The offloading algorithm for D2D-based networks has similar steps as that of IoT networks. The end device, e.g., the LL device, decides either local execution or offloading. The decision engine of the end device calculates the required time and energy for handling the task locally, T exc-LL and E L-exc-LL , based on the currently available resources.
For a positive offloading decision, the LL device offloads the computing task to the first offloading level, i.e., GW device. The GW device receives the offloading request and processes it. The decision engine of the GW device decides either to handle the offloaded task at the GW device or to offload it to the higher level, i.e., the HL device. GW performs similar steps as LL device to make the decision according to (30-37) Electronics 2021, 10, 910 16 of 28 If the GW device fails to handle the requested task, it offloads the task to the nearby HL device. HL device receives the offloading request and processes it to either handle the requested task or offload it to the C-MEC server. The decision engine performs similar steps as LL and GW devices to make the energy and time decisions of offloading.
For a positive offloading decision of the HL device, i.e., D Off-HL = 1, the task is offloaded to the C-MEC server at the edge of the RAN.

Energy-Aware and Latency-Aware Offloading Algorithm for Cellular Network
Once a C-MEC server receives an offloading request from any connected devices, it makes certain processes based on Algorithm 3 to decide whether to handle or offload the received computing task. The C-MEC server has the same structure proposed in [36]. At first, the profiler extracts the task data from the received request. These data include the QoS latency, T QoS , the task size in bits, Z, and the communication latency, T comm , defined by the connected base station. The profiler passes these data to the C-MEC controller, which also receives the information about the current energy level and the available resources of the C-MEC from the resource scheduler. The controller takes its decision of handling or offloading the requested task based on the received data. The C-MEC controller calculates the total energy consumption for handling the task, E h-C-MEC , and then calculates the energy level of the C-MEC, E C-C-MEC , if the task is handled. If the remaining energy, E C-C-MEC , after handling the task, is higher than the predefined threshold level of energy of C-MEC, E thr-C-MEC , the controller then checks the latency to make the time decision of offloading. However, if the remaining energy is less than the pre-announced threshold level, the controller decides to offload the computing task to the core network cloud.
Algorithm 3 Latency-aware and energy-aware offloading algorithm for mobile edge computing (MEC)-based cellular networks.

Performance Evaluation
In this part, the proposed system and the developed algorithms are evaluated over reliable environments. Various scenarios are considered to indicate the performance. The section is divided into two subsections; simulation setup and simulation results. In the first subsection, the main steps of the simulation setup process are introduced, and the simulation parameters are defined. The considered simulation scenarios are well defined. The second subsection discusses the obtained results and the considered performance metrics.

Simulation Setup
For performance evaluation of the proposed system and the comprised algorithms, an event-driven simulator is developed based on [13,18]. The simulator is implemented using Java. The system is simulated for the topology illustrated in Figure 7. The IoT network considered for the simulation purposes is the LoRaWAN, which represents the most common and efficient LPWAN network [40]. LoRaWAN is a wide range IoT network that has been used widely for many applications [41]. The main reason for considering LoRaWAN as the IoT network in the simulation is the market use since LoRa is the most used IoT network [42]. The gateway of the LoRaWAN networks is connected to an I-MEC server to enable the edge services. The main characteristics of the considered LoRaWAN and the connected I-MEC server are indicated in Table 3.    A cellular network of one cell is considered for simulation purposes, with a base station centered at the cell. The base station is connected to the C-MEC edge server to provide computing capabilities to the end-users of the cellular cell. Heterogeneous devices, e.g., smartphones and tablets, are distributed randomly among the cell with the specifications introduced in Table 4. All these devices are assumed to support a D2D interface, wi-fi direct. Wi-fi direct is selected because it is the most common D2D interface with many advantages [43].
The full battery energy of heterogeneous devices of D2D networks and 5G cellular networks is selected as that of real existing devices. HL devices are assumed to be powerful smartphones, e.g., iPhone x and tablets. These devices have a lithium-ion battery (Li-ion) with a capacity of around 3000 mAh at a rate of 3.80 volt-DC. Thus, the full battery energy of such devices is around 20 KJ, and similar steps are performed for the better assumption of the full battery energy of GW and LL devices. The channel parameters are set as in [13]. The core network deploys a core network cloud with the SDN networks. The SDN networks are deployed with the same specification as in [13]. The considered OpenFlow version in the simulation process is OpenFlow 1.4 [44,45].
For IoT networks, each IoT device is assumed to have a computing task from the defined computing tasks, illustrated in Table 5. The assumed tasks are heterogeneous and selected to cover a wide range of IoT applications. Table 6 illustrates the specifications of LL devices' tasks. Each LL device is assumed to have one of the computing tasks introduced in Table 6. Tables 7  and 8 introduce task specifications of GW and HL devices, respectively. Each GW device is assumed to have a computing task from the tasks presented in Table 7, while HL devices have one task for each device; from the tasks presented in Table 8. All considered computing tasks are equivalent to the workloads of real tasks, such as processing web pages.  Two main scenarios are considered for the performance evaluation. In the first scenario, the effect of QoS latency, T QoS , is checked. Three cases are considered for this case; in each case, a certain value of QoS latency is assumed. The first case, case 1, represents the minimum latency, which corresponds to the ultra-low latency applications. In the second case, case 2, the QoS latency is assumed to be higher than that of case 1. This facilitates local executions, even if the resources are limited. The last case, case 3, is set for the highest value of QoS latency. These cases are deployed for both IoT, and D2D enabled 5G cellular cells.
The second scenario is introduced to evaluate the effect of introducing MEC servers to the edge of the IoT networks and also to the edge of the 5G cellular system. Two main cases are considered in this scenario; the first case, case I, is introduced to evaluate the effect of introducing I-MEC servers. In this case, the I-MEC servers are removed, and the percentage of blocked tasks is measured without deploying I-MEC servers. Furthermore, the percentage of blocked tasks is remeasured with the proposed MEC-based structure, where the I-MEC and C-MEC servers are deployed. The percentage of blocked tasks are compared in both systems to evaluate the effectiveness of introducing I-MEC servers.
The second case, case II, is considered for evaluating the effectiveness of offloading computing tasks to GW and HL devices. Furthermore, the effect of adding C-MEC server for D2D-based 5G system. Four systems are considered in this case. The first, system A, is the proposed system structure, with GW, HL and C-MEC. The second system, system B, is the same as the proposed structure, without deploying GW devices. GW devices are considered LL devices. The third system, system C, removes HL devices and all other devices and structures remain the same. For this system, all GW and HL devices are considered LL devices. In the last system, system D, C-MEC server is removed, and all devices are considered to be LL devices.

Simulation Results and Discussion
The results of the first scenario are introduced in Figures 8-13. Figure 8 presents the time required to handle tasks for IoT devices. Each device is assumed to have one task from tasks indicated in Table 5. These tasks are handled either locally, i.e., at IoT device, or offloaded to edge servers, I-MEC and C-MEC. The average time required to handle each task is presented in Figure 8 for the three considered cases. As the QoS latency increases, i.e., moving from case 1 to case 2 and then case 3, the probability of offloading increases. This is to save the IoT devices' energy and prolong the lifetime of devices. Using the proposed structure, with MEC and SDN, all tasks are handled without any blocking, even with the existence of QoS latency constraints.  Figure 9 provides the average latency for handling tasks of LL devices. Each LL device is assumed to have a computing task with the workload indicated in Table 6. A part of these tasks is executed locally at LL devices, while the rest of these tasks are offloaded to nearby GW and HL devices. The offloaded tasks are due to either energy constraints or lake of local computing resources. Figures 10 and 11 present the average latency of GW and HL devices' handling tasks for the three considered cases.   Figure 9 provides the average latency for handling tasks of LL devices. Each LL device is assumed to have a computing task with the workload indicated in Table 6. A part of these tasks is executed locally at LL devices, while the rest of these tasks are offloaded to nearby GW and HL devices. The offloaded tasks are due to either energy constraints or lake of local computing resources. Figures 10 and 11 present the average latency of GW and HL devices' handling tasks for the three considered cases.
(a)     For better discussing the results of the first scenario, the total number of handled tasks at each type of device is calculated. Figure 12 indicates the total number of handled tasks, at IoT devices, LLs, GWs, HLs, I-MEC servers and C-MEC servers, for the three considered cases. In the third case, the QoS latency increase, which provides a time for offloading and thus, the probability of offloading increases. This reduces the total number of local executed tasks in case 2 and case 3 compared to case 1. For the I-MEC and C-MEC, the number of handled tasks increases for cases 2 and 3 than that in case 1. This is because the number of offloaded tasks is increased due to decreasing latency constraints. Furthermore, the total number of local executed tasks and offloaded tasks are calculated to better indicate the effectiveness of the proposed structure. Figure 13 illustrates the total number of the local executed tasks and offloaded tasks for the three cases of the first simulation scenario. The number of offloaded tasks increases as the QoS latency increases, i.e., moving from case 1 to case 2 and then to case 3. This unloads end-devices in heterogeneous considered networks and efficiently makes use of network resources.   For better discussing the results of the first scenario, the total number of handled tasks at each type of device is calculated. Figure 12 indicates the total number of handled tasks, at IoT devices, LLs, GWs, HLs, I-MEC servers and C-MEC servers, for the three considered cases. In the third case, the QoS latency increase, which provides a time for offloading and thus, the probability of offloading increases. This reduces the total number of local executed tasks in case 2 and case 3 compared to case 1. For the I-MEC and C-MEC, the number of handled tasks increases for cases 2 and 3 than that in case 1. This is because the number of offloaded tasks is increased due to decreasing latency constraints. Furthermore, the total number of local executed tasks and offloaded tasks are calculated to better indicate the effectiveness of the proposed structure. Figure 13 illustrates the total number of the local executed tasks and offloaded tasks for the three cases of the first simulation scenario. The number of offloaded tasks increases as the QoS latency increases, i.e., moving from case 1 to case 2 and then to case 3. This unloads end-devices in heterogeneous considered networks and efficiently makes use of network resources.  Results for the first case, case I, of the second scenario are introduced in Tables 9 and  10. Table 9 presents the percentage of blocked tasks of the considered LoRaWAN networks with and without edge computing servers, i.e., I-MEC and C-MEC. This is for the three considered values of QoS latency. Results indicate that the percentage of blocked tasks falls from 30% for LoRaWAN without edge computing servers to zero with the deployment of edge computing servers and offloading algorithm. Furthermore, the introduction of edge servers not only improves the percentage of blocked tasks but also improves the overall energy efficiency of LoRaWAN networks through the deployment of  Figure 9 provides the average latency for handling tasks of LL devices. Each LL device is assumed to have a computing task with the workload indicated in Table 6. A part of these tasks is executed locally at LL devices, while the rest of these tasks are offloaded to nearby GW and HL devices. The offloaded tasks are due to either energy constraints or Electronics 2021, 10, 910 25 of 28 lake of local computing resources. Figures 10 and 11 present the average latency of GW and HL devices' handling tasks for the three considered cases.
For better discussing the results of the first scenario, the total number of handled tasks at each type of device is calculated. Figure 12 indicates the total number of handled tasks, at IoT devices, LLs, GWs, HLs, I-MEC servers and C-MEC servers, for the three considered cases. In the third case, the QoS latency increase, which provides a time for offloading and thus, the probability of offloading increases. This reduces the total number of local executed tasks in case 2 and case 3 compared to case 1. For the I-MEC and C-MEC, the number of handled tasks increases for cases 2 and 3 than that in case 1. This is because the number of offloaded tasks is increased due to decreasing latency constraints. Furthermore, the total number of local executed tasks and offloaded tasks are calculated to better indicate the effectiveness of the proposed structure. Figure 13 illustrates the total number of the local executed tasks and offloaded tasks for the three cases of the first simulation scenario. The number of offloaded tasks increases as the QoS latency increases, i.e., moving from case 1 to case 2 and then to case 3. This unloads end-devices in heterogeneous considered networks and efficiently makes use of network resources.
Results for the first case, case I, of the second scenario are introduced in Tables 9  and 10. Table 9 presents the percentage of blocked tasks of the considered LoRaWAN networks with and without edge computing servers, i.e., I-MEC and C-MEC. This is for the three considered values of QoS latency. Results indicate that the percentage of blocked tasks falls from 30% for LoRaWAN without edge computing servers to zero with the deployment of edge computing servers and offloading algorithm. Furthermore, the introduction of edge servers not only improves the percentage of blocked tasks but also improves the overall energy efficiency of LoRaWAN networks through the deployment of the developed offloading algorithm implemented with the developed edge structure. This is reasonable since IoT devices make use of edge computing resources. Table 10 presents the percentage of the average remaining energy, compared to start energy, for LoRaWAN networks with and without edge system with the offloading algorithm; this is after handling the considered tasks in the first scenario for the three considered values of QoS latency. This implies an average improvement of energy efficiency of 24% after deploying edge computing servers with the developed offloading algorithm. Table 9. Percentage of blocked tasks of IoT networks in case (II) of the second scenario.

Case (1) Case (2) Case (3)
IoT network (without edge) 38% 28% 24% IoT network (with edge) 0 0 0  Figure 14 presents results for the second case, case II, of the second scenario. The number of blocked tasks for each considered system is recorded, as illustrated in Figure 14. Table 11 provides the percentage of blocked tasks for each system in each case. As results indicate, the percentage of blocked tasks of the proposed system, system A, is zero, and it increases as the edge devices are cut. System D achieves the worst results in terms of task blocking since the C-MEC server is removed and edge devices of D2D networks are considered as LL devices that send to the base station. For each system, the percentage of blocked tasks decreases as the QoS latency increases since this gives more time for task handling either by offloading or by limited resources. framework can be used for dense deployment scenarios with higher network scalability. This fit great and efficient for many applications, especially the announced dense deployment applications for next-generation networks. These applications include smart city applications, smart homes and industrial 4.0. All these applications have a challenge of dense deployment and a massive amount of traffic. Integrating IoT with a 5G system provides a way for realizing such applications with the required QoS and scalability requirements.

Conclusions and Future Work
Heterogeneous IoT networks can be integrated with 5G systems via MEC servers, using the proposed system structure. Connecting IoT gateways with MEC servers, I-MEC servers provide paths for 5G edge servers, C-MEC servers. Furthermore, the introduction of edge computing servers to the access networks of heterogeneous IoT networks provides computing resources to IoT devices, which unloads the core networks. This achieves many benefits in terms of network scalability, availability and reliability. The article also introduces D2D communications over cellular cells to unloads the core networks and increases the system availability and scalability. Computing tasks can be managed, over the proposed IoT/5G networks, using the proposed energy-aware and latency-aware offloading scheme. With the proposed IoT/5G structure, the proposed offloading scheme, achieves 30% improvement in blocked tasks and 20% improvement in energy efficiency. Thus, the proposed IoT/5G structure, with the proposed offloading scheme, achieves IoT's main requirements over 5G systems announced by the 3GPP and ITU.
The proposed IoT/5G framework is limited by the capabilities of the I-MEC and C-MEC servers. Higher server capabilities, e.g., energy, processing and storage, achieve higher performance in terms of scalability, availability and energy efficiency; however, this comes on the overall system cost. Thus, our future goals for this work include developing an optimization algorithm for optimizing and allocating resources dynamically and presents the edge intelligence by developing machine-learning algorithms at the edge servers to improve the overall network performance and provide novel services. Moreover, further measures will be examined for key performance indicators (KPIs), such as overall system reliability.  In summary, results indicate that the introduction of MEC, D2D and SDN to IoT networks facilitates integrating IoT networks with 5G systems. Furthermore, the proposed framework can be used for dense deployment scenarios with higher network scalability. This fit great and efficient for many applications, especially the announced dense deployment applications for next-generation networks. These applications include smart city applications, smart homes and industrial 4.0. All these applications have a challenge of dense deployment and a massive amount of traffic. Integrating IoT with a 5G system provides a way for realizing such applications with the required QoS and scalability requirements.

Conclusions and Future Work
Heterogeneous IoT networks can be integrated with 5G systems via MEC servers, using the proposed system structure. Connecting IoT gateways with MEC servers, I-MEC servers provide paths for 5G edge servers, C-MEC servers. Furthermore, the introduction of edge computing servers to the access networks of heterogeneous IoT networks provides computing resources to IoT devices, which unloads the core networks. This achieves many benefits in terms of network scalability, availability and reliability. The article also introduces D2D communications over cellular cells to unloads the core networks and increases the system availability and scalability. Computing tasks can be managed, over the proposed IoT/5G networks, using the proposed energy-aware and latency-aware offloading scheme. With the proposed IoT/5G structure, the proposed offloading scheme, achieves 30% improvement in blocked tasks and 20% improvement in energy efficiency. Thus, the proposed IoT/5G structure, with the proposed offloading scheme, achieves IoT's main requirements over 5G systems announced by the 3GPP and ITU.
The proposed IoT/5G framework is limited by the capabilities of the I-MEC and C-MEC servers. Higher server capabilities, e.g., energy, processing and storage, achieve higher performance in terms of scalability, availability and energy efficiency; however, this comes on the overall system cost. Thus, our future goals for this work include developing an optimization algorithm for optimizing and allocating resources dynamically and presents the edge intelligence by developing machine-learning algorithms at the edge servers to improve the overall network performance and provide novel services. Moreover, further measures will be examined for key performance indicators (KPIs), such as overall system reliability.

Conflicts of Interest:
The authors declare no conflict of interest.