Resource Management in SDN-Based Cloud and SDN-Based Fog Computing: Taxonomy Study

Abstract

Software-defined networks (SDN) is an evolution in networking field where the data plane is separated from the control plane and all the controlling and management tasks are deployed in a centralized controller. Due to its features regarding ease management, it is emerged in other fields such as cloud and fog computing in order to manage asymmetric communication across nodes, thus improving the performance and reducing the power consumption. This study focused on research that were conducted in SDN-based clouds and SDN-based fogs. It overviewed the important contributions in SDN clouds in terms of improving network performances and energy optimization. Moreover, state-of-the-art studies in SDN fogs are presented. The features, methods, environment, dataset, simulation tool and main contributions are highlighted. Finally, the open issues related to both SDN clouds and SDN fogs are defined and discussed.

1. Introduction

Traditional networks rely on network devices to make forwarding and routing decisions by implementing hardware tables that are embedded to the device itself, such as a bridge and router. As well, traffic rules such as filtering and prioritizing are implemented locally in the device. However, software-defined networks (SDN) brought advancement in networking by addressing simplicity, as it aims to reduce the design complexity involved in implementing both the hardware and software of network devices.

The basic concept of SDN is to remove the controllability from network devices and allow one central device, i.e., control unit to control and manage. The control unit is capable of observing the entire network and making decisions regarding forwarding and routing, whereas the task of forwarding is handled by network hardware devices in addition to filtering and traffic prioritization [1]. OpenFlow is a well-known SDN design that follows its basic architecture of decoupling control from the data plane where the controller and switches are communicating through OpenFlow protocols. Switches, on the other hand, contain flow tables and flow entries such as matching fields, counters and sets of actions, while the controller is capable of performing sets of actions on flow entries, such as update, delete and add [2].

Even though cloud computing is a powerful technology, there are some issues and challenges that are still open, such as network mobility, scalability and security issues; thereafter, the possibility to extend software-defined networks in cloud computing has been investigated [3]. Although fog computing brought solutions to handle IoT applications that requires less latency and a higher bandwidth to process data from cloud server to end devices. However, end devices have limited resources that conflict with high demands, besides heterogeneity, which is another challenge introduced in fog computing [4]. Therefore, the current studies are developed for the sake of bringing advantages of SDN to both cloud and fog computing in order to overcome the above-mentioned issues by offering flexible and intelligent resources management solutions.

Furthermore, resource management is a challenging issue, as the difficulty lies with its relatedness to several problems, such as resource heterogeneity, asymmetric communication, inconsistent workload and resources dependency [5,6]. Therefore, this study reviews related research and its contribution to SDN-based cloud in terms of network performance, and energy efficiency. Additionally, it presents state-of-the-art contributions in the field of emerging software defined networks and fog computing. This review contributes to two of the most promising fields in networking; its importance lies in providing a comprehensive review of important and state-of-the-art studies related to SDN cloud and SDN fog. Additionally, it helps researchers to observe related contributions by assessing relative important key information of a specific study. Thus, it can be used as reference for researchers based on their area of concern; since this review classifies novel findings and summarizes it based on several criteria that help researchers to identify, select and contribute to SDN-based cloud/fog accordingly. Moreover, each reviewed research is mapped with corresponding evaluated metrics for ease of configuration of correlated studies based on common interest.

The remainder of this review is organized as follows: Section 2 overviews the methodology of our literature review. Section 3 presents SDN-based cloud resources management solutions that contribute in network performance enhancements. While SDN-based cloud research that contribute in energy efficiency optimization are overviewed in Section 4. Furthermore, Section 5 shows state-of-the-art contributions in SDN-based fog. Section 6 presents open issues of both SDN-based clouds and SDN-based fogs. Finally, Section 7 presents the concluding remarks.

2. Methodology of the Literature Review

This section presents the methodology used for conducting our review. It follows a systematic review where the reviewed studies were either journal articles or conference papers published in IEEE Xplore, Springer, and ScienceDirect datasets. The keywords used for searching include: “SDN + cloud + resources + management”, “SDN + cloud + resources + allocation”, “SDN + cloud + Virtual Machine (VM) + placement” and “SDN + cloud + VM + migration”. As well, “SDN + fog + resources + management”, “SDN + fog + resources + allocation”, “SDN + fog + VM + placement” and “SDN + fog + VM + migration”. After that, results are filtered based on time period from 2019 to 2021. However, please note that we also reviewed the mostly mentioned baseline works related to the SDN cloud to be used as future references. After that, selected researches are analyzed thoroughly and summarized based on a major feature of the research, main tasks of the proposed algorithm, targeted environment of research, main contribution, evaluation tool, dataset and metrics used for evaluation.

3. SDN-Based Cloud Resource Management in Network Performance Improvement

Regarding network performance, there are many algorithms that target different aspects of network performance metrics, such as response time, bandwidth, throughput and other quality of service (QoS) measures in different environments. This section is subdivided based on the focus of the design of resource management algorithms to be either based on QoS constraints, priority-aware, and QoS-aware algorithms. The last section concentrates on VM migration algorithms.

3.1. Resources Management Based on QoS Constraints

The MAPLE system [7] is developed in order to manage resource allocation for both network and computing resources. It is uses effective bandwidth estimation based on server’s traffic traces analysis. Effective bandwidth estimation is used to specify needed bandwidth to process requests without a violation of QoS constraints. The system is composed of a centralized controller, which processes and manages requests, and effective bandwidth agents on servers. The controller performs preprocessing of VM requests by defining QoS targets and resembling incoming requests to be placed as one VM, calculates residual bandwidth estimations and then places VM. After that, network-aware VM placement algorithm is executed, which allocates VM in first fit decreasing approach in order to address the problem of consolidation while achieving QoS constraints. The system is evaluated in emulated virtual datacenter and compared to two network resources algorithms which are Oktopus [8] based on expected mean and expected peak throughputs. Maple managed on achieving lower QoS constraint (delay) than the expected value, as well as allocating jointly resources efficiently. Since the MAPLE system succeeded on maintaining QoS constraints at lower rates than expected, an extension, MAPLEx [9], is developed to further incorporate anti-colocation problem. The algorithm takes an extra input that is number of VM slots that are preserved for a particular request for VM placement. The algorithm is evaluated in an extended testbed compared to MAPLE evaluation. As well, the dataset that used for evaluation consist of three types that are data intensive applications, data serving applications, and data backup applications. The results show that MAPLEx is able to attain satisfied QoS levels while locating network and cloud resources. In addition to MAPLEx, Maple is developed for SDN environments to be MAPLE-Scheduler [10], which is a built-in flow scheduling unit in SDN controller in order to maintain QoS constraints when it is triggered. It is developed for Maple systems in software defined networks. The main feature of Maple-scheduler is that it is capable of observing network status and, in the case of a QoS violation occurs, it reschedules flows dynamically using effective bandwidth estimation (EB) of the top of rack switches, and the effective bandwidth coefficient (EBC) estimation of core switches. Moreover, it uses monitoring agents that are distributed over network (servers and edge switches) in order to acknowledge MAPLE-scheduler regarding the topology of a network, as well as the flow used resources. Basically, flows rescheduling take place when there is violation in QoS for large flows, and the decision will be made based on existing residual bandwidth and maintain load fairness by selecting links with minimum residual bandwidth. Evaluation is conducted against the Equal-Cost Multi-Pathing protocol ECMP, which is a routing protocol that selects the best paths to forward a packet through [11]. The comparison is based on different QoS constraints scenarios. The results show that Maple-scheduler achieves lower QoS violation rates compared to ECMP, as well as maximizes the throughput in all scenarios.

On the other hand, establishing a fair charging model is the motivation behind developing Oktopus [8]. It is proposed to solve unpredictability in network performance in cloud environment; by pursuing a solution that guarantees both parties rights: providers and tenants from perspectives of cost, profits, and QoS constraints. Therefore, virtual network abstractions are established and offered to tenants instead of computing resources by selecting number of VMs, types of VMs, and virtual network connecting them. Hence, virtual network abstraction decouples tenants from infrastructure. Oktopus is an implemented system which is evaluated by simulations and testbed. The results show that datacenters can reduce the overall cost by charging for internal traffics in virtual networks which leads to reduced tenants’ costs and maintained providers’ profit while concurrently preserve performance. Furthermore, to solve the problem of big data processing, resource allocation algorithm is developed by combining SDN and virtual machines, specifically complex event processing machines CEP [12]. Individual CEP contains a load balancer that works dynamically based on data stream arrival. While a performance optimizer works as a load distributor between all CEPs. Resource allocation is performed by taking into consideration three factors, which are: size of data, time and virtual machine. After that, processing big data is performed by considering individual packets. Processing is composed of three parameters, which are: data packet deadline, amount of work, and required time to reach virtual processer. Finally, job scheduling is performed in FIFO manner. The evaluation is conducted in CloudSim simulation tool with SDN setting. Various types of data are used for testing and time is the evaluation metric. Results show the effectiveness of using CEP and SDN in reducing required time to process data stream.

3.2. Priority-Aware Resources Management

This subsection presents algorithms that focused on priority and methods to assign privilege to high priority requests, such as in [13], proposed a prioritization model in cloud environment that uses the crystalline mapping CM algorithm. Main idea is that multiple tenants initiate service requests with a declared priority information, then it will be sent to a central master server. The master server job will prioritize arrived request queues by considering several attributes, which are fairness and maximum revenue. Thus, each tenant will prioritize requests locally first as the initial phase by following the multivariate likelihood ratio algorithm and then sorts the requests based on their scores. Then, the master server will prioritize requests queues from multiple tenants by using the crystalline mapping algorithm, which preserves fairness, isolation and maintain requests orders. The algorithm will maintain requests priority queues by calculating global priority level. After that, from priority queues, the request of maximum estimated revenue is selected for processing. The framework is evaluated in simulation environment, and the evaluation results show that average waiting time, which was declared as fairness property, was almost the same for all processed requests. While requests of maximum revenue were processed first by CM model compared to baseline work.

Whereas, in [14], the authors developed a client’s priority-based resources provisioning model, high or low, in which a cloud provider offers pool of resources for clients. However, some of these resources are provisioned for only high priority clients. Thus, cloud providers increase profit by maintain SLA for such clients whenever there is demand on service at any given time. On the other hand, low priority clients will get access to shared resources, which also will meet their SLA contract. The authors calculated rejection probability, which indicates the probability of rejected clients’ requests in such a way that high priority clients supposed to get the least rejected requests compared to low priory clients. The proposed work was evaluated in simulation environment. They conclude that, the rejection probability shows that number of high priority clients rejected requests is ten times less than rejected requests of low priority clients.

As well, in [15], the resource allocation algorithm was proposed in the software-defined cloud environment SDCC, which considers an application priority as main consideration when placing VMs and provision network bandwidth. An application is previously determined to be either critical or normal in priority criterion. Submitted application requests contain information about VM and flow, which include number of processing cores, core’s capacity, amount of memory, storage size, required bandwidth, source and destination VM, and application priority. However, when application requests are placed, the system will allocate host and network resources based on provided information. Firstly, the system will determine the network topology and hosts and VM grouping decisions is to be made based on hosts connectivity and application, respectively. Then mapping of VM to hosts is performed based on the highest amount of joint resources taking into consideration host connectivity to be either on single group or within closest proximity in order to minimize network traffic. If an application is characterized as critical, then network bandwidth is provisioned by SDN controller via priority queues on link switches. The proposed algorithm is compared with exclusive resource allocation, random allocation, and combination of fit first decreasing and dynamic flow algorithms [7] in the CloudSimSDN simulation tool against the response time and power consumption in two different scenarios (synthetic workload and Wikipedia workload). The proposed algorithm manages to reduce response time in both scenarios. While power consumption is reduced compared to most baseline algorithm for critical workloads. However, there is not significant reduction in power consumption in the case of Wikipedia workloads which indicates that the proposed algorithm is not power demanding. The proposed algorithm fits properly with the critical applications in synthetic workload for response time reduction since bandwidth provisioning algorithm is only triggered when network is intensive with workloads.

3.3. QoS-Aware Resources Management

While some algorithms consider priority as main challenge, others concentrate more on establishing QoS aware algorithms, such as the quality of service aware virilization-enabled routing (QVR) algorithm [16]. It is proposed for SDN environment where isolated slices of physical infrastructure are created, and each slice is dedicated for several tenants. It aims to dynamically support flow allocation and provision end-to-end QoS constraints fulfillment. Performance evaluation shows that QVR capable of reducing number of shared links while accomplishing lowest link congestion rates compared to baselines. In addition to that, EQVMP [17] is the VM placement algorithm, which aims to preserve low power consumptions and QoS-aware method. It combines three techniques to achieve desirable results. The input consists of VM resource demands, topology matrix and VM traffic. After that, EQVMP will reduce number of hops by maintain same number of VMs in multiple groups to reduce cost of traffic load. Then, the VMs are to be sorted in decreasing order based on total resources needed, then mapping VMs to suitable servers take place by selecting a VM, which is best fit to a specific server. In case there is not an ON-server able to accommodate the VM demands, then a new server will be woken up. Finally, load balancing feature in SDN controller is utilized to prevent link congestion by selecting substituted links in case of overutilization. EQVMP achieves nearly optimal results in power savings, however, average delay is high. Nevertheless, it succeeds on achieving outstanding throughput compared to other baselines and this is the reason this algorithm is located under network performance section.

3.4. Contributions on VM Migration Task in Resources Management

Although most of above-mentioned algorithms focused on VM placement, others consider VM migration as part of resource management, which targets QoS such as in [18], authors developed a live VM migration, S-CORE in SDN environment, which is derived from S-CORE in clouds [19]. S-CORE takes into consideration colocation constraints in order to minimize overall communication cost by assigning links with the ratio of over-subscription, as well the cost of bandwidth as the weights. Then, using temporal bandwidth utilization multiplied with aggregate weightings, overall cost is obtained. The system architecture consists of server virtualization to manage VMs through an open-source API; Libvert. While the software switch is Open vSwitch at the hypervisor level and SDN controller with OpenFlow specification. Moreover, the controller modules include topology discovery, host discovery, links weights, flow statistics and REST API with certain endpoints for controllers to migrate VM, depending on the network state. The proposed work is evaluated using simulation: NS3 and testbed, where, link utilization, overall communication cost, VM–VM communication cost, throughput, scalability, and Number of migrations are the evaluation metrics. The results show that there is a significant decrease in congestion, and the overall throughput is enhanced. Furthermore, this algorithm is proposed in clouds datacenters through tokens, which hold information such as the VM ID, communication level value to determine communication cost. As well, each VM has the token will have an advantage of determining the next VM to pass the token based on specific policy such as round robin.

While Remedy [20] is proposed to manage VM migration by managing steady states. It is composed of three components that are bandwidth monitor, which will send VMs list to VM and target selector that determines a host to migrate VMs and VM memory monitor. The system is evaluated in testbed using real datacenter traces that shows the effectiveness of the Remedy system in reducing the unsatisfied bandwidth.

The following table (

Table 1. Summarization table of contributions in network performance.

Ref.	Major Feature	Main Tasks	Env.	Dataset	Eval. Tool	Measured Metrics	Contribution
[7]	Empirical estimation of effective bandwidth using MAPLE controller and EB agents.	VM ensemble placement based on effective bandwidth—VMs are placed on same server (MAPLE)	DCs	300MB dataset as bulk data. Tenants requests	Emulated virtual DCN	QoS overall and average violation rate: proportion of delayed packets.	Placing new VMs while maintaining bandwidth limits Preserve QoS targets under violation rate
[8]	Virtual Networks abstractions	Predictable performance using virtual network abstraction Oktopus system	Cloud datacenters Production datacenters	Data-intensive application Local communication pattern application	Simulation and testbed	Job completion time Utilization Rejected requests Cost and profit	Fair charging model
[9]	Empirical estimation of effective bandwidth	Enable ensemble VM placement with anti-colocation constraint (MAPLEx)	DCs	Data backup application: based on SCP Data analysis application: word count, Hadoop. Data serving application: YCSB ad Cassandra as data client and server	Testbed	QoS violation: overall and average. Reject rate of ensemble placement requests. Link utilization.	Similar QoS violation rate with MAPLE with more rejection rate thus lower mean link utilization.
[10]	Centralized Flow scheduler using effective bandwidth with distributed monitoring agents Joint Allocation	VM placement: observe network changes Reschedule flows dynamically Maintain QoS of VM. Preserve link utilization (MAPLE Scheduler)	SDN	Data applications as bulk data transfer: word count Hadoop, data serving YCSB and SCP. Custom data backup applications.	Testbed	QoS violation rate: overall and average Throughput Link utilization	Significant decrease of overall QoS violation. Better median throughput
[12]	Use of complex event processing machines	Resource allocation, processing of big data stream, job schedulling, and load balancing	SDN-Cloud	Different types of data.	Cloudsim with SDN settings	Time	Effectiveness of both CEP and SDN in reducing time required to process data
[13]	Considers Local & global priority Prioritization model based on crystalline mapping CM algorithm	Statistical information model Local priority model Global crystalline mapping model Maximum Revenue Evaluation.	Clouds	Real world data collection	Simulation	Response Time Revenue	Greater revenue with maintained response time
[14]	Rejection Probability of two priority requests: high & low.	Resource provisioning for different priority requests	Clouds	Based on number of servers = 40,000.	Simulation	Rejection probabilities for different priority requests	Analytical determination of rejection probability for different priority classes for resources allocation problem.
[15]	Local prioritizing of requests (normal/ critical) Heterogeneous Joint allocation	VM allocation Bandwidth provision	SDN-Cloud Computing	Synthetic workload Wikipedia workload	CloudSimSDN	Power Consumption Response time	Reduced response time
[16]	Adaptive feedback management tool: QVR	Jointly virtualization and routing technique with Dynamic flow allocation based on QoS	SDN	Real time and non-real time application	Simulation	Delay Latency Number of shared links	Better tenants’ isolation with enhanced congestion latency and less delay.
[17]	Three-tier algorithm: Hop reduction Power saving Load balancing	Virtual Machine placement considers QoS and power saving: EQVMP	DCN	Not available	Simulation: NS2	Power consumption Average delay Throughput	Significantly increase of system throughput while acceptably decrease power consumption and average delay.
[18,19]	Assigning each link with a weight metric Using colocation and network locality	Live VM migration decoupled from SDN controller: S-CORE	SDN Cloud DC	Traffic is generated by Nping tool in SDN. DC traffic generator in clouds.	Testbed for SDN Simulation: NS3 for Cloud DCs.	Link utilization Overall communication cost and VM-VM communication cost Throughput Scalability Number of migrations	Significant decrease in congestion and enhance overall throughput.
[20]	Steady state management	VM placement: bandwidth monitor, target selector, and VM memory monitor: Remedy	DCN	Real datacenter traces	Simulation	Number of VM migration Percentage reduction in unsatisfied bandwidth Link utilization	Reduce unsatisfied bandwidth

) summarizes above mentioned network performance contributions in terms of major feature of algorithms, main tasks, the environments where the algorithm targets, which evaluation tool is used for testing and the dataset, evaluated metrics and, finally, the major contribution of the algorithm based on its published results. In addition to that,

Table 2. Measured metrics in contributions of the network performance.

Ref.	Power Consumption	Response Time	Qos Violation Rate	Throughput	Link/Network Utilization	Rejection Rate/ Rejected Requests	Delay	Cost	Profit/Revenue	Job/File Completion Time	Number of Shared Links	Latency	Communication Cost	Scalability	Number of Migration	Unsatisfied Bandwidth
[7]			●		●
[8]					●	●		●	●	●
[9]			●		●	●
[10]			●	●
[12]							●
[13]		●							●
[14]						●
[15]	●	●
[16]							●				●	●
[17]	●			●			●
[18]				●	●								●	●	●
[19]				●	●								●	●	●
[20]					●										●	●

, declares evaluation metrics mapped with corresponding research.

4. SDN-Based Cloud Computing Resource Management in Energy Efficiency

Many techniques were employed in order to minimize the power consumption. For example, in [21], the authors used dynamic overbooking of resources in software-defined data centers; dynamic overbooking is claimed to reduce power consumption for both hosts and network resources. Hosts and network resources are overbooked using a dynamic ratio that is determined based on current workloads, which results in the obligation of SLAs. SLA violation was measured based on response time, such that each request is measured in advance to obtain the response time under no overbooking strategy. After that, with proposed overbooking strategy, the number of requests that exceed the response time, which was previously measured, will be considered as a violation of the SLA. The performance is evaluated in the CloudSimSDN simulation tool using Wikipedia workloads as datasets. Extensive experiments were conducted that show that dynamic overbooking has an advantage over static overbooking in terms of decreasing the SLA violation rate and reduced power consumption.

On the other hand, FCTcon [22] deploys a flow completion time FCT for dynamic flow management especially for delay sensitive applications. It aims to decrease the power consumption in data center networks while guarantee the required FCT. It considers the burst nature of the flow. Basically, FTcon will receive requests with specified bandwidth requirements. Then, it will send it, with control knob = 1 to have their proposed bandwidth requirement, to traffic the consolidation unit that will determine what flow will be used, and the rest of flows will be set into sleep mode. After that, the FCT monitor unit will observe the FCT of requests and then acknowledge FCTcon controller in order to modify the control knob that will either be increased or decreased value, which, in turn, will indicate number of flows to be either wake up or sleep by traffic consolidation unit in order to accommodate the FCT. The proposed work is evaluated in testbed environment using Wikipedia and Yahoo traffic files. The results show that FTcon achieves the required request file completion time. Even though it succeeds on coping with the optimal power consumption levels, some of baselines manage to reduce power consumption below the optimal levels. Additionally, PowerFTC [23] works by considering FCT, especially for delay-sensitive flows in order to reduce the power consumption. In order to impose FCT during the consolidation process, it obtains flow paths and switch settings in advance. The evaluation, which is conducted in testbed and simulation, show that, loosely, FCT requirements lead to a higher power saving while the FCT miss ratio is the minimum in all cases.

While DISCO [24] is developed to optimize power consumption in the data center networks by utilizing traffic consolidation, it offers cloud providers with two algorithms, DISCO-F and DISCO-S, to be chosen from based on the most desirable objectives which are power efficiency, scalability and enhancing the delay performance of the data center network. Both Disco-F and Disco -S perform consolidations; however, the difference lies in where the consolidation takes place. DISCO-F performs individual flow consolidation, while DISCO-S performs multiple flow consolidation on individual switches. Using Wikipedia and Yahoo traces, the evaluation results show that, in terms of delay, DISCO algorithms manage to obtain lower delay than most of baseline algorithms. However, regarding power saving, both algorithms consume more power than most baseline algorithms.

Additionally, Elastic Tree [25] is a power optimizer that switches unused resources ON or OFF based on its current usability by monitoring traffic. Additionally, it targets fault tolerant and performance when making the decision of which subnet is supposed to be ON. It works by receiving traffic status by the optimizer, which is implemented with various methods depending on providers’ goals: power saving, performance and fault tolerant. After that, a subnet of topology and flow path are sent to control the software. The system is evaluated in the testbed and proved its capability of supporting performance and robustness while reducing the power consumption.

In addition to that, by analyzing traffic bandwidth usage in datacenter networks, CARPO [26] is proposed to reduce the power consumption through utilizing the minimum subnet of the network. By tracing Wikipedia and Yahoo traces, various traffic peak time and correlation are observed. This leads to conclude that correlation is either to be loosely or negative and consolidation of traffic value is suggested to be designed, depending on the on and off peak values. Therefore, CARPO works by analyzing correlation from previous period, then consolidates traffic based on link capacity and, finally, adapts the link rates. It is evaluated based on power saving, power consumption, packet drop ratio and delay using testbed and simulation. Results show that delay and packet drop ratio are considered higher than baselines; however, CARPO succeeds in power saving.

Moreover, DCTCP is proposed TCP-like protocol that is specified for datacenter networks [27]. It works by estimating cognition degree from fraction of marked packets extracted from Explicit Congestion Notification (ECN). Thus, DCTCP enrich ECN with control schema in order to reduce buffer space while maintain network performance. While a study showed that a scalable network topology; flattened butterfly with commercial switch chips provides full utilization with reduced energy proportional datacenter [28]. Whereas, in [29], VM placement is addressed by combining two approaches that are genetic and the tabue search algorithms to reduce power consumption in cloud environment. Tabue search algorithm is used to enhance genetic algorithm searching by performing mutation operations. GATA algorithm is evaluated using PyCharm with Google Cluster dataset. Results show that GATA manages to reduce power consumption while lowering load balance. Additionally, it gives highest number of optimal solutions for different size problems. However, execution time is affected as it falls in the third-best position compared to four baselines.

As well, HGAPSO is a multi-objective resource allocation based on Euclidean distance, which combines two algorithms: Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) for VM placement. It comprises three main phases: VM allocation, task scheduling, and VM migration. It focused on reducing the power consumption while increasing the resource utilization. The results of evaluation show that HGAPSO consumes more energy and less resources (CPU, RAM and storage). However, it works much better when combined with VM migration in term of both energy consumption and resource utilization [30]. Moreover, multi-objective virtual cluster allocation framework is proposed in [31] based on four tradeoff models, which are VM availability, saving energy, resource utilization and load balance. Then IBBBO algorithm is developed based on natural evolutionary algorithm in which weights are assigned for previous models as a tradeoff relationship. It is tested using CloudSim where power consumption, average resources utilization, risk value and load balance value are measured. Results show that IBBBO manages in obtaining lower risk value, less power consumption, better average resources utilization with the least load balance value compared to most of the baselines.

In addition to that, several algorithms are proposed to find the acceptable balance between energy consumption and SLA violation, such as in [32], a framework for resource management that considers VM placement in such a way that SLA is maintained, and the energy consumption is reduced. It is composed of three modules responsible of cluster management including VM placement, overload detection of each PM CPU and resource prediction for each VM. The number of overloads and number of SLA violation, and power consumption are evaluated. Results show that a smaller number of overloads occurred with an outperforming on minimizing SLA violation. While for power consumption is almost similar with baseline. While a simple linear regression prediction model RobustSLR [33] is proposed to determine the utilization of server’s resources (CPU) and verifies whether it is underloaded or overloaded, as well predicts future utilization of resources. Moreover, eight migration policies are proposed and evaluated. It aims to reduce power consumption as well SLA violation. evaluation is conducted in CloudSim simulation tool with two types of workloads: random and real world (PlanetLab data). Power consumption, SLA violation, overall and average SLA violation are measured. Most proposed algorithms consume less power while violating less SLA compared to baselines. However, one algorithm is an exception, although, it consumes similar amount of power energy compared with baselines but outperforms in reducing SLA violation.

Similarly, predictive anticorrelated VM placement algorithm PACPA [34], which considers predictive resources prior to VM migration such that future demand is investigated in advance. It finds the suitable balance in terms of CPU utilization in order to perform VM placement. Therefore, two VMs are placed if their combined CPU requirement match the maximum demand of a specific host in order to utilizes resources. Moreover, the algorithm uses neural networks as predictive technique in order to dynamically investigate future VM placements. PACPA is evaluated in CloudSim based on energy consumption and SLA violation. It accomplishes best results regarding reducing power consumption with smaller SLA violation compared to baselines. In [35], VM consolidation algorithms are proposed in order to reduce power consumption by taking into consideration not only CPU as a major power consumption element but, also, RAM. The proposed work consists of two managers: global and centralized manager and server level local manager. Then, two algorithms are developed for VM selection and placement to avoid overloaded servers to reduce power consumption as well reduce SLA violation. Moreover, it utilized two thresholds to detect overloaded and underloaded hosts. An evaluation is conducted using CloudSim simulation tool considering power consumption, SLA violation, and performance degradation after migration with static and dynamic threshold as evaluation metrics. The results show that the dynamic threshold brought better results to proposed algorithms especially with power consumption and performance degradation after migration. However, the SLA violation rate is higher than baselines in both thresholds.

On the other hand, some research focused only on power consumption. Several experiments of different policies: FCFS; SJF; different power management techniques (non-power aware NPA, power aware PA, dynamic voltage and frequency scaling DVFS), as well as different power models (linear, square, cubic and square root) are conducted in [36]. The results show that DVFS techniques consume the least power in both allocation policies FCFS and SJF. Moreover, more evaluation is conducted under different power models and results show that cubic model saves more power. Therefore, two algorithms are proposed based on experimental findings the that are DFCFS and DVFS. Comparison between the two proposed algorithms is performed which show that DVFS consumes less power than DFCFS. Whilst PDM [37] is a centralized manager developed for power distribution in cloud datacenters by implementing a minimization cost function, linear optimization model, for uninterruptable power supplies UPS. The manager collects power consumption needed for resources (CPU) from each server through the server power module. Additionally, the local power controller is responsible on imposing server power threshold, both modules are server resides agents. Evaluation is conducted in testbed. The results show that a power assignment applied properly in such a way that it prevents UPSs overloading.

As well, a three-phase framework is proposed in order to manage virtual resources. Profiling and profile updating, which aim to process data and obtain information from and about the tasks, VM and PM [38]. Task classification to three classes that are long, normal and tiny. Then, the application assignment of these classified tasks to appropriate VMs is conducted. Finally, successive virtual machine placement is done by utilizing the pm that are most energy efficient. Three different scales that are large, medium and small datacenter are evaluated with the proposed work based on number of VM s and PM quantity gaps. As well a power saving is measured which result in increased by at most 12% compared to baseline. Moreover, the VM placement algorithm based on genetic algorithm was developed in [39]. The controller in the proposed system takes into consideration cost and computing requirements in order to determine the suitable VM selection type, which is performed through selection manager module. Additionally, the controller will determine the suitable placement via host selection manager module considering the objective of reducing cost and power consumption. However, in order to achieve that, an adaptive genetic algorithm is proposed which perform VM consolidation in lowest number of servers as possible. The proposed work is evaluated in CloudSimSDN with synthetic and real workloads. The power consumption of network devices, power consumption of servers, average response time of both workloads, traffic load, and hourly cost are the evaluation metrics. Results show that proposed algorithm outperforms in reducing power consumption, response time and hourly cost.

Table 3. Summarization of Power Optimization Contributions.

Ref.	Major Feature	Main Tasks	Envi.	Dataset	Evaluation Tool	Measured Metrics	Contribution
[21]	Dynamic overbooking & correlation analysis	Overbooking controller: VM placement & flow consolidation using overbooking ratio and correlation analysis. Resource Utilization Monitor: collects history data of PM utilization metrics, VM & Virtual links.	SDN- clouds	Wikipedia workload	Simulation: CloudSimSDN	Response time Power consumption	Reduced power consumption while reducing SLA violation rate.
[22]	Traffic consolidation which Considers flow completion time requirement. considers delay-sensitive flows	Traffic consolidation method that consider file completion time: FCTcon	DCN	Real DCN traces from Wikipedia and Yahoo.	Simulation & testbed	FCT Power saving	Reduced power consumption maintain file completion time FTC constraint.
[23]	File completion time for delay sensitive flows	Power optimization: PowerFCT	DCN	Wikipedia DCN traces.	Testbed and simulation	Power saving FCT miss ratio	Loosely FCT requirements lead to higher power saving while FCT miss ratio is the minimum in all cases
[24]	Distributed traffic consolidation for individual flows for scalability purposes. Distributed traffic flows consolidation on individual switches for power saving purposes.	Traffic flow consolidation by implementing an optimizer either on servers or switches depends on DCN purposes; power saving or scalability: DISCO-F & DISCO-S algorithms.	DCN	Real DCN traces from Wikipedia, Yahoo, and DCP.	Simulation & testbed.	Power saving Delay	Power saving with reduced computation overhead.
[25]	Using traffic stats and control over flow path.	Power optimization: Elastic Tree	DCN	Traffic generator tool.	Testbed	Power saving Robustness Latency	Ability to reduce power while support performance and robustness.
[26]	Dynamic consoledation of traffic flow based on correlati-on analysis.	Power optimization: correlation analysis, traffic consolidation, and link rate adaptation: CARPO	DCN	Wikipedia and Yahoo traces.	Testbed	Power consumption Power saving Delay Packet drop ratio	Outperform in power saving.
[27]	Early deduction of congestion	Reducing buffer space while maintain performance by impowering Explicit Congestion Notification (ECN): DCTCP	DCN	Three types of traffic: Query Short-message Background	Testbed	Throughput Queue length convergence query delay timeout queries flow and query completion time
[28]	Butterfly topology to reach scalable network, full utilization with reduced power		DCN	Synthetic workloads and production datacenter Traces: uniform and advert workloads.	Simulation	Fraction of time spent at each link Latency	Links operate mostly in low power consumption mode
[29]	Genetic algorithm with the tabue search algorithm	VM placement using Genetic Algorithm where Tabue search algorithm is used to enhance search by performing mutation operations.	Cloud Datacenter	Google Cluster Dataset	Pycharm 3.3	Load balance Power Consumption Number of optimal solutions Execution time	Reduced power consumption while lowers load balance. Also gives highest number of optimal solutions.
[30]	Multi-objective resources allocateon based on Euclidean distance uses Genetic Algorithm (GA) and Particle Swarm Optimization (PSO)	VM allocation Task scheduling VM migration	CloudDC	Heterogenous and homogenous DCs.	Java	Energy consumption Resource utilization	Better results when combined with VM migration
[31]	Natural evolutionary algorithm	Multi-objective VC allocation algorithm based on 4 different models: VM availability, power consumption, resource utilization, and load balance	Cloud DC		CloudSim	Power consumption Average resources utilization, Risk value Load balance value	Lower risk value, least power consumption, better average resources utilization with least load balance value
[32]	Dynamic VM placement	Cluster Managementmodule Overload selection module Resource predicter module	Cloud DC	Google Cluster Dataset		. Number of overloads Number of SLA-violation Power consumption	Less number of overloads occurred with an outperforming on minimizing SLA violation.
[33]	Linear regression prediction model	VM placement based on Linear regression prediction model: Overloaded and underloaded algorithms. Live migration policies.	Clouds DC	PlanetLab & rand workloads	Simulation: CloudSim	Power consumption SLA violation	Reduced power with minimum SLA violation.
[34]	Neural networks	Dynamic VM placement based on joint CPU demands & prediction of future VM placement.	Cloud	Real workloads	Simulation: CloudSim	Energy consumption SLA violation	Best result of reducing power consumption with smaller SLA violation.
[35]	ConsiderCPU and RAM as major power consum-ers	System consists of global and local manager. two algorithms for VM selection and placement with two thresholds for loading detections.	CloudDC	Real dataset: SPECpower benchmark	Simulation: CloudSim	Power consumption SLA violation Performance degradation due to migration	Reduced power consumption and performance degradation with dynamic threshold.
[36]	Experimental evaluation of different policies, power management models, and power models.	Two algorithms which utilize FCFS/ SJF scheduling tasks and DVFC power management using cubic energy model.	Cloud DC	RIKEN Integrated Cluster of Clusters	Simulation: CloudSim	Energy consumption	Reduced power consumption.
[37]	Uninterruptable power supply UPS minimization cost function.	Centralized manager with two server-level agents: server power modular & local power controller. (PDM)	Cloud	Computational-intensive application workload: mathematic prime counting functions.	Testbed	Power (demand, assign, consume) CPU cap CPU frequency	Power assignment applied properly which prevents UPSs overloading.
[38]	Three-phases for VM placement	Three-phases for VM management: Profiling & profile update Task classification & assignment Successive VM placement.	Cloud DC	Google Cluster dataset		VM number PH quantity gap Energy saving	Energy saving is increased by at most 12%
[39]	Genetic Algorithm	VM selection manager Host selection maager	SDN-Cloud	Synthetic and Wikipedia workload	CloudSimSDN	Power consumption in Response time Traffic load, Hourly cost	outperforms in reducing power consumption, response time, and hourly cost.

summarizes above-mentioned power optimization contributions in terms of major feature of algorithms, main tasks, the environments where the algorithm targets, which evaluation tool is used for testing and the dataset, evaluated metrics and, finally, the major contribution of the algorithm based on its published results. Moreover,

Table 4. Measured metrics in the power optimization algorithm.

Ref.	Power Consumption/Saving	Response Time	Flow Completion Time	FCT Miss Ratio	Delay	Robustness	Latency	Packet Drop Ratio	Throughput	Queue Length	Convergence	Timeout Query/Flow and Query Completion Time	Cumulative Fraction of Query Latency and Completion Time	Fraction of Time Spent at Each Link	Load Balance	Number of Optimal Solutions	Resources Utilization	Risk Value	Number of Overloads	SLA Violation	Performance Degradation	CPU Cap/Frequency
[21]	●	●
[22]	●		●
[23]	●			●
[24]	●				●
[25]	●					●	●
[26]	●				●			●
[27]									●	●	●	●	●
[28]							●							●
[29]	●														●	●
[30]	●																●
[31]	●														●		●	●
[32]	●																		●	●
[33]	●																			●
[34]	●																			●
[35]	●																			●	●
[36]	●
[37]	●																					●
[38]	●

shows measured metrics with associated algorithms.

5. SDN-Based Fog Computing

Since cloud computing plays a major rule in data storage, however, fog computing aims to reduce the latency between nodes, especially for delay-sensitive applications. Thus, one of the main reasons to develop SDN-based fog is to ease communication between nodes in order to ensure a reasonable performance. This section presents state-of-the-art researches that are conducted for SDN-fog computing to solve existing challenges specifically towards network performance issues.

An SDN-based solution is proposed in order to solve the complexity of data forwarding in smart grids in fog computing [40]. Three-tier architecture is proposed that consists of IoT as the bottom tier that is responsible on generating data, the middle tier, which is fog, enables smart grids that consist of smart meters (SM) and data collection servers (DCS). While the cloud tier, which is the top layer, consists of SDN controller and responsible in performing processing and permanent storage. Data forwarding uses shortest paths—specifically, Dijkstra Algorithm—with three types of path recovery that are forced when a link failure is encountered during data transmission. The proposed work is simulated in Mininet considering three types of topologies, which are large, medium and small to measure the transmission time, throughput and transmission overhead. The results show that the Dijkstra algorithm has the lowest forwarding time compared to other shortest path algorithms, i.e., Floyed-Warshell and Bellman-Ford. On the other hand, the forwarding time is increased when the path recovery conducted end-to end link failure. However, no link failure case consumes more time than a path recovery conducted through switch-to-switch or switch-to-end cases when the topologies are large and medium, while the number of broken lines has a positive correlation with the forwarding time while a throughput has a negative correlation with the size of topology. While, in [41], multi-layer advanced networking environment middleware is proposed for the SDN-based fog environment to solve heterogeneity and different time requirements by different applications. Based on the application type, a forwarding mechanism is selected by the SDN controller such as by IP, payload content and overlay networks where selection and initiating are performed simultaneously. Real testbed is used for an intensive evaluation that considers mainly latencies and performances. The results show the capability of the proposed middleware to determine the best forwarding mechanism based on the application type and network status.

On the other hand, merging fog computing and SDN in the internet of vehicle is proposed to solve latency problem resulted from generated data by sensors [42]. It consists of five layers, including clusters of Internet of Vehicles (IoV), local and global load balancers, fog servers, SDN controller and cloud servers. Local and global load balancers are designed for local clusters by local load balancers and different clusters which will be performed by SDN controller. Considering CPU utilization as main criteria when determining size of load to be either overloaded or under loaded. A simulation was conducted via OMNET++ and SUMO with a size of vehicles equal to 1000 and evaluated against VANET-Cloud architecture. The results show that the proposed architecture and load balancers capable of increasing resource utilization, decreasing the average response time, outperforming in meeting tasks’ deadlines, and uses lowest bandwidth compared to baseline.

Other research employed machine learning techniques to provide intelligence to SDN controllers, such in [43], a deep reinforcement learning was developed to perform task scheduling in the SDN controller by learning from intelligent agents that led to optimal decisions. The SDN controller is capable to generate and learn from historical datasets, filter and re-represent datasets, produce learning policies, selects nodes and activates rules. The evaluation is conducted via testbed using two datasets for training and testing. The results show that the maximum latency achieved equal to 12.5 milliseconds and saved energy up to 87%.

To solve the problem of packet delays imposed by the SDN controller in fog networking, a prediction model is proposed based on regression model which follow standard process of data mining methodology [44]. The process generally consists of data understanding, which aims to build an initial dataset, and data preparation, which aims to clean dataset, remove redundancy and construct features, as well, modeling, which includes selection and assessments machine learning algorithms that results in selecting random forest algorithm. The results shows that proposed model manages to predict delay values similar to actual values.

Moreover, a solution for SDN-fog services based on machine learning—specifically, reinforcement learning—is proposed in [45] in order to allocate service dynamically based on objectives such as power consumption, security and QoS constraints. Proposed reinforcement learning imposes recurrent neural networks in order to reduce the average response time. The service manager is built-in SDN controller specifies the locations of services to be processed dynamically by taking into consideration overall objective.

In [46], the proposed work goal is to increase storage, by reducing the cache redundancy and improving service placement with limited interactions with controller, therefore merging software-defined networks and named data network in fog computing is introduced. By extracting powerful features of both paradigms i.e., devoting controllability to SDN controller to perform routing, function placement, caching decision and service allocation. While the data plane is managed by forwarding information base, content store and pending interest table, which are offered by NDN stateful data plane. The evaluation is conducted using MATLAB against SDN-IP protocol systems by comparing the number of packets exchanged between the nodes and controller and number of new rules inserted by the controller. The results show SDN-NDN systems are capable of reducing communication between nodes and the controller.

In [47], data collection schema and estimation model for SDN-based vehicular fog computing is proposed. The architecture of SDN-based VFC consists of three layers, which are an infrastructure layer that includes vehicles, a fog layer that includes both regional controllers and local SDN controllers and a cloud layer, which includes a global SDN controller. The main tasks consist of the data collection configuration process, fog station deployments and data routing. A collection of data configuration consists of four main phases that are initialization of data collection parameters that are performed by a global SDN controller as data-oriented or fog-oriented. Then, the data collection parameter the sends collected data to fog stations. After that, users’ requests are sent to fog stations. Finally, update data collection parameters to fit current environment, which include the deployment of fog station and data migration. A model is built to estimate number of fog stations required to be deployed and evaluation is conducted based on real traffic data. The results show that accurate estimation of fog stations is achieved, which depends on data availability and volume of data collection. While the data collection schema is evaluated in Veins Simulation Framework against classical schema to assess the latency, a number of hops to reach the fog station, recovery time and data loss. The results show that a proposed schema decreases the latency and number of hops. As well, less time is required for recovery, and less data loss is achieved.

The machine learning approach is utilized in [48], specifically the K-Nearest Neighbor (KNN) algorithm, together with multi-objective optimization—specifically, Non-dominated Sorting Genetic Algorithm (NSGA-II)—to meet the QoS constraint based on application type, either maximizing the path reliability or minimizing the delay. The system consists of three parts that are network architecture that consists of SDN switches, fog nodes, IoT devices and links associated with QoS parameters, i.e., bandwidth and transmission delay. Then, the communication model and provisioning model. In order to find the link reliability, the system uses machine learning techniques, i.e., KNN algorithm. While finding the optimal path as a tradeoff objective, the system uses the NGSA-II algorithm. An evaluation is conducted using Mininet with two types of applications: delay-sensitive that requires a path with less delay, and computational-intensive, which requires a reliable path. The results show that the system manages to meet each application QoS constraint by selecting required path according to optimal objective. Moreover, less delay and less packet loss are achieved compared to the baseline.

The software defined and multi-access edge computing vehicular network architecture is developed to reduce the latency in vehicular networks [49]. The system uses fuzzy logic for clustering vehicles and OpenFlow algorithm to update flow tables. The architecture consists of four layers, which are: access layer, that consists of forwarding devices. Then the forwarding layer, which consists of high-speed SDN switches. Then multi-edge computing layer that has local controllers that perform resources allocation, and the MEC application platform system, which performs cellular resource allocation, caching in-vehicle message modules, communication decision modules and multimedia broadcast multicast service computing modules. Finally, the control layer which contains global SDN controller. Using fuzzy logic for clustering the vehicles by head vehicle based on velocity, position and signal quality. After that, the SDN client switched flow tables are updated. The simulation was performed in OMNET++ with INET framework using real-time traffic via SUMO. The results show that proposed system manages to achieve the highest packet delivery ratio and lowest average E2E delay compared to the baselines.

Similarly, an offloading mechanism is proposed in [50] that works dynamically by considering nodes selection prior to offloading. The system is composed of fog nodes, which are connected to the SDN switch and whose information is gathered by fog agents. Additionally, core switches that contain several SDN switches that are controlled via SDN controller. As well, the controller that manages both fog nodes and network via the fog orchestration and SDN controller. Finally, the service layer, which includes fog support applications. On the other hand, a dynamic offloading service is triggered when it receives a call from fog orchestration, which is performed when a node sent an offload request to fog orchestration as a result of an overloading condition. However, the decision of which node will receive the offload that was is made after filtering the nominated nodes and ordered it based on the network status and available resources. The evaluation was conducted via the Minint emulation tool and Python, and iPref was used for traffic generation. Selection time, time to transfer data to the offloading node, throughput and available resources ratio are the evaluation metrics. The results showed that the proposed work was outperformed in minimizing the selection time, thus minimizing the delay. As well, it provides better throughput and less response time. As well, an artificial intelligent approach is proposed to overcome the delay and reliability issues in SDN-based IoV-Fog computing [51]. ARTNet architecture consists of the IoV layer which contains onboard devices, the fog layer, which contains base stations and RSU as the static models to provide processing and storage facilities. As well, the architecture contains an AI-based SDN that consists of three modules that are intelligent agents, deep learning and big data analytics. The experimental results show that the proposed algorithm outperforms in reducing the power consumption and latency with respect to the time slot and number of vehicles, especially when the number of IoV is increased. In addition to that, IoV architecture is proposed in [52], which consists of multiple fog clusters that contain several RSUs. RSUs gather information regarding roads and vehicles, then transform this information to RSU controller, which performs decision-making towards resource allocations in the fog layer. While the SDN controller resides in the cloud to obtain global view and forward status of the network with forwarding policies and rules to the RSU controller. Then, the resource allocation model is constructed, which considers several factors, which are the power consumption, delay, load balancing and time execution stability. After that, the proposed algorithm is developed with integration with the evolutionary algorithm by using hieratical clustering features to enhance the many-objective optimization algorithm. The experimental results show that the proposed algorithm outperforms in time execution stability and load balancing objectives.

Even though previous studies followed OpenFlow architecture, however, in [53], a multi-layer SDN Fog paradigm was proposed in order to organize the SDN controllers, locally and remotely, in fog computing using P4 and P4Runtime to overcome certain difficulties imposed by the OpenFlow protocol. Controllers can be built locally near edge devices in order to ease communication, while remote SDN controllers are centralized to control traffic, among other functions that require a global view of the network.

Table 5. Summarization of the contributions in SDN-based fog computing.

Ref.	Major Feature	Main Tasks	Env.	Dataset	Eval.Tool	Measured Metrics	Contribution
[40]	data forwading scheme	data forwarding: Dijsktra algorithm 3 types path recovery	sdn- fog enabled smart grid	3 different sizes topologies	mininet support openflow protocol	transmission time throughput transmission overhead	Dijkstra has the lowest forwarding time compared to other shortest path algorithms Number of broken lines has a positive correlation with forwarding time. throughput has a negative correlation with the size of topology
[41]	Multi-layer routing approach	Multi-layer advanced networking middleware which enables native IP, overlay networking information, and payload content data forwarding mechanisms.	SDN- based Fog Computing	Varying size packets	testbed	latency performance	capability of determining best forwarding mechanism based on application type and network status.
[42]	Resource utilization using local and global load balancers	Five layers architecture Local and global load balancers based on CPU utilization and certain Thresholds	SDN-Fog IoVs	1000 vehicle with three clusters scenario	OMNeT++ and SUMO	Resources utilization average response time percentage of meeting deadlines percentage of bandwidth usage	increase resource utilization Decrease average response time Outperforms in meeting tasks’ deadlines Uses lowest bandwidth
[43]	Delay prediction by Regression model	three steps model: data understanding, preparation, and modeling.	sdn-fogs	uses several datasets with different number of instances.	CRISP-DM methodology.	predicted and actual delay	predicted delay values similar to actual delay values.
[44]	Deep reinforcement learning	Tasks scheduller in SDN controller level	SDN-based IoT	Training set: 1000 episodes Evaluation set: 100 tasks	Testbed	Saved energy Latency	Reduced energy consumption Maximum latency equal to 12.5 milliseconds.
[45]	reinforcement learning	Service manager located to specify services locations based on function goal.	SDN-Fog	steady sequence of successive requests: 10 requests per second	Testbed	Response time	Ability to adopt to medium and high load.
[46]	Named Data Networking	Jointing SDN-NDN: maintaining SDN controller capabilities and stateful dataplane in NDN.	SDN-NDN Fog computing	Varying number of services requests of size 1000	MATLAB	number of exchange packets between controller and nodes Number of injected rules by controller.	communication is reduced between nodes and controller
[47]	Data collection schema	Three layers SDN- Vehicular Fog computing: infrastructure layer, fog layer, and cloud layer.	SDN—based Vehicular Fog Computing.	Real traffic measurements Different size clusters: 10–50 vehicles.	Veins Simulation Framework	accurate model which estimates fog station based on data collection volume. reduced latency and number of hops reduced recovery time and data loss.
[48]	Machine learning: KNN algorithm multi-objective optimization algorithm: NSGA-II	consists of network architecture, communication model, and provisioning model.	SDN- enable IoT-Fog Computing	training data: real life data recorded for time interval of 5 months testing data: real time and delay sensitive. testing data: computationally intensive tasks.	Mininet with POX SDN controller	packet loss end to end delay	paths are selected based on QoS constraint either reliable path/ minimize delay and succeeds on performing less data packets and delay.
[49]	Fuzzy logic	4 layers architecture for SD-MEC: access layer, forwarding layer, multi edge computing layer, and control layer	SDN-based multi access edge for vehicle networks.	Real world traffic conditions.	OMNET++ with INET framework.	delay of in-vehicles messages end to end delay in respect of vehicle density. packet delivery ratio throughput	least average E2E delay highest packet delivery ratio
[50]	two steps offloading- node selection	4-layers architecture: fog layer, core switches, controller, and service layer	SDN-Fog	iPref for traffic generation	Minimit and Python	Selection time Time to transfer date to offloading node throughput Available resources ratio	Outperforms in minmizing selection time thus minmizing delay. As well, provide better throughput and less response time.
[51]	Employing intelleigent agents.	3-layers arcitecture: IoV layer Fog layer Intellegent-SDN layer	SDN-IoV-Fog		IoV based on SDV-F framework	energy consumption energy shortfall latency	Least energy consumption and latency.
[52]	Two types of controllers are employed: RSUC and SDNC	Multi- objective resource allocation	SDN-Fog-Cloud			Stability Load balancing Power consumption Delay	Outperform in time excution stability and load balancing objectives

summarizes the contributions related to SDN fog computing in terms of the major feature of algorithms, main tasks, the environments where the algorithm targets, which evaluation tool is used for testing and the dataset, evaluated metrics and, finally, the major contribution of the algorithm based on its published results. Figure 1 represents the relationship between each category.

6. Open Issues

Open issues related to SDN cloud and SDN fog, which are derived from a deep analysis of the above-mentioned reviewed researches, are listed as follows:

• Power savings is a challenge, along with several related issues such as data transferring, powered devices, and energy wastage.
• There is a great concentration on the limited number of metrics (Table 3 and Table 4) when evaluating the network performance. On the other hand, power consumption is a major metric in power optimization algorithms, but other important measures of network performance are being neglected.
• The network performance can be enhanced by considering the awareness of the resources management algorithm and considering the bandwidth due to its impact on the overall performance. Moreover, control over resources leads to performance improvement.
• There has been always a tradeoff relationship between the network performance and power optimization due to the consequences resulting from overutilized and underutilized resources. Only few researches manage to find a balanced relationship between the two variables.
• Dynamic deterministic of the overall objective from a set of objectives, such as the minimizing delay or maximizing reliability, based on the application type is an open issue.
• Most research focused on the Central Processing Unit (CPU) as the most dominant factors in energy consumption. Even though it is considered as a high-power consumer element in PM, but other important factors are being neglected, such as RAM, network card, and storage.
• Most presented techniques focused only at the server level and ignored the possible contributions on flows/links.
• From state-of-the-art researches in SDN-based fog, we can observe that interrelated fields are presented such as machine learning and artificial intelligent in order to provide a fully programmed environment, which is a promising approach for future researches.
• Identifying and fulfilling requests with higher priority prior to low-priority requests is an issue due to the high volume of data transferred among different devices, which results in congested links and, thus, latency.
• Joining SDN and named data networking (NDN) is a new approach to gain the maximum privilege of both techniques. However, few researches are conducted in this field.
• Furthermore, SDN cloud is an innovative field, which brings a great opportunity for researchers, since most studies either contribute on the field of cloud or SDN.
• The scalability issue, which can be classified into two aspects: resources scalability either scaling up or down in the cloud and fog layers and controllers scalability (local and global) to consider their location and number of required controllers.
• Security is one the main challenges encountered in SDN-Based cloud/fog, including authentication and SDN.
• Legacy SDN such as OpenFlow forces certain architectures when designing controllers that lead to strict environment for the revolution and development, especially in a fog environment.
• Only a few contributions regarding the simulation tools and testbed that facilitates the experiments of SDN-based environments.

7. Conclusions

From above discussion, we can conclude that SDN approach is a promising in the future to both clouds and fog computing due to its management simplicity advantages. This study overviewed novel and state-of-the-art findings in SDN cloud and SDN fog. Reviewed studies have been classified based on the main objective: network performance improvement and energy optimization. Further classifications will be conducted based on algorithm strategies in achieving a desirable goal. The studies are summarized based on major feature of the proposed algorithm, main tasks, environment, evaluation tool, evaluation metric, dataset, and main contribution. After that, we overviewed related open issues, which covers several aspects, from the main objective to simulation tools development. SDN-based cloud/fog architecture plays an important role in scalability and performance. Moreover, merging SDN with other approaches such as artificial intelligent and machine learning to develop flexible and intelligent environments is promising in order to find an optimal solution to multi-objective algorithm and to achieve desirable outcomes in network performance and energy efficiency. As well, new contributions in designing SDN controller architecture other than OpenFlow are suggested.