Novel Multi-Level Dynamic Trafﬁc Load-Balancing Protocol for Data Center

: Typically, the production data centers function with various risk factors, such as for instance the network dynamicity, topological asymmetry, and switch failures. Hence, the load-balancing schemes should consider the sensing accurate path circumstances as well as the reduction of failures. However, under dynamic trafﬁc, current load-balancing schemes use the ﬁxed parameter setting, resulting in suboptimal performances. Therefore, we propose a multi-level dynamic trafﬁc load-balancing (MDTLB) protocol, which uses an adaptive approach of parameter setting


Introduction
The exponential growth of the information technology (IT) industry and its storage demand forces to move toward cloud computing, in which a huge amount of computing and storage resources are provided in large data centers [1,2].These data centers are designed with high-speed, highly configured servers.The cloud servers are used to offer web searches, Amazon Elastic Compute Cloud (EC2), Microsoft azure storage, and personal and business storage [3][4][5].As a consequence, the cloud servers handle a huge amount of data, and the interconnected networks and server resources eventually become unstable, which hits the performance of the network bandwidth [6].Due to this, the network bandwidth, the latency-sensitive web services, and applications are suffering from a long network latency tail [7].An illustrating image of the data center traffic load is shown in Figure 1.
The multi-rooted topologies have been adopted for data center networks.This is to provide the high bisection bandwidth [8].The existing multipath topologies are used to deliver the alternative routing paths between any two end-hosts under different switches.
The load balancing for multiple paths with the utilization of whole network resources can improve throughput as well as reduce latency for data center applications [9].The production data centers are used to operate under several uncertainty factors.The uncertainty factors are mainly the dynamicity of the traffics, unstable links, device heterogeneity, and switch failures [10].To overcome these situations due to uncertain circumstances, the load-balancing technique is seriously effective.
Using the load-balancing technique, the accuracy of the path-sensing conditions and path rerouting according to the path conditions can be improved.However, many of these protocols are unaffected by path conditions; the approach is to split blindly at a fixed granularity.The proposal was made by ignoring traffic conditions; however, it can be seen that it affects the performance, especially when asymmetry arises [7,11].The flowlet-based proposals have been proposed to sense the path conditions.However, it facilitates rerouting only when the flowlet emerges.The flowlet formation depends on various factors that are related to the applications and the transport layer.As a result, using a flowlet-based reaction to the path conditions does not provide a timely reaction to congestion [10].In addition, switch modification-based approaches (CONGA and HULA) are based on switching or hardware modification that imposes impracticalities [12,13].Among other solutions, Hermes has been impressing through its better performance [14].Hermes accomplishes per-flow load balancing to enact faster rerouting following the network status.It applies packets to react timely to overcome uncertainty challenges.However, it also imposes the congestion mismatch and packet reordering as well; in such conditions, Hermes reduces the frequency of rerouting to avoid congestion.Moreover, Hermes uses the unsystematic hashing, round robin, and seeks the best path to route and reroute based on the path conditions as well as flow status.However, stability is a key concern for the condition when flows interact in the network [10].The bursty network congestion has been well addressed in [15][16][17] at the transport and link layers.Their performances can be further improved with the support from the load balancing schemes at network layer.The multi-rooted topologies have been adopted for data center networks.This is to provide the high bisection bandwidth [8].The existing multipath topologies are used to deliver the alternative routing paths between any two end-hosts under different switches.
The load balancing for multiple paths with the utilization of whole network resources can improve throughput as well as reduce latency for data center applications [9].The production data centers are used to operate under several uncertainty factors.The uncertainty factors are mainly the dynamicity of the traffics, unstable links, device heterogeneity, and switch failures [10].To overcome these situations due to uncertain circumstances, the load-balancing technique is seriously effective.Using the load-balancing technique, the accuracy of the path-sensing conditions and path rerouting according to the path conditions can be improved.However, many of these protocols are unaffected by path conditions; the approach is to split blindly at a fixed granularity.The proposal was made by ignoring traffic conditions; however, it can be seen that it affects the performance, especially when asymmetry arises [7,11].The flowlet-based proposals have been proposed to sense the path conditions.However, it facilitates rerouting only when the flowlet emerges.The flowlet formation depends on various factors that are related to the applications and the transport layer.As a result, using a flowlet-based reaction to the path conditions does not provide a timely reaction to congestion [10].In addition, switch modification-based approaches (CONGA and HULA) are based on switching or hardware modification that imposes impracticalities [12,13].Among other solutions, Hermes has been impressing through its better performance [14].Hermes accomplishes per-flow The main contribution of this article is a multi-level dynamic traffic load-balancing (MDTLB) protocol.The key idea is to design a protocol using the functionalities of Hermes, multiple levels of path characterization, and an adaptive approach to setting parameters.Two algorithms are proposed, namely, the judging algorithm and the rerouting algorithm.
The rest of the article is organized as follows.Section 2 discusses the background study, and Section 3 provides the motivation of the study.Section 4 discusses the methodology of the proposed approach.Section 5 presents the result and discussion of the proposed approach, while Section 6 demonstrates the related works.Finally, Section 7 contains the findings and the conclusion.

Background
The advancement of data center and virtualization technologies has led cloud storage.The concept of the cloud computing paradigm delivers through online network accesses shared resources that are configurable as well as flexible resources.Thereby, load-balancing protocol and its approaches are significant in cloud servers.The load-balancing protocols are mainly designed with the consideration of three types of flows (static, dynamic proactive, and dynamic reactive) and the granularity [7].The static flow is then allocated to the available paths through applying the fixed conditions by the use of the hash for each packet header.Conversely, this technique is inflexible to the scenarios of throughput-oriented flows, while it is consigned to the same path.In such a situation, the flows cannot move to other less utilized paths.Meanwhile, the dynamic reactive flow is able to move across any of the available paths subject to available bandwidth.The dynamic reactive flow provides a significant result.However, it increases higher complexity in measuring link utilization, flow counting, and estimating the best flow.On the other hand, the dynamic proactive allows the flows to become assigned to the available paths in a fixed assignment, where the initial assignment is mainly performed according to the available bandwidth.This approach is better compared to static flow and dynamic reactive flow in terms of implementation overhead, flexibility, and performance.Four kinds of granularity are observed such as per packet, per flow, flowlets, and per flow cell.

•
Per packet: Can be used for the best load balance; however, due to the higher reordering, there are chances to attain insignificant reordering.

•
Flowlets: Due to the variation of the candidate path latencies, the flowlet's size changes enthusiastically.If the latency rates are higher, the flowlets become large.It is obvious that it applies per packet and per flow technique.Therefore, the load balancing is then fine as well as coarse-grained.As a result, the flowlets become promising in load balancing for asymmetric paths.Nevertheless, the flowlets causes a small flow reordering, which can interrupt the execution times.

•
Per flow cell: It uses the fixed size with the consideration of tens of packets.The most positive effect of per flow cell is that it uses simplified load balancing to reduce the possible reordering of small flows.However, it rises the reordering for larger flows, which can be fragmented into several flow cells.

Motivation of the Study
This study is focused on improving the performance of the load balancer to avoid the problem of congestion mismatch.This section discusses the concept of the congestion mismatch.

Congestion Mismatch Problem with an Example
To avoid the congestion mismatch issues, the algorithm may consider the flow-based rate of the congestion state using the condition of the paths and the routing.Regarding the event of rerouting, there are chances of mismatches.This is happening because of the sending rate and the rerouting state of the new path.The dynamic rerouting within a flow allows the various congestion states of the paths to adjust the rate of newer paths.This occurrence is then stated as congestion mismatch [14].To illustrate the consequences of the congestion mismatch problem, two scenarios are considered.The first scenario is considered under asymmetric topologies for the same flow sizes, A and B, and the second scenario is for a heterogeneous network.The second scenario presents the load distributions according to link capacities.Figure 2 depicts the dynamic routing protocols of Presto and Digit-Reversal Bouncing (DRB) [7].
In Figure 2a we have a simple three to two leaf-spine topology with a broken link from L0 to S1.Also, we have two types of flows: flow A, which is a data center transmission control protocol (DCTCP) from L0 to L2, and flow B, which is a user datagram protocol (UDP) flow from L1 to L2.The result of congestion mismatch is low throughput and queue oscillations when equal weights are adopted for different paths.The throughput is oscillating around 1 Gbps, as shown in Figure 2c, while the queue size is around 100 KB, as shown in Figure 2b.This is interpreted by the vigorous rerouting; the congestion feedback (i.e., explicit congestion notification (ECN)) of the upper path constrains the congestion window, resulting in throughput loss in the bottom path.Furthermore, when flow B with a larger window shifts from the bottom path to the upper path, the upper path cannot immediately absorb such a burst, causing queue length oscillations.
A variation in the size of the queues appears in the S0-L2 link, whereas the low throughput can be measured in flow B. Furthermore, observing the distribution of loads according to capacities is shown in Figure 3.In the S0-L1 link, L1 suffers from low throughput, and the S0-L1 link has a limitation regarding queue size variations.In Figure 2a we have a simple three to two leaf-spine topology with a broken link from L0 to S1.Also, we have two types of flows: flow A, which is a data center transmission control protocol (DCTCP) from L0 to L2, and flow B, which is a user datagram protocol (UDP) flow from L1 to L2.The result of congestion mismatch is low throughput and queue oscillations when equal weights are adopted for different paths.The throughput is oscillating around 1 Gbps, as shown in Figure 2c, while the queue size is around 100 KB, as shown in Figure 2b.This is interpreted by the vigorous rerouting; the congestion feedback (i.e., explicit congestion notification (ECN)) of the upper path constrains the congestion window, resulting in throughput loss in the bottom path.Furthermore, when flow B with a larger window shifts from the bottom path to the upper path, the upper path cannot immediately absorb such a burst, causing queue length oscillations.
A variation in the size of the queues appears in the S0-L2 link, whereas the low throughput can be measured in flow B. Furthermore, observing the distribution of loads according to capacities is shown in Figure 3.In the S0-L1 link, L1 suffers from low throughput, and the S0-L1 link has a limitation regarding queue size variations.In addition, we show that congestion mismatch remains harmful even if we distribute the flows proportionally to path capacity.To illustrate this, we consider a heterogeneous network with 1 C and 10 C paths, as shown in Figure 3a.We spread the flows using a 1:10 ratio to match the path capacities, and expect both paths to be fully utilized.
However, as shown in Figure 3b, flow A can only achieve an overall throughput of around five Gbps.To understand the reason, assume that the first 10 flow cells go through the 10 C path.Since the path is not congested, the protocol will increase the congestion window without realizing that In addition, we show that congestion mismatch remains harmful even if we distribute the flows proportionally to path capacity.To illustrate this, we consider a heterogeneous network with 1 C and 10 C paths, as shown in Figure 3a.We spread the flows using a 1:10 ratio to match the path capacities, and expect both paths to be fully utilized.
However, as shown in Figure 3b, flow A can only achieve an overall throughput of around five Gbps.To understand the reason, assume that the first 10 flow cells go through the 10 C path.Since the path is not congested, the protocol will increase the congestion window without realizing that the subsequent flow cell will go through the 1 C path.With a large congestion window, the 11th flow cell is sent at a higher rate on the 1 C path, causing a rapid queue buildup on the output port of spine S0 to leaf L2.As the queue length exceeds the ECN marking threshold, the DCTCP will reduce the window upon receiving ECN-marked ACKs without realizing that such a reduction will affect the following flow cells on the 10 C path.As a result, such a congestion mismatch still causes throughput loss and queue oscillations.

Summary
Our analysis leads us to conclude that the congestion mismatch problem occurs due to the continuous path rerouting, which is happening because of the sending rate and the rerouting state of the new path and the dynamic rerouting within a flow that allows the various congestion states of the paths to adjust the rate of newer paths.These outcomes inspire us to handle the congestion mismatch problem by reducing rerouting to the minimum.We overcome the mismatching problem that is proved by improving the flow completion time (FCT) and the throughput, and we implemented our scheme to not reroute until it is a critical situation such as failure or very bad paths, so that the rerouting is minimized as much as possible.

Basic Concept
The existing Hermes applies the thresholds and parameters to depict the path, identify the best possible path through the confirmation of the lower round trip time (RTT) measurement, and lower the rate of congestion mismatch.Hence, the congested path can be identified while there are higher RTT and ECN.It is obvious that the higher RTT can be lowered due to the stack latency of the network, and the higher ECN fraction comes from some of the inaccurate samples.Else, the path is stated to be gray in all of the other cases.This characterization approach is sensitive to the parameters setting.It causes a non-steady performance that may increase the congestion mismatch effect on the performance instead of decreasing it due to the wrong threshold values.However, a multi-level path judging with dynamic thresholds-based heuristic characterization of the path is more realistic regarding reducing a congestion mismatch issue than a simple threshold-based approach.

Design Considerations
The design of the protocol is composed of two parts: an adaptive parameter-setting algorithm, and a rerouting algorithm.The detailed discussion is as below:

Parameters
The parameters setting for flow-related variables parameters is shown in Table 1, the path level variables are presented in Table 2, and the parameters for the developed protocol are provided in Table 3.

Adaptive Parameters Settings
To overcome the parameters setting sensitivity of Hermes, this article proposed two steps of improvement: 1.
The first is to increase the number of path levels to give the scheme more awareness of the path's status.Hence, the path is divided into five types {very good, good, gray, bad, and very bad}, as shown in Figure 4.

2.
The second step is to make the judging dynamic by introducing static and dynamic T RTT thresholds, as shown in Figure 5 and pseudocode one.The user sets the static thresholds, while the dynamic thresholds are set by the system using an adaptive way to determine the best threshold value based on the network load.It is important to note that T RTT can be used as the main judging thresholds.This is because the measurements of T RTT are more accurate and can give a wider range than T ECN to distinguish between path levels.
The judging algorithm is illustrated in pseudocode one and Figure 6.

Rerouting Logic
The proposed algorithm for rerouting logic is designed that will find the best available path for every flow.The idea of the proposed logic can be derived as: if there is new flow, then, the process is to begin to search for very good paths; if the search result does not find any (none), then the process searches for gray if none chose the path randomly.In such cases, the proposed approach chose a path with the smallest local sending rate from the available paths.If there is old flow with a very bad path with a small rate and the sent size is less than size threshold S, then the approach is applied to

Rerouting Logic
The proposed algorithm for rerouting logic is designed that will find the best available path for every flow.The idea of the proposed logic can be derived as: if there is new flow, then, the process is to begin to search for very good paths; if the search result does not find any (none), then the process searches for gray if none chose the path randomly.In such cases, the proposed approach chose a path with the smallest local sending rate from the available paths.If there is old flow with a very bad path with a small rate and the sent size is less than size threshold S, then the approach is applied to find a better path from the very good, good, or gray.If there is no better path than the current one, it leaves and cancels the rerouting process.An illustration can be seen in Figure 7.

Rerouting Logic
The proposed algorithm for rerouting logic is designed that will find the best available path for every flow.The idea of the proposed logic can be derived as: if there is new flow, then, the process is to begin to search for very good paths; if the search result does not find any (none), then the process searches for gray if none chose the path randomly.In such cases, the proposed approach chose a path with the smallest local sending rate from the available paths.If there is old flow with a very bad path with a small rate and the sent size is less than size threshold S, then the approach is applied to find a better path from the very good, good, or gray.If there is no better path than the current one, it leaves and cancels the rerouting process.An illustration can be seen in Figure 7.The logic of rerouting is illustrated in pseudocode two.A, B, C, D, E, F, G, H, and I.As illustrated in Figure 8 and Table 4, the following events describe the system response for this scenario: 1.
Flow A started; the status of paths is unknown, so they are grey; the portion of good paths equals the portion of bad paths (0), so no need to change the dynamic thresholds, and our routing logic will choose any of the paths, let us say P1.

2.
Flow B started; the judging of paths will be bad for P1 due to traffic and grey for P2; the routing logic will choose P2; the portion of bad paths is greater than the portion of good paths, so the dynamic threshold should be increased by 10%.

3.
Flow C started and due to the change of the dynamic threshold from the previous step, the judging logic will be good for P1 and bad for P2; as a result, the routing logic will choose P1, and the portion of good paths equals the portion of bad paths, so no need to change the dynamic thresholds.

4.
Flow D started, the judging logic will be bad for P1 and P2; the routing logic will choose either P1 or P2, let us say P1; the portion of bad paths is greater than the portion of good paths, so the dynamic thresholds should be increased by 10%. 5.
Flow E started, and due to the change of dynamic thresholds from the previous step, the judging logic will be very bad for P1 and good for P2; the routing logic will choose P2, and the portion of good paths equals the portion of bad paths, so no need to change the dynamic thresholds.6.
Flow F started, the judging logic will be very bad for P1 and bad for P2; the routing logic will choose P2, the portion of bad paths is greater than the portion of good paths, so the dynamic thresholds should be increased by 10%.7.
Flow G started, and due to the change of dynamic threshold from the previous step, the judging logic will be very good for P1 and good for P2; the routing logic will choose P1, and the portion of bad paths is smaller than the portion of good paths, so the dynamic thresholds should decrease by 10%.8.
Flow H started, and due to the change of dynamic threshold from the previous step, the judging logic will be good for P1 and good for P2; the routing logic will chose either P1 or P2, let us say P2; the portion of bad paths is smaller than the portion of good paths, so the dynamic thresholds should decrease by 10%.9.
Flow I started, and the judging logic will be good for P1 and bad for P2; thus, the routing logic will choose P1, and the portion of good paths equals to the portion of bad paths, so no need to change the dynamic thresholds.4. Flow D started, the judging logic will be bad for P1 and P2; the routing logic will choose either P1 or P2, let us say P1; the portion of bad paths is greater than the portion of good paths, so the dynamic thresholds should be increased by 10%. 5. Flow E started, and due to the change of dynamic thresholds from the previous step, the judging logic will be very bad for P1 and good for P2; the routing logic will choose P2, and the portion of good paths equals the portion of bad paths, so no need to change the dynamic thresholds.6. Flow F started, the judging logic will be very bad for P1 and bad for P2; the routing logic will choose P2, the portion of bad paths is greater than the portion of good paths, so the dynamic thresholds should be increased by 10%.7. Flow G started, and due to the change of dynamic threshold from the previous step, the judging logic will be very good for P1 and good for P2; the routing logic will choose P1, and the portion of bad paths is smaller than the portion of good paths, so the dynamic thresholds should decrease by 10%.8. Flow H started, and due to the change of dynamic threshold from the previous step, the judging logic will be good for P1 and good for P2; the routing logic will chose either P1 or P2, let us say P2; the portion of bad paths is smaller than the portion of good paths, so the dynamic thresholds should decrease by 10%.9. Flow I started, and the judging logic will be good for P1 and bad for P2; thus, the routing logic will choose P1, and the portion of good paths equals to the portion of bad paths, so no need to change the dynamic thresholds.

Results and Discussion
The proposed algorithm was implemented in a Network Simulator 3 (NS3) environment.The simulation environment was configured using the selected parameters.The simulation parameters are listed in Table 5.We tested MDTLB under a symmetric 4  4 leaf-spine topology, as shown in

Results and Discussion
The proposed algorithm was implemented in a Network Simulator 3 (NS3) environment.The simulation environment was configured using the selected parameters.The simulation parameters are listed in Table 5.We tested MDTLB under a symmetric 4 × 4 leaf-spine topology, as shown in Figure 9, with 64 hosts connected by 10 Gbps links.We simulate a 2:1 oversubscription at the leaf level similar to the typical data center deployments.Figure 9, with 64 hosts connected by 10 Gbps links.We simulate a 2:1 oversubscription at the leaf level similar to the typical data center deployments.We evaluate and measure the performance of the proposed MDTLB by two main metrics: the (normalized) flow completion time (FCT) and the throughput.To do so, general flow, data mining, and the web search workload were used.The observation of the FCT of general flows, with three types of flows, are also shown separately for short flows, long flows, and both short and long flows in Figure 10.Similarly, in Figures 11 and 12, the observation was made for the flow completion time (FCT) of web search and data-mining, respectively.In all of the figures of FCT, the results were normalized to MDTLB protocol in order to have better visualization for comparing results.We evaluate and measure the performance of the proposed MDTLB by two main metrics: the (normalized) flow completion time (FCT) and the throughput.To do so, general flow, data mining, and the web search workload were used.The observation of the FCT of general flows, with three types of flows, are also shown separately for short flows, long flows, and both short and long flows in Figure 10.Similarly, in Figures 11 and 12, the observation was made for the flow completion time (FCT) of web search and data-mining, respectively.In all of the figures of FCT, the results were normalized to MDTLB protocol in order to have better visualization for comparing results.From the above results, the following observation can be made: Under general workload From the above results, the following observation can be made:

Under general workload
As we observe in Figure 10, we use the FCT of short flows, long flows, and a combination of both short and long flows with general flows to compare the MDTLB performance with other protocols.
All of the FCTs were normalized to MDTLB for easy visualization.It is observed that MDTLB achieves less FCT than ECMP, because ECMP suffers more from hash collisions at a higher load.Moreover, MDTLB performs better at an 80% to 90% load compared to Clove and Herms for all types of flows, because MDTLB has better visibility and can react to the congestion.Furthermore, for all ranges expect 0.5-0.6,MDTLB had less FCT than most of the other protocols.MDTLB has lower a FCT than LetFlow, Drill, Presto, and CONGA for the range 0.3-0.5, because we observe that MDTLB reacts to the congestion in a more timely manner.Also, it is observed that MDTLB has achieved less than all of the other protocols in the range 0.7 until 0.9.Another observation is that the performance of MDTLB was more superior in long flows than other protocols, as we see in Figure 10b in comparison with Figure 10a,c.
Here, we focus on the FCT of short flows, long flows, and both short and long flows with the web search and data-mining workloads, which are both widely used workloads from deployed data centers, as shown in Figures 11 and 12, respectively.

Under the web-search workload
We observe that MDTLB has an average performance compared with other protocols.The reason is that the web-search considers bursty small flows.The frequent arrival of flows then creates flow gaps by breaking large flows into small flows.As shown in Figure 11, the FCT of MDTLB achieves one with the increasing load as compared to other protocols.However, excessive rerouting opportunities may negatively affect small flows without caution rerouting.As we can see in Figure 11a,c, the FCT for small flows grow dramatically as the load increases, which is because small flows are broken into several flowlets under heavy load; thus, they are heavily affected by packet reordering and congestion mismatch.In comparison, we observed that the FCT of MDTLB was lower compared with ECMP, Hermes, Clove, LetFlow, and CONGA due to being interpreted by the multi-level path characterization of MDTLB and the adaptive settings of the threshold.

Under the data-mining workload
In Figure 12, we observe that the MDTLB performance is better then Clove, Hermes, and Drill, because the data-mining workload is less bursty, and so the visibility of the network congestion becomes especially important to balance the load effectively.Also, we observe that MDTLB has achieved a higher FCT than DRB. Figure 12b shows that MDTLB outperforms ECMP, CONGA, Clove, Letflow, Presto, DRB, Hermes, and Drill; this is because MDTLB can effectively resolve the collision of large flows due to the multi-level setting aspects in it.However, MDTLB provides an FCT lower than most protocols, including DRB, when the workload increases.Furthermore, our protocol has achieved a lower FCT than Hermes, LetFlow, and Clove, because these solutions without good visibility can hardly balance the traffic.
The average throughput was evaluated for the different flows.Then, a comparison study has been made between the proposed protocol and the existing protocols (shown in Figures 13-15).It is observed that MDTLB has achieved the best throughput compared with existing protocols for datamining flows.It is also evident that the proposed protocol is the top among three existing protocols for general flows with CONGA and Presto, while it was among the top five protocols for web flow.This is consistent with the nature of web-search flows that have small bursts flows.

Related Work
The most recent and related algorithms, schemes, and protocols studies are discussed for the data center load balancing.Table 6 shows the summary of the existing scheme under the algorithm or mechanism functionality.
Researchers have proposed Presto, DRB, and RPS using per-packet per flow cell-based flow congestion load-balancing schemes [7,11].Many researchers had proposed congestion-aware

Related Work
The most recent and related algorithms, schemes, and protocols studies are discussed for the data center load balancing.Table 6 shows the summary of the existing scheme under the algorithm or mechanism functionality.
Researchers have proposed Presto, DRB, and RPS using per-packet per flow cell-based flow congestion load-balancing schemes [7,11].Many researchers had proposed congestion-aware

Related Work
The most recent and related algorithms, schemes, and protocols studies are discussed for the data center load balancing.Table 6 shows the summary of the existing scheme under the algorithm or mechanism functionality.
Researchers have proposed Presto, DRB, and RPS using per-packet per flow cell-based flow congestion load-balancing schemes [7,11].Many researchers had proposed congestion-aware

Related Work
The most recent and related algorithms, schemes, and protocols studies are discussed for the data center load balancing.Table 6 shows the summary of the existing scheme under the algorithm or mechanism functionality.
Researchers have proposed Presto, DRB, and RPS using per-packet per flow cell-based flow congestion load-balancing schemes [7,11].Many researchers had proposed congestion-aware switching techniques to balance the data center loads.However, from the assessment, it is evident that most of the schemes suffer from the congestion mismatch in an asymmetric condition [12,18].
In continuation, to recover, the congestion mismatch issue advanced programmable switches that were engaged [13].The main target was to achieve better visibility and throughput.Flowlet switching had projected balancing the traffic surrounded by parallel paths.Nevertheless, the flowlet has the limitation of reacting to the congestion situations under stable conditions that lead it to have slower convergence and suboptimal performance.A flowlet weighted round-robin based algorithm had been imposed at the end host where the path weight was estimated using piggybacked ECN signals [19].
A multipath transmission control protocol (MPTCP) [20,21] has been using as a transport protocol to routes subflows toward multiple paths.This is because it cannot change paths, which is why the MPTCP routed paths congestion mismatch issue is very low [22][23][24].However, it carries some limitations on the modification of the end host networking stacks, which really gives the challenge for the multiple customers.On top of that, there are bad incast scenarios.A DRILL-based load-balancing solution had appeared, which mainly maintained the per-packet load to overcome the microbursts under heavy load conditions [25].Moreover, it has also a limitation for the asymmetric topologies that leads to the congestion mismatch issue.Another solution, namely flowbender [26], has appeared to reroute the flows blindly whenever the congestions are detected.This technique is used to apply the random and dynamic rerouting to carry the suboptimal performance under high load conditions.In reference [27], random packet spraying (RPS) is proposed.RPS is a random based load-balancing routing algorithm, which randomly assigns packets to one of the available shortest paths to the destination.CLOVE-ECN [28] adopts per-flowlet weighted round-robin attend hosts, where the path weights are calculated based on piggybacked ECN signals, due to their limited visibility and slow reaction to congestion.The authors in [29] presented DiffFlow with the help of SDN using random packet spraying (RPS) and a new load-balancing solution to detect the long flows and forward packets.In [30], the authors presented a mechanism named flow distribution aware load balancing in order to decrease the flow completion time and obtain higher scalability.Furthermore, the DFFR algorithm [31] was proposed to balance the flows in the data center networks.It is a scalable, distributed, and adaptive algorithm designed for maximizing network resources.A data center load-balancer named Beamer is developed to guarantee a stateless mux operation [32].Moreover, in [33], the authors introduced a load-balancing protocol that was based on sampling named Luopan in order to overcome the shortcomings of the existing protocols.

21 Figure 1 .
Figure 1.Data center image of application.

Figure 1 .
Figure 1.Data center image of application.

Symmetry 2019 ,Figure 2 .
Figure 2. Asymmetry because of broken link between L0 and S1, when the link capacity is 10 G [7].

1. for each path p do: 2 .
if f ECN < T ECN and T RTT < T RTT_LowS then type = very good 3. else if f ECN < T ECN and T RTT < T RTT_LowD then type = good 4. else if f ECN > T ECN and T RTT > T RTT_highS then type = bad 5. else if f ECN >T ECN and T RTT > T RTT_highD then type = very bad 6. else type = gray 7. if (n timeout > 3 and no packet is ACKed) or ( f retransmission > 1% and type = bad or very bad), then type = failed end for If (good portion is small) or (good portion is medium and bad portion is big) or (good portion is big and bad portion is big) then changeRatio = ChangeRatio + 10 Else if ( good portion is big and bad portion is not big ) then changeRatio = ChangeRatio − 10 End Symmetry 2019, 11, x FOR PEER REVIEW 7 of 21

Figure 4 .
Figure 4. Block diagram of path judging logic.Figure 4. Block diagram of path judging logic.

Figure 8 .
Figure 8.The example of the multi-level dynamic traffic load-balancing (MDTLB) method.

Figure 8 .
Figure 8.The example of the multi-level dynamic traffic load-balancing (MDTLB) method.

Figure 10 .
Figure 10.Flow completion time of various loads for the general flows in the symmetric topology with three types of flows: (a) short flows FCT; (b) long flows FCT; (c) and overall FCT of both short and long flows.

Figure 10 .
Figure 10.Flow completion time of various loads for the general flows in the symmetric topology with three types of flows: (a) short flows FCT; (b) long flows FCT; (c) and overall FCT of both short and long flows.

Figure 11 .
Figure 11.Flow completion time of various loads for the web search application in the symmetric topology with three types of flows: (a) short flows FCT; (b) long flows FCT; (c) and overall FCT of both short and long flows.

Figure 11 .
Figure 11.Flow completion time of various loads for the web search application in the symmetric topology with three types of flows: (a) short flows FCT; (b) long flows FCT; (c) and overall FCT of both short and long flows.
(a) Short flows FCT (b) Long flows FCT (c) Overall FCT of both short and long flows.

Figure 12 .
Figure 12.Flow completion time of various loads for the data-mining application in the symmetric topology with three types of flows: (a) short flows FCT; (b) long flows FCT; (c) and overall FCT of both short and long Flows.

Figure 12 .
Figure 12.Flow completion time of various loads for the data-mining application in the symmetric topology with three types of flows: (a) short flows FCT; (b) long flows FCT; (c) and overall FCT of both short and long Flows.

Figure 13 .
Figure 13.Average throughput comparison among the protocols for the general load.

Figure 14 .
Figure 14.Average throughput comparison among the protocols for the web search load.

Figure 15 .
Figure 15.Average throughput comparison among the protocols for the data-mining load.

Figure 13 .
Figure 13.Average throughput comparison among the protocols for the general load.

Figure 13 .
Figure 13.Average throughput comparison among the protocols for the general load.

Figure 14 .
Figure 14.Average throughput comparison among the protocols for the web search load.

Figure 15 .
Figure 15.Average throughput comparison among the protocols for the data-mining load.

Figure 14 .
Figure 14.Average throughput comparison among the protocols for the web search load.

Figure 13 .
Figure 13.Average throughput comparison among the protocols for the general load.

Figure 14 .
Figure 14.Average throughput comparison among the protocols for the web search load.

Figure 15 .
Figure 15.Average throughput comparison among the protocols for the data-mining load.

Figure 15 .
Figure 15.Average throughput comparison among the protocols for the data-mining load.