PFDCT: An Enhanced Transport Layer Protocol for Precise Flow Control in Data Centers

Abstract

This paper proposes an enhanced transport layer protocol for precise flow control in data centers, referred to as PFDCT, combining explicit rate control (priority scheduling) and implicit rate control (marking algorithm with dynamic threshold) innovatively, while satisfying the transmission requirements of each data flow. It enables multiple data flows to share the bandwidth fairly, which is rarely achieved by previous data transmission protocols. Our design stems from the observation that different packets in a data center network need to be processed differentially and that the accuracy of the feedback of the indication information to the link state will directly affect the effectiveness of the congestion control mechanism. Our tests show that PFDCT has lower network jitter, better fairness, and significant performance gains over existing proposals (DCTCP, ICTCP, and L2DCT) in terms of Flow Completion Time (FCT) and throughput on average. Furthermore, we discussed the impact of the marking algorithm with dynamic thresholds on the performance of PFDCT, and the results of the experiments show that the addition of the marking algorithm will further improve the performance of PFDCT in terms of packet loss rate, latency, and FCT.

1. Introduction

Data center networks (DCNs) use centralized infrastructures to provide users with diverse network services and large-scale data computing, such as video streaming and cloud computing. A data center network has several characteristics that make it different from the Internet or Local Area Networks (LANs). First of all, data center networks are specially designed for data-intensive communications. It can provide users with large-scale computing services such as network storage, email, and web search. These computing services have a great impact on the transmission bandwidth between distributed components in the network. Higher requirements were put forward. Secondly, there are thousands of densely interconnected servers in the data center, and its hardware scale is much larger than the general LAN infrastructure. How to flexibly manage the communication between servers is also a challenge.

Aggregator applications in data centers, such as web search or MapReduce, often request data from thousands of servers simultaneously, leading to a many-to-one traffic situation. In this case, numerous workers simultaneously establish Transmission Control Protocol (TCP) connections with the aggregation application and transmit the data requested by the application back to the aggregator, causing a large amount of traffic data to be aggregated suddenly to the bottleneck switch. Even worse, bottleneck switches often have very shallow buffers. Therefore, most concurrent TCP connections will inevitably lose a large number of data packets, and in severe cases, multiple TCP timeouts will occur, resulting in a serious drop in network throughput and greatly prolonged data transmission completion time. Researchers refer to this catastrophic network throughput collapse caused by many-to-one concurrent connections as TCP incast [1]. With the explosive emergence of many-to-one aggregator applications in today’s data centers, the TCP incast problem is becoming one of the most serious problems affecting the quality of service of data centers.

The most obvious and serious impact of TCP incast on the data center network is packet loss and even timeout. As shown in Figure 1, For each query, the application in data center network retrieves data from a number of servers. The query cannot complete until responses are received from all servers. However, it is well known that foreground applications generate mostly short flows (for example, 1 KB to 10 KB) [2,3], while background applications generate mostly long flows (for example, 1 MB to 100 MB), which are sensitive to network throughput [4,5]. In other words, once TCP incast causes data block loss, the transmission of short flows cannot meet the delay requirement, and the throughput of long flows will be seriously affected as the query cannot be completed.

A mass of mechanisms and protocols have recently been proposed to solve the data congestion problem caused by the TCP incast problem; however, these schemes usually focus on only one dimension of the complex problem space, ignoring the need for different idiosyncratic flows to be controlled by different data transfer strategies. For example, DCN algorithm [6] only controls long flows, while T-RACKs [7] and CWND [8] focus on improving Flow Completion Time (FCT) of small flows to ensure short FCT of short flows. Other approaches [9,10] do provide excellent performance for long flows but perform poorly for short flows [11,12].

To ensure the transmission of both long and short flows without sacrificing another, we proposed a novel flow-aware and delay-based transport layer protocol for data centers, PFDCT (Transport Layer Protocol for Precise Flow Control in Data Centers). PFDCT utilizes the end-to-end delay, Explicit Congestion Notification (ECN) [13] marking and the characteristics of flows to jointly improve the service quality of the data center network. The main contributions of PFDCT are:

• In order to provide long-term stable throughput for the link, PFDCT only performs ECN marking on long flows, and PFDCT optimizes the traditional ECN fixed marking threshold to a traffic-aware dynamic marking threshold to cope with complex and frequently changing data center network environments.
• PFDCT has developed a priority-based queue management mechanism according to the urgency of data packet transmission, data transfer rate, and data flow size. It simultaneously guarantees the transmission quality of long-flow packets, short-flow packets, and retransmission packets, which constitute almost all traffic in the data center network. The above measures make the ECN marking algorithm of PFDCT more fair and efficient during data transfer.
• Considering the complexity of the model and measurement accuracy, PFDCT combines two kinds of feedback information, the readily available Round-Trip Time (RTT) from the network and the widely supported ECN marking, to obtain the most accurate link status while saving link overhead. The above measures can ensure that the work of PFDCT is more reasonable and efficient.
• We tested the performance of the PFDCT protocol in terms of end-to-end delay, flow completion time, and packet loss rate, and compared it with the scheme under the fixed threshold, finding that the above indicators of PFDCT are better than the fixed-threshold one to varying degrees. Extensive evaluations showed that PFDCT performs much better on FCT compared to DCTCP, ICTCP, and L2DCT in the simple-scale topology. For a link load of 100%, the throughput of PFDCT is 0.41 MB higher than that of L2DCT, which is better than other similar schemes. In the large-scale topology, PFDCT also maintains a steady increase in throughput, with a stable fairness index of 0.55, which is more than 20% higher than ICTCP.

The rest of the paper is organized as follows: Section 2 summarizes the related work to solve the TCP incast problem. Section 3 describes PFDCT in detail. Section 4 analyzes the changes brought by the dynamic-threshold-based ECN marking algorithm to PFDCT. Section 5 presents the implementation of PFDCT and analyzes the performance evaluation. Section 6 concludes the paper.

2. Related Work

Several schemes are proposed to deal with the incast problem in data center networks and we will present some of the most widely known schemes and their corresponding limitations below. To carry out accurate positioning, evaluation, and analysis of the existing scheme, and also make the writing more fluent, we divided the solutions to TCP incast in data center networks into the transport layer solutions and the non-transport layer solutions. We will first introduce the typical schemes of the non-transport layer below.

The scheme of the non-transport layer is mainly to solve the TCP incast problem from the perspectives of the data link layer and the application layer. Quantized Congestion Notification (QCN) [14] and its variants are typical congestion control mechanisms implemented at the link layer. The QCN scheme consists of two parts, one is the congestion point (CP) dynamics and the other is the reaction point (RP) dynamics. The CP’s switch buffer samples incoming packets to determine the source of congestion, and the RP adjusts the sending rate after receiving congestion feedback information. Since most data flows causing TCP incast are transmitted synchronously, QCN does not perform well in many-to-one transmission scenarios. Prajjwal Devkota et al. [3] improved the QCN scheme and attributed the QCN performance degradation to flow variability. They employed adaptive sampling at the switch and traffic adaptive approach at the rate limiter to improve the performance of the scheme in the TCP incast scenario. Unlike the above solutions, application layer solutions always use the method of modifying the end host to avoid incast. CCMA [15] is a concurrent connection management agent for incast mitigation, it is an application-layer agent on the receiver side that manages concurrent TCP connections on behalf of the aggregated application. For multiple co-existing aggregators, CCMA schedules them sequentially to avoid traffic collisions. DSRAL [16], ARS [10], and PICC [17] avoid TCP incast by controlling the number of concurrent TCP connections. However, these methods all assume that the receiver is aware of the remaining available bandwidth, which is usually not valid in real data centers. Since DSRAL (and [10,15,16]) is built into each aggregator application, it inherently cannot control traffic congestion caused by multiple concurrent senders connected to the same receiver.

In the transport layer schemes, a lot of work tries to solve the incast problem by adjusting TCP parameters. For instance, AP [18] uses pacing to change the sending interval. Swift [19] uses the Additive Increase Multiplicative Decrease (AIMD) algorithm to control end-to-end delay, and uses pacing to control packet sending interval in the case of extreme network congestion. Similarly, Packet Slicing [20] enables packet slice to mitigate packet loss and throughput degradation caused by traffic bursts. These methods make it easier to trigger the network’s fast retransmission mechanism rather than Retransmission TimeOut (RTO). Other solution, e.g., CP [21] tries to reduce retransmission latency by reducing $RT{O}_{min}$ (200 ms by default) to the same timescale as network latency. However, these solutions may lead to premature and unnecessary retransmissions, which in turn trigger more timeout events. Especially when the number of TCP concurrency is large [20], the probability of TCP incast does not decrease.

The ECN mechanism improves and extends the TCP/IP protocol, so that the network can enable the terminal to obtain link information without using packet loss as a congestion feedback signal. In DCNs, commodity switches make the ECN marking decision as packets are dequeued. ECN relies on active queue management (AQM) schemes such as Random Early Detection (RED) [22] to track queue changes at bottleneck switches. When the queue length of the switch buffer exceeds the preset threshold, ECN will mark the Congestion Experienced (CE) bit in the Internet Protocol (IP) header, thereby notifying the terminal to adjust the data sending rate. Once an endpoint observes that a packet has experienced congestion, the sender reduces the transmission rate (the window is halved).

Alizadeh et al. identified the user’s demand for DCN transmission, and indicated that the early ECN solution could not provide high-performance services for the data center. In data centers, the number of background flows is small (e.g., 2 or 3) and their congestion windows tend to be synchronized. Additionally, most traffic is considered to be bursty in nature. Therefore, the behavior of the ECN feedback mechanism to halve the congestion window when the link is likely to be congested will cause severe jitter instead of gracefully converging to the available bandwidth [23]. To sum up, 1-bit ECN feedback can only indicate link congestion, but not the degree of network congestion.

In order to achieve fine-grained congestion window adjustment, DCTCP marks packets using the instantaneous queue length and the ECN marking threshold K of a single queue, which be maintained independently for other queues. DCTCP uses the packet marking mechanism of ECN to obtain link feedback. At the same time, considering the limitations of standard ECN, DCTCP adopts a simpler active queue management scheme. It ensures that once the occupancy of the queue exceeds a fixed threshold K, the switch can mark subsequent arriving packets with a CE code point. In order to notify the sender that it is experiencing congestion in a timely manner, DCTCP sends an acknowledgment (ACK) for each packet and communicates back to the sender the exact sequence number of that packet. The ECN-Echo flag is set if and only if the packet has a marked CE codepoint. Through these changes, DCTCP greatly improves the throughput of data center networks, and DCTCP and ECN-based window control schemes have received great attention for their good applicability and performance [24].

Many solutions also try to solve TCP incast problems by changing the ECN tag threshold, changing the position of the ECN marks, and solving the buffer overflow problem caused by microbursts. CEDM [25] merged enqueue and dequeue ECN marks, A-ECN [26] adjusted the packet marking threshold as packets arrive to accommodate multi-queue conditions in the data center network. However, these methods barely avoid timeout upon heavy incast, since in this case packet loss is so massive and bursty that TCP usually receives no ACK at all (i.e., full window loss) [18,20,27]. RTT in traditional TCP/IP networks differs by several orders of magnitude from the block transfer time in data center networks, which means that RTT in traditional IP networks is no longer suitable for data center networks with high latency requirements. To this end, RAPID [28], T-Racks [29], and TCP-EFR [30] have focused on improving the fast retransmission/fast recovery mechanism in traditional networks, starting with repeated ACK, to avoid triggering RTO in the event of congestion and optimize short-flow FCT in response to congestion.

Whether a solution is deployed at the transport or non-transport layer, more and more attention has been paid to the use of packet priority in recent years. PRIN [31] established a model to analyze the causes of the TCP incast phenomenon and reduced the flow completion time of short flows by improving the priority of the last three packets at the end of the block. RAPID [28] assigned a higher priority to retransmission packets to solve the packet loss problem during data retransmission. PTCP [32] calculated the priority of packets based on the size and expiration date of traffic, and then the priority is used to adjust the receiving window and control ACK interval time to minimize the FCT.

To make up for the shortcomings of the above schemes and complement the priority-based schemes, we propose PFDCT. Based on the flow characteristics, PFDCT uses RTT as congestion feedback, which is a congestion control mechanism similar to DCTCP [33]. PFDCT configures different priorities for packets based on the urgency of their delivery. In particular, retransmitted packets are given the highest priority to avoid triggering the RTO mechanism. Once packets overflow from the switch’s shallow buffer and are lost, PFDCT will do its best to complete the delivery of retransmitted packets as quickly as possible. The most significant difference from the above schemes is that PFDCT considers each type of packet (long-flow packet, short-flow packet, and retransmission packet) most commonly seen in the data center, balancing the transmission of each flow and never sacrificing one flow to any other. PFDCT combines the switch-side control with the sender-side control and adjusts the ECN marking threshold dynamically to adapt to the rapidly changing network environment by obtaining the queue length through RTT.

3. PFDCT Design

The design of the PFDCT is presented in this section. The first part introduces the basic idea of the PFDCT protocol, and the second part gives the design details of the protocol. The packet type of PFDCT and the adopted switch model are defined in the latter part. In the end all the algorithms of PFDCT are described, including how to assign priority to all packets, update the ECN marking threshold and adjust the congestion window.

3.1. Basic Idea

DCN supports various applications with complex flow structures. To allocate network resources more reasonably and improve network service quality, PFDCT requires differentiated control on different traffic and uses a more comprehensive, fairer, and more efficient transmission control mechanism to meet the transmission requirements of different traffic. The idea of PFDCT is that the sender will record the size of each flow at the beginning and will configure different priorities for the packets based on the urgency of delivery and the packet size. Only for long flows does PFDCT enable the ECN marking algorithm. Next, by calculating the queue length of the long flow queue, the sender will project the congestion level, which will be used to reduce the congestion window dynamically. In the control process, PFDCT uses a flow-aware adaptive threshold adjustment algorithm to control the transmission of long flows more accurately considering the dynamic network environment and the transmission delay uncertainty.

The system architecture of PFDCT is briefly presented in Figure 2. As shown in the figure, we focus on seven common data packets in the data center network. For the convenience of description, we assume that the users in the figure will request data from servers located in different racks. If the request packet reaches the server through the layer-by-layer switch, the server will send out the data packet corresponding to the connection. When the data packet sent belongs to a long connection, the connection is enabled with ECN marking, and the initial value of the ECN marking threshold is the same as that of DCTCP. Taking the core switch as an example, when the data packet arrives at the switch buffer, the switch will identify the type of the data packet according to the ECT flag and the Pending Interest Table (PIT) table, and put them into different queues for queuing and forwarding them out in order. The queue management method of PFDCT makes the retransmission data packets be forwarded out at the earliest. As the amount of data continues to increase, the traffic injected into the network exceeds the processing capacity of the link, and network congestion occurs. At this time, the sender will update the ECN marking threshold in time, and adjust the congestion window in real time according to the proportion of the statistically marked long-flow data packets. Finally, after the data transmission is completed, the user will send an ACK packet to the corresponding server to confirm the successful receipt of the data.

3.2. Protocol Details

3.2.1. Packet Types

To meet the actual needs of diversified packet identification, PFDCT uses the Differentiated Services Code Point (DSCP) priority specified in the RFC2747 [1] standard in the packet header and uses the first 6 bits of the 8 bits of the Type of Services (ToS) field to assign priority for specific packets (long-flow packets, short-flow packets, and retransmission packets). At the same time, PFDCT only enables the ECN algorithm for the long flow and sets the ECN field of the short flow header to invalid at the data-sending end. The reason for this is that we realized that traditional ECN marking methods indiscriminately drop packets that exceed the queue threshold but the short-flow packets are usually very sensitive to latency. When coexisting with long-flow packets, it often occupies less bandwidth and cannot compete with long-flow packets, so it is in a very disadvantageous position. When the packets exceeding the queue threshold include short-flow packets, the short-flow packets are also discarded. This operation has great interference with the transmission of short flows, which is considered to be the excessive control of ECN on short-flow packets.

3.2.2. Switch Service Model

The switch in this solution supports ECN tag recognition and uses RED queue management algorithm and priority queue management algorithm to identify the priority area of the packet header.

PFDCT sets two thresholds for the RED algorithm, a high threshold and a low threshold ($K_{max}$ and $K_{min}$). When the average queue length exceeds $K_{max}$, all arriving packets are marked by ECN. When the average queue length is between $K_{max}$ and $K_{min}$, each arriving packet will be marked by ECN with a certain probability, which varies with the average queue length. By doing so, RED can reduce the proportion of ECN marked packets when the queue length is between $K_{max}$ and $K_{min}$, which can reduce the overreaction of the sender and thus keep the throughput high. Therefore, PFDCT improves the RED algorithm so that the queue length threshold in RED is the same as that used in PFDCT, which is dynamic and can greatly reduce the packet loss rate in the traditional RED algorithm. In addition, PFDCT will no longer perform ECN marking on short flows and provide higher priority for short-flow packets to reduce the transmission delay of short flows.

PFDCT’s queue management algorithm implements a strict priority policy, where packets are dequeued from bands only if higher priority bands are all empty. It is a sorted queue tray that can have any number of bands, each band being handled by any type of queue tray. There is no limit to the capacity of the queue disk, and data packets can only be discarded by the sub-queue disk (there may be a capacity limit). It should be noted that the queue disk needs at least two sub-queue disks. In the scenario of this experiment, PFDCT added two sub-queue disks to the queue disk, both of which are RED queue disks.

3.2.3. Discriminating Priority Assignment

To further provide high-quality transmission for short and long flows, PFDCT maintains different queuing queues for long and short flows on the switch. The forwarding priorities of the two queues are different, and the priority of short flows is higher than that of long flows. PFDCT brings two innovations in the priority-based forwarding mechanism: (1) it treats the retransmitted packets as the most urgently delivered packets, and (2) it further refines the existing short flows and long flow classification criteria, selectively giving higher priority only to packets requiring shorter FCT.

PFDCT regards the maximum cwnd as a rough measure of short flows and considers a maximum cwnd less than 20 as one of the characteristics of small flows. In general, PFDCT’s flow classification method is based on two features of flows, including flow size and maximum congestion window. If the flow size is greater than 1 MB and the maximum congestion window is greater than 20, classify the flow as a long flow. Otherwise, classify this flow as a short flow. In PFDCT, the retransmitted packet is given the highest priority regardless of whether it belongs to the long-flow packet or the short-flow packet. Short-flow packets are given the second highest priority, while long-flow packets are given the lowest priority.

Algorithm 1 describes how PFDCT works on switches. In our scheme, the switch classifies incoming packets by identifying the ECN field in the packet header and retrieving the PIT table, and forwards these packets out according to their corresponding priority, which will help PFDCT achieve differentiated management of data center network traffic. When the data packet arrives at the switch, PFDCT will use the $IsRetransmitted$ function to determine whether the data packet to be forwarded is a retransmitted data packet, and the return value type of this function is Boolean (lines 1–6). PFDCT will first check whether the sequence number of the data packet already exists in the PIT of the switch. If it already exists, it indicates that the data packet is a retransmission packet, otherwise, the data packet is a fresh packet, then the function returns the retransmission flag value. The $SetPrio$ function is used to assign forwarding priorities to different types of packets. When the data packet is a retransmission packet, PFDCT will give it the highest priority. Otherwise, PFDCT will check the ECN flag. When the flag bit of the data packet is not activated, it means that the data packet is a short-flow data packet, and PFDCT will give the data packet the second-highest priority; otherwise, the data packet is a long-flow data packet, and PFDCT will give the data packet lowest priority of the three (lines 7–13). Next, the algorithm obtains the priority bands (lines 14–15). The $GetBandforPriority$ function is used to assign priority bands to packets of different priorities, that is, to maintain different priority forwarding queues in the switch buffer. Then $DoEnqueue$ function uses the packet filter to sort the packets and return the ret tag. If the filter fails to filter out a packet of that priority, PFDCT obtains the priority of the packet and puts it into the corresponding priority band (lines 16–22). Otherwise, the value of ret is assigned to the priority band, which means there is a packet filter with a corresponding value of ret, and the packet filter is returned (lines 23–25). If the selected band is out of range, the packet is queued on the queue disk and finally, PFDCT returns the packet filter value (lines 26–27).

Algorithm 1 PFDCT Switch

Input: SocketPriorityTag priorityTag Output: prioband 1: function Isretransmitted($seq$) 2:     if seqisrecorded==Yes then 3:         retag $\leftarrow yes$ 4:         prio $\leftarrow thehighest$ 5:     retag $\leftarrow no$ 6:     return retag 7: function Setprio($seq$) 8:     if retag==No then 9:          10:         if ECNisunabled then 11:            prio $\leftarrow themediatepriority$ 12:         prio $\leftarrow thelowestpriorityamongthethree$ 13:     return prio 14: function GetBandForPriority($prio$) 15:     return $prioband\left[prio\right]$ 16: function DoEnqueue($item$) 17:     band $\leftarrow prioband\left[0\right]$ 18:     ret $\leftarrow Classify\left(item\right)$ 19:     if ret==PacketFilter::NOMATCH then 20:          21:         if item.GetPacketpriorityTag then 22:            band $\leftarrow prioband[priorityTag.GetPriority(\left)\right]$ 23:     else 24:         if $ret>=0andret<GetNQueueDisc\left(\right)$ then 25:            band $\leftarrow ret$ 26:     retval $\leftarrow GetQueueDisc\left(\right)\to Enqueue\left(item\right)$ 27:     return retval

3.2.4. Congestion Window Update

In this part, PFDCT uses queue length as auxiliary congestion feedback to mark packets more carefully. At each RTT, PFDCT estimates the current queue length as:

$$L_Q=\frac{RT{T}_c-RT{T}_{min}}{RT{T}_c}·cwnd$$

(1)

where $RT{T}_{min}$ represents the minimum RTT, $RT{T}_c$ represents the current RTT and cwnd represents the current congestion window. Unless otherwise noted, we use the parameter description in

Table 1. Parameters and description.

Parameters	Description
$RT{T}_c$	Current RTT
$RT{T}_{min}$	The minimum RTT
$cwnd$	Current congestion window
$K_i$	The ideal threshold
$K_t$	The dynamical threshold
${\alpha }_{ns}$	Congestion indicator
$L_Q$	The current queue length
g	Weight factor for congestion factor
$F_{ns}$	The percentage of packets that are tagged
h	Weight factor for $L_{QV}$

.

Next, we will discuss how to queue lengths are used to determine congestion levels. When the sender receives the ACK packet, the sender estimates the queue length according to the RTT and further updates the queue length threshold. Specifically, PFDCT marks arriving packets when the instantaneous queue length is larger than a predefined threshold. After all packets in this congestion window are acknowledged, PFDCT can obtain the sum of marked packets. The network congestion degree $F_{ns}$ can be characterized by calculating the proportion of marked packets in the entire window. Larger $F_{ns}$ means more severe congestion. Then the reduction factor in PFDCT is updated once for each data window (approximately one RTT) as follows, and the g value is the same as used in DCTCP:

$${\alpha }_{ns}=(1-g)\times {\alpha }_{ns}+g\times F_{ns}$$

(2)

After that, PFDCT will update the congestion window based on the congestion level. The traditional TCP adopts the AIMD algorithm to update its cwnd. When congestion occurs, AIMD cuts cwnd by half which is appropriate when transferring data over the Internet. However, due to the extremely low latency and shorter flow characteristics of data center networks, the Multiplicative Decrease (MD) algorithm is too aggressive and does not perform well in data center networks. PFDCT improves TCP in data center networks by moderately reducing cwnd according to the congestion degree.

$$cwnd=\left\{\begin{array}{cc}cwnd+1\hfill & \mathrm{if}{F}_{ns}=0\hfill \\ cwnd\times \left(1-\frac{{\alpha }_{ns}}{2}\right)\hfill & \mathrm{if}{F}_{ns}>0\hfill \end{array}\right.$$

(3)

According to Equation (3), when no marked packet ($F_{ns}$ = 0) is received, the congestion window increases by 1. If $F_{ns}>0$ at this time, which means that packets are marked and the network is in the congestion state, PFDCT will reduce the congestion window gently according to the congestion level ${\alpha }_{ns}$ in each round.

3.2.5. Dynamic ECN Marking Threshold Setting

Our suggestion is to decide whether to reduce the sending rate to avoid congestion according to the current congestion level. As the pioneer of the data center network transmission protocol, DCTCP uses the ECN marking scheme to improve TCP and make it suitable for the data center network. However, the fixed ECN threshold is difficult to meet the strict requirements of data centers for traffic transmission due to the complexity of the modern data center network traffic, and the diversity, and dynamics of the network environment. Therefore, the PFDCT protocol uses a dynamic ECN marking threshold algorithm to generate an adaptive marking threshold based on traffic awareness. The algorithm sets an appropriate value for each long flow according to the characteristics of the flow. It should be noted that PFDCT does not use the same marking algorithm for short flows, in other words, short-flow packets will not be constrained. This implementation is supported by the switch model in this scenario, which is detailed in the switch model section.

If the queue length exceeds a given threshold, PFDCT will mark the packet. PFDCT reduces packet loss and retransmission timeouts by keeping the switch’s queue length around a threshold. Specifically, setting a marking threshold for ECN too high can lead to packet accumulation and unnecessary queuing, which introduces high end-to-end transport latency. Conversely, if the threshold is too small, the bottleneck link will not be fully utilized, resulting in reduced throughput. In order to balance queuing delay and link utilization, PFDCT sets an adjustment range for $K_t$, the upper boundary is called $K_u$, and the lower boundary is called $K_l$. Preset an ideal target threshold $K_i$ for the queue length to maintain an appropriate end-to-end queue length. In short, the threshold $K_t$ will be updated as Equation (5). The $L_{QV}$ in Equation (4) indicates the average queue length, that is:

$$L_{QV}=(1-h)\times L_{QV}+h\times L_Q$$

(4)

where h is a weight parameter and satisfies 0 < h < 1.

$$K_t=\left\{\begin{array}{cc}max\left(K_t,K_l\right)\hfill & \mathrm{if}{L}_{QV}>Ki\hfill \\ min\left(K_t,K_u\right)\hfill & \mathrm{if}{L}_{QV}<Ki\hfill \end{array}\right.$$

(5)

Next, we analyze the minimum and maximum queue oscillation to obtain an optimal threshold $K_i$ and achieve low latency without sacrificing the throughput. PFDCT denotes the link capacity (packets/second) as C, the round trip time as RTT, and the number of flows as N. We assume that the N window sizes are synchronized and follow the identical sawteeth, then the queue size at time t is given by:

$$Q\left(t\right)=NW\left(t\right)-C\times RTT$$

(6)

We assume that the queue length of the switch at this point is exactly equal to the threshold sum of each flow, then we can obtain:

$$N{K}_t=Q\left(t\right)$$

(7)

At the next moment, since no congestion is detected on the link, the window of each flow will increase by 1 according to the window modulation algorithm, then:

$$\begin{array}{cc}\hfill Q(t+1)& =N\left(W\right(t)+1)-C\times RTT\hfill \\ \hfill & =NW\left(t\right)+N-C\times RTT\hfill \\ \hfill & =Q\left(t\right)+N\hfill \end{array}$$

(8)

Now the queue length exceeds a preset threshold, all the senders will observe a positive queuing delay, and they respond by decreasing the window size using the multiplicative factor:

$$Q(t+2)=NW\left(t\right)\left(1-\frac{{\alpha }_{ns}}{2}\right)+N-C\times RTT$$

(9)

As mentioned above, the change of the switch queue length always follows the sawtooth model, so it is not difficult to deduce that the queue length at $t+2$ is equal to $Q_{min}$, namely:

$$\begin{array}{cc}\hfill Q_{min}& =Q(t+2)\hfill \\ \hfill & =NW\left(t\right)\left(1-\frac{{\alpha }_{ns}}{2}\right)+N-\mathrm{C}\times RTT\hfill \\ \hfill & =Q\left(t\right)-NW\left(t\right)·\frac{{\alpha }_{ns}}{2}+N\hfill \end{array}$$

(10)

It can be seen from DCTCP that ${\alpha }_{ns}=\sqrt{\frac{2}{W\left(t\right)}}$, and if Equation (7) is substituted into Equation (6), we can calculate the relation between W(t) and $K_t$ and N as:

$$W\left(t\right)=K_t+\frac{C·RTT}{N}$$

(11)

Further, PFDCT converts $Q_{min}$ into an algebraic expression completely related to thresholds $K_t$ and N:

$$Q_{min}=N{K}_t-N\times \sqrt{\frac{N{K}_t+C\times RTT}{2N}}+N$$

(12)

where $Q_{min}>0$, it means that the minimum queue length should be greater than 0 packet to ensure higher link utilization. Thus, we can obtain:

$$K_t+1-\sqrt{\frac{N{K}_t+C\times RTT}{2N}}\ge 0$$

(13)

This results in:

$$K_t\ge \frac{\sqrt{\frac{8\times C\times RTT}{N}-7}-3}{4}$$

(14)

For convenience, the inequality 14 is denoted by $f\left(N\right)$, and it is not difficult to see that $f\left(N\right)$ is the inverse function of the number of concurrent flows N. In other words, if the threshold $K_t$ satisfies the inequality when N = 1, $f\left(N\right)$ will obtain smaller as the number of concurrent flows increases and the threshold $K_t$ will satisfy the inequality. Finally, we can obtain the ideal marking threshold $K_i$ as:

$$K_i=f\left(1\right)=\frac{\sqrt{8\times C\times RTT-7}-3}{4}$$

(15)

After obtaining the ideal threshold $K_i$, PFDCT needs to further determine the upper limit $K_u$ and lower limit $K_l$ of the marking threshold. Considering the burstiness and concurrency of data center traffic, the total threshold of concurrent traffic must be smaller than the buffer capacity M of the bottleneck switch, i.e., $N{K}_u<M$, $K_u<M/N$. After comprehensive consideration, $K_l$ used in this experiment is 0, and $K_u$ is M/N, which will be adjusted according to the bandwidth and delay.

Algorithm 2 describes how PFDCT works on the sender side. The main tasks include simple processing of packet headers to distinguish between long and short flows, obtaining link status in real time, setting dynamic thresholds for long flow queue markers, and modulating congestion windows. Firstly, PFDCT uses the $ShortorLong$ function to read the retransmission flag, flow size, and congestion window size of packets to distinguish between long-flow packets, short-flow packets, and retransmission packets. The division of long-flow packets and short-flow packets should first confirm that they all belong to non-retransmission packets. The sender enables ECN when the $ShortorLong$ function determines that the packet belongs to the long flow, otherwise the sender sets the ECN flag to inactive (lines 1–7). Next, PFDCT calls the $GetQuelength$ function to obtain the queue length in the bottleneck switch using the method demonstrated in Equation (1) (lines 10–12), and the $GetidealTh$ function to obtain the ideal queue marking threshold under the current experimental parameters using Equation (15). PFDCT sets the right-hand expression of inequality 14 to $f\left(N\right)$. It can be seen that $f\left(N\right)$ is an inverse function with respect to the number of concurrent flows N. In other words, if the marking threshold $K_t$ can satisfy the inequality when N = 1, then as N keeps increasing, the value of $f\left(N\right)$ becomes smaller instead, and the threshold $K_t$ must also satisfy the inequality, at which point the value of $f\left(N\right)$ as a function of $f\left(1\right)$ at N = 1 can be set to the ideal marking threshold (lines 7–9). When all messages in the window have been received, PFDCT calls the $ManThresh$ function to update the marker threshold for the ECN. If the average queue length at this point is greater than the ideal threshold, PFDCT will raise the marker threshold, otherwise, it will lower the marker threshold (lines 13–24). Finally, the $IncreaseWindow$ function will calculate a new congestion window value based on the proportion of packets that are marked. That is, when the proportion of marked packets is 0, the congestion window is increased by 1. When the proportion of marked packets is greater than 0 ($F_{ns}$ > 0), the sender reduces the congestion window according to the value of the congestion factor (lines 25–32).

To sum up, PFDCT strictly classifies data center traffic and provides different forwarding priorities to meet its transmission requirements. In particular, PFDCT combines ECN feedback and RTT feedback, which are most common in data center networks, to capture link information more accurately. In order to change the poor adaptability of the fixed marking threshold in DCTCP, PFDCT utilizes the queue length information to make the marking threshold change in real-time according to the network congestion state, further improving the network throughput. The next chapter provides a performance comparison of the two protocols.

Algorithm 2 PFDCT Sender

Input: Flowsize Urgencydegree Output: The updated threshold 1: function ShortorLong($retag,flowsize,cwnd$) 2:     if flowcwnd <20 and txBytes <10 KB then 3:         ectCodePoint $\leftarrow NotECT$ 4:     else 5:         ectCodePoint $\leftarrow ECT$ 6:     return ectCodePoint 7: function GetidealTh($segmentsAcked,bytesInFlight$) 8:     idealTh $\leftarrow \left(sqrt\right(8\ast linkBandwidth\ast RTT-7)-3)/4$ 9:     return idealTh 10: function GetQuelength($RTT$) 11:     quel $\leftarrow cwnd\ast \left(\right(RTT-minRTT)/RTT)$ 12:     return quel 13: function ManThresh($segmentsAcked,bytesInFlight$) 14:     quel $\leftarrow Getquelength\left(RTT\right)$ 15:     avequel $\leftarrow (1-h)\ast avequel+h\ast quel$ 16:     lowerlimofth $\leftarrow queDiscs/totalflownumber$ 17:     upperlimofth $\leftarrow queDiscs/totalflownumber$ 18:     if txBufferHeadSequence>= highTxMark and txBuffersize==0 then 19:         idealTh $\leftarrow GetidealTh\left(\right)$ 20:         if avequelength > idealTh then 21:            newTh $\leftarrow max(newTh,lowerlimofth)$ 22:         else 23:            newTh $\leftarrow min(newTh,upperlimofth)$ 24:     return newTh 25: function IncreaseWindow($segmentsAcked$) 26:     if ackedBytesEcn > 0 then 27:         cwnd $\leftarrow cwnd\ast (1-bytesEcn/2)$ 28:     else 29:         if ackedBytesEcn==0 then 30:            adder $\leftarrow segmentSize\ast segmentSize/cwnd$ 31:            cwnd $\leftarrow cwnd+max(1.0,adder)$ 32:     return cwnd

4. Impact of the Threshold of PFDCT

To illustrate the positive impact of the dynamic threshold-based packet marking algorithm on PFDCT, we test the performance of PFDCT in terms of latency, FCT, and packet loss rate when using dynamic thresholds and fixed thresholds, respectively, and compare them. In addition, we refer to PFDCT with a fixed threshold as PFDCT-ST in this section for convenience.

The network topology used for simulation in this section is a three-level fat tree topology with 250 servers, 25 core switches, and 10 pods, with 5 aggregation switches and 5 Top of Rack (ToR) switches in each pod. This topology is typical in data center networks, also used in the evaluation of recently proposed protocols [4,16,20,34]. Each ToR switch is connected to up to 20 servers with a 1 GB link. All switches have a buffer size of 100 packets, and the packet size is 1.5 KB, the link delay is 10 µs.

In the many-to-one traffic context, clients receive data from servers located in the same rack or randomly selected servers. In each run, we simulated 2000 transfer sessions (or I/O requests), each of size 1 MB and 4 MB. Each I/O request is randomly assigned to a host in the network as a client. The placement of each data block is based on the Hadoop Distributed File System (HDFS) default strategy, more specifically, we assume that the client itself is not the server, and the data blocks are placed on other random nodes on the same rack. If no data block is stored in the same rack, the client will read a data block from a server located in the same rack or from a server chosen at random.

Flow Completion Time. Figure 3 shows the FCT performance comparison curves of PFDCT and PFDCT-ST. When the time is less than 0.02 ms, PFDCT behaves similarly to PFDCT-ST, and when the time is greater than 0.04 ms, the FCT of PFDCT-ST gradually increases and is higher than that of PFDCT. It is not difficult to see that PFDCT has stronger adaptability and can keep the FCT of the data flow stable when the network environment changes, while PFDCT-ST is greatly affected by the network environment changes and the network changes and the FCT of the data flow presents an increasing trend.

Loss rate. It can be seen from Figure 4 that in the case of stable data transmission, the packet loss rate of the network using the PFDCT protocol remains at 0, and the most serious packet loss rate is about 0.002% which can be ignored. Compared with PFDCT, PFDCT-ST loses more packets but it is gratifying that the maximum packet loss rate of PFDCT-ST can still be kept below 0.01%, which may benefit from the priority scheduling strategy.

Delay. It is clear from Figure 5 that the average delay of PFDCT is 50% lower than PFDCT-ST. Moreover, as the data volume increases, PFDCT can still maintain almost zero end-to-end delay, while PFDCT-ST shows an increasing trend. This indicates that the marking algorithm using a fixed threshold can hardly cope with severe data congestion when the data volume increases, using a dynamic threshold is more suitable for a data center network with heavy load and multiple senders, and can effectively avoid TCP incast.

5. Evaluation

We have extensively evaluated PFDCT’s performance through large scale, packet-level simulations and compared it to the state of the art. To do so, we tested PFDCT for throughput, FCT, packet loss, and fairness, analyzing its performance differences with DCTCP, ICTCP, and L2DCT.

5.1. Simulation Setup

We conduct simulation experiments under two different topologies, including simple-scale topology and large-scale topology. We set the minimum RTO of TCP to 200 µs and the average size of packets to 1.5 KB. The default link bandwidth is 1 GB, and the default link delay is 10 µs. Each switch port in the simulation has a shallow buffer that can only hold 100 packets (approximately 100 × 1.5 KB = 150 KB). In the experiment, we set the g value as 1/16, the maximum marking threshold of the switch queue is 60, and the minimum marking threshold is 20 packets.

DCTCP [23]: DCTCP uses a very simple AQM technique on the switch. When the number of packets in the queue is greater than the marking threshold K, packets arriving after that are marked with CE code point. For packets marked with the CE code point, the receiver sets the ECN-echo flag of the corresponding ACK and estimates the proportion of marked packets at the sender. Finally, DCTCP will adjust the congestion window size according to the degree of congestion.

ICTCP [12]: ICTCP calculates the available bandwidth and throughput of the receiving end every two $RT{T}_s$. ICTCP is deployed at the receiving end of the data center, and adjusts the size of the sliding window by detecting bandwidth utilization to fully utilize link resources. It divides the total bandwidth into two parts, the first part is used to receive all traffic and predict the bandwidth utilization, and this value is used to adjust the sliding window of the remaining part of the bandwidth in turn.

L2DCT [35]: L2DCT also uses ECN as congestion feedback, and uses $\alpha$ and traffic weight to adjust the congestion window. $\alpha$ is the same as used in DCTCP, while weights are assigned to the flow by estimating data sent so far. When both long and short flows exist, the short flows will obtain a higher bandwidth share, and when only long flows exist, L2DCT will penalize them. In case of severe congestion, all traffic shares fall back off entirely.

5.2. Simple-Scale Experiments

In this experiment, we measured the impact of the PFDCT protocol on the link throughput and FCT under different loads. Many previous works have shown that the number of concurrent flows is an important parameter affecting the performance of the scheme, so we also compare the performance of various mechanisms when there are different numbers of concurrent flows to verify the practicality of the scheme. In the topology scenario shown in Figure 1, we set up multiple concurrent senders to send data to the same receiver at the same time and all hosts are under the same switch, which is a typical TCP incast scene.

Throughput. In this part we measure the application throughput for PFDCT in a distributed storage setup which involves many-to-one communication. In each run, we simulate different numbers of transport sessions with sizes of 1.5 KB each. Furthermore, we generate different link loads and evaluate how PFDCT works compared to other schemes. We measure the overall throughput from the time the client initiates the transmission until the last server receives the whole data. The results for various loads and concurrent flows are shown in Figure 6. From Figure 6a, we can see that PFDCT performs load balancing between short-flow packets and long-flow packets, and its throughput has always maintained an advantage of around 5 MB above DCTCP even with the increasing number of concurrent flows. As the number of concurrent flows continues to increase, the change in link throughput brought by PFDCT is very similar to ICTCP and L2DCT. We believe that the reason is PFDCT gives short flows more opportunities to transmit and restricts the transmission of long flows that are more capable of providing throughput accordingly.

Keeping the above configuration unchanged and controlling the number of concurrent flows to 30, we tested the performance of PFDCT and other schemes under different loads. As we can see in Figure 6b, the performance of PFDCT is still significantly higher than that of DCTCP, and the throughput of PFDCT is about 30% higher than that of DCTCP when the load is not heavy. For 100% link load, PFDCT shows the best performance that we measured an average throughput of 32.35 MB for PFDCT, a 0.41 MB improvement over the highest value in the comparison schemes.

Flow Completion Time. Next, we tested FCT for various protocols to see how the performance of PFDCT would improve compared to previous transport protocols. We showed the average FCT for multiple protocols with different numbers of concurrent flows in Figure 7a and the specific values can be seen in

Table 2. The average FCT with varying numbers of senders sending 15 KB flows in simple-scale topology.

Flow Numbers	L2DCT (s)	ICTCP (s)	DCTCP (s)	PFDCT (s)
10	4.168	4.311	4.166	4.267
20	3.898	3.904	3.774	3.810
30	3.402	3.419	3.243	3.185
40	2.890	2.910	2.724	2.658
50	2.459	2.448	2.433	2.367
60	2.499	2.441	2.346	2.295
70	2.487	2.463	2.412	2.300
80	2.481	2.472	2.441	2.324
90	2.518	2.472	2.364	2.300
100	2.510	2.488	2.440	2.305
Avg.FCT	2.931	2.933	2.834	2.781

. PFDCT performs the best in scenarios where the number of flows is high (greater than 20) due to priority delivery for short flows and the supported multiple queue management mechanisms.

We fixed the average flow data size at 1.5 MB and the number of concurrent flows at 30 and kept increasing the link load to test the variation of FCT for PFDCT under different loads. Figure 7b shows the statistics of the average FCT of PFDCT, L2DCT, DCTCP, and ICTCP. At high loads, PFDCT reduces the average flow transmission time by more than 250 ms, and the multi-queue management mechanism makes it unnecessary for PFDCT to trim packets belonging to short flows. Under the same conditions, PFDCT reduces FCT by about 15% relative to DCTCP which confirms the network can provide shorter FCT by using a dynamic ECN marking threshold mechanism. Figure 8 further depicts the distribution law of the average FCT when every protocol acts on the network with different loads.

5.3. Large-Scale Experiments

As shown in Figure 9, we used a FatTree topology with 160 servers, which has four topologies A, B, C, and D, and each pod has eight core switches and two aggregation switches. The buffer size of the ToR switch and aggregation switch shown in the figure is 100 data packets, and the buffer size of the core switch is 250 data packets. The link bandwidth between the aggregation switch and the core switch is 10 GB by default, and the link bandwidth between other devices is 1 GB by default. Meanwhile, we set the default values of link delay and RTT to 50 µs and 300 µs, respectively, and run each simulation multiple times to report the average values or the aggregate values.

In each run, we simulate 60 transfer sessions of size 100 KB and 2 transfer sessions of size 2 MB from Pod A to Pod D (traffic one), 70 transfer sessions of size 200 KB, and 5 transfer sessions of size 2 MB from Pod B to Pod D (traffic two), and 90 transfer sessions of size 300 KB in Pod C (traffic three). By doing so, the receiver can receive traffic from the sender located in the same pod (traffic three) and different pods (traffic one and two), respectively, which is a typical data center traffic transfer mode.

Throughput. We plotted the average throughput of traffic one of four mechanisms at varying times in Figure 10 and the specific values can be seen in

Table 3. Throughput comparison of multiple schemes in large-scale topology.

Time (s)	$\mathit{L}2\mathit{DCT}$ (Mbps)	$\mathit{ICTCP}$ (Mbps)	$\mathit{DCTCP}$ (Mbps)	$\mathit{PFDCT}$ (Mbps)
0–5	131.48	22.88	201.31	166.47
5–10	611.81	216.14	757.68	748.86
10–15	1028.46	568.59	1275.68	1286.10
15–20	1444.95	1058.17	1694.90	1816.22
Avg.throughput	797.20	464.44	972.69	998.89

. PFDCT exhibits similar performance to DCTCP because the sender increases the congestion window linearly at begin continuously. It is worth noting that, unlike other protocols, the PFDCT protocol allows the throughput of the link to continue to climb over time. This is because the short flow reaches the sender in time, freeing up a large amount of transmission bandwidth on the link and providing more transmission space for the long flow.

Flow Completion Time. As we can see from Figure 11a, the performance of PFDCT is similar to other schemes since the available bandwidth of the link is sufficient at the beginning. As the data starts to obtain more and more and the network environment becomes more complex, PFDCT can still maintain a low FCT proving that PFDCT can provide more stable transmission performance for the link compared to other protocols. Figure 11b further depicts the distribution law of the average FCT when every protocol acts on the network.

Loss rate. Simulation data for loss rate in large-scale experiments are analyzed and compared in this part. We can see from Figure 12 that PFDCT achieves the same zero packet loss rate as DCTCP and ICTCP, except that the packet loss rate of L2DCT exhibits oscillation below 0.05%.

Fairness. We conducted fairness index tests on PFDCT and other protocols to show that PFDCT can achieve a fair share of bandwidth among various traffic. As can be seen from Figure 13, the fairness index of PFDCT was always on the rise before 8 s, after then the fairness index remained at about 0.55, which confirms that PFDCT can achieve more reasonable bandwidth sharing between short and long flows by giving short flows a higher priority.

6. Conclusions

In this paper, we proposed PFDCT, a transport layer protocol that addresses TCP incast problem for data centers. It is the first transport protocol that committed to congestion control for so many types of traffic in a data center network while performing at least as well as the state of the art concerning goodput and FCT for long and short flows. PFDCT achieves this remarkable combination by integrating AQM and delay-based congestion feedback with joint flow control between sender and intermediate, packet classification, and in-network priority scheduling. This paper also discussed the impact of the marking algorithm with dynamic thresholds on the performance of PFDCT at last, and the results of the experiments show that the addition of the marking algorithm will further improve the performance of PFDCT in terms of packet loss rate, latency, and FCT. The delay-based link state feedback method, congestion window modulation algorithm, and dynamic threshold marking method are deployed on the sending side. The priority-based multi-queue management method only relies on the intermediate device switch to identify packet header specific fields to assist in implementation. PFDCT has good generality and negligible modifications to the switch side of modern data center networks.