Supervised Learning of Neural Networks for Active Queue Management in the Internet

The paper examines the AQM mechanism based on neural networks. The active queue management allows packets to be dropped from the router’s queue before the buffer is full. The aim of the work is to use machine learning to create a model that copies the behavior of the AQM PIα mechanism. We create training samples taking into account the self-similarity of network traffic. The model uses fractional Gaussian noise as a source. The quantitative analysis is based on simulation. During the tests, we analyzed the length of the queue, the number of rejected packets and waiting times in the queues. The proposed mechanism shows the usefulness of the Active Queue Management mechanism based on Neural Networks.


Introduction
Cisco predicts that by 2022, the Internet traffic will increase to 77 exabytes per month due to the rapid development of mobile technologies. The mobile data transfer will increase sevenfold compared to 2017, with an average annual growth of 46% [1]. The rapid increase in the number of Internet users as well as the transmission of multimedia content of increasing quality force the continuous development of data transmission mechanisms.
Wide area networks have their origins in the 1970s and were created for the American army. Thus, the most important aspect of the network based on a distributed architecture was to deliver reliable transmission of data and low connection costs. Unfortunately, the design assumptions proposed at the beginning turned out to be insufficient over the years.
Initially, IP routers handled packets according to the FIFO (First In First Out) rule (the first incoming packet in the queue is the first one to be served) [2]. For such scheduling, packets are dropped when the queue length exceeds the maximum length which results in the retransmission of a large number of packets in a short period of time. For such a network model, it is very difficult to control transmission throughput, delay and packet dropping [3].
To solve this problem, the Internet Engineering Task Force (IETF) proposed Active Queue Management (AQM) mechanisms [4]. These mechanisms preemptively drop packets before queue overflow occurs. In addition, the rejection of a packet should force the sender to reduce the transmission speed, which is provided by TCP congestion window mechanism [5]. The AQM algorithms used with TCP can enhance the efficiency of network transmission [4]. this way is used as a new AQM mechanism. This paper presents its influence on the Internet transmission.
The remainder of the paper is organized as follows: Section 2 describes the current state of the art in this field. Section 3 presents the theoretical background. Section 4 is a description of the structure of the Artificial Neural Network, the data and the experimental methods used to obtain the results for this research. In Section 5 there is a description of the results of the conducted experiments. Section 6 concludes our research.

Related Works
There are many works regarding new AQM algorithms. These mechanisms are compared with existing solutions in terms of transmission parameters such as total number of dropped packets, average queue length, or transmission delays. In the article [7] passive and active queue management mechanisms were compared. Other works focus only on the comparison of the AQM mechanisms [6,29]. The topics of research in network and computer system performance evaluation also include works considering the impact of self-similarity of network traffic on transmission efficiency [10].
Additionally, the fractional order PI controller [30] is used for the Active Queue Management. This research is still under development, and its mechanisms have also been subjected to an analysis of the effect of the degree of self-similarity and long-term dependence of the traffic [31].
A separate group includes studies that have used neural networks to improve the queue management mechanism in TCP networks. The article [32] proposed the AQM mechanism based on reinforcement learning-Q-learning RED. The authors of [33] proposed an ANB-AQM mechanism, in which a back-propagation algorithm was used to train the neural networks to make decisions about accepting or rejecting packets. Article [34] proposes a neural network model, which modifies the REM algorithm, called the Fuzzy Neuron REM (FNREM) mechanism. This mechanism modifies the value of the proportional integral of the REM algorithm, by using the value of the proportional-integral derivative neuron as an indicator of overload.
ANNs were also used to create a new algorithm-Adaptive Neuron Proportional Integral Differential (ANPID) [35]. This mechanism used a single neuron to tune the PID controller coefficients. The authors of [36] presented the results based on the simulation and the real tests in the Linux Kernel, which resulted in the presentation of another adaptive modification of the PID controller using neural networks-the GRPID mechanism.
In article [37] authors presented an improved PID AQM/TCP system based on the network built using the Long Short-Term Memory (LSTM) layers (a specific type of a recurrent layer). It allows to predict queue length in the next step. They used Root Mean Square Error (RMSE) as a loss function. LSTM layers were also used to predict the occurrence of transmission overloads [38].
The research presented in XuIeee is an example of an attempt to use unsupervised learning to create a more efficient AQM mechanism. For that purpose, the Hebbian Learning rule is used and a new adaptive PHAQM algorithm is presented.
Bisoy and Pattnaik [39] used feed-forward neural network to create an AQM mechanism, namely FFNN-AQM. The network consisted of two input neurons, three neurons in a single hidden layer and the single output neuron.
Zhou et al. [40] also presented an adaptive AQM mechanism based on a single neuron whose weights were selected using reinforcement learning rules. The application of reinforcement learning was also used in [41] to build a mechanism to reduce transmission delays.
There are many works on the topic of AQMs based on neural networks. However, in these works, in contrast to our approach, the neural networks were mainly created using reinforcement learning. In addition, the research results did not consider the analysis of the effect of traffic self-similarity and long-term dependence on transmission efficiency.

Theoretical Background
Self-similarity is widely observed in nature, but the term itself was introduced by Mandelbrot in 1960s and it generally means that the portion of the whole object can be considered an image of the whole in a reduced scale. The object is self-similar, when it exhibits the same statistical properties independently of the scale. Mandelbrot described it on the example of the scaled coastlines, which also exhibited self-similarity. This property can also be used in the case of time-series analysis. The degree of self-similarity in this case determines whether Long-Range Dependence (LRD) and Short-Range Dependence (SRD) occur in data. These relationships were observed as early as the middle of the twentieth century, when Sir H. E. Hurst described the occurrence of long-range dependence based on the value of water level fluctuations in the Nile River. Although the terms of self-similarity and LRD are sometimes used interchangeably, they are not the same [42].
A continuous-time series Y(t) is exactly self-similar when the following condition is satisfied: for t ≥ 0, a ≥ 0 and 0 < H < 1. It results in the statistical invariability in different time scales. H is usually used to denote the Hurst exponent/parameter, which expresses the degree of self-similarity. The parameter can take values from range (0; 1), and specific values represent: • H ∈ (0; 0.5): negative correlation-the LRD does not occur (the SRD occurs). • H = 0.5: no correlation. • H ∈ (0.5; 1): positive correlation-the LRD occurs.
It was first proven in [43] that actual network traffic exhibits self-similarity. This work provided the motivation for numerous studies that demonstrated the significant impact of self-similarity on TCP transmissions [44], or to confirm its occurrence in Wide Area Networks (WANs) [45]. Self-similarity results in performance degradations, such as mean queue length enlargement and the increase in packet loss probability [42]. The topic of self-similarity is still relevant in the literature and found its application in e.g., DoS attack detection (e.g., [46]). Our previous works were also related to this topic. They regarded determining the degree of traffic self-similarity expressed by the Hurst parameter and also using data obtained from the IITiS data traffic traces to examine self-similar properties [25]. Self-similarity significantly impacts queue occupancy and transmission performance [47]. For that reason, the samples generated for the purpose of this article are characterized by different degrees of self-similarity.
Artificial Neural Networks have found application in many different domains, e.g., image classification, natural language processing, signal processing etc. Additionally, Deep Learning approaches have become a solution to many problems due to their better ability to extract patterns than shallow learning [48]. The versatility of neural networks has resulted in them also being frequently used in the network traffic domain for tasks including attack detection [49,50], traffic generation [51] and classification of the traffic type [52].
Network traffic and its features are often represented as a time series. To process time-series data with Artificial Neural Networks, different types of networks (e.g., Autoencoders) and layers can be used, namely Recurrent layers (including Long-Short Term Memory (LSTM) layers and Gated Recurrent Unit (GRU) Layers) and 1D Convolutional Neural Networks.
Autoencoders can be built using different types of layers, e.g., Dense Layers or Convolutional layers. The goal of this type of network is to compress input data and then reconstruct it on output [53]. It can be used for the purpose of data denoising, but also anomaly detection. When the neural network is not able to reconstruct the input data well, it suggests that the sample can be anomalous [53].
LSTM layers are often used for the purpose of time-series data processing. Single LSTM units solve the gradient vanishing and exploding issues typical for simple Recurrent Layers and are able to propagate gradients over a long period of time [54]. The key characteristic of this type of layer is that they store the internal state, which enables them to 'remember' the past information [52]. Due to that their internal 'memory' is longer than in the traditional recurrent units.
The alternative for LSTM layers is a GRU layer. It is very similar to LSTM layer, also stores the Long-Time memory of the past information, which is vital for time-series processing. Nevertheless, it is simpler to implement and compute than the LSTM layer, thus more efficient [55].
Additionally, convolutional layers can be used to process time-series data. In this case, time has to be treated as a spatial dimension [22]. In fact, it is an efficient alternative to recurrent layers. In a Convolutional Neural Network, transformed time-series data are processed in turns using convolutional and pooling layers. As a result deep, more abstract representations are generated on the basis of raw data. Processing ends with a classifier part (Multi-Layer Perceptron), which consists of dense layers.

Data Preparation and Neural Networks Training Process
In this paper, we used artificial neural network models to develop an active queue management mechanism. The neural networks were trained to mimic the operation of the AQM based on the fractional order PI α controller mechanism. The training data were generated based on simulation data, and a detailed description of the learning model is given in this section. The neural network model was based on four convolutional layers and two dense layers. After each convolutional layer, the data were normalized and the results were averaged. Additionally, a dropout layer was placed to prevent over-fitting to the learning data. Python and Keras libraries were used to implement the model. The conceptual structure of the model used in this paper is presented in Figure 1. To design this model structure we relied on the experience of our earlier work [25], where the degree of selfsimilarity of network traffic was classified using Convolutional Neural Networks expressed by Hurst parameter.
In order to prepare the training set for the proposed neural network model, network simulations were performed, reflecting the queueing behavior of a fractional order controller PI α . The values of the fractional order PI α controller parameters have been presented in the Table 1. These values were determined based on our previous work [26]. The results of these articles have shown that the choice of controller parameters significantly affects the queue length control properties. The process of choosing proper AQM/PI controller parameters is non-trivial. It has a significant impact on the packet dropping function (i.e., for an integral order α it can strengthen and accelerate the response of a controller). Properly selected AQM parameters should allow us to obtain adaptation to the changing transmission conditions and desired queue behavior. We discussed the influence of these parameters on queue behavior in papers [15]. The controller parameters were chosen in such a manner that controller PI α 1 was the weakest controller, and controller PI α 3 was the strongest one, which implies a large number of packet rejections and ease of maintaining the desired queue length.  To obtain training data for an AQM model based on Convolutional Networks, network simulations were performed using the AQM mechanism. For this purpose, the discrete event simulator SimPy (written in Python) was used. This software is available under the MIT License and has been used in our previous works regarding the evaluation of AQMs [21,26].
Our simulation model was a discrete model of a G/M/1/N queue. The simulation time was divided into discrete time intervals of length dt. Arrival of a packet was generated (or not) in a given time slot by a traffic source. The source of traffic was self-similar and based on Fractional Gaussian Noise (FGN) process. The advantages of such a source have been described previously in the articles [10,15,25].
All experiments considered different degrees of traffic self-similarity expressed using Hurst parameter. In experiments the Hurst parameter changed between H = 0.5 (no correlation) and H = 0.9 (high degree of LRD).
The input intensity coefficient was set to a constant value λ = 0.5. Thus, the simulation packet source always had a constant intensity. Parameter µ represents the time of packet processing and dispatching (probability of taking a packet from the queue). Different values of this coefficient were used in the experiments. The parameter µ took values between µ = 0.5 (moderately stressed system) to µ = 0.15 (highly stressed system). This choice of simulation parameters allowed us to observe all properties of the AQM mechanism.
In our experiments, we considered different numbers of items from queue occupancy history taken into consideration in the samples used to train Convolutional Networks. For simplicity, we refer to this number of samples as 'CNN History'. This length corresponded to the number of time slots in the simulation model that were used as training data for the network. For example CNN = 200 refers to 200 * dt time intervals taken into consideration. Throughout this time, we observed the behavior of the AQM queue. Thus, the training data consisted of: Learning features: (a) The last n items from the queue's occupancy history (CNN History).

2.
Classes: (a) 11 labels that mapped the probability of packet rejection to the current transmission conditions, according to the principle shown in Table 2. Therefore, we considered different lengths of queue occupancy history, because from the perspective of the router, which is a low resource device, minimizing the length of the history would be beneficial. In our study, we tried to determine the minimum acceptable length of n last items of the queue's occupancy history.
For each probability interval, one million one-dimensional learning records were prepared. Therefore, the training set consisted of 11 million records. They contained transmission information such as the length of the queue in each consecutive time slot, the number of dropped packets, and the value of the PI α controller's packet rejection probability function. We present the process of data preparation in Figure 2. This amount of data seemed to be sufficient in comparison with the cardinality of data reported in the literature [56]. Input data prepared in such a manner were used in the process of supervised learning of the neural network models. In order to train the model and minimize the cost function, the optimizer Adaptive Moment Estimation (Adam) was used with the following parameters: where: η is the learning rate, β 1 is the exponential decay rate for the first moment estimates and β 2 is the exponential decay rate for the second moment estimates. The Adam optimizer is expressed by the equation [57]: where v is the first moment, which resembles momentum that records the past normalized gradient, s is the second moment and g denotes the gradient descent.
In both the four convolutional layers and the two dense layers, ReLU was used as the activation function and Sigmoid/Softmax functions were used to determine the activation of the output layer. Categorical cross-entropy was used as a cost function. Figure 1 shows the conceptual structure of a neural network model used for the purpose of active queue management mechanism.
We limited the training process to 10 epochs. This value was sufficient, since the values start to stabilize after only 5-6 epochs, as confirmed by the results in Tables 3-5. We also compare the accuracy of the model, when Softmax activation function (Table 3) and Sigmoid activation function (Table 4) were used in the output layer. Higher results were obtained for the Sigmoid function.
In the case of Softmax function (Table 3), the minimum accuracy was 32.3%, and the maximum 58.9%. For the models in which we applied the Sigmoid activation function for the last layer the minimum accuracy was 48.77% (for the network trained on the data from the PI α 3 controller, where the CNN History = 20), and the maximum 89.46% (for the network trained on the data from the PI α 3 controller, where the CNN History = 1000). Taking all the results into consideration, the best results were obtained for the CNN History ≥ 500, and the worst for the CNN History < 100 (Table 4).
In the case of the model trained on data representing the behavior of three controllers simultaneously and the use of the Sigmoid activation function of the output layer, the maximum accuracy was 72.1% for the CNN History ≥ 500 (see Table 5).

Evaluation of the Neural Network-Based AQM
This section presents the behavior of the trained neural network (as assumed in Section 4 and evaluates its effectiveness as an AQM mechanism. This evaluation was performed using previously described simulation mechanisms. During the study, we evaluated the number of packets dropped from the queue and the average queue occupancy. We compared the effectiveness of the neural network-based AQM mechanism with the results of the PI α controller-based AQM mechanism. We used the network traffic with different degrees of self-similarity during the experiments. To increase the readability of the paper, we present only two extreme cases -the results obtained for a non-self-similar traffic (H = 0.5) and for a traffic with high degree of LRD (H = 0.9).
The intensity of the packet source in the simulation was assumed to be (lambda = 0.5). On the other hand, the packet service time in a system was set to a constant value (µ = 0.25) in order to obtain a heavily loaded system.
In our experiments, we evaluated four separate neural network models. The first three neural networks were trained with the data obtained from controllers PI α 1, PI α 2, and PI α 3. The fourth model was trained with data regarding all of these controllers. In the first phase of the experiment, we considered two neural network models (see Figure 1): the first one with Softmax, and the second one with Sigmoid activation function of the last layer.
A comparison of Tables 3 and 4 shows that although Softmax function is more commonly used in the literature as an activation function of the output layer of the neural network for multiclass classification, Sigmoid function performs better in our case. In the worst case, in which the network obtained accuracy of 32.31%, changing the activation function to Sigmoid resulted in significant accuracy increase (65.65%). Additionally, in the best obtained case accuracy changed from 58.90% to 86.65%. Figures 3 and 4 show average queue lengths for AQM mechanism based on neural network. Detailed results are compared on Tables 6 and 7 for Sigmoid function and on Tables 8 and 9 for Softmax function. Both presented networks imitate the behavior of the first controller-PI α (see Table 1). Comparing the number of discarded packets and the average queue sizes, we find that they are similar regardless of the chosen network activation function in the last layer. As Hurst increases, the number of dropped packets decreases slightly in the case of Sigmoid function (<1%).   Taking into consideration higher accuracy obtained using Sigmoid function, we chose this function to be used in further experiments. Figure 3 compares the behavior of two AQM mechanisms: PI α 1 controller and the CNN-based AQM trained on the data reflecting the behavior of this controller. For the CNN model, different lengths of the last n elements of the queue occupancy history (input to the neural network) were considered. Regardless of the value of n, the resulting queue length distributions are similar to the queue length distribution of the PI α 1 controller. For Poisson traffic (non-self-similar traffic, H = 0.5), the average queue length oscillates between 166 and 176 packets (see Table 6). For highly self-similar traffic (parameter H = 0.9), the average queue length was between 139 and 147 (see Table 7). In this case, all the Convolutional Neural Network models (with different numbers of CNN History) obtained larger values of the average queue length, with fewer packets dropped, than the PI α 1 mechanism. Figure 5 presents the results for stronger AQM mechanism PI α 2. The detailed results of dropped packets numbers and queue lengths are presented in Tables 10 and 11. Because of the fact that the PI α 2 controller was stronger than the one presented above, the obtained average queue lengths were smaller.      Table 9. Detailed results of queue occupancy obtained for CNN model with the last layer activation function Softmax, trained using data regarding PI α 1 controller and parameters: K P = 0.0001, K I = 0.0004, α = −0.4 and H = 0.9.  Figure 6 compares the last pair of controllers: controller PI α 3, with the corresponding models based on Convolutional Neural Networks. The results prove that this controller is the strongest one. The AQM mechanism increased considerably the number of dropped packets and decreased the obtained queue lengths. In the case of traffic without LRD (see Table 12, for parameter H = 0.5) the average queue occupancy oscillateds between 116 and 139 packets, and in the case of traffic characterized by a high degree of LRD (see Table 13, for parameter H = 0.9) between 94 and 121 packets.

AQM
It should be noted that for all three CNN-based AQM models, a more efficient AQM model was obtained compared to the controllers that were used to create the test data. Even for the model that obtained the smallest accuracy during the learning process (48.77%, see Table 4), based on non-integer controller data of order PI α 3, for CNN History < 100, the obtained average queue length was larger than for the base mechanism PI α 3. This situation occurred both for traffic without LRD (see Table 12) and for traffic characterized by a high degree of LRD (see Table 13). Table 11. Detailed results of queue occupancy obtained for CNN model with the last layer activation function Sigmoid, trained using data regarding PI α 2 controller parameters: K P = 0.0001, K I = 0.0004, α = −0.5 and H = 0.9.    In the next simulation step, we evaluated the AQM-CNN mechanism whose learning data were generated from the behavior of all three PI α controllers. Figure 7 shows the queue distribution, and Figure 8 shows the changes in queue occupancy over time. Details of the number of packets dropped and the resulting average queue occupancy are presented in Table 14 for the traffic without LRD and in Table 15, for traffic with a high degree of LRD.

AQM
The results show that when the number of last n elements of queue occupancy history taken as a CNN input is too small (CNN History < 100), then, independent of the degree of self-similarity of the traffic, the number of dropped packets, and the average queue length, approximates the results obtained using the sets of controllers PI α 2 and PI α 3 (see Tables 10-13).
On the other hand, when the considered number of last n queue occupancy history elements is larger (CNN History ≥ 100), the obtained average queue length increases by 46 packets for traffic without LRD (Table 14, for H = 0.5), or by 32 packets, for traffic characterized by a high degree of LRD (see Table 15, for H = 0.9). This means that the resulting queue distribution matches the one of the original and the most efficient controller PI α 1 (Figure 3). This feature indicates that for the AQM model based on Convolutional Networks, as the number of story elements used increases, the ability of the mechanism to adapt to current Internet transmission conditions also improves.

Conclusions
The paper presents a new Active Queue Management mechanism based on Convolutional Neural Networks and supervised learning.
To train the Convolutional Networks used in the experiments, data obtained through simulation have been used. The training data of the CNN model reflect the behavior of the AQM mechanism, based on a fractional order controller PI α .
In our experiments, we took into account the effect of the degree of traffic self-similarity and long-term dependence on the performance of the proposed mechanism.
We also considered the effect of the number of last n elements of the queue occupancy history, used as input of the neural network, on the efficiency of the proposed mechanism.
The best results were obtained for CNN History = 500. The minimum length of CNN History for which results are still acceptable is 100.
In the experiments, neural networks with different number of convolutional layers and different optimizers and cost functions were considered to build the AQM model. After comparing the results obtained with different activation functions, the results have shown that the most efficient model used Sigmoid activation function in the output layer, therefore we chose this function for further experiments. The decisions made in this work were also influenced by our previous work regarding traffic classification in terms of the degree of self-similarity [25].
The most efficient AQM obtained in our study was based on the Convolutional Neural Network model, trained using the data reflecting the behavior of all three PI α controllers jointly.
The results confirmed that the model based on Convolutional Neural Networks can effectively reproduce the results of the classical AQM algorithm and effectively manage the data transmission. Such a model maintains the assumed average number of packets in the queue and reduces the total number of dropped packets, independent of the degree of traffic self-similarity.
It seems that the proposed mechanism exhibits some advantages over previously proposed mechanisms encountered in the literature. Our previous study [26] demonstrated that the reinforcement learning methods are well suited for maintaining the assumed queue size. However, in computer networks, the process of controlling packet traffic is more complex. The objective is to maximize the transmission efficiency. This efficiency is characterized by: throughput, delay, and possible retransmissions. Efficiency of AQM mechanisms is influenced by self-similarity of network traffic. The higher the Hurst parameter value is, the greater problems with correct packet management occur. The proposed solution addresses this problem much more effectively. The biggest disadvantage of this solution is greater computational and memory complexity of solutions based on Convolutional Neural Networks. This complexity may affect the difficulty of implementing this solution in real routers.
In our previous study [58], we used a Linux-based computer as a router. In that study, we used a special router implementation based on a special forwarding mechanism (based on the iptables mechanism), which delivered all packets to the user program implementing AQM. This solution greatly simplifies the research model. Unfortunately, the tests have shown that forwarding packets from kernel space to userspace requires a significant amount of time and is not optimal. In the target solutions the whole implementation should be realized in the kernel of the system. The implementation may be a great challenge on routers with low hardware resources. In such solutions instead of multiplication operations bit shifting is used, which causes calculation errors. For CNN calculations these errors may be too high. However, it seems that the computational power of routers will increase in the future. We want to devote a separate article to the problems of implementing AQMs in real routers.