Multi-Agent-Based Traffic Prediction and Traffic Classification for Autonomic Network Management Systems for Future Networks

Round 1

Reviewer 1 Report

The paper presents the MANA-NMS architecture with implementation considerations for NTPA and NTCA. The implementation is based on the osBrain library.

The main weaknesses of the card are the following.

1) The performance analysis of prediction and classification algorithms is more or less similar to previous results well known in the literature, such as references [23] and [26]. The authors should clarify whether the performance of the compared algorithms is affected by the proposed MAS-based implementation. Does the MAS architecture change the performance of the compared algorithms? If so, why? If not, what is the novelty of the proposed comparison?

2) The authors should clarify how figures 8,9 and 10 were obtained. For example, does figure 8 represent the traffic forecast reported on the y axis after a training period equal to the value reported on the x axis? What is the meaning of "Traffic equal to 75"? Why does the traffic curve decrease after 310 days? Is the traffic curve a cumulative value? Authors should explain the meaning and definition of RMSE and MAPE. Authors should try to provide some explanation as to why MLP gives a higher RMSE and MAPE than other techniques, not just by writing what the reader can directly observe from figures 8 and 9.

3) In general, the performance of a prediction algorithm also depends on the time horizon considered for the prediction. In this paper, the authors do not discuss the prediction horizon. Is the prediction horizon set to the next sample?

4) How was table 3 obtained? What is the meaning of the values shown in table 3?

The curves in figure 8 are almost indistinguishable. It is clear only that GRU and MLP give the worst performance. No differences between the other curves can be clearly observed.

In my opinion, the authors should provide the code used for the simulation to allow the results to be reproduced.

The card has many typos. For instance

line 45: citeHexa

line 189: wight

line 413: a: A)

In figure 7: Feeder

In several figures, this symbol [?] Appears in the caption.

Line 528. The sentence "In section 5.5 ..." is almost useless.

References 28 and 31 are the same.

Author Response

Dear  Reviewer,

We highly appreciated you taking the time to review our works and give us feedback. We have modified the paper accordingly and provided responses to each question. Please find below the response. We have also uploaded the modified version of the article highlighting blue the modified portion of the paper.

The main weaknesses of the card are the following.

1) The performance analysis of prediction and classification algorithms is more or less similar to previous results well known in the literature, such as references [23] and [26]. The authors should clarify whether the performance of the compared algorithms is affected by the proposed MAS-based implementation. Does the MAS architecture change the performance of the compared algorithms? If so, why? If not, what is the novelty of the proposed comparison?

Thank you for the comment. We agree with the reviewer’s argument about the performance of the implemented algorithms didn’t improve much as compared with the existing result in the literature. However, as an ML/DL application research work, the main goal of the paper is not to improve the performance of machine/deep learning models. Instead, our focus is on the implementation of our previous work on MANA-NMS architecture which used a MAS model. In the previously published research works, we focused on the theoretical approach for a dockerized network function design as an agent towards network automation which is a direct improvement over the current microservice-based service design in a service-oriented architecture. The novelty of the work is showing the implementation guideline of the classifier and predictor agents, which can also be extended to designing other types of agents to use as a building block for an autonomous system design. The performance measurement parameters chosen are accuracy, latency, and resource utilization of agents for QoS and resource/energy efficiency when the designed agents are deployed in a containerized and distributed cloud/edge environment. 

 2) The authors should clarify how figures 8,9 and 10 were obtained. For example, does figure 8 represent the traffic forecast reported on the y-axis after a training period equal to the value reported on the x-axis? What is the meaning of "Traffic equal to 75"? Why does the traffic curve decrease after 310 days? Is the traffic curve a cumulative value? Authors should explain the meaning and definition of RMSE and MAPE. Authors should try to provide some explanation as to why MLP gives a higher RMSE and MAPE than other techniques, not just by writing what the reader can directly observe from figures 8 and 9.

We appreciate the reviewer for pointing that out. We have worked on elaborating and clarifying the figures. Figure 8 illustrates the performance of the different Deep learning models. Where the x-axis is day and the y-axis is traffic(Terabit per second (Tbps)). In our experiment, we predict traffic over different days with different DL models (GRU, CNN, MLP, and LSTM) and take the average traffic and compared it with real traffic. Figure 8 shows the LSTM, GRU, MLP, and CNN-based NTPA prediction results against the actual network traffic. As shown in the figure, there is no significant difference between their prediction results while MLP predicts the network traffic differently. Overall, LSTM outperforms other models. 

To measure the accuracy of different models we used two methods: RMSE and MAPE. Figure 9 and Figure 10 demonstrate the accuracy of DL models with respect to different dataset sizes. We have calculated the average for one week, one month, three months, six months, and one year for the MSE and MAPE errors and compared different models. 

As shown in the figures, LSTM is the most accurate model to predict network traffic. Additionally, as the training dataset size grows, the error value decreases. In other words, training the NTPA with larger datasets provides higher prediction accuracy. The RMSE error values for the 1-year long dataset as shown in Figure 9, for LSTM, GRU, CNN, and MLP are  0.03, 0.06, 0.2, and 5.7, respectively. Moreover, the MAPE error values for the 1-year long dataset as shown in Figure 10, for LSTM, GRU, CNN, and MLP are 0.24, 0.44, 0.77, and 7.17, respectively. Both figures indicate that LSTM has the most accurate prediction performance.

3) In general, the performance of a prediction algorithm also depends on the time horizon considered for the prediction. In this paper, the authors do not discuss the prediction horizon. Is the prediction horizon set to the next sample?

The prediction works according to the number of past observations (lags). Our dataset is a series of observations based on choosing the lags to predict future behavior. For instance, to predict the traffic flow in period t, we use the previous month's traffic flow t-1 as a feature.

4) How was table 3 obtained? What is the meaning of the values shown in table 3?

Thank you for the comment. We have elaborated on how we generated the results shown in table three that can be reproducible. 

 5) The curves in figure 8 are almost indistinguishable. It is clear only that GRU and MLP give the worst performance. No differences between the other curves can be clearly observed.

Figure 8 illustrates the performance of the different Deep learning models. Where the x-axis is day and the y-axis is traffic(Terabit per second (Tbps)). In our experiment, we predict traffic over different days with different DL models (GRU, CNN, MLP, and LSTM) and take the average traffic and compare it with real traffic. Figure 8 shows the LSTM, GRU, MLP, and CNN-based NTPA prediction results against the actual network traffic. As shown in the figure, there is no significant difference between their prediction results while MLP predicts the network traffic differently. Overall, LSTM outperforms other models.  

6) In my opinion, the authors should provide the code used for the simulation to allow the results to be reproduced.

Thank you for the suggestions. After we finalize and fix the bugs, the code will be uploaded to GitHub.

Reviewer 2 Report

1.     The authors do not highlight the advantages and necessity of the two proposed agents, which is the major contributions.

2.     The authors do not specify the experimental settings of environment, dataset and algorithm parameters.

3.     Theoretically, KNN performs best among the four ML algorithms, how can DT be optimal in both accuracy and delay? A reasonable explanation is needed.

Author Response

Dear  Reviewer,

We highly appreciate you taking the time to review our works and give us feedback. We have modified the paper accordingly and provided responses to each question. Please find below the response. We have also uploaded the modified version of the article highlighting blue the modified portion of the paper. 

  1. The authors do not highlight the advantages and necessity of the two proposed agents, which is the major contributions.

Thank you for this valuable comment. We have provided and highlighted it in the introduction section. In general, the paper is a continuation of the previous work by the authors on autonomous network management based on agents. Still, the benefit and advantage of designing NTPA and NTCA agents are to use these agents as a service unit in the multi-agent system. In the simulation section, an example of the complete MANA-NMS system architectures are provided incorporating traffic classifier agent and traffic predictor their role in the overall system as an example. 

2. The authors do not specify the experimental settings of environment, dataset and algorithm parameters.

Section 4 which discusses the Implementation Scenario and Conceptual Framework presented the experimental settings of environment, dataset, and algorithm parameters.

3. Theoretically, KNN performs best among the four ML algorithms, how can DT be optimal in both accuracy and delay? A reasonable explanation is needed.

Thank you for this interesting result question. This is somehow a particular case due to the dataset type which is network traffic. In general, for large datasets, DT could be faster. Since DT needs to scan the whole dataset to predict as it doesn't generalize the data ahead of time, KNN tends to be slower for large datasets. However, theoretically, for accuracy, KNN may perform better in some cases. However, our dataset is generated for network traffic. Network traffic data is a combination of different data bytes packets under different network protocols. Such kind of dataset has complex time-varying non-linear relationships that are ideally suited for DT-based classifiers. Moreover, KNN has the problem of the “curse of dimensionality” which makes it more appropriate to use in case of a small number of inputs. If the number of variables grows, the KNN algorithm will have a hard time predicting the output of a new data point. Our dataset for classification has a larger dimension even if the final classification output is three classes. 

Reviewer 3 Report

The paper has a good potential for being appreciated and cited, but it requires some improvement.

Highlight the research gap in existing research and in literature. The importance of the proposed integrated approach with respect to the problem statement should have been in focus.

About the literature review. Each paper should clearly specify what is the proposed methodology, novelty and results with experimentation. At the end of related works, highlight in some lines what overall technical gaps are observed in existing works, that led to the design of the proposed approach. To better delineate the context and the different possible solutions, you can consider the following papers as references: "A Multi Agent Approach for the Construction of a Peer-to-Peer Information System in Grids" in Self-Organ. Auton. Inform.(I) and https://ieeexplore.ieee.org/abstract/document/9409962.

The future scope of the methodology should be extended/highlighted. Improve the conclusion, and clarify the conclusion of this article with its significance for follow-up research.

Author Response

Dear  Reviewer,

We highly appreciate you taking the time to review our works and give us feedback. We have modified the paper accordingly and provided responses to each question. Please find below the response. We have also uploaded the modified version of the article highlighting blue in the modified portion of the paper.

The paper has a good potential for being appreciated and cited, but it requires some improvement.

Thank you for the valuable feedback. 

Highlight the research gap in existing research and in literature. The importance of the proposed integrated approach with respect to the problem statement should have been in focus.

Thank you for the suggestions. We modified it accordingly. 

About the literature review. Each paper should clearly specify what is the proposed methodology, novelty, and results with experimentation. 

Thank you for this comment. We have elaborated the existing works from the proposed perspective. 

At the end of related works, highlight in some lines what overall technical gaps are observed in existing works, that led to the design of the proposed approach.

Thank you for this comment. We have elaborated the existing works from the proposed perspective. 

To better delineate the context and the different possible solutions, you can consider the following papers as references: "A Multi Agent Approach for the Construction of a Peer-to-Peer Information System in Grids" in Self-Organ. Auton. Inform.(I) and https://ieeexplore.ieee.org/abstract/document/9409962.

Thank you for the suggestion of this important related work. We included the paper as a comparative approach for multi-agent-based system design.  

The future scope of the methodology should be extended/highlighted. Improve the conclusion, and clarify the conclusion of this article with its significance for follow-up research.

Thank you for the suggestion. We have further elaborated the scope of our approach while indicating future directions. It is highlighted in blue. 

Round 2

Reviewer 1 Report

The revised paper gives sufficient responses to my comments.

As minor revisions,

In Figure 8 I suggest adding the units (Tbps) in the y-label "Traffic"

There is a typo in the added text: MSE instead of RMSE (line 531).  

Reviewer 2 Report

No further comments.