Optimized MLP-CNN Model to Enhance Detecting DDoS Attacks in SDN Environment
Abstract
:1. Introduction
- Introduction of OptMLP-CNN: This research introduces a novel DDoS-attack-detection method referred to as “OptMLP-CNN”. This method combines two critical elements: SHAP-feature selection and a hybrid neural network architecture. The primary goal is to create an optimized and effective DDoS attack detector.
- Optimization Using Bayesian and ADAM Optimizers: We optimized the proposed model by applying two optimization techniques: the Bayesian optimizer and the ADAM optimizer. These optimization methods fine tune the model to enhance its performance and effectiveness in detecting DDoS attacks.
- In-Depth Analysis of DDoS-Attack-Detection Systems: The research extends beyond introducing a new model by conducting a detailed analysis of DDoS-attack-detection systems. This analysis encompasses systems that incorporate DL techniques and were selected based on specific criteria. The criteria include evaluating and comparing their DDoS-detection performance, the datasets used, optimization methods applied, and the types of systems they are designed to protect.
- Evaluation of Effectiveness of Using Public Datasets: This study evaluates the proposed method by applying it to two publicly available datasets. One of these datasets is deliberately constructed to simulate an SDN environment. The results of this evaluation demonstrate that OptMLP-CNN, the proposed method, achieves a high accuracy rate and outperforms other existing methods in detecting DDoS attacks in the context of SDN infrastructure.
- Promising Solution for SDN Security: This research’s overarching contribution is the presentation of a promising solution for enhancing the security of SDN and addressing the growing threat of DDoS attacks. By introducing and optimizing the OptMLP-CNN method, this study aims to enhance the resilience of SDN environments against DDoS attacks, which have significant security implications in contemporary networked environments.
Motivation
2. Related Works
2.1. Comprehensive Overview
2.2. Key Findings
- Network-Specific Innovations: A significant portion of this research body has been intricately tailored to address the intricate nuances of specific network types. Notably, SDN and SD-VANET networks have emerged as focal points of interest. Researchers have recognized the importance of customizing DDoS-detection approaches to suit these specialized network environments. These tailored strategies delve into the unique characteristics and challenges that SDN and SD-VANET networks present, paving the way for more effective ML-based DDoS detection within these domains.
- Feature Selection and Deep Learning’s Prowess in SDN: In the context of SDN environments, a remarkable discovery has centered around the power of feature-selection techniques. These methodologies meticulously sift through data to pinpoint the most pertinent features, substantially elevating the performance of ML-based DDoS-detection systems. Furthermore, DL techniques have come under the microscope, with researchers diligently assessing their contribution to SDN’s overarching DDoS-detection framework. DL models have showcased an unparalleled aptitude for capturing intricate data patterns and relationships, positioning them as valuable assets in the arsenal against DDoS threats.
- Fine Tuning via Hyperparameter Optimization: Research efforts have introduced a meticulous fine-tuning mechanism to optimize the performance of DDoS-detection systems. Hyperparameter-optimization techniques, including Bayesian optimization and GridSearchCV, have played a central role in systematically adjusting the settings of ML models. This systematic calibration process bolsters the accuracy and efficiency of these models, ensuring that they are finely attuned to the intricate nuances of DDoS detection.
- The Rise of Hybrid Deep Learning Models: A prominent revelation in this landscape pertains to the emergence of hybrid DL models. These innovative architectures skillfully merge multiple DL frameworks or combine DL with traditional ML techniques. The overarching objective is to enhance the interpretability and robustness of DDoS-detection systems. By amalgamating diverse models, these hybrid entities provide a wealth of insights and information that guide the final decision making. This synergy augments the accuracy of DDoS detection and crystallizes the findings, making them more comprehensible and actionable.
3. Background
3.1. Feature Selection
3.2. Optimization
- represents the function modeled by the GP.
- is the mean function of the GP.
- is the GP’s covariance function (kernel).
- X is the set of input points.
- y is the observed data.
- is the point for which we want to make predictions.
- is the mean function of the posterior GP.
- is the covariance function (kernel) of the posterior GP.
- is the Expected Improvement acquisition function.
- is the predicted function value for a candidate point .
- is the best observed function value so far.
- returns the maximum of the values inside the parentheses.
3.3. Deep Learning
4. Proposed Model
4.1. MLP
- Input Layer: This is the network’s first layer, receiving the raw input data. Each neuron in the input layer corresponds to a feature or variable in the dataset, making it a fundamental representation of the data’s dimensions.
- Hidden Layers: MLPs can have one or more hidden layers between the input and output layers. These layers are called “hidden” because they are not directly connected to the outside world (i.e., input or output). Neurons in hidden layers take the weighted sum of inputs from the previous layer, apply an activation function, and pass the result to the next layer. The number of hidden layers and the number of neurons in each layer are hyperparameters that can be adjusted to optimize the model’s performance.
- Output Layer: The final layer produces the model’s output or prediction. The structure of this layer depends on the problem the MLP is designed to solve. For example, in binary classification, there may be a single neuron with a sigmoid activation function, while in multiclass classification, there might be multiple neurons, each representing a class and using a softmax activation function.
- Weighted Sum
- is the activation of neuron j in layer l.
- is the weight of the connection between neuron i in layer and neuron j in layer l.
- is the output of neuron i in layer .
- is the bias of neuron j in layer l.
- Activation Function
- Feedforward Propagation
4.2. CNNs
4.2.1. Convolution Layer
4.2.2. Activation Function
4.2.3. Pooling Layer
4.2.4. Fully Connected Layer
4.3. OptMLP-CNN Detector
Algorithm 1 Proposed MLP Architecture |
Require: training data , testing data Require: hyperparameters for MLP architecture
|
Algorithm 2 Proposed CNN Architecture |
Require: training data , testing data Require: hyperparameters for CNN architecture
|
Algorithm 3 Combined MLP-CNN Detector |
Require: training data , testing data Require: hyperparameters for MLP and CNN architectures Require: SHAP-feature selection and Bayesian optimization parameters Require: Adam optimizer hyperparameters
|
5. Experimental Results
5.1. Dataset
5.2. Performance Metrics
5.3. Detection Phase
6. Conclusions and Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
ADAM | Adaptive Moment Estimation |
ANN | Artificial neural network |
ANOVA | ANalysis Of VAriance |
BT | Bagging Tree |
CNN | Convolutional Neural Network |
DA | Discriminant Analysis |
DDoS | Distributed Denial of Service |
DL | Deep Learning |
DNN | Deep neural network |
DT | Decision tree |
EI | Expected Improvement |
FNN | Feedforward neural network |
GB | Gradient boosting |
GLM | Generalized Linear Model |
GNB | Gaussian Naive Bayes |
GP | Gaussian process |
IG | Information Gain |
IoT | Internet of Things |
KNN | K-nearest neighbor |
LR | Logistic regression |
LSTM | Long Short-Term Memory |
ML | Machine learning |
MLP | Multilayer Perceptron |
MLP-GA | Multilayer Perceptron-Genetic Algorithms |
MRMR | Maximum Relevance Minimum Redundancy |
NB | Naive Bayes |
NS | Not Specified |
RBF network | Radial Basis Function |
ReLU | Rectified Linear Unit |
RF | Random Forest |
SDIoT | Software-Defined Internet of Things |
SDN | Software-Defined Networking |
SGD | Stochastic Gradient Descent |
SHAP | SHapley Additive exPlanations |
SVM | Support vector machine |
tanh | Hyperbolic tangent function |
VANET | Vehicular Ad Hoc Networking |
WSN | Wireless Sensor Network |
XGBoost | Extreme gradient boosting |
References
- Ali, T.E.; Chong, Y.W.; Manickam, S. Machine Learning Techniques to Detect a DDoS Attack in SDN: A Systematic Review. Appl. Sci. 2023, 13, 3183. [Google Scholar] [CrossRef]
- Karnani, S.; Agrawal, N.; Kumar, R. A comprehensive survey on low-rate and high-rate DDoS defense approaches in SDN: Taxonomy, research challenges, and opportunities. Multimed. Tools Appl. 2023, 1–54. [Google Scholar] [CrossRef]
- Setitra, M.A.; Benkhaddra, I.; Bensalem, Z.E.A.; Fan, M. Feature Modeling and Dimensionality Reduction to Improve ML-Based DDoS Detection Systems in SDN Environment. In Proceedings of the 2022 19th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), Chengdu, China, 16–18 December 2022; pp. 1–7. [Google Scholar]
- Setitra, M.A.; Fan, M.; Bensalem, Z.E.A. An efficient approach to detect distributed denial of service attacks for software defined internet of things combining autoencoder and extreme gradient boosting with feature selection and hyperparameter tuning optimization. Trans. Emerg. Telecommun. Technol. 2023, 34, e4827. [Google Scholar] [CrossRef]
- Benkhaddra, I.; Kumar, A.; Setitra, M.A.; Bensalem, Z.E.A.; Lei, H. Prevention of DDoS attacks using an optimized deep learning approach in blockchain technology. Trans. Emerg. Telecommun. Technol. 2023, 34, e4729. [Google Scholar]
- Rashid, M.M.; Khan, S.U.; Eusufzai, F.; Redwan, M.A.; Sabuj, S.R.; Elsharief, M. A Federated Learning-Based Approach for Improving Intrusion Detection in Industrial Internet of Things Networks. Network 2023, 3, 158–179. [Google Scholar] [CrossRef]
- Fox, G.; Boppana, R.V. Detection of Malicious Network Flows with Low Preprocessing Overhead. Network 2022, 2, 628–642. [Google Scholar] [CrossRef]
- Shieh, C.S.; Nguyen, T.T.; Horng, M.F. Detection of Unknown DDoS Attack Using Convolutional Neural Networks Featuring Geometrical Metric. Mathematics 2023, 11, 2145. [Google Scholar] [CrossRef]
- Thakkar, A.; Lohiya, R. Fusion of statistical importance for feature selection in Deep Neural Network-based Intrusion Detection System. Inf. Fusion 2023, 90, 353–363. [Google Scholar] [CrossRef]
- Saha, S.; Priyoti, A.T.; Sharma, A.; Haque, A. Towards an Optimized Ensemble Feature Selection for DDoS Detection Using Both Supervised and Unsupervised Method. Sensors 2022, 22, 9144. [Google Scholar] [CrossRef]
- Türkoğlu, M.; Polat, H.; Koçak, C.; Polat, O. Recognition of DDoS Attacks on SD-VANET Based on Combination of Hyperparameter Optimization and Feature Selection. Expert Syst. Appl. 2022, 203, 117500. [Google Scholar] [CrossRef]
- Habib, B.; Khursheed, F. Performance evaluation of machine learning models for distributed denial of service attack detection using improved feature selection and hyper-parameter optimization techniques. Concurr. Comput. Pract. Exp. 2022, 34, e7299. [Google Scholar] [CrossRef]
- Batchu, R.K.; Seetha, H. On Improving the Performance of DDoS attack detection system. Microprocess. Microsyst. 2022, 93, 104571. [Google Scholar] [CrossRef]
- Wang, S.; Balarezo, J.F.; Chavez, K.G.; Al-Hourani, A.; Kandeepan, S.; Asghar, M.R.; Russello, G. Detecting flooding DDoS attacks in software defined networks using supervised learning techniques. Eng. Sci. Technol. Int. J. 2022, 35, 101176. [Google Scholar] [CrossRef]
- Batchu, R.K.; Seetha, H. An integrated approach explaining the detection of distributed denial of service attacks. Comput. Netw. 2022, 216, 109269. [Google Scholar] [CrossRef]
- Chanu, U.S.; Singh, K.J.; Chanu, Y.J. An ensemble method for feature selection and an integrated approach for mitigation of distributed denial of service attacks. Concurr. Comput. Pract. Exp. 2022, 34, e6919. [Google Scholar] [CrossRef]
- Kshirsagar, D.; Kumar, S. A feature reduction based reflected and exploited DDoS attacks detection system. J. Ambient. Intell. Humaniz. Comput. 2022, 1-13, 393–405. [Google Scholar] [CrossRef]
- El Sayed, M.S.; Le-Khac, N.A.; Azer, M.A.; Jurcut, A.D. A Flow-Based Anomaly Detection Approach With Feature Selection Method Against DDoS Attacks in SDNs. IEEE Trans. Cogn. Commun. Netw. 2022, 8, 1862–1880. [Google Scholar] [CrossRef]
- Akgun, D.; Hizal, S.; Cavusoglu, U. A new DDoS attacks intrusion detection model based on deep learning for cybersecurity. Comput. Secur. 2022, 118, 102748. [Google Scholar] [CrossRef]
- Zhou, L.; Zhu, Y.; Zong, T.; Xiang, Y. A feature selection-based method for DDoS attack flow classification. Future Gener. Comput. Syst. 2022, 132, 67–79. [Google Scholar] [CrossRef]
- Saha, S.; Priyoti, A.T.; Sharma, A.; Haque, A. Towards an Optimal Feature Selection Method for AI-Based DDoS Detection System. In Proceedings of the 2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA, 8–11 January 2022; pp. 425–428. [Google Scholar]
- Fenil, E.; Kumar, P.M. Towards a secure Software Defined Network with Adaptive Mitigation of DDoS attacks by Machine Learning Approaches. In Proceedings of the 2022 IEEE International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI), Chennai, India, 28–29 January 2022; pp. 1–13. [Google Scholar]
- Golchin, P.; Kundel, R.; Steuer, T.; Hark, R.; Steinmetz, R. Improving DDoS Attack Detection Leveraging a Multi-aspect Ensemble Feature Selection. In Proceedings of the NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium, Budapest, Hungary, 25–29 April 2022; pp. 1–5. [Google Scholar]
- Batchu, R.K.; Seetha, H. A generalized machine learning model for DDoS attacks detection using hybrid feature selection and hyperparameter tuning. Comput. Netw. 2022, 200, 108498. [Google Scholar] [CrossRef]
- Bindra, N.; Sood, M. Evaluating the impact of feature selection methods on the performance of the machine learning models in detecting DDoS attacks. Rom. J. Inf. Sci. Technol. 2020, 23, 250–261. [Google Scholar]
- Polat, H.; Polat, O.; Cetin, A. Detecting DDoS attacks in software-defined networks through feature selection methods and machine learning models. Sustainability 2020, 12, 1035. [Google Scholar] [CrossRef]
- Zaki, F.A.M.; Chin, T.S. FWFS: Selecting robust features towards reliable and stable traffic classifier in SDN. IEEE Access 2019, 7, 166011–166020. [Google Scholar] [CrossRef]
- Cauteruccio, F.; Fortino, G.; Guerrieri, A.; Liotta, A.; Mocanu, D.C.; Perra, C.; Terracina, G.; Vega, M.T. Short-long term anomaly detection in wireless sensor networks based on machine learning and multi-parameterized edit distance. Inf. Fusion 2019, 52, 13–30. [Google Scholar] [CrossRef]
- Setitra, I.; Iwahori, Y.; Meziane, A. Walking cycle and walking phases extraction from videos using transfer learning. Procedia Comput. Sci. 2020, 176, 2695–2704. [Google Scholar] [CrossRef]
- González-Nóvoa, J.A.; Busto, L.; Campanioni, S.; Fariña, J.; Rodríguez-Andina, J.J.; Vila, D.; Veiga, C. Two-step approach for occupancy estimation in intensive care units based on Bayesian optimization techniques. Sensors 2023, 23, 1162. [Google Scholar] [CrossRef]
- Hassan, E.; Shams, M.Y.; Hikal, N.A.; Elmougy, S. The effect of choosing optimizer algorithms to improve computer vision tasks: A comparative study. Multimed. Tools Appl. 2023, 82, 16591–16633. [Google Scholar] [CrossRef] [PubMed]
- Taud, H.; Mas, J.F. Multilayer perceptron (MLP). In Geomatic Approaches for Modeling Land Change Scenarios; Springer: Cham, Switzerland, 2018; pp. 451–455. [Google Scholar]
- Desai, M.; Shah, M. An anatomization on breast cancer detection and diagnosis employing multi-layer perceptron neural network (MLP) and Convolutional neural network (CNN). Clin. eHealth 2021, 4, 1–11. [Google Scholar] [CrossRef]
- Li, Z.; Liu, F.; Yang, W.; Peng, S.; Zhou, J. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Trans. Neural Netw. Learn. Syst. 2021, 33, 6999–7019. [Google Scholar] [CrossRef]
- Benkhaddra, I.; Kumar, A.; Setitra, M.A.; Hang, L. Design and Development of Consensus Activation Function Enabled Neural Network-Based Smart Healthcare Using BIoT. Wirel. Pers. Commun. 2023, 130, 1549–1574. [Google Scholar] [CrossRef]
- Elsayed, M.S.; Le-Khac, N.A.; Jurcut, A.D. InSDN: A novel SDN intrusion dataset. IEEE Access 2020, 8, 165263–165284. [Google Scholar] [CrossRef]
- Sharafaldin, I.; Lashkari, A.H.; Hakak, S.; Ghorbani, A.A. Developing realistic distributed denial of service (DDoS) attack dataset and taxonomy. In Proceedings of the 2019 International Carnahan Conference on Security Technology (ICCST), Chennai, India, 1–3 October 2019; pp. 1–8. [Google Scholar]
Ref | Year | Technique | Optimizer | Feature Selection | Dataset | Network |
---|---|---|---|---|---|---|
Thakkar, A. and Lohiya, R. [9] | 2023 | DNN | NS | Fusion of statistical importance using standard deviation | NSL-KDD, UNSW_NB-15, CIC-IDS-2017 | NS |
Setitra, M. A. et al. [4] | 2023 | Autoencoder and XGBoost | Hyperparameter tuning | SHAP | SDNIoT, SDN-NF-TJ | SDN-IoT |
Saha, S. et al. [10] | 2022 | Seven ML, four DL, five unsupervised | NS | 15 individual methods and one ensemble method | UNSW_NB-15 | NS |
Türkoğlu, M. et al. [11] | 2022 | Bayesian optimize-DT | Bayesian optimization | MRMR | Self-generated | SD-VANET |
Habib, B. and Khursheed, F. [12] | 2022 | LR, DT, SVM, KNN, GNB, RF, XGBoost, ANN, CNN | GridSearchCV and random sampling | Chi2, IG, merged Chi2-IG ranking ML classifiers; that is, DT, RF and XGBoost | KDD Cup 99, UNSW_NB-15 | NS |
Batchu, R. K. and Seetha, H. [13] | 2022 | Extreme ML | Memory optimization | Hybrid | CICDDoS-2019 | NS |
Wang, S. et al. [14] | 2022 | SVM, NB, FNN, KNN, GLM, DT, BT | DA, BT | Single feature based on counter within a time period | 1999 DARPA, DASD, InSDN, Self-generated | SDN |
Batchu, R. K. and Seetha, H. [15] | 2022 | KNORA-E, KNORA-U | Hyperparameter tuning | SHAP | CICDDoS-2019 | NS |
Chanu, U. S. et al. [16] | 2022 | MLP-GA, MLP, NB, RBF network | NS | IG, Gain ratio, Chi2, ReliefF, ensemble method | NSL-KDD | NS |
Kshirsagar, D. and Kumar, S. [17] | 2022 | J48 classifier | NS | Combination of IG and correlation (CR) | CICDDoS-2019, KDD Cup 1999 | NS |
El Sayed, M. S. et al. [18] | 2022 | Autoencoder and LSTM | NS | IG, RF | InSDN, CICIDS2017, CICIDS2018 | SDN |
Akgun, D. et al. [19] | 2022 | DNN, CNN, LSTM | NS | IG | CICDDoS-2019 | NS |
Zhou, L. et al. [20] | 2022 | SVM, Guard, KNN | Threshold tuning | A proposed feature-selection method | IMPACT CAIDA | IoT |
Saha, S. et al. [21] | 2022 | Majority Voting | NS | One from filter, wrappers, and embedded methods | UNSW_NB-15 | NS |
Fenil, E. and Kumar, P. M. [22] | 2022 | Six ML | NS | ANOVA | Self-generated | SDN |
Golchin, P. et al. [23] | 2022 | RF, LR, SVM, NB, MLP | NS | A proposed multiaspect ensemble feature selection | CICDDoS-2019, InSDN | SDN |
Setitra, M. A. et al. [3] | 2022 | Eight ML | NS | Adaptation and dimensionality reduction | SDN dataset | SDN |
Batchu, R. K. and Seetha, H. [24] | 2021 | LR, GB, DT, KNN, SVM | Hyperparameter tuning | Hybrid based on Spearman’s correlation and RF | CICDDoS-2019 | NS |
Bindra, N. and Sood, M. [25] | 2020 | LR, KNN, GNB, RF, SVM | NS | One from filter, wrappers, and embedded methods | CICIDS2017 | NS |
Polat, H., et al. [26] | 2020 | SVM, NB, ANN, KNN | NS | One from filter, wrappers, and embedded methods | Self-generated | SDN |
Zaki, F. A. M. and Chin, T. S. [27] | 2019 | C4.5 | NS | Hybrid based on filter and wrapper methods | Self-generated | SDN |
Cauteruccio, F. et al. [28] | 2019 | Combining edge-based method with a cloud-based one | Multiparameter edit distance | Self-selected based on WSN behavior | Self-generated | WSN |
Dataset | Accuracy | Precision | Recall | F1-Score | AUC |
---|---|---|---|---|---|
CICDDoS-2019 | 0.999504 | 0.999009 | 0.999752 | 0.999381 | 0.997901 |
InSDN | 0.999802 | 0.999858 | 0.999717 | 0.999787 | 0.999601 |
Ref | Used Dataset | Accuracy | Precision | Recall | F1-Score | AUC | Approach |
---|---|---|---|---|---|---|---|
Thakkar, A. and Lohiya, R. [9] | NSL-KDD | 99.84% | 99.94% | 98.81% | 99.37% | NS | Deep neural network with fusion of statistical importance by using standard deviation |
UNSW_NB-15 | 89.03% | 95.00% | 98.95% | 96.93% | NS | ||
CIC-IDS-2017 | 99.80% | 99.85% | 99.94% | 99.89% | NS | ||
Batchu, R. K. and Seetha, H. [13] | CICDDoS-2019 | 99.94% | 99.88% | 99.99% | 99.94% | 99.94% | Extreme ML with hybrid feature selection |
Batchu, R. K. and Seetha, H. [24] | CICDDoS-2019 | 99.97% | 98.90% | 99.99% | 99.44% | 99.97% | Gradient boosting with hybrid feature selection based on Spearman’s correlation and RF |
Proposed Model | CICDDoS-2019 | 99.95% | 99.90% | 99.98% | 99.94% | 99.79% | Optimized MLP-CNN with SHAP-feature selection |
InSDN | 99.98% | 99.99% | 99.97% | 99.98% | 99.96% |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Setitra, M.A.; Fan, M.; Agbley, B.L.Y.; Bensalem, Z.E.A. Optimized MLP-CNN Model to Enhance Detecting DDoS Attacks in SDN Environment. Network 2023, 3, 538-562. https://doi.org/10.3390/network3040024
Setitra MA, Fan M, Agbley BLY, Bensalem ZEA. Optimized MLP-CNN Model to Enhance Detecting DDoS Attacks in SDN Environment. Network. 2023; 3(4):538-562. https://doi.org/10.3390/network3040024
Chicago/Turabian StyleSetitra, Mohamed Ali, Mingyu Fan, Bless Lord Y. Agbley, and Zine El Abidine Bensalem. 2023. "Optimized MLP-CNN Model to Enhance Detecting DDoS Attacks in SDN Environment" Network 3, no. 4: 538-562. https://doi.org/10.3390/network3040024
APA StyleSetitra, M. A., Fan, M., Agbley, B. L. Y., & Bensalem, Z. E. A. (2023). Optimized MLP-CNN Model to Enhance Detecting DDoS Attacks in SDN Environment. Network, 3(4), 538-562. https://doi.org/10.3390/network3040024