# Time Series Forecasting of Software Vulnerabilities Using Statistical and Deep Learning Models

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- An in-depth examination of DL models as a method for forecasting the number of vulnerabilities that a software project will contain in the future.
- A comprehensive comparison between DL and statistical models in vulnerability forecasting.
- A methodology for forecasting the number of vulnerabilities related to an individual future timestep, instead of predicting the cumulative number of vulnerabilities until that timestep.

## 2. Related Work

**Regarding the code-based models**, Automated Static Analysis (ASA) is often used for the early identification of security issues in the source code of software projects [19,20]. ASA manages to identify potential security threats that reside in the source code, by applying some pre-defined rules and identifying the violations of these rules. Based on ASA, more advanced models have been also proposed [21]. Siavvas et al. proposed a model that combines low-level indicators (i.e., security-relevant static analysis alerts and software metrics) in order to generate a high-level security score that reflects the internal security level of the examined software [21].

**Regarding the time series-based models**, Alhazmi et al. proposed a time-based model [11]. Their approach is based on the fact that interest in newly released software rises in the beginning, peaks after a while, and then drops as new competitive versions are introduced. Yasasin et al. examined the issue of estimating the quantity of software security flaws in operating systems, browsers, and office applications [18]. They retrieved their data from NVD and they used mainly statistical models such as ARIMA and exponential smoothing. They also investigated the suitability of the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE) in the measurement of vulnerability forecasting. Furthermore, Jabeen et al., conducted an empirical analysis, where they compared different statistical algorithms with ML techniques showing that many of the ML models provide better results [14]. Shukla et al. in their study [26] attempted to include in their model the so called change point in order to assimilate the information about the time point when changes in the code size, popularity or known vulnerabilities can cause changes in the time series distribution. Wang et al. attempted to include the effort factor (i.e., any factor that can depict environmental changes of the software) in their cumulative vulnerabilities discovery model in order to predict trends in the vulnerability time series data [27]. In Table 2, there is a list of the categories of the time series-based models.

## 3. Methodology and Experimental Setup

#### 3.1. Data Collection

#### 3.2. Time Series Modelling

#### 3.2.1. Statistical Models

#### 3.2.2. Deep Learning Models

#### 3.2.3. Accuracy Metrics

#### 3.3. Fitting Time Series Models

#### 3.3.1. Statistical Models

#### 3.3.2. Deep Learning Models

**Data Transformation**

**Models Training**

**optimizer:**The algorithm that is responsible to adjust model weights in order to maximize the loss function.**batch size:**The size of batches, (i.e., mini-batches) given in parallel to the network.**number of hidden layers:**The number of hidden layers that make up the network.**number of neurons:**The number of nodes that make up each hidden layer.**activation function:**The function that converts the input signal of an ANN node into an output signal that will be used as an input signal at the next layer.**number of training epochs:**The number of times when the whole training set “goes through” the training process.**convolutional filter:**The number and the size of the convolutional filters in the network (only for CNN).

#### 3.3.3. Complexity Analysis

- For MLP: W = nH + HK
- For RNN: $W=nH+{H}^{2}+HK$
- For LSTM: $W=4nH+4{H}^{2}+3H+HK$
- For BiLSTM: $W=2(4nH+4{H}^{2}+3H+HK)$

## 4. Results and Discussion

**goodness-of-fit level**, we notice that in each covered dataset (i.e., software project) there is always a statistical model providing higher R${}^{2}$ and lower MAE and RMSE scores than the DL approaches. While in most of the cases the R${}^{2}$ cannot be considered high, possibly creating doubts about how well the models fit the data, both MAE and RMSE are low enough (at least for the cases of Internet Explorer, Ubuntu Linux and Microsoft Office) showing that the models’ predictions are really close to the real values. To provide a visual inspection of the models’ fit capabilities, Figure 3 shows the ARIMA model (in red colour) fitted to the Google Chrome vulnerability dataset (in blue colour). Based on Table 9, ARIMA demonstrated the best fitting performance as regards this particular dataset. As can be seen by inspecting the plot, the ARIMA model has managed to learn the peculiarities (e.g., level, trend) of Google Chrome’s vulnerability evolution patterns to a quite satisfactory extent, with the only exception being a couple of random spikes, where the number of reported vulnerabilities was unusually high. However, it should be noted that, although the model cannot accurately estimate the exact value of the spikes, the predicted values are higher than the mean value of the predictions, meaning that it can indeed capture this trend, i.e., an expected sudden rise in the number of vulnerabilities. This is important as the purpose of vulnerability forecasting models is to facilitate decision making during the development of software products, by detecting potential future trends in the number of reported vulnerabilities.

**predictive power**on unseen data, by inspecting the results presented on Table 9, we can argue that as far as the cases of Internet Explorer, Ubuntu Linux and Microsoft Office are concerned, MAE and RMSE values indicate that both the statistical and DL models are quite efficient in producing 24 steps ahead forecasts. On the other hand, in the cases of Google Chrome and Apple MacOS the models provide forecasts that are quite far from the “ground truth” (i.e., the real values). To provide a visual inspection of the models’ predictive capabilities, Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 show the forecasted values (in red colour) of the last 24 months for each of the five examined software projects (ground truth in blue colour), generated by the best-performing model in each particular case. As can be seen by inspecting these plots, in most cases the models have managed to learn the peculiarities (e.g., levels, trends) of the projects’ vulnerability evolution patterns to a quite satisfactory extent, with an exception in the Apple macOS case where they are struggling to follow the random spikes that reflect unusual high numbers of reported vulnerabilities.

- The statistical and the DL models had a similar performance regarding the forecasting of the number of vulnerabilities in software systems.
- The statistical models achieved a better goodness-of-fit level (i.e., on the training dataset), whereas the DL models had a slight (but not statistically significant) superiority in terms of predictive power on unseen data.
- The model selection process depends on the specific project under examination.
- The vulnerability forecasting task is significantly affected by the nature of the data. When there were unusual spikes and many zero values in data, all the examined models had difficulty in predicting the number of vulnerabilities.

## 5. Conclusions and Future Work

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Shin, Y.; Williams, L. Is complexity really the enemy of software security? In Proceedings of the 4th ACM Workshop on Quality of Protection, Alexandria, VA, USA, 27–31 October 2008; pp. 47–50. [Google Scholar]
- Shin, Y.; Williams, L. An empirical model to predict security vulnerabilities using code complexity metrics. In Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, Kaiserslautern, Germany, 9–10 October 2008; pp. 315–317. [Google Scholar]
- Chowdhury, I.; Zulkernine, M. Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities. J. Syst. Archit.
**2011**, 57, 294–313. [Google Scholar] [CrossRef] - Pang, Y.; Xue, X.; Wang, H. Predicting vulnerable software components through deep neural network. In Proceedings of the 2017 International Conference on Deep Learning Technologies, Chengdu, China, 2–4 June 2017; pp. 6–10. [Google Scholar]
- Li, Z.; Zou, D.; Xu, S.; Ou, X.; Jin, H.; Wang, S.; Deng, Z.; Zhong, Y. Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv
**2018**, arXiv:1801.01681. [Google Scholar] - Neuhaus, S.; Zimmermann, T.; Holler, C.; Zeller, A. Predicting vulnerable software components. In Proceedings of the 14th ACM Conference on Computer and Communications Security, Alexandria, VA, USA, 2 November–31 October 2007; pp. 529–540. [Google Scholar]
- Hovsepyan, A.; Scandariato, R.; Joosen, W.; Walden, J. Software vulnerability prediction using text analysis techniques. In Proceedings of the 4th International Workshop on Security Measurements and Metrics, Lund, Sweden, 21 September 2012; pp. 7–10. [Google Scholar]
- Iqbal, J.; Firdous, T.; Shrivastava, A.K.; Saraf, I. Modelling and predicting software vulnerabilities using a sigmoid function. Int. J. Inf. Technol.
**2022**, 14, 649–655. [Google Scholar] [CrossRef] - Shrivastava, A.; Sharma, R.; Kapur, P. Vulnerability discovery model for a software system using stochastic differential equation. In Proceedings of the 2015 International Conference on Futuristic Trends on Computational Analysis and Knowledge Management (ABLAZE), Greater Noida, India, 25–27 February 2015; pp. 199–205. [Google Scholar]
- National Vulnerability Database. Available online: https://nvd.nist.gov (accessed on 30 July 2022).
- Alhazmi, O.H.; Malaiya, Y.K. Quantitative vulnerability assessment of systems software. In Proceedings of the Annual Reliability and Maintainability Symposium, Alexandria, VA, USA, 24–27 January 2005; pp. 615–620. [Google Scholar]
- Leverett, É.; Rhode, M.; Wedgbury, A. Vulnerability Forecasting: Theory and practice. Digit. Threat. Res. Pract.
**2022**. [Google Scholar] [CrossRef] - Roumani, Y.; Nwankpa, J.K.; Roumani, Y.F. Time series modeling of vulnerabilities. Comput. Secur.
**2015**, 51, 32–40. [Google Scholar] [CrossRef] - Jabeen, G.; Rahim, S.; Afzal, W.; Khan, D.; Khan, A.A.; Hussain, Z.; Bibi, T. Machine learning techniques for software vulnerability prediction: A comparative study. Appl. Intell.
**2022**, 1–22. [Google Scholar] - Karasu, S.; Altan, A. Crude oil time series prediction model based on LSTM network with chaotic Henry gas solubility optimization. Energy
**2022**, 242, 122964. [Google Scholar] [CrossRef] - Altan, A.; Karasu, S.; Bekiros, S. Digital currency forecasting with chaotic meta-heuristic bio-inspired signal processing techniques. Chaos Solitons Fractals
**2019**, 126, 325–336. [Google Scholar] [CrossRef] - Gencer, K.; Başçiftçi, F. Time series forecast modeling of vulnerabilities in the android operating system using ARIMA and deep learning methods. Sustain. Comput. Inform. Syst.
**2021**, 30, 100515. [Google Scholar] [CrossRef] - Yasasin, E.; Prester, J.; Wagner, G.; Schryen, G. Forecasting IT security vulnerabilities—An empirical analysis. Comput. Secur.
**2020**, 88, 101610. [Google Scholar] [CrossRef] - Zheng, J.; Williams, L.; Nagappan, N.; Snipes, W.; Hudepohl, J.P.; Vouk, M.A. On the value of static analysis for fault detection in software. IEEE Trans. Softw. Eng.
**2006**, 32, 240–253. [Google Scholar] [CrossRef] - Gegick, M.; Williams, L. Toward the use of automated static analysis alerts for early identification of vulnerability-and attack-prone components. In Proceedings of the Second International Conference on Internet Monitoring and Protection (ICIMP 2007), San Jose, CA, USA, 1–5 July 2007; p. 18. [Google Scholar]
- Siavvas, M.; Kehagias, D.; Tzovaras, D.; Gelenbe, E. A hierarchical model for quantifying software security based on static analysis alerts and software metrics. Softw. Qual. J.
**2021**, 29, 431–507. [Google Scholar] [CrossRef] - Kalouptsoglou, I.; Siavvas, M.; Tsoukalas, D.; Kehagias, D. Cross-project vulnerability prediction based on software metrics and deep learning. In Proceedings of the International Conference on Computational Science and Its Applications, Cagliari, Italy, 1–4 July 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 877–893. [Google Scholar]
- Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. arXiv
**2013**, arXiv:1301.3781. [Google Scholar] - Kalouptsoglou, I.; Siavvas, M.; Kehagias, D.; Chatzigeorgiou, A.; Ampatzoglou, A. An empirical evaluation of the usefulness of word embedding techniques in deep learning-based vulnerability prediction. In Proceedings of the Security in Computer and Information Sciences: Second International Symposium, EuroCybersec 2021, Nice, France, 25–26 October 2021; Revised Selected Papers. Springer Nature: Berlin/Heidelberg, Germany, 2022; p. 23. [Google Scholar]
- Kalouptsoglou, I.; Siavvas, M.; Kehagias, D.; Chatzigeorgiou, A.; Ampatzoglou, A. Examining the Capacity of Text Mining and Software Metrics in Vulnerability Prediction. Entropy
**2022**, 24, 651. [Google Scholar] [CrossRef] [PubMed] - Shukla, A.; Katt, B. Change Point Problem in Security Vulnerability Discovery Model. In Proceedings of the 2019 International Conference on Software Security and Assurance (ICSSA), St. Pölten, Austria, 25–26 July 2019; pp. 21–26. [Google Scholar]
- Wang, X.; Ma, R.; Li, B.; Tian, D.; Wang, X. E-WBM: An effort-based vulnerability discovery model. IEEE Access
**2019**, 7, 44276–44292. [Google Scholar] [CrossRef] - Yazdi, H.S.; Mirbolouki, M.; Pietsch, P.; Kehrer, T.; Kelter, U. Analysis and prediction of design model evolution using time series. In Proceedings of the International Conference on Advanced Information Systems Engineering, Thessaloniki, Greece, 16–20 June 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 1–15. [Google Scholar]
- Goulão, M.; Fonte, N.; Wermelinger, M.; e Abreu, F.B. Software evolution prediction using seasonal time analysis: A comparative study. In Proceedings of the 2012 16th European Conference on Software Maintenance and Reengineering, Szeged, Hungary, 27–30 March 2012; pp. 213–222. [Google Scholar]
- Raja, U.; Hale, D.P.; Hale, J.E. Modeling software evolution defects: A time series approach. J. Softw. Maint. Evol. Res. Pract.
**2009**, 21, 49–71. [Google Scholar] [CrossRef] - Tsoukalas, D.; Jankovic, M.; Siavvas, M.; Kehagias, D.; Chatzigeorgiou, A.; Tzovaras, D. On the Applicability of Time Series Models for Technical Debt Forecasting. In Proceedings of the 15th China-Europe International Symposium on Software Engineering Education (CEISEE 2019), Lisbon-Caparica, Portugal, 30–31 May 2019. in press. [Google Scholar] [CrossRef]
- Tsoukalas, D.; Kehagias, D.; Siavvas, M.; Chatzigeorgiou, A. Technical Debt Forecasting: An empirical study on open-source repositories. J. Syst. Softw.
**2020**, 170, 110777. [Google Scholar] [CrossRef] - Mathioudaki, M.; Tsoukalas, D.; Siavvas, M.; Kehagias, D. Technical Debt Forecasting Based on Deep Learning Techniques. In Proceedings of the International Conference on Computational Science and Its Applications, Cagliari, Italy, 13–16 September 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 306–322. [Google Scholar]
- Box, G.E.; Jenkins, G.M.; Reinsel, G.C.; Ljung, G.M. Time Series Analysis: Forecasting and Control; John Wiley & Sons: Hoboken, NJ, USA, 2015. [Google Scholar]
- Croston, J.D. Forecasting and stock control for intermittent demands. J. Oper. Res. Soc.
**1972**, 23, 289–303. [Google Scholar] [CrossRef] - Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowl.-Based Syst.
**1998**, 6, 107–116. [Google Scholar] [CrossRef] - Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] - Chung, J.; Gulcehre, C.; Cho, K.; Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv
**2014**, arXiv:1412.3555. [Google Scholar] - Schuster, M.; Paliwal, K.K. Bidirectional recurrent neural networks. IEEE Trans. Signal Process.
**1997**, 45, 2673–2681. [Google Scholar] [CrossRef] - LeCun, Y.; Haffner, P.; Bottou, L.; Bengio, Y. Object recognition with gradient-based learning. In Shape, Contour and Grouping in Computer Vision; Springer: Berlin/Heidelberg, Germany, 1999; pp. 319–345. [Google Scholar]
- Hyndman, R.J.; Koehler, A.B. Another look at measures of forecast accuracy. Int. J. Forecast.
**2006**, 22, 679–688. [Google Scholar] [CrossRef] [Green Version] - Kim, S.; Kim, H. A new metric of absolute percentage error for intermittent demand forecasts. Int. J. Forecast.
**2016**, 32, 669–679. [Google Scholar] [CrossRef] - Seabold, S.; Perktold, J. Statsmodels: Econometric and statistical modeling with python. In Proceedings of the 9th Python in Science Conference, Austin, TX, USA, 28 June–3 July 2010. [Google Scholar]
- Dickey, D.A.; Fuller, W.A. Distribution of the estimators for autoregressive time series with a unit root. J. Am. Stat. Assoc.
**1979**, 74, 427–431. [Google Scholar] - Pmdarima: ARIMA Estimators for Python. Available online: https://alkaline-ml.com/pmdarima/index.html (accessed on 30 July 2022).
- A Python Package to Forecast Intermittent Time Series Using Croston’s Method. Available online: htts://pypi.org/project/croston/ (accessed on 30 July 2022).
- A Python Package that Transforms Features by Scaling Each Feature to a Given Range. Available online: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html (accessed on 30 July 2022).
- Scikit-learn: Machine Learning in Python. Available online: htts://scikit-learn.org/stable/ (accessed on 30 July 2022).
- An End-to-End Open Source Machine Learning Platform. Available online: htts://www.tensorflow.org/ (accessed on 30 July 2022).
- Keras API Models. Available online: htts://keras.io/api/models/ (accessed on 30 July 2022).
- Ruder, S. An overview of gradient descent optimization algorithms. arXiv
**2016**, arXiv:1609.04747. [Google Scholar] - Ding, B.; Qian, H.; Zhou, J. Activation functions and their characteristics in deep neural networks. In Proceedings of the 2018 Chinese Control and Decision Conference (CCDC), Shenyang, China, 9–11 June 2018; pp. 1836–1841. [Google Scholar]
- Early Stopping Technique Provided by Keras. Available online: htts://keras.io/api/callbacks/early_stopping/ (accessed on 30 July 2022).
- Big O Notation. Available online: htts://en.wikipedia.org/wiki/Big_O_notation (accessed on 30 July 2022).
- Tran, Q.T.; Hao, L.; Trinh, Q.K. A comprehensive research on exponential smoothing methods in modeling and forecasting cellular traffic. Concurr. Comput. Pract. Exp.
**2020**, 32, e5602. [Google Scholar] [CrossRef] - Teunter, R.; Sani, B. On the bias of Croston’s forecasting method. Eur. J. Oper. Res.
**2009**, 194, 177–183. [Google Scholar] [CrossRef] - Cuda ToolKit. Available online: htts://developer.nvidia.com/cuda-toolkit (accessed on 30 July 2022).
- Wilcoxon, F. Individual comparisons by ranking methods. Biom. Bull.
**1945**, 1, 80–83. [Google Scholar] [CrossRef]

**Figure 2.**Vulnerability evolution of the five selected projects. (

**a**) Google Chrome; (

**b**) Internet Explorer; (

**c**) Apple macOS X; (

**d**) Ubuntu Linux; (

**e**) Microsoft Office.

**Figure 9.**Comparison of the best statistical and deep learning models per project in terms of Mean Absolute Error.

**Figure 10.**Comparison of the best statistical and deep learning models per project in terms of Root Mean Square Error.

Code-Based Model | Characteristics | Referenced Papers |
---|---|---|

Static analysis-based | Pattern-matching method. It checks violations of pre-defined rules. | [19,20,21] |

Software metrics-based | ML algorithms that utilize software attributes as input. | [1,2,22,25] |

Text mining-based | ML algorithms that process textual tokens that reside in the source code instead of numerical metrics. | [5,6,24,25] |

Time Series-Based Model | Characteristics | Referenced Papers |
---|---|---|

Statistical models | Statistical models such as ARIMA and exponential smoothing, which predict the future number of vulnerabilities over a time period. | [8,9,12,13,14,17,18] |

ML models | ML models that attempt to learn the number of vulnerabilities evolution through time, based on previous knowledge. | [12,14] |

DL models | DL models (i.e., deep neural networks, recurrent neural networks, etc.) that attempt to learn the number of vulnerabilities evolution through time, based on previous knowledge. | [12,17] |

Alhazmi–Malaiya model | A time-series model that attempts to predict the cumulative number of vulnerabilities until a specific time step. | [11] |

Software Project | Domain | Release Date | Open Source | Data Collection Period | Total Vulnerabilities |
---|---|---|---|---|---|

Google Chrome | Browser | 2008 | Partially | 2008–2021 | 2136 |

Internet Explorer | Browser | 1995 | No | 1997–2018 | 1039 |

Apple macOS X | OS | 2001 | No | 2001–2021 | 2175 |

Ubuntu Linux | OS | 2004 | Yes | 2005–2021 | 361 |

Microsoft Office | Office | 1990 | No | 1999–2021 | 347 |

Software Project | a (Level) | $\mathit{\beta}$ (Trend) | $\mathit{\gamma}$ (Seasonal) |
---|---|---|---|

Google Chrome | 0.129 | 0.000 | 0.000 |

Internet Explorer | 0.179 | 0.000 | 0.256 |

Apple macOS X | 1.922 × 10${}^{-8}$ | 6.891 × 10${}^{-13}$ | 4.383 × 10${}^{-9}$ |

Ubuntu Linux | 0.247 | 0.000 | 0.000 |

Microsoft Office | 0.052 | 7.362 × 10${}^{-15}$ | 2.256 × 10${}^{-12}$ |

Software Project | Test Statistic | p-Value | Critical Value (1%) |
---|---|---|---|

Google Chrome | −6.267 | 4.087 × 10${}^{-8}$ | −3.475 |

Internet Explorer | −4.762 | 0.0098 | −3.460 |

Apple macOS X | −7.021 | 6.550 × 10${}^{-10}$ | −3.459 |

Ubuntu Linux | −6.897 | 1.310 × 10${}^{-9}$ | −3.468 |

Microsoft Office | −7.529 | 3.604 × 10${}^{-11}$ | −3.457 |

Software Project | Order (AR, I, MA) | Seasonal Order S (AR, I, MA, m) |
---|---|---|

Google Chrome | (5, 1, 3) | (1, 0, 0, 12) |

Internet Explorer | (2, 1, 4) | (1, 0, 1, 12) |

Apple macOS X | (2, 1, 5) | (1, 0, 1, 12) |

Ubuntu Linux | (1, 1, 2) | (1, 0, 0, 12) |

Microsoft Office | (2, 1, 2) | - |

**Table 7.**The sliding window data for multi-output prediction (X stands for inputs, Y for outputs and m for the monthly observations.)

X_{1} | X_{2} | X_{3} | Y_{1} | Y_{2} |
---|---|---|---|---|

m_{1} | m_{2} | m_{3} | m_{4} | m_{5} |

m_{2} | m_{3} | m_{4} | m_{5} | m_{6} |

m_{3} | m_{4} | m_{5} | m_{6} | m_{7} |

Hyperparameter | MLP | RNNs | CNN |
---|---|---|---|

Number of Layers | 2 | 3 | 3 |

Number of Hidden Layers | 1 | 2 | 2 |

Number of Nodes | 500 | 500 - 50 | - |

Number of Filters | - | - | 256 |

Kernel Size | - | - | 3 |

Weight Initialization Technique | Glorot Uniform (Xavier) | Glorot Uniform (Xavier) | Glorot Uniform (Xavier) |

Learning Rate | 0.01 | 0.01 | 0.01 |

Gradient Descent Optimizer | Adam | Adam | Adam |

Batch Size | 16 | 16 | 16 |

Activation Function | ReLU | tanh | tanh |

Loss Function | mean squared error | mean squared error | mean squared error |

Timesteps | 24 | 24 | 24 |

Software Project | Model | R${}^{2}$-Fit | MAE-Fit | RMSE-Fit | MAE-Test | RMSE-Test |
---|---|---|---|---|---|---|

Google Chrome | Random Walk | −0.578 | 11.949 | 18.634 | 19.083 | 24.244 |

SES | 0.062 | 9.423 | 14.368 | 14.873 | 17.455 | |

TES | 0.136 | 9.680 | 13.791 | 14.248 | 17.697 | |

ARIMA | 0.291 | 8.938 | 12.490 | 14.027 | 17.191 | |

Croston | 0.201 | 9.016 | 12.876 | 15.926 | 21.166 | |

MLP | −0.124 | 10.007 | 13.280 | 13.955 | 15.888 | |

LSTM | −0.037 | 9.308 | 12.782 | 15.440 | 16.675 | |

GRU | −0.011 | 9.243 | 12.642 | 15.285 | 16.474 | |

BiLSTM | −0.227 | 8.981 | 13.887 | 16.890 | 20.388 | |

CNN | −0.066 | 9.604 | 12.936 | 14.634 | 16.291 | |

Internet Explorer | Random Walk | −0.009 | 3.745 | 7.352 | 15.792 | 16.226 |

SES | 0.443 | 2.959 | 5.458 | 10.562 | 11.181 | |

TES | 0.513 | 2.936 | 5.106 | 12.510 | 13.928 | |

ARIMA | 0.546 | 3.018 | 4.928 | 7.051 | 7.679 | |

Croston | 0.637 | 2.535 | 4.951 | 14.635 | 15.103 | |

MLP | −0.159 | 3.838 | 7.801 | 5.680 | 7.233 | |

LSTM | −0.062 | 3.880 | 7.468 | 3.487 | 4.054 | |

GRU | −0.071 | 3.872 | 7.493 | 3.636 | 4.417 | |

BiLSTM | −0.099 | 3.732 | 7.602 | 3.265 | 3.859 | |

CNN | −0.054 | 3.795 | 7.451 | 5.340 | 5.999 | |

Apple macOS X | Random Walk | −0.930 | 11.823 | 18.804 | 77.375 | 82.276 |

SES | 0.030 | 8.769 | 13.330 | 21.383 | 32.705 | |

TES | 0.090 | 8.932 | 12.915 | 21.345 | 32.042 | |

ARIMA | 0.124 | 8.548 | 12.670 | 20.555 | 33.767 | |

Croston | 0.233 | 7.940 | 11.853 | 21.783 | 32.481 | |

MLP | 0.007 | 9.677 | 12.575 | 21.617 | 31.689 | |

LSTM | 0.016 | 9.650 | 12.514 | 20.753 | 32.687 | |

GRU | 0.009 | 9.507 | 12.555 | 20.630 | 33.168 | |

BiLSTM | −0.254 | 8.328 | 14.121 | 19.810 | 36.230 | |

CNN | −0.045 | 10.125 | 12.900 | 20.907 | 32.015 | |

Ubuntu Linux | Random Walk | 0.064 | 1.862 | 3.917 | 1.375 | 2.031 |

SES | 0.278 | 1.760 | 3.440 | 1.513 | 2.014 | |

TES | 0.349 | 1.945 | 3.266 | 1.293 | 1.965 | |

ARIMA | 0.339 | 1.791 | 3.292 | 1.537 | 2.006 | |

Croston | 0.290 | 1.867 | 3.509 | 1.334 | 2.156 | |

MLP | −0.095 | 2.933 | 5.045 | 1.333 | 2.086 | |

LSTM | −0.178 | 2.920 | 5.233 | 1.272 | 1.964 | |

GRU | −0.222 | 2.924 | 5.328 | 1.305 | 1.996 | |

BiLSTM | −0.165 | 2.835 | 5.201 | 1.331 | 2.085 | |

CNN | −0.144 | 2.863 | 5.155 | 1.383 | 1.965 | |

Microsoft Office | Random Walk | −0.554 | 1.369 | 2.305 | 1.208 | 1.671 |

SES | 0.089 | 1.097 | 1.765 | 1.254 | 1.617 | |

TES | 0.152 | 1.102 | 1.702 | 1.323 | 1.682 | |

ARIMA | 0.122 | 1.134 | 1.732 | 1.270 | 1.717 | |

Croston | 0.137 | 1.106 | 1.713 | 1.223 | 1.648 | |

MLP | −0.051 | 1.383 | 1.992 | 1.065 | 1.529 | |

LSTM | .010 | 1.368 | 1.932 | 1.263 | 1.695 | |

GRU | −0.018 | 1.426 | 1.959 | 1.283 | 1.672 | |

BiLSTM | −0.042 | 1.238 | 1.983 | 1.403 | 1.875 | |

CNN | −0.034 | 1.393 | 1.976 | 1.289 | 1.645 |

**Table 10.**Wilcoxon signed-rank test results of the best pairs of statistical and deep learning models.

Software Project | p-Value |
---|---|

Google Chrome | 0.6634 |

Internet Explorer | 0.0001 |

Apple macOS X | 0.8115 |

Ubuntu Linux | 0.8995 |

Microsoft Office | 0.0950 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kalouptsoglou, I.; Tsoukalas, D.; Siavvas, M.; Kehagias, D.; Chatzigeorgiou, A.; Ampatzoglou, A.
Time Series Forecasting of Software Vulnerabilities Using Statistical and Deep Learning Models. *Electronics* **2022**, *11*, 2820.
https://doi.org/10.3390/electronics11182820

**AMA Style**

Kalouptsoglou I, Tsoukalas D, Siavvas M, Kehagias D, Chatzigeorgiou A, Ampatzoglou A.
Time Series Forecasting of Software Vulnerabilities Using Statistical and Deep Learning Models. *Electronics*. 2022; 11(18):2820.
https://doi.org/10.3390/electronics11182820

**Chicago/Turabian Style**

Kalouptsoglou, Ilias, Dimitrios Tsoukalas, Miltiadis Siavvas, Dionysios Kehagias, Alexander Chatzigeorgiou, and Apostolos Ampatzoglou.
2022. "Time Series Forecasting of Software Vulnerabilities Using Statistical and Deep Learning Models" *Electronics* 11, no. 18: 2820.
https://doi.org/10.3390/electronics11182820