Better Not to Use Vulnerability’s Reference for Exploitability Prediction
Round 1
Reviewer 1 Report
The authors presented a new approach using machine learning modelling to predict a vulnerability’s exploitability and impact on the result of this modelling of including references related to Common Vulnerabilities and Exposures (CVE) description and containing proof-of-concept codes.
The initial steps of data preparation that required an extensive work has been well documented and well-presented and I believe that that phase et amount of data used for the analysis is sufficient. In the other side, I believe that the phase of development of a machine learning model is weak and could be improved:
- The authors used just one model with Random Forest algorithm, while it would be much appropriate and richer to compare results of several algorithms instead.
- The conclusion presented in the article are not well supported with a very manor difference on the results of the various dataset. Applying more than one model as suggest in 1- could help at overcoming this limit.
Other notes:
In line 138: “we analyzed all references”, How? Manually?!
Image 3a- red line for first URL not clear
Author Response
Please see the attachment
Author Response File: 
Author Response.docx
Reviewer 2 Report
This manuscript pointed out a flaw of recent exploitability prediction studies that their datasets include answer information, or hosts that provide PoC codes. It was basically well written: the motivation is clearly described with an example and the experiments seemed to be conducted properly.
A weak point of this manuscript is that, according to the evaluation results, the difference with using the PoC host information was not so significant to support the authors' hypothesis. In other words, the reviewer could not deny the possibility that the difference was obtained by chance.
So the reviewer suggests the authors to provide a more reliable evidence for their claim. For example, the reviewer would like to see the prediction result for each test set of the k-fold cross validation, if the same tendency can be observed regardless of the test set. Another experiment with a different value of k or a different learning algorithm may also be helpful.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The paper has been much improved with more supporting results to the initially presented hypothesis. However, the difference in the results of accuracy between the various models is not that large.
Thus, I would like to ask the authors to amend some parts in the paper to emphasis on this point and express the relativity of the results:
1- At abstract level (lines 12 and 13): please mention the relatively low difference of the results briefly.
2- At Discussion level (lines 194-208): please extend the discussion by quantifying the difference of accuracy between the two groups (use numerical results during the discussion).
Also, it would be interesting to compare the results for each machine learning models: for instance it would be good to discuss also the other results- precision, recall and F1 score- with the purpose of comparing the performances of machine learning algorithms.
3- At conclusion level (lines 220-221) the conclusion is well written, please mention here also the relatively low difference between the two groups (which can be presented as a good reason for the proposed future work)
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 2 Report
This manuscript pointed out a flaw of recent exploitability prediction studies that their datasets include answer information, or hosts that provide PoC codes.
Now that a reliable evidence with multiple machine learning techniques has been provided from the authors, I would like to recommend the revised manuscript as acceptance.
An additional comment: there is room for reconsideration in the future study paragraph because it includes an additional experiment, which has already been presented in the revised manuscript.
Author Response
All authors would like to thank you for your time and valuable comments, and these helped us improve the contribution of our manuscript. We made a change in future study part to accommodate your comment as follows:
- We omitted the related sentence mentioning additional experiments using various machine learning algorithms.
We once again would like to thank you for the time and effort in reviewing our manuscript and helping in improving its quality.
Sincerely,
Authors
