Improved Detection and Response via Optimized Alerts: Usability Study

Round 1

Reviewer 1 Report

The proposal is sound and interesting. Description are exhaustive. The paper is well-written. Results are coherent and the conclusions are supported by the results. In my opinion the paper may be accepted in its current form.

Author Response

Thank you for taking the time to review, and your kind comments.

Cheers, Russ

Reviewer 2 Report

The paper presents an exciting and mainly analytical view of how security analysts perceive the outputs of the data science/machine learning method within, for example, the security operation center. The author focused on visual alert output (VAO), text alert output (TAO), and the difference between the adoption of these outputs. On the other hand, the paper consists of excellent justification for the sample size.

The paper describes the data processing, preprocessing, and evaluation of results in detail. I miss a better description, for example, individual scenarios. It would greatly help in understanding the results.

The conclusions are interesting, although they confirm generally known facts (visual alert output is preferred). The addition of option 3 (visual and text alert output addition) also sounds like a known fact.

The age and length of the respondents' experience would also be a suitable addition. It could show the influence of expertise on selecting an appropriate output. Conversely, the 1st line analysts in the security operation center will begin to be "immune" to specific outcomes after some time. It would be interesting to see how the visual alert output and the text alert output affect this fact. It would probably require a larger sample of respondents.

157 - Figure 1. Theoretical Framework: Technology Acceptance Model. Adapted from - I suggest using the short citation

Author Response

Thank you for taking the time to review, and your feedback.

I agree it appears the study concludes what feels obvious to us with knowledge of security operations and the benefits of visualization, yet know prior research attempted quantitatively validate the assumption. I hope this body of work is a step closer to definitive. 

I appreciate the recommendation regarding Figure 1 and have updated as suggested.

Cheers, Russ

Reviewer 3 Report

In this paper, the author examined tactics to reduce security data fatigue, increase detection accuracy and enhance security analyst experience using security alert output generated via data science and machine learning models. The research determined if security analysts utilizing this security alert data perceive a statistically significant difference in usability between security alert output that is visualized versus that which is text-based. Security analysts benefit twofold: the efficiency of results derived at scale via ML models, with the additional benefit of quality alert results derived from these same models. This quantitative, quasi-experimental, explanatory study conveys survey research performed to understand security analysts’ perceptions via the Technology Acceptance Model. The population studied was security analysts working in a  defender capacity, analyzing security monitoring data and alerts. The more specific sample was security analysts and managers in Security Operation Center (SOC), Digital Forensic and Incident  Response (DFIR), Detection and Response Team (DART), and Threat Intelligence (TI) roles. Data analysis indicated a significant difference in security analysts' perception of usability in favor of visualized alert output over text alert output. The study’s results showed how organizations can more effectively combat external threats by emphasizing visual rather than textual alerts. I read the article, it is a somewhat good article with the right balance of theory and practice. I do have the following serious concerns that need corrections during the revisions to improve this article further.

• The abstract should be revised. It should contain bits of the following information: introduction, problems statement, aim/objectives, methodology, findings, the significance of results, and a concluding statement.
• The contribution section should highlight the implication of research with practical applications. In the current form, a single paragraph is not convincing from a technical point of view. Authors need to write contributions with bullets.
• Please make the related work section concise. It would be better to retain 2~3 passages and please provide a critical analysis of existing studies.
• Captions of the table should be written in consistent English.
• It would be better to include a table before the conclusion by including overall improvements of the proposed method compared to previous work.
• Please list the organization of the paper in the introduction section.
• This paper is lengthy, and many sections/subsections are not needed at all. Authors need to delete most sections (IMO 15~20 pages are enough) or provide a figure at the start that shows the connection between sections.
• In section 3, it is suggested to state a problem statement first before discussing the proposed solution.
• Quality of presentation can be improved further. For example, some figures can be placed side by side to reduce the length.
• In the title, the target system/application should be mentioned. The current title is too generic covering almost the whole cybersecurity topic.
• Many abbreviations have been defined two to three times in the paper. The best practice is to define it in one place and use the abbreviated form later.
• Overall, a complete restructuring of the paper is needed by retaining key points only. State the problem, add supportive material, discuss solutions, and add the application area of the research in the security domain.
• English can also be improved as some expressions are hard to understand from the reader's point of view.

Author Response

Responses to feedback (restated here) are provided in line below:

• The abstract should be revised. It should contain bits of the following information: introduction, problems statement, aim/objectives, methodology, findings, the significance of results, and a concluding statement. 
  • Respectfully, and with apologies, but the abstract as written includes every item you ask for, and as abstracts should be, is intentionally short:
    • Introduction: "Security analysts working in the modern threat landscape face excessive events and alerts, a high volume of false positives alerts, significant time constraints, innovative adversaries, and a staggering volume of unstructured data."
    • Problem statement: "Organizations thus risk data breach, loss of valuable human resources, reputational damage, and impact to revenue when excessive security alert volume and a lack of fidelity degrade detection services."
    • Aims/objectives: "This study examined tactics to reduce security data fatigue, increase detection accuracy, and enhance security analyst experience using security alert output generated via data science and machine learning models."
    • Methodology: "This quantitative, quasi-experimental, explanatory study conveys survey research performed to understand security analysts’ perceptions via the Technology Acceptance Model."
    • Findings: "Data analysis indicated a significant difference in security analyst perception of usability in favor of visualized alert output over text alert output."
    • Significance: "The study’s results showed how organizations can more effectively combat external threats by emphasizing visual rather than textual alerts."
• The contribution section should highlight the implication of research with practical applications. In the current form, a single paragraph is not convincing from a technical point of view. Authors need to write contributions with bullets.
  • While already listed in section 4.4 under Implications for practice I've added a bullet list in 4.2
• Please make the related work section concise. It would be better to retain 2~3 passages and please provide a critical analysis of existing studies.
  • There is no related work section, I intentionally cut the literature review to avoid an even longer submission than currently under consideration. See 4.2 for additional explanation regarding previous studies.
• Captions of the table should be written in consistent English.
  • Table numbering and formatting (italics) corrected accordingly. Consistent English is already utilized. 
• It would be better to include a table before the conclusion by including overall improvements of the proposed method compared to previous work.
  • There is no related previous work. See 4.2 for additional explanation regarding previous studies.
• Please list the organization of the paper in the introduction section.
  • Provided as requested.
• This paper is lengthy, and many sections/subsections are not needed at all. Authors need to delete most sections (IMO 15~20 pages are enough) or provide a figure at the start that shows the connection between sections.
  • • Trimmed to the degree possible, now five pages shorter.
• In section 3, it is suggested to state a problem statement first before discussing the proposed solution.
  • Problem statement added as requested.
• Quality of presentation can be improved further. For example, some figures can be placed side by side to reduce the length.
  • Agreed and implemented as requested.
• In the title, the target system/application should be mentioned. The current title is too generic covering almost the whole cybersecurity topic.
  • There are no specific target system/applications, the topic is specific to security events experienced by SOC personnel across a wide array of systems and apps and is, as such, intentionally broad & generic.
• Many abbreviations have been defined two to three times in the paper. The best practice is to define it in one place and use the abbreviated form later.
  • Agreed and corrected per SOC, DFIR, DART & TI, as well as TAM, it's components, and TAO & VAO.

NOTE: A complete literature could be made available as part of Supplementary Materials if believed to be important.

Round 2

Reviewer 3 Report

The authors reviewed the paper addressing my previous comments in a very limited way. Research design and presentation issues still exist in the paper. I do have the following concerns that need corrections during the next revisions to improve this article further.

• Please revise the title of section 4.2 as Original contribution to the body of knowledge.
• Please make a conclusion as a separate section. IMO, the second paragraph is not needed. Authors can write this section with just one paragraph concisely.
• References can be cited in chronological order. Reference 31 has been cited on page #: 07 which is not the correct order.
• Line #: 230, The research questions frame the variables, and the data collected. This sentence should be revised with better English.
• This paper is lengthy, and many sections/subsections make the research design very hard to follow. Authors need to delete most sections (IMO 15~20 pages are enough) or provide a figure at the start that shows the connection between sections.
• The quality of the presentation has become worse in this version. For example, authors need to increase the font size while placing figures side by side.
• Tables are not consistent as well. For example, one can see table 5 and table 12 for comparison.

Author Response

Thank you for your continued review, feedback addressed in-line with your comments:

• Please revise the title of section 4.2 as Original contribution to the body of knowledge.
  • Revised per your request
• Please make a conclusion as a separate section. IMO, the second paragraph is not needed. Authors can write this section with just one paragraph concisely.
  • Revised per your request
• References can be cited in chronological order. Reference 31 has been cited on page #: 07 which is not the correct order. 
  • Reference 31 is redundant there, I simply removed it as such.
• Line #: 230, The research questions frame the variables, and the data collected. This sentence should be revised with better English.
  • I deleted this sentence entirely. It was, as you say, of low quality, and unnecessary.
• This paper is lengthy, and many sections/subsections make the research design very hard to follow. Authors need to delete most sections (IMO 15~20 pages are enough) or provide a figure at the start that shows the connection between sections.
  • I removed Partial Eta Squared (effect size) content as unnecessary.
  • I removed Means Analysis tables and content as unnecessary.
  • Added figure at start as recommended
• The quality of the presentation has become worse in this version. For example, authors need to increase the font size while placing figures side by side.
  • Agreed, revised per your request. Deleted the second Scenarios image as unnecessary and resized the Maximum Visual graphic accordingly, as well as updated other poor figures.
• Tables are not consistent as well. For example, one can see table 5 and table 12 for comparison.
  • Agreed, tables are all cleaned up.

Changes are tracked in the revised attached document.