An Evaluation Framework for Cybersecurity Maturity Aligned with the NIST CSF

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The manuscript titled "An Evaluation Framework for Cybersecurity Maturity Aligned with the NIST CSF" presents a dual-survey methodology to assess and improve cybersecurity maturity within organizations. The authors propose a framework that integrates insights from cybersecurity experts (Group I) and organizational stakeholders (Group II) to evaluate cybersecurity practices across the five core functions of the NIST Cybersecurity Framework (CSF): Identify, Protect, Detect, Respond, and Recover. While the paper is well-structured and addresses a critical need for standardized maturity assessments, there are several major and minor issues that require attention to enhance its clarity, relevance, and overall contribution.

 

Major Issues

1. Limited Generalizability Due to Small Sample Size
   One of the most significant limitations of the study is the small sample size of organizations involved in the experimental analysis. The authors acknowledge this limitation but do not provide a robust justification for why the findings can be generalized to larger or more diverse populations. This undermines the credibility of the proposed framework as a universally applicable tool.
   Suggestion : Expand the scope of the study by including a larger and more diverse set of organizations across different industries and geographical regions. Additionally, discuss the potential scalability of the framework in the context of varying organizational sizes and resources.

2. Overreliance on Subjective Expert Opinions
   The framework heavily depends on subjective ratings provided by cybersecurity experts, which introduces the risk of bias and inconsistency. For instance, the importance levels assigned to NIST CSF functions and controls may vary significantly depending on the expertise and experience of the respondents.
   Suggestion : Incorporate objective metrics or benchmarks to complement expert opinions. For example, integrating quantitative data from real-world cybersecurity incidents or industry standards could enhance the reliability of the assessment.

3. Insufficient Discussion on Practical Implementation Challenges
   While the paper outlines the theoretical framework and methodology in detail, it lacks a comprehensive discussion on practical challenges organizations might face when implementing the proposed approach. For instance, resource-constrained organizations, particularly small and medium-sized enterprises (SMEs), may struggle to adopt such a framework due to cost, technical expertise, or time constraints.
   Suggestion : Include a dedicated section addressing implementation barriers and proposing strategies to overcome them, such as cost-effective tools, training programs, or phased adoption plans.

4. Ambiguity in Algorithm Descriptions
   The algorithms presented in the paper (e.g., Algorithm 1 and Algorithm 2) lack sufficient explanation and clarity. For example, the calculation of importance degrees and the integration of expert matrices are described in a way that may confuse readers unfamiliar with advanced mathematical concepts.
   Suggestion : Provide step-by-step explanations and visual aids (e.g., flowcharts) to clarify the algorithmic processes. Additionally, include a worked-out example to demonstrate how the calculations are applied in practice.

5. Weak Alignment with Emerging Technologies
   The paper does not adequately explore the role of emerging technologies, such as artificial intelligence (AI) and machine learning (ML), in enhancing cybersecurity maturity assessments. Given the rapid advancements in these fields, their omission limits the framework's relevance in addressing modern cybersecurity challenges.
   Suggestion : Discuss how AI/ML techniques could be integrated into the framework to automate threat detection, analyze large datasets, or predict future vulnerabilities.

 

Minor Issues

1. Repetitive Content
   Certain sections of the paper, particularly the descriptions of NIST CSF functions and categories, are repetitive. This redundancy detracts from the flow of the manuscript and makes it less engaging for readers.
   Suggestion : Consolidate repetitive content into a single section (e.g., a dedicated "Background" subsection) and refer back to it as needed.

2. Inconsistent Terminology
   The paper occasionally uses inconsistent terminology, such as referring to "cybersecurity maturity index" and "maturity score" interchangeably without clear definitions. This can confuse readers unfamiliar with the domain.
   Suggestion : Provide a glossary of key terms or include clear definitions when introducing technical concepts. For example, define "cybersecurity maturity index" early in the paper.

3. Formatting and Presentation Issues
   There are minor formatting inconsistencies, such as mismatched fonts in tables, incomplete equations (e.g., Equation 3), and inconsistent use of acronyms (e.g., "NIST CSF" vs. "Cybersecurity Framework").
   Suggestion : Standardize the formatting of equations, tables, and figures to improve readability. Ensure consistent use of acronyms throughout the manuscript.

4. Missing Acknowledgment of Limitations
   While the paper acknowledges some limitations (e.g., small sample size), it does not address potential biases in expert opinions or the environmental impact of frequent cybersecurity assessments.
   Suggestion : Expand the "Limitations" section to address these overlooked aspects.

5. Outdated References
   Some references (e.g., [1], [2]) are from 2017 or earlier, which may not reflect the latest advancements in cybersecurity maturity models and frameworks.
   Suggestion : Update the reference list to include more recent studies (post-2022) to ensure the paper reflects the current state of the field. Some suggested literature is provided below, and citing these articles will strengthen the background of this study.

   DOI: 10.3233/IDT-230284

   https://doi.org/10.1016/j.jksuci.2023.101820

   https://doi.org/10.1016/j.jksuci.2024.101939

   https://doi.org/10.1016/j.asej.2024.102777

   https://doi.org/10.1016/j.asej.2024.102642

   https://doi.org/10.1016/j.jksuci.2024.102164

   DOI: 10.3233/JIFS-231969

   DOI: 10.32604/cmc.2024.047530

Comments on the Quality of English Language

The English could be improved to more clearly express the research.

Author Response

Please check the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper suggests a supplementary tool for assessing cybersecurity maturity levels, guided by the NIST CSF.  Also, the cybersecurity maturity index was introduced and it illustrated how it might improve organizations security posture.

The readers might observe that the paper is focused on the only statistical conception, specifically, average.  It might be interesting for the readers to see some discussion on whether using other conceptions,  say, median, mode, standard deviation, variance, distribution shape, correlation and so on, might improve the suggested approach.

Author Response

Please check the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The paper delves into an interesting area of the cybersecurity maturity model using NIST CSF. The paper is well-written, however, there are a few concerns for the authors to address:
1. Authors should consider replacing No Technical in Figure 1 with Non-Technical as stated in the text.
2 . There are a few grammatical errors such as "Wich", re- sources ... to be addressed
3. "References" is missing as a subsection.
4. The survey structure in section 3 is missing vital information on data quality. Authors should include details of sample size (Total number of surveys sent out), the response rate from the survey and the data extraction/cleaning process. For example, in the questionnaire, how did the authors deal with sections not answered by the respondent? Out of the total respondents (experts), how many had more than 5 years of experience and how many had 2 years of experience? These details are necessary to lend credibility to importance degree calculations (Algorithm 1) 

Author Response

Please check the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The paper can be accepted in the current form.