Assessing Competencies Using Scenario-Based Learning in Cybersecurity

: Recent years have seen a disconnect between much-needed real-world skills and knowledge imparted to cybersecurity graduates by higher education institutions. As employers are shifting their focus to skills and competencies when hiring fresh graduates, higher education institutions are facing a call to action to design curricula that impart relevant knowledge, skills, and competencies to their graduates, and to devise effective means to assess them. Some institutions have successfully engaged with industry partners in creating apprenticeship programs and work-based learning for their students. However, not all educational institutions have similar capabilities and resources. A trend in engineering, computer science, and information technology programs across the United States is to design project-based or scenario-based curricula that impart relevant knowledge, skills, and competencies. At our institution, we have taken an innovative approach in designing our cybersecurity courses using scenario-based learning and assessing knowledge, skills, and competencies using scenario-guiding questions. We have used the National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework and the Ofﬁce of Personnel Management (OPM) Hiring Cybersecurity Workforce report for skills, knowledge, and competency mapping. This paper highlights our approach, presenting its overall design and two example mappings.


Introduction
The increasing call to action for higher education institutions to impart relevant skills, knowledge and competencies to their graduates is leaving a footprint in cybersecurity education. There is a real disconnect between what students are learning and what is expected of them in the real-world, pushing higher education institutions to adopt innovative learning and teaching strategies. Sometimes, a lack of industry partnerships and other resources are limiting institutions in terms of the design of practical, work-based learning. However, many academics have been engaged in designing and delivering project-based and/or scenario-based curricula that impart relevant knowledge, skills, and competencies. Another challenge that academic programs are facing is how to realistically assess competencies and skills. Traditional approaches such as exams, quizzes, and laboratory exercises can only achieve so much, leaving a gap regarding what students are learning and what their employers expect of them. Moreover, many employers are also shifting their focus to competency-based hiring [1], which places an enormous pressure on higher education institutions to design, deliver, and assess relevant and rigorous curricula. Fortunately, the NICE Cybersecurity Workforce Framework [2,3], the Office of Personnel Management's report on Attracting, Hiring, and Retaining a Federal Cybersecurity Workforce [4], and the Cybersecurity Competency Model [5] provide some good reference frameworks that institutions can start with.
Recently, NIST published a draft NICE Cybersecurity Workforce Framework Competencies-request for public comments [6] in which competency is defined as "as a mechanism for organizations to assess learners". They also released an initial draft of competencies-J. Cybersecur. Priv. 2021, 1 540 technical, professional, organizational and leadership-to accompany the draft framework. The public comment space is centered around the following questions: • Should the NICE Framework work roles and competencies be addresse d separately or in tandem? • Some competencies have been defined as type "professional". Should these be included in the NICE Framework competencies? Should they be included as knowledge and skills statements? • Should proficiency levels be incorporated in the NICE Framework Competencies? If yes, then how? • Is the provision of different competency types useful?

Competency Framework
The NICE Cybersecurity Workforce Framework [2], and its subsequent revision [3], serve as fundamental references and resources for describing and sharing information about cybersecurity work roles and the knowledge, skills, and tasks (KSTs) needed for those roles that can strengthen the cybersecurity posture of an organization. The framework organizes cybersecurity work roles into seven categories, namely, Securely Provision, Operate and Maintain, Oversee and Govern, Protect and Defend, Analyze, Collect and Operate, and Investigate, and identifies the KSTs needed for each work role. The complete list of KSTs for each work role can be found in [2] and [3]. The framework serves as a vital resource to bridge the gap between education and industry by providing a common lexicon for organizations to identify and recruit for cybersecurity work roles, and for education and training programs to identify the KSTs and prepare professionals for KSTs and competencies required for those roles. A revised framework was published in November 2020 [3].
The U.S. Office of Personnel Management (OPM) published a report on Attracting, Hiring, and Retaining a Federal Cybersecurity Workforce [4] in October 2018, in which they discussed their Cybersecurity Competency Model. The model was developed using the following two categories based on OPM's collaboration with the National Security Council Interagency Policy Committee Working Group on cybersecurity education and workforce issues.

•
IT infrastructure, operations, computer network defense and information assurance • Domestic law enforcement and counterintelligence The set of competencies was created by surveying a select group of employers in various occupations related to cybersecurity. The entire list of competencies can be found in Appendix B of the OPM report [4]. The scenarios that we have created are mapped to these competencies, as described in the next section.
The Cybersecurity Competency Model [5] was created by the Employment and Training Administration (ETA) in collaboration with the Department of Homeland Security and the National Initiative for Cybersecurity Education (NICE) to develop a comprehensive competency model for the cybersecurity workforce. The Cybersecurity Competency Model defines the latest skill and knowledge requirements needed by individuals to improve the security posture of their organizations. The model incorporates competencies identified in the NICE National Cybersecurity Workforce Framework and complements the framework by including competencies needed by the workforce and those needed by cybersecurity professionals.
In the following sections, we discuss related work on competency assessment and describe scenario-based learning and our approach of using scenario-based learning technique to assess competencies. We illustrate our approach with two example competencies from the competency frameworks discussed in this section.

Related Work on Competency Assessment
In [7], the authors conducted an evaluation of a novel computerized competencybased assessment of computational thinking. The assessment process draws upon a multidisciplinary approach by combining psychometrics, learning sciences, and computing education. Further, the approach utilizes both summative and formative assessments to generate constructive and personalized reports for learners. A competency assessment model for engineering education using discussion forums was proposed by Felicio and Muniz in [8]. In this model, the evaluation feedback is defined by utilizing the Rubrics scoring tool and Bloom's Taxonomy. Related to this work is an ontological knowledge assessment model proposed by Zulfiya, Gulmira, Assel and Altynbek in [9]. In this model, assessments of student competencies at the level of the educational program are undertaken at two lower levels: modules and discipline.
In [10], Grann and Bushway present a visualization of assessed competency using a competency map. The authors argue that MBA students who utilize the competency map demonstrate elevated competency levels and make progress at a greater rate compared to their peers who have not used the visual tool. A study on the competencies of data scientists was presented by Hattingh, Marshall, Holmner and Naidoo in [11]. Their findings were grouped into six competency themes: organizational, technical, analytical, ethical and regulatory, cognitive, and social. Using a thematic analysis, a unified model of data science competency was developed. This is a seminal work for the conceptual development of competency in the discipline, and could contribute to the improvement of the data science workforce.
In an article published in the Infragard journal, Watkins et al. argued that the competency-based learning approach "focuses on mastering each critical knowledge and skill component before moving on to the next one without being constrained to a fixed time period", in contrast to the outcome-based approach which attempts to impart knowledge and skill components within a fixed time period with the objective of reaching a passing level of competence [12].
In [13], Brilingait, Bukauskasa, and Juozapaviciusb proposed a framework using cyber defense exercises as competency-based approach for cybersecurity education, and an assessment framework to measure competency. The framework consists of a sequence of steps for team formation, determination of objectives for each team, exercise flow, and formative assessments based on surveys. Exercises or competitions, as they were proposed in the paper, are deemed to be effective tools for hands-on, competency-based learning. However, they have one major drawback, i.e., they typically provide a learning environment for experienced students and do not, by design, build on one another, from beginner to advanced knowledge levels, unless they are designed to achieve those goals. Further, most of the time, it is not feasible to design competitions in this manner.
These diverse writings on competency assessment underscore the significance of competency-based learning and competency assessment in the learning process. The U.S. Department of Education has emphasized the importance and need for competency-based learning, and has articulated that "by enabling students to master skills at their own pace, competency-based learning systems help to save both time and money. Depending on the strategy pursued, competency-based systems also create multiple pathways to graduation, make better use of technology, support new staffing patterns that utilize teacher skills and interests differently, take advantage of learning opportunities outside of school hours and walls, and help identify opportunities to target interventions to meet the specific learning needs of students" [14]. In this paper, we present our approach for the design of scenario-based, competency-focused learning, and introduce a methodology with which to assess competency through scenarios.

Scenario-Based Learning
Scenario-based learning is firmly based on the theory of contextual learning, i.e., that learning takes place in a context in which it is applied. It subscribes to the idea that knowledge is best acquired and fully understood when situated within its proper context. Using real-life situations, scenario-based learning provides a relatable and relevant learning experience through an immersive and highly engaging approach [15]. Scenario-based learning works best for training when tasks are set that involve serious consequences, which is apt for the field of cybersecurity. It offers a simulated environment or situation in which learners can afford to make mistakes without serious repercussions. As noted by the authors of [16], "Scenario-based learning is grounded in the idea that students learn better through application of authentic tasks in a real-world context". In [17], Iverson and Colky emphasized that scenario-based learning supports the constructivist view that learning is effective when students apply prior knowledge and construct meaning from that knowledge and experience. These constructs emphasize the need for scenario-based learning in disciplines like cybersecurity, where the application of knowledge within specific contexts is important for learning.
In [18], Clark proposes the following checklist for determining whether scenario-based learning is the right option:

1.
Are the outcomes based on skill development or problem solving? 2.
Does it provide a simulated experience in lieu of a real and dangerous situation? 3.
Are the students provided with relevant knowledge for decision making? 4.
Is a scenario based solution cost-and time-effective? 5.
Will the content and acquired skills be sufficiently relevant to justify their inclusion?
Based on the recent shift among employers to competency and skill-based hiring, and the disconnect between imparted knowledge and needed skills and competencies, it is proposed that scenario-based learning is an excellent option for cybersecurity education and training. In this paper, we outline our approach for the design of scenario-based learning to assess competencies in cybersecurity curricula. We also highlight two use cases with two example courses.

Assessing Competencies Using Scenario-Based Learning
One critical question that always needs to be answered is how do we determine whether the learning objectives have been satisfied? In today's world, where skills and knowledge are being used interchangeably more and more, we need a mechanism to ensure that students gain the knowledge, skills, and competencies required to effectively perform their jobs.
Although NIST published a draft of the NICE Cybersecurity Workforce Framework Competencies [6] for public comments, a good starting point would be the US Office of Personnel Management's report on Attracting, Hiring, and Retaining a Federal Cybersecurity Workforce [4] and list of competencies. Our approach to using scenario-based learning in assessing competencies is depicted in Figure 1, and may be summarized as follows:

1.
Start from the definition of a selected competency from the OPM report [4]. Alternatively, a work role can be selected from the NICE Cybersecurity Workforce Framework revision 1 [3].

2.
Which tasks should be performed to satisfy that work role or competency? Choose a set of tasks from the NICE Cybersecurity Workforce Framework [2,3]  Create learning modules incorporating the knowledge areas, skills, and tasks starting at the beginner level and moving up to advanced level. 6.
Results in sequence of courses, starting from foundational course, leading to an intermediate-level course and culminating in a scenario-based experience. 7.
The scenario-based experience will follow the theory of contextual learning with tasks specifically designed to assess the overall competency.

8.
Knowledge, skills, and competencies will be assessed by designing appropriate scenario-guiding questions that students will have to answer as they progress through the scenarios.
all three levels (beginner, intermediate, expert). 5. Create learning modules incorporating the knowledge areas, skills, and tasks starting at the beginner level and moving up to advanced level. 6. Results in sequence of courses, starting from foundational course, leading to an intermediate-level course and culminating in a scenario-based experience. 7. The scenario-based experience will follow the theory of contextual learning with tasks specifically designed to assess the overall competency. 8. Knowledge, skills, and competencies will be assessed by designing appropriate scenario-guiding questions that students will have to answer as they progress through the scenarios.

Our Approach
We discuss two approaches with which to assess competencies using scenario-based learning, both of which map to the NICE Cybersecurity Workforce Framework [2,3]. In our first approach, we start with the knowledge, skills, and tasks that are used to satisfy the chosen competency. The scenarios are created with guiding questions that will lead students to work on the scenario and answer questions as they progress. Each answer is mapped to a set of knowledge, skills, and tasks, thus assessing whether the student is able to perform that task and has assimilated the relevant knowledge and skills. Our second approach follows the revised NICE Cybersecurity Workforce Framework [3] and starts with a work role corresponding to the chosen competency. We then list the tasks that are needed for that work role and to achieve the chosen competency, followed by sets of knowledge and skills needed to perform relevant tasks. Again, scenarios are designed to guide students through a set of tasks and to impart the knowledge and skills required to perform those tasks.
Although we did not map learning outcomes and tasks as suggested in Bloom's taxonomy [19], our expectation is that each learning outcome mapped to a task from the

Our Approach
We discuss two approaches with which to assess competencies using scenario-based learning, both of which map to the NICE Cybersecurity Workforce Framework [2,3]. In our first approach, we start with the knowledge, skills, and tasks that are used to satisfy the chosen competency. The scenarios are created with guiding questions that will lead students to work on the scenario and answer questions as they progress. Each answer is mapped to a set of knowledge, skills, and tasks, thus assessing whether the student is able to perform that task and has assimilated the relevant knowledge and skills. Our second approach follows the revised NICE Cybersecurity Workforce Framework [3] and starts with a work role corresponding to the chosen competency. We then list the tasks that are needed for that work role and to achieve the chosen competency, followed by sets of knowledge and skills needed to perform relevant tasks. Again, scenarios are designed to guide students through a set of tasks and to impart the knowledge and skills required to perform those tasks.
Although we did not map learning outcomes and tasks as suggested in Bloom's taxonomy [19], our expectation is that each learning outcome mapped to a task from the NICE Framework will need to relate to the appropriate action verb in the Bloom's taxonomy depending on the proficiency level and competency required for the corresponding work role. For example, for computer network defense competencies, tasks that are at an advanced proficiency level relate to either the 'analyze', 'evaluate', or 'create' action verbs. We leave it to the instructional designer to use appropriate learning outcomes mapped to NICE Framework tasks that relate to the Bloom's taxonomy action verbs.
For each scenario, assessment instruments are designed and implemented to document the attainment of the learning objectives at an acceptable level, in accordance with policies of the institution/department/program. These instruments include questionnaires and a write-up describing the evolution of the compromise based on the analysis of digital artifacts. Each of the approaches is discussed in detail below, with examples.

Approach 1-Example on Network Defense
From the OPM report [4], we chose the following competency: Competency: Computer Network Defense-Knowledge of defensive measures to detect, respond, and protect information, information systems, and networks from threats.
The scenario was created using two virtual machines: one a Kali Linux attacking machine and the other a Windows 7 victim machine. A vulnerable application running on Windows 7 was targeted for buffer overflow, and the machine was compromised using weaponized code. Several post-exploitation activities were carried out. Each step of the attack, starting from reconnaissance all the way to mission completion mapping to the Lockheed Martin Cyber Kill Chain [20], was supported with artifacts. The scenario was designed to have students investigate the artifacts, analyze and recreate the attack vector, and devise an appropriate countermeasure.
Our goal was to design the scenario and scenario guiding questions that would map to knowledge, skills, and tasks, and which would satisfy the chosen competency. The following knowledge, skills, and tasks (KST) that directly map to the competency were chosen from the NICE Cybersecurity Workforce Framework. Each KST is further classified into beginner, intermediate, and advanced based on its complexity (see Table 1).

Skills
Intermediate S0054: Skill in using incident handling methodologies Advanced S0063: Skill in collecting data from a variety of cyber defense resources S0020: Skill in developing and deploying signatures S0004: Skill in analyzing network traffic capacity and performance characteristics.

Tasks Advanced
T0067: Conduct analysis of log files, evidence, and other information which would be useful to determine the best methods for identifying the perpetrator(s) of a network intrusion T0260: Analyze malicious activity to determine weaknesses exploited, exploitation methods, effects on system and information T0706: Gather information about networks through traffic analysis T0310: Assist in the construction of signatures which can be implemented on cyber defense network tools in response to new or observed threats within the network environment or enclave Table 2 shows how the scenario questions are framed to assess KSTs and competencies following scenario-based learning. In addition, suggested sample artifacts are listed to guide the development of a staged laboratory setting.

Approach 2-Example on Threat Intelligence
From the OPM report [4], we chose Threat Intelligence Analysis and Threat Analyst as the competency and work role, respectively.
This scenario was also created using two virtual machines: a Kali Linux attacking machine and a victim Windows 2016 server machine. The Windows server machine was compromised using a malicious code downloaded on the machine. Several postexploitation activities were carried out. Each step of the attack was backed up with artifacts in this scenario as well. The scenario was designed to have students investigate the artifacts, analyze and recreate the attack vector, create indicators of compromise, and package and share them using a threat sharing platform.
In this scenario, our goal was to design a scenario and scenario guiding questions that would map to the specific tasks required for the selected competency and work role, and then map to the knowledge and skills required to perform the relevant tasks. The knowledge, skills, and tasks (KST) shown in Table 3 were chosen from the NICE Cybersecurity Workforce Framework that directly map to the relevant competency.  Table 4 shows how the scenario questions are framed to assess KSTs and competencies from scenario-based learning. Suggested sample artifacts are listed to guide the development of a staged laboratory setting for this particular scenario. We discussed Clark's checklist in a previous section as a tool for determining whether scenario-based learning is the right option for cybersecurity curriculum design. After discussing our example scenarios, we outline below how the checklist applies to those scenarios.
Both scenarios impart skills and knowledge to students by providing real-world examples and artifacts that would help them recreate cyberattacks and propose mitigating solutions. Outcomes are very much skill-based and are mapped to cybersecurity competencies and tasks. The example scenarios provide students with artifacts based on real-world attacks instead of requiring them to install actual malware on their systems. The artifacts and scenario-guiding questions provide them with relevant knowledge to recreate the attacks at each stage of the cyber kill chain for appropriate decision making. In this sense, both scenarios are cost-and time-effective.

Competency Assessment Rubric
In order to properly assess competencies, a uniform rubric must first be developed and implemented. In Table 5, we apply the rubric as an assessment tool for computer network defense competency. This rubric is an adaptation of the student performance rubric developed in [21] by the Institute for the Development of Excellence in Assessment Leadership (IDEAL) for ABET. In the rubric, specific competency indicators are assessed based on four levels of competency. The assessment process is guided by the description provided for each pair of indicator-competency levels.  The levels of competency are designed to be flexible and subjective. They are bound to be guided by the personal perspective or judgement of the evaluator. As long as these metrics are consistently and uniformly applied, we believe that they are fair and useful metrics. We resist linking these levels to numeric scores to avoid being prescriptive. Instead, we use qualitative terms such as "some", "basic", "advanced", etc. Indicators of the levels of competency include "Unsatisfactory', "Developing", Satisfactory", and "Exemplary."

Preliminary Empirical Data
Preliminary evaluation data on the efficacy of the scenario-based cybersecurity learning approach were collected during an industrial control systems security course. Immediately following the course, participants completed in a post-course survey. The respondents (n = 10) overwhelmingly agreed that the scenario-based laboratory exercises were very helpful to their learning process. Anecdotal comments that were gathered included descriptive qualifiers such as " . . . reinforced some things that we have done and that we have not done.", " . . . the exercises were a lot of help", "they were really everything . . . ", "Overall the hands on were great", " . . . through the exercises I now have a more thorough understanding" and " . . . were the most enlightening portion of the class." Encouraged by these results, we will continue to use this approach and incorporate the assessment levels that were described in the previous section of this paper. However, we need to point out that this learning technique is just one component of a major project, i.e., the development of a cybersecurity competency assessment model, and should be regarded as such.

Conclusions and Future Directions
As employers are increasingly focusing on skills and competencies to assess their employees and hire new graduates, higher education institutions are facing a call to action to design curricula that impart relevant knowledge, skills, and competencies to their graduates, and to devise effective means to assess them. However, there is a lack of comprehensive measures regarding which metrics should be used to assess competencies and what constitutes an effective learning and assessment strategy to ensure that graduates obtain sufficient skills and knowledge to satisfy their employer's needs. In this paper, we have discussed how to design scenario-based learning strategies for cybersecurity courses, and have devised a method to assess competencies using scenario-guiding questions and artifacts. We have designed and delivered a number of courses that use the strategy discussed in this paper and have presented some examples of scenario designs, scenarioguiding questions, and assessment rubrics.
In this paper, we presented examples of how to assess competencies using scenariobased learning. For a given scenario, we mapped the expected knowledge, skills, and tasks (KST). Although significant empirical data are yet to be collected, what we have provided is a foundation for a pedagogical process that is intended to be broadly applied. Using this seminal work, we intend to follow through with extensive data collection and evaluation, as described in the section on future research directions. We believe that the main contributions of this paper are as follows: • a preliminary evaluation of the efficacy of a scenario-based learning approach to cybersecurity; • the construction of a competency assessment model based on existing frameworks and reports; and • the assembly of a generic and functional assessment rubric for competency evaluation.
There is much to do in this domain. A comprehensive list of competencies needs to be developed; this would benefit employers and higher education institutions in terms of effectively assessing skills and competencies at various levels. Higher education institutions need to effectively design assessment tools and techniques with which to measure skills and competencies, and work closely with industry partners to evaluate the effectiveness of those tools and techniques. It is time to ask whether traditional lectures and lab-style delivery of courses are meeting the needs of today's employers and imparting relevant skills and competencies to graduates.
Additional future directions for enhancing competency-based learning, particularly in the area of cybersecurity, include the following: • develop a dynamic and artificial intelligence-based system that provides an effective learning path that is in line with the learner's abilities; • expand the data collection and evaluations of scenario-based learning approaches and identify possible actions for continuous improvement; • design and implement digital and verifiable credentials for cybersecurity competency pathways that are industry-endorsed; and • enable an effective communication mechanism and collaborative platform wherein industry and academia can actively and constantly communicate to address the competency gaps that evolve due to rapid technological advancements.