CYRA: A Model-Driven CYber Range Assurance Platform

Featured Application

Internet of Things, Industrial Internet of Things and 5G Cyber Range.

Abstract

Digital technologies are facilitating our daily activities, and thus leading to the social transformation with the upcoming 5G communications and the Internet of Things. However, mainstream and sophisticated attacks are remaining a threat, both for individuals and organisations. Cyber Range emerges as a promising solution to effectively train people in cybersecurity aspects. A Training Programme is considered adequate only if it can adapt to the scope of the attacks they cover and if the trainees apply the learning material to the operational system. Therefore, this study introduces the model-driven CYber Range Assurance platform (CYRA). The solution allows a trainee to be trained for known and new cyber-attacks by adapting to the continuously evolving threat landscape and examines if the trainees transfer the acquired knowledge to the working environment. Furthermore, this paper presents a use case on an operational backend ICT system, showing how the CYRA platform was utilised to increase the security posture of the organisation.

1. Introduction

Cyber Ranges aims to provide advanced cybersecurity training exercises for both cybersecurity professionals and individuals regardless of their security expertise. Unfortunately, cybercriminals constantly improve their techniques and launch impactful attacks that affect both organisations and individuals, leading to an ever-changing threat  landscape [1,2]. This is exacerbated nowadays by the complexity of the modern 5G, Internet of Things (IoT), and Industrial IoT (IIoT) systems and the lack of security awareness, as users are not able to promptly identify and minimise the impact of a cyberattack and instead act as enablers for the various threat actors to successfully deploy attacks [1,3,4]. According to the UK’s 2021’s Cyber Security Breaches Survey [5], four in ten businesses (39%) and a quarter of charities (26%) report having cybersecurity breaches or attacks in the last 12 months. Even though fewer businesses are identifying breaches or attacks than in 2020 (46%), the risk level is potentially higher than ever under COVID-19, making it harder for businesses to administer cybersecurity measures during the pandemic. One of the survey’s key findings was unprepared staff risk being caught unaware. More specifically, only a total of 14% of organisations train their staff on cybersecurity, and 20% have tested their staff response through Cyber Range exercises. Further to this, Cybint [6] states that 95% of cybersecurity breaches are due to human error.

That being said, cyber-criminals and hackers will try to infiltrate an organisation through its weakest link. The need for not only more skilled cybersecurity professionals but also well-trained individuals regardless of their security expertise is ever-increasing.

Organisations devote significant efforts concerning money and time for training to enhance the personnel’s job-related competencies [7,8,9]. Relevant investments all over the world are continuously increasing, as well as the demand for advanced and effective Training Programmes. Therefore, it is now becoming mandatory for the training provider, to provide evidence that training activities are being fully realised and ensure the training leads to to designated work competencies, such as increases in the security level or the operational performance of the organisation [8,10,11].

Information security training can raise the organisation’s awareness among employees concerning the best practices for cybersecurity and privacy [11,12,13]. Although training and awareness are still critical for all organisations, they also constitute two of the most neglected areas of cybersecurity. Based on relevant researches, only 50% of companies perform some security training programme periodically [14]. Moreover, such programmes are usually encountered by the personnel and the high-level management just as another element on the checklist that has to be fulfilled [15]. Thus, there is no sufficient effort to assess whether employees benefited from a programme or not or if they are applying the learned practices and the acquired skills in their day-to-day activities.

The SANS Institute is highlighting in several of its reports the challenges that must be tackled by businesses providing security training, including programmes of support, time, and resources by people and departments in their customers’ organisations [16]. Academic researchers have also proven that the training efforts can benefit individuals and organisations only when the newly acquired competencies and skills are transferred to the working environment [7,10]. Henceforth, the necessity to analyse and understand the consequences and antecedents of this training process transfer is emerging.

Regarding cyber-security training, although there is an increasing demand for modern Cyber-Range platforms with advance technical features (e.g., emulation, simulation, serious gaming), the transfer of the learned skills and the adjustment of the organisation’s operation has been completely overlooked in almost all cases [17,18,19].

This study presents CYRA, a model-driven CYber Range Assurance Platform that allows (i) the continuous security assurance of the actual operating system, and (ii) the dynamic adaptation of the training procedures in the virtual cyber ranges environment. Initially, the organisation’s technical assets and operational aspects are evaluated via automated and semi-automated mechanisms. Then, the actual security posture is disclosed and security Key Performance Indicators (KPIs) are defined, representing the desired level of protection that we want to accomplish through training. A relevant Cyber Threat and Training Preparation (CTTP) Programme is developed and the main training is begun. Throughout this process, the installed analysis mechanisms, from the initial phase, are continuously assessing wherever the trainees are applying the learned concepts in the actual system. If not, indications are provided to the trainer and the training procedures are adapted based on fully- or semi-automated techniques. Meanwhile, the deployed mechanisms may discover new threats that were not recognised before and drive the creation of additional training content.

The proposed solution has been successfully deployed in three Information and Communications Technology (ICT) domains of smart transportation, smart energy, and healthcare. For each of these pilots, three complete Training Programmes have been created, namely: “security awareness” for staff with no or low-security knowledge, “edge system security administrator” for personnel that require main security knowledge concerning the setting and usage of edge systems, and “backend security manager” for security and privacy experts. The platform has been evaluated under real operational conditions where cyber-range training was provided to actual employees. The security KPIs were set in advance, and the trainers guided the trainees to accomplish them and adapt this knowledge back in their workplace. The main training life-cycle has been presented in our previous works [20], while this paper concentrates on the aforementioned modules for CTTP Models and Training Programmes adaptation and continuous security assurance in the operational system.

The rest of the paper is organised as follows. Section 2 overviews the related studies in this field. Section 3 describes the Cyber-Range Assurance Platform. Section 4 details the creation of training programmes and their underlying elements. Section 5 presents the proposed adaptation tool of the platform and Section 6 a demonstration example. Section 7 outlines an evaluation analysis, where external trainers utilise the CTTP models and programmes editor and assess its usability and user-friendliness. Finally, Section 8 provides the concluding remarks and pointers to future work.

2. Related Works

2.1. Adopting Training in the Workplace

One of the main concerns of noncompliance of the personnel with the defined security policies for an organisation constitutes the main concern [12,18,21]. If employees do not fully comply with these policies, the efficacy of the relevant protection mechanisms is lost. From the various compliance approaches, effective training is considered the most commonly proposed solution. Nevertheless, only a few of the existing studies evaluate the results of professional training in organisations and promote policies for compliance in the working environment [12,18]. The theory is rarely utilised to explain what factors are affecting users’ compliance with security policies or even provide empirical evidence from their actual application. In general, it is suggested that training programmes should make use of methods and contents that are actively engaging trainees and motivate them to undergo systematic cognitive processing of the taught information [17,20]. Apart from modern technological tools (e.g., simulation, serious games), the continuous communication of the trainer with the learners is vital to improving users’ security compliance [12,17,20].

Researches for security training ordinarily incorporate pedagogical principles as the primary methodology to enhance the user’s compliance [18,20,21]. In 1998, Baldwin and Ford [22] defined the transfer of learning to the working environment as “the degree to which trainees effectively apply the knowledge, skills, and attitudes gained in the training context to the job”. A simplified illustration of the training transfer model is depicted in Figure 1.

However, it is estimated that around 10–40% of all training experiences will be transferred to the workplace [7,17,18]. Moreover, as time passes from the completion of the training programme, employees usually becoming less motivated to retain the new operational behaviours (e.g., after one year). Based on the currently provided methods and training solutions, only a small percent of the desired training outcomes will be permanently transferred to the working environment. Thereupon, enhancing the learning transfer becomes the main concern for modern cybersecurity training platforms, which is also one of the main targets of this study and the proposed continuous assurance and adaptation mechanisms that are detailed in the next sections.

2.2. Cyber Ranges Platforms

Surveys for cybersecurity training in critical infrastructures, such as aviation, transportation, healthcare, energy, and nuclear sectors, are presented in [17,18,19,20]. In the last few years, there has been an increasing demand for cybersecurity professionals, which is expected to continue to grow [17]. Cyber Ranges are a promising solution for advanced training that can fill the gap by incorporating educational courses with hands-on experience.

Most of these platforms are implementing automated tools to facilitate the development of virtual labs and scenarios, as well as the adequate means to assess the  trainees [19,20]. Karlzen had identified and surveyed the automation trends of 74 cyber ranges platforms [19].

Fourteen of these platforms are deploying such automated modules. The Austrian AIT Cyber Range utilises automation for the instantiation of virtual machines (VMs) during capture-the-flag (CTF) competitions [23]. During such events, it also uses a module called GameMaker, as the scenario engine for the execution of injections [24]. NATO’s Cooperative Cyber Defence Centre of Excellence (CCDCOE) cyber ranges platform utilises the tool suite EVE and ADAM for situational awareness during exercises [25]. The CCDCOE platform can also combine an automated availability scoring module [26]. Cyber Conflict Exercise [27] marshals a series of automated tools that are needed for cyber ranges. These include components for system configuration and deployment, attack execution, updating of flags, visualisation of changes in the emulated setting, and scoring. The CyTRONE platform incorporates a module, called CyRIS (Cyber Range Instantiation System) [28], for the automated configuration and deployment of services and systems in the training environment. Then, additional modules are used for the execution of attacks [29].

The DETERLab platform deploys virtual training environments based on abstract test definitions [30] and develops the Montage AGent Infrastructure (MAGI) for the automatic run of tests [30]. The Emulab platform allocates the hardware resources, sets the networking parameters, and executes other automated events [31], while the Linktest module can validate the generated emulated elements [32]. Similarly, KYPO uses the PM Portal module to set up and control the deployed exercises [33], while additional tools perform automated attacks in the virtual system and evaluation of the trainees. The LARIAT platform includes the Automatic Live Instantiation of a Virtual Environment (ALIVE) module for the configuration and instantiation of VMs for the emulated setting [34]. This includes the emulation of standardised network services, websites, and email services, etc.

The above-mentioned findings show that the existing educational solutions for cybersecurity are mostly concerned with the training environment configuration while the transfer of the learned concepts in the workplace is neglected. In the work presented in [20], the authors introduced a platform, developed under the EU-funded THREAT-ARREST project, where the trainee has the chance to be trained to counter advanced, known, and new cyber-attacks. CYRA acts as a continuation of this work that not only trains a user against several attacks through organisation-specific or generic training programmes but it also (a) investigates if the gained knowledge was applied to increase the security posture of an actual cyber system (if the trainee was part of an organisation) and (b) adopts the existing training programmes or creates new ones by following the adaptation procedures described in Section 5.

3. The CYber Range Assurance Platform (CYRA)

CYRA (see Figure 2) is a combination of three main components namely, Sphynx’s Security Assurance Platform, the CTTP models and programmes editor and the CTTP Models and Programmes adaptation tool and was developed by Sphynx Technology Solutions AG (https://www.sphynx.ch/ (accessed on 23 April 2021)).

These components work together to provide Cyber Range training programmes that (a) can train users to understand the ever-increasing threat landscape, (b) are tailored to an organisations needs (based on the use of the Security Assurance Platform) and (c) can be adopted to upcoming cyber threats and/or changes of the assessed cyber systems. More details for the three mentioned components can be found below.

• Sphynx’s Security Assurance Platform is a suite of various tools and technologies that enables clients to realise security assessments based on industrial and international standards (e.g., cloud, network, smart metering standards) through continuous monitoring and testing. The platform’s main components are:

  -

  Asset Loader: The component responsible for receiving the cyber system’s asset model for the target organisation. This model includes the assets of the organisation, security properties for these assets, threats that may violate these properties and the security controls that protect the assets.

  -

  Monitoring Module: A runtime monitoring engine built in Java that offers an API for establishing monitoring rules to be checked. This module is made of three sub-modules: a monitor manager, a monitor, and an event collector. The role of the module is to forward the runtime events from the application’s monitored properties and finally obtain the monitoring results.

  -

  Dynamic Testing Module: The Dynamic Testing Module utilises a combination of various open-source penetration testing tools to execute different types of penetration testing assessments. The module actively interacts with a target system to discover vulnerabilities and determine if the vulnerabilities are exploitable. In addition, the module supports the uploading of a report generated from different types of tools and populates the assurance platform based on their results. Finally, as additional functionality, the module can discover and report assets that were not defined in the current asset model of the system.

  -

  Vulnerability Loader Module: The component responsible for loading the known vulnerabilities (of the identified assets) and updating the assurance platform depending on the organisation’s assets included in the assurance model.

  -

  Event Captor: Event Captor is a tool that, based on collected data and triggering events, formulates a rule or a set of rules and pushes the latter towards monitoring the module for evaluation. Data and events are mostly collected through Elastisearch (https://www.elastic.co/ (accessed on 20 April 2021)) based on lightweight shippers (namely Beats), such as Filebeat, MetricBeat, and PacketBeat, that forwards and centralise log data. Data can also be collected through Logstash, an open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to ElasticSearch. Event captor’s are initiated through REST calls from the monitoring module respectively.

• The CTTP Model and Programmes Editor is responsible for creating the CTTP Models and Training Programmes. The editor is offered as a web service through the Cyber Range Platform.
• Lastly, the CTTP Models and Programmes adaptation tool is the tool responsible for adapting the existing training programmes and models or creating new ones in response to upcoming cyber threats and/or changes of the assessed cyber systems.

4. Cyber Threat and Training (CTTP) Models and Programmes

The Cyber Range approach presented herein incorporates emulation, simulation, data fabrication, and serious gaming tools that prepare end-users in defending high-risk cyber systems and organisations to counter advanced, known, and new cyber-attacks. At the core of our model-driven Cyber Range approach (see Figure 3) is the creation of the CTTP Models. A CTTP Model specifies the potential attacks, the security controls of cyber systems against them, and the tools that may be used to assess the effectiveness of these controls. It also drives the training process and aligns it (where possible) with operational cyber system security assurance mechanisms to ensure the relevance of training. The first version of the CTTP models was described in [1]. Subsequently, the authors expanded the CTTP models by adding several attributes that would allow for a better description of the execution tools and the incorporation of security controls and re-structured the CTTP Programmes to better provide its educational scope. Having said that, the CTTP Programme structure is as follows:

Figure 3. CYRA’s UML representation.

Figure 3. CYRA’s UML representation.

To this end, a training programme (course) is made up of one or more training programme contents (thematic lecture and/or virtual lab). The content can include one or more educational materials (e.g., guidelines, handouts), which in turn can contain an evaluation report that the trainee will need to answer to be evaluated on the provided material. The content can also contain a training scenario. The latter includes the organisation’s core cyber system asset model (if the scenario was created for an organisation) and one or more CTTP models defining the tools that will be used for the realisation of the scenario. A training scenario has (at a minimum) two compulsory parts namely: (a) the training delivery parameter model and (b) the core cyber system asset model and the emulation, simulation, serious game, data fabrication models or a combination of them. If a training programme is created for an organisation, the organisation’s core cyber system asset model is also compulsory. The next subsections contain a brief description of the above-mentioned terms.

4.1. Training Programme

A training programme provides a structured learning path for a trainee, as it includes training programme content focusing on particular threats, cyber system components, and assessment tools of a CTTP model and is being used to drive (along with the CTTP Models) the execution of the simulation, emulation, and serious gaming processes.

A training programme includes a number of parameters (required or optional) used to serve different purposes. The basic information of a Programme is: (a) its name, (b) an introduction that allows the trainee to understand the scope of the programme, (c) its purposes (based on ECSO’s taxonomy [35], (d) its vertical sectors (based on ECSO’s Domains), (e) its targeted users (based on NIST NICE Work Roles (https://www.nist.gov/itl/applied-cybersecurity/nice/nice-framework-resource-center (accessed on 20 April 2021))), (f) its offering level (based on Bloom’s Taxonomy [36], and (g) its duration. Then, each training programme includes a trainee profile and learning pathway section where the: (a) roles per targeted users, (b) prerequisites, (c) education scope, and (d) mapping against training schemes (such as ISACA (https://www.isaca.org/ (accessed on 20 April 2021)), SANS (https://www.sans.org/ (accessed on 20 April 2021)), CompTIA (https://www.comptia.org/home (accessed on 20 April 2021)), (ISC)² (https://www.isc2.org/ (accessed on 20 April 2021))) are identified. Lastly, a training programme can be assigned to one or more organisations.

4.2. Training Programme Content

Training programme content is part of one training programme and includes one or more types of educational material and zero or one training scenario. The basic information of a piece of content allows the user to understand its scope such as, the content’s (a) name, (b) offering level (based on CISCO’s certification schemes), ranging from easy to hard difficulty levels, also known as foundation, practitioner, intermediate, and expert), and (c) duration. Both (b) and (c) are subsets of the corresponding training programme’s fields. For instance, a training programme of the “intermediate” offering level cannot include contents of the “expert” offering level.

4.3. Educational Material

Educational material is a subset of guidelines and brochures, etc., which allows the trainee to better understand the substance of the training programme content. It has the following attributes: (a) a name, (b) a difficulty level (also based on CISCO certification schemes), and (c) a URL that points to the online reference of the educational material. Lastly, educational material can have an online evaluation report (i.e., questionnaire with closed-ended questions) that will be used to automatically evaluate the trainee.

4.4. Training Scenario

A training scenario is made up of zero or one core cyber system asset model (i.e., the model that contains the different assets of an organisation as well as their relations) and one or more CTTP models, and it is used to set up the virtual environment that will be used for a specific training programme.

4.5. CTTP Models

CTTP models will determine: (a) the cyber system components and cyber threats covered by a CTTP programme, (b) the ways of simulating components of a cyber-system and the cyber-attacks against it, (c) the components of the system that may be emulated and the ways of emulating them, and (d) the real system operational events that should be monitored and analysed to assess the operational security status of a cyber-system in real-time. The full specification of the CTTP Models is available in [1]. The next subsections include updates that occurred after writing the above-mentioned paper.

4.5.1. Emulation Model

An emulation model includes information for the automated generation and interconnection of emulated cyber system components and is intended to be dynamically parsed by an emulation tool (e.g., virtual infrastructure management solutions based on OpenStack (https://www.openstack.org/ (accessed on 23 April 2021)) or Kubernetes  (https://kubernetes.io/ (accessed on 23 April 2021)). The updates that occurred to the emulation model included the specification of two new attributes, namely (a) actual trace, an attribute that allows an emulation tool to know the structure of an action that the trainee is expected to perform, and (b) a Boolean value called linkAvailable that allows an emulation tool to understand whether a deployed VM should be visible to the trainee or not.

4.5.2. Simulation Model

A simulation model includes information for the simulation of different layers in the cyber systems implementation stack and is intended to be dynamically parsed by simulation components (e.g., NS-3 https://www.nsnam.org/ (accessed on 23 April 2021)). Unlike the emulation sub-model, the simulation one underwent a number of structural updates. More specifically, the updated simulation model structure is as follows:

That being said, a simulation model has the following attributes: (a) a name, (b) the involved tool’s name, (c) a template (optional), (d) the simulation duration (in minutes), (e) the execution speed, (f) a random seed, (g) the initial simulation time, and (h) the deployment mode. Moreover, from a structural perspective, a simulation model consists of a root component (with a unique name and type), which, in turn, consists of some sub-components or attributes.

4.5.3. Serious Game

A serious game model includes information about the following two types of games: PROTECT [37] or Awareness Quiz [38]. Updates occurred only on the creation of the latter. An Awareness Quiz Serious Game model includes the following attributes: (a) the quiz mode, (b) the available lives, (c) the question time, (d) the points added to the score, (e) the points removed from the score, and (f) specific attributes corresponding to the respective quiz mode. If a game shall be played with a predefined quiz (predefined quiz mode), a specific attribute represents the unique identifier of the quiz. If the set of questions for a quiz shall be compiled on the fly for a certain thematic context (context quiz mode), the following specific attributes are included: (a) the number of questions that shall be asked, (b) a minimum difficulty for the considered questions, (c) a maximum difficulty for the considered questions, (d) one or more quiz topics, on the basis of which the thematic selection of relevant questions takes place. A quiz topic is represented by metadata values of a corresponding metadata type.

4.5.4. Data Fabrication

A data fabrication model includes information that will be used for the creation of synthetic events and is intended to be provided to data fabrication tools, e.g., the IBM Data Fabrication Platform developed by IBM Israel [39]. The data fabrication model did not undergo any substantial updates. Its core component is the scenario, which consists of one or more actions/activities, one or more constraints, one or more sub-nets, one or more connections, and one or more applications.

4.5.5. Training Delivery Parameter Model

The training delivery parameter (TDPM) model (previously mentioned as the training programme) is the “orchestrator” of the above-mentioned models.

Content-wise, the updated TDPM consists of the following attributes: (a) a name, (b) a programme type (e.g., single user, red/blue team, etc.), (c) a difficulty (scaling from 1 to 20), (d) a bonus (added to the trainee’s final score if he/she successfully finalised the training earlier than expected), (e) a penalty (subtracted from the trainee’s final score if he/she failed to finalise the programme within the time), (f) the targeted scenario actor(s), (g) a flag value called “follow sequence” that allows the training tool to identify whether the TDPM’s executions need to follow a specific order, (h) the linked vulnerabilities (based on CVE’s) and weaknesses (based on CWE’s), (i) the prerequisites, (j) the mitigation controls, (k) the background legal context, and (l) the involved assets.

It also contains the scenario’s storyline and course of actions as well as the duration. Moreover, it contains the training standards it was based on (e.g., CSA (https://cloudsecurityalliance.org/ (accessed on 20 April 2021)), ISO (https://www.iso.org/home.html (accessed on 20 April 2021)), NIST (https://www.nist.gov/ (accessed on 20 April 2021)), etc.), the action types (based on several frameworks such as MITRE ATT&CK (https://attack.mitre.org/ (accessed on 20 April 2021)), the Cyber Kill Chain  (https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html (accessed on 20 April 2021)), and NIST’s Incident Response [40]), and the targeted roles (based on the NIST NICE (https://www.nist.gov/itl/applied-cybersecurity/nice (accessed on 20 April 2021)) and e-CF Framework (https://www.ecompetences.eu/ (accessed on 20 April 2021))).

Lastly, from a structural point of view, a TDPM includes one or more pieces of execution information. Each piece of execution information can include one or more screens (that will be presented to the trainee). A screen includes the tool that will be presented to the trainee, whether it will be visible (or not), the targeted role (for instance, if a red/blue team TDPM is created, one screen can be presented to the blue team members whereas another to the red/blue team ones), a brief description of a screen, a hint and its impact (if enabled), the max allowed time a trainee could spend on this screen and a flag value that terminates the whole training session, if the trainee fails to successfully finalise the content of the screen within the requested time limit. Lastly, a screen includes one or more expected traces.

An expected trace can be: (a) an evaluation report that the trainee is asked to fill after he/she finalises the tasks presented to the screen, (b) an event captor, which allows the training tool to understand if the expected actions were a result of the trainee, and (c) a serious game report.

5. Cyber Threat and Training Models and Programmes Adaptation Tool

The CTTP models and programmes adaptation tool covers primary forms of analysis of the impact that specific changes in some parts of the programme have on other parts and checks about the completeness (coverage) and consistency of the entire specification of CTTP models and programmes when some parts of the change. CTTP programme adaptation may be along: (a) the scope of the attacks that they cover (e.g., component specific attacks, system-wide attacks); (b) the sophistication of the attacks (e.g., attacks that explicitly or implicitly affect the asset that they compromise); and (c) the type of response expected from the trainee (e.g., preparedness, analysis, immediate incident response, post-incident response). The general principle underpinning the adaptation of CTTP programmes will be to vary the degree of difficulty that the programme presents to the trainee (e.g., going from component specific to system wide attacks, from asset explicit to asset implicit compromising attacks, from preparedness to analysis). Such levels and corresponding adaptations will depend on the type of system, attack, asset, and security property and will be specified explicitly in the CTTP programme. CTTP programme adaptation is being driven by trainee performance (e.g., increasing/maintaining/reducing the difficulty of a programme if trainees meet/fail (marginally)/fail current action expectations). Furthermore, CTTP programmes will be improved based on a continuous process involving including: (a) automatically recorded performance measures regarding the undertaking of the CTTP programme (e.g., programme completion time) and (b) assessing the level of compliance of these actions to expectations set by the cyber system asset model. The CTTP model and programmes adaptation tool is offered as a web application.

In general, the CTTP models and programmes adaptation is separated into three phases. The first phase, namely, pre-evaluation, includes the training programme that was solely based on a cyber system asset model. Thus, they are based on a theoretical background (asset model) where there was no interaction with an actual cyber system. The second phase, namely, pre-adaptation, includes creating new training programmes based on the security assessments conducted in a cyber systems through Sphynx’s Security Assurance Platform (SAP) (https://www.sphynx.ch/products/#assurance-platform (accessed on 18 April 2021)) or the updates of existing ones. Lastly, the third and final phase, namely, post adaptation, includes the adaptation of the phase two programmes based on changes in the asset model (e.g., insertion of a new asset or security control), the introduction of new vulnerabilities as well as changes on an existing vulnerability’s likelihood (i.e., a likelihood’s shift from LOW to HIGH) and finally, the adaptation based on a trainee’s performance.

5.1. Adaptation Based on Sphynx’s Security Assurance Platform

As above-mentioned, a CTTP models and programmes adaptation will be based on the utilisation of Sphynx’s Security Assurance Platform. Four prominent cases that are based on the outcomes of the platform and trigger adaptation actions were identified, (a) when a new security control is inserted to the system, i.e., “a safeguard or countermeasure prescribed for an information system or an organisation designed to protect the confidentiality, integrity, and availability of its information and to meet a set of defined security requirements”, (b) when a vulnerability is detected on an asset that was not vulnerable before (based on SAP’s Vulnerability Loader Component), (c) when a new vulnerability is detected on an existing asset, and (d) when the likelihood of a vulnerability changes from low to high, making it a higher priority training target.

5.1.1. Adaptation Based on Security Controls

An adaptation is triggered when a new security control asset is detected (see Algorithm 1). The condition for this adaptation is checked every time a new asset is inserted (by using SAP’s Asset Loader). If the condition is true, the adaptation tool searches for existing training delivery parameters models that involve the said security control to identify if training programmes that take into consideration if such controls exist. If such models are found, it generates an alert to notify the trainer of their existence. The generated notification also includes information for the training programme content that is assigned (if any), so that it can be assigned to a trainee (via the training tool). Otherwise, the adaptation tool first creates the training delivery parameter model and then creates an alert prompting the trainer to assign to a training programme content. If the condition is false, no adaptation actions are produced.

Algorithm 1: Adaptation based on security controls

5.1.2. Adaptation Based on Vulnerable Assets

New Vulnerabilities on Existing Assets

This adaptation action is triggered when a result of the dynamic testing or the vulnerability assessment module detects vulnerabilities on an asset that was not affected before (see Algorithm 2). This adaptation’s condition is checked by the adaptation tool every time a new result is generated. If the condition is true, the adaptation tool searches for training delivery parameters models that involve the said asset. If such models are found, it generates an alert to notify the trainer of their existence. The generated notification also includes information for the training programme content that is assigned (if any), so that it can be assigned to a trainee (via the training tool). Otherwise, the adaptation tool first creates the training delivery parameter model and then creates an alert prompting the trainer to assign to a training programme content. If the condition is false, no adaptation actions are produced.

Algorithm 2: Adaptation based on existing asset vulnerabilities

New Vulnerable Asset

This adaptation action is triggered when a new asset is inserted (through the cyber system’s asset model) in the assurance tool’s database and the dynamic testing or vulnerabilities assessment component identifies vulnerabilities on it (see Algorithm 3). If the condition is true, the adaptation tool searches for training delivery parameters models that involve the said asset. If such models are found, an alert is generated to notify the trainer of their existence and includes information for the training programme content that is assigned (if any), so that it can be assigned to a trainee. Otherwise, the adaptation tool first creates the training delivery parameter model and then creates an alert prompting the trainer to assign to a training programme content. If the condition is false, no adaptation actions are produced.

Algorithm 3: Adaptation based on a newly inserted vulnerable asset

Increased Likelihood of an Existing Vulnerability

This adaptation action is triggered when a result from the dynamic testing or the vulnerability assessment indicates that a previously discovered vulnerability has changed its likelihood level from low to high (see Algorithm 4). If the action is true, the adaptation tool searches for training delivery parameter models that are linked to the said vulnerability. If such models are found, it generates an alert to notify the trainer of their existence. The generated alert also includes information for the training programme content that is assigned (if any), so that it can be assigned to a trainee. Otherwise, the adaptation tool first creates the training delivery parameter model and then creates an alert prompting the trainer to assign to a training programme content. If the condition is false, no adaptation actions are produced.

Algorithm 4: Adaptation based on a likelihood alteration

5.1.3. Adaptation Based on Trainee’s Performance

The adaptation based on the trainee’s performance is triggered when a trainee completes an evaluation report or finalises a training programme’s content. By doing that, a message is being generated and published to a message broker’s predefined channel that the adaptation tool is listening to in order to initiate the adaptation phase. The message includes information for the completed component (evaluation report or content), the trainee’s expertise, the completion time, and the final score (normalised in 100).

Completion of an Educational Material’s Evaluation Report

Upon the completion of an evaluation report (of an existing educational material), the adaptation tool consumes the correspondent message and initiates the adaptation phase. More specifically, the adaptation tool checks the evaluation report’s difficulty (available in the assurance tool’s database) and the trainee’s expertise (provided in the consumed message) and, based on the algorithm presented in Algorithm 5, it produces an adaptation alert. In general, the adaptation tool can either be used as an alert mechanism where it will notify the trainer of the need to create a new evaluation report or assign an existing evaluation report with different difficulty.

Algorithm 5: Adaptation based on the evaluation report’s score

Completion of a Training Programme Content

Upon the completion of a training programme content, the adaptation tool consumes the correspondent message and initiates the adaptation phase. In general, a training programme content’s session can last for a predefined amount of time and is of particular difficulty. That being said, the adaptation tool checks the completion time, success score, and trainees’ expertise as received from the consumed message and the content’s difficulty as stored in the assurance tool’s database, and based on the algorithm presented in Algorithm 6 it produces the adaptation results. As before, the adaptation tool can either be used as an alert mechanism where it will notify the trainer of the need to create a new evaluation report or assign an existing evaluation report with different difficulty.

Algorithm 6: Adaptation based on the completion time and score of a Training

5.2. Demonstrator

5.2.1. Adaptation Based on Security Controls and New Threats

This subsection provides a demonstration of the “existing asset vulnerability” (see Algorithm 5) adaptation procedures. For the former (see Figure 4), the initial step was to execute a dynamic testing assessment based on a cyber system asset model deployed in the cyber range platform. The assessment results (produced by the execution of the dynamic testing assessment), indicated that several new vulnerabilities were identified for the Apache Tomcat. Based on these results, the adaptation tool checked if a training delivery parameter model with the newly identified vulnerabilities exist. As it did not find one, it proceeded with the automatic creation of a new training delivery parameter model adaptation and pre-populated with a number of parameters such as the involved assets, the CVE and CWE (based on the adaptation scenario) and a few additional parameters including the model’s storyline, the difficulty, the duration, and the scenario actors.

5.2.2. Adaptation Based on Trainee’s Performance

Next, the CTTP models and programmes adaptation tool received a notification that a trainee of “practitioner” expertise finalised the existing “password management” educational material’s evaluation report (of “practitioner” offering level). The trainee’s final score (45) was below the expected score range (greater than or equal to 50). Given that both the trainee and the evaluation report was of “practitioner” level, the CTTP models and programmes adaptation tool automatically created a new evaluation report of “foundation” level, assigned to a new educational material and notified the trainer of the need to re-assess the trainee based on the newly created educational material (see Figure 5).

The above-mentioned procedures were also part of the CTTP models and programmes adaptation tool executed for the smart home use case described in Section 6.

6. The IoT-enabled Smart Home Use Case

6.1. Outline

For the evaluation of CYRA, a complete training programme for an operational backend ICT system was performed. The organisation that took part in the evaluation offers a smart energy solution, where energy is collected in smart homes with solar panels, and the energy distribution is recorded and administrated by the backend system at the company’s premises [41]. This backend system (the cyber system that CYRA was evaluated) is composed of a set of working and backup servers that host the main organisation’s applications.

At the pre-set phase, the system’s architecture was recorded via short interviews with the system’s users and operators. Thereupon, the main digital assets were identified and were uploaded to CYRA through the cyber system asset loader (see Figure 6). Next, an initial vulnerability assessment was executed (through the vulnerabilities loader component) to understand the basic vulnerabilities that the assessed cyber system has. In general, the vulnerabilities loader searches for known vulnerabilities of the asset’s uploaded through the cyber system asset loader obtained from relevant repositories (i.e., NIST’s National Vulnerability Database (NVD) (nvd.nist.gov (accessed on 22 May 2021)). The Common Vulnerability Scoring System (CVSS) (www.first.org/cvss (accessed on 22 May 2021)) of each identified Common Vulnerabilities and Exposures (CVE) record (cve.mitre.org (accessed on 22 May 2021)) was then normalised (in CYRA) in the range of 0–100. Next, three dynamic testings (i.e., through the Dynamic Testing Tool) were performed, disclosing automatically the exact technical details of the running system (e.g., database version) and further discovering services, applications, and configurations of the system that the interviewers were not aware of (i.e., a running instance of telnet on one of the servers). This was the first step of the supported adaptation approaches.

The results from the above-mentioned sources were then prioritised based on their severity and impact as low, medium, high (as described in the previous section). In total, 40 assets were defined for the smart energy ecosystem and 100 CVEs were identified.

Next, the threat landscape for the specific sector and organisation was revisited and a set of the most severe vulnerabilities was concluded. As 10 of the identified vulnerabilities were of HIGH impact and severity, the training was concentrated on them (see this list in Figure 7).

To monitor for potential exploitation of these vulnerabilities, event captors were deployed in the system in order to capture events that could lead to a detection of the violation of the security property these 10 vulnerabilities might violate as well as monitor the real-time behaviour of the systems’ users, as such elements are not disclosed by the aforementioned automated security analysis tools. Generally, the captors continuously gather events for specific and measurable security aspects, including alerts from the anti-virus/anti-malware software or the warnings from the web browser, as well as occurrence frequency of specific operations in the system’s log-files (e.g., how often a user changed its access credentials). Based on these events, the monitoring tool (part of Sphynx’s Security Assurance Platform), gathers the information and continuously evaluates individual users and the overall organisation’s posture. Specific KPIs are defined for the monitored security properties along with the thresholds that we want to satisfy via training. For example, the organisation has set a security policy, where for organisation of critical services, the employees have to change their password every month. An event captor gathers the events of the relevant application log-files and estimates the password update frequency.

Based on the above-mentioned findings, the learning path of a complete training programme was defined in order to cover all these security-related KPIs. The training programme is called “Backend Security Manager” and consists of seven types of training programme content, as detailed in

Table 1. Backend security manager training programme.

Content	CTTP Models	Covered Threats/Vulnerabilities and Defence Techniques
Introduction to Cyber Security	Main lecture and educational material	Introductory knowledge of general security-related issues and the aspects of confidentiality, integrity, availability, authentication, authorisation, non-repudiation, and privacy.
Password management	-Main lecture and educational material -Emulation of a virtual lab for the installation and utilisation of a password manager (i.e., KeePass)	Password management issues, including the problems from weak or easily guessed ones and password cracking, as well as aspects of creating strong passwords, password ageing and update policies, and practical and user-convenient practices for password maintenance.
Phishing and social engineering	-Main lecture and educational material -Emulation of a virtual lab with an email phishing scenario and the use of OpenPGP software (Kleopatra) -Serious game for targeted social engineering on system administrators with the PROTECT game [37]	The everlasting effects of social-engineering with a focus on phishing attacks, as well as email security authentication, integrity, and confidentiality.
Malicious software and patching	-Main lecture and educational material -Emulation of a virtual lab for malicious software analysis and the examination of malware samples with relevant static binary analysis tools (e.g., PEView, EDx, TUBS library analysis [42], etc.)	Software flaws, vulnerabilities, and attacks, as well as relevant protection mechanisms, the need for regular patching, and analysis tools of malicious software and attacker’s tactics.
Network monitoring and security	-Main lecture and educational material -Emulation of a virtual lab for secure network configuration and monitoring of networking traffic with the tool Head(er) Hunter [43]	Networking manipulation and attacks, as well as secure configuration, administration, and operation, with a focus on continuous traffic monitoring and classification.
Digital Forensics (CTF)	-Main lecture and educational material -Emulation of a virtual lab for the examination of a performed attack on the server (i.e., data breach, crypto-mining, or ransomware based on the activated CTTP model) and the utilisation of forensics software (e.g., glogg, volatility, Wireshark, Nmap, etc.)	Advance attacker’s tactics and familiarisation with actual malware and emulated attacks, including tools and methodologies to conduct a digital forensics analysis as a first-responder to security-related incidents.
Attacks on the backend system (Red/Blue team scenario)	-Main lecture and educational material -Emulation of a virtual lab with a vulnerable server that the blue team has to defend (the populated vulnerabilities are defined in the selected CTTP model and are correlated with the defined KPIs) -The red team can be either performed by the trainees via a similar emulated setting or can be completely simulated and triggered automatically or activated manually by the trainer	Acquire advance skills and hands-on experience as the defender and the attacker of a vulnerable system, including tactics and tools to discover the underlying threats and fix them or exploit them, respectively. The red team training involves ethical hacking perspectives and the main benefit for the trainee is to better understand the impacts of attacks on an insecure or poorly safeguarded system and learn how to block or mitigate them in the real working environment.

.

Following the assignment of the above-mentioned training programme to multiple trainees, their performance was continuously evaluated by the training evaluator module [20], while CYRA’s (a) security assurance platform and (b) CTTP models and programmes adaptation tool, continuously evaluated the trainees’ compliance on the operational system and the potential effects on the KPIs’ status, as well as provided adaptation alerts based on the overall trainee’s performance.

6.2. Training and Compliance

Based on our first iterations of the overall security monitoring and adaptation of training for the accomplishment of the designed KPIs, the authors observed similar trainees’ behaviours as in the literature (see Section 2.1). For instance, the fact that a trainee completed the ”password management” content and successfully covered the learnt material, does not mean that he/she is going to adopt the working behaviour and update the password-related habits right away [17,18].

On the other hand, it could take several weeks or even months for the event captors to capture events regarding password updates that should have been performed by the trainees for organisation-critical services (as presented on the learnt password policy). Moreover, some of the trainees never updated their passwords within the monitoring period [12,18].

Therefore, two elements drastically improved the compliance factor, as well as the educational aspects [20]; the CTTP models and programmes adaptation tool and the so-called “discussion sessions”. Such a session usually occurs at the beginning and the end of each training programme and the end of each content (lesson). Within the first session, the trainer summarises the baseline security assessment results. There, it is made clear to the trainees that the goal of the programme is not only the training part but the achievement of a specific improvement for each monitored KPI. The trainees are becoming part of the organisation’s security culture and are becoming aware that their compliance will be assessed throughout the process. Thereupon, the training starts and everyone is familiar with the long-term view. Training evaluation is not the main factor as in a professional certification exam. Here, the educational and compliance aspects are essential. Thus, after each content, the trainer analyses its main concepts and discusses with trainees how they encountered these issues and what problems they faced. At the end of these sessions, any potential misconceptions and open issues are resolved and everyone is aware of the next steps that must be taken. The final discussions are performed at the end of the training, where the current KPIs’ status is assessed and any potential subsequent actions are settled.

As for the CTTP models and programmes adaptation tool, it played an important role in the overall compliance of the learnt materials as, by continuously adapting the educational material’s evaluation reports (either by automatically creating new ones or updating the existing ones) based on the trainee’s performance, helped the trainee to better understand the content described inside the materials, and then apply what he/she learnt, to the examined ICT system.

Concluding, by examining the above-mentioned steps and procedures, one can understand that CYRA incorporates a series of technical and procedural mechanisms that not only helps to train a user but also examines whether the learnt material was applied to an operational cyber system. As for the presented use case, the authors reached the defined KPIs thresholds and the desired compliance level within the monitoring period. Moreover, all 10 severe CVEs were successfully resolved. This period included two weeks for the baseline analysis, four weeks for training (3 h of training per week for each trainee), and two weeks for post-training evaluation and overall feedback.

7. Evaluation Results

7.1. Platform Statistics

CYRA is currently deployed on a virtual machine with its hardware specifications presented in

Table 2. CYRA’s hardware requirements.

No of. vCPUs	RAM	Storage (Gb)	Bandwidth (Mbps)
8	32	120	50

. Although not a priority, the authors intend to create a more lightweight and portable version of the platform before its final release.

To provide meaningful statistics of CYRA’s performance, the authors used Kibana (https://www.elastic.co/kibana (accessed on 26 May 2021)) and Apache JMeter (http://jmeter.apache.org/ (accessed on 26 May 2021)). The former was used to continuously monitor the platform’s performance in terms of resource usage, while the latter was used to test the performance of the exposed REST APIs.

As depicted in Figure 8, CYRA’s mean CPU usage while the platform is at a normal use (e.g., creation and retrieval of new CTTP models and training programmes) is only at 3%, whereas its memory usage is equal to 24.9%. When the CTTP Models and Programmes adaptation tool and Sphynx’s Security Assurance Platform were utilised, one can observe that the mean CPU usage was increased to 30.5% while the memory usage remained at the same levels (see Figure 9). In general, CYRA’s CPU and memory usage was observed to increase to around 70% only when dynamic testing and monitoring assessments were executed simultaneously.

CYRA’s availability (excluding planned downtime) for the monitored period is at 99.99% (see Figure 10) with around four pings per minute (see Figure 11).

As for the user’s interaction with the platform, it is observed that CYRA receives around 2000 hits every roughly 15 min (see Figure 12).

Lastly, JMeter is used to measure the elapsed time per CYRA’s API. JMeter’s Glossary [44] states that it measures the “load time” as the difference between the time when the request was sent and the time once a response has been completely received and the “latency” as the time from just before sending the request to just after the first response has been received. For the sake of brevity, only two REST API metrics are presented. The first one is responsible for retrieving all the training programmes, while the latter is responsible for retrieving all the training scenarios.

As presented in Figure 13, the load time to fetch 547,689 bytes of data for the training programme’s REST API is 1410 ms.

Lastly, as for the training scenarios, the load time to fetch 325,846 bytes of data for the Training Scenario’s REST API is 1222 ms (see Figure 14).

7.2. CTTP Model Editor Evaluation

At the time of writing, 12 external trainers and 20 trainee’s of different sectors (e.g., healthcare, maritime, IoT and 5G) and expertise (foundation, practitioner, intermediate, expert) evaluated the CTTP models and programmes editor (part of the cyber range assurance platform). Only the trainers were asked to submit an evaluation report, as they were responsible for presenting the CTTP Models and programmes editor to the trainee’s, and incorporate their comments into the final report. Most of the users (10 out of 12) have a technical background, while the rest are general IT personnel (see Figure 15a). Five of the participants considered themselves of “practitioner” level in terms of their cyber-security skills, knowledge and abilities, one as “foundation”, four as “intermediate” and two as “expert” (see Figure 15b).

The four levels of expertise are:

• Foundation: The trainer has a basic security knowledge and no experience in creating cyber range programmes.
• Practitioner: The trainer has an advanced security knowledge and limited experience in creating cyber range programmes.
• Intermediate: The trainer is a security expert with limited expertise in creating cyber range programmes.
• Expert: The trainer is a security expert with high expertise in creating cyber range programmes.

The participants were asked a total of 14 questions in order to evaluate the usefulness of the CTTP models, programmes and the editor. As shown in Figure 16, most of the participants found the CTTP models and programmes editor both easily accessible and user friendly.

Next, the participants were asked about the virtual tour feature of CTTP models and programmes editor as well as the editor’s waiting times. The former was created to facilitate a user on the creation of the CTTP models and programmes. This was deemed necessary as elsewhere, the user would need to go through several documents to be in a position to create CTTP models and programmes. As shown in Figure 17a, all of the participants either agreed (4 out of 12) or strongly agreed (8 out of 12) that this feature is useful. As for the waiting times, all the participants either agreed or strongly agreed that the waiting times were acceptable (see Figure 17b). In general, it took an experienced participant around 8 min to fully define a training programme.

Following, the participants were asked about the scope of each CTTP model and the difficulty of creating a training programme through the CTTP model editor (see Figure 18). For the former, most of the participants (10 out of 12) either agreed or strongly agreed that the scope of CTTP models is well defined, while 2 out of 12 kept a neutral stance (see Figure 18a). As for the latter, 11 out of 12 found it either easy or normal to create a training programme, whereas one participant found it difficult (see Figure 18b).

Succeeding, the participants were asked if the CTTP model and programmes editor can sufficiently model (a) existing working system’s security aspects (see Figure 19a), (b) professional training programmes’ educational scope ((see Figure 19b) and (c) the typical application environment in the participants’ domain (see Figure 19c). The participants seemed to form a common view for all three questions as they either agreed or strongly agreed that the CTTP model and programmes editor has sufficient modelling capabilities.

Lastly, almost all the participants were extremely satisfied with their interaction with the CTTP models and programmes editor (see Figure 20).

8. Conclusions and Future Work

This paper presented our novel model-driven Cyber Range approach by introducing CYRA, a Cyber Range assurance platform that includes: (a) the CTTP models and programmes editor, a tool that supports the specification of CTTP models and programmes that are at the core of the platform’s model-driven cyber range training approach, (b) Sphynx’s security assurance platform that enables customised and continuous assessment of the security and privacy of a cyber system and comprehensive risk management and (c) the CTTP models and programmes adaptation tool, a tool that covers primary forms of analysis of the impact that specific changes in some parts of the programme have on other parts and checks about the completeness (coverage) and consistency of the entire specification of CTTP models and programmes when some parts of it change.

In addition to the above, the manuscript included a validation of CYRA in an operational backend ICT system of a THREAT-ARREST pilot. The results presented in the use case shows the organisation increased its security posture by utilising (a) the CTTP models and programmes editor to create organisation-specific training programmes, (b) Sphynx’s security assurance platform to enable customised and continuous assessment of the organisation’s backend ICT system and (c) the CTTP models and programmes adaptation tool to adapt to the newly identified threats.

In general, the overarching objective of these efforts is to create a platform that would allow an organisation to increase its security posture or maintain it at the highest possible level by: (a) creating training programmes that would enable its personnel to understand the continually intensifying threat landscape regardless of the personnel’s security background, (b) continually assess the security and privacy of the organisation’s cyber systems and (c) adopt to the threat landscape and trainee’s skills by creating new training programmes or modifying existing ones.

In terms of the next steps, efforts will focus on (a) extending the CTTP models to support the specification of models for a wider range of emulation and simulation tools, (b) extending the CTTP models and programmes adaptation tool to support multiple forms of adaptation, including the introduction of machine learning algorithms that will allow the tool to proactively adapt to the ever-increasing threat landscape and (c) defining adaptation procedures based on user and entity behavioural analytics (UEBA). The authors plan to provide a number of training programmes in an open repository to which interested parties can refer and retrieve the programmes of interest.

Finally, the authors aim to assess the usability of CYRA by extending the evaluation invitations to the whole THREAT-ARREST consortium and its personnel after the final release of the platform, as well as external parties.