A Review of Cyber-Ranges and Test-Beds: Current and Future Trends
2.2. Aim and Objectives
- Survey recent CRs and TBs research.
- Analyse and review state-of-the-art trends, scenarios and applications.
- Establish a taxonomy for future CRs/TBs.
- Equip cyber-security professionals with the relevant skills to combat cyber-threats in next-generation highly inter-connected, multi-domain infrastructures.
2.3. CRs and TBs Survey
2.3.1. Classification and Research Criteria
- “Cyber-Ranges” + (“Military” + “Defense” + “Intelligence”) or (“Industry” + “Commercial”) or (“Education” + “Research”)
- “Test-Beds” + (“IoT” or “Smart Grid” or “Cloud”) + cyber
2.3.2. Selection Criteria
2.3.3. Extraction Criteria And Results
3. Related Work
4. Systematic Review
4.1. Cyber-Ranges and Test-Beds
- Application Domains: A total of 44 CRs were categorised, including the CRs surveyed by [16,19]. Figure 4 shows that CRs have been predominantly used for academic purposes in education and research at 31%. The result differs from that of Davis and Magrath  of 2013, where the predominant use of CRs was in the training for cyber-security, a paradigm shift in the main application. The trend is also consistent with the findings presented in Section 2; that the bulk of CR papers were published by the academic community reporting on applications in teaching, learning, and research; followed by Enterprises and Commercial organisations, as well as, Military Defence and Intelligence for training purposes such as cyber-defence preparedness both at 24% respectively. The use in of CRs in Government was at 15% rate, while other application areas such as Law Enforcement, Service Providers, and Open-Source constitute only 2% of the manuscripts surveyed.
- Types:Figure 5 shows that public and federated CRs are predominant in use at 30% respectively, private at 24%, a combination of Public-Private at 11%, a combination of Federated-Public-Private at 3% and Federated-Public at 2%. A link between the cyber-security preparedness application with the type of technology is evident. The predominant domain of application is for academic purposes and the institutions that provide education and research are mostly public with international collaborative perspectives, thereby suggesting an inter-relationship.
- Team Formation: Team formations are central to training through exercises emulating operations. Teams are formed depending on the type of exercise; (1) Red team acts as adversaries by launching attacks on the network system; (2) Blue team is responsible for defending against an adversary attack; (3) White team for administrative management; (4) Purple team sets objectives for offensive and defensive strategies; (5) Green team is responsible for maintaining network efficiency; (6) Grey team conducts non-malicious activity; and (7) Yellow team acts as a motivator during each exercise. From the survey Red-Blue team formation is most prominent at 67%, an indication that many CRs are dedicated to cyber-attack and defence training and exercises, followed by Red-Blue-Grey teams at 9% and others such as Red-Blue-Green, Red-Blue-Green-White-Yellow and Red-Blue-Purple with 8% each as shown in Figure 6. The training of teams on operational environments is restrictive in the scope of threat conditions that can be established as it compromises business continuity.
- Methods of Experimentation:Figure 7 highlights that simulation is the most common implementation methodology at 60%, followed by emulation at 38%, overlay at 8%, and finally live scenario demonstrations at 4%.
4.2.1. Core Technologies
4.2.2. Infrastructure Technologies
4.2.3. Front-End Technologies
5. Scenarios and Applications
5.1. CRs and TBs Scenarios
Scenario Design, Validation and Deployment
- Design: The definition of functional and non-functional as well as user and team-related requirements are essential pre-requisites in the design of a CR scenario. While the functional requirements pertain to the services the system provides, non-functional requirements describe how the system reacts to inputs and its dynamic responses. The team-related requirements are the tools and resources inherent within the exercise for use by teams . Furthermore, ab initio a set of attack trees based on an understanding of how an attacker can gain access to the domain under study is imperative to an effective attack scenario design. Thus a comprehensive vulnerability assessment must be established, and coupled with the impact scenarios, are combined produce a set of attack trees, the foundation for establishing a representative real-life breach condition and in turn enabling an evaluation of the optimum countermeasures to arrest the attack [45,57].
- Validation: Russo et al.  report on a framework for automating model validation of scenarios through a Scenario Definition Language (SDL) on the OASIS Topology and Orchestration Specification for Cloud Application (TOSCA) . SDL/TOSCA based implementations automate the validation of the scenario against specified design errors, such as incorrect hardware/software bindings. The approach translates a SDL design into a Data Log specification, before verifying if the specification satisfies the goals of the scenario. A design modification is triggered whenever the validation fails, otherwise the scenario is automatically deployed. While developed for CR applications, the solution is also applicable to TBs but is dependent on the domain of study, most relevant in attack scenarios in targeting Control Centres (Figure 14).
- Deployment: A number of other approaches to activating scenarios have been reported. CRACK  are a SDL/TOSCA scenario definition, design and deployment languages and Automated Deployment of Laboratory Environments Systems (ADLES) , an open source specification language and associated deployment tool, achieve the same goals. ADLES provides an instructor a tool-set to design, specify, and semi-automatically deploy the training scenario together with tutorials as well as competitions. Furthermore, efficient sharing of classes together with the associated computing environment are provisioned to participants. The ADLES deployment begins with the verification and fixing of Master instances by converting them into templates followed by the use of these instances to clone services, create virtual networks and folders. The full exercise scenario on the specified virtualisation platform is then deployed. While these implementations are current state-of-art deployments, it is important to acknowledge that within the foreseeable future, the effectiveness of these tools will be diluted as the sophistication and complexity of cyber-attacks evolve powered through AI-based and Bio-Inspired attack strategies, motivating the need to migrate to Real-Time Auto-configurable systems.
5.2. CRs and TBs Applications
- Industrial and Commercial: IBM X-Force Command Centre  is the first commercial malware simulator that tests for the security of systems. At the heart of the simulator is a mobile Command Cyber Tactical Operations Center (C-TOC) that provides CR and watch floor services. The C-TOC can be configured both as an immersive training CR, a platform for Red teaming and capture-the-flag competitions, as well as a watch floor for special security events. The Ixia Breaking Point, advertised as providing CR capabilities , is also a commercially available. The single rack-mountable appliance provides traffic generation and a ‘Strike Pack’ of network security and malware attacks. Exata is yet another commercially available simulation-based CR. A number of emulation-based CR are currently on offer, a good example being the ATC .
- Education and Research: Cohen  presents the development of SECUSIM , a highly customisable system with integrated Graphic User Interface (GUI) capabilities, the first example of the education and research community creating a training platform for simulating the impact of attacks on computer networks . The University of Illinois has developed the Real Time Immersive Network Simulation Environment (RINSE) in 2006, also primarily for training . Other implementations in academia for modelling computer networks and intrusion detection systems (IDSs) attacks include the Virginia CR , Emulab , Virtualised CR , ARENA . NetENGINE  has been designed for training on the strategies to combat cyber-attacks in large IP networks comprising a Virtual Cyber-Security Testing Capability (VCSTS) for the automated testing of new devices to assess its security robustness before deployment .
- Military, Defence and Intelligence: Davis and Magrath  assert that the USA Air Force (USAF) used CR around 2002, an element of the Simulator Training Exercise Network (SIMTEX) referred to as the Black Demon. The first reported CR was the Defence Advanced Research Projects Agency (DARPA)’s National Cyber Range (NCR) representing the foundation in the training of their military, defence and intelligence agencies on cyber warfare initiated by the United State military in 2009 as a consequence of the US Department of Defence classified military computer networking infrastructure being significantly compromised in 2008 . Although NCR was largely a military-sponsored initiative, its use and application cut across the military, commercial, academic, and Government sectors [16,17]. Fourteen (14) CR applications in Military, Defense and Intelligence have been recorded to date ranging from CRATE , DoD CR , CAAJED , SAST , StealthNet , LARIAT  to INL . Their role is not only to train the security agencies of sovereign countries on counter cyber-terrorism and warfare, but also to protect the nation’s critical infrastructure such as Naval, Power and Aviation. SoftGrid  and CyberVan  are examples of TBs found in these application sectors.
- Smart Grids: The predominate area of application for TBs is Smart Grids owing to the reliance for the effective operation of an ever-evolving power network on an enabling communication network with information flow managing the power delivery. Consequently, the security of equipment and the critical signals that control the power system becomes essential for the safe, flexible and uninterrupted provision of the supply of energy.A Smart Grid Test-bed can be cast as two simulation environments (Figure 15), one for the power, the other for the cyber/communication network. Co-simulator segmentation is a necessity as a hacker can target operations within both networks . Here, TBs are classified into two categories; off-line and real-time. An off-line environment is the most prevalent approach realised, most readily, by SCADA systems ; refer to Table 3 for details. OMNET++ or NS2 are invariably at the core of most cyber simulators, with the TCP/IP protocol used to communicate between simulators. Synchronisation is central to the co-ordination of operations in the two domains. Real-time TBs have been proven to facilitate efficient training outcomes .
- IoT Devices: In the recent past, Internet-of-Things (IoT) architectures have evolved rapidly characterised by a growing complexity of inter-connections of an ever-increasing number of nodes (‘things’). The proliferation of highly connected environments translates into an enhanced spectrum of vulnerabilities/opportunities for cyber criminals. Furthermore, IoT-inspired data-driven solutions have been adopted by key industry sectors as a means to implement business transformation. Securing network infrastructures consisting for example, of medical records, financial credential information against breaches becomes even more challenging. The training of security operators in these new classes of threats is essential. IoT Test-Beds that can simulate different kinds of attacks play an important role in supporting the delivery of dynamically changing training requirements. Example IoT Test-Beds have been reported in [67,68,69].
5.3. CRs and TBs Realisation
- Evaluate the weaknesses in the infrastructure; the architecture of the local network, past attacks, and the current security strategy should be examined.
- Map the basic solutions; e.g., a firewall or control of external devices.
- Asses current security policies; to enhance the security level of an infrastructure.
- Training by simulation of attack exercises and scenarios; the definition of an appropriate virtual TB for training should adopt the following steps;
- Step 1A classification of the infrastructure is an essential step in advance of the realisation of the TB. Validation tests on the infrastructure may are required for an accurate classification.
- Step 2The vulnerabilities of the infrastructure should be identified; the localisation of vulnerabilities is important in informing on the security deficiencies within the infrastructure.
- Step 3Selection of the most appropriate software dependent on the application domain and the infrastructure. As an example, in the Smart Grid environment, OPAL-RT can be chosen to simulate the electric power and a discrete event network simulator to simulate the communication network.
- Step 4A modular approach is adopted to describe the infrastructure, with the input/output of each module verified.
- Step 5A database of different tests and scenarios is created, fundamental for the validation of the TB.
- Step 6Users are trained to respond to a range of attacks and threat scenarios, with the relevant reports being extracted from the interface module of the TB.
6. Analysis and Taxonomies
6.1.7. Attack Types
7. Training Methods
- Gamification: Gamification has been adopted to make Cyber-Security training more engaging and motivating . The principle is to enhance exercises through a compelling experience utilising graphics and play. The aim is to enrich the challenge, engagement, as well as motivating the trainees owing to increased levels of interaction. The concept of ‘Attacker-centric Gamification’ was introduced by Adams and Makramalla in  with the goal empowering trainees to assume the roles of attacker combining gamification with entrepreneurial perspectives with an emphasis on surfacing their abilities, skills, knowledge, motivation, and resources .
- Mock Attack Training: The training method, developed by Sadeh et al. , embodies an approach that senses user actions which expose that user’s infrastructure to cyber threats. The action could be as a result of a mock attack delivered to the user through a messaging service from any device, a wireless communication service or a fake malware application. The system selects the most appropriate training from a list of available training routines based on the users’ reaction to the message in so ding delivering the most targeted training.
- Role-Based Training: One practical training approach through CR-enabled scenarios is to assign unique roles to trainees. Such roles, for example emulating or taking the place of a hacker in a real life situation, cyber offensive operator, cyber defender, or training instructor  can be dynamic depending on the exercise, defined or selected using databases that contain predefined roles. Furthermore, customisation to better emulate real-life enterprise is also possible.
- Exercises: Competitions such as [90,91,92,93,94], are aimed at developing problem solving techniques, proficiency, teamwork and cyber defense skills by providing the participants with sets of hands-on Cyber-Security exercises in real-world scenarios to the participants. ‘Catch-The-Flag’ is an example, a distributed, wide-area security hacking competition involving multiple teams. ‘Cyber Defense Exercise (CDX)’ is another form of such an exercise in a larger setting where an inter-agency academy of an institution competes in the design, implementation, management and defend a network of computers. CDX is established by setting objectives, selecting an approach, defining a network topology, creating a scenario, stipulating the rules and choosing the right metrics with which to determine the lessons learnt .
8. The Future of CRs and TBs
8.1. Future Trends
- Real-Time Auto-configurable Systems: MIT’s Lincoln Laboratory developed an advanced tool for Cyber-Ranges referred to as Automatic Live Instantiation of a Virtual Environment (ALIVE), , a range application extension to LARIAT. ALIVE has the capability of ingesting configuration files from Common Cyber Event Registration (CCER) to automate the building out of Virtual Machines and networking infrastructure of the CR . In addition to the capability to create virtual networks, it can also automate most of the system network build-outs, creating end hosts, routers, firewalls, and servers needed to support traffic generation. The host software packages and user accounts can also be installed.The Cyber-Range Instantiation System (CyRIS), an open source tool for facilitating cyber-range creation , can execute efficient instantiation of cyber-ranges automatically. CyRIS automatically aids in the preparation and management of CRs using a pre-defined specification provided by the scenario managers or instructors. The tool contains both basic functions for establishing the infrastructure as well its security settings.ALPACA  is one of the modern auto-configurable CR with the facility to set user-specified constraints to generate complex cyber-ranges. The core of the implementation are an AI planning engine, a database of vulnerabilities and machine specific configuration parameters with the ability to generate a VM that includes the sequences of vulnerabilities and exploits.
- Smart, Mobile and Integrated Technologies: Pharos , a TB for Mobile Cyber-Physical Systems, is aimed at supporting mobile cyber-physical system evaluation in live networks. It is a networked system of independent mobile devices with its fundamental building block based on Proteus (an autonomous mobile system with highly modular software and hardware), with the capability of relating with each other and with networks of embedded sensors and actuators. Push-button repeatability facilitating the recreation of the same scenarios multiple times is an important feature of the TB.Cybertropolis  is aimed at breaking the paradigm of CRs and TBs by providing what is referred to as Cyber-electromagnetic (CEMA) range facilities, which merges the features of CRs and TBs to yield a hybrid type of cyber-security training system. Cybertropolis was developed as a one-of-kind cyber-range that can be used in the areas of industrial control systems, cyber-physical devices, IoT and wireless systems. The platform provides the ability to create a heterogeneous network consisting of virtual Information and Communication Technology (ICT) systems with integrated live cyber-physical systems, live Radio Frequency (RF), and Internet of Things (IoT) systems into a virtual environment.
- Training with Augmented Reality Technology: Augmented Reality (AR) is increasingly viewed as an important dimension in learning in different domains and is being considered as another impactful technology in future CR and TB training. AR offers the possibility of interaction with different parts of the systems, in so doing enriching the training owing to enhanced visualisation. Augmented reality TB or CR create a new interactive experience able to modify the trainee view of the progression of attacks. AR solution also gates portable solutions, as an example, the attack reaction could be modelled anywhere without infrastructure dependence. The environment can be modified and the programmer can add new attack scenarios.
8.2. Future Technologies
- 5G/6G Technologies: 5th and 6th Generation (5G, 6G) networks will transform services using mobile and wireless network infrastructures by provisioning connections with advantageous features ranging from low latency with high network bandwidth capability through to machine-to-machine communication. 5G solutions enable better services using Virtualisation and Cloud technologies , extending to Network Functions Virtualisation (NFV) which enhances server virtualisation to network devices. Tranoris et al. , utilised these capabilities to demonstrate real-time remote monitoring and video streaming between Vehicle-to-Vehicle (V2V) in an assisted overtaking application , showcasing the potential impact from emerging 5G and beyond. Mitra and Agrawal , described a highly futuristic connected society—“smart living”: Vehicle Ad-hoc Networks (VANET) cloud for network connected transport systems managing dynamic real-time traffic demands; and massive M2M communications. West  also added that the revolution will bring about IoT-enabled health services while Letaief et al.  postulate that 6G will bring about ubiquitous AI-based services. The self-same capabilities present leveraging opportunities for CR and TB engineers and users to provide a seamless, faster, and low latency-based CR and TB deployments using virtual machines, sandboxes or containerised technologies.
- Containerisation Technologies: The impact of hypervisor-enabled virtualisation technology in CRs/TBs cyber-warfare training has been highly beneficial. VMs provide the required isolation from operational networks but present users with real-life training scenarios. The deployment of applications to implement VMs on data centres needs a dedicated guest operating system on each VM, on occasion different from the host operating system. Containerisation technology has been introduced as light-weight virtualised technology to that of VMs in order to manage these concomitant accrued overheads. A study conducted by Bhardwaj and Krishna  compared the use of the pre-copy VM migration scheme with that of the LXD/CR container migration technique, concluding that the use of latter reduces system downtime by 76.66%, migration time by 65.55%, scalability (volume of data transferred) by 76.63%, throughput (number of transferred pages) by 76.78%, overhead costs were reduced with regards to CPU utilisation by 55.89% and RAM utilisation by 76.52%. Thus, containerised technology costs less, guarantees more system up-time and saves times. Other studies that highlight the benefits of containerised technologies are Lovas et al. , on their software container-based simulation platform in order to achieve scalability and portability; Mucci and Blumbers  to gain flexibility, reduce complexity while providing extensibility; and Kyriakou et al. , to ease deployment, management and resilience of their cloud-based environment.
8.3. Future Application Areas
- Smart Cyber-Physical Systems: Smart Cyber-Physical Systems (sCPS) are large-scale software intensive and pervasive systems, that are intelligent, self-aware, self-managing and self-configuring . In line with other data driven artificially intelligence powered systems, sCPS utilise multiple data streams to manage real-world processes efficiently and through these offers a broad range of new applications and services in housing, hospital, transportation and automobile applications.In recent times, cyber-attackers have up-skilled their skills through AI techniques to automate attacks, augment their strategies, launch more sophisticated attacks and by implication increase the success rates [112,113]. ICT tools and AI techniques have not only enriched the opportunities for cyber-attackers as a new form of threat landscape has suddenly emerged. There is a pressing obligation for Cyber-Range based training to evolve as a consequence implementing the detection as well as informing on optimum mitigation of these new threat dynamics.
- Smart Cities and Industry 4.0: The 4th Industrial revolution, also referred to as Industry 4.0, are data driven, network connected, digitalised industrial systems, heralding an era of automated manufacturing and service delivery with strong potential of process optimisation, imbued with new business practices. The evolution is, however, not without its attendant new cyber-threats. CyberFactory#1  is designed to proffer a solution between future digital factories and security threats gated by digitalisation. The principles on which the environment is established are conscious design, development, and demonstration of a System-of-Systems embracing the technical, economical, human and societal dimension of future factories . The platform demonstrates sets of major enabling capabilities that foster optimisation and resilience of next generation manufacturing and service delivery industries. As the evolution unfolds, there is a need to continue to propose new solutions capable of mitigating the dilemma between the deployment of future factories/smart cities and cyber-threats.A body of available literature stresses that cyber threats and privacy concerns will increase significantly in smart systems due to high degrees of network inter-connectivity; Reys et al. in , Baig et al. in , Vitunskaite et al. in , Mylrea et al. in , Srivastava et al. in , Aldairi et al. in , Cerrudo et al. in , Alibasic et al. in  and Braun et al. in . Wang et al. in  and Farahat et al. in  focused on data security as well as threat modeling for smart city infrastructures. Vattapparamban et al. in  expect that drones will be used in service delivery in highly connected smart cities environments of the future and hence will become a factor in defining the scope of cyber-attacks. Li et al. in  report on the intelligent management of network traffic to avoid congestion while reducing Cyber-Security concerns in Smart cities.
- Aerospace and Satellite Industries: The evolution of the aerospace and satellite industries and the significant contribution the sector makes to the health of the economy has made them a central interest for cyber-attacks. CRs and TBs are essential to model the impact of cyber-attack effects and enhance the ability of protecting this critical infrastructure. The goal is to understand and overcome the spectrum of possible attacks by taking into account the sensitivity of information used. Virtualisation using a simulation-based system is a potential solution to implement TBs, but a total recognition of several parts of such critical infrastructure should be studied. The prediction of the hacker’s strategies and aims remain the core to understanding the optimum countermeasure against class of attack.
Conflicts of Interest
- Denning, D.E. Stuxnet: What has changed? Future Internet 2012, 4, 672–687. [Google Scholar] [CrossRef][Green Version]
- Lallie, H.S.; Shepherd, L.A.; Nurse, J.R.C.; Erola, A.; Epiphaniou, G.; Maple, C.; Bellekens, X. Cyber Security in the Age of COVID-19: A Timeline and Analysis of Cyber-Crime and Cyber-Attacks during the Pandemic. arXiv 2020, arXiv:cs.CR/2006.11929. [Google Scholar]
- Leyden, J. Stuxnet’a Game Changer for Malware Defence’; The Register: London, UK, 2010. [Google Scholar]
- Mitchell, R. After Stuxnet: The new rules of cyberwar. Computerworld 2012, 5, 1. [Google Scholar]
- Benson, P. Computer Virus Stuxnet a ‘Game Changer’, DHS Official Tells Senate. Available online: http://edition.cnn.com/2010/TECH/web/11/17/stuxnet.virus/index.html (accessed on 11 December 2020).
- Aldawood, H.; Skinner, G. Educating and raising awareness on cyber security social engineering: A literature review. In Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Wollongong, NSW, Australia, 4–7 December 2018; pp. 62–68. [Google Scholar]
- Zwilling, M.; Klien, G.; Lesjak, D.; Wiechetek, Ł.; Cetin, F.; Basim, H.N. Cyber Security Awareness, Knowledge and Behavior: A Comparative Study. J. Comput. Inf. Syst. 2020, 1–16. [Google Scholar] [CrossRef]
- Newhouse, W.; Keith, S.; Scribner, B.; Witte, G. National initiative for cybersecurity education (NICE) cybersecurity workforce framework. NIST Spec. Publ. 2017, 800, 181. [Google Scholar]
- Giuliano, V.; Formicola, V. ICSrange: A Simulation-based Cyber Range Platform for Industrial Control Systems. arXiv 2019, arXiv:1909.01910. [Google Scholar]
- Yonemura, K.; Sato, J.; Komura, R.; Matsuoka, M. Practical security education on combination of OT and ICT using gamfication method. In Proceedings of the 2018 IEEE Global Engineering Education Conference (EDUCON 2018), Tenerife, Spain, 17–20 April 2018; pp. 746–750. [Google Scholar]
- Van der Velde, D.; Henze, M.; Kathmann, P.; Wassermann, E.; Andres, M.; Bracht, D.; Ernst, R.; Hallak, G.; Klaer, B.; Linnartz, P.; et al. Methods for Actors in the Electric Power System to Prevent, Detect and React to ICT Attacks and Failures. arXiv 2020, arXiv:2003.06185. [Google Scholar]
- Vozikis, D.; Darra, E.; Kuusk, T.; Kavallieros, D.; Reintam, A.; Bellekens, X. On the Importance of Cyber-Security Training for Multi-Vector Energy Distribution System Operators. In Proceedings of the 15th International Conference on Availability, Reliability and Security, online, online, 25–28 August 2020. [Google Scholar] [CrossRef]
- Okoli, C.; Schabram, K. A Guide to Conducting a Systematic Literature Review of Information Systems Research. SSRN Electron. J. 2010. [Google Scholar] [CrossRef][Green Version]
- Okoli, C. A Guide to Conducting a Standalone Systematic Literature Review. Commun. Assoc. Inf. Syst. 2015, 37, 43. [Google Scholar] [CrossRef][Green Version]
- Bures, M.; Klima, M.; Rechtberger, V.; Bellekens, X.; Tachtatzis, C.; Atkinson, R.; Ahmed, B.S. Interoperability and Integration Testing Methods for IoT Systems: A Systematic Mapping Study. In Software Engineering and Formal Methods; de Boer, F., Cerone, A., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 93–112. [Google Scholar]
- Davis, J.; Magrath, S. A Survey of Cyber Ranges and Testbeds. Available online: https://apps.dtic.mil/dtic/tr/fulltext/u2/a594524.pdf (accessed on 11 December 2020).
- Ranka, J. National Cyber Range. Available online: https://apps.dtic.mil/dtic/tr/fulltext/u2/a551864.pdf (accessed on 11 December 2020).
- Leblanc, S.P.; Partington, A.; Chapman, I.M.; Bernier, M. An Overview of Cyber Attack and Computer Network Operations Simulation. In Proceedings of the 2011 Military Modeling & Simulation Symposium, Boston, MA, USA, 3–7 April 2011; pp. 92–100. [Google Scholar]
- Priyadarshini, I. Features and Architecture of the Modern Cyber Range: A Qualitative Analysis and Survey. Ph.D. Thesis, University of Delaware, Newark, DE, USA, 2018. [Google Scholar]
- Yamin, M.M.; Katt, B.; Gkioulos, V. Cyber Ranges and Security Testbeds: Scenarios, Functions, Tools and Architecture. Comput. Secur. 2020, 88, 101636. [Google Scholar] [CrossRef]
- Radziwill, N.M. Virginia cyber range. Softw. Qual. Prof. 2017, 19, 46. [Google Scholar]
- Sommestad, T. Experimentation on Operational Cyber Security in CRATE. Available online: http://www.sommestad.com/teodor/Filer/Sommestad%20-%202015%20-%20Experimentation%20on%20operational%20cyber%20security%20in%20CRATE.pdf (accessed on 11 December 2020).
- Pernik, P. Improving Cyber Security: NATO and the EU. Available online: https://icds.ee/wp-content/uploads/2010/02/Piret_Pernik_-_Improving_Cyber_Security.pdf (accessed on 11 December 2020).
- Ferguson, B.; Tall, A.; Olsen, D. National cyber range overview. In Proceedings of the 2014 IEEE Military Communications Conference, Baltimore, MD, USA, 6–8 October 2014; pp. 123–128. [Google Scholar]
- Mudge, R.S.; Lingley, S. Cyber and Air Joint Effects Demonstration (CAAJED). Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.460.476&rep=rep1&type=pdf (accessed on 11 December 2020).
- Meitzler, W.D.; Ouderkirk, S.J.; Hughes, C.O. Security Assessment Simulation Toolkit (SAST) Final Report; Technical report; Pacific Northwest National Lab.: Richland, WA, USA, 2009.
- Varshney, M.; Pickett, K.; Bagrodia, R. A live-virtual-constructive (LVC) framework for cyber operations test, evaluation and training. In Proceedings of the 2011-MILCOM 2011 Military Communications Conference, Baltimore, MD, USA, 7–10 November 2011; pp. 1387–1392. [Google Scholar]
- Chi, S.D.; Park, J.S.; Lee, J.S. A role of DEVS simulation for information assurance. In International Workshop on Information Security Applications; Springer: Berlin/Heidelberg, Germany, 2003; pp. 27–41. [Google Scholar]
- Liljenstam, M.; Liu, J.; Nicol, D.; Yuan, Y.; Yan, G.; Grier, C. Rinse: The real-time immersive network simulation environment for network security exercises. In Proceedings of the Workshop on Principles of Advanced and Distributed Simulation (PADS’05), Monterey, CA, USA, 1–3 June 2005; pp. 119–128. [Google Scholar]
- Brown, B.; Cutts, A.; McGrath, D.; Nicol, D.M.; Smith, T.P.; Tofel, B. Simulation of cyber attacks with applications in homeland defense training. In Proceedings of the Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Defense and Law Enforcement II, Orlando, FL, USA, 21–25 April 2003; pp. 63–71. [Google Scholar]
- Kuhl, M.E.; Sudit, M.; Kistner, J.; Costantini, K. Cyber attack modeling and simulation for network security analysis. In Proceedings of the 2007 Winter Simulation Conference, Washington, DC, USA, 9–12 December 2007; pp. 1180–1188. [Google Scholar]
- Zhou, M.; Lang, S.D. A Frequency-Based Approach to Intrusion Detection. Available online: https://d1wqtxts1xzle7.cloudfront.net/56628641/dosdos.pdf?1527000071=&response-content-disposition=inline%3B+filename%3DA_Frequency_Based_Approach_to_Intrusion.pdf&Expires=1607693224&Signature=BsARTy9SFcwtnh3em1-GAbLEdtohfiBZqLiuH5ODCyL5hife4Ga95~urso2IqlZxQnaAN~GepEHt8ovbOD9-XdAXMfLvKQ6PCBdFX5fhWYE2ZOGWfgcBjEy16pMT9egqXlZkSr4e-raPjDQy6GxeQRTTe4Z~tlQTCtdI6dn6GOGQssPh3lv5bwwbi6CnPBScFCv823Gwfbkk2rPp5WyLGRMKFaXlqRdzmFrqYKdUpsTeRK-bNXBjViTrD6hlbA0iFl3ELVh431TX1OC1elbrdZWaQL8sZPmi6YmM5XNH2SJ5RyqawQ1HrHjA9-Jhs3kMs5rqRw9l857KsGg9G4Whwg__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA (accessed on 11 December 2020).
- Rossey, L.M.; Cunningham, R.K.; Fried, D.J.; Rabek, J.C.; Lippmann, R.P.; Haines, J.W.; Zissman, M.A. LARIAT: Lincoln adaptable real-time information assurance testbed. In Proceedings of the IEEE Aerospace Conference, Big Sky, MT, USA, 9–16 March 2002; p. 6. [Google Scholar]
- Pederson, P.; Lee, D.; Shu, G.; Chen, D.; Liu, Z.; Li, N.; Sang, L. Virtual Cyber-Security Testing Capability for Large Scale Distributed Information Infrastructure Protection. In Proceedings of the 2008 IEEE Conference on Technologies for Homeland Security, Waltham, MA, USA, 12–13 May 2008; pp. 372–377. [Google Scholar]
- Anderson, R.S. Cyber Security and Resilient Systems; Technical report; Idaho National Laboratory: Idaho Falls, ID, USA, 2009.
- Siaterlis, C.; Garcia, A.P.; Genge, B. On the use of Emulab testbeds for scientifically rigorous experiments. IEEE Commun. Surv. Tutor. 2012, 15, 929–942. [Google Scholar] [CrossRef]
- Mayo, J.; Minnich, R.; Rudish, D.; Armstrong, R. Approaches for Scalable Modeling and Emulation of Cyber Systems: LDRD Final Report. Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.463.1793&rep=rep1&type=pdf (accessed on 11 December 2020).
- Thomas, J.; Meunier, P.; Eugster, P.; Vitek, J. Mandatory access control for experiments with malware. In Proceedings of the 10th Annual Information Security Symposium, West Lafayette, IN, USA, 9–10 April 2009. [Google Scholar]
- Brueckner, S.; Guaspari, D.; Adelstein, F.; Weeks, J. Automated computer forensics training in a virtualized environment. Digit. Investig. 2008, 5, S105–S111. [Google Scholar] [CrossRef]
- Tsai, P.W.; Yang, C.S. [email protected] TWISC: A network security experiment platform. Int. J. Commun. Syst. 2018, 31, e3446. [Google Scholar] [CrossRef]
- Herold, N.; Wachs, M.; Dorfhuber, M.; Rudolf, C.; Liebald, S.; Carle, G. Achieving reproducible network environments with INSALATA. In IFIP International Conference on Autonomous Infrastructure, Management and Security; Springer: Cham, Switzerland, 2017; pp. 30–44. [Google Scholar]
- Chadha, R.; Bowen, T.; Chiang, C.Y.J.; Gottlieb, Y.M.; Poylisher, A.; Sapello, A.; Serban, C.; Sugrim, S.; Walther, G.; Marvel, L.M.; et al. Cybervan: A cyber security virtual assured network testbed. In Proceedings of the MILCOM 2016—2016 IEEE Military Communications Conference, Baltimore, MD, USA, 1–3 November 2016; pp. 1125–1130. [Google Scholar]
- Gunathilaka, P.; Mashima, D.; Chen, B. Softgrid: A software-based smart grid testbed for evaluating substation cybersecurity solutions. In Proceedings of the 2nd ACM Workshop on Cyber-Physical Systems Security and Privacy, Vienna, Austria, 28 October 2016; pp. 113–124. [Google Scholar]
- Haines, J.W.; Rossey, L.M.; Lippmann, R.P.; Cunningham, R.K. Extending the darpa off-line intrusion detection evaluations. In Proceedings of the DARPA Information Survivability Conference and Exposition II. DISCEX’01, Anaheim, CA, USA, 12–16 June 2001; pp. 35–45. [Google Scholar]
- Hahn, A.; Kregel, B.; Govindarasu, M.; Fitzpatrick, J.; Adnan, R.; Sridhar, S.; Higdon, M. Development of the PowerCyber SCADA security testbed. In Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research, Oak Ridge, TN, USA, 21–23 April 2010; pp. 1–4. [Google Scholar]
- Hindy, H.; Brosset, D.; Bayne, E.; Seeam, A.; Bellekens, X. Improving SIEM for Critical SCADA Water Infrastructures Using Machine Learning. In Computer Security; Katsikas, S.K., Cuppens, F., Cuppens, N., Lambrinoudakis, C., Antón, A., Gritzalis, S., Mylopoulos, J., Kalloniatis, C., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 3–19. [Google Scholar]
- Tam, K.; Jones, K. Cyber-SHIP: Developing Next Generation Maritime Cyber Research Capabilities. Available online: https://pearl.plymouth.ac.uk/handle/10026.1/14949 (accessed on 11 December 2020).
- Hildebrand, E.; Flinterman, R.; Mulder, J.; Smit, A. Clusus: A Cyber Range for Network Attack Simulations. Available online: http://resolver.tudelft.nl/uuid:2588ee53-0249-476f-aa53-54d7a368dc8e (accessed on 11 December 2020).
- Hallaq, B.; Nicholson, A.; Smith, R.; Maglaras, L.; Janicke, H.; Jones, K. CYRAN: A hybrid cyber range for testing security on ICS/SCADA systems. In Cyber Security and Threats: Concepts, Methodologies, Tools, and Applications; IGI Global: Hershey, PA, USA, 2018; pp. 622–637. [Google Scholar]
- Ghaleb, A.; Zhioua, S.; Almulhem, A. SCADA-SST: A SCADA security testbed. In Proceedings of the 2016 World Congress on Industrial Control Systems Security (WCICSS), London, UK, 12–14 December 2016; pp. 1–6. [Google Scholar]
- Čeleda, P.; Čegan, J.; Vykopal, J.; Tovarňák, D. Kypo—A platform for cyber defence exercises. Available online: https://is.muni.cz/repo/1319597/2015-NATO-MSG-133-kypo-platform-cyber-defence-exercises-paper.pdf (accessed on 11 December 2020).
- Vykopal, J.; Ošlejšek, R.; Čeleda, P.; Vizvary, M.; Tovarňák, D. Kypo Cyber Range: Design and Use Cases. Available online: https://repozitar.cz/repo/26486/ (accessed on 11 December 2020).
- Almalawi, A.; Tari, Z.; Khalil, I.; Fahad, A. SCADAVT-A framework for SCADA security testbed based on virtualization technology. In Proceedings of the 38th Annual IEEE Conference on Local Computer Networks, Sydney, Australia, 21–24 October 2013; pp. 639–646. [Google Scholar]
- Benzel, T.; Braden, R.; Kim, D.; Neuman, C.; Joseph, A.; Sklower, K.; Ostrenga, R.; Schwab, S. Experience with deter: A testbed for security research. In Proceedings of the 2nd International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, TRIDENTCOM, Barcelona, Spain, 1–3 March 2006. [Google Scholar]
- Richmond, M. ViSe: A virtual security testbed. Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.2205&rep=rep1&type=pdf (accessed on 11 December 2020).
- Marrocco, D. Design and Deployment of a Virtual Environment to Emulate a Scada Network within Cyber Ranges. Ph.D. Thesis, Politecnico di Torino, Torino, Italy, 2018. [Google Scholar]
- Ten, C.W.; Liu, C.C.; Govindarasu, M. Vulnerability assessment of cybersecurity for SCADA systems using attack trees. In Proceedings of the 2007 IEEE Power Engineering Society General Meeting, Tampa, FL, USA, 24–28 June 2007; pp. 1–8. [Google Scholar]
- Russo, E.; Costa, G.; Armando, A. Scenario design and validation for next generation cyber ranges. In Proceedings of the 2018 IEEE 17th International Symposium on Network Computing and Applications (NCA), Cambridge, MA, USA, 1–3 November 2018; pp. 1–4. [Google Scholar]
- Binz, T.; Breitenbücher, U.; Kopp, O.; Leymann, F. TOSCA: Portable automated deployment and management of cloud applications. In Advanced Web Services; Springer: New York, NY, USA, 2014; pp. 527–549. [Google Scholar]
- Russo, E.; Costa, G.; Armando, A. Building Next Generation Cyber Ranges with CRACK. Comput. Secur. 2020, 95, 101837. [Google Scholar] [CrossRef]
- De Leon, D.C.; Goes, C.E.; Haney, M.A.; Krings, A.W. ADLES: Specifying, deploying, and sharing hands-on cyber-exercises. Comput. Secur. 2018, 74, 12–40. [Google Scholar] [CrossRef]
- Cohen, F. Simulating cyber attacks, defences, and consequences. Comput. Secur. 1999, 18, 479–518. [Google Scholar] [CrossRef]
- Park, J.S.; Lee, J.S.; Kim, H.K.; Jeong, J.R.; Yeom, D.B.; Chi, S.D. Secusim: A tool for the cyber-attack simulation. In Proceedings of the International Conference on Information and Communications Security, Xi’an, China, 13–16 November 2001; pp. 471–475. [Google Scholar]
- Lynn, W.F., III. Defending a new domain-the Pentagon’s cyberstrategy. Foreign Aff. 2010, 89, 97. [Google Scholar]
- Hammad, E.; Ezeme, M.; Farraj, A. Implementation and development of an offline co-simulation testbed for studies of power systems cyber security and control verification. Int. J. Electr. Power Energy Syst. 2019, 104, 817–826. [Google Scholar] [CrossRef]
- Poudel, S.; Ni, Z.; Malla, N. Real-time cyber physical system testbed for power system security and control. Int. J. Electr. Power Energy Syst. 2017, 90, 124–133. [Google Scholar] [CrossRef]
- Waraga, O.A.; Bettayeb, M.; Nasir, Q.; Talib, M.A. Design and implementation of automated IoT security testbed. Comput. Secur. 2020, 88, 101648. [Google Scholar] [CrossRef]
- Kim, Y.; Nam, J.; Park, T.; Scott-Hayward, S.; Shin, S. SODA: A software-defined security framework for IoT environments. Comput. Netw. 2019, 163, 106889. [Google Scholar] [CrossRef]
- Lee, S.; Lee, S.; Yoo, H.; Kwon, S.; Shon, T. Design and implementation of cybersecurity testbed for industrial IoT systems. J. Supercomput. 2018, 74, 4506–4520. [Google Scholar] [CrossRef]
- Wang, Y.; Nguyen, T.L.; Xu, Y.; Shi, D. Distributed control of heterogeneous energy storage systems in islanded microgrids: Finite-time approach and cyber-physical implementation. Int. J. Electr. Power Energy Syst. 2020, 119, 105898. [Google Scholar] [CrossRef]
- De La Torre, G.; Rad, P.; Choo, K.K.R. Implementation of deep packet inspection in smart grids and industrial Internet of Things: Challenges and opportunities. J. Netw. Comput. Appl. 2019, 135, 32–46. [Google Scholar] [CrossRef]
- Kumar, A.; Lim, T.J. A Secure Contained Testbed for Analyzing IoT Botnets. In Proceedings of the International Conference on Testbeds and Research Infrastructures, Shanghai, China, 1–3 December 2018; pp. 124–137. [Google Scholar]
- Alves, T.; Das, R.; Werth, A.; Morris, T. Virtualization of SCADA testbeds for cybersecurity research: A modular approach. Comput. Secur. 2018, 77, 531–546. [Google Scholar] [CrossRef]
- Siracusano, G.; Salsano, S.; Ventre, P.L.; Detti, A.; Rashed, O.; Blefari-Melazzi, N. A framework for experimenting ICN over SDN solutions using physical and virtual testbeds. Comput. Netw. 2018, 134, 245–259. [Google Scholar] [CrossRef]
- Papadopoulos, G.Z.; Gallais, A.; Schreiner, G.; Jou, E.; Noel, T. Thorough IoT testbed characterization: From proof-of-concept to repeatable experimentations. Comput. Netw. 2017, 119, 86–101. [Google Scholar] [CrossRef][Green Version]
- Liu, X.F.; Shahriar, M.R.; Al Sunny, S.N.; Leu, M.C.; Hu, L. Cyber-physical manufacturing cloud: Architecture, virtualization, communication, and testbed. J. Manuf. Syst. 2017, 43, 352–364. [Google Scholar] [CrossRef]
- Bernieri, G.; Miciolino, E.E.; Pascucci, F.; Setola, R. Monitoring system reaction in cyber-physical testbed under cyber-attacks. Comput. Electr. Eng. 2017, 59, 86–98. [Google Scholar] [CrossRef]
- Flauzac, O.; Gonzalez, C.; Nolot, F. Developing a distributed software defined networking testbed for IoT. Procedia Comput. Sci. 2016, 83, 680–684. [Google Scholar] [CrossRef][Green Version]
- Ashok, A.; Sridhar, S.; McKinnon, A.D.; Wang, P.; Govindarasu, M. Testbed-based performance evaluation of attack resilient control for AGC. In Proceedings of the 2016 Resilience Week (RWS), Chicago, IL, USA, 16–18 August 2016; pp. 125–129. [Google Scholar]
- Deshmukh, P.P.; Patterson, C.D.; Baumann, W.T. A hands-on modular laboratory environment to foster learning in control system security. In Proceedings of the 2016 IEEE Frontiers in Education Conference (FIE), Erie, PA, USA, 12–15 October 2016; pp. 1–9. [Google Scholar]
- Adhikari, U.; Morris, T.; Pan, S. WAMS cyber-physical test bed for power system, cybersecurity study, and data mining. IEEE Trans. Smart Grid 2016, 8, 2744–2753. [Google Scholar] [CrossRef]
- Ashok, A.; Wang, P.; Brown, M.; Govindarasu, M. Experimental evaluation of cyber attacks on automatic generation control using a CPS security testbed. In Proceedings of the 2015 IEEE Power & Energy Society General Meeting, Denver, CO, USA, 26–30 July 2015; pp. 1–5. [Google Scholar]
- Bauer, S.W. What Is Classical Education? Available online: http://www.bluegrasschristianacademy.org/wp-content/uploads/2015/06/Classical-Education-_-The-Well-Trained-Mind.pdf (accessed on 11 December 2020).
- Brynielsson, J.; Franke, U.; Varga, S. Cyber situational awareness testing. In Combatting Cybercrime and Cyberterrorism; Springer: Cham, Switzerland, 2016; pp. 209–233. [Google Scholar]
- Boopathi, K.; Sreejith, S.; Bithin, A. Learning cyber security through gamification. Indian J. Sci. Technol. 2015, 8, 642–649. [Google Scholar] [CrossRef]
- Adams, M.; Makramalla, M. Cybersecurity skills training: An attacker-centric gamified approach. Technol. Innov. Manag. Rev. 2015, 5. [Google Scholar] [CrossRef]
- Bellekens, X.; Jayasekara, G.; Hindy, H.; Bures, M.; Brosset, D.; Tachtatzis, C.; Atkinson, R. From Cyber-Security Deception to Manipulation and Gratification Through Gamification. In HCI for Cybersecurity, Privacy and Trust; Moallem, A., Ed.; Springer International Publishing: Cham, Switerzerland, 2019; pp. 99–114. [Google Scholar]
- Sadeh-Koniecpol, N.; Wescoe, K.; Brubaker, J.; Hong, J. Mock Attack Cybersecurity Training System and Methods. US Patent US9558677, 31 January 2017. [Google Scholar]
- Toth, P.; Klein, P. A role-based model for federal information technology/cyber security training. NIST Spec. Publ. 2013, 800, 1–152. [Google Scholar]
- Glumich, S.M.; Kropa, B.A. Defex: Hands-On Cyber Defense Exercise for Undergraduate Students. Available online: https://apps.dtic.mil/dtic/tr/fulltext/u2/a560055.pdf (accessed on 11 December 2020).
- Conklin, A. The use of a collegiate cyber defense competition in information security education. In Proceedings of the 2nd Annual Conference on Information Security Curriculum Development, Kennesaw, GA, USA, 22–23 September 2005; pp. 16–18. [Google Scholar]
- Dodge, R.; Ragsdale, D.J. Organized cyber defense competitions. In Proceedings of the IEEE International Conference on Advanced Learning Technologies, Joensuu, Finland, 30 August–1 September 2004; pp. 768–770. [Google Scholar]
- Augustine, T.; Dodge, R.C. Cyber Defense Exercise: Meeting Learning Objectives Thru Competition. 2006. Available online: https://tinyurl.com/y254qtj3 (accessed on 11 December 2020).
- Mattson, J.A. Cyber defense exercise: A service provider model. In Proceedings of the Fifth World Conference on Information Security Education, West Point, NY, USA, 19–21 June 2007; pp. 81–86. [Google Scholar]
- Patriciu, V.V.; Furtuna, A.C. Guide for Designing Cyber Security Exercises. Available online: http://www.wseas.us/e-library/conferences/2009/tenerife/EACT-ISP/EACT-ISP-28.pdf (accessed on 11 December 2020).
- Braje, T.M. Advanced Tools for Cyber Ranges; Technical report; MIT Lincoln Laboratory: Lexington, KY, USA, 2016. [Google Scholar]
- Talbot, J.; Pikula, P.; Sweetmore, C.; Rowe, S.; Hindy, H.; Tachtatzis, C.; Atkinson, R.; Bellekens, X. A Security Perspective on Unikernels. In Proceedings of the 2020 International Conference on Cyber Security and Protection of Digital Services (Cyber Security), Dublin, Ireland, 15–19 June 2020; pp. 1–7. [Google Scholar]
- Pham, C.D. On Automatic Cyber Range Instantiation for Facilitating Security Training. Available online: https://dspace.jaist.ac.jp/dspace/bitstream/10119/14161/3/paper.pdf (accessed on 11 December 2020).
- Eckroth, J.; Chen, K.; Gatewood, H.; Belna, B. Alpaca: Building Dynamic Cyber Ranges with Procedurally-Generated Vulnerability Lattices. In Proceedings of the 2019 ACM Southeast Conference, Kennesaw, GA, USA, 18–20 April 2019; pp. 78–85. [Google Scholar]
- Fok, C.; Petz, A.; Stovall, D.; Paine, N.; Julien, C.; Vishwanath, S. Pharos: A Testbed for Mobile Cyber-Physical Systems. Available online: https://www.cs.odu.edu/~nadeem/classes/cs795-CPS-S13/papers/eval-001.pdf (accessed on 11 December 2020).
- Deckard, G.M. Cybertropolis: Breaking the paradigm of cyber-ranges and testbeds. In Proceedings of the 2018 IEEE International Symposium on Technologies for Homeland Security (HST), Woburn, MA, USA, 23–24 October 2018; pp. 1–4. [Google Scholar]
- Tranoris, C.; Denazis, S.; Guardalben, L.; Pereira, J.; Sargento, S. Enabling Cyber-Physical Systems for 5G networking: A case study on the Automotive Vertical domain. In Proceedings of the 2018 IEEE/ACM 4th International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS), Gothenburg, Sweden, 27 May–3 June 2018; pp. 37–40. [Google Scholar]
- Urquhart, C.; Bellekens, X.; Tachtatzis, C.; Atkinson, R.; Hindy, H.; Seeam, A. Cyber-security internals of a skoda octavia vRS: A hands on approach. IEEE Access 2019, 7, 146057–146069. [Google Scholar] [CrossRef]
- Mitra, R.N.; Agrawal, D.P. 5G mobile technology: A survey. ICT Express 2015, 1, 132–137. [Google Scholar] [CrossRef][Green Version]
- West, D.M. How 5G technology enables the health internet of things. Brook. Cent. Technol. Innov. 2016, 3, 1–20. [Google Scholar]
- Letaief, K.B.; Chen, W.; Shi, Y.; Zhang, J.; Zhang, Y.J.A. The roadmap to 6G: AI empowered wireless networks. IEEE Commun. Mag. 2019, 57, 84–90. [Google Scholar] [CrossRef][Green Version]
- Bhardwaj, A.; Krishna, C.R. A Container-Based Technique to Improve Virtual Machine Migration in Cloud Computing. IETE J. Res. 2019, 1–16. [Google Scholar] [CrossRef]
- Lovas, R.; Kardos, P.; Gyöngyösi, A.Z.; Bottyán, Z. Weather model fine-tuning with software container-based simulation platform. IDOJÁRÁS/Q. J. Hung. Meteorol. Serv. 2019, 123, 165–181. [Google Scholar] [CrossRef][Green Version]
- Mucci, D.; Blumbergs, B. TED: A Container Based Tool to Perform Security Risk Assessment for ELF Binaries. Available online: https://www.scitepress.org/Papers/2019/73716/73716.pdf (accessed on 11 December 2020).
- Kyriakou, A.; Sklavos, N. Container-based honeypot deployment for the analysis of malicious activity. In Proceedings of the 2018 Global Information Infrastructure and Networking Symposium (GIIS), Thessaloniki, Greece, 23–25 October 2018; pp. 1–4. [Google Scholar]
- Delicato, F.C.; Al-Anbuky, A.; Kevin, I.; Wang, K. Smart Cyber–Physical Systems: Toward Pervasive Intelligence Systems. Future Gener. Comput. Syst. 2020, 107, 1134–1139. [Google Scholar] [CrossRef]
- Kaloudi, N.; Li, J. The ai-based cyber threat landscape: A survey. ACM Comput. Surv. (CSUR) 2020, 53, 1–34. [Google Scholar] [CrossRef][Green Version]
- Brundage, M.; Avin, S.; Clark, J.; Toner, H.; Eckersley, P.; Garfinkel, B.; Dafoe, A.; Scharre, P.; Zeitzoff, T.; Filar, B.; et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv 2018, arXiv:1802.07228. [Google Scholar]
- Bécue, A.; Fourastier, Y.; Praça, I.; Savarit, A.; Baron, C.; Gradussofs, B.; Pouille, E.; Thomas, C. CyberFactory# 1—Securing the industry 4.0 with cyber-ranges and digital twins. In Proceedings of the 2018 14th IEEE International Workshop on Factory Communication Systems (WFCS), Imperia, Italy, 13–15 June 2018; pp. 1–4. [Google Scholar]
- Reys, N. Smart Cities and Cyber Threats, ControlRisks. Available online: https://www.controlrisks.com/-/media/corporate/files/our-thinking/insights/smart-cities-and-cyber-threats/smart-cities-article.pdf (accessed on 11 December 2020).
- Baig, Z.A.; Szewczyk, P.; Valli, C.; Rabadia, P.; Hannay, P.; Chernyshev, M.; Johnstone, M.; Kerai, P.; Ibrahim, A.; Sansurooah, K.; et al. Future challenges for smart cities: Cyber-security and digital forensics. Digit. Investig. 2017, 22, 3–13. [Google Scholar] [CrossRef]
- Vitunskaite, M.; He, Y.; Brandstetter, T.; Janicke, H. Smart cities and cyber security: Are we there yet? A comparative study on the role of standards, third party risk management and security ownership. Comput. Secur. 2019, 83, 313–331. [Google Scholar] [CrossRef][Green Version]
- Mylrea, M. Singapore’s Smart City: Securing It from Emerging Cyber Threats. Available online: https://dr.ntu.edu.sg/handle/10356/80353 (accessed on 11 December 2020).
- Srivastava, S.; Bisht, A.; Narayan, N. Safety and security in smart cities using artificial intelligence—A review. In Proceedings of the 2017 7th International Conference on Cloud Computing, Data Science & Engineering-Confluence, Noida, India, 12–13 January 2017; pp. 130–133. [Google Scholar]
- AlDairi, A. Lo’ai Tawalbeh Cyber security attacks on smart cities and associated mobile technologies. Procedia Comput. Sci. 2017, 109, 1086–1091. [Google Scholar] [CrossRef]
- Cerrudo, C. Hacking Smart Cities. Available online: https://docs.huihoo.com/rsaconference/usa-2015/hta-t10-hacking-smart-cities.pdf (accessed on 11 December 2020).
- Alibasic, A.; Al Junaibi, R.; Aung, Z.; Woon, W.L.; Omar, M.A. Cybersecurity for smart cities: A brief review. In Proceedings of the International Workshop on Data Analytics for Renewable Energy Integration, Riva del Garda, Italy, 23 September 2016; pp. 22–30. [Google Scholar]
- Braun, T.; Fung, B.C.; Iqbal, F.; Shah, B. Security and privacy challenges in smart cities. Sustain. Cities Soc. 2018, 39, 499–507. [Google Scholar] [CrossRef]
- Wang, P.; Ali, A.; Kelly, W. Data security and threat modeling for smart city infrastructure. In Proceedings of the 2015 International Conference on Cyber Security of Smart Cities, Industrial Control System and Communications, (SSIC), Shanghai, China, 5–7 August 2015; pp. 1–6. [Google Scholar]
- Farahat, I.; Tolba, A.; Elhoseny, M.; Eladrosy, W. Data security and challenges in smart cities. In Security in Smart Cities: Models, Applications, and Challenges; Springer: Cham, Switzerland, 2019; pp. 117–142. [Google Scholar]
- Vattapparamban, E.; Güvenç, İ.; Yurekli, A.İ.; Akkaya, K.; Uluağaç, S. Drones for smart cities: Issues in cybersecurity, privacy, and public safety. In Proceedings of the 2016 International Wireless Communications and Mobile Computing Conference (IWCMC), Paphos, Cyprus, 5–9 September 2016; pp. 216–221. [Google Scholar]
- Li, Z.; Shahidehpour, M. Deployment of cybersecurity for managing traffic efficiency and safety in smart cities. Electr. J. 2017, 30, 52–61. [Google Scholar] [CrossRef]
|ST, NS||Fed||R, B,|
|IBM X-Force||||N||N||Y||N||N||N||N||C||N||Prv||R, B||VM,|
|Florida CR||||Y||Y||N||N||N||N||Y||C||PT, EH,|
|Regent CR||||Y||Y||Y||N||N||N||N||N||RA, M,|
|Wayne CR||||N||Y||Y||N||N||N||N||C||EH, CTF,|
|[email protected] TWISC|||
|Aca: Academic or Research||H: Hardware||RA: Ransomware Attacks|
|AS: Attack Simulation||LE: Law Enforcement||S: Simulation|
|B: Blue||M: Monitoring||SB: Sandbox|
|C: Cloud||MDI: Military, Defense and Intelligence||SP: Service Provider|
|CSE: Cyber Security Exercise||ME: Method of Experimentation||SS: System Security|
|CTF: Catch-The-Flag||N: No/Not Available||ST: Security/Software Testing|
|DF: Digital Forensic||NS: Network Security||TE: Testing Environment|
|E: Emulation||O: Overlay||TV: Threat and Vulnerability|
|EC: Enterprise and Commercial||OS: Open Source||VCN: Virtual Clone Network|
|EH: Ethical hacking||P: Purple||VM: Virtual Machine|
|Fed: Federated||Prv: Private||VPN: Virtual Private Network|
|Gn: Green||PT: Penetration Testing||W: White|
|Govt: Government||Pub: Public||Y: Yellow|
|Gy: Grey||R: Red||Y: Yes/Available|
|ine Classification||Ref.||Year||Core Technology||Infrastructure Technology||Front-End Technology|
|ine Cyber-SHIP||||2019||Live Scenario||Not Available||Not Available|
|ine ICSRange||||2019||Simulation||Not Available||Not Available|
|ine [email protected]||||2018||Emulation||Available||Available|
|ine Virginia CR||||2017||Simulation||Available||Available|
|ine DoD CR||||2014||Simulation||Available||Available|
|ine SCADAVT-A||||2013||Live Scenario||Available||Available|
|ine NCR (DARPA)||[17,24]||2011||Emulation||Available||Available|
|ine ATC||||2008||Live Scenario||Not Available||Not Available|
|ine RINSE||||2005||Simulation||Not Available||Not Available|
|Integrate generators controllers||Cannot integrate hardware systems|
|||2020||Smart Grid||OPAL-RT||Medium||Test-Beds cyber events||Cyber attack needed to validate|
|||2020||IoT||Open source platform||High||Extensive analysis/Automated tests||Time analysis needed to show the efficiency of Test-Beds|
|||2019||Smart Grid (PSCADA)||OMNET||Low||DoS/FDI||Lack of testes and scenarios|
|||2019||IoT (SCADA)||–||Medium||DNS attack||security tests required|
|||2019||IoT||QEMU emulator||Low||DDOS attack||Few attacks are tested|
|||2018||Water storage (SCADA)||–||High||Packet injection/ARP spoofing/DoS||Real-time implementation discussion missed|
|||2018||Information centric network||CONET||High||Input traffic pattern||Adding further experiences|
|||2017||Smart Grid||OPAL-RT||High||Access to communication link||Real-time implementation of OPF model|
|||2017||IoT||FIT/IoT LAB||Low||No attack tested||lack of security tests|
|||2017||Cloud||Open source||Low||No attack tested||lack of security tests|
|||2017||Industrial control system (SCADA)||–||Medium||Availability attack/Integrity attack||Detection tool should be implemented|
|||2017||IoT(SCADA)||Hardware-based Test-Beds||High||5 kinds of attack||Test-Bed with IDS|
|||2016||IoT||Software-based OpenFlow switches||Low||No attacks||Software defined networking testbed|
|||2016||Smart Grid||Real Time Digital Simulator||Low||Man-in-the-Middle attack||Test Bed with Attack Resilient Control algorithm|
|||2016||SCADA||C++||Low||Denial of Service attack||Test-Bed based on SCADA simulation environment (SCADA-SST)|
|||2016||SCADA||–||Low||Man-in-the-Middle Attack||Test-Bed using CPS topology|
|||2016||Power System||Real Time Digital Simulator (RTDS)||Medium||Aurora Attack/Network Based Cyber-Attacks||WAMS cyber-physical Test-Bed|
|||2015||Power System||Real Time Digital Simulator (RTDS)||Medium||Measurement attack/Control attacks||PowerCyber CPS security Test-Bed|
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Ukwandu, E.; Farah, M.A.B.; Hindy, H.; Brosset, D.; Kavallieros, D.; Atkinson, R.; Tachtatzis, C.; Bures, M.; Andonovic, I.; Bellekens, X. A Review of Cyber-Ranges and Test-Beds: Current and Future Trends. Sensors 2020, 20, 7148. https://doi.org/10.3390/s20247148
Ukwandu E, Farah MAB, Hindy H, Brosset D, Kavallieros D, Atkinson R, Tachtatzis C, Bures M, Andonovic I, Bellekens X. A Review of Cyber-Ranges and Test-Beds: Current and Future Trends. Sensors. 2020; 20(24):7148. https://doi.org/10.3390/s20247148Chicago/Turabian Style
Ukwandu, Elochukwu, Mohamed Amine Ben Farah, Hanan Hindy, David Brosset, Dimitris Kavallieros, Robert Atkinson, Christos Tachtatzis, Miroslav Bures, Ivan Andonovic, and Xavier Bellekens. 2020. "A Review of Cyber-Ranges and Test-Beds: Current and Future Trends" Sensors 20, no. 24: 7148. https://doi.org/10.3390/s20247148