1. Introduction
Over the past years, the growing dependency of the business-critical applications and processes on network technologies and services has expanded the threat landscape to a large extent. Today networks constitute the main vector for attacks [
1]. Thus, considering network security is crucial for ensuring business continuity in light of the growing threats.
Network security mainly concerns the security architectural needs that describe network segmentation (a.k.a., security zoning); security constraints of network devices connecting the communicating end user systems; and security constraints of the data being transferred across the communication links [
2]. The ultimate aim boils down to preventing illegitimate access to the data assets that carry vital information and influence the decisions relating business critical operations [
3]. An appropriate design of the network architecture provides many advantages (e.g., isolation of low trust systems, limitation of a security breach’s scope, costs savings) [
4].
Currently, there exists numerous network security controls (e.g., firewalls, intrusion detection and prevention systems, VPNs, etc.) to implement network security [
5,
6,
7,
8]. Each time a security control (e.g., firewall or a proxy) is added or removed to the network, it will impact the applications running over the network, the quality of service of the network and also the cost of the network with new purchase and installation [
9]. For instance, adding a DMZ (de-militarized zone) into an existing network provides a perimeter defense by isolating the internal network devices that are accessible from the public network. Nevertheless, the DMZ significantly alters the network architecture. In this regard, considering network security too late in the network development cycle (i.e., post deployment of network designs) would render additional costs and complexity in terms of incorporating any changes to the existing infrastructures. The difference in the return of security investments considered at early or at late stages of the system development cycle can range from 12 to 21 percent [
10]. Therefore, it is necessary to examine network security at earlier stages, i.e., right from the planning stages of networks architecting. This perspective is known as security-by-design, which enforces the consideration of security right from the requirements analysis.
Therefore, network security requirements play a critical role since they impact the decisions related to the implementation of security controls [
2]. Indeed, bad network security requirements can lead to ineffective and costly security or worth security holes in the network security design. An effective network security requirement engineering is needed to help organizations in capturing cost-effective security solutions that protect their networks against malicious attacks while meeting the business requirements.
Requirements Engineering (RE), in general, is a sub-discipline of system/software engineering, which subsumes the activities of gathering (a.k.a. eliciting), evaluating and documenting system/software requirements. Security requirements engineering (SRE) methods extend RE to the security context, by enabling the integration of risk analysis concepts. They help organizations in capturing security requirements by analyzing the business risk impact of potential threats. However, the diversity of SRE methodologies leads security requirements engineers to an open question: How to choose one? Utilization of various procedures and tools, choosing unsuitable RE instruments or utilization of inadequate techniques for eliciting requirements are among the most frequently reported issues [
11]. Several comparative studies on SRE methodologies are available. Nonetheless, their evaluation is often based on a set of ad hoc criteria, hence these comparison criteria may not fit the SRE needs of every company. Thus, a systematic method must be proposed to properly capture the actual company’s needs regarding an SRE methodology.
In this article, we address this issue by proposing a requirements engineering-based evaluation methodology. It helps in characterizing the appropriateness of a SRE methodology by capturing the SRE needs of the stakeholders as well as the quality characteristics of good security requirements (a.k.a. quality criteria). We developed this evaluation methodology during the IREHDO2 project where we could interview security experts involved at each step of the security process. In this project, security experts of an aircraft company wanted to improve their security process in order to increase the assurance on the final security solution enforced on their aircraft networks. More precisely, they were interested in enhancing their security requirement practices. This group of security experts included security requirement engineers, risks analysts as well as security testing experts who are involved at different levels of the security process. Our task in this project consisted in proposing the best SRE methodology which will help them in writing good security requirements. However, each security expert had a different point of view on what could constitute a good SRE methodology. As consequence, setting generic evaluation criteria appropriate to any organization is not a good idea. A better approach is to develop a process to elicit these evaluation criteria and their weight from the stakeholders. We applied this methodology to our project context. This article consolidates and extends previous contributions [
12,
13,
14,
15] to provide in a single and self-contained document explaining the whole methodology. The contributions are as follows:
The description of a whole methodology for evaluating SRE based on a set of generic evaluation criteria that shall be refined for each organizations.
The application of this methodology to the IREHDO2 project context.
An evaluation of three SRE methodologies based on the feedback from security experts.
The rest of the article is structured as follows. In
Section 2, we provide an overview of the existing comparative studies. Next, we present our proposed evaluation methodology and its strategy in
Section 3. In
Section 4, we present a deep study of the characteristics of good security requirements. This work provides a basis for eliciting the characteristics of a good SRE methodology. In
Section 5, we describe a process and a tool to elicit SRE methodologies evaluation criteria. Each evaluation criterion must be associated with an evaluation metric. In
Section 6, we describe the implementation of our evaluation methodology in the context of IREHDO2. We present our evaluation process to evaluate three SRE methodologies (i.e., STS [
16], KAOS [
17] and SEPP [
18]). At the end, we highlight the pros and cons of each of these three SRE methodologies elicited characteristics serving as evaluation criteria. Finally, we conclude and present our future works in
Section 7.
2. Related Works
In the literature, there exists numerous works related to comparative studies that offer varying dimensions of SRE evaluation perspectives. In below, we first present an overview of the related works and later we enlist the three main issues that we identified in their evaluation perspectives. Note that, our literature review does not provide the synthesis or classification of survey studies such as in [
19,
20].
N. Mayer [
21] provided a comparative study of the SRE methodologies based on a domain model consisting of 14 security concepts. These concepts are categorized under three groups: asset-related (e.g., business asset, system asset), risk-related (e.g., risk, impact, threat, threat agent and attack method) and risk treatment related concepts (e.g., security requirements, control).
Fabian et al. [
22] proposed a conceptual framework to support the comparative study of existing SRE methodologies. When compared with the modelling framework in [
21], this work highlights similar concepts related to risk analysis but also extend them with granular security analysis concepts based on the security confidentiality, integrity and availability properties. In addition, it considers the concepts related to the multi-lateral security requirements analysis that stresses on the compromise between conflicting stakeholders’ security needs [
23].
Munante et al. [
24] extended the previous comparison frameworks [
21,
22] with additional evaluation concepts related to model-driven engineering (e.g., tool support, SRE modelling language, formalism). This work concludes that KAOS [
25] and secure i* [
26], are the most compatible for formal SRE modelling.
Souag et al. [
27] proposed a comparison framework to provide a systematic mapping of the reusable concepts and patterns within the existing SRE methodologies. Accordingly, research contributions were classified into 5 categories: security patterns, taxonomies and ontologies, templates and profiles, catalogues and generic models and miscellaneous.
Uznov et al. [
28] proposed a comparative analysis of security engineering methodologies using a list of 12 characteristics of security methodologies. This work provides guidelines on the selection of methodologies upon their comprehensiveness, applicability and uniqueness from the perspective of their adaptability to an industrial use.
Mellado et al. [
29] provided a systematic review of comparative studies to analyze the internal/external verification and documentation support of the SRE methodologies. The evaluation is principally based on the quality characteristics of good requirements (e.g., traceability, comprehensibility, etc.) referred from the internal standard IEEE 830 [
30].
N. Mead [
31] provided a comparative analysis of the requirements elicitation techniques based on some criteria such as learnability, client acceptance and durability of the requirements elicitation techniques, tools support etc.
Nhlabatsi et al. [
32] proposed a comparative study of SRE methodologies in order to evaluate their support capability regarding the evolution of secure software during the change management process. Accordingly, the criteria address different perspectives such as the modularization, component architectures, change propagation and change impact analysis.
Niazi et al. [
33] developed a Requirements Engineering Security Maturity Model to assist software development organizations to better specify the requirements for secure software development. They conducted a questionnaire survey based on security requirements practices derived from Sommerville’s requirements engineering practices [
34] to end up with 79 practices classified into 7 categories. This work has the same approach involving stakeholders. However, they focus on assessing the maturity level of security requirements practices, while we aim at helping organizations in choosing a security requirement engineering methodology.
Although these related works discussed many interesting strategies and aspects, they are not sufficient to evaluate the goodness of SRE methodologies because:
Issue A: Evaluation criteria coverage. SRE methodologies aim at covering the whole SRE process (i.e., elicitation, evaluation and documentation) [
22]. From our study, we noted that the majority of the evaluation studies except [
29] lack a full emphasis on the whole SRE process [
35]. While some works [
21,
22,
31] concentrate on evaluating the extent of support to security requirements elicitation at earlier stages; some others [
27] focus on evaluating the extent of support to documentation in terms of reusable patterns or modelling initiatives.
Issue B: Evaluation criteria affirmation. The evaluation criteria must result from a systematic process or standard method, such as [
36], to facilitate reusability. However, in the majority of works, the proposed evaluation criteria were ad hoc and lack affirmation on why the criteria were good enough to be considered for the evaluation [
28,
31,
32]. In this aspect only [
29] can be noted as an exceptional case as the evaluation is based on the international standard [
30].
Issue C: Lack of stakeholders’ involvement. This issue concerns the involvement of stakeholders’ views while formulating the evaluation criteria. As highlighted in [
37], the requirements process is a human endeavour, and so the requirements method or tool must be able to support the need for stakeholders to communicate their ideas and obtain feedback. In our context, stakeholders refer to the SRE methodology users (e.g., requirements engineers). However, from our observation we found that none of these comparative studies involved SRE experts while formulating the evaluation criteria for an SRE methodology. Moreover, almost all of the works acknowledge (either implicitly/explicitly) the significance of involving stakeholders’ perspectives in the SRE process. For instance, [
22] includes a criterion that concerns the integration of stakeholders’ views to support multilateral security requirements analysis.
Altogether, we can conclude that there exists no generic evaluation methodology that can help in identifying the best suitable SRE methodology. Evaluation of an SRE methodology is an enduring problem. As long as new SRE methodologies keep arriving, this necessity of evaluation persists. Having this in mind, we decided to develop a generic evaluation methodology independent from the SRE context of use so that it can be instantiated to varying SRE contexts (e.g., network security, software security).
4. How to Qualify Good Security Requirements?
The common goal of any research contribution in the SRE community is ultimately to facilitate capturing good security requirements [
29,
39,
40,
41]. If the security requirements are error-prone (i.e., ambiguous, incorrect, inconsistent) then the security design will be ineffective regardless of the successful implementation of efficient security solutions [
42]. For instance, let’s take as an example the security requirement that states: “The data flow between device1 and device2 must be encrypted by a strong encryption algorithm”. This requirement is ambiguous, because it does not explicitly define what a strong encryption algorithm means. This ambiguity may lead to false assumptions concerning the rigor of security protection, which eventually impacts the decisions related to the secure solutions.
The RE literature features numerous works describing different characteristics of good security requirements (a.k.a. quality criteria) such as unambiguous, traceable, consistent, etc. These characteristics help to verify and validate the quality aspects (i.e., the goodness) of security requirements. Nonetheless, there exists no consensus on how to characterize a good security requirement. In addition, there is no one complete and consistent set of characteristics [
43].
In this context, we studied 185 definitions from 17 literature works depicted in
Table 1. Different sources have listed different sets of criteria defining different characteristics of good requirements. The objective of this survey was to identify and study all the concepts discussed in each of the definitions, thereby analyzing the possibility to reach a consensus.
Since, the list being huge and the definitions complex, we developed a semantic graph-based analysis tool using the python graph module NetworkX [
55]. This allows us to formalize the definition and analyze the similarities or differences between the definitions. The nodes in the graph represent the criterion names and the concepts concerning the respective definitions. The edges correspond to the reference links expressing the semantic relation between a criterion and its associated concepts or between some criteria. For labelling the reference links (edges), we used a standard format by associating the abbreviation of the criteria with the source ID of the respective authors in
Table 1 that propose the criterion. This ensures the unique interpretation of the reference links in the graph. For instance, the edge labelled M8 from node “modifiability” and node “express each requirements separately” means IEEE 830 1998 [
30] (Source ID 8) refers to expressing each requirements separately in the definition of modifiability.
Figure 2 shows the complete view of the resulting semantic graph generated using Gephi tool [
56]. The first result of this analysis produced a complex graph containing 212 nodes and 278 edges. The graph projects a total of 71 criterion nodes and 141 concept nodes. Every node is connected in the graph, highlighting the complexity of the semantic dependencies between the quality criteria. Such graph exhibits similarities between criteria. Indeed, when two criteria C1 and C2 refer, respectively, to a set of concepts D1 and D2 in their definitions such that D1 ⊆ D2, thus we can conclude that criterion C2 is more generic than criterion C1. In addition, the semantic graph showed ambiguities in some definitions that make self-references.
Figure 3 depicts an example of ambiguous definitions in which the definitions precise (PS), concise (CS) and clear (CL) from the authors result in a cyclic referencing. This opens a new problem that might require a proper research to analyze the semantic meaning of each criterion definition.
In our evaluation methodology context, our objective is defining an exhaustive list of the existing characteristics that can be integrated within any SRE process. Suitably, we needed something simpler that can be used as a standard basis for eliciting the characteristics of a good SRE methodology. The semantic graph helped us in identifying a total of 20 distinctive criteria definitions. However, we have observed many variations in their corresponding definitions (e.g., different authors, for similar criteria, have defined different names). This entailed into defining a weaving methodology to express these variations.
The main goal of our weaving methodology is to showcase the similarities in the criteria propositions of the unified list in a tabular format that featured a weaving strategy.
Table 2 consolidates our weaving results, which extends the results in [
12]. Firstly, we gave a unique identifier as a reference to each class of quality characteristics. We denote the term criterion (represented as C) that gives us a list of criteria as C1 to C20 (see
Table 2). In addition, we added a one-line definition to each criterion. If the characteristic is defined in ISO 29148, we use the standard definition. Otherwise, we take references from respective authors if the characteristic description is straight forward (e.g., C8). When the characteristic descriptions seemed ambiguous (e.g., C10), we provide our own interpretation.
Secondly, we used colors (i.e., blue and orange) and typographical emphasis (i.e., bold, underlined and italic) to differentiate some special cases as follows: Criterion definitions are highlighted in blue color when defined by only a single author (e.g., C13). When different authors employ different names to describe similar quality characteristics (e.g., C3), then the respective criterion definition is emphasized in italic text. When a single author employs different names to describe similar quality characteristics (e.g., C3 by S13), then the respective criterion definition is emphasized in orange. Contrarily, when the same criterion name is used to describe different quality characteristics (same or different authors), then the characteristic name reflecting the uncommon mapping is underlined (e.g., C6 by S5 and C15 by S15).
Thirdly, we distinguish the applicability of a criterion. If mentioned
All the Applicability column in
Table 2, the respective quality characteristic requires to verify the whole set of requirements specifications document (e.g., C2). If mentioned
Each, the respective quality characteristic targets only a single (or a specific set of) requirement(s) (e.g., C20). See the Applicability column in
Table 2. Finally, we defined credibility scores to show the frequency of citations. We used qualitative scale high, medium and low. Credibility is given
high when cited by at least 15 authors;
medium with at least 8 authors; and
low when at least 4 authors;
very low when cited by less than 4 authors.
Altogether,
Table 2 overviews our comprehensive review of the non-consensus issue concerning the characteristics definitions.
Criterion C1 ensures that final set requirement specifications sufficiently express all the needs of stakeholders, respecting all the considerable aspects and scenarios. The difficulty in fulfilling this criterion corresponds to identification of all considerable aspects such as stakeholder security and risk management objectives. In a way this criterion insists on efficient requirements elicitation and risk analysis process. The common keyword used to represent this criterion is complete with credibility high. Applicability of this requirement implies either to an entire set of requirements or to a set of requirements.
Criterion C2 ensures that all requirements are compatible and consistent with one another. Accordingly, this criterion C2 insists on verifying if there exist any conflicts in terms of contradicting requirement statements, improper representation of viewpoints or possibility of incompatible interpretations of a statement, etc. The difficulty in fulfilling this criterion corresponds to establishment the right level of trade-off. This criterion indirectly contributes to satisfaction of requirement completeness. The common keyword used is consistent with credibility as high. Applicability of this criterion concerns with either an entire set of requirements or a group of requirements.
Criterion C3 ensures that all those derived requirements within the document are accomplishable within the given constrains. ISO defines some of the considerable constraints such as time, cost and process control, financial, technical, legal and regulatory. In addition, dependency constrains and domain constraints [
17] can also be considered. Constraints can be viewed in two ways, one as they are imposed by stakeholders and the other based on operational context. On the whole, this criterion insists on identifying and acquiring all the possible constraints in terms of financial or technological implementations. Applicability of this criterion concerns with either an entire set of requirements or a group of requirements or an individual requirement. Credibility of this criterion is also
high. The common keyword used is feasible, affordable and other keywords are realism and legal.
Criterion C4 ensures that all requirements within the document are well categorized and well documented in a structured manner so that it is maintainable with fewer changes. Credibility of this criterion is low and common keyword used is structured.
Criterion C4 ensures that all requirements within the document are prioritized and well documented in a structured manner. Credibility of this criterion is low and common keyword used is structured.
Criterion C5 ensures that specified within the document are traceable in both forward and backward ways. Credibility of this criterion is high and common keyword used is traceable. Some sources have highlighted different aspects in the same context; hence different keywords were used accordingly. The keywords are cohesiveness, allocated, satisfied/qualified.
Criterion C6 ensures requirements derived do not specify the implementation details of the solution instead it specifies what is needed. Credibility of this criterion is medium and the keywords used are implementation free, external observability, design independent and abstract.
Criterion C7 ensures that the document containing all set of derived requirements is modifiable and adaptable to changes. It is to note that this is similar to a Meta characteristic to criterion C4 (well structured). Credibility of this criterion is medium and the keywords used are modifiable, adaptability.
Criterion C8 ensures that there is no redundancy of information corresponding requirement needs. It insists during the requirements elicitation process, one must clearly be able to distinguish between redundant stakeholder needs and non-redundant stakeholder needs. Credibility of this criterion is low and the common keyword used is non-redundant.
Criterion 9 ensures that all the requirements in the document are uniquely identifiable. This criterion helps to achieve the traceability feature (C5). Credibility of this criterion is low and the common keyword used is unique.
Criterion C10 ensures that completeness feature of an individual requirement. In a way it insists on verifying if the stakeholder need is sufficiently elicited. Credibility of this criterion is low and the keywords used are adequacy and validity.
Criterion C11 ensures that requirements are derived using simple terminology without usage of technical jargon. Technical jargon corresponds to terminology used by different teams working in different areas of business operational environments. For example, terminology used in software development environment is difficult to be understood by individuals belonging to organizational environment. Hence, this criterion enforces that the derived requirement must be comprehensible to all the readers of the document with in the business environment. Credibility of this criterion is medium and the common keyword used is comprehensibility. Some sources have highlighted different aspects in the same context; hence accordingly different keywords used. They are customer or user orientation and clear.
Criterion C12 ensures that the derived requirements are precise enough and does not lead to any misinterpretations. It is to note that this criterion is different from the previous one C11 (comprehensibility). C11 insists on the aspect that there is no difficulty in the comprehension of the text (phrase or sentence), in the way it was written (focus on terminology). In addition, C12 insists on the aspect that the content of the text maintains careful precision while expressing the idea so that it does not lead to misinterpretation of the idea. In end, this criterion emphasizes on the verification that comprehension of the text is not wrong. It focuses on punctuation and meaning of terminology or vocabulary used. In a way, this criterion can be viewed as a meta-characteristic of the criterion C11. Credibility of this criterion is high and the common keyword used is unambiguous. Another keyword used is precise.
Criterion C13, it ensures that requirements derived can be measured with some quantifiable values. For example, consider a requirement need “a service must be available to all the customers”. This need cannot be measured and while eliciting such needs, it is important to elicit measurable information. For this derived requirement for this need can say “a service must be available on an average to ‘x’ number of customers at ‘t’ units of time”. This way, the requirements can be measured. Credibility of this criterion is low and the common keyword used is measurable.
Criterion C14 ensures that the derived requirement specifies what is needed and it has not got any unnecessary information. It is to note that this criterion complements the criterion C10 (adequacy). Credibility of this criterion is medium and the common keywords used are necessary and mandatory. Some sources have highlighted different aspects in the same context; hence accordingly different keywords used. They are bounded, pertinence and relevance.
Criterion C15 ensures that requirement must possess accurate and up to date information. Credibility of this criterion is medium and the common keyword used is correct.
Criterion C16 ensures that one requirement derives one need. For example, if a requirement need says “entrance to aircraft allowed to customers with boarding pass and special emergency pass”. This is not singular or atomic in nature. It is speaking allowing customers of two different types. One can split this into two as: “entrance to aircraft allowed to customers with boarding pass” and “entrance to aircraft allowed to customer’s special emergency pass”. This way it helps to defined more precisely what does it mean by saying special or emergency. Accordingly, we can say that this criterion C16 contributes towards C13 (measured). Credibility of this criterion is low and the keywords used are singular and atomic.
Criterion C17, it ensures that each of the requirements is verifiable against the constraints, standards and regulations to ensure the correctness of the requirements. This criterion somewhere again falls between C10 (adequacy) and C15 (Accurate). Credibility of this criterion is high and the common keyword used is verifiability.
Criterion C18, it ensures the formulation of requirement must follow some standard so that they are understandable globally. Credibility of this criterion is low and the common keywords used are requirement language criteria and devoid of escape clauses.
Criterion C19, it ensures that requirements must be formulated in such a way that they are reusable. This criterion emphasizes on the using some common pattern for similar type of requirement needs. Credibility of this criterion is rare and the common keyword is usability.
Criterion C20, it ensures that every requirement should be identified with some metadata such as attributes, acceptance criteria. This way, it facilitates in their validation and evaluation. Credibility of this criterion is rare and the keyword used is Metadata.
Since the interpretation of a definition can differ between different people with varying expertise levels, the table contents can raise several questions. This implies that our weaving methodology strategy provides a way to trigger a discussion between experts. This will lead to brainstorming, which serves our purpose of initiating the elicitation process of our RE based evaluation methodology.
6. Evaluation of Existing SRE Approaches
This section describes the final step of our SRE evaluation methodology. It concerns the assessment of the SRE approaches against the evaluation criteria refined in step2. Our evaluation is a qualitative assessment that is performed in collaboration with the security experts. We organized two meeting sessions (i.e., precisely 3 h brainstorming and discussions for each) with a time gap of around 2 months.
We performed the evaluation in two iterations. In the first iteration, we—the authors of this article—acted as stakeholders (i.e., SRE analysts) and tested three SRE approaches, i.e., STS, Secure KAOS and SEPP using an example scenario. First, a description of the system-to-be is presented to three different persons (playing the roles of RE analysts) from our research group whose initial knowledge fits the aforementioned methodologies the best. Then, each person has been asked to elaborate the requirements for the system-to-be using the methodology that was assigned to him/her. During this process, the persons (acting as SRE analysts) were not allowed to communicate with each other during the scenario analysis phase. Each of them had come up with a different list of security requirements for the system-to-be. The resulting SRE models have been presented during a meeting that involved four security experts. Their feedback was recorded as our initial evaluation results, which enabled us to identify and select the SRE approach found interesting to the experts at first instance.
In the second iteration, the actual security experts are directly asked to test the SRE approach selected during the first iteration. The feedback of this second iteration enabled us to finalize the evaluation results. In each iteration a different use case scenario is employed in order to ensure the credibility of the capture of the evaluation experience. In below, we provide the details of the evaluation performed in two iterations.
6.1. Evaluation Iteration 1—Use Case Scenario 1
We first present the use case scenario employed in the first iteration. The scenario concerns a task related to the aircraft maintenance process ensuring compliance to some safety regulations. The objective of this task is to monitor the on-board aircraft system by verifying specific parameters (e.g., fuel indicator) in order to ensure the readiness of the aircraft prior to taking the next trip. The Onboard aircraft system consists in two applications, i.e., aircraft control application and monitoring application (Cf.
Figure 5). These two applications are connected to each other via an internal avionic bus network (represented by a red double arrow line in
Figure 5).
The aircraft control application captures the observed parameters from various sensors and indicators of the aircraft system and transmits them to the monitoring application. Every time the aircraft has landed, the responsible person (i.e., maintainer) must connect his/her laptop to the monitoring application in order to fetch the monitored parameters. However, there is no direct access to the monitoring application. The connection is permitted only using the secured network connection within the secured premises of the aircraft. The maintainer must carry the laptop to the airport ground in order to access the network. In this process, the maintainer is assumed to be trusted, while the laptop is untrusted.
The business risk impact of this scenario is expressed in terms of integrity and availability of the monitored parameters. These parameters influence the critical decision process that decides whether the aircraft is ready to fly again or has to be retained for further inspection. Non-availability of these parameters may delay the inspection process that may incur loss in terms of reducing the number of trips per day. While the lack of integrity may cause serious problems when the faulty aircraft is assumed as correctly functioning and permitted to take next trip.
Therefore, the aircraft network security design solutions must ensure a trusted transmission of parameters from the monitoring application to the aircraft control application and then to the laptop of the maintenance people. It should also ensure that the access to the aircraft control application is restricted and the laptop is allowed to connect to the monitoring application only. This implies network security requirements to closely monitor communication traffic between the laptop and the monitoring application. In this regard, some example security measures may include: a VPN connection with a strong encryption algorithm, configuring firewalls, authentication on the WIFI access point or router. The security experts wanted to confirm that these measures are sufficient to mitigate the risks. In addition, they wanted to verify the cost of the network security solutions to compare different choices; e.g., maintenance people can potentially connect to the aircraft using an Ethernet cable or a wireless connection.
The first iteration concerns the implementation of the aforementioned scenario using three SRE methods: Secure KAOS, STS and SEPP. The discussion part consolidates our observations on the SRE modelling experience of each method. We highlight the relative evaluation criteria in parenthesis to facilitate the understanding of which part of the observations are subjects to which evaluation criteria. Finally, we provide the initial feedback and evaluation results of the first iteration.
6.1.1. Scenario Implementation Using Secure KAOS
KAOS [
17] (Knowledge Acquisition automated Specification or Keep All Objectives Satisfied), is a goal-oriented RE methodology developed as a joint collaboration of research work between the University of Oregon and the University of Louvain (Belgium) in 1990. The KAOS methodology mixes the top-down and bottom-up approaches. The goal modelling activity focuses on eliciting goals and refining them into sub-goals until they are atomic. Goal refinement is specified via AND/OR constructs. When a goal cannot be refined further, it is called as a requirement of the system-to-be and is assigned to an agent (to either environment or system agents) represented. When a requirement concerns an environment agent (e.g., human), it is called an expectation. Altogether, identifying and refining security goals and assigning them to agents are the core activities that are represented as goal and responsibility models. Authors of KAOS propose a formal specification language based on temporal logic. They also specify formal generic refinement patterns to validate goals refinements.
Our experimentation was conducted using the free trial version of Objectiver [
60] the official tool maintained by Respect-IT.
Figure 6 depicts a sample of the goal model specified in our example scenario context. It required some effort to become acquainted with the tool and its terminology through the help of guidelines and cited references (refer to R
M2.1.1 and R
M2.1.3 criteria). In particular, the security experts expressed some concerns regarding the formal temporal logic language (R
M2.1.1). They prefer the textual notation even it is more ambiguous.
KAOS drives the RE analyst to define agents later in the RE process. As a consequence, it does not help in expressing the relation between the agents and their interaction dependencies which is important for the network design. For instance, while defining network agents (e.g., an access control system) in our scenario analysis, we encountered issues when we needed to add a new device in the network architecture. It was difficult to express the dependencies between network agents. In this regard, a conceptual diagram of the network would have been more concise and clearer. As a consequence, we had a difficulty in specifying network security requirements on network agents, particularly relative to security zoning (RM3.1). Furthermore, KAOS provides features to express abstractions and the notation provides a good support to achieve traceability by linking goals and sub-goals (RM4.1). However, it does not guide the security requirement engineers in asking the right questions at the right time to structure the security engineering process (RM3.2).
Regarding integration of risk analysis concepts, the link between security requirements and risk information is explicitly expressed using the concepts of obstacles/anti-goals that are represented as relations such as obstruction/resolution. For example, in
Figure 6, security goal “PROTECTION NEED: monitoring data should be available when connected” resolves anti-goal “maintenance unavailable”. The resolution link is expressed using green arrowhead. These anti-goals can be further refined such as ‘normal’ goals resulting in the specification of attack trees. Obstacles/Anti-goals include two risk attributes likelihood and criticality (R
M6.2) while security goals are attributed with ordinal priority scale (R
M6.3). However, there is no explicit relationship defined between the priority of a security goal/requirement and the risk of an associated obstacle. In addition, it helps in observing the environmental constraints upon the goals through domain properties. (e.g., physical laws). For instance, the trust assumptions on agents are expressed using the domain properties (Cf.
Figure 6).
6.1.2. Scenario Implementation Using STS
STS (Secure Socio-technical Systems) [
16] implements the agent-oriented approach which mainly focuses on early elicitation of security requirements based on the social dependency interaction relationships. Similar to KAOS, the STS framework offers the possibility to create composite goals via the AND/OR constructs. However, the respective goals and sub-goals are determined in the scope of each actor. An actor can be either a role or an agent. This methodology mainly defines three views: social view, authorization view and information view. The social relationships between actors are manifested by the relationships such as goal delegation and resource provision. This is modelled in the social view. Delegated goal implies the dependency of the interaction. An actor depends on another actor to achieve a goal. In addition, the social relationships between actors may also include the exchange of documents (informational resources) that contain necessary information for the achievement of a goal. STS captures this exchange via the relationship: document transmission (resource provision). The ownerships of the resources are modelled in the authorization view. This view also permits to explicitly define the authorization rules over the access permissions on the documents (resources). Another view called the information view allows to define the primitives and relationships that permit the differentiation between the information (content) and the representation of the information (document).
STS is supported by a tool of the same name. The tool is not open source but it is freely available to public use (RM2.1.2). The usage of a simple terminology as well as the user-friendly tool took less effort to become used to the overall concepts and terminology (RM2.1.1 and RM2.1.3). The security experts provided a positive feedback concerning the comprehensibility aspects during the first intermediary meeting.
Unlike KAOS, STS does not support full-fledged traceability of the security goals being refined into security requirements (R
M4.1). From the STS point of view, the security needs are implicitly expressed either in the form of security constraints attributes (e.g., integrity/availability/non-repudiation, etc.) over the interaction dependencies in the social view; or in the form of constrained permissions (e.g., read/modify/produce/transmit) upon the provision of the resources in the authorization view.
Figure 7 highlights the security constraints over delegated goals in blue dashed rectangles, which cannot be refined further into network security requirements (R
M3.1).
On the other hand, STS provides a good support to express network agents and interaction dependencies. However, we had an issue in handling the number of dependencies between multiple network agents as the social view grows. Furthermore, from the STS modelling perspective, the abstractions are expressed using the three modeling views (i.e., social view, authorization view, information view). This abstraction perspective respects separation of concerns specific to authorization security problem context only. Similarly to KAOS, STS does not guide the security requirement engineers in asking the right questions at the right time to structure the security engineering process in order to build a security architecture (RM3.2).
In STS, the threat analysis can show the effects of a threating event over goal trees and goal/resource relationships. In addition, it defines attributes (implicit) to link countermeasures to threat events (R
M6.2). However, the threat analysis is limited to threat event propagation. Thereby it does not facilitate the expression of security needs upon the threats related to environmental risks. In addition, it does not have a facility to prioritize goals or threats such as in KAOS (R
M6.3).
Figure 7 shows that the threat event “maintenance unavailable” threatens the goal “make aircraft maintenance”. Likewise, the threat event “bad maintenance” threatens three goals. When the threats are propagated, the threat impact traces back to the root objective node. Respectively, the “maintenance unavailable” threatens the business objective “run business”.
6.1.3. Scenario Implementation Using SEPP
SEPP (Security Engineering Process using Patterns) [
18] implements the problem frames approach. In contrast with STS and KAOS, this approach views the stakeholder’s requirements as constrained behavioral characteristics expected within the real physical world. A machine represents the system-to-be that helps to achieve the desired behavior. Problem world represents a part of the real world, in which the machine operates. Problem Domains represent different active entities (agents) in a given problem world upon which the machine operates in order to achieve the desired behavior. Casual domains (noted C in
Figure 8) represent system agents. Lexical domains (noted X in
Figure 8) represent data. Biddable domains (noted B in
Figure 8) represent human agents. A problem is characterized as dependencies in terms of the shared phenomena (agents interaction attributes) between the machine and the problem domains, in a given problem context. A problem frame is therefore a problem pattern representing the common characteristics of a recognized class of problems. It separates the problems from its solution.
SEPP extends the concepts and terminology of the PF to security problem analysis in a particular SRE context. It guides the RE analyst to define security problem patterns, called as security problem frames (SPF), separately from the security solution patterns, called as concertized security problem frames (CSPF). Accordingly, they have introduced four security problem frames (SPF) and eight concretized security problem frames for authentication, confidential data transmission, distribution of secrets and integrity preserving data transmission. It is to note that, CSPF are the next abstraction level instantiations of the security problem frames which define some generic security mechanisms to solve security problems.
Figure 8 depicts a sample of our secure problem frames specification for integrity protection scheme. Security requirements are constrained on the communication network domain. The links between domains depict that there are some shared phenomena (i.e., shared attributes). The non-availability of a tool for SEPP made our experimentation hard (R
M2.1.2). It took some extra effort to become acquainted with the concepts and terminology even with the help of the cited references (R
M2.1.1 and R
M2.1.3). As a consequence, we have designed all the SPF and CSPF patterns models (for the scenario) manually which consumed more time (R
M2.1.3). In addition, there is a traceability issue (R
M4.1) as well. During our experimentation, we had other issues in knowing all the acting domains in a network environment, in particular during the early stages. A network design in hand will facilitate the problem analysis.
From the SEPP perspective, security problems (security goals) are identified using the what-if analysis technique similar to the Hazard analysis [
61]. However, it does not provide explicit linking between the risk and the associated security goal (R
M6.2). This makes the risk definition implicit. The constraints on the security requirements are expressed in terms of pre-conditions attributes of SPF. These are the formalized conditions that must be satisfied by the problem environment “on prior”, i.e., before applying the security problem frame. Similarly, the post-conditions attributes correspond to the formal expression of the security requirements.
Furthermore, SEPP facilitates a two-level abstraction through separation of security problems from solutions in generic perspective. However, this abstraction perspective respects the separation of concerns specific to the authorization security problem context, which does not help to explicitly describe the separation of concerns of the agents/actors specific to the network security problem context (RM3.2). Lastly, similar to STS, it does not support attribute security requirements with either priority or implementation cost related information (RM6.3 and RM6.4).
6.1.4. Evaluation at Iteration 1—Results and Discussion
In
Table 6, we resumed the evaluation results of the SRE methodologies to provide a consolidated view for comparison. First column displays the high-level root goals of evaluation criteria (Cf.
Figure 4). Second column lists the elicited criteria that correspond to the leaf nodes in
Figure 4. The remaining three columns display the respective measurement scales for each of the three SRE methods.
Each approach exhibits different features. STS is interesting when it concerns comprehensibility factors (i.e., RM2.1.1, RM2.1.2, RM2.1.3), Secure KAOS is interesting in terms of traceability aspects (RM4.1) as well as integration of risk analysis information. On the other hand, the modelling terminology and notation of SEPP seemed difficult to learn and adapt. Therefore, comparatively, STS and secure KAOS seem to be more suitable to our SRE context compared to SEPP. However, some of our requirements’ RM were not supported by any of the three methodologies (i.e., RM3.1, RM3.2, RM6.1 and RM6.4).
The criteria RM3.1 and RM6.1 are notably related to the network SRE context that concerns the integration of network security zoning. Without this support, it would be difficult to elicit network security requirements at early stages. Furthermore, the requirement concerning the abstraction characteristic goal cannot be respected by the three SRE methods (RM3.2). As a consequence, expressing the security problem context of all the stakeholders including the business analysts, and the security analysts of all systems connected to the network is still an issue. Finally, none of the three methodologies allows to help in considering the cost of the security requirements (RM6.4).
6.2. Evaluation Iteration 2—Use Case Scenario 2
During the first iteration of the evaluation, the authors of this article acted as stakeholders and tested the SRE models, which were then presented to the actual stakeholders (i.e., security experts) to capture their comments. The second iteration encompasses active participation of stakeholders for evaluating the STS modelling notation that holds highest rating in terms of comprehensibility aspects. The purpose of this second evaluation is capturing the direct feedback from real users’ experience of SRE modelling in STS using a new use case scenario. In below, we first provide the scenario in plain-text description as it was presented to us at the initial step.
The scenario contains two business needs (Cf.
Figure 9). The first business need concerns the safety measures related to the aircraft. It implies that the aircraft needs to be updated with correct load information through a portable device. The update center is a sub-component in the aircraft equipment, which reads these load parameters and performs an auto-update. The aircraft network security design must ensure the secure transmission of these parameters between the mobile devices to the on-board update center application. In addition, the direct connectivity of the mobile devices with the aircraft equipment is forbidden.
The second business need concerns the in-flight entertainment services to improve passengers’ inflight condition. It describes the necessary functionalities of the aircraft equipment (e.g., passenger entertainment panel) to entertain the passengers during their flight journey. For example, one functionality is to provide the real-time flight status to the passengers. This functionality requires connectivity between the aircraft control panel, centralized on-board entertainment system component and the passengers’ equipment panels. The security needs concern the availability of the aircraft’s information (e.g., speed, altitude, temperature, distance, etc.) to the passengers accessing the respective entertainment system display panels. Respectively, the aircraft inflight network security design must ensure the availability of correct information to the passengers.
6.3. Evaluation Iteration 2—STS Modelling and Feedback Discussion
From our initial analyses, we highlighted that STS provides no support to describe separation of concerns. In this experiment, we investigate the integration of SABSA layered approach to STS modelling in order to improve the separation of concerns.
SABSA [
9] (Sherwood Applied Business Security Architecture) is a methodology framework for enterprise security architecture and service management. The main intention of this framework is to guide security architects in their task. It defines a six-layered architecture model; each layer reflecting the respective stakeholder view in the process (
Figure 10). The Business view and Architect’s view together correspond to the strategy and planning phases of the system engineering process. The Business view deals with the elicitation of high-level security objectives and constraints that are important to do business. It is essential to analyze the context of the business before building a secure system. This view facilitates the specification of the business security needs (e.g., reputation), operational goals and risk impacts (e.g., monetary loss) that drive the necessity of a secure system. The Architect’s view concerns the integration of protection objectives (e.g., application security services, network security services, data security services, etc.) that are necessary to secure the business security needs. The Designer’s view and builder’s view together, correspond to the designing phases of the system security engineering process. The Designer’s view deals with the specification of the security services (e.g., network entity authentication, data integrity protection, etc.) that are necessary to satisfy the control objectives. The builder’s view deals with the mapping of the technical security mechanisms (e.g., firewalls, hashing, etc.) to the logical security services. Finally, tradesman view (LV5) corresponds to the final implementation phase of the system security engineering process. The latest view called operational manager’s view corresponds to the maintenance management of the system security engineering process after the deployment.
SABSA is not strictly a security requirement engineering approach. This method provides a broad picture to understand the conventional security requirements engineering process starting from the perspective of the business system security architecture. The SABSA also includes a matrix that describes a wide variety of aspects that a modeler should be able to express (Cf.
Figure 11) with the help of six interrogative questionnaires (what/why/how/who/where/when). This matrix guides the security architect in asking the right questions at the right time. Thus, it structures the security engineering process.
In SABSA, network security requirement analysis (e.g., related to security zoning) starts from the designer’s view (i.e., LV3). Since we demonstrated in the first iteration of evaluation that STS does not support to derive network security zoning; therefore, we confine this second experiment to the first two views of SABSA only (i.e., Business view and Architect’s view). This implied that any discussion related to network security requirement analysis has been purposefully discarded. This strict confinement to the high-level abstraction improved the elicitation activity by provoking a long discussion. Indeed, this strict confinement to the SABSA abstraction has helped the team in focusing more on ‘WHAT’ perspectives of security needs rather than ‘HOW’ perspective of security needs. During this experimentation, we observed that the elicitation part of the scenario seemed clear and provoked a useful discussion.
Initially, the description of this scenario fits in just few lines. However, during the process of SRE modelling, the provoked discussion has developed the scenario details which resulted in an STS social model. In particular, new agents were introduced that were not mentioned during the textual description of the scenario. In addition, we observed that the goals of the maintainer agent were refined and improved as well. However, the second business need related to the passenger’s entertainment system remained unchanged.
The overall feedback of the STS notation from the two iterations confirms that STS modelling is interesting and easy to adapt. The overall experiment lasted 2 hours, which proves that STS modelling concepts can be learnt quickly (RM2.1.3). However, the STS modelling requires the users to know the agents and their respective initial goals before beginning the process. Our experiment shows that it is not the case because information on agents as well as their goals is evolving gradually during the elicitation process.
Although STS modelling concepts are interesting and easy to adapt, it does not help to trace the gradual growth of the social model, i.e., where to begin or how agents are introduced in the model. In addition, considering the traceability or risk analysis aspects, STS provides less support while compared to Secure KAOS (see
Table 6). Indeed, the first two layers of SABSA modelling emphasize on risk analysis and risk control objectives. Therefore, STS modelling language is an interesting choice to apply together with the SABSA layering views. However, it requires some enhancements in terms of integrating risk analysis concepts and traceability attributes.