Probabilistic Measurement of CTI Quality for Large Numbers of Unstructured CTI Products

Abstract

This paper addresses the critical challenge of evaluating the quality of Cyber Threat Intelligence (CTI) products, particularly focusing on their relevance and actionability. As organizations increasingly rely on CTI to make cybersecurity decisions, the absence of CTI quality metrics challenges the assessment of intelligence quality. To address this gap, the article introduces two innovative metrics. Relevance ($Re$) and Actionability ($Ac$) are designed to evaluate CTI products in relation to organizational information needs and defense mechanisms. Using probabilistic algorithms and data structures, these metrics provide a scalable approach for handling large numbers of unstructured CTI products. Experimental findings demonstrate the effectiveness of metrics in filtering and prioritizing CTI products, offering organizations a tool to prioritize their cybersecurity resources. Furthermore, experimental results demonstrate that, using the metrics, organizations can reduce candidate CTI products by several orders of magnitude, understand weaknesses in defining information needs, guide the application of CTI products, assess CTI products’ contribution to defense, and select CTI products from information sharing communities. In addition, the study has identified certain limitations, which open avenues for future research, including the real-time integration of CTI into organizational defense mechanisms. This work significantly contributes to standardizing the quality evaluation of CTI products and enhancing organizations’ cybersecurity posture.

1. Introduction

Cyberattacks continuously threaten organizations worldwide, seeking to compromise their assets’ confidentiality, integrity, and availability. In the past decade, the majority of threat actors have become professionals, with the European Union Agency for Cybersecurity (ENISA), distinguishing the majority of them into the following groups: state-sponsored actors, cybercrime actors, hacker-for-hire actors, and hacktivists [1]. At the same time, these actors can organize highly sophisticated and coordinated attacks, such as disinformation attacks [2], side channel attacks [3,4], distributed denial of service [5], and supply chain attacks [6].

This hazardous environment has led organizations to adopt new defense mechanisms, with CTI having a prominent role in their defense arsenal. CTI is the field where data from various sources are collected, analyzed, and assessed about threat actors and their motivation, attacks’ methodology, and victims to produce intelligence that helps organizations prevent or predict a cyberattack and follow intelligence-based decision making [7]. Although most cybersecurity specialists understand the importance of CTI and believe that the quality of CTI meets their standards, they are also concerned about missing actionable CTI due to the large scale of data that must be processed daily [8].

The large number of CTI sources and data that organizations use in their daily defense against threat actors makes the identification of actionable and relevant CTI a problem that lies in the area of Big Data since the 5Vs (velocity, volume, value, variety, and veracity) differentiate the generated intelligence and its application within defense mechanisms. In addition, relevance and actionability are key quality factors of CTI, introducing the dimension of quality of CTI as an alternative point of view to manage this problem.

In general, the information within CTI encompasses both structured and unstructured data, serving as the input or output of a threat intelligence process [7]. Therefore, identifying the most relevant and actionable CTI products for an organization or a security analyst based on the CTI quality of a large-scale data flow is the problem that this work deals with. To formulate this problem, we have determined the following research questions.

• RQ1: In what ways can the relevance and actionability of unstructured CTI products be defined and quantitatively assessed?
• RQ2: What methodologies can be employed to rigorously assess the CTI products in relation to the organizations willing to use them?
• RQ3: In what manner can the proposed methodologies be systematically applied to extensive datasets?

To answer these questions, we have developed the Relevance ($Re$) and Actionability (Ac) CTI quality metrics that leverage probabilistic data structures and algorithms to face the large-scale unstructured CTI products, which are the main contribution of this work. Toward the development of $Re$, we discuss the information needs of an organization in terms of CTI and introduce the concept of analyzing an organization as an open system in the context of CTI. Similarly, in the case of $Ac$, we analyze the decision-making process in the context of cybersecurity of an organization, and we propose an innovative modeling approach for it, which drives us in the definition of the metric in relation to the defense mechanisms of an organization. Finally, in the last part of this work, we implement the metrics and experimentally measure them against a dataset of CTI products.

The remainder of this paper is organized as follows. Section 2 presents the related work and alternative approaches in the bibliography. Section 3 presents the background of this work is divided into three subsections; in Section 3.1, we present the key concepts and definitions related to this work; in Section 3.2, we formally define the problem described in the introduction and in Section 3.3, we present the algorithmic and mathematical background of this work. The proposed metrics are presented in Section 4. In Section 5, we propose an implementation of the two metrics and explain the implementation assumptions. In addition, the experimental results of the application of the proposed metrics in unstructured CTI products are analyzed. Finally, in Section 6, we present our conclusions and future work.

2. Related Work

In the bibliography, few works deal with massive unstructured CTI data quality. There is a perplexity between the quality of CTI sources and the quality of the produced intelligence (CTI products). Moreover, the formal identification of actionable and relevant CTI quality factors remains a significant research challenge, due to the diverse methodologies employed by researchers. Only a part of the bibliography deals with CTI quality; for example, Tale et al. [9] discuss the quality of large unstructured data. They propose the construction of a data quality profile for a dataset using only a sample for their analysis that captures the general features, such as type, format, and data domain. They use the data quality profile as input in their unstructured big data quality assessment model to evaluate the overall quality of the dataset. But their work is domain-agnostic and does not focus on CTI.

Zibak et al. [10] assess the quality of threat intelligence by identifying key dimensions—such as accuracy, actionability, interoperability, provenance, relevance, reliability, and timeliness—utilizing a systematic literature review and a Delphi study with input from 30 experts. They differentiate between threat data and intelligence, emphasizing their distinct purposes and quality priorities. Their study reveals a lack of standardized metrics for assessing intelligence quality and stresses the need for aligning intelligence products with user needs. Challenges like privacy issues, limited resources, and underdeveloped threat intelligence programs obstruct quality evaluation. Although their study does not delve into the technical aspects of CTI quality assessment or the factors added by the volume of CTI data, it lays the groundwork for developing metrics and frameworks to improve the quality and efficacy of threat intelligence.

Tundis et al. [11] present a feature-oriented approach to automate the evaluation of open-source CTI sources, with a particular emphasis on Twitter. They introduce a CTI relevance score designed to measure the significance of sources by leveraging metadata and word embeddings, prioritizing both timeliness and informativeness. Their experiments show enhancements in the early detection of cyber threats, minimizing alert delays while preserving accuracy. Nevertheless, their research concentrates on the quality of the sources and does not address the quality of the intelligence they produce. Specifically, they aim to measure the relevance of each Twitter user (considered the CTI source in their study) involved in CTI generation, but do not assess the quality features of the intelligence itself.

Azevedo et al. [12] propose the PURE platform to generate enriched Indicators of Compromise (IoCs) that improve the quality characteristics of IoCs collected from different sources of Open Source Intelligence (OSINT). To succeed in the development of enriched IoCs, the authors combine filtering, deduplication, and clustering techniques based on the similarity of IoCs. However, PURE has been designed to handle average data volumes and the authors do not explain the cost of calculating the respective similarity indices.

Schaberreiter et al. [13] propose a methodology for evaluating the trust of CTI sources. Their method is based on the calculation of ten parameters (extensiveness, maintenance, false positives, verifiability, intelligence interoperability, compliance, similarity, timeliness, and completeness) on Structured Threat Information Expression (STIX) objects and the continuous estimation of a trust indicator for each source. However, the authors follow a closed-word assumption, namely that the CTI shared by the sources comprises the entire worldview of threat intelligence, which is contrary to the belief of unknown unknowns in threat intelligence [14].

Yang et al. [15] introduce an automated approach to evaluate CTI quality by merging feed trustworthiness with content availability. They represent CTI feed interactions using a correlation graph and implement an iterative algorithm to evaluate the originality of the feed. Machine learning is applied to assess CTI content based on features, such as multi-source verification, completeness, and timeliness. Their approach integrates feed trustworthiness and content availability, incorporating a dynamic aging factor for real-time adaptability. However, because their study focuses on CTI quality factors of trustworthiness and availability, it does not address the specific characteristics of CTI consumers.

In their work, Schlette et al. [16] tackle the challenge of evaluating and illustrating the quality of CTI. They introduce a set of data quality dimensions and metrics, specifically designed for the STIX format, facilitating both automated and expert-led assessments of CTI artifacts. Additionally, they employ visual analytics to display quality indicators, enabling security analysts to interpret and engage in CTI quality evaluations. They also emphasize the significance of measuring CTI quality and its potential benefits for threat intelligence information sharing platforms. However, their approach relies on the structured nature of STIX, utilizing specific attributes, thereby limiting its applicability to structured CTI products.

Zhang et al. [17] propose the Threat Intelligence Automated Assessment Model (TIAM) for the quality assessment of large-scale CTI. More specifically, they aggregate IoCs extracted from STIX objects and sparse text-based intelligence to assess the overall intelligence. Then, they correlate the IoCs with the adversarial Tactics, Techniques, and Common Knowledge (ATT&CK) knowledge base [18] to identify potential attack techniques. TIAM defines the alert level, the created time, the external references, and the Common Vulnerability Scoring System (CVSS) (only in the case of vulnerabilities) as the assessment features of IoCs and proposes weighted scores for each evaluation. However, the authors do not explore quality factors, such as accuracy, timeliness, and completeness, and how these quality factors can be measured in the case of large-volume CTI.

While the field of CTI quality assessment still demands deeper investigation, its role in enhancing CTI sharing, boosting cybersecurity resilience, and mitigating overall risks has been evident since the inception of CTI. As highlighted by Wagner et al. [19], collaborative CTI sharing network participants must be adept at filtering relevant and actionable CTI products to prevent analysts from being overwhelmed with vast volumes of data. Additionally, as noted by Geras and Schreck [20], utilizing CTI products in a targeted manner based on their quality is essential because it contributes to real-time decision-making and reduces potential risks associated with false positive incidents, influencing an organization’s response strategies. This study addresses CTI quality challenges by proposing and applying two novel metrics, $Re$ and $Ac$.

The rest of the related bibliography refers to the use of big data techniques to handle CTI and its quality.

Tao et al. [21] proposed a modified classified protection model based on CTI and big data analysis techniques, in which CTI is utilized in awareness and detection of the defense mechanism. However, the system that implements the modified model is based on the bulk consumption of CTI without evaluating the quality of the data. At the same time, no false-positive/negative detection statistics are presented.

Marchetti et al. [22] propose the AUSPEX framework for the detection of advanced persistent threats (APTs). AUSPEX utilizes internal and external sources of raw data (e.g., logs, OSINT) and CTI to detect APTs, which are then analyzed using big data techniques. However, CTI is used only in the form of blacklists without mentioning their effectiveness.

Wheelus et al. [23] propose a big data architecture specialized in collecting and analyzing CTI data by combining existing and widely used big data techniques. Moreover, they use the proposed architecture to demonstrate its capabilities in a series of CTI problems, such as malware-type detection. However, they do not explore the quality of the results and how the latter can become part of CTI products.

Finally, Martins et al. [24] develop the Automated Event Classification and Correlation platform, which combines classification, trimming, enrichment, and clustering techniques to improve the quality of events on threat intelligence platforms. They propose a unified taxonomy that seeks to simplify the categorization of a threat. However, they do not explain what quality factors are intended to improve and how to measure this improvement.

Table 1. Summary of related work.

Article	Strengths	Weaknesses
Tale et al. [9]	evaluation of quality on unstructured big data	domain agnostic, not focusing on CTI
Zibak et al. [10]	identification of key CTI quality dimensions and CTI quality challenges	lack of analysis on technical implementation
Tundis et al. [11]	CTI relevance score, combination of different CTI quality factors	focus on CTI sources and not on the CTI products
Azevedo et al. [12]	CTI quality-based enrichment of IoCs	not designed to handle big data
Schaberreiter et al. [13]	evaluation of CTI sources trust, continuous estimation of a CTI source indicator	closed-world assumption about the threat intelligence
Yang et al. [15]	combination of various quality factors for the measurement of CTI products trustworthiness and availability	not addressing the characteristics of CTI consumers
Schlette et al. [16]	introduces a set of STIX-based data quality dimensions	limited to structured CTI products
Zhang et al. [17]	quality assessment of large-scale CTI	not exploring the CTI quality factors for the assessment but specific features of IoCs
Tao et al. [21]	big-data analysis of CTI for the improvement of defense mechanisms	bulk consumption of CTI remains a problem, no CTI quality evaluation
Marchetti et al. [22]	big-data analysis of CTI	no measurement of CTI effectiveness
Wheelus et al. [23]	big-data based processing of CTI	not exploring the quality of the produced intelligence
Martins et al. [24]	pipeline-based processing of CTI products to improve their quality	not targeting on the improvement of specific quality factors

provides a summary of the findings from examining the bibliography of related work in CTI quality.

Through a comprehensive review of the pertinent literature, we have identified several research gaps, resulting in the formulation of the research questions delineated in Section 1: (a) the CTI quality factors, particularly relevance and actionability, lack precise definitions, and the metrics used for quality evaluation are not explicitly correlated with these quality factors, (b) there is a tendency to evaluate the quality of CTI products interdependently of the consumer’s (e.g., organizations) characteristics that intend to use them, and (c) the employment of CTI quality measurement on large-scale datasets remains limited.

3. Background

3.1. Key Concepts

3.1.1. Unstructured CTI Products

The results of a threat intelligence process are called CTI products [7]. Those results can take a structured or unstructured format. Structured CTI products [25] are formatted following a standard such as STIX [26], whereas unstructured CTI products are written in natural language or do not follow a well-defined standard [27]. For the remainder of this paper, we focus on unstructured CTI products and use unstructured text data that contain information about cybersecurity as our experimental base.

3.1.2. Relevance CTI Quality Factor

Relevance is a crucial CTI quality factor [28], which determines the level at which the content of a CTI product meets the informational needs of a specific CTI consumer employing it in decision making [7,10]. Pawlinski et al. [29] state that a piece of information (e.g., a CTI product) is considered relevant if it is “applicable” in the area of the one that uses it, where the term “area” describes the networks, software, and hardware comprising one’s systems. Moreover, Deliu et al. [30] stress that an effective CTI process should result in relevant CTI products. In summary, the relevance CTI quality factor reflects the level at which a CTI product is considered to have a cybersecurity impact on the CTI consumer. In this context, “impact” pertains to modifications within the system, which may manifest as alterations in the system architecture, implementation of security controls, and the commencement of cybersecurity protocols (e.g., incident management).

3.1.3. Actionability CTI Quality Factor

In general terms (that is, including system decisions, e.g., Intrusion Detection System (IDS) detection and experts’ decisions), the actionability CTI quality factor expresses the immediacy of the use of a CTI product in a decision-making process [28]. In addition, actionability appears to be a compound quality factor that integrates other CTI quality factors such as relevance, completeness, ingestibility, accuracy, and timeliness [19,28,29]. In general, actionability reveals at what level a CTI product can initiate a decision by a CTI consumer at a given moment [7,10].

3.1.4. Relevance vs. Actionability

At this point, we need to clarify the relationship between the two quality factors and why it is important for organizations to be aware of both factors in the context of CTI. In simple terms, we can say that an actionable CTI product is always relevant to an organization, but a relevant CTI product is not always actionable. The main reasons for this proper-subset relationship between relevance and actionability are time and the organization’s ability to use CTI products. Within the scope of CTI, organizations need to recognize the importance of the relevance quality factor to determine which CTI products pertain to them (such as CTI products related to the software the organization employs). Conversely, it is essential for an organization to consider the actionability quality factor to ascertain whether a CTI product can be used by its current capabilities (e.g., by an IDS) at any specific time.

3.2. Problem Definition

Consider a set of CTI sources S, which produce a large number of unstructured CTI products that have the characteristics of big data’s 5Vs (velocity, volume, value, variety, and veracity). In this case, the fundamental research question is how we can evaluate the quality of those unstructured CTI products in reference to a given organization C that aims to use them as input to a decision-making or CTI process and wants to avoid investing resources in the analysis of unrelated data. In this context, we investigate the development of two metrics, $Re$ and $Ac$, to quantify the relevance and actionability of CTI quality factors for unstructured CTI products, respectively.

At this juncture, it is crucial to explore the challenges linked to the nature of unstructured CTI products, which stem from their unstructured text-based format. Evaluating any quality factor of unstructured CTI products necessitates recognizing the obstacles in handling large quantities of text data. The primary challenges include the inherent ambiguity of natural languages, the limits on scalability in text mining [31], and the identification of content. However, in our situation, the proposed metrics effectively address these challenges by employing techniques and algorithms that are specifically designed to handle the typical issues encountered in text mining.

3.3. Probabilistic Algorithms and Data Structures

Probabilistic algorithms and data structures have been proposed in the bibliography [32] to handle problems in the area of Big Data. Those algorithms, being nondeterministic by definition, utilize mainly hashing techniques, and their results include a “tolerable” error. Next, we present two categories of probabilistic algorithms and data structures: those focusing on the similarity problem, and those focusing on the membership problem. We use the first category of algorithms on the definition of the relevance metric ($Re$), and the second category of algorithms on the definition of the actionability metric ($Ac$).

3.3.1. Probabilistic Algorithms and Data Structures of Similarity Category

Probabilistic algorithms and data structures in the similarity category handle problems such as finding the nearest neighbor for a given document, detecting duplicates, and clustering. This similarity category includes algorithms and data structures such as MinHash [33]. Similarity expresses the level of resemblance between two objects (for example, documents). To handle this problem numerically, objects are usually represented as sets of features called canonical forms in the case of documents. Then, the Jaccard similarity is used to calculate the percentage of their common features (i.e., similarity). Formally, the Jaccard similarity for two documents $d_1,d_2$, is given by the following formula: $J(d_1,d_2)={\displaystyle \frac{|d_1\cap d_2|}{|d_1\cup d_2|}}$, where $|d_1\cap d_2|$ represents the count of features shared by both $d_1$ and $d_2$, and $|d_1\cup d_2|$ signifies the total count of distinct features present in either $d_1$ or $d_2$. It is important to highlight that we have opted for Jaccard similarity among various document similarity metrics [34,35] as it is integral to the theory of the $MinHash$ algorithm.

$MinHash$ implements Locality Sensitive Hashing (LSH) [36]. The basic idea is that when similar documents are hashed with an LSH algorithm, they are highly likely to produce hash values in a close range. An LSH function generally ensures that the collision probability for similar documents is higher than for two irrelevant, random documents. An LSH algorithm combines two functions: Locality-Sensitive Bucketing, which maps documents in a hash table of buckets indexed by the hash values, and the Finding Similar Documents function, which searches the hash table for a given document d and returns its candidate documents, then calculates the similarity of d and the returned documents to find those that have a similarity above a certain threshold.

In the case of $MinHash$, let us consider a set of indexed features (words) that we are looking for in documents, and let us say that a document (d) contains a number of these features. If we construct a bit array for the indexes and set the indexes of the features of d to 1 and randomly permutate the indexes, then the $MinHash$ value of d is the position of the left-most set bit on the permutated bit array. If we choose k-random permutations, we construct the $MinHash$ signature for d by assigning each of the respective $MinHash$ values to a k-length vector. For a set of documents $d_i$, $MinHash$ creates the $MinHash$ signature matrix, where the rows correspond to permutations and the columns to documents. In its implementations, $MinHash$ uses a random hash function. Moreover, it has been proven that for documents $d_1,d_2$, the probability that their signatures on the $MinHash$ signature matrix are equal is the Jaccard similarity of those documents $P(minhashsig\left(d_1\right)=minhashsig\left(d_2\right))=J(d_1,d_2)$.

3.3.2. Probabilistic Algorithms and Data Structures of Membership Problem Category

Probabilistic algorithms and data structures belonging to the category of the membership problem are tasked with deciding whether an element is a member of a dataset or not [32]. The category of membership problem includes algorithms and data structures like the Bloom filter [37] and the Cuckoo filter [38]. A Cuckoo filter is a data structure that leverages Cuckoo hashing.

Hash functions, particularly cryptographic hash functions (e.g., SHA256), are widely used in cybersecurity and play a crucial role in probabilistic algorithms and data structures. Generally speaking, a hash function maps an arbitrary size of data to a fixed-length hash value: $h\left(x\right)\to y_{(k-length)},$ y$:avalueoffixedk$-length. A hash table is a dictionary that comprises a m-length unordered array of k-buckets indexed by key $kϵ[0,m-1]$. An element x is inserted in the bucket with the key $k=h\left(x\right)$, where h is a hash function of the range $[0,m-1]$.

Cuckoo hashing utilizes two hash functions instead of one to index a new element to the cuckoo hash table, namely an array of buckets where an element has two candidate buckets, one for each hash function. A new element is inserted into one of these two buckets if it is empty; otherwise, the algorithm randomly selects one of the two occupied buckets and inserts the element, moving the existing element to its alternative candidate bucket. The process repeats until an empty bucket is found or until a maximum number of displacements is reached. Lookup and deletion are performed by determining the candidate buckets of an element by computing the two hashes.

A cuckoo filter is a variation of a cuckoo hash table, but instead of key-value pairs, it stores fingerprints (f) of a predefined length (p). A cuckoo filter consists of a hash table with a bucket capacity b. The indices of the candidate buckets for an element x are calculated by applying the following three equations: $f=h\left(x\right)modp$, $i=h\left(x\right)modm$, and $j=(i\oplus h(f\left)modm\right)modm$. The interesting characteristics of a cuckoo filter with respect to the membership problem are that false positives are possible with probability $P_{fp}\approx {\displaystyle \frac{2b}{2^p}}$. In contrast, false negatives are impossible (that is, $P_{fn}=0$). Moreover, cuckoo filters support dynamic addition and deletion.

4. Proposed Algorithms

4.1. Methodology

To define the proposed metrics $Re$ and $Ac$, we adopt a structured methodology for developing CTI metrics [27]. This methodology is outlined in the eight steps of

Table 2. CTI Quality Metrics Development Methodology.

Step	Description
1	Based on the CTI data or sources, try to identify what can better express their quality and name this metric M.
2	Determine the set of variables X necessary to calculate M.
3	Define function F, which computes metric M.
4	Analyze X and F to determine subjectivity and objectivity $\mathrm{\Gamma }$.
5	Analyze F to determine the performance of M (i.e., time complexity of M calculation), P.
6	Analyze F to determine the precision of M, A.
7	Conduct sensitivity analysis on M to determine B.
8	Construct metric $M=(Q,F(X\left)\right)$

. Briefly, in Section 4.2.1, Section 4.2.2, Section 4.3.1 and Section 4.3.2, we address all the aspects related to STEP-1 of the methodology for the $Re$ and $Ac$ metrics, respectively. Section 4.2.3 and Section 4.3.3 provide an in-depth analysis of the application of the remaining steps for developing these two CTI metrics. By employing this systematic approach, we integrate the quality characteristics of a metric into its definition, enabling a theoretical evaluation of different metrics.

4.2. Defining the Relevance CTI Quality Metric

To quantify the relevance of a CTI product, we need to determine the reference point to which we define and measure the CTI quality metric, which in our case is organization C (cf. Section 3.2). As discussed in Section 3.1, the relevance quality factor is related to the informational needs of C and the applicability and the potential impact of a CTI product on C. However, to define the Relevance quality metric, $Re$, we need to analyze in more detail what organization C is and how those abstract notions (i.e., informational needs, applicability, impact) are defined within C.

4.2.1. Determining Organization C

According to Scott and Davis [39], an organization can be described as an open system [40], meaning that it comprises parts that operate as one, interacting with the environment to achieve its goal. As a result, an organization is modeled as a system that receives input (i.e., materials, human resources, capital, technology, information), applies a transformation (i.e., work activities, management activities, technology, and operations methods) and generates output (i.e., products, information, financial, and human impact). The environment that affects an organization can be further divided into three layers [41]:

• the general environment
• the task environment and the internal environment.

The general environment affects all organizations almost equally and includes international, technological, natural, sociocultural, economic, and legal/political aspects. The task environment includes customers, competitors, suppliers, and the labor market, which interact directly with an organization. Finally, the internal environment includes the interorganizational aspects of employees, management, and culture, which handle the transformation of input to output.

4.2.2. Organization Aspects and the Relevance CTI Quality Metric

The environments described in Section 4.2.1 determine the information needs, the applicability, and the impact of a CTI product on an organization. The core idea of the proposed quality metric $Re$ is the observation that an organization is interested in information that helps it survive (i.e., applicability and impact) and obtains this information as input from each of its environments. Information needs are usually expressed as questions, e.g., which vulnerabilities affect our information systems? Which business areas were attacked more last year?

At the same time, information applicability and impact can act as a filter for potential answers to information needs. For example, let us assume that an organization receives CTI products related to cyber attacks in business areas similar to those in which the organization operates; moreover, those cyber attacks are based on the exploitation of a specialized operating system. In this case, the organization needs to know about cyber attacks against the business areas in which it operates (information need). Consequently, it investigates the received CTI products to determine whether the exploited operating system is part of its information systems (applicability) and whether the version of the operating system it uses can be exploited (impact).

To focus more on CTI, we identify the following relative aspects of each environment [41] that can be a source of threat information for an organization: international, technological, customers, competitors, suppliers, and employees. Each of these aspects can be related to the input, the transformation process, and the output of an organization. Hence, the fundamental question is as follows: How can we use those environmental aspects to identify the potential relationship of a CTI product with an organization? To answer this question, we observe that a threat can impact the inputs, the transformation process, or the output of an open system. For this reason, we adopt and extend the notion of the information landscape [42] to combine the information needs of the organization related to a CTI product and the applicability and impact of the information that a CTI product delivers to an organization. For each organization, we propose three information landscapes: input landscape ($L_I$), transformation process landscape ($L_{TP}$), and output landscape ($L_O$).

Landscape $L_I$ includes the information needs related to the suppliers, competitors, and capital sources of an organization. Those needs arise from the potential later impact against suppliers, competitors, and capital sources that a cyber threat can cause to an organization.

Landscape $L_{TP}$ includes information needs related to an organization’s business activities (e.g., business areas that an organization operates), its internal operations (e.g., Human Resource Management (HRM)) and its information systems (that is, the information needs of $L_{TP}$ reflect the three risk assessment tiers described in National Institute of Standards and Technology (NIST) Special Publication (SP) 800-30 [43]).

Finally, landscape $L_O$ includes information needs related to an organization’s products (e.g., does a threat actor focus on a specific brand product?) and providing services. In the context of the Relevance metric, these landscapes are materialized as unstructured text documents that contain detailed expressed information needs.

4.2.3. Relevance Metric Generic Calculation Mechanism

The generic idea of calculating the relevance metric is based on $MinHash$ Signatures estimated between the information landscapes of an organization C and a CTI product. These probabilities represent the level at which a CTI product responds to the information needs of an organization. Then, a weighted average of those probabilities is calculated, representing the $Re$ metric. Figure 1 depicts the generic mechanism for calculating the $Re$ metric.

To define in detail the proposed metric $Re$, we follow the eight-step methodology of [27] as outlined in Table 2.

STEP 1

As a first step, we name the metric in development. According to our analysis in Section 4.2.1 and Section 4.2.2, we intend to develop a metric $Re$, which measures the relevance of the CTI product P with the organization C, taking into account the information needs of C.

STEP 2

Having defined the metric’s name, we need to identify and determine the variables that contribute to the metric’s calculation. So, we determine the set of variables X that calculates $Re$. Based on our analysis in Section 4.2.2, X includes the three landscapes $L_I$, $L_{TP}$, and $L_O$, and the CTI product P. So, $X=\{L_I,L_{TP},L_O,P\}$.

At this point, we need to analyze in more detail what these variables contain and how they can be constructed. First, we turn our attention to the three proposed landscapes. Each of the variables, $L_I$, $L_{TP}$, and $L_O$, is the textual representation of the information needs of C. Therefore, $L_I$ should contain the textual description of the information needs of C about its suppliers, competitors, and capital sources. $L_{TP}$ should contain the textual description of the information needs of C about its business activities, internal operations, and information systems. Finally, $L_O$ should contain the textual description of the information needs of C about its products and services. These variables can be constructed and updated using formal taxonomies (e.g., Domain Industry Taxonomy (DIT) [44]) that describe business activities, products, services, capital sources, etc., or C can construct them by writing its information in a text format (e.g., competitor A has the following business activities; we use version K of the information system B). The accurate determination of information needs in the context of CTI is a future research challenge. However, Section 5 presents an example of how organization C can construct them. Here, P represents the textual form of a CTI product.

STEP 3

Here, the methodology [27] focuses on the definition of the mathematical function that calculates the metric. So, in this step, we define a function F that calculates the $Re$ metric. As we have previously mentioned, the idea is that when we only have to select a few CTI products from a bunch of millions of CTI products, represented in text format, we select those that are close to the information needs of an organization. We calculate the value of $Re$ by averaging the three equally-weighted $MinHash$ values ($M{H}_1,M{H}_2,M{H}_3$) of the CTI product P and the information landscapes of organization C. So, we define F as follows:

$$F\left(X\right)=Avg(M{H}_1(P,L_I),M{H}_2(P,L_{TP}),M{H}_3(P,L_O))$$

(1)

STEP 4

In this step, we determine the subjectivity and objectivity of $Re$, which is defined as $\mathrm{\Gamma }$ in [27] and takes one of the values in $\{SO,SS,OS,OO\}$. A metric is characterized as subjective or objective by examining two characteristics: the involvement or not of the human factor in the determination of its variables and the deterministic or non-deterministic characteristic of metric F. We observe that the human factor is involved in the determination of $L_I$, $L_{TP}$, and $L_O$ because these variables express the information needs of an organization C. Therefore, $Re$ is estimated from subjective data. Moreover, using the $MinHash$ function in the calculation of $Re$ introduces a non-deterministic component because it requires the selection of a number of random hash functions. So, we can say that $Re$ is a subjective metric of subjective data. In summary, we infer that $\mathrm{\Gamma }=SS$.

STEP 5

In this step, the methodology theoretically estimates the performance of metrics F, which is usually expressed as time complexity, by analyzing the algorithm that calculates F. We use Algorithm 1 to estimate the performance of $Re$, expressed as time complexity. As stated in [33], a $MinHash$ algorithm that uses k hash functions has a time complexity of $O\left(kn\right)$. The algorithm that calculates $Re$ comprises the application of three $MinHash$ functions and the calculation of the average value of their results. Therefore, the performance of calculating $Re$ is $O\left(kn\right)$ because the three $MinHash$ functions are applied independently, and the average value is calculated only once. Consequently, the time complexity for calculating $Re$ is $O\left(kn\right)$ (which is deemed more effective when compared to using other document similarity measures, such as brute-force comparison with a complexity of ($O\left(n^{2}d\right)$)). It is important to note that performance is articulated in terms of time complexity based on the methodology outlined by [27], and serves as a comparative measure among different metric implementations. Since the population size of CTI products under evaluation is typically one, we do not undergo a space complexity analysis of the algorithm we propose. This implies that the algorithm (i.e., the three $MinHash$) is executed with a specific CTI product in mind. For the subsequent CTI product, the memory from the prior computation can be reused. This situation differs from time complexity, where n represents the number of distinct text documents serving as inputs to the algorithm. Additionally, we provide the experimental findings related to time and space measurement in Section 5.3.4.

Algorithm 1$Re$ Metric Calculation Algorithm

Require: P, $L_I$, $L_{TP}$, and $L_O$ $F,l_1,l_2,l_3\leftarrow 0$ $l_1\leftarrow M{H}_1(P,L_I)$ $l_2\leftarrow M{H}_2(P,L_{TP})$ $l_3\leftarrow M{H}_3(P,L_O)$ $F\leftarrow Avg(l_1,l_2,l_3)$

STEP 6

In this step, the methodology theoretically analyzes the accuracy of F which, in simplicity, explains how close to the real value of the metric is the calculated one. The calculation of $Re$ includes a nondeterministic factor (that is, introduced by $MinHash$). Specifically, $MinHash$ estimates the $Jaccard$ similarity of two given documents with an error probability $ϵ$ related to the number of hash functions k. In simple words, the accuracy of $MinHash$ increases as the number of hash functions used increases ($ϵ\approx {\displaystyle \frac{1}{\sqrt{k}}}$[45]), with a cost on the storage space required for the calculation of $MinHash$. Consequently, the accuracy of $Re$ depends on the errors of the $MinHash$ functions of Algorithm 1 ($F({ϵ}_1,{ϵ}_2,{ϵ}_3)$). Thus, $A=Avg({\delta }_1,{\delta }_2,{\delta }_3)$, where ${\delta }_1=J_1{ϵ}_1$ ($J_1$ the Jaccard similarity of P and $L_I$), ${\delta }_2=J_2{ϵ}_2$ ($J_2$ the Jaccard similarity of P and $L_{TP}$), and ${\delta }_3=J_3{ϵ}_3$ ($J_3$ the Jaccard similarity of P and $L_O$).

STEP 7

In this step, the methodology performs a sensitivity analysis of the metric F to theoretically determine how sensitive F is to changes in its input variables. We use the elementary effects method [46] to perform a theoretical sensitivity analysis of F. To apply the elementary effects method, we first identify the variables of X whose change organization C does not control. Of the four variables, $X=\{L_I,L_{TP},L_O,P\}$, only P can be changed by actors that do not belong to C. On that note, we observe that P is the input of three different $MinHash$ functions that comprise F, and each of these has one of the variables $L_I,L_{TP},L_O$, as the second input. Applying the elementary effects method, we consider the t selected levels (a selected level represents a discrete value in which a variable can be set during the application of the elementary effects methods) at which P and $\overline{Y}$ (a new CTI product that is derived by randomly causing minor changes in P) can be set, where $\mathrm{\Omega }$ is their discretized input space. Then, the elementary effect of P, $E{E}_P$, is as follows:

$$E{E}_P={\displaystyle \frac{F(L_I,L_{TP},L_O,\overline{Y})-F(L_I,L_{TP},L_O,P)}{\mathrm{\Delta }}}$$

(2)

where $\mathrm{\Delta }ϵ\{{\displaystyle \frac{1}{t-1}},1-{\displaystyle \frac{1}{t-1}}\}$ and $\overline{Y}=P\pm \mathrm{\Delta }$. Then, the distribution $F_P$ of $E{E}_P$ is derived by randomly sampling $\overline{Y}$ from $\mathrm{\Omega }$. According to the elementary effects method, the sensitivity measures are the mean (${\mu }_P$) and standard deviation (${\sigma }_P$) of the distribution $F_P$, and the mean of the absolute values (${\mu }_P^{*}$) of $\left|E{E}_P\right|$ of the respective distribution $\left|E{E}_P\right|\sim G_P$. Here, ${\mu }_P$ assesses the influence of P in $Re$, ${\mu }_P^{*}$ again assesses the influence of P in $Re$ simultaneously handling negative values of $E{E}_P$, and ${\sigma }_P$ reveals the total effects of the interactions between the variable P and the variables $L_I$, $L_{TP}$, and $L_O$. Following the sampling approach proposed in the elementary effects method [46], we conclude that for the distributions $F_P$, $G_P$ derived from r samples, we calculate ${\mu }_P$ as follows:

$$\begin{array}{c}\hfill {\mu }_P={\displaystyle \frac{1}{r}}\sum _{j=1}^{r}E{E}_P={\displaystyle \frac{1}{r}}\sum _{j=1}^r{\displaystyle \frac{F(L_I,L_{TP},L_O,\overline{Y_j})-F(L_I,L_{TP},L_O,P_j)}{{\mathrm{\Delta }}_j}}=\\ \hfill {\displaystyle \frac{1}{r}}\sum _{j=1}^r({\displaystyle \frac{Avg(M{H}_1(L_I,\overline{Y_j}),M{H}_2(L_{TP},\overline{Y_j}),M{H}_3(L_O,\overline{Y_j}))}{{\mathrm{\Delta }}_j}}-\\ \hfill {\displaystyle \frac{Avg(M{H}_1(L_I,P_j),M{H}_2(L_{TP},P_j),M{H}_3(L_O,P_j))}{{\mathrm{\Delta }}_j}})\end{array}$$

(3)

To simplify the calculation, let us explore how a change $\mathrm{\Delta }$ in P affects the value of $MinHash$ between P and $L_I$. We consider that n-hash functions construct the $MinHash$ signatures, $Sig\left(X\right)$ (i.e., n-length signature). Then, $M{H}_1(L_I,P)$ is given by the following formula:

$$M{H}_1(L_I,P)={\displaystyle \frac{{\sum }_{i=1}^n{\delta }_{[Sig\left(L_I\left[i\right]\right)-Sig\left(P\left[i\right]\right)],0}}{n}}$$

(4)

where $\delta$ is Kronecker’s delta. Similarly, for $Y=P\pm \mathrm{\Delta }$, we calculate the following:

$$M{H}_1(L_I,Y)={\displaystyle \frac{{\sum }_{i=1}^n{\delta }_{[Sig\left(L_I\left[i\right]\right)-Sig\left(Y\left[i\right]\right)],0}}{n}}$$

(5)

and we name the difference between the two values ${\alpha }_{L_I}=M{H}_1(L_I,Y)-M{H}_1(L_I,P)$. Focusing on the effect of $\mathrm{\Delta }$ in $MinHash$ calculation, we observe that it is only related to the probability of $\mathrm{\Delta }$ affecting the bitwise comparison of the two $MinHash$ signatures. Specifically, if we define $k\left[i\right]=Sig\left(L_I\right)\left[i\right]-Sig\left(Y\right)\left[i\right]$ and $l\left[i\right]=Sig\left(L_I\right)\left[i\right]-Sig\left(P\right)\left[i\right]$, then $\mathrm{\Delta }$ always affects ${\alpha }_{L_I}$ except for $P(k\left[i\right]-l\left[i\right]=0)=1$. This is equivalent to the probability that the $MinHash$ value of P and Y is equal to one, $P(M{H}_1(Y,P)=1)$ which is always almost one because Y derives from P by adding a small $\mathrm{\Delta }$. This means that the value of $M{H}_1(L_I,P)$ is practically not affected by $\mathrm{\Delta }$. It is only affected by the similarity estimation error of $MinHash$ that absorbs the error introduced in $M{H}_1$ by $\mathrm{\Delta }$. Moreover, this error is related to the number of hash functions, n, used for the construction of $MinHash$ signatures [32]. As a result, we can consider ${\alpha }_{L_I}$ as $\mathrm{\Delta }$-independent, becoming equal to zero for small values of deltas. Similarly to ${\alpha }_{L_I}$, we define ${\alpha }_{L_{TP}}$, and ${\alpha }_{L_O}$. Hence, Equation (3) is transformed into the following:

$$\begin{array}{c}\hfill {\mu }_P={\displaystyle \frac{1}{r}}\sum _{j=1}^r\left({\displaystyle \frac{Avg{({\alpha }_{L_I},{\alpha }_{L_{TP}},{\alpha }_{L_O})}_j}{{\mathrm{\Delta }}_j}}\right)\end{array}$$

(6)

In addition, for the calculation of ${\mu }^{*}$ and $\sigma$, we have the following:

$$\begin{array}{c}\hfill {\mu }_P^{*}={\displaystyle \frac{1}{r}}\sum _{j=1}^r|E{E}_{Pj}|={\displaystyle \frac{1}{r}}\sum _{j=1}^r\left({\displaystyle \frac{\left|Avg{({\alpha }_{L_I},{\alpha }_{L_{TP}},{\alpha }_{L_O})}_j\right|}{{\mathrm{\Delta }}_j}}\right)\end{array}$$

(7)

$$\begin{array}{c}\hfill {\sigma }_P^2={\displaystyle \frac{1}{r-1}}\sum _{j=1}^r(E{E}_{Pj}-{\mu }_P{)}^2={\displaystyle \frac{1}{r-1}}\sum _{j=1}^r({\displaystyle \frac{Avg{({\alpha }_{L_I},{\alpha }_{L_{TP}},{\alpha }_{L_O})}_j}{{\mathrm{\Delta }}_j}}-{\mu }_P{)}^2\end{array}$$

(8)

In conclusion, from ${\alpha }_{L_I}$, we can infer that for small values of deltas regarding ${\alpha }_{L_{TP}}$, ${\alpha }_{L_O}$, ${\mu }_P$ (Equation (6)) and ${\mu }_P^{*}$ (Equation (7)), P has a constant influence on $Re$, while for large values of deltas, the value of $Re$ depends on the deviation of $\mathrm{\Delta }$. Additionally, from ${\sigma }_P$ (Equation (8)), we can infer that the interactions between P and variables $L_I,L_{TP}$, and $L_O$ depend on the magnitude of the change ($\mathrm{\Delta }$). In simple words, for small changes, we expect $Re\left(P\right)\approx Re\left(Y\right)$, while for large changes, we expect $Re\left(P\right)\ne Re\left(Y\right)$. The threshold $\mathrm{\Delta }$ needs to be determined experimentally, for it to be considered large enough.

Having performed the sensitivity analysis, we determine the behavior quality factor [27], as $B=({\mu }_P,{\mu }_P^{*},{\sigma }_P)$, for $Re$.

STEP 8

In summary, we define the following:

$$\begin{array}{c}\hfill Re=(\{SS,O\left(kn\right),B=({\mu }_P,{\mu }_P^{*},{\sigma }_P),A=Avg({\delta }_1,{\delta }_2,{\delta }_3)\},F(L_I,L_{TP},L_O,P))\end{array}$$

(9)

4.3. Defining the Actionability CTI Quality Metric

Similar to Section 4.2, to quantify the actionability of a CTI product, we set organization C as a reference point for the definition and measurement of $Ac$. As mentioned in Section 3.1, actionability is related to decision-making processes, especially cybersecurity decision-making processes. As a result, to define the $Ac$ metric, we first need to analyze what a cybersecurity decision-making process is.

4.3.1. Cybersecurity Decision-Making Process and Actionability

The decision-making process has been analyzed in the literature in different contexts. However, from a cybersecurity perspective, Magna et al. [47] have modeled the decision-making process and analyzed the types of information needs that are required for a successful choice of action (decision). At the same time, Cotae et al. [48] distinguished three categories of decision-making in cybersecurity: decision-making (a) under-certainty, (b) under-risk, and (c) under-uncertainty. Those categories reflect the state of knowledge of a decision-making system at the moment when a decision is made. From this aspect, a CTI product is actionable when it transforms a decision-making process from an under-uncertainty to an under-risk or under-certainty process.

Furthermore, in the literature [43,49], the actions that a decision-making process can result in are summarized in three capability categories: (a) prevention, (b) detection, and (c) response/recovery. Consequently, a CTI product is actionable when it leads to a state change in these capabilities (e.g., adding new rules to an IDS, which is part of the detection capability). In this study, we model these capabilities as cognitive agents, calling them defense mechanisms to define the quality metric of actionability of CTI products, $Ac$.

4.3.2. Defense Mechanism Modeling

In this study, we define a defense mechanism as a cognitive agent [50,51]. This means that a defense mechanism comprises an inference engine and a knowledge base. In addition, a defense mechanism can receive input from the environment (by communicating), perceive the environment, and generate an output that affects its environment or updates its knowledge base (see Figure 2a).

Consider a typical rule-based IDS as an example of how we can model a system with detection capability as a defense mechanism. This IDS consists of a knowledge base (i.e., the IDS’s ruleset), which stores the alert rules, and a reasoning mechanism that generates alerts. Furthermore, it receives input as new alert rules or network traffic. In the first case, the IDS updates its knowledge base, while in the second case, the IDS’s reasoning mechanism compares the network traffic with the alert rules and takes an action (e.g., generates an alert or not).

From a CTI quality perspective, the moment a defense mechanism receives a CTI product ($P_i$) as input, it has its knowledge base in a given state $s_{t_i}$. Then, we say that a CTI product causes an action (that is, it is actionable) when the defense mechanism updates its knowledge base to a new state $s_{t_{i+1}}$. So, we can express the actionability of a CTI product in terms of a defense mechanism as the conditional probability of its knowledge base changes to the new state $s_{t_{i+1}}$ given the current state of the knowledge base $s_{t_i}$ and the input $P_i$. In summary, we say $Ac\left(P_i\right)=Pr\left(s_{t_{i+1}}\right|s_{t_i},P_i)$.

4.3.3. Actionability Metric Generic Calculation Mechanism

In the case of an organization C, with a set of defense mechanisms $D={\left\{d_t\right\}}_{t=1}^n$, all of which have their knowledge base in a randomly defined state $s_{t_i}$ with $i\in \mathbb{N}$, a CTI product independently of its other quality factors, can be actionable if it leads to a decision that changes one or more of the states of the knowledge base of defense mechanisms.

We observe that to measure actionability, we must examine whether a CTI product finally leads to any knowledge base change. However, if we explore all the CTI products individually, we do not need to have a quality metric. So, in the case of organization C, we propose an approximation of the overall actionability of a CTI product based on the observation that the states of the knowledge base of the defense mechanisms are the result of decisions taken against CTI products that have previously appeared. So, we associate each knowledge base state ($s_{t_i}$) of the defense mechanism with the respective set of CTI products $S_{t_{P\left(i\right)}}=\{P_{t1},...,P_{ti}\}$, which leads to this state. We estimate the actionability of a given CTI product $P_{ti+1}$ in the organization C as the average probability that $P_{ti+1}$ is a member of these sets $S_{t_{P\left(i\right)}}$. In the case of large amounts of unstructured CTI products, to estimate this probability, we propose the use of Cuckoo filters. Specifically, we construct a Cuckoo filter, $CF\left[S_{t_{P\left(i\right)}}\right]$, for each $S_{t_{P\left(i\right)}}$ (that is, a Cuckoo filter for each defense mechanism), and calculate the actionability of a new CTI product, $P_{ti+1}$, based on the membership test of it against the Cuckoo filters. Figure 2b presents the generic calculation mechanism of the proposed actionability metric.

Similarly to Section 4.2.3, we apply the eight-step methodology of Table 2. For the sake of brevity, we avoid explaining the purpose of each step as we do in Section 4.2.3.

STEP 1

Based on our analysis in Section 4.3.1 and Section 4.3.2, we propose a metric $Ac$, which measures the actionability of the CTI product P from the defense mechanisms of an organization C.

STEP 2

Following the previous discussion, we observe that the variables involved in the calculation of $Ac$ are the CTI product, P, for which we estimate its actionability and n Cuckoo filters (that is, equal to the number of defense mechanisms of organization C). So, the set of variables, X, used for the estimation of $Ac$ is as follows: $X=\{P,CF\left[S_{P\left(1\right)}\right],...,CF\left[S_{P\left(n\right)}\right]\}$.

STEP 3

Based on the methodology of Table 2, we have to define the function F that computes the metric $Ac$. As mentioned previously, the idea is to measure the actionability of a CTI product P in relation to the organization C, by testing the potential membership of P in the sets of CTI products that have contributed to the knowledge base state of each defense mechanism of C. All defense mechanisms are equally weighted in membership testing. So, we test P against each constructed Cuckoo filter and average the total score. Specifically, we define the function F as follows:

$$F\left(X\right)={\displaystyle \frac{{\sum }_{i=1}^n\left({\delta }_i\right)}{n}},where{\delta }_i=\left\{\begin{array}{c}1,ifPmember ofCF\left[S_{P\left(i\right)}\right]\hfill \\ 0\hfill \end{array}\right.$$

(10)

STEP 4

Similarly to Section 4.2.3, in this step we determine the objectivity and subjectivity, $\mathrm{\Gamma }$, of $Ac$. We observe that none of the variables of F are affected by a human factor, so we infer that $Ac$ is calculated by objective data. On the other hand, the membership decision that a Cuckoo filter provides comes with a probability of false positive [32], which depends on the Cuckoo filter’s constructive characteristics. So, the decision part of F is non-deterministic and we consider that F is a subjective function applied to objective variables. Hence, we infer that $\mathrm{\Gamma }=SO$.

STEP 5

To estimate the performance M of Table 2 for $Ac$, we use the Algorithm 2, from which we observe that the algorithm performs n lookup on the Cuckoo filters of $O\left(1\right)$ cost in time complexity [38], and a division of $O\left(1\right)$. So, $Ac$ has a performance of $O\left(n\right)$ expressed in time complexity.

Algorithm 2$Ac$ Metric Calculation Algorithm

Require: P, $CF\left[S_{P\left(1\right)}\right]$,…, $CF\left[S_{P\left(n\right)}\right]$ $F,sum\leftarrow 0$     for i in range(1,n) do        if $P\in CF\left[S_{P\left(i\right)}\right]$ then            $sum+=1$        end if     end for $F\leftarrow sum/n$

STEP 6

Although the characteristics of the Cuckoo filters, $CF\left[S_{P\left(i\right)}\right]$, introduce a nondeterministic bias, at the time that F computes the value of $Ac$, $CF\left[S_{P\left(i\right)}\right]$ are stable and do not change by F. So, based on Algorithm 2, we infer that F is deterministic and does not introduce bias in the computation of $Ac$, thus the accuracy of, A, and Table 2 is equal to one, $A=1$.

STEP 7

Similarly to Section 4.2.3, we apply the elementary effects method for the theoretical sensitivity analysis of F. Once again, only changes in the variable P of X are not controlled by C. Furthermore, we observe that P is tested against n Cuckoo filters regarding its potential membership in the set of CTI products that construct each of those Cuckoo filters. In this case, we again assume the t selected levels and the elementary effect of P, $E{E}_P$, is as follows:

$$E{E}_P={\displaystyle \frac{F(\overline{Y},CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right])-F(P,CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right])}{\mathrm{\Delta }}}$$

(11)

where $\mathrm{\Delta }ϵ\{{\displaystyle \frac{1}{t-1}},1-{\displaystyle \frac{1}{t-1}}\}$ and $\overline{Y}=P\pm \mathrm{\Delta }$. Then, the mean (${\mu }_P$), the mean of the absolute values (${\mu }_P^{*}$), and the standard deviation (${\sigma }_P$) of the distribution $F_P$ of $E{E}_P$, which is obtained by randomly sampling $\overline{Y}$ from $\mathrm{\Omega }$, assess the influence of P in $Ac$ and reveal the total effects of the interactions between the variable P and the variables $CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right]$. So, we have for the calculation of ${\mu }_P$:

$$\begin{array}{c}\hfill {\mu }_P={\displaystyle \frac{1}{r}}\sum _{j=1}^{r}E{E}_P={\displaystyle \frac{1}{r}}\sum _{j=1}^r{\displaystyle \frac{F(\overline{Y},CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right])-F(P,CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right])}{{\mathrm{\Delta }}_j}}=\\ \hfill {\displaystyle \frac{1}{r}}\sum _{j=1}^r\left({\displaystyle \frac{\frac{{\sum }_{i=1}^n\left({\delta }_{\overline{Y_j}i}\right)}{n}-\frac{{\sum }_{i=1}^n\left({\delta }_{P_{j}i}\right)}{n}}{{\mathrm{\Delta }}_j}}\right)=\end{array}$$

(12)

We notice that $Y=P\pm \mathrm{\Delta }$ and the two sums $s_{\overline{Y}j}={\displaystyle \frac{{\sum }_{i=1}^n\left({\delta }_{\overline{Y_j}i}\right)}{n}}$ and $s_{Pj}={\displaystyle \frac{{\sum }_{i=1}^n\left({\delta }_{P_{j}i}\right)}{n}}$ are independent because they sum the membership decisions of the Cuckoo filters, which are based on hash functions; as a result, even a minor ${\mathrm{\Delta }}_j$ change can cause a different decision in the Cuckoo filters because the avalanche effect of the hash functions affects them. So, we can consider the difference $a_j=s_{\overline{Y}j}-s_{Pj}$ as a random value, which is independent of P. So, we have the following:

$$\begin{array}{c}\hfill {\mu }_P={\displaystyle \frac{1}{r}}\sum _{j=1}^r\left({\displaystyle \frac{a_j}{{\mathrm{\Delta }}_j}}\right)randomvalueindependentfrom\overline{Y}andP\end{array}$$

(13)

For the calculation of ${\mu }^{*}$ and $\sigma$, we have the following:

$$\begin{array}{c}\hfill {\mu }_P^{*}={\displaystyle \frac{1}{r}}\sum _{j=1}^r\left({\displaystyle \frac{|a_j|}{{\mathrm{\Delta }}_j}}\right)randomvalueindependentfrom\overline{Y}andP\end{array}$$

(14)

$$\begin{array}{c}\hfill {\sigma }_P^2={\displaystyle \frac{1}{r-1}}\sum _{j=1}^r({\displaystyle \frac{a_j}{{\mathrm{\Delta }}_j}}-{\mu }_P{)}^{2}independentfrom\overline{Y}andP\end{array}$$

(15)

In conclusion, we can infer from ${\mu }_P$ (Equation (13)) and ${\mu }_P^{*}$ (Equation (14)) that P has an influence on $Ac$ which does not depend on the magnitude of change $\mathrm{\Delta }$. Furthermore, ${\sigma }_P$ (Equation (15)) reveals that the interactions between P and the variables $CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right])-F(P,CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right]$ do not depend on the magnitude of the change ($\mathrm{\Delta }$).

After performing the theoretical sensitivity analysis, we determine the behavior quality factor [27], as $B=({\mu }_P,{\mu }_P^{*},{\sigma }_P)$, for $Ac$.

STEP 8

Finally, we define the following:

$$\begin{array}{c}\hfill Ac=(\{SO,O\left(n\right),B=({\mu }_P,{\mu }_P^{*},{\sigma }_P),A=1,F(P,CF\left[S_{P\left(1\right)}\right],\dots ,CF\left[S_{P\left(n\right)}\right]))\end{array}$$

(16)

Please note that the built-in false positive rate (i.e., $P_{fp}\approx {\displaystyle \frac{2b}{2^p}}$) associated with the Cuckoo Filters, as outlined in Section 3.3.2, has a minor impact on the measurement of $Ac$. Specifically, let us assume the worst-case scenario in which all the decisions taken by the n $CF\mathrm{s}$ are false positives. Then, the probability that $Ac=1$ equates to a notably small value ${\left({\displaystyle \frac{2b}{2^p}}\right)}^n$, since the hash value length p is significantly greater than the bucket size b.

5. Implementation on a Hypothetical System—Experiments

In this part, we implement the metrics proposed in Section 4 by developing experimental environments in parallel. In addition, we present a way for an organization to adjust one of the proposed metrics to its own security and environmental requirements. For the rest of this section, we set the experimental environments of the two metrics in Section 5.1 and Section 5.2, and present the experimental results of them in Section 5.3 and Section 5.3.4.

5.1. Experimental Environment of Relevance Metric

An organization C is expected to define its information needs as part of its risk management process. For example, we can assume that an organization using Amazon Web Services S3 buckets to store its clients’ data needs information related to cybersecurity violations against S3 buckets. More specifically, assume that C has a supplier $S_1$. In that case, the information needs of C regarding $S_1$ are not limited to, but can be expressed by the following queries:

• Does $S_1$ face any cyberattack?
• Are $S_1$ products affected by any vulnerability?

Obviously, to determine its information needs regarding $S_1$, C needs to expand the previous list with queries related to the products, the business activities, etc. Moreover, the answer to some queries (for example, “What are the common cyber threats that affect the agricultural business area?”) can result in more sub-queries (for example, “What ransomware has been used against other agricultural companies?”) whose answers should be added as responses to the information needs of C.

To simulate the definition of information needs as those distributed across the three landscapes, $L_I$, $L_{TP}$, and $L_O$, we emulate inter-organizational processes utilizing artificial intelligence (AI) and business ontologies. So, in our experiments, we create random organizations $C_1,C_2,...C_n$ and determine their functions by creating organizational profiles. Those profiles are constructed by randomly selecting entities (e.g., business areas) from ontologies and structured naming schemes. The use of ontologies ensures that the randomly created profiles follow a common logic. For example, when the information technology business area is selected, then the business activities or the products cannot belong to the agricultural business area. An organizational profile is a simplified textual description of an organization’s environment, and it is introduced here to overcome the need for a detailed organizational description (e.g., operations, customers, suppliers, functions, business areas, products, etc.). For simplicity, we construct those profiles to match the three landscapes.

Table 3. Relevance Metric Landscapes’ Datasets.

Landscape	Information Needs of	Ontology and Datasets
Input Landscape ($L_I$)	Suppliers	Companies [52]
	Competitors
	Capital Sources	Financial Industry Business Ontology (FIBO) [53]
Transformation Process Landscape ($L_{TP}$)	Business Activities	Nomenclature statistique des Activites economiques dans la Communaute Europeenne (NACE) [54], Domain Industry Taxonomy (DIT) [44]
	Internal Operations	General Process Ontology (GPO) [55]
	Information Systems	Common Platform Enumeration (CPE) [56]
Output Landscape ($L_O$)	Products Services	FIBO [53], Product Types Ontology (PTO) [57], European Core Conceptual Framework (ECCF) [58]

presents the utilization of the ontologies. Then, we use AI to develop the information needs, in the form of queries, for the three landscapes (see Figure 3a).

Furthermore, for the selection of CTI products, we have deployed an OpenCTI server [59], which collects various CTI products from many sources. In our experiments, we randomly select some of those products and calculate $Re$.

Details of those sources and the configuration of the OpenCTI server are available as part of the source code of this work. Figure 3b depicts the functionality of the experimental environment for $Re$.

5.2. Experimental Environment Actionability Metric

To implement and evaluate the experimental environment of $Ac$, we consider an organization C, that has k defense mechanisms in place that are equally distributed in the three capability categories (that is, prevention, detection, and response/recovery).

In this case, we used the sources of an OpenCTI server to randomly collect t CTI products. We distributed the collected CTI products equally in k sets, each representing the knowledge base of a defense mechanism. Then, we used these k sets to construct the Cuckoo filter of each defense mechanism (see Figure 4a). Finally, we used these Cuckoo filters to calculate $Ac$ of different CTI products collected again from the OpenCTI server. Figure 4b depicts the experimental environment for the calculation of $Ac$.

5.3. Analysis of Experimental Results

Following the assumptions made in the setup of the experimental environment of the metrics, in this section, we present the results of the calculation of the metrics against CTI products collected from various sources (see

Table 4. CTI Products Sources.

CTI Sources	Num. of CTI Products in Dataset	Num. of CTI Products in Validation Dataset
MITRE ATT&CK [18], CISA KNOWN VULNERABILITIES [61], CVE [62], ALIENVAULT [63], FEEDLY [64], MALPEDIA [65], MISP FEEDS [66], MITRE ATLAS [67], TWEETFEED [68]	32,012	5000

). In our case, the gathered CTI products vary in context and complexity, which is a common situation for numerous organizations [60]. Therefore, the CTI products studied include both straightforward CTI feeds and more detailed incident reports. The experiments utilized a total of 32,012 CTI products, with 5000 reserved as a validation set, and the remainder employed in setting up the environments.

Additionally, in the interest of thoroughness,

Table 5. Experimental Organizations Characteristics of $Re$ metric.

Org.	Input Landscape (${\mathit{L}}_{\mathit{I}}$)			Transformation Process Landscape (${\mathit{L}}_{\mathit{TP}}$)			Output Landscape (${\mathit{L}}_{\mathit{O}}$)
	Num. of Suppliers	Num. of Competitors	Num. of Capital Sources	Num. of Business Activities	Num. of Internal Operations	Num. of Information System	Num. of Products	Num. of Services
$rc1$	12	16	10	18	10	18	1	8
$rc2$	12	6	6	8	4	18	3	1
$rc3$	14	19	7	12	10	15	5	9
$rc4$	19	6	6	16	6	18	5	5
$rc5$	17	18	5	10	1	16	7	7
$rc6$	7	12	11	14	7	17	1	2
$rc7$	9	14	4	7	9	19	9	1
$rc8$	12	14	11	19	9	13	5	3
$rc9$	19	5	7	13	10	18	9	3
$rc10$	9	13	2	5	7	11	10	3

and

Table 6. Experimental Organizations Characteristics of $Ac$ metric [KB: Knowledge Base, DM: Defense Mechanism].

Org.	Num. of DMs	Num. of CTI Products in KB of a DM	Total Num. of CTI Products in the KBs of the DMs
$ac1$	17	179	3043
$ac2$	15	477	7155
$ac3$	3	1789	5367
$ac4$	18	293	293
$ac5$	27	346	346
$ac6$	5	507	507
$ac7$	12	804	804
$ac8$	14	417	417
$ac9$	4	1917	1917
$ac10$	8	1116	8298

detail the properties of the organizations generated at random for the experiments concerning the $Re$ and $Ac$ metrics, respectively.

5.3.1. Relevance Metric Experimental Results Analysis

$Re$ calculation quantifies the relationship between the content of a CTI product and the information needs of organizations. As explained in Section 5.1, we have constructed artificial organizations to evaluate the calculation of $Re$ against their information needs, such as those expressed through the landscapes. In Figure 5a, we present the $Re$ calculation distribution of one hundred CTI products randomly selected from the validation dataset against the ten artificial organizations. We observe that the distribution of the calculation of the metric varies between 0.001 and 0.20, with a mean value of approximately 0.025 for all organizations, which demonstrates the capabilities of $Re$ as a filter for CTI products.

To better explain the previous conclusion, in Figure 6b, we present the comparison of CTI products with the highest calculation of the metric against the artificial organizations of this experiment, in which we observe that only a small percentage of CTI products have an $Re$ higher than the mean value. So, to demonstrate how an organization can use the metric as a filter to decide which CTI products are worth further examination, let us assume that the organization $rc1$ defines a threshold of $Re=0.10$ for choosing CTI products as interesting. From this, we can infer that in the case of our experiment, $rc1$ we will have to examine two CTI products out of the hundred for which $Re$ has been calculated. Moreover, using the mean calculation of $Re$ (see Figure 5a), $rc1$ can estimate the number of potential CTI products that require further examination and, by extension, the required resources.

In addition, an organization can assess the thoroughness of its information needs for each landscape by analyzing the way its landscape contributes to the overall calculation of the metric. In Figure 6a, we present the average metrics calculations for each landscape per organization. Based on that, an organization can identify significant differences between landscapes (e.g., organization $rc3$), which can be an indicator of an insufficient definition of the information needs of a landscape.

In summary, an organization can employ $Re$ in several ways, which include the following:

• Serving as a filter for unstructured CTI products. The organization can define a $Re$ threshold, allowing only those CTI products for further analysis where the calculated $Re$ surpasses this set threshold.
• Acting as a resource predictor for the assessment of CTI products. This utilizes the distribution traits of calculated $Re$ from prior unstructured CTI products and the $Re$ threshold. For instance, if the $Re$ values for one thousand unstructured CTI products adhere to a normal distribution (i.e., mean $\mu$ and standard deviation $\sigma$) and the organization establishes a threshold of $\mu +3\sigma$, then a cybersecurity manager can predict that out of every thousand unstructured CTI products received, only about 10 will necessitate further analysis, enabling efficient resource allocation.
• Functioning as an indicator of how well-defined the organization’s information needs are by analyzing each landscape’s impact on the $Re$ calculation.

5.3.2. Actionability Metric Experimental Results Analysis

The $Ac$ calculations of the CTI products against the artificial organizations of the experiment quantify the probability that a CTI product will change the state of their defense mechanisms. In Figure 5b, we present the calculation distributions of $Ac$ of one hundred CTI products across the organizations. We observe that most of the calculations are in the range of $Ac=0.75$ and $Ac=0.99$ which comes from the fact that each organization is represented by a number of defense mechanisms and their respective Cuckoo filters, which vary from five to twenty, whose knowledge bases have been formed by the limited amount of CTI products and sources mentioned in Table 4. However, $Ac$ can be applied as a filter for CTI products (e.g., setting a threshold on the mean value of each organization), and an organization can infer from these distributions how useful the CTI products consumed for them are and, by extension, evaluate the quality of the CTI sources.

Furthermore, in Figure 7a, we present the calculations of the CTI products $Ac$ per defense mechanism in the organization $Ac10$. The diagram reveals whether a CTI product can affect the state of each defense mechanism. In that way, an organization can choose which of its defense mechanisms a specific CTI product can be applied to, minimizing the resources required for a more detailed examination of a CTI product.

In addition, in Figure 7b, we present the comparison of the $Ac$ calculations of the CTI products between the different organizations. We observe that the calculations appear to follow a similar pattern between them, which is evident from the fact that the knowledge bases of their defense mechanisms have been created by randomly selecting CTI products from the same pool. However, even under this constraint, we observe how organizations with similar defense mechanisms can use the metric to select CTI products that are useful for their environment. For example, let us assume that two organizations with similar defense mechanisms participate in a CTI information sharing community, if the first organization calculates $Ac$ of one CTI product and shares this value within the community, then the second organization can use this information as a selection criterion for the CTI product.

In summary, the prospective application of $Ac$ by the organization can be summarized as follows:

• Analogous to $Re$, $Ac$ can be employed as a filter for unstructured CTI products by setting a threshold for $Ac$ and selecting only those CTI products for further review whose calculated $Ac$ is above the threshold.
• It can act as an assessment metric for a CTI source by comparing the average $Ac$ values of unstructured CTI products sourced from different CTI sources.
• It can act as a means to accurately apply a CTI product by evaluating how the CTI product might impact each defense mechanism.
• It can function as a selection criterion for CTI products among organizations that have a similar arsenal of defense mechanisms and are part of a threat intelligence sharing community.

5.3.3. Relevance and Actionability Metrics Experimental Results of an Organization

In this section, we examine the experimental results of the measurement of $Re$ and $Ac$ of a number of selected CTI products for two of the organizations created randomly. Specifically, in

Table 7. Organization $rc3$ Profile.

Landscape	Profile
Input Landscape ($L_I$)	Num. of Suppliers: 14 (e.g., GRIVE)
	Num. of Competitors: 19 (e.g., M.A.P.L.E)
	Num. of Capital Sources: 7 (e.g., SPDR S&P 500 ETF Trust)
Transformation Process Landscape ($L_{TP}$)	Num. of Business Activities: 12 (e.g., “auxiliary to financial services”)
	Num. of Internal Operations: 10 (e.g., Information Transport Process)
	Num. of Information Systems: 15 (e.g., XR3Player)
Output Landscape ($L_O$)	Num. of Products: 5 (e.g., carpets, food products)
	Num. of Services: 9 (e.g., community services)

, we present the profile of the organizations $rc3$. For simplicity, we give the numbers and an example of the different entities that comprise each landscape; however, the profile is available in this paper’s source code. Furthermore, in

Table 8. Organization $ac9$ Structure.

Number of Defense Mechanisms	Number of CTI Products in Knowledge Base of a Defense Mechanism	Total Number of CTI Products in the Knowledge Bases of the Defense Mechanisms
4	1917	7668

we present the structure of $ac9$, particularly the number of defense mechanisms that comprise the $ac9$ and the number of CTI products assigned to their knowledge bases. In

Table 9. CTI Products used for the Measurement of $Re$ and $Ac$.

Measurement of $\mathit{Re}$ Metric		Measurement of $\mathit{Ac}$
Product	Remark	Product	Remark
P1	Poll Vaulting Report	P1	NETBIOS Scanner Report
P2	Wrong Sphere Vulnerability Report	P2	Cross-Site Scripting Vulnerability Report
P3	OT URL Activity Report	P3	SQL Injection Attack Report
P4	Linux Kernel Vulnerability Report	P4	Wrong HTTP Header Encoding Report
P5	Firmware Buffer Overflow Vulnerability Report	P5	Wrong HTTP Header Encoding Report

, we present the CTI products for which $Re$ and $Ac$ have been calculated for $rc3$ and $ac9$, respectively. Finally, in Figure 8a,b, we present the measurement of $Re$ and $Ac$ of the CTI products in Table 9 for organizations $rc3$ and $ac9$, respectively.

In the case of organization $rc3$, the organization’s experts can use $Re$ to select the $P1$ (Poll Vaulting Report) for further examination and analysis between the five presented CTI products. By examining the profile of $rc3$ (e.g., $rc3$ offers “community services”), we can validate that $P1$ is probably relevant to it. Similarly, in the case of $ac9$, we observe that the organization’s experts can use the $Ac$ to select $P1$ (NETBIOS Scanner Report), $P2$ (Cross-Site Scripting Vulnerability Report), and $P3$ (SQL Injection Attack Report) for further investigation regarding their applicability in the organization’s defense mechanisms. Moreover, we observe that the CTI products (P1–5) have a high $Ac$ value and are related to web/network traffic, which is explained by the fact that the knowledge bases of $ac9$’s defense mechanisms consist of such a type of information.

5.3.4. Experimental Results of $Re$ and $Ac$ Performance

To empirically validate the performance of $Re$ and $Ac$, as discussed in Section 4.2.3 and Section 4.3.3, we once again employ the organizations $rc3$ and $ac9$, respectively. For this purpose, we have designed ten random sets of CTI products ($s_1,s_2,\dots ,s_{10}$) by incrementally adding one hundred CTI products to each subsequent set, beginning with one hundred. Subsequently, we compute the execution time required to calculate $Re$ and $Ac$ for each individual set. Figure 9 illustrates the observed outcomes. Additionally, we include measurements of memory usage for the computation of $Re$ and $Ac$ (Refer to Figure 9).

The experimental outcomes validate the theoretical performance evaluations of $O\left(kn\right)$ for $Re$ and $O\left(n\right)$ for $Ac$, respectively.

5.3.5. Discussion

After detailing the experimental implementation, it is necessary to clarify why, at this stage, we must refrain from comparing the proposed metrics with other CTI quality assessment methodologies or metrics, and why this will be considered a focus of our future research endeavors in the area of CTI quality.

Although numerous works cite the significance of measuring CTI quality [10,69], few [13,70] delve into the technical implementation details seen in other fields [71]. On the other hand, there are implementations often involving CTI quality metrics that are deeply integrated into larger systems, making them hard to isolate for comparative measurement [12,15,17]. Furthermore, the differentiation between structured and unstructured CTI products necessitates a potential comparison to use purpose-specific datasets, a topic for which, to the best of our knowledge, there is a scarcity of research. Moreover, the methodologies and metrics for CTI quality, particularly within threat intelligence platforms [7], remain proprietary and inaccessible to the public. Available CTI quality metrics, regardless of their structured or unstructured focus, are utilized within specialized contexts [70], needing replication to facilitate metric comparison. Thus, we suggest this comparison as a potential avenue for future research.

6. Conclusions

In summary, this study presents a novel approach to evaluating the quality of unstructured CTI products through the introduction of Relevance ($Re$) and Actionability ($Ac$) metrics. These metrics provide a structured and scalable approach for evaluating CTI products, aligning them with the unique informational needs and defense mechanisms of organizations. Although the study acknowledges certain constraints, such as the limited scope of the datasets used, it provides a solid foundation for future research and practical applications. By addressing these limitations and exploring future research opportunities, this work establishes the foundations for more robust, adaptive, and automated cybersecurity solutions to enhance the utility of CTI in the ever-changing cybersecurity domain. In the remainder of this section, we examine the contribution of this study in Section 6.1, discuss its limitations and the underlying assumptions in Section 6.2, and explore potential future research directions in Section 6.3.

6.1. Contribution

In this paper, we explore the aspects of CTI quality, focusing on the relevance and actionability of unstructured CTI products. The increasing reliance on CTI in cybersecurity and the necessity for organizations to evaluate the quality of the intelligence they consume have introduced significant challenges in evaluating the quality of these products. Organizations rely on relevant and actionable CTI products to make cybersecurity decisions, but the lack of standardized metrics complicates this process. This paper proposes two innovative metrics, Relevance ($Re$) and Actionability ($Ac$), to address these challenges.

The research questions posed in this study aimed to assess how the relevance and actionability of CTI products can be defined and quantitatively calculated. The main contribution of this paper is the proposal of two metrics: $Re$ and $Ac$. The proposed metrics are designed to incorporate the characteristics of organizations by evaluating the quality of CTI products in relation to organizations’ information needs and defense mechanisms, respectively. To answer the first and second research questions, we have introduced the concepts of informational landscapes, which is a notion introduced for the first time on CTI to the best of our knowledge, and the modeling of the organizations’ defense mechanisms as cognitive agents. Furthermore, by leveraging probabilistic algorithms and data structures, these metrics provide a scalable approach to assess the quality of CTI products, ensuring their applicability in large datasets of CTI products, and answering our third research question. Based on the information currently available to us, it is the first time that probabilistic data structures and algorithms are used in the context of CTI quality evaluation. In addition, the experimental findings indicate that these metrics can serve as effective filters for organizations seeking to prioritize their CTI analysis efforts, ensuring that only the most pertinent information consumes their cybersecurity resources. To summarize, Section 4.2 and Section 4.3 provide answers to the RQ1, while Section 3.3, Section 4.2.2 and Section 4.3.2 address the RQ2. Lastly, RQ3 is covered in Section 4.2.3, Section 4.3.3 and Section 5.3.

The main advantages of our approach, in contrast to similar endeavors, can be outlined in the following key points:

• Our metrics are designed using a systematic methodology that facilitates a clear analysis of any component.
• The proposed metrics are independent of any specific system, allowing for their integration into existing systems.
• Metrics evaluation yields explicit numerical results, which are essential in data quality metrics [72], and are not part of a comprehensive framework that transparently demonstrates the application of the calculated results.
• The metrics are independent of the CTI product structure, enabling the potential evaluation of structured CTI products, given that a simple data cleaning technique is applied to their inputs.

Additionally, due to the abstract concepts on which the proposed metrics are based, they are resilient to the dynamic nature of cyber-threats. Simultaneously, they can be seamlessly integrated with established frameworks such as MITRE ATT&CK [18], because the provided CTI products include extensive textual descriptions that can serve as input for the proposed metrics. Specifically, we can easily pinpoint within campaigns like C0010 [73], Night Dragon [74], and Operation Spalax [75], textual content about particular business areas, functions, and systems in an organization, which will, in the case of, for example, $Re$, elevate the computed value.

Furthermore, at this point, we have to discuss the benefits of the proposed metrics in comparison with asking an AI model how relevant or actionable a CTI product is for an organization. First, we should recognize that the use of AI makes it possible to evaluate the relevance and actionability of CTI products. However, an AI model trained with organization’s classified data (e.g., models of IT systems, architecture of an organization’s IT environment, business activities, etc.) introduces a significant risk to an organization’s cybersecurity posture because an attacker capable of exploiting this model would identify and exploit the organization’s weaknesses [76,77]. In addition, it is difficult for an organization to know the level of efficacy that an AI model has when it continuously handles a large number of CTI products. In comparison, the proposed metrics do not rely on any communication with models or algorithms hosted outside of the organization. At the same time, the proposed metrics use algorithms that are designed to handle a large amount of data by definition.

6.2. Limitations and Assumptions

Despite the contributions of this work, we need to recognize that our study faced several constraints, which open future research paths.

Initially, the limited scope of the datasets employed to validate the proposed metrics compelled us to make certain assumptions during the design of experimental organizations. This led to the artificial creation of their information needs and knowledge bases for defense mechanisms, potentially restricting the generalizability of our results. As mentioned in Section 5, the scarcity of CTI datasets prompted us to assemble the dataset for this study by collecting CTI products using an OpenCTI server along with specific CTI sources for a designated timeframe. This mirrors the approach a new organization might adopt for CTI. Nonetheless, this method might have limited the data to particular attack types and tactics. Furthermore, in crafting the landscapes, we implemented AI to develop the experimental organizations’ information needs, possibly introducing bias based on the state of the AI model used. However, we consider this approach more advantageous than the manual landscape constructions by a small team of researchers.

Secondly, due to their probabilistic nature, the proposed metrics come with an acceptable margin of error that might impact precision in certain situations. Nevertheless, this paper does not explore such scenarios for the sake of simplicity. To clarify this limitation, it is important to delve into the nature of probabilistic data structures. These structures are intentionally designed to trade some degree of accuracy for reduced time and space complexity. Consequently, their application introduces a minor error, the analysis of which is outside the scope of this study and will be addressed in our future research.

Third, to maintain minimal complexity, we have kept the design and implementation of the experimental environments simple, but we must acknowledge that incorporating the metrics into an organization’s current processes and systems can be difficult. This difficulty arises from the complexity of pinpointing information needs and managing defense mechanisms within an organization because these tasks involve, on one side, individuals not specialized in cybersecurity, and on the other side, a range of mostly proprietary systems and technologies.

6.3. Future Work

In summary, looking towards the future, our findings give rise to several open research questions:

• How can we experimentally evaluate the CTI quality metrics, including common data analytics methods like precision-recall and confusion matrices?
• How can all characteristics of probabilistic data structures be utilized in the computation of metrics (e.g., dynamic updates in Cuckoo filters)?
• Is it possible to clearly and systematically define organizational information needs in the CTI context?
• How can these metrics be employed to incorporate selected CTI products into the knowledge bases of organizations’ defense mechanisms in real-time?
• In what ways can these metrics be integrated into existing CTI frameworks and systems?

Based on these research questions, we propose the following future research directions:

• Developing benchmark datasets specifically for CTI.
• Investigating the experimental comparison of CTI quality metrics.
• Examining the use of alternative probabilistic data structures for measuring CTI quality.
• Formally defining the informational needs of organizations concerning CTI.
• Researching the real-time integration of CTI products into organizational defense mechanisms, grounded in their CTI quality.
• Investigating the integration of CTI quality metrics into existing frameworks and open-source systems.

Our future work will concentrate on employing these metrics for the real-time integration of CTI products into organizational cyber defenses. We aim to assess the impact of CTI quality in enhancing an organization’s cybersecurity posture. This will involve developing automated tools for real-time CTI assessment and integration, aligning the metrics with current cybersecurity frameworks, and exploring their application in collaborative environments like threat-sharing platforms.