Evaluating the Effectiveness of Information Security Management Systems: An Analysis Framework and Key Metrics

Abstract

As large scale digitization continues to reform business processes, one critical challenge organizations are currently facing is managing the staggering amount of data flowing. Further, with large datasets comes the added complexity of insuring a cyber secure environment and shielding the information security management system (ISMS) from undesirable manipulations. Today’s drastic rise of cyberattacks urges the need for effective security frameworks to guard against unauthorized access and malicious acts impeding business operations. The latter of which compelled organizations to adopt holistic information security approaches, commonly implemented via ISMS frameworks. Further, to maintain an effective ISMS, ongoing monitoring and measurements are highly required. Considering the aforementioned points, this paper explores how organizations measure the effectiveness of their ISMS focusing on key performance indicators, metrics, and foundational components involved in information security management by categorizing metrics into governance, risk, and incident response as well as determining the maturity level based on ISO alignment, the presence, specificity and automation of KPIs. Based on empirical interviews with eight diverse organizations, the research findings reveal a wide range of maturity among organizations, from those lacking clear defined KPIs to those with sophisticated multi-layered systems. While special attention is paid to incident-response management, companies with a strong ISMS stand out because they use automated and proactive metrics for strategic reporting, whereas companies with a weaker ISMS often do not have organized KPIs and depend on random manual audits. Based on these results, the present work suggests an analysis framework for evaluating ISMS effectiveness. While previous studies have struggled to define clear ISMS measurement practices, this paper aims to provide insights on measurements by identifying the core building blocks of ISMS and revealing how they are evaluated to drive continual ISMS improvement.

1. Introduction

Digitalization of business processes is a two-sided challenge; while it improves organizations’ capabilities, it also creates vulnerabilities. It has become essential in most organizations to adapt to this transformation, and the more digital assets the organization owns, the more it is a concern for it regarding what and how data is being stored and who is accessing it. These are the investigatory questions to be asked when a data breach occurs, and hence it is becoming important for organizations to develop a strategic information security plan for the whole information flow process, ensuring a secure environment with less exposure to malicious cyber activities.

Thus, implementing an ISMS is becoming the backbone of every organization, and failing to properly design and implement an effective one makes organizations exposed to cyberattacks [1]. A notable example of such risks is illustrated by the SingHealth cyber attack [2]. It demonstrates the potential consequences of poor implementation of an ISMS and the exposure it poses for critical national infrastructure. It started as a phishing email and ended as an advanced persistent threat (APT) because of poor human awareness and the lack of technical controls such as a firewall configuration to prevent unauthorized access through remote desktop protocol (RDP). Similarly, the Twitter cyberattack in 2020 highlighted the vulnerabilities in technologically advanced platforms due to insufficient ISMS. The breach started with a social engineering attack that targeted vulnerabilities in Twitter employees’ transition to remote-working systems. The major issues in this case were insufficient training of employees, which led to excessive privilege access in breach of the principle of least privilege, and lack of multi-factor authentication for sensitive systems and internal monitoring systems [3]. These issues highlight that effective security depends on both strong technology and well-trained people working together in a proactive defense. Furthermore, this emphasizes the urgent need for having and maintaining a robust ISMS to ensure public discourse and user trust.

ISMS encompasses a set of documents and tools addressing data/information from multiple angles and it covers four aspects: data collection, data risk analysis, data security assessment, and safety protection [4]. Such systems involve processes, tools, policies and personnel and although standards have been developed and maintained to help organizations manage their data, organizations are still struggling to effectively implement a successful ISMS and they eventually fail to maintain a cybersecure data flow process [5]. In addition to data management systems, processes and operations systems are also included within the cybersecurity sphere, making it critical for organizational safety and business continuity. Platforms and industrial control systems are prone to cyberattacks, as demonstrated by the Stuxnet attack on Iran’s nuclear facilities. Safeguarding these systems also requires the implementation of an effective ISMS, highlighting the need for a structured, risk-based comprehensive approach to security. A way to integrate Operational Technology (OT) security into an ISMS is described in Handbook on Operational Technology and its Security [6].

According to a recent study by Grenefalk et al. [5], challenges associated with the implementation and maintenance of ISMS remain relatively unexplored and there is still a lack of understanding of their effect on information security. Furthermore, according to Haufe et al. [7], there is not a specific process framework for security management that clearly differentiates between ISMS processes and the security measures controlled by ISMS processes. Adding to this, and according to the literature review, there is a lack in measurement approaches of the effectiveness of ISMS and frameworks that combine technical and organizational aspects of ISMS. Based on those findings, this research analyzes challenges and success factors related to ISMS’s robustness. Additionally, the present work aims to understand and identify how to measure ISMS effectiveness according to governance, risk and incident response. Those are to be considered the foundational pillars of ISMS and they should be included in every part of its assessment process.

2. Research Methodology

In order to provide a comprehensive understanding of theoretical and practical approaches for evaluating ISMS effectiveness, this study combines a systematic literature review with qualitative insights from interviews to scrutinize the metrics applied.

Semi-structured open-ended interviews [8] were used during 2025 to collect qualitative information about how organizations, of various sizes and in both private and public sectors, measure ISMS effectiveness, taking into account pragmatic methods and contextual elements. The purpose of semi-structured open-ended interviews is to provide flexibility while keeping the measurement of ISMS effectiveness foremost.

The interview outline was organized into three core dimensions:

1.

Effectiveness and Measurements: How organizations define and track the success of their security systems.

2.

Governance, Risk, and Incident Response: Exploring the integration of critical security pillars, which is essential for assessing alignment with ISO standards.

3.

Challenges and Improvements: Identifying gaps that the analysis framework must address to ensure continual ISMS improvement.

The interview process yielded the necessary data to classify organizations according to their maturity level and evaluate their adherence to ISO standards and best practices. The full interview outline is provided in Appendix A.

The selected organizations, including both public and private companies, were active in the Cybersecurity Node North project at Luleå University of Technology, Sweden, mainly funded by the EU Regional Growth Fund and the regions of Norrbotten and Västerbotten. The eight respondents were professionals responsible for IT, IT services and/or cybersecurity, and profoundly knowledgeable about ISMS and organizational metrics/KPIs and they requested their data to be anonymized. The respondents worked at the following private or public organizations:

• Luleå Energi AB: Founded in 1971, Luleå Energi AB is a municipally owned energy company based in Luleå, Sweden, responsible for supplying electricity, district heating, and broadband infrastructure in Luleå and surrounding areas.
• Nordic Information Control AB: A software company founded in 2021 in Stockholm. They offer a cloud-based platform (the NIC Platform) that automates regulatory compliance and information security efforts for organizations, addressing frameworks like NIS2 [9], DORA [10], GDPR [11], and ISO27000 [12].
• Skellefteå Kommun: A municipality in the region of Västerbotten County, northern Sweden.
• Pensionsmyndigheten: The Swedish Pensions Agency is a national public authority that administers and pays out Sweden’s pensions, it also provides guidance and information on all parts of the pension system.
• Luleå University of Technology: A public university founded in 1971 located in Norrbotten County, Sweden, with its main campus in Luleå, and other campuses in Kiruna, Skellefteå, and Piteå.
• Pitea Energi AB: Founded in 1909, Pitea Energi AB is an energy company based in Piteå, Sweden. It provides essential infrastructure and utility services in the region, delivers and manages electricity distribution (hydropower and solar), district heating, broadband infrastructure, renewable electricity retail, and local electricity production.
• Elastisys AB: Cloud-native security and compliance company, founded in 2011 as a spin-off from Umeå University’s research group in distributed systems and cloud computing.
• Boliden AB: Founded in 1920s, Boliden AB is a Swedish multinational mining company and they operate in Sweden, Finland, Norway, Portugal and Ireland.

The purpose of having multiple types of organizations from different industries, and with different focuses, is to get an in depth understanding of what metrics are used and how ISMS effectiveness is maintained, as well as which perspectives are of interest, considering the similarities and differences between the industries and organizations. Although organizations differ, they share common challenges and objectives when it comes to improving the effectiveness of their ISMS implementations. The interview process followed the structure to be outlined as follows: Firstly, semi-structured interviews with open-ended questions were used, enabling the respondents to provide detailed responses and provide any additional information where suitable [13]. Then, collected data were, during the interviews, corroborated with the respondents using a shared TEAMS application window. The duration of the interviews was approximately one to one and a half hours. Then, the collected data were structured, displayed, visualized and analyzed [14]. During the interview, respondents were given the opportunity to accept and approve the interview content. Furthermore, they were required to approve any work written using the data collected. The study was based on conducting interviews across eight organizations from different sectors ensuring that the observed “maturity spectrum” and KPI trends are applicable across other organizational contexts. Rigor, in terms of reliability, was upheld using an interview form and the respondents could see what was written in real time via Microsoft Teams. Inter-coder was not used as we wrote into Word, the responses were later analyzed manually. The data analysis followed the process by Huberman et al. [14] for structuring, displaying, and visualizing data. Regarding systematic categorization, the analysis adapted these methods to detect common patterns and categorized results into governance, risk, and incident response. Concerning the maturity scale used for classifying

Table 1. Overview of organizations based on KPI effectiveness, maturity level and observed trends.

Org.	KPI	Key Trends	Comments
1	No defined KPIs. Relying solely on audits.	Very good at writing policies and instruction. Low employee compliance.	Lack of structured KPI framework for ISMS effectiveness measurement. ISMS appears formalized in paper but not effectively integrated into daily operations. Limited awareness of metrics.
2	No defined KPIs. Partial implementation of ISO 27001/27002 [15]	Formally aligned with ISO standard. Decision-making appears ad hoc rather than policy-driven. ISMS progress likely depends on a few individuals rather than institutional structures.	Correlate the risk management process. Increase management engagement. Capacity building.
3	Number of security incidents. Inventory and classification of information assets. Compliance with security policy (ISO 27001).	Automated surveillance and policy compliance monitoring. Practical, internally adapted approach (internal).	Gap between theoretical ISMS frameworks and practical implementation. Lack of structured set of KPIs to assess effectiveness.
4	Incident response, preventive controls, phishing simulation, compliance, system availability. Not fully automated.	Partial implementation of KPIs. Employee awareness: phishing simulations. ISMS still being developed across multiple entities.	Clearly defined and relevant KPIs that align with ISMS best practices. Awareness of continuous improvement through compliance and audit mechanisms.
5	Microsoft Defender Secure Score. Incident detection and response. Security awareness: phishing click rates, training completion. Risk reduction over time. Aligns with ISO 27001 principles.	Heavy reliance on Microsoft Defender Secure Score. Strong security culture focus.	Mature and well structures ISMS measurement approaches.
6	Measures system up-time and availability. Measures the speed and efficiency of incident detection ad response. Tracks the number of reported security incidents. Measures customer trust and satisfaction with security and service reliability. Assesses ISMS maturity and compliance against ISO 27001 standards. Tracks identified risks, mitigations, and timeliness of mitigation actions.	Achieving 85 percent ISO27001 maturity. Continuous improvement approach. Automation and strategic reporting. Strong alignment and readiness for regulatory changes like NIS2.	Future focus could include trend analysis over time. Balanced measurement approach (technical and organizational).
7	Internal Control Framework Coverage Tracks how many information assets (IS) have been properly classified according to policy. Measures implementation and adherence to the organization’s library of security controls (ISO 27002). Evaluates conformity with ISO 27001, 27002, and internal guidelines. Uses a GRC platform (ifacts) to monitor compliance status and control effectiveness across entities. Each business entity defines additional KPIs based on its unique operational and risk context.	A cultural transformation driven by the rising prominence of cyber threats. The organization’s progress is metrics-dependent, decision-making and enhancement are guided by measurable outcomes. Structured governance.	Mature and dynamic ISMS. This organization’s ISMS is framework structured, adaptable to different adaptable, and culturally evolving continuously. Proactive approach to information security.
8	Scoring model based on the NIST cybersecurity framework. Platform Integration: measures how ISMS connects with other organizational systems. Tracks training, participation and completion. Risk mitigation activities. Issue management: assess how well raised issues are addresses based on risk severity and prioritization.	NIST Alignment and Practicality. Leadership involvement indicating a top-down culture of responsibility and engagement, supporting long-term maturity. Connecting the ISMS to other platforms reflects a trend toward integrated data-driven security management, improving visibility and efficiency.	Expand the risk mitigation tracking to ensure a closed-loop process connecting identified risks with corrective actions, and measurable impact reduction. ISMS is technically robust and strategically aligned, though currently IT-heavy.

Note: Gray-scale gradient indicates maturity levels, with light gray denoting low maturity and dark gray denoting high maturity.

’s content, it followed a basic three-tier scale (low, medium, high) based on ISO alignment, the presence, specificity, and automation of KPIs. This offers a reproducible metric for how organizations were ranked in Table 1. Finally, the results were evaluated and approved by the respondents. This was completed via email exchange.

3. Literature Review

Few studies on the evaluation of the effectiveness of ISMS provide specific metrics or KPIs to measure effectiveness, they mostly emphasize frameworks or qualitative improvements rather than providing a comprehensive list of metrics. A study conducted by Farn et al. outlines a wide range of metrics such as risk-exposure level, asset-protection coverage, threat-mitigation rate, and vulnerability-reduction rate, and thus addresses both risk management and effectiveness within an ISMS framework. Further, another study by Safonova et al. [16] introduces a way to measure how well an organization’s ISMS works by adding an effectiveness control to the usual process-based approach which is already recommended in ISO/IEC 27001:2013 [15] and ISO/IEC 27005:2018 [17].

In order to provide a thorough assessment of large-scale IT systems, Danniil’s study [18] comprises ISMS metrics covering incident-response, governance and risk management. These include mean time to detect (MTTD) and mean time to respond (MTTR) which are the average time an organization takes to detect and respond to an incident, policy compliance rates, and audit non-conformity rates. These metrics demonstrate how to assess ISMS adherence to standards, and risk-reduction percentages. The study also mentions how employee awareness and participation in security procedures may be measured by security metrics such as training completion or phishing test success rates. In the same context, Wahydin et al. [19] highlight GRC in their paper by integrating ISMS metrics through the IT Balanced Scorecard and Critical Success Factors. In another article by Negi et al. [20], they describe a framework for integrating Information Technology (IT) and Operational Technology (OT) to strengthen cyber resiliency. In order to align security with operational goals and standards like ISO/IEC 27001, it emphasizes governance through clear policies, leadership, and cross-departmental coordination. The work highlights the importance of incident-response plans and metrics that measure the average time it takes to detect an incident and to respond and resolve it (MTTD and MTTR), along with ongoing audits, to evaluate and strengthen ISMS effectiveness against evolving threats. Wen et al. [21] states that in order to comply with ISO 27001 by having a shared governance, they incorporated ISMS measures into a blockchain-based security framework, mentioning how governance can be measured by indicators like policy enforcement and smart contract compliance. The mentioned framework adds real-time monitoring to enable incident-response measures like MTTD and MTTR. While scalability and security are being assessed by ISO/IEC 27004:2016 [22] aligned metrics, such as transaction throughput, system up-time, and access-control effectiveness. Another study conducted by Totty et al. [23], highlights policy compliance rates and alignment with standards such as ISO 27001 as governance measurements and it also covers other measures like vulnerability exposure levels and risk mitigation efficacy when it comes to examining risk management. As for the organization’s ability to be cyber-resilient, they emphasize incident-response metrics (MTTD and MTTR) and their implementation in accordance with ISO 27004:2016 requirements. These latter metrics are pointed out in the study by Bin et al. [24], where they evaluate the Mobile Threat Defense cybersecurity approach by using an attack surface model to comply with both ISO 27001 and ISO/IEC 27004:2016. The same study highlights other measurements such as attack-success probability, attack cost, system resilience and reconfiguration cost. Similarly, in the context of measurements for ISMS effectiveness, a research process conducted by Fauzi et al. [25] has integrated the use of the COBIT 2019 framework, evaluating IT workforce performance by analyzing current practices and suggesting improvements with an emphasis on skills and better alignment with governance goals, while ensuring compliance with ISO 27001 and 27005 by applying some metrics like MTTR, policy compliance and training efficacy. The same study highlights further measurements to align with ISO 27004:2016, such as IT staff-retention rates, training ROI and competency score.

Further existing studies reveal diverse approaches to evaluate ISMS effectiveness, with varying focus on quantitative and qualitative metrics, but not as a comprehensive method. Boehmer et al. [26] emphasize the use of quantitative KPIs to measure the effectiveness of different policies and control sets. They also highlight the importance of ISMS alignment with company objectives, through ISO 27001 compliance, control effectiveness and risk management. On the other hand, Hsu et al. [27] analyze broad financial figures such as return on assets and stock-market performance, revealing that ISO27001 has little to no impact on those financial figures. The study suggests that firms adopt ISO 27001 mainly for regulatory compliance. Furthermore, it lacks detailed ISMS metrics, with governance implied through management commitment and limited attention to risk management; however, it lacks specific measures.

In another study conducted by Szczepaniuk et al. [1], they evaluate how public organizations manage their ISMS, which revealed their continuous monitoring of incident rates; adherence to General Data Protection Regulation (GDPR), Network and Information Systems regulations (NIS), and the confidentiality, integrity, and availability (CIA) triad; and dependency on audit results and regulatory conformity to highlight governance gaps. Additionally, other metrics to measure ISMS effectiveness are revealed by Grenefalk et al. [5], who draw attention to top-management support, leadership engagement, a security culture, and ISO27001 certification and audit outcomes. In another study by Humpert-Vrielink et al. [28], they highlight that many organizations, across all levels, lack the capability to evaluate ISMS effectiveness, and based on an organization’s influencing factors, they suggested a model for developing meaningful KPIs addressing the management system and technical, risk and human aspects of ISMS.

Moreover, Haufe et al. [7] outline key ISMS tasks, internal audits, and risk reviews but do not define specific KPIs, instead they focus on establishing a framework for ISMS core processes and their criteria. For detailed effectiveness metrics, they recommend referring to ISO27004:2016. In a later study, Haufe et al. [29] propose a resource management process for ISMS to enhance transparency and cost accountability, using budget planning and cost allocation as compliance and risk-linked metrics under ISO 27001.

Survey-based metrics were proposed by Baker et al. [30] to reflect governance and compliance by implementing control and policy enforcement, emphasizing quality over quantity.

The studies above revealed that ISMS is not evaluated as a whole system but rather as separate entities where each entity is evaluated individually. This implies, to some extent, that business units lack internal communication and alignment with an existing framework which will eventually have an impact on the whole system effectiveness. On the other hand, research discloses the use of different metrics, implying the non-unification of metrics and the absence of a holistic approach for evaluating ISMS effectiveness.

4. Findings

To identify the metrics used to evaluate ISMS effectiveness in today’s operations, interviews were conducted with eight organizations from different sectors and of varying sizes (Regulatory Compliance, Pension, Education, Energy, Cloud Computing, Municipality, Mining). To analyze the results, we detected common patterns and categorized responses into three foundational pillars: governance, risk, and incident response. Additionally, we outlined key differences and provided insights into approaches to ISMS effectiveness measurement.

The findings illustrated a set of approaches, ranging from no metrics at all to sophisticated automated evaluation systems. As each company was analyzed individually on important KPIs with corresponding maturity levels and notable trends, an overview is presented in Table 1 below highlighting such data with identified trends and further comments on observations of interest. The table uses a gray-scale gradient to illustrate maturity levels, with light gray denoting low maturity and dark gray denoting high maturity.

The scoring rubric used for Table 1 is based on a three-tier maturity scale (low, medium, high) that evaluates an organization’s ISMS effectiveness using four main criteria: ISO alignment (cf. 27001, 27004), presence, specificity, and automation of KPIs. To preserve the confidentiality and anonymity requested by the respondents, the table was designed in a way that the numbering of organizations (1–8) does not correspond to the alphabetical listing of companies provided in the Section 2.

The table above highlights the varied organizational maturity levels, with incident management as a common key focus, more opportunities for less mature organizations to adopt structured and automated KPIs aligned with standards like ISO 27001. For detailed evaluation of those KPIs, an analysis framework was developed (Figure 1) that categorizes the indicators, assesses their maturity, and probes trends and gaps for further recommendations. By synthesizing multiple insights from diverse organizations, we can better identify the key ingredients needed to develop a well-rounded framework for evaluating ISMS effectiveness.

The analysis framework, as illustrated in Figure 1, pinpoints and encompasses key elements in evaluating the effectiveness of ISMS and each of those elements relates to the different deployment stages of ISMS. The summation of those key elements reflects the comprehensive approach which includes setting metrics, ensuring employee compliance, automating processes, strategic reporting, aligning with ISO standards along with the correlation of governance, risk management and incident response. Integrating agentic AI with this approach would provide continuous monitoring, interpretation and improvement of the effectiveness of ISMS.

The framework was designed by synthesizing insights from eight participating organizations (ranging from energy and mining to public authorities) to identify the “key ingredients” for a structured analysis framework for evaluating ISMS effectiveness. Each component corresponds to specific patterns found in the empirical data:

• Defining Metrics: Derived from the observation that low-maturity organizations (like Organizations 1 and 2) lacked defined KPIs entirely, whereas high-maturity ones (like Organization 8) used robust scoring models.
• Automation and Strategic Reporting: Based on the finding that organizations like Organizations 6 and 7 achieved higher effectiveness through automation and strategic alignment with leadership.
• Alignment with ISO Standards: Derived from the fact that several organizations (like Organizations 3 and 6) used ISO 27001/27002 as their primary measurement baseline.
• Connecting GRC with Incident Response: Directly addresses the “guidance gap” where separates entities (governance vs. risk) often lack internal communication, a trend identified in Organizations 3 and 4.
• Employee Compliance and User Awareness: Synthesized from the widespread use of phishing simulations and training completion rates as proactive maturity indicators.

4.1. Categorization of KPIs

The responses reveal a variety of KPIs and approaches to evaluating ISMS effectiveness. Below is a breakdown of the KPIs mentioned, categorized by governance/compliance/policy adherence, risk management and incident-response management:

4.1.1. Governance, Compliance and Policy Adherence

In this KPI category, we outline how and if an organization’s (listed above) ISMS efficiently supports their decision-making and oversight, as well as how it ensures adherence to internal policies, standards, and regulatory requirements.

• Organization 1: No KPIs, but auditing is mentioned; struggles with policy implementation and employee compliance.
• Organization 3: The KPIs are inventory and classification compared to policy, and automated monitoring of compliance.
• Organization 4: The KPIs are compliance tracking measures (on-going alignment to standards and regulations, and audit-tracking of non-conforming events).
• Organization 7: The KPIs are adherence to ISO 27002 guidelines, internal control framework, and specific measurement system based on entity requirements.
• Organization 8: The KPIs are NIST score, CIS controls, GRC tools, BICS, documentation, IT governance alignment, and strategic business alignment.

The above findings reveal that organizations are employing different KPIs to track their governance, compliance and adherence, while other organizations are not showing any compliance-monitoring measurements. This calls for setting a strong foundation prior to any ISMS implementation, allowing continuous monitoring and alignment with regulatory frameworks.

4.1.2. Risk Management

As for risk management, this section examines how and whether those organizations are measuring the effectiveness of their ISMS as an approach reducing risk exposure and if controls are mitigating the most significant threats.

• Organization 3: The KPI is partial risk analysis. However, the risk management processes are not yet correlated.
• Organization 5: The KPIs are related to risk reduction.
• Organization 6: The KPIs comprise risk registry and timeliness of risk-mitigating actions.
• Organization 7: The KPIs pertain to risk-based measurements tied to entity requirements.
• Organization 8: The KPIs contain risk scoring (impact/likelihood), mitigation activities, prioritization, risk ownership and response guidelines.

Risk management is of concern to some organizations, and while some have illustrated their maturity by implementing metrics, others have not proved how they conduct risk management. This implies that risk management is still largely treated as an ad hoc approach that does not support proactive action-facing cyber incidents.

4.1.3. Incident-Response Management

When it comes to detecting and responding to threats, this section looks further into what KPIs are in place to measure ISMS effectiveness.

• Organization 3: The KPIs are number of incidents and constant surveillance of the environment.
• Organization 4: The KPIs include incident management-related measures such as number of incidents and response time.
• Organization 5: The KPIs contain a security scanner reporting on incidents and risk score.
• Organization 6: The KPIs comprise incident-response time and incident-health ticketing (number of incidents).
• Organization 7: The KPIs are based on their GRC platform reports, and further they are based on risks specific to each business entity.
• Organization 8: The KPIs include incident plans, business continuity, improvements to post-incident management, awareness programs, and historical incidents.

As for incident-response readiness, common metrics are used across a few organizations illustrating the use of a unified management approach for incident response. This approach could be adopted by the organizations that have not implemented a clear management strategy for their incident response.

4.2. Maturity Assessment

The maturity levels were derived using a basic evaluative scale that analyzed four primary factors for each organization: ISO alignment, presence, specificity, and automation of KPIs:

1.

Low Maturity (only few components of the analysis framework):

• When an organization lacks defined KPIs and relies instead on vague, unstructured processes (e.g., Organizations 1, 2 and 3).
• These organizations often rely on manual audits and “paper-based” compliance where policies are written but not adequately integrated into daily operations.

2.

Medium Maturity (some or around half of the components of the analysis framework):

• KPIs are clearly structured and implemented, but the automation and scope remain limited (e.g., Organizations 4 and 5).
• These organizations demonstrate progress in specific areas—like using Defender Secure Scores or phishing simulations—but lack a comprehensive approach to strategic reporting.

3.

High Maturity (majority or all components of the analysis framework):

• When an organization defines and establishes specific, comprehensive KPIs with high levels of automation and consistent reporting to management (e.g., Organizations 6, 7 and 8).
• These organizations view ISMS as a dynamic “backbone” and they employ advanced scoring models and GRC platforms to provide an effective implementation of ISMS.

This assessment underpins the lack of metrics evaluating ISMS effectiveness, with only a few organizations showing proof of their readiness to cyberattacks and ensuring a high-performance ISMS is protecting their information flow process.

From the findings above, Figure 2 below demonstrates how organizations perform based on metrics for evaluating their ISMS effectiveness. Each block corresponds directly to one of the five core components defined in the analysis framework (Figure 1), and the total height of the bar reflects the organization’s overall maturity and the robustness of its ISMS. Thus, the highest bars signify a strong foundation characterized by comprehensive, automated KPIs, while the lowest bars indicate a weak, reactive foundation relying on manual processes and audits. The data suggests that maturity is directly proportional to the quality and automation of KPIs rather than the simple volume of data collected.

Based on the interview findings regarding KPIs used by the eight organizations to measure the effectiveness of their ISMS, and based on the insights from the categorization and maturity assessment, the next section of this study addresses the relevance of these KPIs, highlighting significant gaps and corresponding recommendations to enhance ISMS effectiveness across the evaluated organizations. The evaluation of the results with the respondents indicated that the analysis framework and Table 1 were adequate and reflected upon reality.

5. Analysis of Findings

Insights collected from the eight interviews indicate that a variety of KPIs and metrics are employed to evaluate the effectiveness of ISMS, serving as essential tools for monitoring security effectiveness, though their implementation shows considerable variations across organizations. Organizations range from those lacking clear defined KPIs to those with advanced multi-layered systems. This indicates that KPIs are not universally standardized but rather customized to an organization’s needs, size, and sector. Among the KPIs employed, incident management stands out as a core focus shared by a number of organizations and measured through metrics such as response time, number of incidents, and risk mitigation. Other common indicators include compliance with policies and standards such as ISO 27001/27002, employee awareness, and operational reliability. Furthermore, technology-driven KPIs such as the Microsoft Defender Secure Score also highlight a growing shift toward automated, data-focused evaluations. There are further preventive measures as part of the KPIs. Examples of such are phishing simulations, incident detection and response, and business continuity, which also highlights detective/preventive/proactive measures as part of the KPIs.

The findings suggest that automation is the primary method of managing ISMS and evaluating its effectiveness; however, some organizations continue to struggle with the foundational blocks of automated KPIs resulting in ad hoc, audit-focused strategies that may overlook comprehensive security insights.

As shown in Figure 1, the analysis framework encompasses KPIs and metrics for assessing ISMS maturity and measuring organizational sophistication in security management. Maturity is evaluated on a scale from low to high, taking into consideration automation, integration and comprehensiveness of metrics. Moreover, it is revealed that ISMS maturity is directly related to the quality, automation, and existence of KPIs. Higher maturity involves not simply more metrics but how they are strategically connected to standards, policies and organizational objectives. This assessment pinpoints that maturity is continuously developing and can be enhanced by addressing gaps such as limited automation or unaligned risks.

Some patterns and gaps were identified through interviews such as the universal emphasis on live incident management and the absence of measurable KPIs in firms with low maturity. Organizations with well-defined and implemented KPIs have clearly proven their ability to evaluate their ISMS effectiveness in comparison to their peers. For instance, low-maturity firms such as Organizations 1, 2, and 3 can gain insights from higher-maturity-level firms, such as 6, 7, and 8, on adopting ISO-aligned automated KPIs and more standardized methodologies to boost their security resilience. Additionally, the findings from Table 1 and Figure 2, indicate a shift to increased user awareness and compliance with standards reflecting the adaptive nature of the ISMS, dealing with current threats while proactively maintaining the anticipation shifts in the cyber landscape, including AI and other holistic risk mitigation approaches. Parts of the KPIs proposed could be collected using automation and agentic AI measurements. This will not exclude human participation as they need to verify that the automatic or agentic output is accurate, relevant and reliable. Some of the KPIs may have multiple components that needs to be pre-processed, normalized, verified and validated prior to being used in the calculation/summarization. Examples of those KPIs are listed in Table 1; for instance, incident detection and response, implementation and adherence to the organization’s library of security controls, conformity with ISO 27001 and 27002, and internal guidelines.

The research findings also reveal some gaps, including the overlooked user awareness and the narrowed focus approach. Key trends in ISMS effectiveness show increasing focus on ISO standard alignment and the use of phishing simulations as awareness tools, suggesting that indicators are becoming more proactive, rather than just reactive. In contemporary business operations, where cyber threats are unpredictable, these insights highlight the fact that evaluation of the effectiveness of ISMS is no longer optional for compliance, risk management, and business continuity, it is now an imperative. The findings also disclose some difficulties encountered by some organizations such as policy implementation (Organization 2), as well as opportunities for customized metrics (Organization 7) assisting leaders and decision-makers and prioritizing the usage of tools like GRC systems to enhance maturity and cybersecurity resilience. Furthermore, the wide spectrum of maturity levels within and across various sectors (for instance, Municipality and Energy) indicates that universal approaches are inadequate, indicating the need for customized and scalable KPIs. This is of great relevancy for policy-makers, regulators, educators, and researchers to develop frameworks that encourage proactive ISMS practices that are vital for the continuous enhancement of the safety of cyberspace and enabling reasoned policy decisions in light of the sharp increase in data breach incidents.

This study aims to provide a guideline for information security managers, CISOs, GRC teams, and executive leadership responsible for security oversight, while supporting continuous planning and monitoring ISMS effectiveness. The findings also offer a practical reference for evaluating the maturity of an organization’s ISMS by identifying which KPIs are commonly adopted, highlighting the significant role of automation, and exposing the impact of ISO 27001/27002 standards alignment on the overall effectiveness of ISMS.

The insights can be generalized to some extent, particularly in relation to common themes such as the prevalent use of incident-management metrics and the increasing emphasis on user awareness (e.g., phishing simulations), automation and strategic reporting practices. However, this can be limited by the small sample size and the variability in sector-specific needs, regulatory policies, and resource availability. Therefore, while the findings offer strong directional guidance, they should be adapted to each organizational context.

The findings showcased above can significantly improve organization’s effectiveness, reduce costs, and lessen the amount of repetitive manual workload that needs to be done where automated structured KPIs improve the response to incidents and cut down on manual incident reports, which help improve reporting compliance and security resource expenditure. As an example, automating monitoring and using GRC tools reduces regulatory costs and enhances employee accountability which is achieved through awareness programs while improving decision-making, which is demonstrated by organizations that have achieved automation maturity. In low-maturity organizations, which lack defined KPIs, are dependent on manual audits, and have lesser ISMS effectiveness, eventually face greater challenges.

The lack of defined KPIs is a significant concern for Organizations 2 and 3. Organization 2 is conducting audits as the only means of verification and has a reactive immature ISMS which is policy-heavy and light in implementation. Organization 3 has some level of compliance with ISO 27001 and 27002 and conducts risk analyses, yet the absence of KPIs stunts risk management from being a cohesive process. Organization 4 is more advanced in the automation of incident management, preventive controls, phishing simulations, compliance, system backups, and system up-time. However, slow automation of compliance and preventive controls slows down the process. Organization 5 uses Microsoft Defender Secure Score and the phishing click rate as KPIs, demonstrating a lack of a multi-faceted approach to KPIs as more holistic factors. Organization 2 has documented compliance with policies by employees, and this is an assumption for Organizations 1 and 3 which ignore user awareness metrics. Organization 3 is also missing risk management processes which, combined with incomplete risk analyses, stifles effective risk measurement and risk mitigation. Finally, Organizations 2, 3, 4, and 5 do not engage in structured and regular strategic reporting unlike Organizations 6 and 7, which may hinder strategic alignment and oversight.

6. Discussion and Conclusions

The paper contributes to the literature by addressing the lack of precise metrics for evaluating ISMS effectiveness. While other studies and work focus on qualitative enhancements or frameworks, this study reveals and provides measurable KPIs which can be customized to the specific requirements of an organization, thus improving the relevance of ISMS effectiveness evaluations.

The outcomes of this research offer significant managerial contributions by providing actionable insights and tools that empower managers to improve their decision-making and the strategic alignment of ISMS within their organizations. These insights are particularly beneficial for managers seeking to implement ISMS frameworks while maintaining the balance between efficiency, compliance, and the organization’s objectives.

From a practical perspective, this research supports ISMS implementation and evaluation by offering a combination of recommendations covering both technical and non-technical aspects regarding automation, employee training awareness, formalized risk frameworks, and management dashboards specific to individual organization needs, allowing efficient operationalization of systematic refinements within ISMS.

Based on the findings collected from interviews conducted with each organization, the following future actions are highly advised to improve ISMS effectiveness for a better evaluation of ISMS. In order to address the gaps in Organizations 2 and 3, which do not have established defined KPIs, it is necessary to put into action foundational metrics for incident management (such as number of incidents, average response time), compliance (for instance percentage of resolved audit findings), and risk reduction (including number of risks mitigated over time). Organization 2 should set training and phishing awareness policies, ensuring completion and click rates. While Organization 3 is advised to perform a comprehensive ISO 27001 risk assessment, establish a risk registry and KPIs for timeliness of mitigation. As for Organization 4, they ought to improve their efficiency through automation and utilization of GRC tools such as dashboards to reduce manual errors. Organization 5 needs to broaden their focus beyond technical metrics by including policy compliance and training effectiveness KPIs. On policy adherence, Organizations 1, 2, and 3 can improve employee adherence with awareness campaigns, phishing simulations, and micro-training by implementing KPIs associated with participation and human-error incidents. Additionally, Organization 3 should formalize its risk management plan by integrating KPIs to measure mitigation coverage and response time. Finally, Organizations 2 and 5 should adopt an implementation of strategic KPI reporting, similar to Organization 6, by using dashboards that align ISMS with organizational goals and enhance overall oversight.

The study identified a variety of ISMS maturities, and these findings are contextualized in Northern Europe where legislation frameworks such as GDPR and the NIS2 Directive significantly influence governance structure and KPI selection. The latter implies that the emphasis on automated incident reporting and compliance to specific metrics might be required as a regulatory response than an internal strategic decision. This might be considered as a limitation for the applicability of the analysis framework to organizations operating outside the EU legal jurisdictions. Furthermore, the procedural requirement that respondents consent to their content might have led to social desirability bias protecting the organization’s reputation by sanitizing their data about significant occurrences and system failures protecting company reputation. However, we felt during the interviews that the answers were candid and genuine.

This paper focuses on evaluating the effectiveness of ISMS rather than providing a direct comparative scoring against established maturity models like ISO 27004 or CMMI. The proposed analysis framework uses existing standards as maturity indicators. For example, the study identifies that “High Maturity” organizations are characterized by their alignment with ISO 2700X standards and their ability to automate their reporting. Furthermore, the research proposes its own maturity assessment: low, medium, and high based on the presence, specificity, and automation of KPIs. A core novelty of this framework is the focus on GRC interconnectivity by analyzing the relational dynamics between governance, risk, and incident response, providing a comprehensive, integrated methodology for evaluating the whole ISMS effectiveness.

While standards like ISO/IEC provide well-defined controls, their misinterpretation or partial implementation often fails to align with actual ISMS objectives. The analysis framework is intended to solve this by providing a universal, objective assessment of whether those practices are actually effective in a live environment. The literature unfolded how qualitative descriptions extensively relied on evaluating ISMS effectiveness, revealing a scarcity of integrated frameworks that combine qualitative and quantitative KPIs for comprehensive evaluation and continuous improvement of the effectiveness of ISMS. Humpert-Vrielink et al. [28] highlighted in their study the limitations of traditional security measurement approaches. Consequently, although ISMS has been incorporated in many organizations, they continue to struggle and face challenges regarding their effective implementation and determining the adequate approach to adapt. The analysis framework in this paper addresses these issues and provides a potential solution to reach an effective implementation of an ISMS. Thus, the paper’s results are aligned with Humpert-Vrielink et al. but also provide a solution. Further, aligned with Grenefalk et al. [5], we also want to draw attention to top management support and engagement, resulting in a security culture. This is paramount for the success of any effective ISMS implementation. Not only that, but acknowledging the fact that existing standards, such as ISO/IEC, provide well-defined controls, their misinterpretation or partial implementation does not align with ISMS objectives. In addition, the interviews revealed how practices and business operations are not harmonized with ISMS effectiveness. These challenges call for an urgent need for a comprehensive approach fueled with agentic AI, supporting both qualitative and quantitative KPIs, enabling a universal objective ISMS assessment.