A Methodology for Quantitative Security Evaluation of Operating Systems: Scenario-Based Comparison of Qubes OS and Windows 11

Abstract

Securing users’ endpoint devices is a highly important part of organizations’ overall security posture. The vast majority of cyber attacks either begin with endpoint compromise or use it as an effective method for lateral movement and privilege escalation. In this article we propose a new methodology for quantitatively assessing the overall security provided by operating systems, based on implemented mitigations of MITRE ATT&CK techniques. The proposed approach enables reproducible scenario-based comparisons of operating system security and can support security-oriented decision-making in organizational endpoint protection strategies. Moreover, it allows for quantitative assessment of operating systems under specific cyber attack scenarios, expressed as collections of adversaries’ utilized ATT&CK techniques, facilitating comparison across multiple operating systems under identical scenarios. We apply this methodology to Qubes OS and Windows 11, showcasing measurable differences in how both operating systems mitigate cyber threats.

1. Introduction

Cybersecurity is a broad field, from the security of user workstations and mobile devices (endpoint systems) through network and cloud security [1,2], to the protection of critical infrastructure and industrial control systems [3,4]. Due to the size and diversity of the field, it is difficult to conceptualize and catalog all the possible attacks on information technology (IT) systems and their mitigations. However, projects such as the MITRE ATT&CK framework attempt to create a comprehensive list of tactics, techniques, and procedures (TTPs) used by adversaries during cyber attacks  [5].

When reporting cyber attacks, many organizations make use of the MITRE ATT&CK framework to map each of the steps that an adversary has taken when performing the attack. The framework also provides a list of possible mitigations that can help defend against each step of an analyzed attack in the future. It is difficult, if not impossible, to implement all possible mitigations, but according to the report by Tidal Cyber and the Cyentia Institute [6], the most effective mitigations for the techniques observed and reported throughout the cyber threat landscape would be M1040, M1038, and M1028. These stand for Behavior Prevention on Endpoint, Execution Prevention, and Operating System Configuration, respectively [5]. All three relate partially or fully to the security of endpoint devices specifically. This underscores the still-inadequate state of endpoint security and showcases a need for more effective ways to protect users’ systems against diverse types of attacks.

Despite the widespread adoption of the MITRE ATT&CK framework for attack reporting and mitigation guidance, no established methodology exists for leveraging it to quantitatively compare the security postures of different operating systems. Given that the choice of operating system is one of the most fundamental decisions affecting endpoint security, this represents a gap worth addressing.

The purpose of this article is to propose and design a methodology for evaluating the security of a given operating system, based on the MITRE ATT&CK Enterprise framework. The proposed methodology allows for a quantitative description of an operating system’s resistance to specific tactics and techniques as well as providing a numerical basis for comparisons between different systems. This approach also allows researchers to calculate how well a chosen system would fare against a specific attack scenario, when broken down into steps based on MITRE ATT&CK.

The novelty of this approach lies in using MITRE ATT&CK technique mitigations as a quantitative scoring basis for cross-OS security comparison from the adversary’s perspective. Existing works in this area focus on risk assessment [7,8], organizational decision-making [9], vulnerability counts [10], or usability studies [11], but do not provide a reproducible framework for comparing how different operating systems mitigate documented attacker behavior at the endpoint level.

The article is organized into seven sections. Section 2 presents the current state of the art in the scientific and industry literature related to the security of operating systems and virtual machines. Next, Section 3 presents pertinent details about the security architectures of Qubes OS and Windows 11. Section 4 details the proposed quantitative security testing methodology and outlines two example security scenarios. Afterwards, Section 5 presents the results that were gathered using our methodology for Qubes OS and Windows 11. Section 6 discusses what can be inferred from the results, including a comparison between the effective security levels of the analyzed operating systems. Then, Section 7 presents our conclusions, including the limitations of the proposed methodology, as well as future research directions. Finally, Appendix A provides the complete data for both operating systems in tabular format.

2. Related Works

As far as the industry and scientific literature on the subject at hand is concerned, very few works address the topic of Qubes OS security specifically. We identified several works that tackle related subjects, such as methodologies utilizing MITRE ATT&CK to different ends and analyses of virtualization- and compartmentalization-based approaches to security.

Ahmed et al. [7] provided a comprehensive approach to assessing cybersecurity risks using the MITRE ATT&CK framework. Their method consists of organizational and threat modeling as well as assessing the probable impact of selected tactics and techniques. They utilized critical attack route simulations using attack scenarios from two APT groups, Lazarus and menuPass. Their approach finds certain similarities with our research, but focuses on overall risk assessments and not OS-level mitigations specifically.

Oruc, Amro and Gkioulos [8] proposed a new method of assessing cyber risks using the MITRE ATT&CK Enterprise and ICS frameworks. Their approach assigns each integrated navigation system component on a ship a set of related ATT&CK techniques. They then assign each combination a risk score from four categories, low, medium, high and critical, calculated based on the likelihood, impact and detectability of such a technique when applied to a component. Their resulting four-level categorization can be considered similar to the approach in our article, although the measured properties (risk vs. mitigation) present different approaches to utilizing the ATT&CK framework.

Mohamed, Hefny and Darwish [9] designed a cyber-defense approach that integrates the MITRE ATT&CK framework with multi-criteria decision-making methods. Through determining the relative importance of included techniques, they applied multi-criteria decision-making methods to create decision matrices that can be fine-tuned on a per-organization basis. Their approach allows organizations to utilize ATT&CK to prioritize the implementation of high-priority security controls to maximize gains in their cybersecurity posture.

Lefeuvre et al. [12] described the historical and contemporary landscape of compartmentalization, while providing recommendations and guidelines for approaching and implementing compartmentalized software systems. They demonstrated measurable security benefits stemming from such approaches, through an analysis of 61 systems and 211 research efforts in this field. While demonstrating that compartmentalized software can be a highly effective means of minimizing the impact of exploits, they also noted several obstacles to widespread adoption, such as the reliance on manual methods, maintenance burdens of legacy mechanisms, and non-standardized abstractions.

Issa, Murray, and Ernst described the usability of the user interfaces of converged multi-level secure (MLS) systems [11]. They observed the behavior of 21 participants using the Cross Domain Desktop Compositor (CDDC) and judged, through a series of tests, whether the users would behave securely, maintaining correct isolation between security domains throughout the duration of the experiment. Their findings indicate that such a method of compartmentalization shows significant promise for security-critical tasks, as most (19 out of 21) participants managed to succeed in behaving securely, even under more demanding scenarios. This result is of special importance for laying out the assumptions of this research, as the CDDC’s design bears a resemblance to that of Qubes OS and it is likely that it was inspired by it, considering that its patent application was filed five years after the first alpha release of Qubes OS [13]. This lets us assume that basic security measures will be followed by users using Qubes OS, who will not immediately break its security model through user error.

Sharma and Khilji presented a method for evaluating an element of the security of Qubes OS based on the number of reported and fixed security vulnerabilities in the Fedora project [10]. This represents an alternative approach to estimating the security of an operating system. Although the method we implement differs from this one, it is nonetheless a useful approach that could be used to examine the security of an operating system more holistically.

Singh and Somani proposed a taxonomy of cross-virtual machine attacks and possible defense mechanisms [14]. They separated cross-VM attacks into five categories, each requiring different conditions and presenting unique challenges for VM security. Certain types of the described attacks are already accounted for by Qubes OS’s or Xen’s built-in security mechanisms, but this taxonomy is nonetheless useful regardless of the utilized hypervisor and distribution.

Motiee et al. conducted a study of whether Windows users followed the principle of least privilege in relation to Windows’s User Account Control (UAC) [15]. Their findings demonstrate that, depending on interpretation methods, 69–94% of users did not follow the principle successfully and consented to fake/malicious prompts, demonstrating that Windows’s UAC prompt can be thought of as click-through in most cases.

Tidal Cyber and the Cyentia Institute recently performed an in-depth meta-analysis of the most common attack techniques based on MITRE ATT&CK classification [6]. It is a very helpful source of quantitative data and has served multiple purposes in this research.

It is worth mentioning that current events [16] have triggered renewed interest in digital sovereignty across European nations [17], such as Denmark [18] and Germany [19,20]. Qubes OS, as a free and open-source (FOSS) project [21], could be an effective tool to achieve such goals, while increasing the security of organizations. In particular, this is not just the realm of theory, as organizations such as The Guardian [22] and Freedom of the Press Foundation [23] have already deployed Qubes OS for security-critical workloads and there do exist standards compatible with ISO-27001 that include dedicated compliance controls for this operating system [24].

To the best of our knowledge, no peer-reviewed work to date applies MITRE ATT&CK technique mitigations as a quantitative basis for comparing the security postures of different desktop operating systems. The works presented above either focus on risk assessment, organizational decision-making, or vulnerability counting, but do not provide a reproducible framework for cross-OS security comparison at the endpoint level. This is the gap that the present article aims to address.

3. Security Architectures of Operating Systems

Desktop operating systems are continuously increasing in complexity and new security measures are constantly being implemented to combat known and emerging threats. This section presents the main security features of two operating systems selected to validate the proposed methodology in practice: Qubes OS and Windows 11.

3.1. Qubes OS Security Architecture

Qubes OS is a free and open-source (FOSS) desktop operating system designed with security at its core. It relies on virtualization provided by the Xen project in order to create and manage virtual machines (VMs), called qubes [25,26]. By default, Qubes OS provides GNU/Linux-based guest operating systems (Fedora, Debian, and Whonix) for usage in the userspace. The management domain (virtual machine), called dom0, runs Fedora Linux, and users are free to choose any other operating system for the user domains. The project also supports (either officially or through community efforts) the running of other GNU/Linux-based operating systems, with Windows 10/11 integration through Qubes Windows Tools [27], as well as community support for the Mirage unikernel OS [28].

The Qubes OS security architecture expects users to create separate qubes for separate trust levels, such as separating casual Internet browsing from e-mail, password management, or work-related activities. The OS creates a collection of such virtual machines by default during installation, but the user is free to change this configuration as needed. Qubes OS also creates the sys-usb, sys-net and sys-firewall virtual machines by default in order to separate out the USB and networking stacks to protect from various kinds of vulnerabilities and attacks. Figure 1 presents the typical configuration of virtual machines on a Qubes system. (The exception is the GuiVM/AdminVM separation. As of Qubes 4.3, these are by default integrated as one virtual machine—dom0. It is possible to opt in to GuiVM/AdminVM separation after installation.) On a live Qubes OS installation, each virtual machine displays its windows on the unified desktop and is marked with colored window borders, allowing users to easily tell which trust level each window belongs to.

In order to provide additional security and usability and decrease disk usage, Qubes OS makes use of the template system. Templates are clean installations of OS distributions that provide their root file system to other qubes, called AppVMs [29]. Figure 2 illustrates this process. In essence, each AppVM has its own private storage, which is persistent across reboots but shares the same root image, changes to which are not saved when the AppVM is shut down. This allows users to install software and perform other system modifications in a single qube (the TemplateVM for a number of AppVMs), with the modifications propagating to all derived AppVMs after a reboot [29].

This provides numerous advantages, including, but not limited to, the following.

• Security. Using the template system makes it significantly easier to apply system updates to many virtual machines at once. AppVMs also do not have write access to the templates’ original root images, which means that any malware infections occurring in an AppVM should be contained to just that qube.
• Performance. Using a common root file system and initially empty private images allows for very rapid creation of new AppVMs.
• Storage. Having just one root file system per template significantly cuts down on the storage requirements of running a large number of virtual machines.

In addition to TemplateVMs and AppVMs, Qubes OS also provides two additional types of virtual machines. They are: DisposableVMs and StandaloneVMs. The former work similarly to AppVMs, with a root file system based on a TemplateVM, but their private image is also cleared upon shutdown. This makes them ideal for riskier actions, such as opening untrusted files or URLs, as malware should not have the opportunity to achieve any persistence without significant user interaction [31]. StandaloneVMs are the opposite—they are not template-derived (unless cloned from one) and have full persistence for all of their file systems. This makes them useful for certain development tasks and for usage as typical virtual machines [32].

Qubes OS includes additional security measures such as USB and RPC policies, full-disk encryption (FDE) and many others, but providing an exhaustive list of the implemented measures is outside of the scope of this article. When evaluating the security mitigations and scenarios in Section 4 and Section 5, any measures that may affect the results will be mentioned. Security features inherited from Fedora Linux (the default AppVM template in Qubes) will also be considered.

3.2. Windows 11 Security Architecture

Although Windows is a very popular desktop and moderately popular server operating system, Windows 11 specifically introduced several new security features that are worth highlighting. This section presents some of the newer and less commonly known security features introduced by Microsoft for this major release of Windows. Most of the information contained in this section is based on the Windows 11 Security Book [33].

One of the most immediately evident features that differentiates Windows 11 from Windows 10 is its requirement for the presence of a Trusted Platform Module (TPM) 2.0 chip in the hardware of the host device [33] (p. 8). Windows 11 utilizes the TPM device as a hardware root of trust to provide security features such as Trusted Boot, Windows Hello, BitLocker, System Guard, Measured Boot and Microsoft Azure Remote Attestation [33] (pp. 8, 18, 22, 44, 64).

Windows 11 devices come with UEFI Secure Boot enabled as a “baseline security feature of all Windows 11 PCs” [33] (p. 12). This, combined with the TPM-provided root of trust and Measured Boot, is called Trusted Boot and is enabled by default on Windows 11 devices, impeding the capabilities of rootkits and similar types of malware [33] (p. 15).

On hardware with an IOMMU, Windows 11 enables Kernel Direct Memory Access (DMA) Protection by default. This increases the level of protection against drive-by DMA attacks through Thunderbolt/USB4 DMA to kernel memory. It is worth mentioning that, contrary to Microsoft’s claims in the Security Book, this only protects against specifically the DMA attack class, not arbitrary types of “physical attack wherever people work” [33] (p. 11).

Windows 11 also introduces several security features based on virtualization. These are as follows.

• Virtualization-based security (VBS) utilizes hardware virtualization features to create an isolated environment for hosting certain security-critical features and lays the groundwork for other features based on itself.
• Hypervisor-Protected Code Integrity (HVCI) is based on VBS and runs Kernel Mode Code Integrity (KMCI) isolated in the containment of VBS to reduce the likelihood of attacks on kernel-mode code.
• Local Security Authority (LSA) Protection, also called RunAsProcessProtectedLight (RunAsPPL), utilizes VBS to protect the Local Security Authority Subsystem Service (LSASS) from certain types of attacks, such as process injection.
• Credential Guard uses VBS to protect Active Directory (AD) credentials from attacks such as Pass-the-Hash or Pass-the-Ticket [33] (pp. 10, 51).

The first three security features are enabled by default for clean Windows 11 installs, while Credential Guard requires Windows Enterprise or Windows Education licenses [34].

Windows 11 also includes several enabled-by-default features related to exploit mitigations. These are: Data Execution Prevention (DEP), Address Space Layout Randomization (ASLR), and Control Flow Guard (CFG). DEP marks certain memory pages as non-executable, helping prevent exploits from running code from the heap, stack, and memory pools. ASLR randomizes the memory addresses where programs get loaded for images that opted in at compile time. CFG performs additional runtime checks on indirect function call targets and helps protect against memory corruption vulnerabilities [35].

Microsoft has also continuously expanded its Microsoft Defender Antivirus and other default app protection solutions with additional functionality. Worth mentioning are: Tamper Protection, which protects Microsoft Defender components from unauthorized modification [33] (p. 32); Smart App Control (SAC), enabled by default on clean Windows 11 installs, which leverages cloud-based intelligence to predict whether apps are safe to run [33] (pp. 37, 38); and the Vulnerable Driver Blocklist (VDB), which utilizes a seldom-updated blacklist to restrict loading vulnerable drivers to help protect against attacks on the NT kernel [33] (p. 39). Windows 11 also enables App-Bound Encryption (ABE) by default for select apps (such as Microsoft Edge). This aims to make it more difficult for attackers to extract cookies and saved credentials from applications such as web browsers, but bypasses are documented, with forensics companies offering such services as part of their tooling [36]. Microsoft is also developing a new feature called Enhanced Phishing Protection (EPP), but as of yet it only operates in audit mode by default [33] (p. 50).

On the authentication control layer, Microsoft has added Server Message Block (SMB) signing as a requirement, which protects against tampering and replay attacks targeting SMB packets [33] (p. 28). Windows 11 also adds a default account lock-out policy for new devices (10-minute delay after 10 failed login attempts) to protect against brute-force attacks on local and Remote Desktop (RD) accounts [33] (p. 53). As with previous Windows versions, Windows 11 also includes User Account Control (UAC) as an elevation control mechanism, but bypass mechanisms are “regularly discovered and […] used in the wild” [37] as well as being considered effectively click-through for most use-cases [15].

3.3. Architectural Comparison

The two operating systems represent fundamentally different approaches to endpoint security. Qubes OS relies on hardware-enforced isolation through the Xen hypervisor, compartmentalizing user activities into separate virtual machines with minimal trust between domains. Its security model assumes that individual compartments may be compromised and focuses on limiting the blast radius of such events. Windows 11, by contrast, operates within a monolithic kernel environment and employs a layered defense strategy combining detection-based controls (Microsoft Defender and Smart App Control), exploit mitigations (DEP, ASLR, and CFG), and virtualization-based security for select sensitive components such as Credential Guard. Its model primarily aims to prevent compromise within a single shared environment. This architectural contrast makes the pairing particularly informative for evaluating the proposed methodology across systems with distinct security philosophies.

4. Methodology

The study’s methodology consisted of two parts. Firstly, we created a table with MITRE ATT&CK v18 tactics and techniques (TTs), scoring and describing what kinds of mitigations each operating system had in relation to those TTs. Secondly, to showcase the methodology’s applicability to cyber attack scenario-based analyses, two suitable attack campaigns were chosen and analyzed using the created TT mitigation scoring table.

MITRE ATT&CK was chosen as the foundation for this methodology because its mitigation dimension helps to assess OS security from the adversary’s perspective, which ATT&CK is uniquely suited for due to its comprehensive coverage of real-world TTPs, not merely cataloging which techniques apply to a given platform. Importantly, the methodology scores the quality and completeness of each OS’s mitigations, meaning that a technique applicable to both systems but well-mitigated by only one will be reflected in the results. CVE-based approaches were considered but tend to reflect vulnerability discovery rates rather than architectural security posture [10]. Complementary frameworks such as MITRE D3FEND could enrich future iterations and are discussed in Section 7.2.

For the purposes of the study, two operating systems representing contrasting architectural approaches to endpoint security were selected: Qubes OS r4.3.0 [38] and Windows 11 24H2 [39]. This pairing serves as a test of the methodology’s applicability across architecturally distinct systems. Qubes OS was chosen for this study as it is often highly acclaimed by cybersecurity experts [40]. For comparison with a commonly used operating system, Windows 11 was chosen due to its significant adoption by end users, being the second most widely used operating system worldwide after Android [41], dedicated to mobile devices.

4.1. Mitigations

The tactics and techniques analyzed in this study were compiled by filtering the entirety of the MITRE ATT&CK Enterprise Matrix [5] for TTs affecting both the Linux and Windows platforms. Techniques specific only to one platform and not the other (such as Windows Background Intelligent Transfer Service jobs, T1197) were removed. Tactics not specifically related to OS security, that is TA0043: Reconnaissance and TA0042: Resource Development, were removed as out of scope. Additionally, tactics TA0002, TA0008, TA0011 and TA0010 were excluded. The rationale for the exclusion of those tactics is as follows. TA0002: Execution was excluded as a large part of its techniques target non-endpoint environments (ESXi, containers, cloud and serverless) and the remaining techniques largely describe abuse of legitimate OS functionality, mitigation of which is usually the domain of dedicated Endpoint Detection and Response (EDR) solutions and not OSs themselves. Additionally, compartmentalization-based effects on execution containment are partially captured by the analysis of techniques in Privilege Escalation and Defense Evasion. TA0008: Lateral Movement was removed, as most of the techniques therein concern themselves with movement between separate networked systems with significant reliance on OS-specific sub-techniques, which were difficult to reconcile between the selected OSs. TA0011: Command and Control encompasses techniques related to Command and Control (C2) communication. Most of its related techniques, such as Data Obfuscation, Data Encoding and Application Layer Protocol, do not find an application to endpoint OS security specifically. A similar approach was used when excluding TA0010: Exfiltration, as most of its techniques, for example Exfiltration Over Alternative Protocol, are not in scope for OS-level mitigations, being more the domain of Intrusion Detection/Prevention (IDPS) and Data Loss Prevention (DLP) systems.

Finally, techniques affecting external, non-endpoint systems (such as T1190: Exploit Public-Facing Application) were left out as well. As a result, the comparison focuses strictly on endpoint-resident OS-level mitigations and does not capture network-layer defenses. This reduces confounding factors outside the OS’s direct control, at the cost of a narrower scope. For each of the selected techniques, the following tasks were performed.

• The MITRE technique definition was consulted from the official MITRE website, including sub-techniques, procedure examples, mitigations, and detection strategies.
• Official documentation sources, technical reports, and scientific articles were consulted in order to describe the details of how an operating system mitigates the specified technique.
• If such sources proved inadequate, practical attack simulations utilizing specific tactics were performed on live installations of the chosen operating systems.
• Based on the information gathered, a mitigation score and the corresponding description were assigned to the technique for each of the operating systems.

Mitigation details and scores were written and calculated with the following assumptions.

• It was assumed that the user is using Qubes OS correctly (for example separating work, email, and miscellaneous browsing into appropriate AppVMs), but overall they use the default configuration (meaning only sys-usb and sys-firewall were configured as disposable; the user does everything else in AppVMs). (AppVMs are virtual machines that, in short, have an ephemeral root file system taken from a TemplateVM and a persistent /home directory. Disposable VMs have all of their block storage devices configured as ephemeral.) We believe that this assumption is reasonable on the basis of the research findings of Issa et al. [11], discussed in Section 2.
• Mitigation scores were defined so that Low represents next to no mitigation or measures that are trivially easy to bypass. Medium represents reasonable mitigation efforts that may still be bypassable under certain circumstances or complete mitigation via domain isolation but without specific in-domain mechanisms. High is taken to mean complete or near-complete mitigation that would be difficult to circumvent without significant user interaction.
• Critically, it was assumed that threat actors are sophisticated and are able to bypass standard detection-based security measures (such as through the use of polymorphic malware and indirect execution), but do not use zero-day exploits.

To illustrate the scoring process in practice, Appendix B presents worked examples for three representative techniques.

Mitigation Score Calculation

The mitigation scores were then converted into numerical values, with Low assigned a value of 0 points, Medium assigned a value of 1 point, and High assigned a value of 2 points. This means that within one ATT&CK tactic, successful and complete mitigation (with a score of High) of all of the tactic’s techniques would grant an operating system a score of $2\times n$, where n denotes the total number of techniques within the tactic. A complete lack of mitigations would result in a score of 0.

The findings of this research were compiled into a table and used to calculate interesting quantitative indicators in the form of charts and totals tables. Section 5 focuses on that information, as well as the analysis of chosen security scenarios based on synthesized mitigation data. It is worth mentioning that the careful selection of the aforementioned assumptions is important, as they are bound to have a moderate to notable impact on the overall results. This is further elaborated on in Section 7.1.

4.2. Scenarios

In order to be compatible with the presented methodology, security scenarios must be defined in relation to MITRE ATT&CK TTs, as this is the industry-standard method of decomposing a complex attack scenario into its component parts. For the purposes of this work, two comprehensive scenarios were chosen to demonstrate what the application of this methodology to security incident scenarios would entail.

The first selected scenario concerned the 2019–present Conti ransomware campaign, carried out by the Russian Wizard Spider hacking group. The attack has been carried out on a large scale, with the primary targets being North American government agencies and corporations. In this attack’s lifecycle, the ransomware is usually distributed via spear phishing and stolen credentials. After execution, the malware escalates privileges through process injection and performs discovery through the abuse of legitimate system functionality, such as using system APIs to discover processes, remote systems and network shares. It then moves laterally by accessing saved credentials, exploiting Kerberos and SMB, and impairs defenses by stopping security and backup-related services. Finally, the malware exfiltrates sensitive data and encrypts the user’s local copy, demanding a ransom for decryption and threatening targets with public release of their data if they do not comply. Full information about the attack, including an extended set of utilized tactics and techniques, can be found on the CISA Alerts page [42] and Conti’s dedicated page in the MITRE ATT&CK’s Software catalog [43].

Figure 3 presents a chart made in the MITRE ATT&CK Navigator [44] that showcases the TTPs used in the Conti ransomware campaign as reported by MITRE [43], along with utilized sub-techniques. The annotations in parentheses are generated by the Navigator and denote the number of highlighted sub-techniques out of the total available for a given technique. The layer presents the full campaign as documented by MITRE, including tactics outside the scope of this study’s analysis (TA0002: Execution and TA0008: Lateral Movement), in order to provide a complete picture of the attack chain. Only the techniques falling within the tactics defined in Section 4.1 were used for the scoring analysis presented in Section 5.

The second scenario concerns the high-profile SolarWinds compromise, also carried out by a Russian actor, APT29, codenamed Iron Ritual. It resulted in the compromise of the SolarWinds Orion software suite, giving APT29 access to government and corporate organizations across North America, Europe, Asia and the Middle East. The campaign was carried out through a supply chain attack using a compromised software update that allowed attackers to execute arbitrary code in victims’ environments and gain persistence through scheduled tasks. The attackers then used a wide range of techniques, such as stealing account credentials and password spraying, to elevate privileges and move laterally. Similarly to Conti, the malware also disabled various services related to security and monitoring. The ultimate goals of the attacks on each target were varied, and some are not known to date. Most attacks were aimed at gaining a foothold in the infrastructure and exfiltrating confidential data, sometimes using the compromised organizations’ systems to move on to higher-value targets, such as government agencies. Information about this campaign has been taken from MITRE ATT&CK’s comprehensive documentation in its Campaigns catalog [45].

The analysis of per-OS mitigations of the aforementioned scenarios was performed using the mitigations table created according to the process described in Section 4.1. For each of the scenarios analyzed, the table was filtered to include only the tactics and techniques utilized by each of the campaigns, as detailed on their CISA [42] and MITRE [43,45] pages. The per-tactic mitigation percentages were then calculated in a manner analogous to the main table, by tallying up the mitigation scores for each tactic’s techniques and dividing them by the maximum achievable score.

5. Results

This section presents the processed results of determining the mitigation degrees of MITRE ATT&CK tactics, as chosen in Section 4. It also presents example applications of this methodology to attack scenarios selected in Section 4. The complete unprocessed results and raw data are presented in Appendix A.

5.1. Mitigations

Figure 4 presents overall mitigation results, with the y-axis representing the percentage of maximum mitigation achieved per tactic by each operating system. While Qubes OS came out ahead in every tactic, the difference between operating systems’ mitigation percentages varied considerably. In general, the conducted research has found that Qubes OS’s security mitigations outperformed Windows 11’s across all analyzed MITRE ATT&CK tactics.

The tactic where Qubes OS achieved the highest mitigation percentage of 80% was TA0006: Credential Access, while the one where Windows 11 scored the highest with 43% successful mitigation was TA0005: Defense Evasion. The lowest scores were also observed within different domains, with Qubes OS scoring its lowest mitigation percentage of 58% in TA0040: Impact and Windows 11 scoring the lowest mitigation score of just 5% in TA0007: Discovery.

In terms of intra-tactic differences, the analyzed operating systems differed greatly, as can be seen in Figure 5. Qubes OS’s lead over Windows 11 is seen most strongly in the TA0007: Discovery and TA0009: Collection tactics, with 59 and 61 pp, respectively. The difference between the two operating systems turned out to be the lowest for TA0003: Persistence and TA0004: Privilege Escalation, with a 35 pp difference for the former and 27 pp for the latter. These differences are discussed in detail in Section 5.

Table 1. Summary statistics for all tactics.

Metric	Qubes OS	Windows 11
Total	143/222	56/222
Total (%)	64%	25%
Min	58 pp	5 pp
Median	64 pp	32 pp
Max	80 pp	43 pp
$\mu$	65 pp	25 pp
$\sigma$	7 pp	14 pp
CV	10%	52%

presents the totals and summary statistics for all analyzed MITRE ATT&CK tactics. Throughout the space, Qubes OS managed to achieve a mitigation score of 64%, which is more than 2.5 times higher than Windows 11’s score. As mentioned before, Qubes OS’s minimum score is still noticeably higher than Windows 11’s maximum, showcasing no overlap between the two operating systems’ score distributions. A similar median and mean points to Qubes OS’s distribution being more symmetric, while Windows’s distribution is left-skewed due to very weak tactics dragging the average down. Despite having a lower mean ($\mu$), Windows’s standard deviation ($\sigma$) is twice as large as Qubes’s. This, along with a more than five-times-larger coefficient of variation (CV) points to Windows’s protections being more “hit or miss” than consistent, while Qubes’s protections are more consistent.

The denominator of 222 represents the maximum achievable score, calculated as $2\times n$ summed across all analyzed tactics, where n is the number of techniques per tactic and 2 corresponds to the highest mitigation score (High).

5.2. Scenarios

The results of the Conti ransomware campaign attack scenario analysis are shown in Figure 6. In general, a rough approximation of the distribution shape in Figure 4 can be seen, with a local maximum in TA0003: Persistence becoming a global one, and Qubes OS once again scoring visibly higher than Windows 11 in TA0007: Discovery.

In the case of this tactic, Qubes OS achieved a mitigation percentage of 50%, while Windows 11 offered no mitigation whatsoever. In the specific case of the Conti ransomware campaign scenario, mitigation scores for Qubes OS and Windows 11 differ less than in the general case, even reaching identical mitigation percentages for three of the tactics.

A similar situation can be seen in the mitigation results for the Solar Winds campaign presented in Figure 7—Qubes OS managed to achieve measurably better mitigation percentages than Windows 11, while this time scoring similarly in one tactic, TA0005: Defense Evasion. Once again, Qubes OS also managed to achieve satisfactory mitigation percentage scores for tactics in which Windows 11 did not manage to mitigate anything effectively: TA0007: Discovery and TA0009: Collection.

The examples presented illustrate the practical applicability of the proposed security research methodology to scenario-based evaluations. It is worth noting that for scenarios with a good break-down into sub-techniques and procedures it would be worthwhile to extend the mitigations table with even more granular scores to help increase the accuracy and applicability of our proposed methodology.

6. Discussion

The results presented in Section 5 suggest that the security-by-isolation model introduced by Qubes OS has measurable advantages over a more traditional approach presented by Windows 11. In particular, the fact that Qubes OS’s lowest tactic mitigation score (58%) was still found to be higher than Windows 11’s highest (43%) may indicate an advantage caused by architecture-level considerations, not single-tactic outliers. This is confirmed by Qubes OS’s significantly lower coefficient of variation of 10% compared to Windows’s 52%. In attack scenarios that rely on multiple stages and dozens of techniques, such stable coverage can turn out to be critical for preventing a breach.

Qubes OS’s highest score in Credential Access suggests that isolating workflows into separate security domains can significantly reduce attack pathways to credential exposure. Windows’s best performance in Defense Evasion aligns with Microsoft’s recent investment in platform security features such as virtualization-based protections [33].

The large difference between Windows’s and Qubes OS’s mitigation scores in Discovery and Collection point to the inherent difficulty of effectively mitigating attacks based on abusing legitimate system functionality. An application running locally on a Windows system faces few to no obstacles to calling system APIs that allow it to discover system information—a majority of the techniques from TA0007: Discovery do not generate permission prompts, nor do most of them require elevated privileges. An isolation-based approach may be one of the more effective ways to mitigate this, as a stream of permission pop-ups from software asking for access to system APIs might quickly cause such dialogs to lose relevance, for example with User Account Control dialog boxes on Windows [15]. It is worth mentioning that certain elements of Windows’s Discovery and Collection mitigation scoring would likely change if a deployed EDR solution’s behavioral anomaly detection engine were to be taken into account. Such a solution may or may not flag and block assumed usage of certain adversarial techniques, such as T1010: Application Window Discovery or T1057: Process Discovery. In such a situation, the mitigation scoring table and graphs would exhibit visible differences. Their current contents are based on the assumptions previously detailed in Section 4.1.

It is worth noting that the results presented above treat all analyzed techniques equally. As discussed in Section 7.2, integrating technique weighting based on real-world prevalence data or organization-specific threat models could shift the relative standings, particularly for tactics where one operating system dominates a small number of high-impact techniques.

There is one area in which Windows 11’s protections proved consistently more mature than ones implemented by default on Qubes OS. Windows 11 leads in techniques such as T1542: Pre-OS Boot, where Secure Boot being enabled by default provides protections that are absent on the Qubes OS side. It is of course possible for users to enroll their own Secure Boot keys for Qubes and even implement advanced features like Measured Boot with Heads [46], but this is not the default configuration as of the time of writing. For even greater security, certain Qubes-certified hardware offers Dasharo TrustRoot (Intel Boot Guard) [47], but this configuration is inherently dependent on availability of both firmware and hardware support.

All of the above showcases that our proposed methodology is applicable across operating systems with notably different architectures and could be extended to an even broader variety of such systems. Through a well-known scaffolding in the MITRE ATT&CK framework, the methodology also lends itself to independent verification of results and community contribution models—perhaps in the form of a continuously updated database covering a range of commonly used operating systems that would allow for the simultaneous reduction in duplication of effort and discussion among researchers and security professionals.

7. Conclusions

Throughout this article, a new methodology for quantitatively evaluating the security of operating systems has been put forward, together with an example of how to apply such a methodology to scenario-based analyses. The methodology has been applied to compare the security mitigation levels of Qubes OS and Windows 11. Our findings indicate that Qubes OS’s architectural security-through-isolation approach provides measurable benefits to the security of endpoint devices. Qubes OS achieves higher overall mitigation percentages in all ATT&CK tactics, while conceding ground to Windows 11 in a select few techniques, mostly related to Secure Boot. To the best of our knowledge, this represents the first reproducible framework for comparing OS security postures at the endpoint level using ATT&CK technique mitigations from the adversary’s perspective.

7.1. Limitations

It is worth reiterating that the proposed methodology is quite sensitive to initial assumptions. For example, performing this analysis with an assumed unsophisticated adversary would likely change the results, as:

(1)

Operating systems that rely on detective security measures to a significant degree, such as Windows 11, would be able to successfully block many more classes of attacks.

(2)

Unsophisticated adversaries are less likely to write malware targeting GNU/Linux-based desktop operating systems.

Additionally, while the three-point scale (Low, Medium and High) used in this work lends itself to easier understanding and helps perform analyses for larger numbers of ATT&CK techniques, expanding it to, for example, a five-point scale could give more fidelity. Such an approach could also be beneficial in the context of boundary conditions—a researcher’s decision to put an “on the edge” component into the Medium or High class can have a notable statistical impact that would be lowered by increasing the number of points on the overall scale. Additionally, involving multiple independent raters in the scoring process and performing inter-rater reliability analysis could help quantify the degree of subjectivity inherent in the current approach.

It is also worth noting that while practical attack simulations were performed on live installations where documentation proved inadequate, a significant majority of mitigation scores were assigned based on official documentation, technical reports, and scientific articles. Systematic penetration testing or controlled malware trials across all analyzed techniques would further strengthen the validity of the assigned scores, although the scale of such an undertaking would be considerable given the number of techniques evaluated.

Furthermore, the analysis presented in this article is based on default, clean-install configurations for both operating systems. Windows 11’s mitigation scores would likely improve under enterprise-grade hardening, such as enabling Credential Guard, Attack Surface Reduction (ASR) rules, or stricter Microsoft Defender policies. Similarly, Qubes OS results could shift with more aggressive compartmentalization strategies, such as the exclusive use of DisposableVMs, or conversely decrease with less disciplined usage patterns. Quantifying this configuration sensitivity is an avenue for future research, as noted in Section 7.2.

Finally, both the MITRE ATT&CK framework and the operating systems themselves evolve constantly and rapidly. This means that future research using this methodology may encounter certain difficulties comparing its results with the findings of its predecessors if a long period of time has elapsed. This, unfortunately, seems to be unavoidable in such a fast-paced industry as IT, although we believe that grounding the methodology in a well-established industry standard helps to future-proof it somewhat.

7.2. Future Directions

Future research can easily expand on the body of knowledge by utilizing our proposed methodology in at least three ways. First, additional operating systems can be introduced into the comparison in order to see which ones perform the best under identical assumptions. Second, the assumptions can be changed, offering insight into how the analyzed operating systems deal with differing levels of adversary sophistication or usage patterns. Third, comparisons could be performed between different configurations of the same operating system, for example to provide an indication of how an OS’s security has changed over the course of many versions or to investigate the effectiveness of certain hardening methods.

Additionally, integrating complementary frameworks such as MITRE D3FEND into the methodology could provide a more structured way of mapping the defensive techniques employed by each operating system to the ATT&CK techniques they counter.

Furthermore, integrating technique weighting based on real-world prevalence data, such as that provided by Tidal Cyber and the Cyentia Institute [6], or multi-criteria decision-making approaches, such as those proposed by Mohamed et al. [9] and Ahmed et al. [7], could allow the methodology to reflect the relative importance of individual techniques rather than treating them equally. Such weighting could also be fine-tuned on a per-organization basis to better reflect specific threat models and operational contexts.

Lastly, the analysis presented in this article deals with MITRE ATT&CK tactics and techniques. It would be possible to conduct a more in-depth analysis that also takes into account specific sub-techniques and procedures. Although the authors believe that such an approach would be useful, the ratio of the amount of effort required compared to the outcome would quite possibly be higher than in the case of analyzing just the tactics and techniques. Nevertheless, this remains a worthwhile avenue for future research.