An Empirical Assessment of Endpoint Security Systems Against Advanced Persistent Threats Attack Vectors

Advanced persistent threats pose a significant challenge for blue teams as they apply various attacks over prolonged periods, impeding event correlation and their detection. In this work, we leverage various diverse attack scenarios to assess the efficacy of EDRs and other endpoint security solutions against detecting and preventing APTs. Our results indicate that there is still a lot of room for improvement as state of the art endpoint security systems fail to prevent and log the bulk of the attacks that are reported in this work. Additionally, we discuss methods to tamper with the telemetry providers of EDRs, allowing an adversary to perform a more stealth attack.


Introduction
Cyber attacks are constantly evolving in both sophistication and scale, reaching such an extent that the World Economic Forum considers it the second most threatening risk for global commerce over the next decade [9]. The underground economy that has been created has become so huge to the point of being comparable to the size of national economies. Contrary to most cyberattacks which have a 'hit-and-run' modus operandi, we have advanced persistent threats, most widely known through the abbreviation APT. In most cyber attacks, the threat actor would try to exploit a single exploit or mechanism to compromise as many hosts as possible and try to immediately monetise the abuse of the stored information and resources as soon as possible. However, in APT attacks, the threat actor opts to keep a low profile, exploiting more complex intrusion methods through various attack vectors and prolong the control of the compromised hosts. Indeed, this control may span several years, as numerous such incidents have shown.
Due to their nature and impact, these attacks have received a lot of research focus as the heterogeneity of the attack vectors introduces many issues for traditional security mechanisms. For instance, due to their stealth character, APTs bypass antiviruses and therefore, more advanced methods are needed to timely detect them. The goal of an Endpoint Protection Platform (EPP) is prevent and mitigate endpoint security threats such as malware. Going a step further, Endpoint Detection and Response (EDR) systems provide a more holistic approach to the security of an organisation as beyond signatures, EDRs correlate information and events across multiple hosts of an organisation. Therefore, individual events from endpoints that could fall below the radar are collected, processed, and correlated, providing blue teams with a deep insight into the threats that an organisation's perimeter is exposed to.
Despite the research efforts and the advanced security mechanisms deployed through EPPs and EDRs, recent events illustrate that we are far from being considered safe from such attacks. Since APT attacks are not that often and not all details can be publicly shared, we argue that a sanity check to assess the preparedness of such security mechanisms against such attacks is deemed necessary. Therefore, we decided to conduct an APT group simulation to test the enterprise defences' capabilities and especially EDRs, covering also some EPPs. To this end, we opted to simulate an APT attack in a controlled environment using a set of scripted attacks which match the typical modus operandi of these attacks. Thus, we try to infiltrate an organisation using spear-phishing and malware delivery techniques and then examine the IOCs and responses produced by the EDRs. We have created four such use case scenarios which are rather indicative and diverse enough to illustrate the weak points of several perimeter security mechanisms, focusing more on EDRs.
Based on the above, the contribution of our work is dual. First, we illustrate that despite the advances in static and dynamic analysis, as well as multiple log collection mechanisms that are applied by state of the art EDRs, there are multiple ways that a threat actor may launch a successful attack without raising suspicions. As it will be discussed, while some of the EDRs may log fragments of the attacks, this does not imply that these logs will trigger an alert. Moreover, even if an alert is triggered, one has to consider it from the security operations center (SOC) perspective. Practically, a SOC receives multiple alerts and each one with different severity. These alerts are prioritised and investigated according to this severity. Therefore, low severity alerts may slip below the radar and not be investigated, especially once the amount of alerts in a SOC is high [18]. Furthermore, we discuss how telemetry providers of EDRs can be tampered with, allowing an adversary to hide her attack and trails. To the best of our knowledge, there is no empirical assessment of the efficacy of real-world EDRs and EPPs in scientific literature, nor conducted in a systematic way to highlight their underlying issues in a unified way. Beyond scientific literature, We consider that the closest work is MITRE Engenuity 1 ; however, our work provides the technical details for each step, from the attacker's perspective. Moreover, we differ from the typical APT capabilities that are reported for each known group using and modifying off the shelf tools. Therefore, this work is the first one conducting such an assessment. By no means should this work serve as a guidance on security investment on any specific EDR solution. As it will be discussed later on, the outcomes of this work try to point out specific representative attack vectors and cannot grasp the overall picture of all possible attacks that EDRs can mitigate. Indeed, customisation of EDRs rules may significantly change their efficacy, nevertheless, the latter depends on the experience of the blue teams handling these systems.
The rest of this work is organised as follows. In the following section, we provide an overview of the related work regarding EDRs and APT attacks. Then, we present our experimental setup and detail the technical aspects of our four attack vectors. In Section 4, we evaluate eleven state of the art EDRs and assess their efficacy in detecting and reporting our four attacks. Next, in Section 5 we present tampering attacks on telemetry providers of EDRs and their impact. Finally, the article concludes providing summarising our contributions and discussing ideas for future work.

.1 Endpoint detection and response systems
The term endpoint detection and response (EDR), also known as endpoint threat detection and response (ETDR), is coined by A. Chuvakin [7] back in 2013. As the name implies, this is an endpoint security mechanism that does not cover the networking. EDRs collect data from endpoints and send them for storage and processing in a centralised database. There, the collected events, binaries etc., will be correlated in real-time to detect and analyse suspicious activities on the monitored hosts. Thus, EDRs boost the capabilities of SOCs as they discover and alert both the user and the emergency response teams of emerging cyber threats.
EDRs are heavily rule-based; nevertheless, machine learning or AI methods have gradually found their way into these systems to facilitate finding new patterns and correlations. An EDR extends antivirus capabilities as an EDR will trigger an alert once it detects anomalous behaviour. Therefore, an EDR may detect unknown threats and prevent them before they become harmful due to the behaviour and not just merely the signatures. While behavioural patterns may sound ideal for detecting malicious acts, this also implies many false positives; that is, benign user actions considered malicious, as EDRs prioritise precision over recall. Therefore, SOCs have to deal with sheer amounts of noise as many of the received alerts are false [5]. This is the reason why Hassan et al. recently introduced Tactical Provenance Graphs (TPG) [12]. They reason about the causal dependencies between the threat alerts of an EDR and improve the visualisation of multistage attacks. Moreover, their system, RapSheet, has a different scoring system that significantly reduces the false positive rate. Finally, an EDR can perform remediation or removal tasks for specific threats.
Despite the significant boost in security that EDRs bring, the overall security of the organisation highly depends on the human factor. In the case of the blue teams, the results against an attack are expected to greatly vary between fully trained teams in Incident Response and teams that solely respond to specific detected threats and are dependent on the output of a single security tool. However, both teams are expected to be triggered by and later investigate the telemetry from EDRs. Since the experience and the capacity of the blue team depends on multiple factors which are beyond the scope of our work, in this study we focus on the telemetry of the EDRs, the significance that they label events, and whether they blocked some actions.
Nevertheless, we highlight that not all EDRs allow the same amount of customisation nor implementation of the same policies. Moreover, blue teams cannot have the experience in all EDRs to configure them appropriately as each team will specialise in a limited set of solutions due to familiar-ity with a platform, marketing or even customer policies. Moreover, not all blue teams face the same threats which may significantly bias the prioritisation of rules that blue teams would include in an installation, let alone the client needs. The above constitute diverse factors that cannot be studied in the context of this work. On the contrary, we should expect that a baseline security when opting in for all possible security measures should be more or less the same across most EDRs. Moreover, one would expect that even if the EDR failed to block an attack, it should have at least logged the actions so that one can later process it. However, our experiments show that often this is not the case.

Advanced persistent threats
The term advanced persistent threat (APT) is used to describe an attack in which the threat actor establishes stealth, long-term persistence on a victim's computing infrastructure. The usual goal is to exfiltrate data or to disrupt services when deemed necessary by the threat actor. These attacks differ from the typical 'hit and run' modus operandi as they may span from months up to years. The attacks are launched by high-skilled groups, which are either a nation state or state-sponsored.
As noted by Chen et al. [6], APT attacks consist of six phases: (1) reconnaissance and weaponization; (2) delivery; (3) initial intrusion; (4) command and control; (5) lateral movement; and (6) data exfiltration. Complimentary to this model, other works [11,20] consider attack trees to represent APTs as different paths may be used in parallel to get the foothold on the targeted resources. Thus, information flows are often used to detect APTs [3] along with anomaly detection, sandboxing, pattern matching, and graph analysis [1]. The latter implies that EDRs may serve as excellent means to counter APT attacks.
In many such attacks, threat actors use fileless malware [15], a particular type of malware that does not leave any malicious fingerprint on the filesystem of the victim as they operate in memory. The core idea behind this is that the victim will be lured into opening a benign binary, e.g. using social engineering, and this binary will be used to execute a set of malicious tasks. In fact, there are plenty of binaries and scripts preinstalled in Windows or later downloaded by the OS and are either digitally signed or whitelisted by the operating system and enable a set of exploitable functionalities to be performed. Since they are digitally signed by Microsoft, User Account Control (UAC) allows them to perform a set of tasks without issuing any alert to the user. These binaries and scripts are commonly known as Living Off The Land Binaries and Scripts (and also Libraries), or LOLBAS/LOLBINS [4].

Cyber kill chain
Cyber kill chain is a model which allows security analysts to deconstruct a cyber attack, despite its complexity, into mutually nonexclusive phases [13]. The fact that each phase is isolated from the others allows one to analyse each part of the attack individually and create mitigation methods and detection rules that can facilitate defence mechanisms for the attack under question or similar ones. Moreover, blue teams have to address smaller problems, one at a time which is far more resource efficient than facing a big problem as a whole. In the cyber kill chain model we consider that a threat actor tries to infiltrate a computer network in a set of sequential, incremental, and progressive steps. Thus, if any stage of the attack is prevented, then the attack will not be successful. Therefore, the small steps that we referred above are crucial in countering a cyber attack and the earlier phase one manages to prevent an attack, the smaller impact it will have. While the model is rather flexible, it has undergone some updates to fit more targeted use cases, e.g. Internal Cyber Kill Chain to address issues with internal malicious actors; such as a disgruntled or disloyal employee.
MITRE's ATT&CK [21]is a knowledge base and model which tries to describe the behavior of a threat actor throughout the attack lifecycle from reconnaissance and exploitation, to persistence and impact. To this end, ATT&CK provides a comprehensive way to categorize the tactics, techniques and procedures of an adversary, abstracting from the underlying operating system and infrastructure. Based on the above, using ATT&CK one can emulate threat scenarios 2 or assess the efficacy of deployed defense mechanisms against common adversary techniques. More recently, Pols introduced the Unified Kill Chain 3 which extends and combines Cyber Kill Chain and MITRE's ATT&CK. The Unified Kill Chain addresses issues that are not covered by Cyber Kill Chain and ATT&CK as, among others, it models adversaries' behaviours beyond the organizational perimeter, users' roles etc.

Experimental Setup
In this section, we detail the preparation for our series of experiments to the EDRs and EPPs. Because our goal is to produce accurate and reproducible results, we provide the necessary code where deemed necessary. To this end, we specifically design and run experiments to answer the following research questions: • RQ1: Can state of the art endpoint security systems detect common APT attack methods?
• RQ2: Which are the blind spots of state of the art endpoint security systems?
• RQ3: What information is reported by EDRs and EPPs and which is their significance?
• RQ4: How can one decrease the significance of reported events or even prevent the reporting?
Using ATT&CK is a knowledge base and model, one can model the behaviour of the threat actor that we emulate as illustrated in Figure 1. Due to space limitations, we have opted to use a modified version of the standard ATT&CK matrix and used a radial circular dendrogram. In this work, we perform an empirical assessment of the security of EDRs. The selected EDRs were selected based on the latest Gartner's 2021 report 4 , as we included the vast majority of the 4 https://www.gartner.com/en/documents/4001307/magic-quadrant-for-endpoint-protection-platforms leading EDRs in the market. The latter implies that we cover a big and representative market share which in fact drives the evolution and innovation in the sector. In our experiments, we opted to use the most commonly used C2 framework, Cobalt Strike 5 . It has been used in numerous operations by both threat actors and 'red teams' to infiltrate organisations [22].
Moreover, we used a mature domain; an expired domain with proper categorisation that will point to a VPS server hosting our Cobalt Strike team-server. This would cause less suspicion and hopefully bypass some restrictions as previous experience has shown with parked domains and expired domains 6 . We issued a valid SSL certificate for our C2 communication from Let's Encrypt 7 to encrypt our traffic. Figure 2 illustrates our domain and its categorisation. Cobalt Strike deploys agents named 'beacons' on the victim, allowing the attacker to perform multiple tasks on the compromised host. In our experiments, we used the so-called malleable C2 profile 8 as it modifies the beacon's fingerprint. This masks our network activity and our malware's behaviour, such as the staging process, see Listing 6 in Appendix. Please note that it has been slightly formatted for the sake of readability.

Attack Vectors
We have structured four diverse yet real-world scenarios to perform our experiments, which simulate the ones used by threat actors in the wild. We believe that an empirical assessment of EDRs should reflect common attack patterns in the wild. Since the most commonly used attack vector by APT groups is emails, as part of social engineering or spear phishing, we opted to use malicious attached files which the target victim would be lured to execute them. Moreover, we should consider that due to the high noise from false positives that EDRs report, it is imperative to consider the score that each event is attributed to. Therefore, in our work we try to minimise the reported score of our actions in the most detailed setting of EDRs. With this approach we guarantee that the attack will pass below the radar.
Based on the above, our hypothetical threat actor starts its attack with some spear-phishing emails that try to lure the target user into opening a file or follow a link that will be used to compromise the victim's host. To this end, we have crafted some emails with links to cloud providers that lead to some custom malware. More precisely, the attack vectors are the following: • A .cpl file: A DLL file which can be executed by double-clicking under the context of the rundll32 LOLBINS which can execute code maliciously under its context. The file has been crafted using CPLResourceRunner 9 . To this end, we use a shellcode storage technique using Memory-mapped files (MMF) [17] and then trigger it using delegates, see Listing 1.  Listing 1: Shellcode execution code from CPLResourceRunner.
• A legitimate Microsoft (MS) Teams installation that will load a malicious DLL. In this regard, DLL side-loading 10 will lead to a self-injection, thus, allowing us to "live" under a signed binary. To achieve this, we used the AQUARMOURY-Brownie 11 .
• An unsigned PE executable file; from now on referred to as EXE, that will execute process injection using the "Early Bird " technique of AQUARMOURY into werfault.exe. For this, we spoofed the parent of explorer.exe using the PROC THREAD ATTRIBUTE MITIGATION POLICY flag to protect our malware from an unsigned by Microsoft DLL event that is commonly used by EDRs for processes monitoring.
• An HTA file. Once the user visits a harmless HTML page containing an IFrame, he will be redirected and prompted to run an HTML file infused with executable VBS code that will load the .NET code provided in Listing 2 perform self-injection under the context of mshta.exe.
In what follows, we solely evaluate EDRs against our attacks. Undoubtedly, in an enterprise environment one would expect more security measures, e.g., a firewall, an antivirus, etc. However, despite improving the overall security of an organisation, their output is considered beyond the scope of this work.

Code Analysis
In the following paragraphs, we detail the technical aspects of each attack vector.

HTA
We used C# and the Gadget2JScript 12 tool to generate a serialized gadget that will be executed into memory, see Listing 2. ETWpCreateEtwThread is used to execute the shellcode by avoiding common APIs such as CreateThread(). Note that in the background, RtlCreateUserThread is used 13 . 4 string key = " mysecretkeee " ; 5 shellcode = xor ( xored , Encoding . ASCII . GetBytes ( key ) ) ; 6 uint old = 0; 7 // Gets current process handle 8 IntPtr procHandle = Process . G e t C u r r e n t P r o c e s s () . Handle ; 9 // Allocation and then change the page to RWX 10 IntPtr a ll o cM em Ad d re ss = Vi rtualAll ocEx ( procHandle , IntPtr . Zero , ( uint ) shellcode . Length , MEM_COMMIT | MEM_RESERVE , 11 PAGE _READWRI TE ) ; 12 V i r t u a l P r o t e c tE x ( procHandle , allocMemAddress , ( UIntPtr ) shellcode . Length , PAGE_EXECUTE_READWRITE , out old ) ; 13 // Write the shellcode 14 UIntPtr bytesWritten ; 15 W r i t e P r o c e s s M e m o r y ( procHandle , allocMemAddress , shellcode , ( uint ) shellcode .
Length , out bytesWritten ) ; 16 E t w p C r e a t e E t w T h r e a d ( allocMemAddress , IntPtr . Zero ) ; Listing 2: Code to allocate space and execute shellcode via EtwpCreateEtwThread.

EXE File
The main idea behind this attack is a rather simplistic code injection using executing our shellcode using the QueueUserAPC() API before the main method. It will launch a sacrificial process with PPID spoofing and inject to that. The file will employ direct system calls in assembly to avoid hooked functions. It should be noted that the Windows Error Reporting service (werfault) is an excellent target for injection as a child werfault process may appear once a process crashes, meaning the parent can be arbitrary. This significantly impedes parent-child relation investigation. Notably, once used with the correct flags, it can avoid suspicions [19]. Find the relevant code in Listing 3. Listing 3: Execution of shellcode into a child process with CIG and spoofed PPID via the "EarlyBird" technique using Nt* APIs.

DLL Sideloading
In this case, we used the Brownie -Koppeling projects to create an evil clone of a legitimate DLL from system32 and added it to the folder of MS Teams so that our encrypted shellcode will be triggered under its process. Moreover, since MS Teams adds itself to the startup, this provides us persistence to the compromised host. Note that EDRs sometimes tend to overlook self-injections as they consider that they do not alter different processes.
In Listing 5 we illustrate the shellcode execution method. It is a classic CreateThread() based on local injection that will launch the shellcode under a signed and benign binary process. Unfortunately, the only problem, in this case, is that the DLL is not signed, which may trigger some defence mechanisms. In the provide code, one observe the usage of VirtualProtect(). This was made to avoid direct RWX memory allocation. In Listing 4 we can see the usage of assembly syscalls.
Finally, it should be noted that for the tests, the installation will be placed and executed in the Desktop folder manually. Figure 3 illustrates that MS Teams allows for DLL hijacking.

EDR and EPP evaluation
In what follows paragraphs, we evaluate fourteen state of the art EDRs and five EPPs against our attacks. To this end, we provide a brief overview of each EDR and its features. Then, we proceed reporting which features were enabled and discuss how each of them performed in the attack scenario. EDRs and EPPs are listed in alphabetical order.

BitDefender GravityZone Plus
BitDefender GravityZone Plus is the company's flagship including EDR, EPP, and SandBox capabilities. Its use of common telemetry providers is exemplary as far as the tests are concerned and tries to make the most out of them with a highly intelligent engine which correlates the information that in turn leads to immediate blocking and remmediation as well as a robust console.

CPL
This vector was blocked as a behavioural alert of cobalt strike, as illustrated in Figure 4.

HTA
This vector was instantly detected as malicious and was blocked, see

DLL
This vector was blocked but did not raise a major alert. However, its events were included in another attack vector detection as illustrated in Figure 4.

EXE
The product is very dependant on UM Hooks, in this case the content was not blocked nor raised any alert/event as it uses syscalls.

Carbon Black Response
Carbon Black is one of the leading EDR solutions. Its true power comes from its telemetry and its ability to extensively monitor every action performed on a system, such as registry modifications, network connections etc., and most importantly, provide a SOC friendly interface to triage the host. Based on the telemetry collected from the sensor, a comparison to several IoCs. The latter will be aggregated into a score which depending on its value, will trigger an alert. Moreover, when considering EDRs, configuration plays a vital role. Therefore, in this case, we have a custom SOC feed for detections based on IOCs that Carbon Black processes. Also, the feeds can be query-based, meaning that alerts will be produced based on results yielded by searches based on the events that Carbon Black processes, including but not limited to, registry modifications, network connections, module loadings.
This EDR relies heavily on kernel callbacks and a lot of its functionalities reside in its network filtering driver and its file system filtering driver. For several detections, user-mode hooks are also used. As an example, consider the detection of memory dumping (DUMP PROCESS MEMORY). As mentioned in Carbon Black's documentation, userland API hooks are set to detect a process memory dump. Another example is the detection of script interpreters loaded into memory (HAS SCRIPT DLL). As mentioned in the documentation, a driver routine is set to identify processes that load an inmemory script interpreter.

Enabled settings
Carbon Black Response is different in terms of logic and use case. Its main purpose is to provide telemetry and not to proactively act. Moreover, its scope is to assist during an investigation as it does not include blocking capabilities but is a SOC friendly software that gives in-depth visibility. Its power is closely related to the person behind the console as beyond triaging hosts, its detection rely on feeds that can be customized and produce alerts. In our case we used some default feeds such as ATT&CK feed and Carbon Black's Community Feed as well as a custom corporate feed. As illustrated in Figure 6, an alert was triggered due to the abnormal name, location and usage of Shell32.dll. Carbon Black is well aware of malicious .cpl files in this case, but it cannot clearly verify whether this activity is indeed malicious. Therefore, the event is reported with a low score. Figure 7 illustrates on the right side the IOCs that were triggered.

HTA
The .hta file was detected due to its parent process as a possible CVE and for a suspicious loaded module. Carbon Black is aware of both LOLBAS and LOLBINS and timely detected it.

Carbon Black Cloud
Further to Carbon Black Response, we also tested Carbon Black Cloud, the flagship of the company when it comes to endpoint protection. It reserves the same capabilities for highly efficient telemetry processing and collection with blocking as an additional feature and NGAV capabilities.
It follows the same principles as the Response version with extended core functionality.

Enabled settings
We used the Advanced policy created by VMWare with all settings set to block.

EXE-DLL
Both attacks were successful without triggering any alert.

CPL-HTA
Both attacks were detected by Carbon Black Cloud raising a major alert, however, none of them was blocked, see Figure 10. Quite interestingly, the HTA related IOC was loading the CLR.DLL under the context of the aforementioned LoLbin. This is a clear indication of .NET code running under this process and of the well-known G2JS tool. As for the CPL file, the same classic detection of abnormal CPL files applied. The detections were reported as non-blocked.

Enabled settings
For Check Point Harmony, we used an prevent mode where possible and enabled emulation/behavioural (antibot antiexploit), and did not turn on safesearch setting to prevent checks of hashes.

HTA-CPL
For the HTA attack vector, a medium alert was raised, but the attack was not blocked. see Figure  11. In the case of the CPL, the attack was blocked, and an alert was raised in the console, see Figure  12.

EXE
The EXE attack vector was detected and blocked, see Figure 13.

DLL
The DLL attack vector was not blocked nor detected.

Cisco Secure Endpoint (ex AMP)
AMP is Cisco's EDR which provides endpoints with prevention, detection, and response capabilities, as well as threat hunting. Moreover, it uses cloud-based analytics and machine learning to timely detect threats.

Enabled settings
In this EDR we used the "Standard Protection Policy" activating the "Malicious Script Blocking" feature.

CPL-HTA
Both attacks were blocked. In the case of the CPL file, the file was quarantined, while in HTA case, the process was killed, see Figure 14.

DLL
In the case of the DLL attack vector we noticed that while the attack was blocked, see Figure 15, the provided alert was for exploit blocking. Therefore, we opted to perform the same attack, but with a different application. Indeed, the problem seemed to be the specific application, so once we used another app but the same technique, the attack was successful.

EXE
This attack vector was successful and raised no alert.

Enabled settings
For OpenEDR we used the the preconfigured profile that claims to offer maximum security namely "Level 3 Security (Max)"

CPL-EXE
Both attacks were blocked by the EDR using Commodo's containment technology. While the files were sent to console, no alert was raised, see Figure 16.

CrowdStrike Falcon
CrowdStrike Falcon combines some of the most advanced behavioural detection features with a very intuitive user interface. The latter provides a clear view of the incident itself and the machine's state during an attack through process trees and indicators of attacks. Falcon Insight's kernel-mode driver captures more than 200 events and related information necessary to retrace incidents. Besides the classic usage of kernel callbacks and usermode hooks, Falcon also subscribes to ETWTi 14 .
When it comes to process injections, most EDRs, including Falcon, continuously check for Windows APIs like VirtualAllocEx and NtMapViewOfSection prior to scanning the memory. Once Falcon finds any of these called by any process, it quickly checks the allocated memory and whether this was a new thread created from a remote process. In this case, it keeps track of the thread ID, extracts the full injected memory and parses the .text section, the Exports section, the PE header, the DOS header and displays the name of the PE, start/stop date/time, not limited to the export address of the loaded function.
As for the response part, it provides extensive real-time response capabilities and allows the creation of custom IOAs based on process creation, network connections, file creation, among others.

Enabled settings
For this EDR we used an aggressive policy enabling as much features as possible. It was a policy already used in a corporate environment with its goal being maximum protection and minimum disruption.

DLL -CPL -HTA
None of these three attack vectors produced any alerts and allowed the Cobalt Strike beacon to be executed covertly.

EXE
Quite interestingly, the EXE was detected, although direct system calls were used to bypass user-mode hooking. Note that the alert is of medium criticality. Also, please note the spoofed parent process in Figure 17.

Cylance PROTECT
Cylance is one of the first players according to their claims to integrate AI and ML to elevate their product to an 'NGAV' (Next Generation Anti Virus). Cylance PROTECT was used as a refernce point by many researchers in the past 15 due to the fact that it was too much dependent on user mode hooks and it was an excellent example of how new-age protections can be bypassed both engine-wise and telemetry-wise. The solution has evolved much since then, yet, the strategic approach towards detections seems to be built upon the same traditional basis with an AI flavor.

Enabled settings
The configuration of Cylance PROTECT is a very straightforward approach. For our experiments, we set all possible protections to 'on' and 'block & quarantine'; where applicable, and enabled verbose logging.

EXE
The EXE attack vector was not detected by the Cylance PROTECT.

DLL-CPL-HTA
The DLL attack vector was blocked, see Figure 18. Notably, for the rest two attack vectors (CPL and HTA), despite the fact that we performed several tests, both were not executed under Cylance PROTECT. However, despite the fact that we enabled verbose logging, no alert nor any blocking took place according to the provided interface during the our experiments.

Cynet
Cynet's core resides in its Cynet Endpoint Scanner and some of the main capabilities it promotes are memory based artifact scanning and detections as well as 'memory pattern alerts' which run on a real-time basis and provide continuous monitoring of the system process-wise. Most of its detections and protections are based on its minifilter driver and its machine learning-based algorithm for static detections, which is what it makes it an Next-Generation Antivirus (NGAV) according to Cynet. It also exploits memory scanning as mentioned above to detect generic threats and common ETW providers; not EtwTI for the time being. Finally, Cynet uses network filtering methods to gain further insight.
The company's approach, as they informed us, is based on a multi-layered mindset which tries to provide as much info as possible based on the telemetry sources. An example of this mindset is that LSASS dumping protection is based on the kernel level which tackles user-mode dumping techniques that rely on handle opening to LSASS. However, on top of monitoring of command line utilities further monitoring of strings like "procdump" or "lsass" takes place to guarantee more insight.
As a more generic example of kernel level process handle monitoring and protection we could take a look at the ObRegisterCallbacks routine which is essentially Microsoft's kernel based hooking of process and thread handles intended to be used mainly by AV solutions, multiple examples exist online on how one can protect processes using this approach commonly combined with access mask filtering 16 . Cynet also employs a simple yet effective (when the ransomware is target agnostic) protection for ransomware based on decoy files, among others which has proven to be a lifesaver in multiple cases. Moreover, memory pattern alerts are string-based detections according to one of their articles.
While most of the attacks were succesful, Cynet claims that some discrepancy has resulted to a malfunction of the memory scanner which immediately tackles generic malware, such us Cobalt Strike that we used. After the assessment, the vendor immediately committed and timely responded pushing a number of updates and managed to fix many of the issues, including the ones related to the memory scanner, see Figure 19.

Enabled settings
In Cynet we used an aggressive configuration enabling prevent mode, when possible, and enabled all features including ADT, Event monitoring etc.

HTA-EXE-DLL
All three attack vectors were successful without raising any alert.

Elastic EDR
Elastic EDR is one of the few open source solutions in the market. It is built upon the well-known ELK stack allowing for advanced search and visualisation capabilities and its open nature allows for further customisation.

Enabled settings
We enabled all prevention settings and available sources, e.g. file modifictions.

DLL
The DLL attack was detected and blocked once it touched the disk, see Figure 20.

EXE-HTA
Both attacks were successfully launched and did not raise any alert.

ESET PROTECT Enterprise
ESET PROTECT Enterprise is a widely used endpoint solution that uses behaviour and reputation systems to mitigate attacks. Moreover, it uses cloud sandboxing to prevent zero-day threats and full disk encryption for enhanced data protection. The EPP uses real-time feedback collected from million of endpoints using, among others, kernel callbacks, ETW (Event Tracing for Windows), and hooking. ESET PROTECT Enterprise allows fine-tuning through editing XML files and customising policies depending on users and groups. For this, blue teams may use a file name, path, hash, command line, and signers to determine the trigger conditions for alerts.
We used ESET PROTECT Enterprise with the maximum available predefined settings, see Figure  22 without further fine tuning.

Enabled settings
For this EPP we used the predefined policy for maximum security, as stated by ESET in the console. This makes use of machine learning, deep behavioural inspection, SSL filtering, PUA detection and we decided to hide the GUI from the end user.

EXE-DLL
Both these attack vectors were successfully executed, without the EPP blocking and reporting any alert, see Figure 23.

CPL-HTA
The CPL and HTA attacks were correctly identified and blocked by ESET PROTECT Enterprise, see Figures 25 and 24, respectively. It should be noted that the memory scanner of ESET correctly identified malicious presence but falsely named the threat as Meterpreter.

F-Secure Elements
F-Secure Elements can have several products under it, for this experiment, two products were tested, namely Endpoint Protection Platforms (EPP) and Endpoint Detection and Response solutions (EDR). Both solutions collect behavioural events from the endpoints, including file access, processes, network connections, registry changes and system logs. To achieve this, the Elements use Event Tracing for Windows among other capabilities. While F-Secure Elements EDR uses machine learning for enrichments, human intervention from cyber-security experts is often used. The EDR also features built-in incident management. Moreover, after a confirmed detection, F-Secure Elements EDR has built-in guidance to facilitate users in taking the necessary steps to contain and remediate the detected threat.

Enabled settings
In terms of our experiments, we experimented with both the EPP and the EDR solution enabling all features available, including DeepGuard. We also included browsing control based on reputation, and the firewall was up and running. In the first version of the manuscript, only the results of the EPP were included. Notably, all of the launched attacks were successful, and F-Secure Elements EPP reported no alerts, see Figure 26.
However, after collaborating with F-Secure, it was discovered that the initial test was only done for the EPP solution. As such, F-Secure assisted in setting up the licensing for the EDR product so that we can perform the test from our environment. In order to make sure that no new detections are considered, in this configuration, the database license was downgraded to an earlier date: June 18, 2021. We tested these attacks against the F-Secure EDR twice. There were three attacks detected during these tests. Two of these attacks were detected immediately, while the third one had a time delay of 5 hours during the initial. Since F-Secure downgraded the databases, there was some confusion that led to the misconfiguration of the backend systems. Once the misconfiguration was

F-Secure EPP
In the case of F-Secure EPP no attack was detected nor blocked, see Figure 26 as also validated by the vendor.

F-Secure EDR
In the case of F-Secure EDR, as already discussed, two experiments were conducted in collaboration with F-Secure. There were three attacks detected during these tests. Two of these attacks were detected immediately, while the third one had a time delay of 5 hours during the initial. experiment. According to F-Secure this was due to the database downgrade which caused a misconfiguration of the backend systems. After some resolution from the vendor side, the delay for that particular attack was reduced to 25 minutes. Due to the nature of EDR products, none of the attacks was blocked. It should be noted that, as illustrated in Figure 27, the two attack vectors were merged into one attack from the EDR, where one of them was marked with a medium alert. However, the merging of the attacks can be attributed to their timing. Finally, the EXE attack vector was successful in all cases. A brief detection history regarding the detections that the F-Secure Elements EDR collected is illustrated in Figure 28.

FortiEDR
FortiEDR is heavily based on its simulation mode which we did not use due to time constraints and the nature of the experiments, its a training session for it to learn and understand the function of the organization. It makes the most out of the callbacks and tries identify and block the unmapped code and its dynamic behaviour in the infection process. According to our experiments these alerts occur in cases where reflective injection is performed as we have observed this alert in several tools that use the aforementioned technique,also , as mentioned in the description the alert is related to files loaded form memory. Also, the COM activity for the HTA was blocked.

Enabled settings
In FortiEDR we used an aggressive setting with all features enabled and block mode everywhere.

CPL-HTA-EXE-DLL
FortiEDR managed to detect and block all attack vectors as illustrated in Figures 29, 30, 31, and 32.

Harfang Lab Hurukai
Harfang Lab is a new player in the market. Their EDR tries to provide extensive visibility and some highly needed analysis features including real-time disassembly and Yara based capabilities to detect malicious patterns and extend the initial detections. Although blocking and prevention is not their man goal, the product can be tuned accordingly and provide those capabilities according to their R&D team.

Enabled settings
The experiments were conducted in collaboration with the Harfang Labs' team and they were set to provide the highest level of possible detection, however, it was not set in block mode.

CPL-EXE
The CPL attack vector was detected as a suspicious execution, see 33. The EXE attack vector was also detected, see Figure 34 due to the fact that the sacrificial process werfault.exe did not have any flags (even dummy) as it normally does. Practically, if dummy values would be present, the

DLL-HTA
Both attack vectors were not detected by the EDR.

ITrust ACSIA
ITrust claims that its security solution ACSIA, that it positions as an EDR is a game changer with a totally different approach that would prevent the attack before the payload reaches the target. Their solution is not based on traditional telemetry sources and does not have a custom mini-filter, but seems to be based on tools like a log collector.

Enabled settings
All the settings were enabled by the vendor in the provided environment.

CPL-HTA-EXE-DLL
All attack vectors passed without any alert being raised by the EDR. Interestingly, considering the fact that the legitimate C runtime installation needed for the experiments triggered an alert, see Figure 35 we attempted to use a malicious .msi file to further trigger the solution to verify that all  components were working as they should. The attack was not blocked and the installation was found clean, the correct functionality was verified by the vendor too.

Kaspersky Endpoint Security
Kaspersky Endpoint Security is an endpoint security platform with multi-layered security measures that exploits machine learning capabilities to detect threats. Moreover, this EPP agent serves also as the EDR agent also facilitating vulnerability and patch management and data encryption.

Enabled settings
In our experiments, we enabled all security-related features in every category. However, we did not employ any specific configuration for Web and Application controls. More precisely, we created a policy and enabled all options including behavior detection, exploit and process memory protection, HIPS, Firewall, AMSI and FileSystem oritection modules. The actions were set to block and delete all malicious artifacts and behaviors.

CPL-HTA-EXE
In the case of CPL, HTA, and EXE attack vectors, Kaspersky Endpoint Security timely identified and blocked our attacks, see Figure 36. More precisely, the EXE and CPL processes were killed after execution, while the HTA was blocked as soon as it touched the disk.

DLL
Our DLL attack was successfully launched and no telemetry was recorded by Kaspersky Endpoint Security.

McAfee Endpoint Protection
McAfee Endpoint Protection is among the most configurable and friendly to the technical user solutions, it allows reacting to specific process behaviours, i.e. remote memory allocation, but also to proactively eliminate threats by reducing the options an attacker has based on a handful of options such as blocking program registration to autorun. We decided to leverage this configurability and challenge McAfee Endpoint Protection to the full extend and only disabled one rule blocking execution from common folders such as the Desktop folder. The rationale behind this choice is usability since activating this rule would cause many usability issues in an everyday environment.
In our experiments, we managed to successfully bypass the restrictions using our direct syscalls dropper and allocate memory remotely as well as execute it. The latter is an indicator that the telemetry providers and processing of the information is not efficient.

Enabled settings
For this EPP, we decided to challenge McAfee since it offers a vast amount of settings and a lot of option for advanced users such as memory allocation controls etc. It was also quite interesting that some policies were created by default to block suspicious activities such as our HTA's execution. We opted to enable all options without exception apart from one that was block execution from user folders and would cause issues in a corporate environment.
An excerpt of the settings that were enabled is illustrated in Figure 37.

HTA-CPL
Both HTA and CPL-based attacks were identified and blocked. However, it should be noted that the HTA attack was blocked due to the applied policy of blocking execution of all HTA files, see Figure  38.

EXE-DLL
Both the EXE and DLL-based attacks were successfully executed without being identified by McAfee Endpoint Protection nor producing any telemetry.

McAfee Endpoint Protection infused with the MVision EDR
Further to the aforementioned test, we conducted the tests again with the collaboration of McAfee engineers extending our testing with a newer version that included their EDR solution too.

Enabled settings
The policy was configured for optimal security in collaboration with McAfee's engineers.

DLL-EXE
Both attack vectors passed without triggering any alert.

CPL-HTA
The CPL attack was blocked by the EPP, as in the previous experiment, see Figure 39. The HTA attack was detected with a low alert, but was not blocked, see Figure 40. The above validate the original results of the EPP and extend it with a minor alert for the EDR.

Microsoft Defender for Endpoints (ex. ATP)
Microsoft Defender for Endpoints is heavily kernel-based rather than user-based, which allows for great detection capabilities. The beauty of MDE lies in the fact that most of the detection capability lies in Windows itself, albeit not utilised unless the machine is onboarded. For these tests, the EDR was set to block mode to prevent instead of merely detecting. Its telemetry sources include kernel callbacks utilised by the WdFilter.sys mini-filter driver. As previously mentioned callbacks are set to 'intercept' activities once a condition is met, e.g. when module is loaded. As an example of those consider: • PsSetCreateProcessNotifyRoutine(Ex) -Process creation events.
They also include a kernel-level ETW provider rather than user-mode hooks. This comes as a solution to detecting malicious API usage since hooking the SSDT (System Service Dispatch Table) is not allowed thanks to Kernel Patch Protection (KPP) PatchGuard (PG). Before moving on we should note a different approach taken by Kaspersky to hook the kernel it made use of its own hypervisor. This comes with several downsides as it requires virtualization support 17 . Since Windows 10 RS3, the NT kernel is instrumented using EtwTi functions for various APIs commonly abused for process injection, credential dumping etc. and the telemetry available via a secure ETW channel 18 . Thus, MDE heavily relies on EtwTi, in some cases even solely, for telemetry. As an example of the ETWTi sensor, consider the alert below 41. It is an alert produced by running our EXE payload on a host that MDE is in passive mode. Note that although our payload uses direct system calls, our injection is detected.
Due to the fact that the callbacks operate at the kernel level (Ring 0), an attacker needs to have high integrity level code execution in a machine to blind them or render them useless successfully. An attacker may choose any one of the following three techniques to achieve this: • Zero out the address of the callback routine from the kernel callback array that stores all the addresses.
• Unregister the callback routine registered by WdFilter.sys.
• Patch the callback routine of WdFilter.sys with a RET(0xc3) instruction or hook it.
Due to the nature of the ETWTi Sensor telemetry, it is not possible to blind the sources from a medium-IL context and needs admin/a high-IL context. Once this is achieved, an attacker may employ any one of the following methods: • Patch a specific EtwTi function by inserting a RET/0xC3 instruction at the beginning of the function so that it simply returns without executing further. Not KPP-safe, but an attacker may avoid BSOD'ing the target by simply restoring the original state of the function as soon as their objective is accomplished. In theory, Patch Guard may trigger at any random time, but in practice, there is an extremely low chance that PG will trigger exactly during this extremely short interval.
Due to interaction with Microsoft, we performed three iterations of the tests to see how they perform against the attack vectors in collaboration with their team due to the fact that important changes had been pushed to the platform. To this end, the results are organised according to each iteration.

Enabled settings
We enabled all the basic features including the tamper protection, the block mode option and auto investigation. Most is handled in the background and the admins are able to configure connection to intune which was out of scope. We also enabled file and memory content analysis using the cloud that will upload suspicious files and check them.

Original experiments
CPL -EXE -HTA Most of these vectors were detected as soon as they touched the disk or were executed. Find the relevant alerts in Figure 42. Note that for the .cpl file, despite the fact that the EDR detected it, it was executed with a fully functional beacon session.See Figure 43.
Find below the relevant auto-investigation started for this MDE incident, including all the alerts produced. Note that till successful remediation and full verdict, the investigation may take a lot of time. See Figure 44 DLL Figure 43: Details about the alerts produced from MDE.
The DLL side-loading attack was successful as the EDR produced no alerts nor any suspicious timeline events. Figure 45 illustrates the produced telemetry. Notice the connection to our malicious domain and the uninterrupted loading of our module.

CPL
The CPL attack vector was detected in the new test but was not blocked, replicating the behaviour of the original test.
DLL-EXE-HTA All three attack vectors were detected and blocked in the new tests.

CPL
We noticed that CPL attack vector was detected with the modified IOCs but was not blocked immediately. In fact, the payload was running for around 15 minutes before being blocked in memory by MDE.
DLL In the revised experiment we opted to change the MS Teams target binary in order to use Licencing UI. Contrary to the previous test with the updated MDE, the attack was successful and was not detected by MDE.
EXE-HTA Both attack vectors were detected and blocked by MDE.

Minerva Labs
Minerva Labs is a new player in the market with its main goal being user-mode based prevention based on deception and target emulation. The main idea behind this is that Minerva uses several modules to detect and prevent activities ranging from sandbox detection used by malware to fingerprinting target environments to process injection detection and LoLBin usage prevention. The product uses a multi-layered approach in general. The aforementioned LoLBin detection is one of the best examples as if a malicious LoLBin-based file like a CPL or an HTA is dropped into a writable by the user directory, it would be blocked regardless whether it is malicious, a text file with a renamed extension or a signed .cpl file like appwiz.cpl. The next stage with regards to the process created using that cpl, somewhat similar to the approach of Carbon Black, would be to check with other modules, such as the process injection prevention. The philosophy is being a full blown ring 3 rootkit with app control capabilities and false positive prevention via custom whitelisting through a right-click process.

Enabled settings
We used all modules as offered by Minerva including their ruleset which comes as a part of their EDR. The engineers verified the correct function of the tenant.

EXE
The .exe used direct system calls which were not detected by the solution also using a werfault process which has an arbitrary parent was not abnormal.

CPL
The .cpl file was blocked by the LoLBin protection as it was, but the technique used for injection by itself was not blocked.

HTA -DLL
The HTA file was blocked both on the LoLbin execution level and the injection level. Moreover, the DLL file was blocked on the process injection level, see Figure 46.

Panda Adaptive Defense 360
Panda is a well-known solution that was categorized by Gartner for 2021 and 2019 as a "niche player". Its detections are based on kernel callbacks and ETW mostly as far as the vectors are concerned. It provides the user with a UI on which the entire attack paths can be seen and according to the vendors provides the clients with "unified EPP and EDR capabilities to effectively detect and classify 100% of processes running on all the endpoints within your organization".

Enabled settings
We created a policy for maximum active protection.

CPL
The CPL attack vector was detected and blocked but only the host had an alert about it, see Figure  47.

EXE
In this case, the attack was successful and after some time an alert was raised, see Figure 48.

DLL -HTA
Both attack vectors were successful and raised no alert.

Palo Alto Cortex
Palo Alto Cortex is one of the most intriguing solutions of the market at the moment as they advertise extended next level detection and features such as multilevel detection approach, including hooking and in memory protections. Palo Alto is already a leader in the network protection market their main goal is to move towards a holistic mindset.

Enabled settings
Our configuration included blocking exploitation attempts, malware, PE examination, and taking full advantage of the solution's cloud integration-analysis. We opted for behavioural quarantine and data gathering for investigation (a Pro version feature) but no telemetry collection as it is by default.

EXE
The WildFire analysis found zoom.exe benign, but the behavioral analysis blocked it at runtime and accurately pointed out the APC based injection attack, as well as the syscall usage, see Figure 49.

CPL-DLL-HTA
All three attack vectors were not detected nor blocked by the EDR.

Sentinel One
Sentinel One has sophisticated AI-based behavioural analysis features that make stealth infiltration and tool execution rather difficult. Among others, Sentinel One collects ETW telemetry and monitors almost all parts of the system. It uses kernel callbacks to collect information such as process creation, image load, thread creation, handle operations, registry operations. It also produces detailed attack paths and process tree graphs.
Also, Sentinel One recently released a new custom detection engine called STAR. With STAR custom detection rules, SOC teams can turn queries from Deep Visibility, SentinelOne's EDR data collection and querying mechanism, into automated hunting rules that trigger alerts and responses when rules detect matches. STAR also allows users an automated way to look at every endpoint event collected across their entire fleet and evaluate each of those events against a list of rules.
Howerver, our results indicate that the Sentinel One has severe issues in handling PowerShellbased post-exploitation activities. Thus, one could easily run tools such as PowerView using just some IEX cradles.

Enabled settings
For this solution we decided to enable all the features needed using the buttons in the console to use its engines including static and behavioral AI, script, lateral movement, fileless threat detection etc. Moreover, we enabled all the features Deep Visibility provides apart from the full disk scan and data masking. We also chose to kill processes and quarantine the files.
Sentinel One has some new features that when the first tests were conducted were in test mode, meaning that they were not used and also required custom configuration to be enabled.

EXE -HTA -CPL
Notably, none of these attack vectors issued an alert to Sentinel One. With the test features enabled all three attack vectors that passed were blocked since the EDR was targeting the core of the payloads, thus, the shellcode itself. These features are now integrated in the EDR.

DLL
As soon as the folder with the MS-Teams installation touched the disk, an alert was triggered indicating that the malicious DLL was unsigned, and this could be a potential risk. As it can be observed in Figure 50, the high entropy of our DLL was detected as an IoC. The IoC was correct as our shellcode was AES encrypted. It should be noted that previous experiments with Sentinel One with low entropy files (using XOR encoding) passed the test without any issues implying that the actual issues were due to the high entropy of the DLL.

Sophos Intercept X with EDR
Sophos Intercept is one of the most well-known and trusted AVs/EDRs. It has been previously used as a test case for user-mode hook evasion 19 . The EDR version provides a complete view of the incidents and really detailed telemetry, as well as a friendly interface with insightful graphs. Some of its features can be seen Figure 51.

Enabled settings
In the case of Sophos, the configuration was simple and intuitive for the user. Therefore, we enabled all offered features, which provided protection without usability issues.

EXE
This was the only vector that worked flawlessly against this EDR. In fact, only a small highlight event was produced due to its untrusted nature because it was not signed. PPID spoofing worked, and no alerts were produced, but the activities of werfault.exe were logged by Sophos, e.g. the connection to our domain.See Figure 52

DLL
Unfortunately, the malicious DLL could not be loaded, yet the EDR produced no alert. Interestingly, the application was executed normally without the DLL in the folder. We assume that there might be some interference due to the EDR's process protection features as the payload was functioning normally.

CPL
As soon as the .cpl file was executed, an alert was produced, the process was blocked, and the attack path in Figure 53 was created. As it can be observed, detailed telemetry was produced about the system's activities.

Symantec Endpoint Protection
Symantec Endpoint Protection is a well-known solution and among the most used ones in multiple industries. It combines a highly sophisticated static detection engine with emulators. The latter considers anti-evasion techniques, addressing packed malware obfuscation techniques and detects the malware that is hidden inside even custom packers. Symantec Endpoint Protection uses a machine learning engine to determine whether a file is benign or malicious through a learning process. Symantec Security Response trains this engine to recognise malicious attributes and defines the machine learning engine's rules to make detections. Symantec leverages its cloud service to confirm the detection that the machine learning engine made. To protect endpoint devices, it launches a specially anti-malware mechanism on startup, before third-party drivers initialise, preventing the actions of malicious drivers and rootkits, through an ELAM driver 20 . The EDR is highly configurable and easy to adapt to everyday enterprise life with a powerful HIDS and network monitoring which enable it to identify and block network-based lateral movement, port scans, as well as common malware network behaviour, e.g. meterpreter's default HTTPS communication.

Enabled settings
We enabled the default features using the default levels of protection. They were enough to provide adequate protection without causing issues.

HTA
In our attacks, Symantec Endpoint Protection managed to identify and block only the HTA attack, see Figure 56. However, no alert was raised to the user.

CPL-EXE-DLL
All three attack vectors (CPL, EXE, and DLL) were successful, without the EPP identifying, blocking them or producing any alert.

Symantec Endpoint Security (SES) Complete
Compared with Symantec Endpoint Security, the Complete version, beyond being newer, it provides EDR telemetry, and more insight on what is happening in the EDR.

Enabled settings
As Symantec ES Complete is comprised of several modules there are different policies that are responsible for each module's function. The active policies and their configurations respectively were the following: • Exploit Protection -MEM Policy: We used the default policy in active mode with no extra admin-defined applications.
• Firewall Policy: There was no particular need in tuning the firewall further than the default applied protections.
• Detection and Response: Maximum telemetry was the goal with no optimization in terms of log collection and storage. We also applied extensive logging to both of the LoLbins used as a proactive measure.
• Adaptive Protection Policy: We denied all the MITRE mapped techniques including the HTTP traffic blocking of RunDLL32.
• Malware Policy: We increased the blocking level to 3 and monitoring level to 4, a common measure to increase detection efficiency and decrease False Positives and therefore noise.
• IPS Policy: We enabled the module with no whitelists.
• System Policy: We used the default policy with enabled tamper protection.
• Integrity Policy: We enabled integrity checks.

HTA
The HTA attack vector was detected and blocked statically.

CPL-DLL-EXE
While all the attack three vectors were detected and an alert was raised, SES Complete did not block them although the "RunDll32 blocking HTTP traffic" rule was set to on.

Trend Micro Apex One
Apex One is a well-known solution and ranked among the top ones on Gartner's table. Its overall features beyond the basic protection and firewall capabilities include predictive machine learning and can also be used for offline protection. The lightweight, offline model helps to protect the endpoints against unknown threats even when a functional Internet connection is not unavailable. Security Agent policies provide increased real-time protection against the latest fileless attack methods through enhanced memory scanning for suspicious process behaviours. Security Agents can terminate suspicious processes before any damage can be done. Enhanced scan features can identify and block ransomware programs that target documents that run on endpoints by identifying common behaviours and blocking processes commonly associated with ransomware programs. You can configure Security Agents to submit file objects containing previously unidentified threats to a Virtual Analyzer for further analysis. After assessing the objects, Virtual Analyzer adds the objects it determined to contain unknown threats to the Virtual Analyzer Suspicious Objects lists and distributes the lists to other Security Agents throughout the network. Finally, Behaviour Monitoring constantly monitors endpoints for unusual modifications to the operating system and installed software. According to our research, Apex One uses network, kernel callbacks, hooking; in both kernel and usermode, ETW, and AMSI to perform behavioural detection. More specifically, for ETW Apex One uses a data collector called TMSYSEVT ETW.

Enabled settings
In Apex One we leveraged as much features as possible that were presented in the policy editor such as the EDR's smart scanning method, intelliscan, scanning of compressed files, OLE object scanning, intellitrap (a feature used to combat real time compression of malware), ransomware protection (behavioural protection against ransomware, not needed for our tests), anti exploit protection, monitoring of newly encountered programs, C&C traffic filtering, and of course predictive machine learning. Finally, we configured the EDR to block all malicious behaviour.

EXE-DLL-CPL-HTA
After collaboration with Trend Micro we performed the experiments in the provided environment. Notably, all attack vectors were successful. However, there were three generic alerts with low criticality that were raised notifying that, e.g. an HTA or a CPL file were opened. The latter does not necessarily mean that there was a malicious usage.  Table 1 illustrates an aggregated overview of our findings. Evidently, from the 112 attacks that were launched, more than half of them were successful. It is rather alarming that only a handful of the EDRs managed to detect all of the attacks. More precisely, the bulk of the successful attacks, issued no alert to the endpoint solution to at least inform the corresponding team that an attacks has been launched. Even more, there are many cases of successful attacks with different kind of alerts many of which denote that the blocking capabilities of many of these solutions are more limited than expected.   Table 1: Aggregated results of the attacks for each tested solution. Notation: Highlighted row denotes acknowledged results by the vendor. : Successful attack, Successful attack, raised medium alert, •: Successful attack, raised minor alert, : Successful attack, alert was raised •:Unsuccessful attack, no alert raised, : failed attack, alerts were raised. †: In two experiments supplied by the vendor, in the first it was detected after five hours, in the second it was detected after 25 minutes.

Aggregated results
: Initial test was blocked due to file signature, second one was successful with another application. ∇: The attack was detected by the EDR but was blocked after 15 minutes. ⊕: Blocked by filetype (LOLBIN module), but the technique passed.

Tampering with Telemetry Providers
Apart from finding 'blind spots' for each EDR there is also the choice of 'blinding' them by tampering with their telemetry providers in various ways. Unhooking user-mode hooks and utilising syscalls to evade detection is the tip of the iceberg [2]. The heart of most EDRs lies in the kernel itself as they utilise mini-filter drivers to control file system operations and callbacks in general to intercept activities such as process creation and loading of modules. As attackers, once high integrity is achieved, one may effectively attack the EDRs in various ways, including patching the ETWTi functions of Defender for Endpoints and removing callbacks of the Sophos Intercept X to execute hacking tools and remain uninterrupted. Note that our goal during the following POCs was not to raise any alert in the EDR consoles, something that was successfully achieved.

Attacking Defender for Endpoints
In what follows, we present two attacks, both executed manually using WinDBG. To circumvent the Patch Guard protection mechanism, we performed all actions quickly to avoid introducing noise that could trigger the EDR. Note that the EDR was in passive mode for this test since we are only interested in silencing the produced alerts.

Manually Patching Callbacks to Load Unsigned Drivers
In this case, our process will be manually patching some of the contents of the PspLoadImageNotifyRoutine global array, which stores the addresses of all the registered callback routines for image loading. By patching the callback called SecPsLoadImageNotify, which is registered with the mssecflt.sys driver, we are essentially blinding the EDR as far as loading of drivers is concerned.
It is important to note here how the EDR detects whether the Driver Signature Enforcement (DSE) is disabled. Strangely, the alert about a possibly disabled DSE is triggered once an unsigned driver is loaded. Therefore, the MDE assumes that since an unsigned driver has been loaded, the DSE was disabled. See Figure 59. Then, after the callback is patched, we will zero-out the g CiOptions global variable whose default value is 0x6 indicating that DSE is on. Then, we load our driver using the OSR driver loader utility. Afterwards, we reset the g CiOptions variable and the patched callback to avoid a possible bug check by Patch Guard, and thus our system crashing. See Figure 60.

Manually Patching an ETWTi Function to Dump LSASS without Alerts
In this POC, we manually patch the EtwTiLogReadWriteVm function, which is responsible for the telemetry of the NtReadVirtualMemory syscall, which is called from MiniDumpWriteDump which is used by many Local Security Authority Subsystem Service (LSASS) dumping tools. We are using the Outflank-Dumpert tool [8] to dump the LSASS memory that uses direct syscalls, which may evade most common EDRs but not MDE, see Figure 61.
Find below the procedure we followed to achieve an 'undercover' LSASS dump. Note how we convert the virtual address to the physical address to execute our patch successfully. This is because this is a read-only page we want to write at, and any forced attempt to write there will result in a blue screen of death. However, we may write on the physical address without any trouble. Notably, while timeline events will most likely be produced, no alert will be triggered that will make SOCs investigate it further.

Further attacks
Further to the above and after requested by Microsoft, we performed several attacks, including tampering to MDE. We timely notified Microsoft that MDE is subject to several attacks, the bulk of which is originating from signed drivers. A typical example is the case of PowerTool (see for more details see Section 5.3) which can crash MDE without any alert or resistance 63. In essence, MDE crashed by simply right clicking on the service/process and deleting a crucial executable.
Further to our notifications, we noticed that Microsoft issued a specific ASR rule 21 without acknowledging our reports as according to their claims the original development for the patch was  initiated before our notification and the release coincided with our reports.

Attacking Sophos Intercept X
For this EDR, our approach will is quite different. We utilise a legitimate and signed driver that is vulnerable, and by exploiting it, we may access the kernel and load a custom unsigned driver. The tools we will be using are going to be TelemetrySourcerer 22 that will provide us with the unsigned driver that will actually suppress the callbacks for us, and we will communicate with it through an application that will provide us with a GUI, as well as gdrv-loader 23 that will exploit the vulnerable driver of Gigabyte and load our driver. Beyond Sophos Intercept X, TelemetrySourcerer can be 22 https://github.com/jthuraisamy/TelemetrySourcerer 23 https://github.com/alxbrn/gdrv-loader used in other EDR referred in this work, but for the sake of simplicity and clarity, we use it only for this EDR use case here. Note that the EDR was in block mode for these tests, but we managed to bypass it and completed our task without raising any alerts, see Figures 64 and 65.
Once we suppress all the callbacks by the sophosed.sys driver, the EDR cannot monitor, among others, process creations and filesystem activities. Therefore, one may easily execute arbitrary code on the tools without the EDR identifying them, e.g. one may launch Mimikatz and remain uninterrupted, clearly showing the EDR's inability to 'see' it, see Figure 66 Nevertheless, the user-mode hooks are still in place. Therefore, tools like Shellycoat of AQUARMOURY and the Unhook-BOF 24 for Cobalt Strike may remove them for a specific process or the beacon's current process, see Figure 67.

Attacking BitDefender
In this case we opted to use a "legitimate tool" to issue process termination from the kernel and successfully kill all BitDefender related processes which resulted into the product shutting down without any alert on the console. To this end, we used PowerTool 25 is a free anti-virus and rootkit utility. It offers the ability to detect, analyze, and fix various kernel structure modifications and 24 https://github.com/rsmudge/unhook-bof 25 https://code.google.com/archive/p/powertool-google/  allows a wide scope of the kernel. Using PowerTool, one can easily spot and remove malware hidden from normal software. The concept in this case was to use a defence related tool with a signed driver 26 to leverage the kernel to kill the protection mechanisms 27 . To verify the results we executed mimikatz, see Figures 68 and 69. Bear in mind that tampering with the kernel may cause some instabilities, meaning that this tool may trigger a blue screen of death situation.
As for the internal working of the driver, the technique used is rather common. It uses the ZwTerminateProcess() API to kill the process combined with several other APIs to access the process of interest. Perhaps the most important one in this case is KeStackAttachProcess(), see Figure 70, which will attach to the address space of the target process prior to terminating. It should be highlighted that similar methods have been used by APTs in the wild 28 . 26 https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-kestackattachprocess 27 http://www.rohitab.com/discuss/topic/40788-2-ways-to-terminate-a-process-from-kernel-mode/ 28 https://news.sophos.com/en-us/2021/05/11/a-defenders-view-inside-a-darkside-ransomware-attack/

Attacking FortiEDR
During our experiments we noticed a behaviour that could be leveraged to attack FortiEDR. More precisely, we noticed that while FortiEDR managed to block a malicious kernel exploit 29 , namely WindowsD 30 ,it did not do it instantly. This allowed for a window of opportunity, wide enough to disable DSE, see Figure 71. WindowsD is a 3rd party "jailbreak" so administrators can remove some intrusive defensive features introduced in modern windows versions. Currently, it can disable: • Driver signing, including WHQL-only locked systems (secureboot tablets).
• Read-only, "invulnerable" registry keys some software and even windows itself employs.
Its main purpose is to exploit a signed, legitimate but vulnerable driver in order to access the kernel level and perform the "jailbreaking" from the ring-0. In our case we will install the tool which 29 http://kat.lua.cz/posts/Some_fun_with_vintage_bugs_and_driver_signing_enforcement/#more 30 https://github.com/katlogic/WindowsD  will disable DSE and then create a service for an unsigned driver.
Although an alert was triggered and the attack was finally blocked according to the EDR report, WindowsD was successfully executed. This allowed us to disable FortiEDR by injecting into its processes from the kernel mode and intentionally causing them to become dysfunctional.Using the Kinject 31 driver we performed kernel mode shellcode injection using APCs.Then, after installing the driver and injecting a calc shellcode file to all three processes, although the processes of FortiEDR seemed to remain running they were "bricked", see Figure 72.
It should be noted that using the above method with MDE, one could inject into MsMpEng, Figure 73 and LSASS and dump it from within itself, without raising any alert or the MDE blocking

Conclusions
Throughout this work, we went through a series of attack vectors used by advanced threat actors to infiltrate organisations. Using them, we evaluated state of the art EDR solutions to assess their reactions, as well as the produced telemetry. In this context, we provided an overview for each EDR and the measures used to detect and respond to an incident. Quite alarmingly, we illustrate that no EDR can efficiently detect and prevent the four attack vectors we deployed. In fact, the DLL sideloading attack is the most successful attack as most EDRs fail to detect, let alone blocking it. Moreover, we show that one may efficiently blind the EDRs by attacking their core which lies within  their drivers at the kernel level. In future work, we plan to assess positive, false negative, false positive results produced by different EDRs to measure the noise that blue teams face in real-world scenarios. Moreover, the response time of EDRs will be measured as some EDRs may report attacks with huge delays, even if they have mitigated them. These aspects may significantly impact the work of blue teams and have not received the needed coverage in the literature.
Beyond Kaspersky's hooking solution, vendors may opt for other approaches 32 with possible stability issues. However, most vendors prefer to use cloud sandboxes for analysis as this prevents computational overhead. It should be noted that attackers may use signed drivers and hypervisors, e.g. Kaspersky's to launch their attacks and hook the kernel without issues in rootkits.
Unfortunately, no solution can provide complete security for an organisation. Despite the significant advances in cybersecurity, an organisation needs to deploy a wide array of tools to remain secure and not solely depend on one solution. Moreover, manual assessment of security logs and a holistic overview of the events are needed to prevent cyber attacks, especially APTs. Due to the nature of the latter, it is essential to stress the human factor [14,16,10], which in many cases is the weakest link in the security chain and is usually exploited to get initial access to an organisation.  Organisations must invest more in their blue teams so that they do not depend on the outputs of a single tool and learn to respond to beyond a limited set of specific threats. This will boost their capacity and raise the bar enough to prevent many threat actors from penetrating their systems. Moreover, by increasing their investments on user awareness campaigns and training regarding the modus operandi of threat actors the organisation's overall security will significantly increase. Finally, the introduction of machine learning and AI in security is expected to improve the balance in favor of the blue teams in mitigating cyber attacks as significant steps have already been made by researchers. Advanced pattern recognition and correlation algorithms are finding their way in security solutions, and EDRs in particular, detecting or even preventing many cyber attacks in their early stages, decreasing their potential impact.