Hybrid Analysis Model for Detecting Fileless Malware

Abstract

Fileless malware is a type of malware that does not rely on executable files to persist or propagate. Unlike traditional file-based malware, fileless malware is more difficult to detect and remove, posing a significant threat to organizations. This paper introduces a novel hybrid analysis model that combines static and dynamic analysis techniques to identify fileless malware. Applied to four real-world and two custom-created fileless malware samples, the proposed model demonstrated its qualitative effectiveness in uncovering complex behaviors and evasion tactics, such as obfuscated macros, process injection, registry persistence, and covert network communications, which often bypass single-method analyses. While the analysis reveals the potential for significant damage to organizational reputation, resources, and operations, the paper also outlines a set of mitigation measures that cybersecurity professionals and researchers can adopt to protect users and organizations against threats posed by fileless malware. Overall, this research offers valuable insights and a novel analysis model to better address and understand fileless malware threats.

1. Introduction

In today’s world, cyber threats are on the rise, with malware being a familiar term for many. Malware refers to any software or program that has been developed with the intention of harming or exploiting a computer system, network, Android device [1], or other digital device. It encompasses various types, including viruses, worms, Trojans, ransomware, spyware, adware, and rootkits, each employing different methods to execute malicious actions. These programs are designed to gain unauthorized access to the target system in order to steal information, disrupt system operations, or completely destroy the system. Malware can be created and transmitted to the targeted system in many ways, such as email attachments, infected software downloads, or through malicious websites, and the attacker’s intention will depend on the type of malware they use. The impact of malware, particularly ransomware, is significant, costing organizations billions of dollars and resulting in massive data theft, as highlighted by current statistics [2].

Table 1. Notable ransomware attacks and associated losses [2].

Name and Release Year of the Attack	Financial and Data Loss
Ransomware Attack at Circuit Board Maker (2025)	377 GB of SQL files and documents
Ransomware attack on Finastra (2024)	400 GB data stolen
Ransomware attack on Swisspost (2022)	1.6 TB data
Ransomware attack on Costa Rica Government (2022)	$30 million/day
Ransomware attack on Impressa (2022)	50 TB data
Ransomware attack on Kronos (2021)	Unknown
Ransomware attack on Colonial Pipeline (2021)	$4.4 million
Sodinokibi (2019)	$200 million
SamSam (2018)	$6 million
NotPetya (2017)	$10 billion

summarizes notable ransomware attacks and their resulting financial or data losses, from the 400 GB stolen in the 2024 Finastra breach to the $10 billion in damages from the NotPetya attack in 2017.

It is crucial for organizations to understand the potential consequences of having a compromised system, network, or software. Such a situation can lead to various losses, including data theft, privacy breaches, financial loss, and reputation damage. The severity of these losses varies with the malware type, e.g., ransomware can encrypt system files, rendering them inaccessible until a ransom is paid, while destructive viruses can delete important files or data altogether, causing significant data loss and disrupting daily business operations. Without proper backups or data recovery mechanisms, the results can be catastrophic for the entire organization. A notable example is the ransomware attack on the Colonial Pipeline, which halted fuel pipeline operations for several days and resulted in widespread fuel shortages, severely affecting the organization’s reputation and finances [3].

Malware that generates excessive traffic can also disrupt regular business operations. This overload can crash servers and consume network bandwidth, leading to downtime, loss of productivity, and revenue. Lastly, spyware and adware may monitor keystrokes, track network activities, and bombard users with intrusive advertisements, all of which raise security and privacy concerns.

Malware is typically classified into two types: file-based and fileless malware. File-based malware requires the host to execute a file to successfully gain access to the targeted device and carry out its malicious activities. In contrast, while fileless malware may rely on a file being executed in the traditional sense, it still necessitates some form of user interaction, such as opening a file or clicking a link embedded with the malicious code. Due to signature-based techniques, executable malware is traditional malware, which is easily detectable by antivirus software and Windows Defender. However, fileless malware poses a significant challenge for detection since it is not present on the disk but resides in memory as a process that can be detected through behavior analysis.

Static and dynamic analysis are common methods used to analyze executable malware. Static analysis involves examining the code without executing the file to identify malicious signatures, patterns, and potential vulnerabilities. In contrast, dynamic analysis involves executing the file in a controlled environment to observe its behavior. However, these methods are often not effective against fileless malware, as they operate directly in memory without relying on traditional executable formats. Traditional static analysis may miss fileless threats because there are no malicious executables to scan on disk, and purely dynamic analysis can be challenging because fileless techniques may not exhibit obvious malicious behavior until triggered in specific ways. To address these challenges, this paper proposes a hybrid analysis approach that combines static and dynamic analysis for detecting and analyzing fileless malware more efficiently. The proposed model aims to detect stealthy in-memory attacks that could evade detection by a single method.

Contributions and Plan of the Paper

This paper presents a hybrid malware analysis approach designed specifically to address the growing threat posed by fileless malware. By integrating static code analysis with dynamic behavior monitoring, this approach can detect advanced fileless malware that can evade traditional detection methods. The methodology plays a crucial role in helping organizations proactively identify complex malware, thereby improving their ability to protect their systems. It effectively uncovers obscured code and examines memory behavior to enhance security. The key contributions of this work are outlined as follows.

• This paper introduces a novel hybrid malware analysis model that combines static malware analysis with dynamic behavior monitoring, thereby enhancing the accuracy of detection and providing deeper insights into fileless malware.
• Using both real-world and newly created fileless malware samples, this research demonstrates the practical effectiveness of the proposed model, validating its effectiveness across various scenarios.
• A comparative analysis of different malware samples is presented, providing valuable insights into their similarities and differences in behavior, e.g., payload delivery, process creation, registry modification, and network communication. Additionally, some common mitigation techniques and best practices are discussed for addressing fileless malware.

The remainder of this paper is organized as follows. Section 2 presents the background study of malware, including the lifecycle of fileless malware and an overview of malware analysis techniques. Section 3 reviews related work and existing literature on fileless malware and detection methods. Section 4 outlines the proposed hybrid model’s design and methodology, including the research environment for analyzing malware, the process of malware creation, an overview of real-world and newly created fileless malware samples, and the components of the proposed model. Section 5 includes a detailed hybrid analysis, comparing selected fileless malware samples while discussing their characteristics and potential mitigation strategies. Finally, Section 6 concludes this research and provides directions for future work.

2. Background

Organizations today heavily depend on technology to operate effectively and efficiently. However, this reliance also brings about an increased risk in the form of malware. It is crucial for organizations to understand the characteristics and impacts of malware, as these malicious programs can cause severe damage to an organization’s systems, data, and overall operations.

2.1. Types of Malware

Different types of malware affect organizations in various ways, depending on their ability to steal data or perform harmful actions. The most common types of malware include viruses, worms, Trojans, ransomware, spyware, adware, and rootkits. Each type has a unique modus operandi and threat profile:

• A virus attaches itself to legitimate software or files and duplicates itself when opened. It can spread to other files or systems, potentially causing damage to data, files, and hardware with weaknesses [4,5]. For example, in 1998, thousands of users were forced to replace their computer’s motherboard after the CIH virus (also known as the Chernobyl virus) corrupted the machine’s BIOS code [6]. Beyond self-replication, viruses may carry malicious payloads, such as deleting files or stealing information. A notable example is the ILOVEYOU virus [7], which spreads through an email attachment. When the user opened the infected file, it propagated not only to the user’s contact list but also to the files on the user’s computer.
• A worm is a type of malware that can self-propagate across networks without user involvement. They reproduce by exploiting network vulnerabilities, consuming bandwidth, slowing down systems, and sometimes crashing networks. Notable examples include the Conficker [8] and SQL Slammer [9] worms.
• A Trojan (or Trojan horse) masquerades as legitimate software or files to trick users into executing it. Once executed, it can provide attackers access to the infected system, allowing them to steal sensitive data, install additional malware, or exploit the system in harmful ways. Notable examples include Zeus [10] and Emotet or backdoor Trojans [11]. These days, Trojans are often used as delivery mechanisms for other malware, e.g., attackers may use a Trojan to drop ransomware onto a system.
• Ransomware is one of the most popular and destructive types of malware. It encrypts files on the targeted machine. Then, it demands a ransom (usually in cryptocurrency) for the decryption key. This type of malware can spread via phishing emails, online links, and software downloads. It is important to note that paying the ransom does not guarantee that the victim will receive the decryption key to unlock their files; the attackers may ask for more money or vanish entirely. An example of this is the WannaCry [12] ransomware attack, which resulted in significant financial losses for both organizations and individuals.
• Spyware is designed to quietly collect information about a victim’s computer activities without their knowledge or consent. It can capture keystrokes, passwords, screenshots, or other sensitive data and transmit them back to the attacker. An example of this is FinFisher spyware [13], a sophisticated spying tool that has been used to monitor individuals and organizations.
• Adware often appears in the form of inappropriate adverts on the victim’s computer when they interact with illegal or untrusted websites or download cracked software. Adware is frequently embedded with legitimate software or obtained through shady websites, and it can cause system slowdowns, prompt inappropriate ads, and often lead to programs or processes crashing, e.g., Super-Fish [14], search bar, or pop-up adware.
• Rootkit malware modifies the operating system or uses low-level functions to hide itself and other malicious processes from detection. These rootkits can install additional malware, steal data, or provide unauthorized access to the victim’s machine to the attacker. They are often difficult to detect and remove, making them a significant threat to computer security, as seen with Fuzzbunch rootkits [15].

These malware examples can thus be broadly divided into two categories: file-based and fileless malware.

• File-based malware requires the host device to execute a file to be installed successfully on the targeted device.
• Fileless malware is a type of cyber threat that operates differently from traditional malware. Although it is referred to as fileless, it is not entirely fileless; instead, the host device does not need to execute the file to be victimized. Instead, the host device can become compromised without executing a conventional file. This type of malware often resides within macros or JavaScript code embedded in a file or link. When the user opens these documents or clicks on the links, the malware activates, putting the device at risk.

2.2. Fileless Malware Life Cycle

File-based and fileless malware differ not only in their structure but also in their lifecycle. Typical file-based malware is generated on the attacker’s device and delivered to the target through methods such as phishing emails or malicious websites, where it is stored as an executable file on the attacker’s disk. In contrast, fileless malware operates through four distinct stages: delivery, code injection, persistence, and execution. The classification of steps involved in fileless malware attack processes is illustrated in Figure 1, which explains the process of fileless malware from creation to execution. This process has three stages: the creation of malware, the attack vector, and the execution of malware on the target machine.

• In the creation of the malware stage, an intruder creates malware that can be embedded in Microsoft (MS) files, which can be PowerShell scripts, Javascript (JS) or Windows Management Instrumentation (WMI) objects, or other executables. To bypass the antivirus system or other defense systems in the target machine, the intruder uses obfuscation techniques. Intruders host the malware online to download it later through MS files to run the script directly into memory or through PowerShell or CMD and avoid detection. Furthermore, the intruder develops Visual basic (VBA) scripts inside MS files to download and execute the file on the target machine.
• In the second stage (attack vector), the malicious MS file will be sent to the target through different means, such as phishing, fake URLs and documents, or through spam emails.
• In the third stage (execution), malware is downloaded from an external source through MS file(s) and performs its intended tasks. Some intruders develop malware to hide among legitimate processes to avoid detection.

Attackers often use various techniques to gain unauthorized access to a target system. One such technique involves creating a malicious payload and uploading it to the Internet for later access via a link. Another approach is to write code in languages like VBA or JavaScript, which can download the payload during runtime and execute it using PowerShell.

Stages of Fileless Malware Lifecycle

The lifecycle of fileless malware includes different stages from creation to execution, which are as follows:

• Code Injection: This step is part of the creation of malware, which involves the mechanism of hiding the malicious function within the file, allowing it to download a malicious payload while remaining undetectable throughout the process. This can be performed using various techniques such as obfuscation, encryption, and anti-analysis methods.
• Delivery: This step is part of the attack vector and it comprises how that malicious file can be transferred to the target user, either through phishing emails or links from malicious websites. However, the primary objective is to gain access to the system’s RAM using this code. Once access is gained, the attacker can then download and execute the payload, thereby providing themselves with entry into the system.
• Execution: This stage is part of the last stage of the process in Figure 1. In this stage, the main malicious payload is executed through scripting engines like PowerShell, WScript, or Office macro functions. This step is crucial for attackers as it allows them to execute their malicious payload without being detected by traditional security measures. By utilizing trusted system processes, such as PowerShell or legitimate applications’ macro capabilities for execution, fileless malware can blend into normal activity and evade triggering classic antivirus alerts.
• Persistence: This stage is also part of the last stage of the process in Figure 1, and it is another critical aspect of this process, where the attacker wants continuous access to the system. As most malware operates in the RAM and is eliminated when the system is restarted, attackers often opt to store their malicious software in the registry. This allows them to regain access to the system even after it has been rebooted, ensuring persistence in their infiltration efforts.

2.3. Malware Analysis

Malware analysis is the process of investigating the properties and behavior of suspicious software to understand its functionality and potential impact. There are multiple approaches to malware analysis, each providing unique insights into the malware’s operation. In the literature, these methods are categorized into the following types: Static, Dynamic, and Hybrid.

2.3.1. Static Analysis

Static analysis is a rule-based detection technique that follows a set of rules and is performed in a controlled environment without running the malware. It examines the functions graph, Portable Executable (PE) headers, Application Programming Interface (API) calls, opcodes, and code analysis [16] to determine whether a file is malicious or not. Based on these features, it is assumed that this method works with signature-based techniques and requires fewer computational resources. This method is considered a faster approach to identifying malware compared to other methods because it does not require the file to be executed.

However, static analysis has important limitations. It often requires significant expertise, as analysts must interpret code or disassembled instructions to infer behavior. Moreover, attackers commonly employ techniques like code obfuscation and encryption to thwart static analysis [17]. Attackers can use various code obfuscation techniques to alter the code signatures and syntax of the original code, thereby performing similar tasks, and encrypt the strings to make them harder to detect. Additionally, attackers can use packaging and dead code insertion techniques to bypass antiviruses and Windows Defender.

Despite these challenges, static analysis remains a critical early detection tool. To detect and mitigate such malware, system administrators use a variety of specialized tools for static analysis: disassemblers (e.g., Ghidra, IDA Pro, Radare2) help break down binaries into readable code, and PE analysis tools (e.g., PEiD, PEview, PEStudio) examine the structure of executables for anomalies. Online scanners like VirusTotal are also used to cross-check files against a wide array of antivirus engines. These tools help in effective detection, mitigation, and designing response strategies in static analysis.

2.3.2. Dynamic Analysis

Dynamic analysis is a technique used to study the behavior of malware. It involves executing the malware in a controlled environment and monitoring its activities with various tools. The aim is to identify the processes and actions performed by the malware. During the analysis, after executing the file, all activities are monitored to detect abnormal operations in the systems, such as connections to external networks, CRUD (Create, Read, Update, Delete) operations on file directories, and modifications to the registry. Tools like ProcMon (Process Monitor) can capture real-time events, such as file and registry operations, while network analyzers like Wireshark or Microsoft Network Monitor can record outbound traffic for signs of data exfiltration or malicious communications. Other utilities, such as Any.Run (an interactive sandbox), Cuckoo Sandbox, Process Hacker 2, RegShot, and Redline, provide additional visibility into process activity, memory usage, and system changes during execution.

Dynamic analysis requires more computational power and resources than static analysis. It involves more comprehensive resources, experts, the use of virtualization, various detection tools, constant activity monitoring, and specialized tools. According to the authors of [18], dynamic analysis outperforms static analysis in terms of detection accuracy. However, its high computational requirements can make it less accessible for some organizations.

2.3.3. Hybrid Analysis

This model is a combination of static and dynamic analysis. The hybrid approach first performs static analysis on a file to flag any immediately suspicious features. If the static analysis indicates potential malicious intent but cannot conclusively identify the threat, the analysis proceeds to a dynamic phase. During dynamic monitoring, the file is executed in a sandbox environment to observe its behavior and confirm any malicious actions.

The hybrid model is generally more effective than relying on static or dynamic analysis alone, especially against modern malware that uses evasion techniques. Some malware may bypass static analysis by using obfuscation techniques or drop-in malware files that are only activated when a user opens a file, such as in the case of fileless malware. Conversely, malware might lie dormant or perform checks to detect sandbox environments, thereby evading pure dynamic analysis. As a result, the hybrid model is generally considered more effective than single analysis models for gaining a comprehensive understanding of malware and its behavior.

The hybrid model uses tools similar to those used in both types of analysis, e.g., OLEVBA (a static analysis tool) to extract and inspect macro code in Office documents, Any.Run to run the malware and observe high-level behaviors. However, it is advisable to select tools based on the analysts’ experience and the organization’s budget, rather than using all tools from both analyses.

3. Related Work

Fileless malware operates by injecting malicious code into the memory of a target system, rather than relying on a physical executable file. This makes traditional detection techniques, such as opcode and PE header analysis, ineffective, as these methods only examine file structures rather than the memory space where fileless malware operates. In short, fileless malware does not reside in file headers, making it much harder to identify and mitigate [19]. The authors performed memory forensics to detect fileless malware, which requires extensive knowledge of memory, registry, and processes to understand the proper behavior of malware and detect it. The fileless malware was delivered to the target by some means, for example, documents that downloaded the malware later on. To address this challenge, the proposed hybrid model analyses the file before execution using various tools to detect signatures indicative of external file downloads, enhancing our ability to identify potential threats before they can execute.

The authors of [20] classified fileless malware into two categories: script-based and RAM-resident-based fileless malware. The research suggests that RAM-resident malware is particularly challenging for antivirus software to detect, as it scans the hard disk for malicious activity and checks signature files, rather than examining memory. It is often assumed that, if a process is already running on the system, it is not considered suspicious. In contrast, script-based malware is the most popular method, due to its automation capabilities. This type of malware takes advantage of built-in functionalities in Microsoft Office, Windows applications, and Windows PowerShell, which are primarily designed for automation. Attackers use these functionalities to download their malicious scripts and directly execute them, bypassing the scanning process.

The author of [21] described a fileless attack against a bank by the Lurk Trojan. This attack allowed the hackers to gain unauthorized access to sensitive user data and compromise online banking services without the users’ knowledge or consent. The attackers used two methods to infect the bank’s servers and systems. The first method involved two commands, “regsrv32” and “netsh add helper dll”. The second method involved using the ShellIconOverlayIdentifiers branch of the system registry.

According to the author of [22], fileless malware was launched through Microsoft Office. This malware launches a malicious PowerShell script through JavaScript code, which is executed directly in legitimate memory via JavaScript code, without creating any files or folders on the disk. Once installed, the malware lodges itself directly into the registry and remains there as long as the computer runs. Upon restarting, it reloads itself from the registry into memory, making detection by antivirus software and Windows Defender particularly difficult.

The authors of [23] analyzed fileless malware and recommended security mechanisms to defend against it. The authors analyzed malware, such as Poweliks, which remains persistent in memory and is executed in base64 encoded format. In addition, they also carried out a detailed analysis of Gootkit, Phasebot, and Emotet malware, observing their lifecycle, infection technique, and persistence.

The author of [24] analyzed a phase bot, which is a type of fileless rootkit that gains unauthorized access to the victim’s system and steals its information using a form-grabbing approach. The phase bot was designed to move its code around automatically and remained encrypted within the system’s registry, making it much harder to detect and remove.

In a study conducted by Sudhakar [25], various types of fileless malware were analyzed, including Kovter, Powerware, Lurk Trojan, Phasebot, Poweliks, Code Red, and SQL Slammer. The author explained how these types of malware remain hidden within the system registry and memory, and also provided techniques to counteract their attacks. In another study, by Lee et al. [26], an in-depth analysis of Poweliks and Kovter was conducted, detailing the methods these types of malware use to conceal themselves in the memory and registry. The study also provided techniques for identifying fileless malware through memory analysis methods.

The author of [19] conducted an analysis of fileless malware using memory forensics techniques. This analytical approach focused on the attack strategies and characteristics of the chosen sample, which simplifies feature extraction and decreases the processing load. The author used only a memory forensics approach rather than a static one, as the latter is resource-intensive and requires experts to analyze the registries carefully.

Graeber [27] identified Windows-based fileless malware by using Windows Management Instrumentation and PowerShell. PowerShell scripts were used to automate different processes and retrieve data from file systems and registries. The author developed a unique detection method to detect and remove Windows-based fileless malware. However, his experiments showed that achieving 100% accuracy in detecting fileless malware is impossible, as it can quickly spread and cause significant damage.

The authors of [28] focused on analyzing and classifying PE files to distinguish between malicious and benign files. This classification was accomplished by mining the behavioral aspects of API calls within these files. They proposed a feature selection algorithm that identifies unique APIs among the API calls found in PE files. Similarly, in another study conducted by Sami et al. [29], a framework was developed to accomplish the same goal of classifying PE files as benign or potentially dangerous based on API call mining. The framework comprised four main components: a PE analyzer, a feature generator, a feature selector, and a classifier. The framework’s PE analyzer extracted Windows API calls from the PE files, which represent the interactions between the executable file and the Windows operating system. These sets of API calls were then utilized by the feature generator to create various features, including information about the frequency, order, and structure of these API calls for each PE file. The feature selection method was then employed to select the most significant and differentiating features from the pool of generated features. This step aimed to identify the most critical aspects of API call behavior that could determine whether a PE file is malicious or safe. Finally, the classifier algorithm was used to classify the PE files as either benign or harmful based on the selected features and their associated attributes.

Recent research has increasingly explored the application of machine learning (ML) techniques to detect fileless malware, using behavioral and memory-based features. One such approach proposed an ML-based detection technique for fileless malware focusing on feature analysis [30]. They extracted the features from a dataset of malicious and non-malicious malware samples by executing them on a virtual machine and using memory dump. Using k-fold cross-validation, the study evaluated several models and found Random Forest to be the most effective, achieving 93.33% accuracy. However, this research totally focused on memory analysis using memory dump and feature selection, which is a resource-intensive and time-consuming task. It also requires in-depth knowledge of memory analysis and expertise in ML.

Another study proposed a hybrid model using static and dynamic API calls to eliminate the noise of both techniques [31]. This model was a two-stage learning approach where the XGBoost algorithm acted as a feature extractor feeding into an Artificial Neural Network (ANN). It also used a similarity-based feature to detect obfuscated malware. This approach was evaluated using seven ML techniques with 10 K-fold cross-validation and resulted in higher accuracy results compared to the related work. However, this technique relied solely on API calls for detections, limiting its ability to analyze deeper behavioral indicators.

Another study proposed a model for detecting fileless malware by analyzing memory-resident fragments of PE files using a one-dimensional convolutional neural network (1CNN) [32]. The authors created a custom dataset by executing benign and malicious samples in virtual machines and extracting memory dumps at regular intervals. By experimenting with different fragment sizes and positions, they demonstrated that 4096-byte fragments from the head of PE files resulted in the highest accuracy (97.48%) in classifying malware. However, the approach had limitations, including reliance on specific fragment lengths and positions, which may not generalize well against heavily obfuscated or relocated malware code.

In [33], the authors proposed a forensic framework that analyzed memory dumps to detect and investigate fileless malware using static and dynamic inspection techniques. It was evaluated on 10 real-world malware samples, achieving up to 90% success in unpacking malicious code, 80% in extracting command-and-control configurations, and 70% in recovering injected code. However, the system was limited by high computational overhead, making it more suitable for post-incident analysis than real-time detection.

In summary, the existing fileless malware analysis techniques all have limitations, e.g., existing static analysis techniques are unable to detect fileless malware, which is downloaded and executed directly in memory. On the other hand, dynamic analysis techniques require extensive knowledge of the registry, memory, and malware behaviors. The ML-based models often do not explain how the malware behaves. Additionally, there is a risk of false positives if the malicious code does not execute for a sufficient duration or is not captured during the precise detection window. These limitations signify the need for a hybrid model that offers a comprehensive and systematic methodology for hybrid fileless malware analysis. In response to this need, the proposed model outlines a sequential and comprehensive workflow that integrates a suite of specific off-the-shelf tools. This methodology is designed for in-depth, manual, and semi-automated analysis of individual fileless malware samples. The novelty of the proposed approach lies in its ability to complement broader, often ML-driven detection frameworks by improving the balance between detection accuracy and understanding, and aiding in more effective threat response.

4. Proposed Hybrid Model

This section presents the design and methodology of the proposed hybrid malware analysis model. It includes the research environment for analyzing malware, an overview of real-world and newly created fileless malware samples, and the design of a fileless malware analysis model.

4.1. Design of the Hybrid Model

This section presents the generic design of the proposed hybrid model for analyzing real-world and newly created malware samples. Figure 2 illustrates the workflow of the analysis process. In the initial static phase, the file is scanned with specialized tools to extract any embedded scripts or macros and to compare the file against known malware signatures. If the static analysis indicates suspicious content but not a definitive result, the model proceeds to the dynamic phase, where the file is executed in a controlled environment for close observation. In the implementation of the static analysis phase, the OLEVBA tool (version 0.54) is used to extract Visual Basic macro code from Microsoft Office-based files, including XLS, Word, and PowerPoint documents. This helps identify any hidden or obfuscated script that could trigger malicious activity. Additionally, VirusTotal is used to scan static properties of files and compare them with the existing dataset of malware. If either OLEVBA or VirusTotal reveals clear indicators of malware, the model can already raise an alert about the file.

If the static analysis is inconclusive, it is moved to dynamic analysis. In this stage, a file is run within a virtual machine sandbox to monitor its behavior. An online sandbox service, any.run, is utilized initially, as it conveniently visualizes process creation and network calls triggered by the file. A more detailed manual dynamic analysis is performed using other tools, which are detailed in Section 4.4.

By integrating these steps, the proposed hybrid model ensures that, even if a malware sample evades detection in one phase, it can be caught in another. Static analysis might catch a known malicious macro or suspicious strings before the malware runs. If static analysis misses it, dynamic analysis might observe the malware’s malicious behavior (like connecting to a strange domain) and flag it. Together, the two phases provide defense in depth in the analysis process.

4.2. Environment

All malware creation and analysis activities were conducted exclusively within isolated virtual machine environments. Specifically, VMWARE workstation 16 PRO was used to create three virtual machines: One Kali Linux for development and performing static analysis, one Windows 11 Home Edition for VBA coding in Word documents, and a separate Windows 11 Home Edition as the target for the malware and performing dynamic analysis. These virtual environments were completely disconnected from any internal or public networks, except for controlled and monitored outbound connections for malware command and control simulations, which were strictly contained within the virtual network. This setup effectively prevented any unintended propagation or compromise of external systems. Additionally, no human subjects were involved, and no real-world systems were impacted during the process, thus adhering to responsible research conduct.

4.3. Tools Used for Developing Malware

In order to generate custom fileless malware samples for this work, namely Sample 5 and Sample 6, the following tools were used:

• NGROK: This is a tool that allows interaction between devices connected to the Internet across different networks and local networks.
• MSFVENOM: This is a Linux-based tool used to generate payloads for penetration testing. In this research, it is utilized to create a PowerShell-based payload that can be embedded in other files or executed separately.
• MSFConsole: This is used to create a listener for the payload and interact with the victim’s machine to perform other activities.
• VBA Macro Word: Visual Basic coding is embedded in macros of Word documents provided by Microsoft. It downloads scripts from online hosting and executes them in memory using PowerShell or the command line.

These tools and techniques mirror what sophisticated attackers do, thereby providing realistic samples for testing the proposed hybrid model.

4.4. Tools Used for Malware Analysis

To analyze both the real-world and custom malware samples, the following tools are used:

• Olevba: This is a Python-based utility of oletools used to analyze Visual Basic scripts embedded in macro-enabled Microsoft Office documents for any suspicious activities. It enables security analysts to extract and examine VBA code from Office files and assess the potential security risks associated with malicious functions present in the script.
• Hashrat: This is a tool used to generate a hash of the file. It supports multiple hash formats, including MD5, SHA1, and SHA2, along with multiple file extensions, which enhances its adaptability.
• VirusTotal: This is an online tool that scans and analyzes files for viruses and suspicious activities using pre-defined signatures provided by various antivirus engines and security tools. It maintains a file reputation database that shows historical scan results and detection rates for specific files, helping users understand any identified malicious activities.
• Xarchiver: This tool is designed to split the packages of a file into multiple components. This research uses the tool to discover embedded templates within Word documents and identify any abnormalities. It is available as an offline command-line tool, as well as a graphical user interface (GUI). For this research, the GUI version of the tool is used to extract the embedded packages from Word documents.
• Any.Run: This is an online tool to capture process activity and network traffic flow of files, helping to detect suspicious activities. It offers features that allow users to analyze potentially malicious files in a controlled environment, examine their behavior, and assess associated security risks. The tool enables testing of files on various platforms and architectures, although not all features are available in the free version. In the free version, file execution is limited to Windows 7 32-bit architecture. However, users still have access to features such as monitoring network activities and interacting with access control. Additionally, functions like registry modifications, process creation, and execution are available to all users.
• ProcMon: This provides real-time monitoring of Windows system operations and activities by capturing information related to processes, file system activities, the registry, and network activities. It enables system administrators to focus on specific processes, operations, or types of activities, enabling them to identify and analyze particular issues or behaviors. By applying filters, administrators can extract detailed logging information and view it in a hierarchical tree structure that shows these processes and their child processes.
• Process Hacker 2: This is an open-source Windows task manager and system monitoring tool that enables users to view and manage running processes in real time, providing detailed information, such as CPU and memory usage, thread details, loaded modules, and network connections.
• Microsoft Network Monitor: This is a network protocol analyzer and packet sniffer tool that enables users to capture and analyze network traffic in real time. Additionally, it offers a process tree view, allowing users to manage and troubleshoot traffic related to specific processes or applications.
• RegShot: This is a registry comparison tool that can be used to identify changes made to the Windows Registry before and after executing a malware sample. This tool is invaluable for understanding how malware impacts system settings.

The analysis was conducted using a combination of static and dynamic tools to thoroughly examine each malware sample. A key feature of our proposed approach is its modular design, which eliminates dependence on any single tool. This flexibility allows the model to adapt to changes in the cybersecurity landscape. For example, Any.run can be replaced with another sandbox, like Cuckoo, and OLEVBA can be substituted with a different VBA parser, such as ViperMonkey. Since OLEVBA is a public tool, specific versions can be accessed via GitHub to ensure reproducibility (in this research OLEVBA version 0.54 is used), while Any.run is a cloud-based service that maintains a consistent environment.

4.5. Real-World Malware Samples

In this study, four samples of fileless malware belonging to the Trojan family were downloaded from the public malware repositories MalwareBazaar [34] and GitHub [35]. For the purpose of this research, these samples are referred to as Sample 1, Sample 2, Sample 3, and Sample 4. Each of these malware samples is a macro-enabled Microsoft Word document (.doc) known to exhibit fileless behavior. They contain macros that download and execute PowerShell scripts or drop additional payloads into memory during their execution.

Each of the four samples was selected to represent different sub-variants of fileless techniques, e.g., one sample uses a macro to run PowerShell directly, another uses embedded shellcode, etc. Using these real-world samples allowed us to validate the proposed hybrid analysis model against known fileless attack patterns.

4.6. Newly Created Malware Samples

In addition to the real-world samples, two different types of fileless malware were developed to simulate novel or less common attack strategies. These are referred to as Sample 5 and Sample 6. These samples were created exclusively for academic research purposes. The aim was to understand the operational behaviors of fileless malware, test the proposed hybrid analysis methodology, and contribute to the development of more effective detection and mitigation strategies. There was no malicious intent, nor was there any aim to cause harm outside of the controlled research environment.

• Sample 5: The first is rootkit malware that allows attackers to gain unauthorized access to the targeted system. This malware was created using MSFVENOM tool, which generates a PowerShell-based payload that can be embedded in other files or executed separately. NGROK is used to gain access because it allows interaction between Internet-connected devices across different networks. MSFConsole was used to create a listener for the payload and to interact with the victim’s system. The payload was hosted on an online hosting service called Hostinger and was downloaded through VBA Macro code. Visual Basic coding is embedded into macros of Microsoft Word documents, enabling the script to be downloaded from an online hosting service and executed in memory through PowerShell. Figure 3 illustrates the creation process for this sample, including how the macro fetches the script. Additionally, Figure 4 shows a snippet of the malicious VBA macro code embedded in the Word document for Sample 5, which initiates the download of the script onto the target machine.
• Sample 6: The second is ransomware malware that encrypts files on the targeted system. This malware was created using a Python (version 3.13.2) script developed in Visual Studio Code. This script uses cryptography libraries to encrypt files on the victim’s computer with a specific encryption key. This script was then hosted online (again using Hostinger for external accessibility). For delivery, similar to Sample 5, a VBA macro is embedded in a Word document that, upon enabling, downloads the Python script from the remote host. In this case, the macro executes the Python script via CMD. Figure 5 and Figure 6 illustrate snippets from the generated scripts for Sample 6.

5. Results and Discussion

This section provides an analysis of both real-world and newly created fileless malware samples. These malware are not completely fileless; they each demonstrate the core idea of fileless malware: using a legitimate doc file to download scripts or execute malicious code in memory. Additionally, we will discuss a comparative analysis of selected fileless malware samples in terms of their classification, files dropped, processes created, registry changes and network communication.

5.1. Analysis of Sample 1

This malware differs from traditional malware that uses macros to download and execute a malicious program in PowerShell through Visual Basic. Instead, it uses a macros template method that downloads the malicious program directly, making it undetectable by macro detection tools or Visual Basic detectors. OLEVBA was used to scan Sample 1 for VBA macros. The tool revealed that no Visual Basic scripts were present in the document, as illustrated in Figure 7a. However, suspicion of template injection suggested that the lack of macros indicated the malware might be hidden elsewhere. The file was then submitted to VirusTotal, which underwent scanning and was flagged as a malicious file by 22 antivirus software vendors, as illustrated in Figure 7b. This indicates that the file is capable of executing activities with malicious intent, which are not encoded in Visual Basic macros, as evidenced by their undetection by Olevba.

The macro template method used by Sample 1 essentially works by combining the Word document with additional hidden files (such as .rels relationship files or settings XML files) that reference external content. These files can be accessed through the Xarchiver tool to inspect the internal structure of the Word document. The tool found embedded relationships that pointed to an external URL, as illustrated in Figure 7c. Please note that the figure indicates that the file was communicating with a non-secure external website.

By performing static analysis, it becomes evident that the file is not legitimate. There were no macros, but the presence of an external template reference (especially to a suspicious domain) is a red flag. In order to better understand the behavior of this malware and any associated activities, dynamic analysis is conducted. For this purpose, the malware is uploaded to Any.Run. The results, displayed in Figure 7d, suggest that the UDP connections were attempting to connect to multiple IP addresses. This is suspicious, as a Word document usually does not require connections to other IPs. The fact that Sample 1 was trying to contact external servers indicated it was attempting to download something malicious.

To investigate the network indicators, the reputation of one of the IP addresses was checked through “AbuseIPDB”, a community-driven online tool that allows users to report and review abusive IP addresses. The tool revealed that this particular IP address has a poor reputation. It has been flagged as a red flag by the majority of community members, as illustrated in Figure 7e.

Interestingly, Any.Run’s automated analysis did not explicitly label the document as malicious, probably because the external server hosting the malicious template was offline or unreachable at the time of analysis. When the document tried to download the payload and failed (due to the server being down), it did not produce any immediate harmful effects or error messages; it simply continued to function like a normal document. This scenario highlights a limitation of dynamic analysis: if malware’s key actions cannot occur (e.g., the command server is offline), the analysis might not catch it.

To gather more evidence at the network level, the Microsoft Network Monitor tool is used to identify any unfamiliar or suspicious communications. This tool allowed us to detect network activity related to the document while filtering out any unrelated traffic. It was observed that the document was repeatedly attempting to communicate with the targeted website (as shown in Figure 7f), but it received no response from the server. This behavior is abnormal and could potentially pose a greater risk than it currently appears to. Given all these findings, we have classified this file as malicious. Although the sandbox did not flag it as suspicious due to the missing payload, the hybrid analysis model allowed us to confidently classify the file as malware.

5.2. Analysis of Sample 2

Sample 2 is another malicious Word document but, unlike Sample 1, it follows a more typical macro malware pattern. It contains an actual macro that, when enabled, attempts to download a malicious program from a third-party online service as a string and execute it via PowerShell and Visual Basic. It employs an obfuscation within the macro to conceal its true intent, potentially helping it evade simple detection.

For static analysis, the file is submitted to Olevba to identify any VBA scripts within the document. The analysis revealed a lengthy script containing encrypted and obfuscated strings, which is considered a red flag, as illustrated in Figure 8a. However, it is not 100% guaranteed that the file is malicious; it is merely marked as suspicious based on the syntax and function names present in the code, as shown in Figure 8b. Next, the file was uploaded to VirusTotal, where 42 vendors flagged it as malicious, as illustrated in Figure 8c. Such a high detection count is a clear confirmation that this file is known malware or contains known malicious patterns. To understand how the malware behaves when executed, dynamic analysis is conducted. First, the malware is uploaded to the online sandbox, Any.Run. On Any.Run, once macros were enabled, the malicious behavior became evident. The Word process for Sample 2 launched a series of suspicious actions: it created a cmd.exe process that subsequently initiated other processes and activities, as illustrated in Figure 8d. There were also external communications triggered via the command line. However, while Any.Run showed that something bad was happening (processes and possibly network pings), it did not provide granular details on what those processes were doing internally.

To explore detailed process activities, the document was then opened in a controlled environment. By clicking “enable content” (which enables macros), a CMD process was initiated, which started performing numerous operations on the system, such as creating and deleting files in various directories (C:\Windows\, system folders, and user directories) within seconds, as shown in Figure 8e. This kind of rapid file activity is not normal for Word or CMD unless an installation or script is running. Shortly after, the CMD process terminated itself, and a new process named “searchprotocolhost.exe” (this is actually a legitimate Windows process related to indexing) started. The searchprotocolhost.exe process then began its own set of activities, such as creating and deleting various files within the same folders and attempting to override access policies, as well as tampering with .dlls and other files. All the while, the malware was collecting information from the system (likely user data or system credentials) and packaging it. The logs indicated that the malicious process eventually transmitted this data out to the attacker, as illustrated in Figure 8f.

In summary, Sample 2 is malware that infiltrates the system, gathers data, and attempts to exfiltrate it. Static analysis confirmed the file’s malicious nature, while dynamic analysis provided further details on its harmful processes and impacts on the file system and network.

5.3. Analysis of Sample 3

Sample 3 is another macro-enabled Word file that exhibits classic fileless malware traits. It uses a macro to download additional malware components via the document and relies on both obfuscation and process injection techniques to carry out its payload.

Running Olevba on Sample 3 revealed that the document’s macro contains a significant amount of obfuscated code embedded with lengthy strings and an encrypted format, as shown in Figure 9a. This suggests that the macro likely contains an encoded payload or uses string obfuscation to hide URLs and commands. Despite the obfuscation, due to the presence of suspicious terms used in the script, such as SHELL, Olevba marked the file as a suspicious file, as shown in Figure 9b. The Olevba code analysis indicated that, although the macros’ content could not be easily read, their structure and keywords align with those of known malware-laden macros. Next, Sample 3 was scanned with VirusTotal. It was flagged by 38 antivirus vendors as malicious, as seen in Figure 9c. This indicates that either the macro’s signature or the patterns of its obfuscation are recognized by many security engines. When Sample 3 was executed in a sandbox environment (Any.Run), the system immediately marked it as malicious. The Word document, upon enabling the macro, began creating multiple processes, and these processes attempted to establish external communication. Specifically, Any.Run noted that the Word process launched a cmd.exe process, which in turn launched two noticeable child processes: msg.exe and PowerShell.exe, as illustrated in Figure 9d. The presence of a PowerShell process triggered from a Word macro is a significant red flag; this is a common technique for fileless malware, as PowerShell can directly execute code from memory or fetch scripts from the web. Typically, msg.exe is a legitimate Windows utility used for sending messages over a local network. However, if it appears in an unusual context, it may indicate that malware is either exploiting this utility for malicious purposes or using its name as a disguise for its own code. Meanwhile, the PowerShell.exe process likely carried out the core payload execution, as PowerShell can run scripts that perform various tasks, including downloading additional payloads and modifying system configurations. During the Any.Run analysis, it was observed that these processes (especially PowerShell.exe) were interacting with external websites that were not considered secure.

For detailed process and traffic analysis, this file was executed in a local virtual controlled environment designed to capture such activities. Monitoring the network activity with Microsoft Network Monitor showed no signs of suspicious activity, as all traffic was directed to the official website of Microsoft templates, as illustrated in Figure 9e. However, upon enabling the macro content in a Word file, the Process Hacker tool showed the creation of a cmd.exe process, followed by the immediate creation of a PowerShell.exe process. These two processes were created simultaneously, as shown in Figure 9f.

With ProcMon, we looked at what the cmd.exe process was doing once it started. The logs indicated that a CMD process executed various activities through the CMD kernel, including interactions with the system, Windows, and public folders. New files were being created and deleted within mere seconds, which raised suspicions, as illustrated in Figure 10a. Additionally, the CMD process created a new process, msg.exe, which then began to disseminate malware to other connected systems, as shown in Figure 10b. This mirrored what we observed in Sample 2: an indicator of either a script or commands that quickly drop payloads, execute them, and then clean up (delete them) to avoid detection. It is important to note that this activity was conducted in a virtual environment, and due to resource limitations, all observations could only be tracked through logs shown in Figure 10c.

After msg.exe activities finished, it disappeared from the file system, and a new process, PowerShell.exe, began a new set of actions (see Figure 10d). This process interacted with various system files and even sensitive areas of the OS (like the session manager, system32 directory, drivers, etc.), as shown in Figure 10d,e. The PowerShell script aimed to steal session information and tamper with system components, indicating an attempt to escalate privileges and maintain persistence within the OS. PowerShell was also observed modifying the system’s security certificates and root trust settings, adding its own certificates or signatures, as shown in Figure 10e. By doing so, the malware could mark its malicious components as trusted or intercept secure traffic. The malware, through PowerShell, modified the registry to ensure that it could automatically trigger itself at system startup (another persistence technique). Network activity monitoring reveals that the PowerShell process established connections to external domains, giftcard2naira.com and online-theorie.driveddy.com. Both of these are highly likely to be malicious or compromised domains used by the attackers. Communicating with such domains could mean the malware is exfiltrating data or checking in with a control server, as shown in Figure 10f.

Our proposed approach revealed issues with the macro through static analysis, while dynamic analysis provided the exact sequence of the malware’s attempts to take over the system.

5.4. Analysis of Sample 4

Sample 4 is another type of macro-based fileless malware that disguises its malicious payload by using a filename resembling a well-known domain, specifically “microsoft.com”. It seems to employ a strategy similar to that of Sample 3, but it features different specifics regarding the processes it executes and the network contacts it establishes.

Olevba detection on Sample 4 indicated a macro that is heavily obfuscated with extremely long encrypted strings and even chunks of HTML code embedded in it, as shown in Figure 11a,b. The presence of HTML in the macro suggests that the macro might be constructing some script or data dynamically. Olevba flagged the entire obfuscated VBA script as suspicious due to the obfuscation, as shown in Figure 11c. The entire macro was difficult to read, suggesting possible malicious intent. When scanned with VirusTotal, Sample 4 was identified as malicious by 43 different antivirus vendors, as shown in Figure 11d. This high detection count reinforced that the file’s static signature or behavior is well-known among security circles as malicious.

In Any.Run, as soon as Sample 4 (with macros enabled) was executed, the service marked it as malicious. This is because the Word document downloaded and ran an executable named “microsoft.com” (with an “.exe” extension hidden behind what looks like a domain name). This fake “microsoft.com” process then began communicating externally. Two key suspicious activities were observed: The first suspicious activity involved the initiation of the “microsoft.com” exe file, and the second suspicious activity was its attempt to access an external domain name (m4tz0of0xi8o3brr.com) (see Figure 11e). This indicates the malware is likely trying to either fetch further instructions or exfiltrate data as part of its routine.

By monitoring the network activity of the Word document in Microsoft Network Monitor, no suspicious activity was detected, as shown in Figure 11f. This was likely because, similar to earlier samples, the malware was attempting to blend in by contacting a domain that appears related to Microsoft (or possibly it had not yet initiated malicious communications within the captured timeframe). We then observed the “microsoft.com” process running in the Process Hacker tool. It indicated that a process named “microsoft.com” was created simultaneously, which raised suspicion (see Figure 12a). It was observed that it had no visible window (running in the background), and it was started by the Word process’s macro. At the same time, this “microsoft.com” process was also identified in Microsoft Network Monitor. It was trying to send multiple requests to m4tz0of0xi8o3brr.com, which was querying a PHP page to download content, as illustrated in Figure 12b.

When the “Enable Content” option (which activates macros) was clicked in the Word document, logs generated in Process Monitor indicated that the “microsoft.com” process was created. That process then executed a number of operations via the CMD: interacting multiple times with system directories, such as System32. New files were created and then quickly removed. This behavior is similar to that observed in Samples 2 and 3, suggesting that the malware’s payload executes a series of commands or scripts to set itself up or propagate, while also erasing any traces of its activity (see Figure 12c). The external requests to the malicious domain were also captured in the logs; after making those requests, the “microsoft.com” process closed its network connections. This indicates an attempt at stealth: performing necessary online actions, such as downloading an encryption key or sending stolen information, and then disconnecting to minimize the risk of detection (see Figure 12d). However, it was observed that, even after closing its outward connections, the “microsoft.com” process did not terminate. The process continued to run in the background (see Figure 12e), likely monitoring the system or waiting for triggers (like persistence until system shutdown). It might have been set to monitor for certain conditions or simply keep a backdoor open, waiting for further commands. It was noted that the process did not terminate until the computer was rebooted or shut down, which is typical of malware that wants to maintain its presence.

This behavior demonstrates a multi-stage attack that starts with legitimate-looking network activity and is followed by a covert, persistent process for data exfiltration or additional malicious actions. This emphasizes the essential importance of thorough process monitoring in hybrid analysis.

5.5. Analysis of Sample 5

Sample 5 is newly created fileless malware embedded in a Word document. It serves as a backdoor Trojan with rootkit-like behavior, delivered via a Word macro. The purpose of Sample 5 was to create a fileless scenario in which a malicious script is delivered and executed entirely in memory, establishing a reverse TCP connection to an attacker. In this sample, a macro runs a PowerShell command to download a malicious script from the Internet and execute it, thereby not writing the main malicious program to disk.

Sample 5’s macro and payload were designed in such a way that typical antivirus software at the time of execution might not detect them as malicious. When we analyzed the file on VirusTotal during development, it was observed that no antivirus vendors flagged it as harmful. Figure 13b shows that, at the time of analysis, zero security vendors detected Sample 5. This is because it was newly developed malware that had not been shared. This situation highlights the potential dangers of such fileless techniques, as signature-based tools often fail to detect them initially.

Using Olevba on Sample 5, Visual Basic code was found in the file. Olevba flagged it as suspicious primarily because of the presence of keywords like “PowerShell” and “ShellExecute” in the macro, which are not common in benign documents, as illustrated in Figure 13a. The static analysis thus indicated that the macro could launch a system command. For dynamic analysis, the file is first uploaded to Any.Run, which detected the Word document creating a PowerShell process and making external network connections, confirming suspicious activity (see Figure 13c). While Any.Run provided an overview of the processes, it lacked detailed information on specific process activities and traffic flow.

To obtain this, the malware was executed in a controlled local environment and monitored network activities using Microsoft Network Monitor. A process named “PowerShell.exe” was identified as communicating with the same website as Any.Run (see Figure 13d). Additionally, by monitoring the log activities in the ProcMon, it was found that “PowerShell.exe” attempted to manipulate the sessions and files in the File Explorer to establish a TCP reverse connection, as shown in Figure 13e. Following the execution of the “PowerShell.exe” process, Microsoft Network Monitor observed other processes establishing connections with the NGrok IP address (see Figure 13f), confirming the successful reverse shell. Thus, the malware succeeded; it connected back to the attacker, which in a real scenario would allow the attacker to run commands on the victim’s system remotely.

Given that no antivirus detected it and it functioned as intended, Sample 5’s analysis highlights the effectiveness of custom fileless attacks. Without behavioral monitoring, such an attack might bypass initial defenses. However, the proposed hybrid approach was effective: statically, it was identified as suspicious and, dynamically, its malicious behavior, including dubious external connections and its impact, were observed on system sessions.

5.6. Analysis of Sample 6

Sample 6 is the second custom fileless sample, which simulates a fileless ransomware scenario. Instead of using PowerShell, this sample’s macro downloads a Python script (acting as ransomware) and executes it via the Windows CMD. The purpose of this approach was to evaluate whether the proposed analysis model could effectively handle a situation where the payload is not standard shellcode or PowerShell, but rather a scripting language like Python.

Running Olevba on Sample 6’s Word document revealed the presence of a Visual Basic macro (see Figure 14a). The macro references downloading something and executing via CMD. It was flagged because of terms like “Shell” and possibly a URL in the macro (see Figure 14b). This indicated that the macro was likely malicious, attempting to retrieve a file from the Internet and execute it. Additionally, the file was uploaded to VirusTotal. Similar to Sample 5, at the time of analysis, no antivirus engines flagged Sample 6 as malicious (see Figure 14c).

Then, Sample 6 was uploaded to Any.Run, which did not detect it as malicious (see Figure 14d), indicating its evasion capabilities against automated sandboxes. To investigate the detailed process activities, the document was opened in a controlled environment, and macros were enabled. Microsoft Network Monitor showed that the document was communicating with an external website, as illustrated in Figure 14e, and a system process was initiated, enabling the script to download byte streams from a protocol-based website (see Figure 14f).

By observing the memory usage and processes associated with the Word document using the Process Hacker 2 tool (Figure 15a), it was identified that the Word document dropped the file on the hard drive from an external website and then encrypted the data using Python’s encryption modules. This temporary file drop, though quickly removed, is a critical dynamic indicator. By monitoring with ProcMon and applying specific filters, information was gathered that indicated the Word document initiated the “MsMpEng.exe” module, illustrated in Figure 15b. The Python.exe file then queried the file’s information, encrypted it, and created a new file with a similar name but containing the encrypted data, as shown in Figure 15c.

The sophisticated nature of Sample 6 highlights the need for comprehensive dynamic analysis within a hybrid framework to fully uncover its malicious functionality; a single static or network-focused tool might have missed it.

5.7. Comparative Analysis

This section presents a comparative analysis of all malware analyzed in this research in terms of their classification, the files they drop, the processes they create, the registry changes they make, and their network communication.

Table 2. Comparative analysis.

Malware	Classification	Delivery Method	Network Communication	Process	File Dropped	Evasion Technique
Sample 1	Trojan (sdrop)	Word file	Moveis-schuster-com.ga	N/A	None	Code Obfuscation
Sample 2	Trojan (ole2)	Word file	N/A	cmd.exe, Luajit.exe, SearchProtocol, Host.exe	Luajit.exe (removed)	Code Obfuscation
Sample 3	Trojan (mrzi)	Word file	Giftcard2naira.com	cmd.exe, Msg.exe, PowerShell.exe	Msg.exe	Code Obfuscation, Process Injection
Sample 4	Trojan (w97m)	Word file	M4tz0of0xi8o3brr.com	Microsoft.com	Microsoft.com	Code Obfuscation
Sample 5	Trojan (backdoor)	Word file	Trendusfashion.com, 2.tcp.ngrok.io	PowerShell.exe	None	Code Obfuscation
Sample 6	Ransomware	Word file	Trendusfashion.com	cmd.exe, Python.exe, PowersheMsMpEng.exe	Microsofttemplate.py	Code Obfuscation

illustrates the general properties of all malware analyzed in this study. This comparative analysis of six malware samples (four real-world and two newly created) reveals commonalities and distinctions in their operational tactics. All samples were delivered through malicious Word files, typically via phishing or cracked software, though they could also be embedded in XLS files. Samples 1 to 5 are classified as Trojans, which include subtypes like sdrop or backdoor, mrzi, w97, and ole2, while Sample 6 is categorized as ransomware.

A consistent observation across most samples is their attempt to connect to suspicious external domains (e.g., giftcard2naira.com, M4tz0of0xi8o3brr.com, Trendusfashion.com) for purposes such as data exfiltration or payload delivery. Notably, Sample 5 used ngrok.io, a tunneling tool, to establish covert communication, while Sample 4 attempted to mimic legitimate traffic by communicating with microsoft.com. This highlights the evolving evasion techniques employed by attackers. Common system processes like cmd.exe and PowerShell were consistently exploited across samples to execute commands, with Sample 6 also utilizing Python.exe to run its Python-based ransomware script.

While some malware samples, such as Sample 2 (Luajit.exe) and Sample 6 (Microsofttemplate.py), temporarily dropped files to maintain persistence or execute their payload, others, like Sample 1 and Sample 5, demonstrated truly fileless behavior by leaving no persistent traces on disk. All analyzed samples employed code obfuscation techniques to evade static detection. Sample 3 additionally used process injection to hide itself within legitimate workflows. The data highlights attackers’ reliance on everyday tools and obfuscation methods, alongside occasional advanced techniques like tunneling or scripting to maximize damage and evade detection. The hybrid analysis model proved crucial in uncovering these diverse evasion tactics, particularly the dynamic behaviors that bypassed initial static or sandbox checks.

Table 3. Registry changes.

	Sample 1	Sample 2	Sample 3	Sample 4	Sample 5	Sample 6
Keys Added	24,036	17	3	15	3	4
Keys Deleted	36,767	23	4	2	4	0
Values Added	29,083	42	16	39	4	9
Values Modified	230	80	124	110	66	84
Values Deleted	115,142	41	4	10	0	0

lists the values and key changes in the registry observed after malware execution, providing insights into their persistence mechanisms and system impact. Sample 1 shows an extremely high volume of registry operations, including 24,036 keys added, 36,767 keys deleted, 29,083 values added, 230 values modified, and 115,142 values deleted. This aggressive behavior is typical of extensive system changes or anti-analysis techniques, with a significant number of deletions indicating a deliberate effort toward aggressive cleanup or disruption. In contrast, Samples 2–4 showed more moderate changes, with a greater focus on value modifications rather than extensive additions or deletions. This behavior aligns with stealthier, persistent malware aiming to maintain a low profile. The newly created malware samples (Sample 5 and Sample 6) in this research show notably lower registry footprints, emphasizing value modifications over additions or deletions. This suggests that these samples have a more targeted functionality compared to the others. The diverse patterns of registry interaction highlight the varied persistence strategies employed by fileless malware, emphasizing the need for comprehensive registry monitoring within a dynamic analysis model.

5.8. Discussion

The detection of fileless malware poses unique challenges in cybersecurity due to its sophisticated evasion techniques. Unlike traditional malware that resides on disks and creates files, fileless malware operates directly in memory and uses the system’s legitimate tools to carry out its functions. It may also use names similar to system processes, making it significantly harder to detect with conventional methods that rely solely on static or dynamic behavior analysis. Furthermore, the memory-oriented nature of fileless malware complicates the preservation of evidence for forensic analysis, especially after a system reboot, as volatile memory contents are cleared. Additionally, fileless malware often encrypts its payload and employs anti-debugging techniques, making it even harder to analyze, and requiring extensive knowledge of memory and registry operations to differentiate between malicious and legitimate changes.

Recognizing these inherent limitations of single-method approaches, the proposed hybrid model aims to bridge this detection gap by integrating both static and dynamic techniques in a systematic workflow. While the current scope of this paper does not include a quantifiable performance metric for the model’s effectiveness across a large dataset, we acknowledge this as a critical direction for future work. Generating metrics, such as detection rate, false positive rate, false negative rate, precision, recall, and F1-score, would require systematic testing of the entire hybrid workflow against a diverse and labeled dataset of fileless and benign samples. This is a critical next step for future work. However, to highlight the novel contribution and effectiveness of the proposed approach,

Table 4. Comparison of static, dynamic, and hybrid analysis.

Metric	Static Analysis	Dynamic Analysis	Proposed Hybrid Model
Detection Rate	50% (3/6)	66.67% (4/6)	100% (6/6)
False Positive Rate	16.67%	0%	0%
False Negative Rate	50%	33.33%	0%

presents a comparative analysis of individual static and dynamic results alongside the proposed hybrid analysis in terms of detection rate, false positives, and false negatives. The detection rate is calculated as follows:

$$DetectionRate={\displaystyle \frac{DetectedSamples}{TotalMaliciousSamples.}}$$

(1)

The false negative rate is calculated as follows:

$$FalseNegativeRate={\displaystyle \frac{UndetectedMaliciousSamples}{TotalMaliciousSamples.}}$$

(2)

The false positive count is calculated in a similar way if benign samples are tested.

Table 4 presents a comparative analysis of static, dynamic, and the proposed hybrid analysis approaches based on three key performance metrics based on the six malware samples analyzed. Static analysis achieved a 50% detection rate, with the remaining samples not detected primarily due to obfuscation techniques employed by the malware. It also produced a 16.67% false positive rate, largely because it detected malware based on keywords used in the script (e.g., flagging a file as malicious if “Shell” was used for any purpose), and a 50% false negative rate as static tools failed to detect certain malware files. Dynamic analysis showed improvement with a 66.67% detection rate, though it notably failed to detect the newly created malware in this research. It achieved a 0% false positive rate, and performed better than static analysis with a 33.33% false negative rate. The proposed hybrid model, however, achieved a 100% detection rate, with 0% false positive and 0% false negative rates. This outcome validates the main argument of the paper that integrating static and dynamic techniques in the proposed workflow effectively addresses the inherent limitations of each method. As a result, it provides a comprehensive and highly accurate approach for detecting sophisticated fileless malware that might otherwise evade detection.

Our analysis repeatedly demonstrated scenarios where a legitimate process (like Word) exhibited illegitimate behavior (launching a command shell or connecting to a third party), which is a key indicator of a fileless attack. This highlights the importance of employing hybrid analysis approaches that do not rely on a single indicator. However, static analysis still plays a vital role by allowing the detection of threats at earlier stages, e.g., identifying malicious macro code or embedded scripts in documents before they can prevent an attack entirely.

The limitations in addressing fileless malware extend beyond just technical challenges. Many traditional security solutions struggle to detect and prevent these threats effectively because they primarily focus on file-based indicators. The memory-oriented nature of fileless malware complicates the preservation of evidence for forensic analysis, especially after a system reboot. Additionally, fileless malware often encrypts its payload and employs anti-debugging techniques, making it even harder to analyze. This kind of analysis requires extensive knowledge of memory and registry operations to differentiate between changes made by malware and those made by legitimate software or services.

In this research, we have seen examples of these challenges, such as malware that hides in registry keys or uses common processes to mask its activity. This highlights the importance of employing hybrid analysis approaches that do not rely on a single indicator.

Comparison with Malware Detection Approaches

Table 5. Comparison with existing malware detection approaches.

Approach	Primary Focus	Core Techniques	Level of Automation	Accuracy (If Applicable)	Strengths	Limitations
[30]	Behavior analysis	ML models	Semi-automated	93.33%	Feature extraction, obfuscation detection	Trained on limited features, expertise analysis required, and resource intensive
[32]	Enhancing accuracy	1CNN	Manual	97.48%	Byte fragmentation	Reliance on fragment lengths and positions
[33]	Memory dumps analysis	Static and dynamic inspection techniques	Manual	90% success in unpacking malicious code, 80% success in extracting command-and-control configurations	Inspection at multiple level	Higher computational overhead and limited real- time analysis
[19]	Memory forensics	Feature extraction based on attack strategies and characteristics	Automated	N/A	Reduced processing load, and improved feature selection	Resource intensive, expertise required for analysis, and solely focused on API calls
Proposed model	In-depth behavioral analysis	Multi-tool workflow (static and dynamic tools)	Manual	N/A	Granular behavioral insights, evasion resistance, practical for detection of fileless malware through tools, obfuscation detection code analysis, and external traffic detection	Time consuming and limited to per-sample analysis

provides a comparative analysis of existing malware detection approaches, outlining their methodologies, automation levels, reported accuracies, strengths, and limitations. While many ML-based and memory forensics models demonstrate high accuracy in automated detection, they often face challenges, including dependency on specific datasets, computational overhead, and reliance on particular feature sets, such as API calls or memory fragments. Additionally, these algorithmic models often do not fully explain how the malware behaves, and there is a risk of false positives if the malicious code does not execute long enough or is not captured during the specific detection window due to inherent algorithmic limitations. In contrast, the proposed hybrid model adopts a multi-tool workflow that combines both static and dynamic tools for in-depth behavioral analysis of fileless malware. This approach provides detailed behavioral insights and enhanced evasion resistance through a practical, per-sample analysis process. Its strengths include obfuscation detection, comprehensive code-level analysis (e.g., through OLEVBA), and detailed external traffic detection by performing practical analysis on real-world fileless malware. While this model does not have a quantifiable accuracy at this stage, the experiments have qualitatively demonstrated its effectiveness in detecting all six analyzed samples. However, a notable limitation is that it is time-consuming, as it is designed to analyze one file at a time. However, this limitation can be overcome by automating the hybrid analysis using ML and AI. This automation would address various scalability challenges, such as increasing throughput and speed, reducing resource usage, and enabling automated reporting. Furthermore, it would facilitate the integration of threat intelligence with deep learning and parallel file analysis, which can significantly reduce analysis time from hours to just seconds or minutes.

5.9. Mitigation Measures

Malware poses a significant threat to individuals, businesses, and organizations worldwide, infiltrating systems, stealing sensitive data, disrupting operations, and causing financial harm. To combat these risks, it is essential to implement strong mitigation measures. Security experts suggest effective strategies and best practices to prevent malware from entering systems or to mitigate its effects. The following are key mitigation measures to consider:

• Enhanced User Awareness and Training: User education is crucial for effectively mitigating malware threats. Regular security training programs should educate employees about malware risks, phishing attacks, and safe online behavior. It is important to include phishing awareness in these programs to help users identify phishing emails and malicious links. Training should also cover safe browsing habits, and organizations should prioritize software updates and patch management to prevent malware exploitation.
• Antivirus and Anti-Malware Software: To protect against malware, deploy reputable antivirus and anti-malware software on all endpoints and ensure they are updated regularly. While antivirus alone is not sufficient for fileless threats, it remains a necessary component of a layered defense.
• Firewalls: Configure host-based firewalls to control network traffic and prevent unauthorized access to systems. Proper firewall rules can block unwanted inbound connections (reducing risk from certain network-based worms) and also restrict outbound traffic. Implementing application whitelisting and blacklisting at the firewall can limit unauthorized applications.
• Intrusion Detection and Prevention Systems: Implement intrusion detection and prevention systems to actively monitor and block malicious activities in real time. These systems can be tuned to detect patterns similar to those observed in our analysis, e.g., a sudden surge of outbound connections from an internal host or known exploit signatures.
• Network-Level Security: Employ multiple measures at the network level to contain and reduce malware spread. Network segmentation, secure web gateways, email filtering, DNS filtering, and VPNs are essential measures of network-level security. Network segmentation limits lateral movement, secure web gateways block access to malicious websites, email filtering scans for malware attachments, DNS filters block malicious domains, and VPNs require multi-factor authentication for secure remote work in an organization.
• Principle of Least Privilege: Enforce the principle of least privilege for user accounts and software processes. Each user or service should have only the minimum privileges necessary for their role. This way, even if malware infects a user’s session, it may not have the rights to install itself persistently or access sensitive parts of the system.
• Multi-Factor Authentication (MFA): Use MFA for critical systems and accounts, especially for remote access and administrative accounts. It is a crucial security measure that prevents malware from easily bypassing sensitive systems and data, even if credentials are stolen.
• Incident Response Plan (IRP): Develop and maintain an IRP for malware incidents that includes procedures for detection, containment, eradication, and recovery. It should also define roles and communication flow when responding to an incident. Staying informed about emerging threats enables early detection and mitigation.
• Data Backup: Regularly back up critical data and systems, verify their integrity, and securely store them off-site for ransomware recovery. In the case of a ransomware attack, having recent backups can mean the difference between a minor inconvenience and a major crisis. Backups should be kept in a manner that malware cannot easily delete or encrypt them.
• Security Information and Event Management (SIEM) Implementation: Deploy a SIEM system to centralize and analyze logs and security events, providing real-time alerts for potential malware activity. Real-time alerts for patterns indicative of fileless attacks can significantly cut down response times.
• Updates and Patch Management: Prioritize security patch installation for operating systems, applications, and firmware to prevent malware attacks. Establish a formal patch management process, test patches, schedule cycles, and use automated tools for efficient patching. Keeping systems up to date can prevent certain infection vectors entirely.
• Email Phishing Protection: Implement email filtering and anti-phishing solutions to detect and block phishing attempts, while also applying content filtering to scan email attachments for malware and block suspicious content.
• Memory Forensics with Deep Learning: Installing AI-powered tools, such as process hollowing, to continuously scan volatile memory for signs of code injection. Neural networks identify subtle patterns, like malicious shellcode or unauthorized reflective .dll loading, which evade traditional scans.
• Deception-Based AI Honeypots: AI-managed decoy processes should be deployed, or fake credentials should be stored in memory. When fileless malware interacts with these traps, the AI logs its behavior, which helps threat intelligence identify patterns. This information can then be used to update detection models and counter future attacks.
• Reinforcement Learning for Endpoint Detection and Response (EDR): Enhance EDR systems by integrating reinforcement learning. The AI learns the best responses to malicious process chains and automatically isolates compromised endpoints. In the context of our analysis, such a system might have automatically killed a Word process that started running PowerShell and CMD simultaneously, preventing the malware from completing its routine.

By implementing a combination of these mitigation measures, organizations can establish a strong defense against both traditional and fileless malware threats. Our study emphasizes that relying on a single defensive measure is inadequate. Instead, adopting a defense-in-depth strategy (including user education, preventive controls, detection mechanisms, and response capabilities) provides the most effective protection against the constantly evolving landscape of malware attacks.

6. Conclusions and Future Work

The increase in cyber-attack incidents and their complexities has highlighted the importance of detailed malware analysis to protect an organization’s reputation, resources, and business operations. With the rise of both file-based and fileless malware, advanced analysis methods are now necessary.

This study focuses on fileless malware and introduces a new hybrid analysis technique that combines both static and dynamic analysis methods. By examining four real-world fileless malware samples and two custom-created samples, the research provides a thorough understanding of the behavior of fileless malware. The findings reveal that fileless malware is particularly stealthy, allowing it to operate silently within infiltrated systems and often evade traditional detection measures. The proposed analysis method takes a comprehensive approach to address the complex nature of fileless malware, using advanced tools and a combination of techniques to overcome the individual limitations inherent in each individual method. This leads to more accurate detection and deeper insights into fileless malware. The study also highlights the essential mitigation measures that individuals and organizations should adopt to protect themselves from external malware threats. We are confident that the proposed analysis method will be instrumental in combating the growing menace of fileless malware, and we believe that its importance will only continue to grow in the coming years.

Future work will focus on enhancing the model by integrating AI and ML techniques to improve the automated detection and classification of fileless malware. To statistically validate the model’s performance, we will conduct extensive systematic testing of the entire hybrid workflow using a larger, diverse, and labeled dataset that includes both fileless malware and benign samples. This approach will allow us to calculate key performance metrics, such as detection rate, precision, recall, and F1-score, utilizing robust methods like cross-validation. Additionally, the hybrid model will be benchmarked against widely used malware detection tools, including VirusTotal, Any.Run, and Cuckoo Sandbox, as well as other relevant hybrid analysis techniques from recent literature. This benchmarking process will quantitatively demonstrate the effectiveness, robustness, and generalizability of our model. Furthermore, we plan to expand our Windows-centric approach to other operating systems, particularly Linux, to proactively respond to the increasing threat of fileless malware across different platforms.