Chaotic-Based Shellcode Encryption: A New Strategy for Bypassing Antivirus Mechanisms

Abstract

This study employed chaotic systems as an innovative approach for shellcode obfuscation to evade current antivirus detection methods. Standard AV solutions primarily rely on static signatures and heuristic analysis to identify malicious code. However, chaotic systems employ dynamic and unpredictable encryption methods, significantly obstructing detection efforts. The utilization of various chaotic maps for shellcode encryption facilitates the generation of multiple unique variations from the same functional code, each exhibiting distinct unpredictability due to the inherent nonlinearity and sensitivity of chaotic systems to initial conditions. The unpredictability of these situations poses a considerable challenge for antivirus software in recognizing consistent patterns, resulting in decreased detection rates. The findings from our experiments demonstrate that chaos-driven encryption methods significantly outperform traditional encryption techniques in terms of evading detection. This paper emphasizes the potential of chaos theory to enhance malware evasion strategies, offering a sophisticated approach to bypassing modern antivirus protections while ensuring the effectiveness of malicious payloads.

1. Introduction

Innovative methods for concealing malicious code are continually evolving in the ongoing conflict between malware authors and antivirus (AV) solutions [1]. Malware, an abbreviation for malicious software, encompasses a wide range of harmful programs designed to disrupt, damage, or gain unauthorized access to computer systems. These include viruses, worms, trojans, and ransomware that can steal private information, damage files, or take over computers. Malware assaults usually result in significant financial losses, reputational damage, and operational disruptions for both individuals and enterprises, which can have catastrophic consequences. Shellcode, commonly used to exploit software vulnerabilities, consists of code segments that execute arbitrary instructions on a target system, constituting a significant element of malware attacks. It is an effective weapon in the cybercriminal’s arsenal; once injected into the target machine’s memory, it enables attackers to seize control or execute malicious actions. Investigating malware and shellcode is crucial for developing more effective defensive strategies, as malware significantly contributes to cybersecurity threats. Malware developers must enhance their techniques to maintain efficacy as AV technology advances [2].

Obfuscation techniques such as metamorphism and polymorphism focus on altering code structure to hide it from detection. These approaches adhere to detectable transformation patterns that AV systems can finally learn to recognize [3], regardless of their capacity to generate numerous versions of the same virus. Chaotic encryption provides a unique benefit due to the inherent nonlinearity and unpredictability of chaotic systems which introduce an additional degree of complexity. Chaotic encryption generates highly randomized, dynamic encryptions that lack established protocols, in contrast to traditional obfuscation. This significantly complicates the ability of AV softwareto recognize consistent indicators in the code, hence enhancing the malware’s capability to circumvent even the most sophisticated detection systems. By examining chaotic encryption, researchers may explore more robust evasion techniques that not only outperform traditional methods but also establish new frontiers in malware defense and cybersecurity.

This study’s main contribution is demonstrating how chaotic encryption techniques can be used to make shellcode more stealthy, so it can avoid detection by AV programs. Using the nonlinearity and high sensitivity of chaotic systems, our suggested method creates an encrypted shellcode that is both dynamic and unexpected. The experimental results reveal that this method is light years ahead of more conventional methods of evasion. This research presents a fresh viewpoint on how current AV technology could be sidestepped by showcasing the possibilities of chaos theory in malware evasion tactics.

This study is organized as follows: Section 2 examines pertinent literature, Section 3 delineates the technique and encryption procedure, Section 4 presents the experimental findings, and Section 5 concludes with prospective research avenues.

2. Related Work

2.1. Antivirus Detection Methods

Commonly used in malware creation, obfuscation is used to hide malicious code [4]. Traditionally, AV systems have mostly depended on signature-based detection, which detects malware by comparing a file against a database of known hostile signatures [5]. This method struggles to identify new or altered malware, including polymorphic or metamorphic versions that change their appearance with each iteration [6], even though it is successful against established threats. Heuristic-based detection tries to identify malware depending on suspicious behavior patterns in order to overcome this constraint. However, this approach may overlook malware that dynamically alters its own behavior [7] and is prone to false positives.

Antivirus (AV) systems have progressively included artificial intelligence (AI) and machine learning (ML) to improve their detection power as malware develops in sophistication. With anomaly detection as a basic use, ML-based systems examine patterns in large-scale datasets to identify previously unidentified threats. These complex algorithms highlight deviations that might point to hostile activities and train models on proper software behavior. Complementing this method, behavioral-based detection tracks file access patterns, network traffic, and system calls to identify malware activities. Leading this technological arms race, deep learning models—including convolutional neural networks (CNNs) and recurrent neural networks (RNNs)—are being used to forecast hostile behavior before it occurs [8,9].

2.2. Polymorphic and Metamorphic Malware

Malware creators have devised metamorphic and polymorphic approaches [10] to avoid signature-based detection. Complicating detection, polymorphic malware encrypts its payload, decrypting it during runtime, and changes its encrypted form with every iteration. By rewriting its own entire code during each infection, metamorphic malware goes a little further and generates functionally comparable but structurally distinct variations [11]. Though they rely on modifications that AV systems can ultimately learn to recognize [12], these techniques can evade both signature-based and heuristic detection methods.

Recent years have seen hackers merging metamorphic and polymorphic methods with packing and virtualization-based obfuscation [13]. Packers encrypt or compress executables, therefore, preventing AV analysis until runtime. Virtualization-based obfuscation adds complexity to the execution environment [14] by converting malware code into bespoke virtual machine (VM) bytecode, therefore, complicating detection. These hybrid methods challenge even contemporary AV systems, necessitating advanced detection methods such as dynamic analysis in sandbox settings to monitor malware’s interactions with the operating system and uncover evasive behavior [15].

2.3. Obfuscation Techniques for Malware Evasion

Malware analysis and detection are complicated by obfuscation methods including instruction replacement, garbage code insertion, control flow flattening, and instruction sequencing [16]. These techniques complicate the reverse-engineering of the code and prevent its discovery. Different tools and frameworks apply these approaches, so traditional AV solutions lose their efficiency.

Nevertheless, AV technologies are evolving to counter obfuscation techniques as they become more advanced. Modern AV systems use methods such as machine learning and dynamic analysis to identify unusual execution patterns or excessive complexity in program behavior [17], thereby detecting obfuscated code. Although obfuscation is still useful to some extent, the constant arms race between malware creators and antiviral programs emphasizes the drive for constant innovation in both malware methods and cybersecurity protections.

2.4. Well-Known Evasion Tools

2.4.1. TheFatRat

Particularly used to produce payloads that elude AV detection [18], the popular utility TheFatRat aids in constructing obfuscated malware. Using a range of obfuscation and encryption methods, it lets users develop backdoors and payloads challenging for AV systems to detect. TheFatRat adds additional levels of obfuscation to evade signature-based detection and automates the injection of malicious payloads into permitted programs.

2.4.2. Shikata Ga Nai

This is a somewhat popular encoding method in the Metasploit framework. Shikata Ga Nai employs polymorphic encoding to produce several variations of the same payload, therefore, altering the shellcode’s appearance without compromising its performance. This method continually modifies the structure of the shellcode, consequently defeating signature-based AV detection [19].

2.4.3. Veil Framework

Veil is another obfuscation tool designed to generate payloads that can evade AV systems. It allows users to encode payloads using different methods, including encryption and polymorphic techniques, to avoid detection by static and heuristic analysis. Veil supports various languages and formats for payload generation and focuses on bypassing real-time protection [20].

2.4.4. ROPInjector

Using return-oriented programming (ROP), ROPInjector is a tool for translating shellcode into its ROP equivalent and injecting it into portable executable (PE) files with the aim of AV evasion [21,22,23]. ROPInjector uses innocuous code gadgets already existing in the target PE, unlike conventional polymorphism techniques requiring writable code sections, therefore, making the injection process more difficult for AV software to detect. By converting shellcode into ROP code, the tool essentially thwarts behavioral detection techniques [24] and signature-based detection systems. New static analysis methods for ROP code have recently been presented, allowing for the examination of challenging payloads without first having to replicate the ROP activation environment. Furthermore, improving the identification of complex ROP-based assaults [25], a guessing technique helps in the discovery of gadget origins in payloads from network traffic or documents.

2.4.5. Movfuscator

Designed to replace traditional x86 instructions with equivalent MOV instructions, Movfuscator is an obfuscation tool [26]. The Turing completeness specific to the MOV instruction makes this transformation feasible, as it lets all the instructions be replaced with sequences of MOV instructions while maintaining the original logic of the program. MOV-based obfuscation is unique, so no particularly effective and accessible deobfuscation techniques have yet been developed [27]. Deobfuscation tools such as de-movfuscator can only partially deobfuscate the code, offering a starting point, but they are not yet able to completely eradicate this kind of obfuscation [28].

2.5. Chaotic Systems in Cryptography

Chaotic systems have been investigated across several domains, including cryptography, data encryption, and secure communications, owing to their intrinsic unpredictability, sensitivity to the beginning circumstances, and nonlinear dynamics [29]. Chaotic systems in cryptography have been used to create encryption algorithms that produce extremely unpredictable key streams, which are challenging to anticipate or replicate without access to the starting parameters [30,31]. Investigations in this domain have demonstrated that chaotic systems can generate secure and efficient encryption methods appropriate for settings with constrained computational capabilities [32]. J. Fridrich suggested the encryption of $N\times N$ images using two-dimensional chaotic maps [33]. The author demonstrated that two-dimensional chaotic maps when modified and discretized, can facilitate the development of symmetric block encryption systems characterized by robust diffusion qualities, exemplified by the Baker map for safe encryption. Another study presented a multimedia communication method utilizing 2D alteration models and integrating several chaotic maps for shuffling and substitution [34]. This exhibits robust resilience to assaults and enhanced performance relative to conventional approaches. The use of chaos theory in shellcode obfuscation remains inadequately examined, presenting the potential for innovative strategies in malware avoidance.

Despite these developments, the use of chaos theory for obfuscating shellcodes is still unexplored territory. In order to improve malware evasion tactics, this research proposes a new route for current malware creation and tests these techniques in real-world AV systems using lightweight chaotic maps.

Table 1 provides a comparative analysis of various evasion techniques, highlighting the advantages and limitations of each method in circumventing AV detection.

Table 1. Comparison of evasion techniques: advantages and disadvantages of various methods for shellcode detection evasion.

Method	Advantages	Disadvantages
TheFatRat	Automates payload injection and applies multiple obfuscation techniques, challenging AV detection.	Susceptible to detection by advanced AV systems, as many AV tools recognize common patterns in generated payloads.
Shikata Ga Nai	Polymorphic encoding alters the appearance of shellcode without changing functionality, defeating static analysis.	Limited effectiveness, as AV systems can learn to detect the encoding pattern with multiple iterations.
Veil Framework	Supports multiple languages and formats for payload generation, using encryption and polymorphic techniques.	Less effective against dynamic analysis and sandboxing; may still be flagged by heuristic or behavioral-based AV systems.
ROPInjector	Converts shellcode into return-oriented programming (ROP) gadgets, complicating detection.	Complex to implement and limited by the availability of suitable ROP gadgets within the target environment.
Movfuscator	Replaces instructions with equivalent MOV instructions, creating unique obfuscated code.	Limited deobfuscation techniques exist, but AV systems may still detect certain patterns; high performance overhead.
Chaotic Encryption (Proposed)	Generates randomized, dynamic encryption with chaotic systems, offering high unpredictability and evasion capabilities.	Sensitive to parameter tuning; if the parameters are not optimized, the system may stabilize, reducing evasion effectiveness.

3. Methodology

This section delineates the encryption procedure using a chaotic system. Algorithm 1 employs a chaotic system to produce random values (chaotic values) for the encryption of a specified plaintext (shellcode) P. The method processes each byte of the plaintext, extracting chaotic values from the system, which are then normalized to a range between 0 and 255. If the chaotic value is zero, a new chaotic value is created, as XOR with zero does not effectuate any encryption as $p\oplus 0=p$. The resultant ciphertext C is derived by executing the XOR operation between the plaintext byte and the chaotic key.

Algorithm 1: Chaotic Encryption Algorithm

Input: Plain text (shellcode) P Output: Cipher text C

As shown in Algorithm 2, because XOR is a reversible operation, the plaintext P can be recovered from the ciphertext C using the same chaotic key k obtained from the system [35].  

Algorithm 2: Chaotic Decryption Algorithm

Input: Cipher text (encrypted shellcode) C Output: Plain text P

Chaotic systems have the benefit of producing consistent random values k from identical beginning circumstances, hence guaranteeing the congruence of encryption and decryption keys. The values produced by chaotic systems demonstrate significant randomness and unpredictability, making it difficult for AV to identify discernible patterns or signatures inside the encrypted ciphertext.

This ensures that the encryption is reversible and the original data may be recovered when needed, given that the identical parameters for the chaotic system are used for decryption. The Lorenz system [36], Chua’s circuit system [37], and the Rossler system [38] use the Runge–Kutta fourth Order (RK4) method for solving first-order ordinary differential equations (ODEs) [39,40], offering an effective equilibrium between precision and computational efficiency [41]. It is especially advantageous for chaotic systems because of its stability and accuracy in extended simulations.

$$k_1=f(t_n,y_n)$$

(1)

$$k_2=f\left(t_n+{\displaystyle \frac{dt}{2}},y_n+{\displaystyle \frac{k_1·dt}{2}}\right)$$

(2)

$$k_3=f\left(t_n+{\displaystyle \frac{dt}{2}},y_n+{\displaystyle \frac{k_2·dt}{2}}\right)$$

(3)

$$k_4=f\left(t_n+dt,y_n+k_3·dt\right)$$

(4)

$$y_{n+1}=y_n+{\displaystyle \frac{dt}{6}}\left(k_1+2k_2+2k_3+k_4\right)$$

(5)

In this method, $k_1$, $k_2$, $k_3$, and $k_4$ are intermediate slopes used to estimate the solution at the next step $y_{n+1}$. The final result is a weighted average of these slopes, making RK4 highly accurate for solving first-order ODEs. The method allows for higher precision and stability compared to lower-order methods, especially when applied to chaotic systems such as the Lorenz, Chua’s circuit, and Rössler systems.

3.1. Logistic Map Encryption/Decryption

The logistic map is a mathematical function that models chaotic behavior in dynamic systems. It is widely studied in chaos theory because it demonstrates how complex, unpredictable behavior can emerge from simple, nonlinear equations. The logistic map is defined as follows:

$$x_{n+1}=r·x_n·(1-x_n)$$

(6)

• $x_n$: The current state of the system, typically within the range $0<x_n<1$.
• r: The control parameter.
• n: The iteration index.

The most interesting dynamics occur when r is between 3.57 and 4, where the system transitions from periodic behavior to full chaos [42]. The implementation source code is from Listing A1 in Appendix A.1.

3.2. Henon Map Encryption/Decryption

For the Henon Map, the values of x (and y) can vary depending on the parameters a and b. The Henon Map is defined as follows:

$$x_{n+1}=1-a{x}_n^2+y_n$$

(7)

$$y_{n+1}=b{x}_n$$

(8)

• a and b: The control parameters that influence the system’s dynamics.
• $x_n$ and $y_n$: The state variables at iteration n.

The system exhibits chaotic behavior for typical values $a=1.4$ and $b=0.3$ and is widely studied in chaos theory [43]. The implementation pseudocode is from Listing A2 in Appendix A.1.

3.3. Lorenz System Encryption/Decryption

The Lorenz system is a system of three coupled, nonlinear differential equations originally developed to model atmospheric convection. The system is known for its chaotic behavior, which emerges under specific parameter conditions. The Lorenz system is defined as follows:

$${\displaystyle \frac{dx}{dt}}=\sigma (y-x)$$

(9)

$${\displaystyle \frac{dy}{dt}}=x(\rho -z)-y$$

(10)

$${\displaystyle \frac{dz}{dt}}=xy-\beta z$$

(11)

• $\sigma$, $\rho$, and $\beta$: Control parameters that influence the system’s dynamics.
• x, y, and z: Initialize the start value.

For $\sigma =10$, $\rho =28$, and $\beta =8/3$, the system exhibits the well-known Lorenz attractor. The implementation source code is from Listing A3 in Appendix A.1.

3.4. Chua’s Circuit System Encryption/Decryption

Chua’s circuit, composed of standard electronic components, became one of the first recognized circuits to demonstrate chaos. Chua’s circuit is highly sensitive to its initial conditions, which is a hallmark of chaotic systems. This can be described by the following set of coupled nonlinear ordinary differential equations as

$${\displaystyle \frac{dx}{dt}}=\alpha \left(y-x-g\left(x\right)\right)$$

(12)

$${\displaystyle \frac{dy}{dt}}=x-y+z$$

(13)

$${\displaystyle \frac{dz}{dt}}=-\beta y$$

(14)

where $\alpha$ and $\beta$ are system parameters. $g\left(x\right)$ is a piecewise linear function modeling the nonlinearity introduced by Chua’s diode and is defined as

$$g\left(x\right)=\left\{\begin{array}{cc}{m}_{1}x+m_1-m_0,\hfill & \mathrm{if}x\le -1\hfill \\ m_{0}x,\hfill & \mathrm{if}-1\le x\le 1\hfill \\ m_{1}x+m_0-m_1,\hfill & \mathrm{if}x\ge 1\hfill \end{array}\right.$$

• $\alpha$ and $\beta$: Control parameters for the system’s dynamics.
• $m_0$ and $m_1$: Parameters defining the nonlinearity of $g\left(x\right)$.
• x, y, and z: Initialize the start value.

The typical values for chaotic behavior are $\alpha =9.0$ and $\beta =14.286$ [44]. For the above parameter values, Chua’s circuit exhibits chaotic behavior, making it a widely studied system in chaos theory. The implementation source code is from Listing A4 in Appendix A.1.

3.5. Rössler System Encryption/Decryption

The Rössler system is a simple three-dimensional dynamical system that exhibits chaotic behavior. It was introduced as a model of chemical reactions but has since been widely used to study chaos. The Rössler system is defined as follows:

$${\displaystyle \frac{dx}{dt}}=-y-z$$

(15)

$${\displaystyle \frac{dy}{dt}}=x+ay$$

(16)

$${\displaystyle \frac{dz}{dt}}=b+z(x-c)$$

(17)

• a, b, and c: Control parameters that determine the system’s behavior.
• x, y, and z: Initialize the start value.

For $a=0.2$, $b=0.2$, and $c=5.7$, the system exhibits the well-known Rössler attractor, a chaotic attractor that is widely studied in chaos theory [45]. The implementation source code is from Listing A5 in Appendix A.1.

4. Experiment

For this test, the computer language C# was used with .NET Framework 4.7.2 on Windows and .NET 8.0 on Linux. The goal of this study was to evaluate the efficiency of chaotic encryption in evading AV detection on Kleenscan.com, an online sandbox platform that aggregates data from over 30 AV companies. This platform enables users to submit files and analyze the detection probabilities for various AV engines. The platform was chosen because it enables a comprehensive evaluation of the effectiveness of chaotic encryption in escaping detection, and it offers a wide choice of commercial AV solutions. Penetration testers and red teams can benefit greatly from utilizing this method because it ensures a thorough and fair evaluation of the effectiveness of encryption across a variety of AV systems. The shellcode is created with the msfvenom tool; the unencrypted original shellcode, generated via the msfvenom command, is depicted in Figure A1 and Figure A2 in Appendix B.2, and will be shown for 64-bit Windows and Linux staged reverse TCP [46].

4.1. Windows Shellcode Execution

As described in Listing A6 in Appendix A.2, this code demonstrates the dynamic execution of shellcode by using platform invoke (P/Invoke) to call functions from kernel32.dll, such as VirtualProtect, CreateThread, and WaitForSingleObject. These functions are used to allocate memory, change the memory protection settings, and create a thread to execute the shellcode [47,48].

4.2. Linux Shellcode Execution

As described in Listing A7 in Appendix A.2, this code uses P/Invoke (Platform Invocation Services) in .NET 6.0 to execute unmanaged code by interfacing with Linux’s system calls via libc. Specifically, it allocates memory, copies executable code into that memory, changes the memory’s protection to allow execution, and then runs the code [49].

4.3. Parameters in Chaotic Systems

We conducted experiments on encryption using the logistic map with the parameter $r=3.99$ and the initial condition $x_0=0.5$; the Henon map with the parameters $a=1.4$, $b=0.3$, and initial conditions $x_0=0.1$, $y_0=0.0$; the Lorenz system with the parameters $\sigma =10$, $\rho =28$, $\beta =2.667$, and initial conditions $x_0=0.1$, $y_0=0.0$, $z_0=0.0$; Chua’s circuit with the parameters $\alpha =10$, $\beta =14$, $m_0=-0.14$, $m_1=0.28$, and initial conditions $x_0=0.1$, $y_0=0.0$, $z_0=0.0$ and the Rössler system with the parameters $a=0.2$, $b=0.2$, $c=5.7$, and initial conditions $x_0=0.1$, $y_0=0.0$, $z_0=0.0$. We also conducted the worst case for encryption using the logistic map with the parameter $r=20$ and the initial value condition $x=2$.

5. Results

5.1. Execution Time

This experiment involved measuring the encryption duration of a shellcode on both Windows and Linux platforms utilizing various chaotic systems. The shellcodes for Windows and Linux were produced using msfvenom, measuring 510 bytes and 130 bytes, respectively. We performed encryption with five distinct chaotic systems: the logistic map, the Henon map, the Lorenz system, Chua’s circuit, and the Rössler system. The execution durations for each system were assessed utilizing an AMD Ryzen Threadripper 2950X 16-Core Processor (3.5 GHz). Table 2 summarizes the execution timings for both the Windows and Linux shellcodes.

Table 2. Processing times for encrypting/decrypting Windows and Linux shellcodes using different chaotic systems.

Chaotic System	Windows Execution Time (µs)	Linux Execution Time (µs)
Logistic Map	425.2	463.8
Henon Map	754.5	668.9
Lorenz System	1677.5	1368.3
Chua’s Circuit	1268.3	1431.4
Rössler System	1855.3	1313.1

5.2. Ensuring Chaotic Behavior in Encryption

The data were analyzed pre- and post-encryption using the logistic map encryption method, as shown in Figure 1; corresponding chaotic keys were produced throughout the procedure. Although the capacity to reverse the encryption with the appropriate keys is retained, the chaotic nature of the logistic map significantly alters the shellcode. This demonstrates the efficacy of chaotic encryption in altering the original data’s structure while guaranteeing accurate decryption. For example, the first byte from the shellcode is 0xFC and the key is 0xFE, while the end first encrypted byte is 0x02. This process continues for each subsequent byte.

Figure 1. Data before and after logistic map encryption with keys.

The shellcode files exhibit keys and encrypted data, as shown in Figure 2. The value’s derivation from the Henon map, which might have negative outputs, affects the result. The value from the second byte 0xFFFFFFAC is negative in the two complement representations. The XOR operation using 0xFF only examines the least significant 8 bits of the negative integer, yielding 0xFF ⊕ 0xAC = 0x53. This transpires because limited to a byte (8 bits), the negative integer 0xFFFFFFAC (in two’s complement) is interpreted as 0xAC, leading to the XOR operation producing 0x53.

Figure 2. Data before and after Henon Map encryption with keys.

5.3. Worst Encryption Case

The worst-case scenario of the experiment occurs when the chaotic system stabilizes and becomes predictable. As shown in Figure 3, this sample has been identified as malicious and represents the worst sample for logistic map encryption with the parameter $r=20$ and the initial value condition $x=2$. In this case, the value from the third byte 0x80000000 is negative in the two complements. The XOR operation with 0x83 considers only the least significant 8 bits of the negative integer, resulting in 0x83 ⊕ 0x00 = 0x83. This occurs because constrained to a byte (8 bits), the negative integer 0x80000000 (in two’s complement) is interpreted as 0x00, resulting in the XOR operation yielding 0x83. The corresponding byte in the encrypted data remains identical to the original shellcode byte. This results in sections of unencrypted data, which could reduce the overall encryption effectiveness.

Figure 3. Data before and after logistic map encryption with worst case.

5.4. Handling 0x00 Values in Chaotic Encryption

Figure 4 illustrates the significance of omitting 0x00 in the design of Algorithms 1 and 2, using the logistic map as a case study. To determine the system’s resilience, it is important to assess whether the chaotic sequence produced by the logistic map converges to stability under specific parameters. Through the examination of the sequence’s behavior regarding stability in different places, we can ascertain that omitting 0x00 preserves the desired chaotic characteristics. This verification phase guarantees that the system does not settle into a discernible pattern, hence maintaining the encryption’s unpredictability. When XOR encryption meets 0x00 chaotic value at $r=0.35,x=0.5$, in the Logistic Map, the process fails to transform the original data, leading to unchanged bytes and compromising the encryption’s efficacy. Upon configuring the method to exclude 0x00, each XOR operation consistently conceals the data, guaranteeing that the encryption process remains resilient and safe. This idea is widely applicable to encryption methods that employ chaotic maps to generate encryption keys.

Figure 4. Data before and after logistic map encryption including 0x00 keys.

5.5. Disassemble

Part of the original shellcode and encrypted data were disassembled using Defuse Security’s Online x86/x64 Assembler and Disassembler (https://defuse.ca/online-x86-assembler.htm accessed on 4 November 2024) [50]. As shown in Figure 5 and Figure 6 (the disassembled codes are in Appendix B.1), following CISC (complex instruction set computing) architecture, this assembly language uses a precise understanding of past instructions to ascertain the limits of later ones in decoding. The variable-length character of CISC instructions means that a single bit mistake may interfere with the decoding of the next instructions, therefore, influencing the whole program’s run. In the disassembly of the encrypted data, certain segments are marked as BAD. These labels indicate that the encrypted bytes do not translate to valid opcodes, resulting in decoding errors when attempting to interpret them as executable instructions. This is an expected outcome in the encryption process, as the XOR operation with chaotic keys produces byte sequences that no longer represent meaningful or valid instructions.

Figure 5. Disassembly of the shellcode and encrypted data for Windows staged x64 reverse TCP shellcode.

Figure 6. Disassembly of the shellcode and encrypted data for Linux staged x64 reverse TCP shellcode.

In Figure 7 (with the disassembled code in Appendix B.1), the red-highlighted parts denote code segments that are identical in both the original shellcode and the encrypted variant. This consistency suggests that some segments of the shellcode remain unaltered by the encryption process. This partial encryption produces discernible patterns in the encrypted data, which may be detected by antivirus. The continued existence of identifiable parts of code shows the importance of an enhanced encryption strategy that completely hides the payload, thereby decreasing the chance of antivirus detection.

Figure 7. Disassembly of the shellcode and encrypted data for Windows staged x64 reverse TCP shellcode with 0x00 keys.

5.6. Scan Report

5.6.1. Comparison Other Proposed Method

The comprehensive report is included in Appendix B.3. As indicated in Table 3, we analyzed the samples by generating executable files using msfvenom and shikata_ga_nai with five iterations, PwnWind in FatRat, and the Veil framework to produce Windows malware for reverse TCP shell control. The results show that a large number of AV providers effectively identified the malware samples produced by various evasion methods. Although there were minor variations in the detection rates across the tools, all the significant AV tools detected attempts from msfvenom, PwnKind, shikata_ga_nai, and Veil, showing their limited efficacy in evading detection. Despite their higher likelihood of detection by AV software, these tactics remain prevalent today [51].

Table 3. Scan results for other evasion tools for Windows reverse shell.

Vendor	Msfvenom Native (exe)	shikata_ga_nai in 5 Iterators	PwnWind in FatRat	Veil Framework
AdAware	Trojan.Metasploit.A	Trojan.Metasploit.A	Gen:Variant.Bulz.632241	Gen:Variant.Kryptik.175
Alyac	Trojan.Metasploit.A	Trojan.Metasploit.A	-	Gen:Variant.Razy.575638
Arcabit	Trojan.Metasploit.A	Trojan.Metasploit.A	Trojan.Bulz.D9A5B1	Trojan.Kryptik.175
Avast	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
AVG	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
Avira	TR/Crypt.XPACK.Gen7	TR/Crypt.XPACK.Gen7	HEUR/AGEN.1234945	TR/Crypt.XPACK.Gen7
Bullguard	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
ClamAV	Win.Malware.Metasploit-10022275-0	Win.Trojan.MSShellcode-6360728-0	-	-
Comodo Linux	-	-	TrojWare.MSIL.Rozena.C	TrojWare.MSIL.TrojanDownloader.Small.H
Crowdstrike Falcon	Threat detected.	-	Threat detected.	-
Emsisoft	Trojan.Metasploit.A	Trojan.Metasploit.A	Gen:Variant.Bulz.632241	Gen:Variant.Kryptik.175
F-Prot	W64/S-c4a4ef26!Eldorado	W64/S-c4a4ef26!Eldorado	-	-
F-Secure	TR/Crypt.XPACK.Gen7	Trojan.TR/Crypt.XPACK.Gen7	Heuristic.HEUR/AGEN.1306275	Trojan.TR/Crypt.XPACK.Gen7
G Data	Trojan.Metasploit.A	Trojan.Metasploit.A	Gen:Variant.Bulz.632241	Gen:Variant.Kryptik.175
IKARUS	Trojan.Win64.Meterpreter	Trojan.Win64.Meterpreter	Trojan.MSIL.Rozena	Trojan-Downloader.MSIL.Tiny
Immunet	Win.Malware.Metasploit-10022275-0	Win.Trojan.MSShellcode-6360728-0	-	-
Kaspersky	Trojan.Win64.Packed.gen	Trojan.Win64.Packed.gen	-	-
Max Secure	Trojan_W64_090622_Packed_gen_YR	Trojan_W64_090622_Packed_gen_YR	-	-
McAfee	Trojan-FJIN!D8D12B41E269 trojan !!!	Trojan-FJIN!C681A28426B4 trojan !!!	-	Trojan-Veil-FLRH!FC0B92454BC0 trojan !!!
Microsoft Defender	Trojan:Win64/Meterpreter!pz	Trojan:Win64/Meterpreter!pz	Trojan:Win32/Rozena.D!bit	Trojan:MSIL/Tiny.EM!MTB
NANO	-	-	-	Multiple Threats Detected (Trojan.Win32.TrjGen.dbjaze, Trojan.Win32.TrjGen.dkksls)
NOD32	Win64/Rozena.M trojan	Win64/Rozena.J trojan	MSIL/Rozena.C trojan	MSIL/TrojanDownloader.Tiny.BQ trojan
Norman	Win32:MsfShell-V	Win32:ShikataGaNai-B	Win32:TheFatRat-A	Win32:Evo-gen
SecureAge APEX	Malicious	Malicious	-	Malicious
Seqrite	Trojan.Dynamer.S4605	Trojan.Dynamer.S4605	-	-
Sophos	Mal/Swrort-J	Mal/Swrort-J	Troj/Tiny-DJ	Troj/Rozena-D
Vba32	-	-	Trojan.MSIL.PShell.gen	-
VirITExplorer	Trojan.Win32.Generic.BZPS	Trojan.Win32.Generic.BZPS	-	Backdoor.Win32.Meterpreter.BG
VirusFighter	ATK/Meter-A	ATK/Swrort-J	ATK/FatRat-H	ATK/Dloadr-EFR
ZoneAlarm	HEUR:Trojan.Win64.Packed.gen	HEUR:Trojan.Win64.Packed.gen	HEUR:Trojan-Downloader.Win32.Generic	HEUR:Trojan.Win32.Generic

5.6.2. Proposed Method

A detailed report is contained in Appendix B.3. The data shown in Table 4 and Table 5 indicate that using encryption alone, without further evasion techniques, enhances the efficacy of avoiding detection by all AV vendors.

Table 4. Scan results for Windows x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map	Henon Map	Lorenz System	Chua’s Circuit System	Rössler System
AdAware	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
Arcabit	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
Avast	Win32:MsfShell-V	-	-	-	-	-
AVG	Win32:MsfShell-V	-	-	-	-	-
Avira	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641	HEUR/ AGEN.1216641
ClamAV	Win.Malware.Metasploit-10022275-0	-	-	-	-	-
Crowdstrike Falcon	Threat detected.	-	Threat detected.	Threat detected.	-	Threat detected.
Emsisoft	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
G Data	Generic.ShellCode.Marte.4.EFC3787B	-	-	-	-	-
IKARUS	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt
Immunet	Win.Malware.Metasploit-10022275-0	-	-	-	-	-
Microsoft Defender	Trojan:Win64/ Meterpreter.B	Trojan:Win64/ Meterpreter.B	Trojan:Win64/ Meterpreter.B	-	-	-
Norman	Win32:MsfShell-V	-	-	-	-	-
ZoneAlarm	HEUR:Trojan.Win32.Generic	-	-	-	-	-

Table 5. Scan results for Linux x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map	Henon Map	Lorenz System	Chua’s Circuit System	Rössler System
AdAware	Gen:Variant.MSILHeracles.120280	-	-	-	-	-
Avast	ELF:Agent-DDL	-	-	-	-	-
AVG	ELF:Agent-DDL	-	-	-	-	-
eScan	Gen:Variant.MSILHeracles.120280[ZP]	-	-	-	-	-
Norman	ELF:Agent-DDL	-	-	-	-	-

According to Table 6, these samples are the least favorable due to the encryption keys being consistently fixed at certain values.

Table 6. Worst case for scan results for Windows x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map
AdAware	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359F
Arcabit	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359F
Avast	Win32:MsfShell-V	Win32:MsfShell-V
AVG	Win32:MsfShell-V	Win32:MsfShell-V
Avira	HEUR/AGEN.1216641	HEUR/AGEN.121664
ClamAV	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0
Crowdstrike Falcon	Threat detected.	-
Emsisoft	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359
G Data	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.7E57359F
IKARUS	Trojan.MSIL.Crypt	Trojan.MSIL.Cryp
Immunet	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0-
Microsoft Defender	Trojan:Win64/Meterpreter.B	Trojan:Win64/Meterpreter.B
Norman	Win32:MsfShell-V	Win32:MsfShell-V
ZoneAlarm	HEUR:Trojan.Win32.Generic	-

According to Table 7, when the Logistic Map is applied with a 0x00 key for encryption, the resulting samples are even more vulnerable to detection by AV software. The consistent use of a zero key significantly reduces the obfuscation strength, leading to high detection rates similar to those observed with unencrypted samples.

Table 7. Worst case for scan results for Windows x64 staged reverse TCP shellcode before and after encryption.

Vendors	Original	Logistic Map with 0x00 Keys
AdAware	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
Arcabit	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
Avast	Win32:MsfShell-V	Win32:MsfShell-V
AVG	Win32:MsfShell-V	Win32:MsfShell-V
Avira	HEUR/AGEN.1216641	HEUR/AGEN.1216641
ClamAV	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0
Crowdstrike Falcon	Threat detected.	-
Emsisoft	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
G Data	Generic.ShellCode.Marte.4.EFC3787B	Generic.ShellCode.Marte.4.1CC7DB19
IKARUS	Trojan.MSIL.Crypt	Trojan.MSIL.Crypt
Immunet	Win.Malware.Metasploit-10022275-0	Win.Malware.Metasploit-10022275-0
Microsoft Defender	Trojan:Win64/Meterpreter.B	Trojan:Win64/Meterpreter.B
Norman	Win32:MsfShell-V	Win32:MsfShell-V
ZoneAlarm	HEUR:Trojan.Win32.Generic	-

This study highlights the efficacy of the proposed chaotic encryption methods in evading antivirus detection. We have expanded the results to present detailed detection outcomes across various AV engines, emphasizing the successes and limitations of each chaotic encryption system for enhanced clarity in the results. The logistic map encryption with a zero key exemplifies a worst-case scenario, highlighting its increased vulnerability to AV detection resulting from inadequate obfuscation. The findings demonstrate the comparative efficacy of each chaotic encryption method with the corresponding detection rates from various antivirus vendors. This enhances information security in penetration testing by providing more effective techniques for evading antivirus systems. The analysis and enhanced obfuscation techniques provide important insights for cybersecurity professionals, especially those involved in penetration testing, as they aim to comprehend and address potential detection mechanisms.

6. Conclusions

In the current research, we explored the use of chaotic systems for shellcode encryption as a novel method of preventing AV detection. Traditional AV systems, which rely primarily on signature-based and heuristic methods, have been increasingly challenged by sophisticated methods of evasion such as polymorphic and metamorphic malware, obfuscation, and packing. Our research focuses on using chaotic maps as an obfuscation strategy for shellcode encryption, taking advantage of the inherent unpredictability and sensitivity to the beginning factors of chaotic systems.

The experimental findings show that chaotic systems, such as the logistic map and Henon map, have promising potential to generate random encryption patterns that can evade static AV detection techniques. By adding chaos theory into malware evasion tactics, we demonstrated how a chaotic system may successfully conceal shellcode, making detection by modern AV systems harder. This method additionally emphasizes the lightweight character of the chaotic system, making it suitable for resource-constrained situations.

Overall, this study contributes to the developing subject of malware evasion by providing a fresh perspective on the exploitation of nonlinear dynamic systems such as chaos theory. The results support the potential of chaotic systems to improve malware stealthiness, but further investigations in varying real-world scenarios are needed.

7. Future Work

Although this research has shown encouraging results with chaotic systems for shellcode encryption, various possibilities for further study remain.

7.1. Dynamic Malware Analysis

Chaotic encryption should be examined in order to prevent behavioral detection because AV systems use dynamic analysis in sandboxes. Chaos systems’ persistence against contemporary AV solutions may be tested under more clearly defined and varied sandbox limitations.

7.2. Real-World Deployment

By circumventing sophisticated AV systems, red teams might imitate advanced threats and assess organizational defenses [52]. Security experts might test their networks and AV systems’ resistance to elaborate obfuscation by adding chaotic encryption to Cobalt Strike or Metasploit. This method will help to improve cybersecurity offensive and defense methods.   

Appendix A.1. Encryption and Decryption

Listing A1. Pseudocode for logistic map.

Function LogisticMapEncrypt(data, r, x0)       Initialize x = x0       Initialize encryptedData as a byte array of the same length as data       Initialize keys as an integer array of the same length as data       For each index i in data             Repeat                   Update x as r * x * (1 - x)                   Set chaoticValue as the integer part of x * 255             Until chaoticValue is not equal to 0             Store chaoticValue in keys at position i             Set encryptedData at position i to data[i] XOR chaoticValue       End For       Return encryptedData and keys End Function

Listing A2. Pseudocode for Henon map.

Function HenonMapEncrypt(data, a, b, x0, y0)       Initialize x = x0       Initialize y = y0       Initialize encryptedData as a byte array of the same length as data       Initialize keys as an integer array of the same length as data       For each index i in data             Repeat                   Calculate newX as 1 - a * x * x + y                   Calculate newY as b * x                   Set chaoticValue as the integer part of newX * 255             Until chaoticValue is not equal to 0             Store chaoticValue in keys at position i             Set encryptedData at position i to data[i] XOR chaoticValue             Update x to newX             Update y to newY       End For       Return encryptedData and keys End Function

Listing A3. Pseudocode for Lorenz system.

Function LorenzSystemEncrypt(data, sigma, rho, beta, x0, y0, z0)       Initialize x = x0, y = y0, z = z0       Set time step dt = 0.01       Initialize encryptedData as a byte array of the same length as data       Initialize keys as an integer array of the same length as data       For each index i in data             Repeat                   // Calculate RK4 steps using LorenzDerivatives                   (k1x, k1y, k1z) = LorenzDerivatives(x, y, z, sigma, rho, beta)                   (k2x, k2y, k2z) = LorenzDerivatives(x + 0.5 * k1x * dt, y + 0.5 * k1y * dt, z + 0.5 * k1z * dt, sigma, rho, beta)                   (k3x, k3y, k3z) = LorenzDerivatives(x + 0.5 * k2x * dt, y + 0.5 * k2y * dt, z + 0.5 * k2z * dt, sigma, rho, beta)                   (k4x, k4y, k4z) = LorenzDerivatives(x + k3x * dt, y + k3y * dt, z + k3z * dt, sigma, rho, beta)                   // Update x, y, z using RK4 formula                   x = x + (dt / 6) * (k1x + 2 * k2x + 2 * k3x + k4x)                   y = y + (dt / 6) * (k1y + 2 * k2y + 2 * k3y + k4y)                   z = z + (dt / 6) * (k1z + 2 * k2z + 2 * k3z + k4z)                   Set chaoticValue as integer part of x * 255             Until chaoticValue is not equal to 0             Store chaoticValue in keys at position i             Set encryptedData at position i to data[i] XOR chaoticValue       End For       Return encryptedData and keys End Function Function LorenzDerivatives(x, y, z, sigma, rho, beta)       Set dx = sigma * (y - x)       Set dy = x * (rho - z) - y       Set dz = x * y - beta * z       Return (dx, dy, dz) End Function

Listing A4. Pseudocode for Chua’s circuit system encryption.

Function ChuasCircuitEncrypt(data, alpha, beta, m0, m1, x0, y0, z0)       Initialize x = x0, y = y0, z = z0       Set time step dt = 0.01       Initialize encryptedData as a byte array of the same length as data       Initialize keys as an integer array of the same length as data       For each index i in data             Repeat                   // Calculate RK4 steps using ChuaDerivatives                   (k1x, k1y, k1z) = ChuaDerivatives(x, y, z, alpha, beta, m0, m1)                   (k2x, k2y, k2z) = ChuaDerivatives(x + 0.5 * k1x * dt, y + 0.5 * k1y * dt, z + 0.5 * k1z * dt, alpha, beta, m0, m1)                   (k3x, k3y, k3z) = ChuaDerivatives(x + 0.5 * k2x * dt, y + 0.5 * k2y * dt, z + 0.5 * k2z * dt, alpha, beta, m0, m1)                   (k4x, k4y, k4z) = ChuaDerivatives(x + k3x * dt, y + k3y * dt, z + k3z * dt, alpha, beta, m0, m1)                   // Update x, y, and z using RK4 formula                   x = x + (dt / 6) * (k1x + 2 * k2x + 2 * k3x + k4x)                   y = y + (dt / 6) * (k1y + 2 * k2y + 2 * k3y + k4y)                   z = z + (dt / 6) * (k1z + 2 * k2z + 2 * k3z + k4z)                   Set chaoticValue as integer part of x * 255             Until chaoticValue is not equal to 0             Store chaoticValue in keys at position i             Set encryptedData at position i to data[i] XOR chaoticValue       End For       Return encryptedData and keys End Function Function ChuaDerivatives(x, y, z, alpha, beta, m0, m1)       // Piecewise-linear function g(x) for Chua’s diode       If x <= -1             Set g = m1 * x + (m1 - m0)       Else if x >= 1             Set g = m1 * x + (m0 - m1)       Else             Set g = m0 * x       End If       // Derivatives of the Chua’s Circuit equations       Set dx = alpha * (y - x - g)       Set dy = x - y + z       Set dz = -beta * y       Return (dx, dy, dz) End Function

Listing A5. Pseudocode for Rössler system encryption.

Function RosslerSystemEncrypt(data, a, b, c, x0, y0, z0)       Initialize x = x0, y = y0, z = z0       Set time step dt = 0.01       Initialize encryptedData as a byte array of the same length as data       Initialize keys as an integer array of the same length as data       For each index i in data             Repeat                   // Calculate RK4 steps using RosslerDerivatives                   (k1x, k1y, k1z) = RosslerDerivatives(x, y, z, a, b, c)                   (k2x, k2y, k2z) = RosslerDerivatives(x + 0.5 * k1x * dt, y + 0.5 * k1y * dt, z + 0.5 * k1z * dt, a, b, c)                   (k3x, k3y, k3z) = RosslerDerivatives(x + 0.5 * k2x * dt, y + 0.5 * k2y * dt, z + 0.5 * k2z * dt, a, b, c)                   (k4x, k4y, k4z) = RosslerDerivatives(x + k3x * dt, y + k3y * dt, z + k3z * dt, a, b, c)                   // Update x, y, z using RK4 formula                   x = x + (dt / 6) * (k1x + 2 * k2x + 2 * k3x + k4x)                   y = y + (dt / 6) * (k1y + 2 * k2y + 2 * k3y + k4y)                   z = z + (dt / 6) * (k1z + 2 * k2z + 2 * k3z + k4z)                   Set chaoticValue as integer part of x * 255             Until chaoticValue is not equal to 0             Store chaoticValue in keys at position i             Set encryptedData at position i to data[i] XOR chaoticValue       End For       Return encryptedData and keys End Function Function RosslerDerivatives(x, y, z, a, b, c)       Set dx = -y - z       Set dy = x + a * y       Set dz = b + z * (x - c)       Return (dx, dy, dz) End Function

Appendix A.2. Executing Shellcode

Listing A6. Pseudocode for executing shellcode on Windows.

Function ExecuteShellcode()       // Load necessary Windows API functions       Load GetProcAddress from kernel32.dll       Load GetModuleHandleA from kernel32.dll       // Define delegates for system calls       Define RtlMoveMemoryDelegate as a pointer to RtlMoveMemory function       Define VirtualProtectDelegate as a pointer to VirtualProtect function       Define CreateThreadDelegate as a pointer to CreateThread function       Define WaitForSingleObjectDelegate as a pointer to WaitForSingleObject function       Set PAGE_EXECUTE_READWRITE = 0x40       // Initialize shellcode or encrypted data       Set data as byte array with shellcode       If shellcode is encrypted by a chaotic system like Lorenz, decrypt data using LorenzSystemEncrypt(…)       If shellcode is original (e.g., from msfvenom), use data directly for execution.       // Obtain addresses for required functions       Set kernel32Handle = GetModuleHandleA("kernel32.dll")       Set virtualProtectAddr = GetProcAddress(kernel32Handle, "VirtualProtect")       Set createThreadAddr = GetProcAddress(kernel32Handle, "CreateThread")       Set waitForSingleObjectAddr = GetProcAddress(kernel32Handle, "WaitForSingleObject")       // Create delegate instances for system calls       Set VirtualProtect = Create delegate for VirtualProtect using virtualProtectAddr       Set CreateThread = Create delegate for CreateThread using createThreadAddr       Set WaitForSingleObject = Create delegate for WaitForSingleObject using waitForSingleObjectAddr       // Allocate memory for shellcode       Set addr = Allocate memory of size data.Length       // Copy shellcode data to allocated memory       Copy data to addr       // Change memory protection to PAGE_EXECUTE_READWRITE       Call VirtualProtect on addr with size data.Length, new protection PAGE_EXECUTE_READWRITE       // Create a thread to execute shellcode       Set hthread = Call CreateThread with start address addr       // Wait for thread to complete       Call WaitForSingleObject with hthread and timeout 0xFFFFFFFF End Function

Listing A7. Pseudocode for executing shellcode on Linux.

Function ExecuteShellcodeLinux()       // Load necessary Linux system functions       Load mmap from libc.so.6       Load mprotect from libc.so.6       // Define constants for memory protection and mapping flags       Set PROT_READ = 0x1       Set PROT_WRITE = 0x2       Set PROT_EXEC = 0x4       Set MAP_PRIVATE = 0x02       Set MAP_ANONYMOUS = 0x20       // Initialize shellcode or encrypted data       Set data as byte array with shellcode       If shellcode is encrypted by a chaotic system like Lorenz, decrypt data using LorenzSystemEncrypt(…)       If shellcode is original (e.g., from msfvenom), use data directly for execution.       // Allocate memory with read, write, and execute permissions       Set size = data.Length       Set addr = mmap(0, size, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0)       // Copy shellcode data to allocated memory       Copy data to addr       // Change memory protection to read and execute only       Call mprotect on addr with size and PROT_READ | PROT_EXEC       // Create delegate to execute shellcode       Set dataDelegate = Create delegate for data at addr       // Execute shellcode       Call dataDelegate() End Function

Appendix B.1. Disassembly

Listing A8. Disassembly of staged Windows shellcode in 64 bits.
0:	fc	cld
1:	48 83 e4 f0	and	rsp,0xfffffffffffffff0
5:	e8 cc 00 00 00	call	0xd6
a:	41 51	push	r9
c:	41 50	push	r8
e:	52	push	rdx
f:	51	push	rcx
10:	56	push	rsi
11:	48 31 d2	xor	rdx,rdx
14:	65 48 8b 52 60	mov	rdx,QWORD PTR gs:[rdx+0x60]
19:	48 8b 52 18	mov	rdx,QWORD PTR [rdx+0x18]
1d:	48 8b 52 20	mov	rdx,QWORD PTR [rdx+0x20]
21:	4d 31 c9	xor	r9,r9
24:	48 8b 72 50	mov	rsi,QWORD PTR [rdx+0x50]
28:	48 0f b7 4a 4a	movzx	rcx,WORD PTR [rdx+0x4a]
2d:	48 31 c0	xor	rax,rax
30:	ac	lods	al,BYTE PTR ds:[rsi]
31:	3c 61	cmp	al,0x61
33:	7c 02	jl	0x37
35:	2c 20	sub	al,0x20
37:	41 c1 c9 0d	ror	r9d,0xd
3b:	41 01 c1	add	r9d,eax
3e:	e2 ed	loop	0x2d
40:	52	push	rdx

Listing A9. Disassembly of staged Linux shellcode in 64 bits.
0:	31 ff	xor	edi,edi
2:	6a 09	push	0x9
4:	58	pop	rax
5:	99	cdq
6:	b6 10	mov	dh,0x10
8:	48 89 d6	mov	rsi,rdx
b:	4d 31 c9	xor	r9,r9
e:	6a 22	push	0x22
10:	41 5a	pop	r10
12:	6a 07	push	0x7
14:	5a	pop	rdx
15:	0f 05	syscall
17:	48 85 c0	test	rax,rax
1a:	78 51	js	0x6d
1c:	6a 0a	push	0xa
1e:	41 59	pop	r9
20:	50	push	rax
21:	6a 29	push	0x29
23:	58	pop	rax
24:	99	cdq

Listing A10. Disassembly of encrypted Windows shellcode in 64 bits.
0:	02 4a 89	add	cl,BYTE PTR [rdx-0x77]
3:	c2 72 16	ret	0x1672
6:	ce	(bad)
7:	0b 2c 93	or	ebp,DWORD PTR [rbx+rdx*4]
a:	b9 4b 1e be 6f	mov	ecx,0x6fbe1e4b
f:	eb 91	jmp	0xffffffffffffffa2
11:	e5 ec	in	eax,0xec
13:	a1 99 43 a0 dd 9a 5b	movabs	eax,ds:0x99cd5b9adda04399
1a:	cd 99
1c:	bb a2 c0 81 af	mov	ebx,0xaf81c0a2
21:	b7 23	mov	bh,0x23
23:	8c 81 23 91 30 a7	mov	WORD PTR [rcx-0x58cf6edd],es
29:	36 06	ss (bad)
2b:	9d	popf
2c:	ce	(bad)
2d:	b6 32	mov	dh,0x32
2f:	cf	iret
30:	96	xchg	esi,eax
31:	8f	(bad)
32:	b2 f3	mov	dl,0xf3
34:	f8	clc
35:	3d 61 80 7b 01	cmp	eax,0x17b8061
3a:	a7	cmps	DWORD PTR ds:[rsi],DWORD PTR es:[rdi]
3b:	a1 6b 36 fc 87 a5 5f	movabs	eax,ds:0xbf3b5fa587fc366b
42:	3b bf
44:	96	xchg	esi,eax

Listing A11. Disassembly of encrypted Linux shellcode in 64 bits.
0:	cf	iret
1:	fd	std
2:	60	(bad)
3:	2f	(bad)
4:	da 67 b4	fisub	DWORD PTR [rdi-0x4c]
7:	1b 64 1a 2e	sbb	esp,DWORD PTR [rdx+rbx*1+0x2e]
b:	57	push	rdi
c:	6e	outs	dx,BYTE PTR ds:[rsi]
d:	27	(bad)
e:	57	push	rdi
f:	98	cwde
10:	86 f7	xchg	bh,dh
12:	b7 74	mov	bh,0x74
14:	a6	cmps	BYTE PTR ds:[rsi],BYTE PTR es:[rdi]
15:	04 2e	add	al,0x2e
17:	c7	(bad)
18:	7f d3	jg	0xffffffffffffffed
1a:	3e 9a	ds (bad)
1c:	c9	leave
1d:	e0 0a	loopne	0x29
1f:	8a df	mov	bl,bh
21:	90	nop
22:	3b 1d 50 c2 e1 3f	cmp	ebx,DWORD PTR [rip+0x3fe1c250]        # 0x3fe1c278
28:	85 38	test	DWORD PTR [rax],edi

Listing A12. Disassembly of encrypted Windows shellcode in 64 bits including 0x00 keys.
0:	ea	(bad)
1:	4f 81 e4 f0 e8 cc 00	rex.WRXB and r12,0xcce8f0
8:	00 00	add	BYTE PTR [rax],al
a:	41 51	push	r9
c:	41 50	push	r8
e:	52	push	rdx
f:	51	push	rcx
10:	56	push	rsi
11:	48 31 d2	xor	rdx,rdx
14:	65 48 8b 52 60	mov	rdx,QWORD PTR gs:[rdx+0x60]
19:	48 8b 52 18	mov	rdx,QWORD PTR [rdx+0x18]
1d:	48 8b 52 20	mov	rdx,QWORD PTR [rdx+0x20]
21:	4d 31 c9	xor	r9,r9
24:	48 8b 72 50	mov	rsi,QWORD PTR [rdx+0x50]
28:	48 0f b7 4a 4a	movzx	rcx,WORD PTR [rdx+0x4a]
2d:	48 31 c0	xor	rax,rax
30:	ac	lods	al,BYTE PTR ds:[rsi]
31:	3c 61	cmp	al,0x61
33:	7c 02	jl	0x37
35:	2c 20	sub	al,0x20
37:	41 c1 c9 0d	r9d,0xd
3b:	41 01 c1	add	r9d,eax
3e:	e2 ed	loop	0x2d
40:	52	push	rdx

Appendix B.2. Shellcode Generation

Figure A1. Generation of staged Windows shellcode in 64 bits using msfvenom.

Figure A2. Generation of staged Linux shellcode in 64 bits using msfvenom.

Appendix B.3. Scan Report

All URLs in Appendix B.3 are accessed on 4 November 2024.

Table A1. The scan report for Windows staged malware with other obfuscation tools from Kleenscan.

Notes	Scan Result URL
Msfvenom	https://kleenscan.com/scan_result/946515ebfb97c794a3fbdf8b253d847d6830b2f5c7b7b5f592095dd56322abaf
Encode by shigata_ga_nai	https://kleenscan.com/scan_result/0ed3421008a9792bd4e5c1404b6448ef860251e8f3301f51764bc3488c90373a
Pwnwind in Fatrat	https://kleenscan.com/scan_result/0cd88320ee1aed918efcf8e78c7c1da43d27a528811fd5ba71ca9542fdceb8bc
Veil Framework	https://kleenscan.com/scan_result/f36b017c19229c6135654e16c7776d6d99f1bf848c51117423cb3965830a0e3e

Table A1. The scan report for Windows staged malware with other obfuscation tools from Kleenscan.

Notes	Scan Result URL
Msfvenom	https://kleenscan.com/scan_result/946515ebfb97c794a3fbdf8b253d847d6830b2f5c7b7b5f592095dd56322abaf
Encode by shigata_ga_nai	https://kleenscan.com/scan_result/0ed3421008a9792bd4e5c1404b6448ef860251e8f3301f51764bc3488c90373a
Pwnwind in Fatrat	https://kleenscan.com/scan_result/0cd88320ee1aed918efcf8e78c7c1da43d27a528811fd5ba71ca9542fdceb8bc
Veil Framework	https://kleenscan.com/scan_result/f36b017c19229c6135654e16c7776d6d99f1bf848c51117423cb3965830a0e3e

Table A2. The scan report for Windows x64 staged shellcode from Kleenscan.

Notes	Scan Result URL
No Encryption Shellcode	https://kleenscan.com/scan_result/2a79cfe73f5114a13d7e9139f82b768b6a690ff844684ea99fce8d98532cc911
Logistic Map Encryption	https://kleenscan.com/scan_result/9a6ca9a7221675c91088baa0e92e62699608b248b1a78aec11ebf49b9ac38ded
Henon Map Encryption	https://kleenscan.com/scan_result/70fb5b29d61e2064a8e30fea1ab4e3b547eff7fef204351778d4125bbeaa19b2
Lorenz System Encryption	https://kleenscan.com/scan_result/fb28f0c703418878a45b147df853af92ff931e878bf4258b0f5af70017c2a38a
Chua’s Circuit System Encryption	https://kleenscan.com/scan_result/f1b3bab9edfb2f3720baeb39d4015bc95ae2ff22aa2e6454a5c1252d9babd781
Rössler System	https://kleenscan.com/scan_result/200b9570e6ad1cc84cf9f6afb953b43e95eb24ae05ced954ad8e2be8b6d7ae6f

Table A2. The scan report for Windows x64 staged shellcode from Kleenscan.

Notes	Scan Result URL
No Encryption Shellcode	https://kleenscan.com/scan_result/2a79cfe73f5114a13d7e9139f82b768b6a690ff844684ea99fce8d98532cc911
Logistic Map Encryption	https://kleenscan.com/scan_result/9a6ca9a7221675c91088baa0e92e62699608b248b1a78aec11ebf49b9ac38ded
Henon Map Encryption	https://kleenscan.com/scan_result/70fb5b29d61e2064a8e30fea1ab4e3b547eff7fef204351778d4125bbeaa19b2
Lorenz System Encryption	https://kleenscan.com/scan_result/fb28f0c703418878a45b147df853af92ff931e878bf4258b0f5af70017c2a38a
Chua’s Circuit System Encryption	https://kleenscan.com/scan_result/f1b3bab9edfb2f3720baeb39d4015bc95ae2ff22aa2e6454a5c1252d9babd781
Rössler System	https://kleenscan.com/scan_result/200b9570e6ad1cc84cf9f6afb953b43e95eb24ae05ced954ad8e2be8b6d7ae6f

Table A3. The scan report for Linux x64 staged shellcode from Kleenscan.

Notes	Scan Result URL
No Encryption Shellcode	https://kleenscan.com/scan_result/d19cfebe80fc46ede1cca891b27bb99f147458292d1023788362e9c08b08f41a
Logistic Map Encryption	https://kleenscan.com/scan_result/070b4889528b78a3e58a80fe313a2fd5823b5473c8ae91cbd8fe6603c65d8910
Henon Map Encryption	https://kleenscan.com/scan_result/cc5c9c218ac16489c55d18cb56883f7e994216cfea63763e26ab419fd7ca5cd7
Lorenz System Encryption	https://kleenscan.com/scan_result/7a50c86fc750d8982712a14b3c3b0860d39e569eb56bf542279335fb69b3b7c9
Chua’s Circuit System Encryption	https://kleenscan.com/scan_result/99fdc85e2f28d0d0f3cfec9f02b0fb9534a6d195af3d4171ae9d0c82b28181aa
Rössler System	https://kleenscan.com/scan_result/909e99b779650af2fa6f4a5c4ef5e16f09f558151e66d7308eed2ac50a29f989

Table A3. The scan report for Linux x64 staged shellcode from Kleenscan.

Notes	Scan Result URL
No Encryption Shellcode	https://kleenscan.com/scan_result/d19cfebe80fc46ede1cca891b27bb99f147458292d1023788362e9c08b08f41a
Logistic Map Encryption	https://kleenscan.com/scan_result/070b4889528b78a3e58a80fe313a2fd5823b5473c8ae91cbd8fe6603c65d8910
Henon Map Encryption	https://kleenscan.com/scan_result/cc5c9c218ac16489c55d18cb56883f7e994216cfea63763e26ab419fd7ca5cd7
Lorenz System Encryption	https://kleenscan.com/scan_result/7a50c86fc750d8982712a14b3c3b0860d39e569eb56bf542279335fb69b3b7c9
Chua’s Circuit System Encryption	https://kleenscan.com/scan_result/99fdc85e2f28d0d0f3cfec9f02b0fb9534a6d195af3d4171ae9d0c82b28181aa
Rössler System	https://kleenscan.com/scan_result/909e99b779650af2fa6f4a5c4ef5e16f09f558151e66d7308eed2ac50a29f989

Table A4. The worst case for the scan report for Windows x64 staged shellcode from Kleenscan.

Notes	Scan Result URL
No Encryption Shellcode	https://kleenscan.com/scan_result/d19cfebe80fc46ede1cca891b27bb99f147458292d1023788362e9c08b08f41a
Logistic Map Encryption	https://kleenscan.com/scan_result/7866ed2f5d852c5f910013cded9fbc5bc9c5efc496c02f4d18cfd190d7de7961
Logistic Map Encryption with 0x00 keys	https://kleenscan.com/scan_result/48553fa830bd3b124b6ae501e2d53e1a2043e585f8bf763176211b502190f597

Table A4. The worst case for the scan report for Windows x64 staged shellcode from Kleenscan.

Notes	Scan Result URL
No Encryption Shellcode	https://kleenscan.com/scan_result/d19cfebe80fc46ede1cca891b27bb99f147458292d1023788362e9c08b08f41a
Logistic Map Encryption	https://kleenscan.com/scan_result/7866ed2f5d852c5f910013cded9fbc5bc9c5efc496c02f4d18cfd190d7de7961
Logistic Map Encryption with 0x00 keys	https://kleenscan.com/scan_result/48553fa830bd3b124b6ae501e2d53e1a2043e585f8bf763176211b502190f597