Abstract
In this study, we propose a method for successfully evading antivirus detection by encoding malicious shellcode with fountain codes. The Meterpreter framework for Microsoft Windows 32-bit and 64-bit architectures was used to produce the shellcode used in this investigation. The experimental results proved that detection rates were substantially decreased. Specifically, the number of detected instances using antivirus vendors for 32-bit shellcode decreased from 18 to 3, while for 64-bit shellcode, it decreased from 16 to 1. This method breaks up a malicious payload into many packets, each with their own distinct structure, and then encodes them. This obfuscation approach maintains the shellcode’s integrity, ensuring correct code execution. However, in the persistence phase of the penetration testing process, this method offers an additional means of evading antivirus techniques.
1. Introduction
Global internet users interact with software, retrieve information from search engines, and engage in entertainment due to the internet’s swift proliferation. Cybercriminals attack unsecured systems via the internet, jeopardizing user privacy and data integrity. Despite the prevalent utilization of antivirus software, innovative malware methodologies frequently circumvent detection, rendering systems susceptible to criminal access. Organizations often use endpoint detection software in conjunction with additional security measures to safeguard assets. Nonetheless, the deployment of hardware and software may be inadequate for preventing and identifying all destructive tactics employed by hackers. An increasing number of security specialists now perform penetration testing to identify systems’ vulnerabilities. During this process, specialists may exploit weaknesses to establish backdoors and maintain access []. However, they will face increasing technological challenges in doing so due to the continuous innovation of antivirus software technologies. Attack and defense are in conflict with one other in this respect [].
Reconnaissance, weaponization, delivery, exploitation, installation, command and control (C2), and actions form the framework often known as the Cyber Kill Chain []. This model accounts for the assault life-cycle and the moments at which defense techniques may be executed successfully. Both attackers and cybersecurity experts can gather open-source intelligence, or OSINT, to identify possible risks and vulnerabilities []. Although cybersecurity tools can detect multiple payloads from the popular Exploit-DB [], an issue-free system cannot be guaranteed []. The Cyber Kill Chain’s installation phase is the subject of the current study []. To improve shellcode obfuscation and lower antivirus detection rates, this study presents a method for encoding shellcodes using fountain codes.
This study emphasizes the application of fountain codes to evade antivirus measures for penetration testing and red team evaluations. Although these approaches may illustrate the constraints of existing security solutions, they also present considerable ethical dilemmas. The possible abuse of these tools for nefarious ends requires their rigorous regulation and supervision. Researchers and practitioners must guarantee these technologies’ responsible use, emphasizing the enhancement of cybersecurity defenses rather than facilitating exploitation.
The structure of this study is as follows: Section 2 reviews current advancements in shellcode obfuscation techniques and fountain code applications. Section 3 details the proposed methodology, focusing on the encoding and decoding processes using fountain codes. Section 4 presents the experimental setup and results, including shellcode generation, memory allocation, detection rates, and entropy analysis. Section 5 analyzes the limitations, decoding process, and security implications of the findings. Finally, Section 6 summarizes the contributions and proposes potential directions for future research.
2. Related Studies
2.1. Evasion and Detection Technique
Mechanisms for antivirus detection employ signatures, behavioral detection, heuristic detection, and sandboxing to recognize malware []. Antivirus evasion mechanisms used by hackers involve techniques such as inserting meaningless code and modifying signatures, obfuscating payloads, and encrypting malicious code using XOR. Additionally, they utilize polymorphisms with complex methods to complicate analysis, as well as process injection, through DLL injections or shellcode []. Current antivirus evasion methods include obfuscation, polymorphism, and encryption []. Obfuscation modifies the appearance of code without altering its functionality [], while polymorphism ensures that each payload instance differs in structure [].
Geng et al. (2024) thoroughly surveyed malware evasion techniques, classifying them into three categories: transformation-based, concealment-based, and attack-based strategies. The authors proposed a strategy-driven framework that integrated these techniques from the viewpoint of malware authors, illustrating how attackers employ various methods to circumvent detection. The study identified trends in evasion techniques, emphasizing the challenges faced by defenders and proposing future research directions to improve malware detection capabilities [].
The shikata_ga_nai and powershell_base64 tools from msfvenom’s encoder are widely recognized for their ability to evade signature-based antivirus systems through polymorphic XOR encryption []. Similarly, Veil-Evasion generates diverse payloads in multiple languages (e.g., Python, C, and PowerShell) using advanced obfuscation techniques. Other methods, such as converting malicious code into ROP devices to inject PE headers [,], and tools like TheFatRat, combining encryption with anti-debugging strategies, further highlight the evolution of antivirus evasion techniques [].
Control flow manipulation techniques, such as thread-hiding, suspending-threads, multi-threading, and self-debugging, bypass anti-debugging mechanisms by concealing debuggers and altering malware execution []. Past research on payload attack vectors for Windows and Linux highlights simplified malware injection techniques, such as ShellSwap, which utilizes jmp instructions for shellcode hopping, achieving an 88% success rate in evading detection [].
Another study proposed the use of Multifaceted Deep Generative Adversarial Networks (MDGANs) to detect mobile malware, leveraging grayscale images from Android APKs, and API call sequences processed through GoogleNet and LSTM. The GAN-generated data enhanced the training set, achieving a 96.2% classification accuracy, surpassing the accuracy rates of existing malware detection frameworks []. While the proposed method leverages GANs to enhance training data, it heavily relies on existing malware datasets.
2.2. Fountain Code
Fountain codes, such as Luby Transform (LT) codes and Raptor codes, are engineered for efficient data transfer []. LT codes employ probabilistic methods to produce random data segments, facilitating effective decoding with a small number of packets. Raptor codes enhance LT codes by incorporating a preprocessing phase, which increases decoding efficiency and minimizes the necessary packet count, rendering both techniques appropriate for high-loss situations [,]. Raptor codes augment LT codes via preprocessing, thus improving decoding success rates and enhancing transmission efficiency []. These techniques proficiently address data loss and enhance performance in transmission systems, especially in contexts with unstable networks.
These are essential in network communication, especially in contexts with significant packet loss, such as wireless networks and peer-to-peer systems []. Their capacity to produce an unlimited number of encoded packets makes them exceptionally effective in unstable contexts, such as multicast and broadcast networks []. Decoding may commence upon receiving a sufficient quantity of packets, thus minimizing latency and enhancing data recovery efficacy. These codes are particularly efficient for real-time multimedia streaming, facilitating seamless video playing over shaky connections via incremental decoding []. Moreover, they are extensively utilized in cloud storage and content distribution networks (CDNs), as they guarantee dependable transmission and enhance storage management []. Fountain codes are, thus, crucial in contemporary network systems, as they reduce packet loss and retransmissions [].
3. Methodology
3.1. Procedure
Algorithm 1 depicts the encoding process utilizing fountain codes. To augment data unpredictability and randomness, the procedure began by encoding the original shellcode while incorporating redundancies. Then, the Fisher–Yates shuffle was utilized to randomize the order of the created packets []. This randomization guaranteed that the data could be divided into an unlimited number of encoded packets, ensuring strong obfuscation against detection.
| Algorithm 1: Shellcode encoding and validation using fountain code. |
Data: Original Shellcode Data Result: Validated and High-Entropy Encoded Shellcode  |
3.2. Encode and Decode
Fountain codes improve data transmission efficiency by generating encoded packets through random slice selection and XOR operations. Our method streamlined the encoding process by randomly selecting integers from within a range based on the number of slices (X). This approach balanced simplicity and performance, ensuring obfuscation and effective antivirus evasion, regardless of possible data loss.
Algorithm 2 illustrates the encoding process of the simplified fountain code. The encoding procedure produced x packets, each denoted by , where I represents the set of chosen slice indices and C represents the encoded data.
Algorithm 3 shows that the encoder produces n packets , with each packet represented as and the whole collection of encoded data represented by . The decoded indices aggregates the indices from each accurately decoded slice, and the decoder operates by processing up to x slices. The encoding and decoding techniques guarantee that the reconstructed shellcode corresponds to the original shellcode P, thus maintaining data integrity.
The time complexity of the Simplified Fountain Code Encoder Algorithm (SFCEA) was primarily determined via the encryption step, which processed one packet of the encoded shellcode, resulting in a complexity of . The algorithm then selected d unique indices from X slices and performed XOR operations on each slice, yielding a total complexity of , where l represents the slice size.
For the Simplified Fountain Code Decoder Algorithm (SFCDA), the determination of the time complexity involved processing each packet through XOR operations and decryption. This resulted in a complexity of . Both algorithms were scaled to reflect the size of the data being processed.
Although more advanced methods such as the Robust Soliton Distribution or preprocessing in Raptor codes can enable optimization, this simplified approach maintained a hierarchical structure, resulting in an overall time complexity of . This design choice prioritized simplicity and comprehensibility, making it suitable for shellcode production, albeit with higher computational overhead.
| Algorithm 2: Simplified Fountain Code Encoder (SFCEA) |
Input: P: byte array of shellcode, l: slice size (in bytes), s: encryption seed Output: Encoded packet where I: slice indices; C: encoded data
|
| Algorithm 3: Simplified Fountain Code Decoder Algorithm (SFCDA) |
Input: : received encoded packets; l: size of each slice; s: encryption seed; t: total size of the shellcode Output: : Byte array for the decoded plaintext shellcode
|
3.3. Evaluating Data Randomness
To determine data randomness using shellcode and encoded data, scholars often use entropy [,]. The entropy method used in this study for determining the shellcode was previously used in [].
To measure randomness, the normalized Shannon entropy was used to evaluate the unpredictability of encoded data:
where
Higher entropy indicates greater obfuscation and complexity, which is essential for avoiding detection.
4. Experiment and Results
4.1. Shellcode Generation and Execution
For our experiments, we used msfvenom, a payload generation tool from the Metasploit framework, to create the shellcode. This tool is commonly employed in penetration testing and research, enabling the customization of payload type, architecture, and output format []. The generated shellcode was executed using C# within the .NET Framework 4.7.2 [].
The shellcode execution workflow used followed a series of steps, as detailed in Appendix A.3:
- Memory Allocation: The VirtualAlloc function reserves memory withPAGE_EXECUTE_READWRITE permissions to prepare an execution environment for the shellcode;
- Shellcode Loading: Marshal.Copy transfers the shellcode into the allocated memory;
- Thread Creation: A new thread is created using CreateThread, with its entry point set to the shellcode memory address;
- Thread Synchronization: The WaitForSingleObject function ensures that the thread completes execution before the process terminates.
This process ensured secure and efficient shellcode execution while maintaining precise control over memory and thread management.
4.2. Encoding Successes and Failures Compared
This experiment created shellcode using the msfvenom command, as shown in Table 1. The LHOST argument represents the attack’s IP address.
Table 1.
The shellcode generated from Msfvenom.
Each input parameter produced 25 samples. As seen in Table 2, Table 3, Table 4 and Table 5, short slices and insufficient packets prevented proper decoding. These results show that greater slices resulted in closer entropy between encoded and original data. An attacker could remotely control a target machine using a reverse shell. By connecting across a port on the victim’s device, the attack bypassed firewalls []. The entropy results presented summarize the impacts of varying packet counts and slice sizes on the encoding and decoding success rates, as well as on the entropy of the encoded data. Key observations include the following:
Table 2.
Entropy results sorted by packet count and slice size for x86 Meterpreter reverse shell with lower value.
Table 3.
Entropy results sorted by packet count and slice size for x64 Meterpreter reverse shell with lower value.
Table 4.
Entropy results sorted by packet count and slice size for x86 Meterpreter reverse shell with higher value.
Table 5.
Entropy results sorted by packet count and slice size for x64 Meterpreter reverse shell with higher value.
- Redundancy and Packet Count: Increasing packet counts ( 500, 750, 1000, 2000, 3000, 4000) enhanced redundancy, improving decoding success rates. However, the improvement diminished as M exceeded , indicating that redundancy beyond this threshold provided limited additional benefits.
- Impact of Slice Size: Smaller slice sizes () led to consistent decoding failures due to higher fragmentation and insufficient redundancy. Conversely, larger slice sizes () achieved 100% decoding success, demonstrating their ability to balance redundancy and fragmentation effectively.
- Entropy Trends: Larger slice sizes increased the encoded entropy (Encode Entropy (MIN) and Encode Entropy (MAX)), achieving values closer to those of the original shellcode entropy. For example, with , Encode Entropy (MAX) remained consistently high () across varying packet counts, indicating effective data mixing.
- File Size Growth: Both packet count and slice size contributed to larger file sizes. For instance, increasing M from 1000 to 4000 nearly quadrupled the file size. This trade-off reflected the balance between ensuring decoding success and maintaining reasonable file sizes.
Table 2 and Table 3 present the lower entropy results for x86 and x64 shellcode, while Table 4 and Table 5 provide the higher entropy results. These results demonstrate the correlation between slice size and entropy, decoding success rates, and file size. The observations include the following:
- Smaller slice sizes () consistently fail to decode due to insufficient redundancy;
- Higher slice sizes () achieve optimal entropy while maintaining a balance between file size and decoding reliability.
These findings underscore the importance of parameter selection in optimizing shellcode obfuscation and detection evasion while preserving practical usability.
4.3. Scan Results for Malware
4.3.1. Well-Known Evasion Tools
To compare our approaches with the evasion tactics presented in Appendix A.1, we utilized widely recognized encoding and encryption tools. In such experiments, the direct use of the original msfvenom executable backdoor during penetration testing on systems with antivirus software is inadvisable, as it has been identified by many antivirus vendors. Moreover, multiple antivirus engines have exhibited significant decreases in detection when employing shikata_ga_nai with 10 and 20 encoding iterations. While this method effectively circumvents certain signature-based detection systems, numerous antivirus programs still categorize the encoded payload as hazardous. The 32-bit malware payload was developed in C# and designed for execution via the Veil-Evasion program (version 3.1.14). We created 64-bit malware for further comparison with PwnWind, one of TheFatRat’s evasion modules.
4.3.2. Proposed Method
All samples were subjected to testing on Kleenscan.com, a professional malware scanner that incorporates over 30 distinct antivirus vendors and does not disclose files during malware assessment []. Based on the data shown in Table 6, we concluded that the shellcode from the original file was generated via Meterpreter sourced from many vendors. Our method successfully encoded the shellcode, decreasing it from 18 to 3 and from 16 to 1. All of these findings were evaluated in October 2024.
Table 6.
Virus scan results for original and encoded shellcode.
Table 7 and Table 8 compare the detections between the original and encoded shellcode. This experiment demonstrated that, similar to the shellcode, the detection description was altered, reducing the detection rate. In the table, the antivirus providers are organized alphabetically according to the initial letter of each name. The 32-bit shellcode comprises 354 bytes with parameters defined by a seed number of , 16 slices, and 1000 packets, whereas the 64-bit shellcode consists of 510 bytes with the same parameters for seed number, slices, and packets.
Table 7.
Antivirus scan results for original and encoded (x86) shellcode in the best-case scenario.
Table 8.
Antivirus scan results for original and encoded (x64) shellcode in the best-case scenario.
Using this approach, it was possible to build encoded shellcode in a unique manner. As shown in Table 9, we used this method to generate malware that cannot scan all samples, which was ideal. As the sample that contained parameters generated via 32-bit shellcode in 354 bytes had a seed number of , 16 slices, and 250 packets, this sample was identified as dangerous using 12 different antivirus engines. Furthermore, according to Table 10, the sample in question was a 64-bit shellcode contained inside 510 bytes. It had a seed number of , 16 slices, and 2000 packets. It was identified as dangerous by five different antivirus engines. However, despite the fact that the encoded virus removed the characteristics from Msfvenom, it was necessary to combine additional tools and approaches to enhance the evasion strategy.
Table 9.
Antivirus detection results for original and encoded x86 shellcode in the worst-case scenario.
Table 10.
Antivirus detection results for original and encoded x64 shellcode in the worst-case scenario.
5. Discussion
5.1. Worst-Case Scenario Analysis
The dnSpy tool was utilized to decompile binaries created by the .NET Framework and .NET Core, enabling a deeper understanding of the encoding process []. To analyze the worst-case scenario, we used this tool to deconstruct the encoded shellcode and compare it with its original form.
Figure 1 illustrates the comparison of the original and encoded 32-bit shellcode. We observe that the initial 14 bytes of both the original and encoded shellcode are identical (in decimal format). This lack of encryption represents a significant flaw in the encoding process, as it does not obfuscate critical shellcode features effectively.
Figure 1.
A comparison of the 32-bit shellcode before and after encoding in the worst-case scenario.
Similarly, Figure 2 illustrates the comparison for the 64-bit shellcode. The first 14 bytes of the original and encoded shellcode remain unchanged (in decimal format), indicating that encoding scatters the data rather than encrypting it. This behavior, coupled with a larger slice size, generates a more extended shellcode structure, inadvertently providing antivirus software with additional features to classify the file as malicious.
Figure 2.
A comparison of 64-bit shellcode before and after encoding in the worst-case scenario.
In addition, the encoded shellcode does not sufficiently obfuscate key API call parameters, allowing security tools to identify it as malicious. As demonstrated in Figure 3 and Figure 4, the disassembled shellcode reveals clear patterns of API calls and their associated parameters. These patterns, when not sufficiently obscured, are strong indicators of malicious activity. For example, in both x86 and x64 shellcode, the parameters for memory allocation APIs such as VirtualAlloc can be directly identified. These parameters include the following:
Figure 3.
A comparison of 32-bit shellcode before and after encoding in the worst-case scenario.
Figure 4.
A comparison of 64-bit shellcode before and after encoding in the worst-case scenario.
- PAGE_EXECUTE_READWRITE (0x40), which grants executable permissions to allocated memory, a common feature of malicious payloads;
- MEM_COMMIT (0x1000), indicating the allocation type.
Additionally, the sequence of push, mov, and call instructions in the disassembly clearly prepares and invokes API calls, such as VirtualAlloc, through an indirect call ebp. This behavior is consistent with known malicious patterns in shellcode, where API addresses are resolved dynamically to evade static analysis.
- Key Observations:
- Parameter Consistency: The hardcoded values for key API parameters (e.g., 0x40, 0x1000) are easily detectable and align with typical malicious shellcode behavior;
- Indirect API Calls: The use of indirect calls (e.g., call ebp) is a common obfuscation technique but can still be identified through dynamic analysis tools or advanced heuristic detection;
- Entropy Disparity: While the encoded shellcode increases overall entropy, certain regions, such as API call parameters, retain low entropy and remain detectable.
These findings highlight a critical flaw in the encoding process: it does not fully obfuscate key API call parameters, commonly used by antivirus engines to classify the payload as malicious.
5.2. Analysis of Encoding and Decoding Processes
Successful decoding relies on the following conditions:
- The attacker must collect enough packets (at least N independent packets, where N is the number of slices);
- The attacker must know the index set of each packet;
- The attacker must know the initial seed value s used in the XOR operation.
- Challenges for attackers:
- Packet Combination Space: Given M packets, if , the number of possible combinations iswhich grows exponentially as . For example, .
- Seed Brute Force: Since the seed s is a single byte, the attacker needs to try only possibilities.
- Overall Complexity: Assuming each attempt involves solving N sparse linear equations, with a computational complexity of , the total brute-force complexity is as follows:
- Analysis of low success rate:
The decoding process depends on whether the matrix A is full-rank. For randomly selected index sets , the success probability of decoding is approximately
when , the success rate drops significantly.
- Information-Theoretic Proofs:
- Entropy Increase and Obfuscation: After fountain code encoding, the entropy H of the packet data approaches the maximum entropy :Attackers cannot extract meaningful patterns through statistical analysis.
- Irreversibility of XOR: Without knowledge of s and packet indices, attackers cannot reconstruct valid slices.
5.3. Future Studies
During testing, we tried to preserve Meterpreter’s characteristics by employing the presented strategy to eliminate distinguishing traits. Although this experiment focused on randomly encoding and creating unique samples, more strategies should be included in future studies. Encoding unique harmful features on network traffic [] requires adjusting or developing our own shellcode to identify command and control (C2), even though this can be accomplished using fileless malware []. The anti-VM approach should also be considered []. It can integrate different methods, making forensic and reverse engineering more difficult []. This approach can also be used for PowerShell scripts [] and to obtain packet data from the internet.
5.4. Limitations
Our study has specific limitations. One limitation pertains to the Microsoft Windows operating system, which requires the installation of the .NET Framework to run the virus. The .NET Framework is pre-installed on contemporary operating systems, but earlier ones require manual installation. This experiment may be executed using .NET Core using the C# programming language, and it is compatible with installation and execution on a Linux system []. The function from is analogous to the Win32 API , while from is analogous to the Win32 API .
6. Conclusions
This study utilizes fountain code to obfuscate shellcode, enhancing its resistance to detection by security systems. The proposed encoding method incorporates randomness, markedly enhancing entropy and diminishing detection rates among multiple antivirus systems and thus demonstrating its efficacy in evasion. The method streamlines decoding, rendering it appropriate for shellcode production pipelines; however, the encoding process is slower than that of more advanced techniques like LT or Raptor codes. The simplicity of fountain code facilitates its practicality and ease of integration, balancing usability and evasion capability. The results emphasize the importance of parameter tuning, especially in the optimization of the slice size, packet count, and entropy. Increased slice sizes and packet counts enhance randomness and decoding success, but they also lead to larger file sizes. This highlights the significance of selecting parameters strategically according to specific use cases, thus optimally balancing evasion effectiveness and resource efficiency.
Author Contributions
Conceptualization, G.-C.H., T.-H.L. and K.-C.C.; methodology, G.-C.H. and T.-H.L.; software, G.-C.H.; validation, G.-C.H. and T.-H.L.; formal analysis, G.-C.H.; investigation, G.-C.H. and T.-H.L.; resources, T.-H.L. and K.-C.C.; data curation, G.-C.H.; writing—original draft preparation, G.-C.H.; writing—review and editing, T.-H.L. and K.-C.C.; visualization, G.-C.H.; supervision, T.-H.L. and K.-C.C. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
Data are contained within the article.
Conflicts of Interest
The authors declare no conflicts of interest.
Appendix A. Details
Appendix A.1. Scan Details for Original and Evasion Tools
Table A1.
Antivirus scan results for original and evasion (x86) tools.
Table A1.
Antivirus scan results for original and evasion (x86) tools.
| No. | AntiVirus | Msfvenom Staged Reverse Shell (*.exe) | shikata_ga_nai with 10 Iterators | shikata_ga_nai with 20 Iterators | Veil-Evasion in C# |
|---|---|---|---|---|---|
| 1 | AdAware | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Gen:Variant.Kryptik.175 |
| 2 | Alyac | Trojan.CryptZ.Gen | Trojan.CryptZ.Gen | Trojan.CryptZ.Gen | Gen:Variant.Razy.575638 |
| 3 | Arcabit | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Trojan.Kryptik.175 |
| 4 | Avast | Win32:Meterpreter-C | Win32:SwPatch | Win32:ShikataGaNai-C | Win32:Evo-gen |
| 5 | AVG | Win32:Meterpreter-C | Win32:SwPatch | Win32:ShikataGaNai-C | Win32:Evo-gen |
| 6 | Avira | TR/Patched.Gen2 | TR/Patched.Gen2 | TR/Patched.Gen2 | TR/Crypt.XPACK.Gen7 |
| 7 | CalmAV | Win.Trojan.Swrort-5710536-0 | Win.Trojan.MSShellcode-6360728-0 | Win.Trojan.MSShellcode-6360728-0 | - |
| 8 | Comodo Linux | TrojWare.Win32.Rozena.A | TrojWare.Win32.Rozena.A | TrojWare.Win32.Rozena.A | TrojWare.MSIL.TrojanDownloader.Small.H |
| 9 | Emsisoft | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Gen:Variant.Kryptik.175 |
| 10 | F-Prot | W32/Swrort.A.gen!Eldorado | W32/Swrort.A.gen!Eldorado | W32/Swrort.A.gen!Eldorado | - |
| 11 | F-Secure | Trojan.TR/Patched.Gen2 | Trojan.TR/Patched.Gen2 | Trojan.TR/Patched.Gen2 | Trojan.TR/Crypt.XPACK.Gen7 |
| 12 | G Data | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Trojan.CryptZ.Marte.1.Gen | Gen:Variant.Kryptik.175 |
| 13 | IKARUS | Trojan.Win32.Swrort | Trojan.Win32.Swrort | Trojan.Win32.Swrort | Trojan-Downloader.MSIL.Tiny |
| 14 | Immunet | Win.Trojan.Swrort-5710536-0 | Win.Trojan.MSShellcode-6360728-0 | Win.Trojan.MSShellcode-6360728-0 | - |
| 15 | Max Secure | Packed_Win32_BDF_a_1_YR | Packed_Win32_BDF_a_1_YR | Packed_Win32_BDF_a_1_YR | - |
| 16 | McAfee | Swrort.i trojan !!! | Swrort.i trojan !!! | Swrort.i trojan !!! | Trojan-Veil-FLRH!A132AFFEC851 trojan !!! |
| 17 | Microsoft Defender | Trojan:Win32/Meterpreter.RPZ!MTB | Trojan:Win32/Meterpreter.A | Trojan:Win32/Meterpreter.A | Trojan:MSIL/Tiny.EM!MTB |
| 18 | NANO | Virus.Win32.Gen-Crypt.ccnc | Virus.Win32.Gen.ccmw | Virus.Win32.Gen-Crypt.ccnc | Multiple Threats Detected |
| 19 | NOD32 | Win32/Rozena.AA trojan | Win32/Rozena.AA trojan | Win32/Rozena.AA trojan | MSIL/TrojanDownloader.Tiny.BQ trojan |
| 20 | Norman | Win32:Meterpreter-C | Win32:SwPatch | Win32:ShikataGaNai-C | Win32:Evo-gen |
| 21 | Quick Heal | Trojan.Swrort.A | Trojan.Swrort.A | Trojan.Swrort.A | - |
| 22 | SecureAge APEX | Malicious | Malicious | Malicious | Malicious |
| 23 | Seqrite | Trojan.Swrort.A | Trojan.Swrort.A | Trojan.Swrort.A | - |
| 24 | Sophos | Mal/EncPk-TZ | Mal/EncPk-TZ | Mal/EncPk-TZ | Troj/Rozena-D |
| 25 | TrendMicro | - | BKDR_SWRORT.SM | BKDR_SWRORT.SM | - |
| 26 | VirITExplorer | Trojan.Win32.Rozena.AA | Trojan.Win32.Rozena.AA | Trojan.Win32.Rozena.AA | - |
| 27 | VirusFighter | Mal/EncPk-ACE | Mal/EncPk-ACE | Mal/EncPk-ACE | ATK/Dloadr-EFR |
| 28 | Zillya | Trojan.RozenaGen.Win32.2 | Trojan.RozenaGen.Win32.2 | Trojan.RozenaGen.Win32.2 | - |
| 29 | ZoneAlarm | HEUR:Trojan.Win32.Generic | HEUR:Trojan.Win32.Generic | HEUR:Trojan.Win32.Generic | HEUR:Trojan.Win32.Generic |
Table A2.
Antivirus scan results for original and evasion (x64) tools.
Table A2.
Antivirus scan results for original and evasion (x64) tools.
| No. | AntiVirus | Msfvenom Staged Reverse Shell (*.exe) | shikata_ga_nai with 10 Iterators | shikata_ga_nai with 20 Iterators | PwnWind Module in TheFatRat |
|---|---|---|---|---|---|
| 1 | AdAware | Trojan.Metasploit.A | Trojan.Metasploit.A | Trojan.Metasploit.A | Gen:Variant.Zusy.486491 |
| 2 | Alyac | Trojan.Metasploit.A | Trojan.Metasploit.A | Trojan.Metasploit.A | - |
| 3 | Arcabit | Trojan.Metasploit.A | Trojan.Metasploit.A | Trojan.Metasploit.A | Trojan.Zusy.D76C5B |
| 4 | Avast | Win32:MsfShell-V | Win64:Evo-gen | Win64:Evo-gen | Win64:TrojanX-gen |
| 5 | AVG | Win32:MsfShell-V | Win64:Evo-gen | Win64:Evo-gen | Win64:TrojanX-gen |
| 6 | Avira | TR/Crypt.XPACK.Gen7 | TR/Crypt.XPACK.Gen7 | TR/Crypt.XPACK.Gen7 | - |
| 7 | CalmAV | Win.Malware.Metasploit-10022275-0 | Win.Trojan.MSShellcode-6360728-0 | Win.Trojan.MSShellcode-6360730-0 | Win.Malware.Agent-9870952-0 |
| 8 | Crowdstrike Falcon | - | Threat detected. | - | - |
| 8 | Emsisoft | Trojan.Metasploit.A | Trojan.Metasploit.A | Trojan.Metasploit.A | Gen:Variant.Zusy.486491 |
| 9 | F-Prot | W64/S-c4a4ef26!Eldorado | W64/S-c4a4ef26!Eldorado | W64/S-c4a4ef26!Eldorado | - |
| 10 | F-Secure | Trojan.TR/Crypt.XPACK.Gen7 | Trojan.TR/Crypt.XPACK.Gen7 | Trojan.TR/Crypt.XPACK.Gen7 | - |
| 11 | G Data | Trojan.Metasploit.A | Trojan.Metasploit.A | Trojan.Metasploit.A | Gen:Variant.Zusy.486491 |
| 12 | IKARUS | Trojan.Win64.Meterpreter | Trojan.Win64.Meterpreter | Trojan.Win64.Meterpreter | - |
| 13 | Immunet | Win.Malware.Metasploit-10022275-0 | Win.Trojan.MSShellcode-6360728-0 | Win.Trojan.MSShellcode-6360730-0 | Win.Malware.Agent-9870952-0 |
| 14 | Kaspersky | Trojan.Win64.Packed.gen | Trojan.Win64.Packed.gen | Trojan.Win64.Packed.gen | - |
| 15 | Max Secure | Trojan_W64_090622_Packed_gen_YR | Trojan_W64_090622_Packed_gen_YR | Trojan_W64_090622_Packed_gen_YR | - |
| 16 | McAfee | Trojan-FJIN!102811008CF1 trojan !!! | Trojan-FJIN!F999F6FE19FA trojan !!! | Trojan-FJIN!765AE5EF0526 trojan !!! | - |
| 17 | Microsoft Defender | Trojan:Win64/Meterpreter!pz | Trojan:Win64/Meterpreter!pz | Trojan:Win64/Meterpreter!pz | Trojan:Win64/Rozena.IG!MTB |
| 18 | NOD32 | Win64/Rozena.M trojan | Win64/Rozena.J trojan | Win64/Rozena.J trojan | Win64/Rozena.HH trojan |
| 19 | Norman | Win32:MsfShell-V | Win64:Evo-gen | Win64:Evo-gen | Win64:TrojanX-gen |
| 20 | Quick Heal | HackTool.Metasploit.S9212471 | HackTool.Metasploit.S9212471 | HackTool.Metasploit.S9212471 | - |
| 21 | SecureAge APEX | Malicious | Malicious | Malicious | - |
| 22 | Seqrite | Trojan.Dynamer.S4605 | Trojan.Dynamer.S4605 | Trojan.Dynamer.S4605 | - |
| 23 | Sophos | Mal/Swrort-J | Mal/Swrort-J | Mal/Swrort-J | - |
| 24 | VirITExplorer | Trojan.Win32.Generic.BZPS | Trojan.Win32.Generic.BZPS | Trojan.Win32.Generic.BZPS | - |
| 25 | VirusFighter | ATK/Meter-A | ATK/Swrort-J | ATK/Swrort-J | ATK/FatRat-G |
| 26 | Zillya | - | - | - | Trojan.Rozena.Win64.43244 |
| 27 | ZoneAlarm | HEUR:Trojan.Win64.Packed.gen | HEUR:Trojan.Win64.Packed.gen | HEUR:Trojan.Win64.Packed.gen | HEUR:Trojan.Win32.Generic |
Appendix A.2. Scan Report from Kleenscan.com
Table A3.
The scan report for this experiment.
Table A3.
The scan report for this experiment.
Appendix A.3. The Implemented Pseudocode
| Listing 1. Pseudocode for executing the decoded shellcode using FountainDecoder. |
|
References
- Wu, J.; Wu, J. Security risks from vulnerabilities and backdoors. In Cyberspace Mimic Defense: Generalized Robust Control and Endogenous Security; Springer: Cham, Switzerland, 2020; pp. 3–38. [Google Scholar]
- Slayton, R. What is the cyber offense-defense balance? Conceptions, causes, and assessment. Int. Secur. 2016, 41, 72–109. [Google Scholar] [CrossRef]
- Yadav, T.; Rao, A. Technical aspects of cyber kill chain. In Proceedings of the Security in Computing and Communications: Third International Symposium, SSCC 2015, Kochi, India, 10–13 August 2015; Proceedings 3. pp. 438–452. [Google Scholar]
- Hassan, N.; Hijazi, R. Open Source Intelligence Methods and Tools; Springer: New York, NY, USA, 2018. [Google Scholar]
- Exploit-DB. Available online: https://www.exploit-db.com/ (accessed on 22 October 2024).
- Yang, H.; Park, S.; Yim, K.; Lee, M. Better not to use vulnerability’s reference for exploitability prediction. Appl. Sci. 2020, 10, 2555. [Google Scholar] [CrossRef]
- Yamin, M.; Ullah, M.; Ullah, H.; Katt, B.; Hijji, M.; Muhammad, K. Mapping tools for open source intelligence with cyber kill chain for adversarial aware security. Mathematics 2022, 10, 2054. [Google Scholar] [CrossRef]
- Dube, T.; Raines, R.; Grimaila, M.; Bauer, K.; Rogers, S. Malware Target Recognition of Unknown Threats. IEEE Syst. J. 2013, 7, 467–477. [Google Scholar] [CrossRef]
- Zhang, H.; Li, B.; Li, W.; Zhu, L.; Chang, C.; Yu, S. MRCIF: A Memory-Reverse-Based Code Injection Forensics Algorithm. Appl. Sci. 2023, 13, 2478. [Google Scholar] [CrossRef]
- Samociuk, D. Antivirus evasion methods in modern operating systems. Appl. Sci. 2023, 13, 5083. [Google Scholar] [CrossRef]
- Taghizadeh, S.; Masmooie, M. COCASER: A Code Obfuscation Method in Cyber Attacks for Enhancing Stealth and Evasion of Reverse Shell Payloads. In Proceedings of the 2023 International Conference on Computational Intelligence, Networks and Security (ICCINS), Mylavaram, India, 22–23 December 2003; pp. 1–6. [Google Scholar]
- You, I.; Yim, K. Malware Obfuscation Techniques: A Brief Survey. In Proceedings of the 2010 International Conference on Broadband, Wireless Computing, Communication and Applications, Fukuoka, Japan, 4–6 November 2010; pp. 297–300. [Google Scholar]
- Geng, J.; Wang, J.; Fang, Z.; Zhou, Y.; Wu, D.; Ge, W. A Survey of strategy-driven evasion methods for PE malware: Transformation, concealment, and attack. Comput. Secur. 2024, 137, 103595. [Google Scholar] [CrossRef]
- Wu, M.; Hsu, F.; Hunag, J.; Wang, K.; Liu, Y.; Chen, J.; Wang, H.; Yang, H. MPSD: A Robust Defense Mechanism against Malicious PowerShell Scripts in Windows Systems. Electronics 2024, 13, 3717. [Google Scholar] [CrossRef]
- Andriesse, D.; Bos, H.; Slowinska, A. Parallax: Implicit Code Integrity Verification Using Return-Oriented Programming. In Proceedings of the 2015 45th Annual IEEE/IFIP International Conference On Dependable Systems And Networks, Rio de Janeiro, Brazil, 22–25 June 2015; pp. 125–135. [Google Scholar]
- Kaushik, K.; Sandhu, H.; Gupta, N.; Sharma, N.; Tanwar, R. A systematic approach for evading antiviruses using malware obfuscation. In Emergent Converging Technologies And Biomedical Systems: Select Proceedings of ETBS 2021; Springer: Singapore, 2022; pp. 29–37. [Google Scholar]
- Kunwar, R.; Sharma, P.; Kumar, K. Malware Analysis of Backdoor Creator: Fatrat. Int. J. Cyber-Secur. Digit. Forensics 2018, 7, 72–80. [Google Scholar] [CrossRef]
- Afianian, A.; Niksefat, S.; Sadeghiyan, B.; Baptiste, D. Malware dynamic analysis evasion techniques: A survey. ACM Comput. Surv. (CSUR) 2019, 52, 1–28. [Google Scholar] [CrossRef]
- Strachanski, F.; Petrov, D.; Schmidbauer, T.; Wendzel, S. A Comprehensive Pattern-based Overview of Stegomalware. In Proceedings of the 19th International Conference on Availability, Reliability and Security, Vienna, Austria, 30 July–2 August 2024; pp. 1–10. [Google Scholar]
- Mazaed Alotaibi, F.; Fawad. A multifaceted deep generative adversarial networks model for mobile malware detection. Appl. Sci. 2022, 12, 9403. [Google Scholar] [CrossRef]
- Byers, J.; Luby, M.; Mitzenmacher, M.; Rege, A. A digital fountain approach to reliable distribution of bulk data. ACM SIGCOMM Comput. Commun. Rev. 1998, 28, 56–67. [Google Scholar] [CrossRef]
- Hayajneh, K. Memory-based lt codes for efficient 5g networks and beyond. Electronics 2021, 10, 3169. [Google Scholar] [CrossRef]
- Wang, L.; Niu, F.; Jin, J.; Yu, L. Analysis and Research on Secondary LT Coding Anti-Eavesdropping Scheme Based on LT Code Degree-1. Appl. Sci. 2023, 13, 11296. [Google Scholar] [CrossRef]
- Shokrollahi, A. Raptor codes. IEEE Trans. Inf. Theory 2006, 52, 2551–2567. [Google Scholar] [CrossRef]
- Richardson, T.; Urbanke, R. Modern Coding Theory; Cambridge University Press: Cambridge, UK, 2008. [Google Scholar]
- Stockhammer, T.; Shokrollahi, A.; Watson, M.; Luby, M.; Gasiba, T. Application layer forward error correction for mobile multimedia broadcasting. In Handbook of Mobile Broadcasting; Auerbach Publications: Boca Raton, FL, USA, 2008; pp. 239–278. [Google Scholar]
- Fan, B.; Tang, B.; Qu, Z.; Ye, B. Network Coding Approaches for Distributed Computation over Lossy Wireless Networks. Entropy 2023, 25, 428. [Google Scholar] [CrossRef] [PubMed]
- Hung, T.; Kaushal, S.; Hsiao, H. Content distribution network for streaming using multiple Galois fields. In Proceedings of the 2021 IEEE International Symposium on Circuits and Systems (ISCAS), Daegu, Republic of Korea, 22–28 May 2021; pp. 1–5. [Google Scholar]
- Liu, J.; Gao, Q.; Wang, X.; Zhou, X.; Li, S.; Zhang, H.; Cui, X. FCMPR: A multi-path secure transmission method based on link security assessment and fountain coding. Int. J. Intell. Netw. 2024, 5, 275–285. [Google Scholar]
- Durstenfeld, R. Algorithm 235: Random permutation. Commun. ACM 1964, 7, 420. [Google Scholar] [CrossRef]
- Shannon, C. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef]
- Bang, J.; Kim, J.; Lee, S. Entropy Sharing in Ransomware: Bypassing Entropy-Based Detection of Cryptographic Operations. Sensors 2024, 24, 1446. [Google Scholar] [CrossRef] [PubMed]
- Kim, W.; Kang, S.; Kim, K.; Kim, S. Detecting ShellCode Using Entropy. KIPS Trans. Comput. Commun. Syst. 2014, 3, 87–96. [Google Scholar] [CrossRef]
- Alhamed, M.; Rahman, M. A Systematic Literature Review on Penetration Testing in Networks: Future Research Directions. Appl. Sci. 2023, 13, 6986. [Google Scholar] [CrossRef]
- Metula, E. Managed Code Rootkits: Hooking into Runtime Environments; Elsevier: Amsterdam, The Netherlands, 2010. [Google Scholar]
- Kaushik, K.; Aggarwal, S.; Mudgal, S.; Saravgi, S.; Mathur, V. A novel approach to generate a reverse shell: Exploitation and Prevention. Int. J. Intell. Commun. Comput. Netw. Open Access J. 2021, 2, 83–93. [Google Scholar] [CrossRef]
- Kleenscan. Available online: https://kleenscan.com/ (accessed on 22 October 2024).
- Manna, M.; Case, A.; Ali-Gombe, A.; Richard, G., III. Memory analysis of. net and. net core applications. Forensic Sci. Int. Digit. Investig. 2022, 42, 301404. [Google Scholar] [CrossRef]
- Cheng, T.; Lin, Y.; Lai, Y.; Lin, P. Evasion Techniques: Sneaking through Your Intrusion Detection/Prevention Systems. IEEE Commun. Surv. Tutor. 2012, 14, 1011–1020. [Google Scholar] [CrossRef]
- Nasereddin, M.; Al-Qassas, R. A new approach for detecting process injection attacks using memory analysis. Int. J. Inf. Secur. 2024, 23, 2099–2121. [Google Scholar] [CrossRef]
- Mills, A.; Legg, P. Investigating anti-evasion malware triggers using automated sandbox reconfiguration techniques. J. Cybersecur. Priv. 2020, 1, 19–39. [Google Scholar] [CrossRef]
- Lee, G.; Kim, M.; Yi, J.; Cho, H. Pinicorn: Towards Automated Dynamic Analysis for Unpacking 32-Bit PE Malware. Electronics 2024, 13, 2081. [Google Scholar] [CrossRef]
- Varlioglu, S.; Elsayed, N.; Varlioglu, E.; Ozer, M.; ElSayed, Z. The Pulse of Fileless Cryptojacking Attacks: Malicious PowerShell Scripts. In Proceedings of the SoutheastCon 2024, Atlanta, GA, USA, 15–24 March 2024; pp. 571–580. [Google Scholar]
- Wu, M.; Hsu, F.; Huang, J.; Wang, K.; Hwang, Y.; Wang, H.; Chen, J.; Hsiao, T.; Yang, H. Enhancing Linux System Security: A Kernel-Based Approach to Fileless Malware Detection and Mitigation. Electronics 2024, 13, 3569. [Google Scholar] [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).