A Deep Learning Framework for Enhanced Detection of Polymorphic Ransomware

Abstract

Ransomware, a significant cybersecurity threat, encrypts files and causes substantial damage, making early detection crucial yet challenging. This paper introduces a novel multi-phase framework for early ransomware detection, designed to enhance accuracy and minimize false positives. The framework addresses the limitations of existing methods by integrating operational data with situational and threat intelligence, enabling it to dynamically adapt to the evolving ransomware landscape. Key innovations include (1) data augmentation using a Bi-Gradual Minimax Generative Adversarial Network (BGM-GAN) to generate synthetic ransomware attack patterns, addressing data insufficiency; (2) Incremental Mutual Information Selection (IMIS) for dynamically selecting relevant features, adapting to evolving ransomware behaviors and reducing computational overhead; and (3) a Deep Belief Network (DBN) detection architecture, trained on the augmented data and optimized with Uncertainty-Aware Dynamic Early Stopping (UA-DES) to prevent overfitting. The model demonstrates a 4% improvement in detection accuracy (from 90% to 94%) through synthetic data generation and reduces false positives from 15.4% to 14%. The IMIS technique further increases accuracy to 96% while reducing false positives. The UA-DES optimization boosts accuracy to 98.6% and lowers false positives to 10%. Overall, this framework effectively addresses the challenges posed by evolving ransomware, significantly enhancing detection accuracy and reliability.

1. Introduction

While technological innovation has yielded substantial advancements, it has contemporaneously amplified security vulnerabilities, particularly concerning malicious software (malware) such as viruses, worms, Trojans, rootkits, and ransomware [1,2,3]. Ransomware, emerging as a significant threat since the 1980s, is engineered to impede access to critical data and disrupt operational continuity [4,5]. Distinct from other malware, ransomware employs data encryption or locking mechanisms to extort victims, impacting both individual users and enterprise operations [6,7]. Its capacity to compromise operational data and system resources renders it particularly pernicious. The proliferation of accessible development toolkits has facilitated the exploitation of unpatched systems and software vulnerabilities by malicious actors [8,9]. The WannaCry attack, leveraging the Eternal Blue vulnerability, serves as a stark illustration, compromising 230,000 systems across 150 nations and incurring losses exceeding USD 8 billion [2]. Ransomware’s reach into Industrial IoT and Supervisory Control and Data Acquisition (SCADA) systems now most certainly threatens critical infrastructure [10]. This expansion unveils profound vulnerabilities and systemic risks, foreshadowing societal catastrophes without robust mitigation strategies and methodologies.

Notable ransomware attacks on Industrial Control Systems (ICSs) critical infrastructure, such as the Colonial Pipeline attack in 2021, show the extensive disruption these attacks can cause to essential services and the economy [11]. These incidents highlight the need for stronger cybersecurity measures, particularly for Operational Technology (OT) systems, which manage critical infrastructure. The National Institute of Standards and Technology (NIST) has developed guidelines to improve OT security to emphasize the importance of performance, reliability, and safety in defending against any threats [11].

1.1. Problem and Motivation

The escalating threat of ransomware demands new, effective detection strategies to catch attacks early. While current approaches fall into three main categories—process-centric, data-centric, and resource-centric—they all struggle against modern ransomware’s sophisticated evasion tactics [8,9,12].

• Process-centric methods analyze running programs, but they often need complete data to work, which makes early detection difficult.
• Data-centric approaches monitor user files for unusual changes. However, they frequently produce too many false alarms because they cannot easily tell the difference between normal file changes and malicious ransomware activity. This leads to slow responses.
• Ransomware’s polymorphic nature (meaning it can constantly change its appearance) and its ability to mimic legitimate software further complicate early detection. The critical pre-encryption phase is especially challenging; it is so brief and elusive that traditional analysis struggles to collect enough data for timely and accurate detection [13,14].

Some researchers have tried to identify features of “immature” pre-encryption ransomware, but these features quickly become outdated due to ransomware’s rapid evolution [13,15]. This is where advanced techniques come in. Generative Adversarial Networks (GANs) show promise for generating realistic, synthetic attack patterns, which could help overcome the problem of not having enough real-world data. However, standard GANs have their own issues:

• Their Minimax loss function struggles to accurately model the complex differences between real and synthetic ransomware data, particularly during that critical pre-encryption phase.
• GANs are also vulnerable to the “curse of dimensionality,” meaning they need effective ways to select important features. Traditional feature selection methods often are not adaptable enough to keep up with evolving ransomware [16].

Furthermore, building and training robust detection models, like Deep Belief Networks (DBNs), is complicated by the risk of overfitting—where the model performs well on training data but poorly on new, unseen data. Existing methods for deciding when to stop training (epochs) do not fully account for the unpredictable nature of ransomware [17,18,19,20].

Our research addresses these challenges head-on. We have developed an innovative framework that enhances GAN-based early ransomware detection. This framework incorporates novel mechanisms to adapt to changing feature significance and dynamically adjust training, significantly improving detection accuracy and preventing overfitting through a new three-phase approach [21,22]. Our goal is to provide a more robust, adaptable, and effective solution to combat the ever-growing threat of ransomware.

1.2. Research Questions and Objectives

This work introduces an enhanced framework for early ransomware detection. Our approach addresses critical challenges in current detection methods by (i) generating synthetic data to overcome scarcity in the ransomware’s pre-encryption phase, (ii) employing incremental learning for dynamic updates to feature relevance, and (iii) utilizing a Bayesian approach to estimate uncertainty for optimized training adjustments.

1.2.1. Research Questions

To achieve the overarching goal of improved ransomware detection, this research investigates the following key questions:

• To what extent does GAN-based data augmentation mitigate pre-encryption data scarcity and enhance ransomware early detection?
• How can mutual information feature selection techniques be integrated into a DBN-based ransomware detection model to enable dynamic feature relevance reassessment with the incorporation of new data?
• How can an uncertainty-based estimation approach be utilized to determine the optimal number of training epochs for a DBN-based ransomware detection model?

1.2.2. Objectives

This research aims to significantly enhance early ransomware detection by addressing key limitations in existing approaches. Our specific objectives are as follows:

• Introduce an enhanced GAN-based data augmentation module with a novel Bi-Gradual Minimax loss function. Existing GAN models struggle to accurately bridge the gap between real and synthetic data distributions, especially with scarce pre-encryption data. Our new loss function, employing a gradual up-weighting coefficient, minimizes this divergence, enabling the generation of highly realistic artificial ransomware attack patterns. This directly tackles data scarcity, a major obstacle to early detection, and is expected to significantly improve accuracy.
• Propose an Incremental Mutual Information Selection (IMIS) technique to enhance model adaptability. Traditional feature selection methods often become outdated due to the dynamic nature of ransomware. IMIS dynamically reassesses feature relevance as new data integrates into our custom DBN-based model. This on-the-fly adaptation allows continuous learning and improved detection accuracy against evolving threats, ensuring long-term effectiveness.
• Present an Uncertainty-Aware Dynamic Early Stopping (UA-DES) criterion and method to optimize DBN training. Overfitting is a common challenge in DBN training, particularly in determining optimal epochs. Existing dynamic approaches often fall short due to the inherent variability of ransomware behavior. By leveraging Bayesian approximation to accurately estimate uncertainty, UA-DES aims to determine the optimal number of training epochs, leading to improved model generalization, best-in-class prediction performance, and effective overfitting prevention.

1.3. Early Detection Value Proposition

This paper introduces the Polymorphic Ransomware Deep Learning (PRDL) framework, a novel and sophisticated solution designed to significantly improve the early detection of evasive, shapeshifting ransomware during its brief pre-encryption phase, a critical window before any data is locked or ransom demands are made. Traditional cybersecurity methods often fail in this phase due to limited observable behavior, constant code mutations, and a lack of model adaptability. To overcome these challenges, the PRDL framework integrates a GAN-augmented DBN and introduces three key innovations:

• Realistic Ransomware Pattern Generation: A novel Bi-Gradual Minimax Loss function uses a gradual up-weighting coefficient to enable the GAN to produce highly realistic artificial ransomware attack patterns.
• Dynamic Feature Adaptation: An Incremental Mutual Feature Selection (IMIS) technique allows the model to continuously adapt to evolving ransomware behaviors by reassessing feature relevance.
• Optimized Training for Timely Detection: The Uncertainty-Aware Dynamic Early Stopping (UA-DES) technique, based on Bayesian approximation, precisely determines the optimal training epochs for the DBN, ensuring efficient and early detection.

Beyond technical enhancements, the PRDL framework contributes strategically to ransomware defense by providing

• First-of-its-Kind Taxonomy of Ransomware Attack Success Factors: This work introduces a groundbreaking categorization of factors contributing to successful ransomware attacks, particularly within industrial SCADA environments. This offers essential insights for risk mitigation and targeted defense [11,14].
• Comprehensive Ransomware Attack Model for Industrial Systems: We have developed a detailed model illustrating ransomware’s operational mechanisms in industrial systems, crucial for designing effective prevention and mitigation strategies.
• Advanced Situational Awareness for Prediction: The framework enhances the ability to predict ransomware attacks by integrating diverse evidence, enabling proactive intervention. It also establishes a benchmark for evaluating future ransomware prediction methodologies in Industrial Control Systems (ICSs) [10].

1.4. Overview of the 3-Phase Framework

The proposed framework, named a “custom GAN-augmented Deep Belief Network (DBN) model,” tackles these challenges through a three-phase approach:

Phase 1: Generating Realistic Ransomware Data (GAN Pre-encryption Data Augmentation)

• What it does: Uses a GAN to create synthetic, yet realistic, ransomware attack patterns.
• Why it is important: Real-world pre-encryption ransomware data is scarce. GANs help overcome this data insufficiency, allowing the detection model to be trained on a much larger and more diverse set of potential attack scenarios. The framework employs a “Bi-Gradual Minimax loss function” to make the synthetic data even more accurate.

Phase 2: Smartly Selecting Key Indicators IMIS

• What it does: Employs an IMIS technique to identify and prioritize the most important “features” or indicators of ransomware behavior.
• Why it is important: Ransomware constantly evolves, meaning what was a useful indicator yesterday might not be today. IMIS dynamically reassesses which features are most relevant as new data becomes available, ensuring the detection model focuses on the most current and effective signals, while also avoiding redundant information.

Phase 3: Optimizing the Detection Model (Deep Belief Network with Uncertainty-Aware Dynamic Early Stopping)

• What it does: Trains a DBN using the augmented data and selected features. It introduces a UA-DES criterion.
• Why it is important: Training deep learning models can lead to “overfitting,” where the model becomes too good at recognizing the training data but performs poorly on new, unseen data. UA-DES helps determine the optimal point to stop training, preventing overfitting and improving the model’s ability to accurately detect new and evolving ransomware threats. It does this by considering the uncertainty in the model’s predictions.

Essentially, our deep learning framework creates its own “training data” for rare ransomware behaviors, intelligently picks out the most relevant clues, and then trains a powerful detection system in a way that makes it highly adaptable and accurate against the ever-changing threat of polymorphic ransomware, especially in critical infrastructure like SCADA systems.

1.5. Example Use Case

Imagine a security team at a water treatment plant that uses SCADA systems to control its operations. They are constantly concerned about ransomware attacks, especially new, “polymorphic” variants that traditional antivirus often misses. Here is how this framework could be “used”:

Data Augmentation (Phase 1)

• Scenario: The security team has logs from a few past ransomware incidents, but these are limited, especially for the very early, pre-encryption stages.
• Framework in Action: The GAN component of the framework would take these limited real-world ransomware logs and generate thousands of synthetic but highly realistic “pre-encryption” ransomware behaviors. These synthetic logs would include subtle system changes, file access patterns, or network communication anomalies that might occur just before encryption. This massively expands the training data available to the security team’s detection system.

Feature Selection (Phase 2)

• Scenario: As new, real-world (or even synthetic) data comes in from various sources—perhaps from threat intelligence feeds about emerging ransomware or from internal simulations—the security team needs to know which indicators are still relevant. For example, a previous ransomware might have always accessed a specific registry key, but a new variant might not.
• Framework in Action: The IMIS component would continuously analyze all the available data (real and synthetic). It would dynamically identify which specific actions, API calls, or network patterns are currently the strongest indicators of ransomware behavior, and which ones are becoming less relevant for detection. For instance, it might determine that unusual file system enumeration combined with specific network beaconing is a very strong indicator for a new variant, while a previously important indicator (like a specific DLL injection) is now less reliable. The system automatically prioritizes these most relevant “features” for the detection model.

Model Training and Detection (Phase 3)

• Scenario: The security team wants their SCADA system’s security software to detect ransomware as early as possible, without generating too many false alarms (e.g., stopping a legitimate system update). They also need the model to be robust and adaptable.
• Framework in Action: The DBN, trained on the GAN-augmented data and using the IMIS-selected features, would constantly monitor the water treatment plant’s SCADA systems. The UA-DES part of the framework ensures the DBN is trained just enough to be highly accurate but not so much that it overfits (leading to false positives). If the DBN detects a sequence of behaviors (based on the dynamically selected features) that strongly resembles ransomware, it would trigger an alert. For example, it might flag a sudden, unusual attempt to access specific operational data files followed by an attempt to modify system services, even if the exact pattern has not been seen before but falls within the learned “ransomware behavior space.” This early alert allows the security team to intervene before any critical systems are encrypted or damaged.

1.6. Multi-Step Deep Learning Approach

There is a Glossary of Acronyms just before the References, and an Appendix with a small set of ransomware case reviews, data samples, and code snippets. Section 1 introduces the problem of early ransomware detection, outlining research questions, objectives, and contributions. Section 2 analyzes current early ransomware detection models, examining state-of-the-art techniques and key research gaps. Section 3 details the proposed framework, research methodology, and experimental architecture, including dataset characteristics and evaluation metrics. Section 4 presents the design and development of the proposed ransomware detection framework, including (i) GAN-based data augmentation, (ii) IMIS-driven dynamic feature selection, and (iii) DBN-based detection model training (UA-DES). Detailed analyses, encompassing process descriptions and the experimental results, are provided for each phase. Section 5 synthesizes the research findings, highlights key contributions and implications, and proposes future research directions.

2. Related Work

The operational benefits of SCADA system deployment are undeniable; however, this proliferation has created substantial cybersecurity challenges [23]. SCADA systems are inherently complex, comprising diverse components and protocols that frequently exhibit interoperability issues and operate under disparate standards [24]. Moreover, their growing connectivity to public networks, intended to facilitate interoperability, dramatically increases their vulnerability to cyber threats. These threats include malicious software specifically crafted to steal sensitive information, gain unauthorized access, and disrupt vital industrial processes, as shown in Figure 1 [23,25].

Malicious software has evolved significantly since its first appearance in the late 1970s [26]. Advanced strains, such as viruses, worms, Trojans, and ransomware, now pose significant threats [25,27,28]. Ransomware stands out as a particularly insidious threat, leveraging encryption to render data inaccessible and exploiting victims’ fear of critical resource or sensitive information loss to extort ransom payments [29,30,31,32,33,34]. Ransomware attacks are often irreversible because they use encryption, and their financial profitability motivates adversaries to develop increasingly advanced versions [13,35]. Moreover, the financial impact of ransomware attacks is substantial, with ransom demands ranging from USD 300–USD 700 for individuals and USD 10,000–USD 17,000 for enterprises [36]. In some cases, victims have paid as much as USD 400,000 in a single month [37], and the FBI’s Internet Crime Complaint Center reported losses of approximately USD 18 million from ransomware attacks between April 2014 and June 2015 [38,39].

Probably the most famous attack, although not targeted at ICSs, in recent memory (Feb. 2024) was the (Russian speaking) hacker gang known as BlackCat (also as ALPHV), receiving a payment of USD 22 million via Bitcoin. Though the victims made the ransomware payment, UnitedHealth Group CEO Andrew Witty shared on 1 May 2024 to Congress that Change Healthcare did not obtain its data back. Total damages are estimated to be almost USD 3B (BlackCat operates as a Ransomware-as-a-Service (RaaS) platform, allowing other cybercriminals to use its ransomware in exchange for a share of the profits. While most RaaS models let affiliates retain about 70% of their earnings, BlackCat offers a more generous share, allowing affiliates to keep 80–90% of their profits (see “Breaking Down the BlackCat Ransomware Operation,” CIS, 7 July 2022. Available: https://www.cisecurity.org/insights/blog/breaking-down-the-blackcat-ransomware-operation (accessed on 1 March 2025), and a USD 10M + USD 5M = USD 15M reward is being offered). Though this was not an attack on ICSs, the attack is estimated to have resulted in a total impact of USD 2.87 billion, including USD 1.7 billion in direct response costs. Change Healthcare covers about 50% of medical claims in the US for around 900,000 physicians, 33,000 pharmacies, 5500 hospitals, and 600 laboratories. This incident underscores the catastrophic potential of single-point-of-failure systems, a stark warning as our world increasingly relies on operational monopolies. Refer to Appendix A for more detailed discussion of the methods and tactics for this and the other attacks mentioned in this article).

Given the escalating threat of ransomware, understanding the factors driving attack success is crucial. This research leverages the developed framework to analyze these factors, specifically within SCADA system attack models. By elucidating the tools and mechanisms employed during attacks, the model facilitates the development of effective security measures tailored to safeguard SCADA environments [40,41,42].

2.1. Ransomware Attacks and Trends

On 21 February 2024, Change Healthcare, a key healthcare technology unit of UnitedHealthcare Group, was severely disrupted by a ransomware attack attributed to the ALPHV (also known as BlackCat) ransomware group. This breach, as reported by The Washington Post and later confirmed by UnitedHealth Group, exposed the data of an estimated 190 million Americans and crippled critical operations across the U.S. healthcare system. The immediate consequences were widespread; pharmacies experienced significant disruptions in processing insurance claims and paying providers, often forcing them to resort to manual workarounds and cash transactions. This incident serves as a stark reminder of the acute vulnerability of essential healthcare infrastructure to the devastating effects of ransomware attacks, highlighting the critical need for robust cybersecurity measures within the sector. Consequently, pharmacies experienced widespread disruptions in processing insurance claims and paying providers, forcing them to rely on manual workarounds and cash transactions. This incident serves as a stark reminder of the acute vulnerability of essential healthcare infrastructure to the devastating effects of ransomware.

Beyond healthcare, Phobos ransomware has remained a persistent threat since May 2019, utilizing a Ransomware-as-a-Service model to target government, education, and other critical infrastructures. These groups employ techniques such as network intrusion, data exfiltration, and encryption, prompting U.S. government agencies to issue urgent mitigation guidelines.

The year 2021 witnessed two devastating ransomware attacks that laid bare the vulnerabilities of U.S. critical cyber-physical infrastructure. In May, the Colonial Pipeline attack crippled fuel supplies across the East Coast, sparking widespread panic buying, severe shortages, and soaring prices. This disruption forced the company to capitulate, paying a USD 4.4 million ransom. Just weeks later, JBS, a global meat processing giant, suffered a similar fate, halting operations in the U.S., Canada, and Australia and ultimately succumbing to a USD 11 million ransom demand. Both attacks were executed using sophisticated ransomware variants, notably Ryuk, highlighting the escalating threat to vital sectors.

These incidents highlight the growing sophistication and persistence of ransomware campaigns and their capacity to inflict widespread economic and social disruptions. They also emphasize the urgent need for organizations managing critical infrastructure to implement robust cybersecurity measures and ensure regular system updates and patching to mitigate these escalating risks.

2.2. Ransomware Success Factors

Ransomware attacks have become increasingly effective due to several contributing factors, which can be broadly categorized into operational-related and resource-related factors.

2.2.1. Operational-Related Factors

Ransomware attacks are often successful due to a combination of factors: end-user vulnerabilities, insufficient backup policies, and the use of anonymous payment methods. End-users often underestimate their vulnerability, believing that only critical organizational resources are targeted. This misconception allows attackers to exploit unprotected endpoints through phishing, malicious links, and social engineering [43,44]. Luo and Liao [45] highlight the importance of user education across organizations to mitigate these risks. Secondly, ineffective backup policies hinder data recovery and business continuity during attacks. Organizations may neglect user-level data backups, leaving sensitive information exposed, and individuals often rely on default backup settings, further exacerbating the problem [23,46,47,48]. Finally, the use of cryptocurrencies like Bitcoin provides attackers with safe and anonymous payment methods, enabling them to demand and receive ransoms without revealing their identities. This anonymity has fueled the growth of ransomware operations, including Ransomware-as-a-Service models [49].

2.2.2. Resource-Related Factors

Ransomware attacks exploit several resource-related vulnerabilities: data vulnerabilities, susceptibility to service disruptions, and a lack of specialized expertise.

• Data Vulnerabilities: IoT and SCADA devices collect and transmit critical data used in control systems. Ransomware attacks targeting these data storages can encrypt or hijack (inject false data) operational data, which damages systems. Data servers, Human–Machine interfaces, and connected networks within SCADA systems are particularly sensitive due to their exposure to external networks [50,51].
• Service Disruption: Resource-limited devices often cannot easily recover from service disruptions caused by ransomware. Locking up ransomware, which disables device services without encryption, is a common tactic against such systems. Also, attackers can easily disable critical services required for operational decision-making [52].
• Lack of Expertise: Industrial systems like SCADA devices often run on specific operating systems that have not undergone enough research by the cybersecurity community. This lack of focus creates vulnerabilities that attackers exploit through zero-day attacks. The low interest in securing SCADA systems among researchers and industry professionals makes the issue bigger, and that can make them attractive targets [53].

2.3. Current Detection and Prevention Methods

Ransomware defenses are categorized into prevention, detection, and prediction. Prevention uses predefined rules and procedures to block attacks in real-time, but is static and vulnerable to human error due to reliance on manual configurations [54]. Detection models identify threats with similar behaviors by relying on known attack patterns, forming a proactive defense. Prediction enhances these approaches by anticipating future attacks. However, traditional prediction models fail to account for ransomware’s dynamic nature and obfuscation techniques like polymorphism and metamorphism [26,27].

2.3.1. Ransomware Prevention

Prevention research emphasizes factors that enhance preventive measures, including organizational size, security posture, and ransomware sophistication [55]. Studies also assess how business continuity disruptions, recovery timeframes, and the number of affected devices impact prevention effectiveness. Also, studies assess how business continuity disruptions, recovery timeframes, and the number of affected devices influence prevention effectiveness. Other contributors to ransomware’s success, such as anonymous payment systems and easily accessible development tools, are frequently studied in isolation, which limits their practical applicability [56]. As discussed in Section 2.2.2, SCADA systems pose additional challenges due to their unique architectures and resource constraints. Additionally, user behavior significantly influences malware spread. Factors like internet usage, peer-to-peer applications, and technical expertise play a role [57]. Research using routine activity theory indicates that legitimate software use has a weak correlation with malware infections, while illegitimate software use significantly increases risk [29].

2.3.2. Gaps in Ransomware Prediction and Detection

Truly effective ransomware prediction demands a synergistic blend of operational and situational data. Operational data, gleaned from ransomware processes, must be contextualized by situational intelligence, encompassing the system’s security posture and real-time state. This fusion enables models to adapt to the evolving and dynamic nature of ransomware. Alarmingly, a significant portion of current models remain tethered to historical data alone, rendering them woefully inadequate against sophisticated, contextually aware attacks.

Advanced approaches include machine learning and deep learning models. For example, GANs generate new malware samples to train detection engines on unseen variants [30]. Models like MalDeepNet and MalGAN predict malware behavior trends and create adversarial samples to enhance black-box detection systems [31,32]. Supervised learning models [33] and regression-based neural networks [58] predict ransomware behavior by analyzing past attack data, but their lack of situational awareness reduces their effectiveness for adaptive threats.

Likewise, lightweight models like those designed for IoT devices incorporate context ontologies for feature extraction [40], which reduces computational complexity. However, these models often block behavioral data, which is critical for understanding ransomware’s adaptive strategies. Similarly, deep learning approaches like LSTM provide early malware predictions on Android devices by analyzing data from active sessions [42], but limited input data disrupts accuracy for long-term predictions.

Addressing the inherent limitations of current predictive models necessitates a paradigm shift: the fusion of operational data with comprehensive situational and threat intelligence. This integration is not merely an enhancement; it is a critical imperative, enriching the model’s knowledge base and empowering it to dynamically adapt to ransomware’s rapidly evolving behaviors and diverse environmental influences.

Table 1. Studies related to ransomware behavior prediction.

Author	Problem	Solution	Method	Tools	Empirical	Limitation
[33]	The detection of ransomware based on past attack data is not suitable to detect novel, zero-day attacks, which are common nowadays.	The behavioral patterns extracted from the dynamic analysis of ransomware during the execution time were used to train a prediction model.	Support vector machines (SVM) were used to build the prediction model based on the behavioral data.	ScikitLearn, and Pandas	Yes	This approach also uses historical behavior to predict future ones. This is not suitable for evasive ransomware that uses obfuscation and polymorphism to change its behavior from time to time.
[34]	Advanced malware can obfuscate much of its traces through many mechanisms, such as metamorphic engines. Therefore, the detection of such malware has become a significant challenge for malware analysis mechanisms.	A regression model to predict advanced malware based on a selected set of significant features extracted from a dataset of malware runtime data.	The dataset is created by executing real-world malware samples and capturing the behavioral data into trace files.	N/A	Yes	The model was trained using historical data of existing and known malware samples. The dataset does not contain the future behaviors necessary for accurate prediction models.
[35]	Sophisticated Android malware families often implement techniques aimed at avoiding detection. Split-personality malware, for example, behaves benignly when it detects that it is running on an analysis environment such as a malware sandbox, and maliciously when running on a real user’s device.	Exploiting sandbox detecting heuristic prediction to predict and automatically generate bytecode patches.	An Andronew, a heuristic approach, was used based on API calls collected during the execution time of the malware.	Sandbox	Yes	The heuristics were performed based on historical data, which limits the ability of this approach to predict the future behavior of malware
[41]	Zero-day malware attacks are challenging due to the polymorphic nature of the malware.	Generating synthesized malware samples based on existing malware signatures derived from the static analysis of malware payloads.	GAN algorithm to generate artificial malware samples.	Keras, and Tensorflow	Yes	The static analysis adopted by the study does not reveal the behavioral aspect of the malware as polymorphism works during the runtime. In addition, the packing and encryption techniques used by sophisticated malware prevent the static analysis from exploring the malware features.
[42]	Existing malware detection is not accurate enough.	A cluster-based detection engine that is trained based on artificial patterns represents the trending of malware behavior.	GAN algorithm to create malware patterns.	N/A	N/A	There was no evidence of the applicability and efficacy of the model.
[7]	Malware authors have the ability to reveal the features used by detection models.	MalGAN model that attacks black-box machine-learning detection models.	A substitute detector to fit the black-box malware detection system.	N/A	Yes	The data used for model training were general and limited to malware operational behavior. The context was not captured.
[43]	The ransomware changes its behavior which makes it difficult to detect.	The study studies data collected from the ransomware process and its interaction with the file system.	It used malware development toolkits to create ransomware samples.	ADMMutate, Clet, and Phatbot	Yes	The study is limited to the ability of the tools to manually create samples, which makes it impractical to have a diversified dataset.
[44]	Detecting novel malware attacks is difficult as the behavior changes continuously.	The model examines the patterns in the data and studies the evolution of the malware behavior.	It used a collection of data from previous malware infections to train a logistic regression algorithm.	N/A	Yes	Relying on the evolution of the attack behavior to forecast future attacks is not sufficient to visualize the sophisticated malware attacks.
[46]	The new types of malware tend to be more difficult to detect than older ones. This has made content-based, signature-based, and pattern-matching techniques less effective in detecting and preventing ransomware attacks.	Utilized the neural network algorithm to predict the future occurrences of ransomware and malware attacks over time.	Time-series regression-based neural network algorithm model.	TensorFlow, Keras, NumPy, Matplotlib, and Pandas	Yes	The model concentrates solely on data pertaining to processing operations, disregarding the context in which the process was executed.
[47]	Existing ransomware attack predictions are not tailored for IoT systems that are diverse and resource-constrained environments.	A technique for predicting ransomware using contextual data and utilizing a context ontology to gather information characteristics of ransomware attacks against the IoT.	An ontology approach with SVM.	N/A	Yes	Relying only on contextual data and ignoring the behavioral data is insufficient for modeling the characteristics of the evasive malware attacks.
[49]	Detection solutions alone are no longer enough to protect against malware due to the increasing rate of zero-day attacks.	An early prediction of malware attacks in Android devices was proposed. By capturing the implicit contextual relations between various data, the model predicts the suspicious behavior of a running process using data collected during the early stages of the attack within the same session.	LSTM and ensemble learning.	N/A	Yes	This approach is inadequate in terms of the necessary data required for an accurate prediction and is unable to anticipate the future behavior of the malware.
[49]	Due to the obfuscation techniques employed by advanced malware, detection is no longer enough, and there is a need for methodologies to predict future behavior instead.	A rapid sequence snapshot analysis was used to make the prediction decisions.	A set of random snapshots were taken from the APIs and permission data and used to train an ensemble LSTM model that is used for the prediction.	Tensorflow	Yes	The LSTM was trained on historical data only, which assumes that these historical attack patterns are likely to reoccur in future attacks. This does not hold, especially with the use of obfuscation and polymorphic strategies adopted by the malware to change the attack behavior.
[59]	Existing approaches to detect the malware need to collect enough data which takes more time, during which the sabotage has likely already been inflicted by the time of detection.	Predicting the behavior based on a short snapshot of behavioral data.	An ensemble RNN. The method was able to predict the attack within 5 s with an accuracy of around 94%.	Keras, and Tensorflow	Yes	The method relies on historical data to predict the behavior. This approach is not suitable for obfuscated behavior that tries to show a major difference between past and future attacks.

provides a critical overview of existing ransomware behavior prediction studies, dissecting their problem statements, proposed solutions, methodologies, tools, and inherent limitations.

2.4. Addressing Prevention Gaps

Existing ransomware defense solutions often focus narrowly on operational and/or managerial aspects, but they ignore situational factors that significantly influence the effectiveness of these strategies. To address this limitation, we propose a comprehensive framework that combines both approaches, allowing the development of situationally aware solutions designed for the context of specific attacks. Moreover, the proposed framework emphasizes predictive situational awareness to trigger active defenses and assess the effectiveness of hardening strategies. By integrating evidence from diverse sources, the model could identify potential ransomware threats early and allow for timely preventive measures. This framework not only provides direction for future research but also serves as a baseline for evaluating current investigations and future work that can build on our framework to create adaptive and context-aware solutions for ransomware prediction and prevention.

To summarize, the capacity of ransomware to masquerade as legitimate software and irreversibly encrypt data renders early detection paramount yet arduous. The reliance on robust cryptographic mechanisms and the rapid evolution of attack patterns impedes conventional detection methodologies, frequently resulting in failures due to data insufficiency within the critical pre-encryption window and the malware’s polymorphic nature. To mitigate these challenges, we use a dual-pronged strategy: (1) leveraging GANs to augment sparse datasets with realistic synthetic samples, and (2) employing adaptive feature selection via mutual information to dynamically prioritize salient features. This approach addresses data scarcity and dimensionality complexities, ensuring model efficacy against evolving ransomware tactics. Furthermore, DBNs are investigated as detection architectures, with training optimizations such as early stopping to mitigate overfitting. This integrated approach enhances detection accuracy, synergistically combining robust modeling techniques with adaptive strategies to counter ransomware’s dynamic behavior.

3. Methodology

This research introduces a novel adaptive feature selection methodology to counter evolving ransomware. We use Incremental Mutual Information and batch processing to update our detection model with the latest attack patterns. We also address model design and training by using dynamic stopping criteria that account for ransomware’s uncertainty and variability. This comprehensive approach aims to boost early ransomware detection accuracy, prevent overfitting, and create more reliable detection models, significantly advancing proactive cybersecurity. The process involves augmentation, feature selection, and model training/testing. First, GANs synthesize attack samples, mimicking real ransomware behavior. Next, in the feature selection phase, we identify critical features from ransomware’s runtime data. This adaptive feature selection keeps our detection model resilient against changing threats. Finally, a DBN is trained on this augmented data with selected features, resulting in a robust early ransomware detection model. Figure 2 shows the full architectural overview.

3.1. Phase 1: GAN Pre-Encryption Data Augmentation

The GAN is used to generate artificial attack data that mimics ransomware behavior during the early stages of the attacks. The early stage of the attack is the timeframe that precedes data encryption, also called pre-encryption. The lack of data and attack patterns is a major challenge that early detection solutions face. In this phase, this data insufficiency challenge is addressed by using GAN to generate data that compensate for the missing and/or unavailable attack patterns. The underlying principle behind GANs involves a game-theoretic approach, where two neural networks, the generator and the discriminator, are trained simultaneously through adversarial processes.

The generator’s job is to create data. Initially, it produces data pretty much at random. Over time, it learns to generate more realistic data, aiming to make its outputs indistinguishable from real data. It does not have access to the actual data but learns from the feedback it receives from the discriminator. The discriminator acts as a critic or judge. It reviews both the real data, and the fake data produced by the generator. Its task is to distinguish between the two, identifying what is real and what is fake. The better the discriminator becomes at distinguishing real from fake, the more pressure it puts on the generator to improve its data generation capabilities.

3.2. Phase 2: Incremental Mutual Information Selection (IMIS)

Mutual information-based incremental selection (IMIS) is employed to extract the most salient feature subset for the ransomware detection model. This technique leverages mutual information to quantify the relevance of candidate features while mitigating redundancy with already selected features. Its non-parametric nature renders it adaptable to both linear and non-linear relationships, and categorical and continuous data types. Unlike correlation, which is limited to linear dependencies, mutual information captures the full spectrum of variable relationships, making it a robust tool for feature selection in complex datasets characterized by intricate, non-linear dependencies.

The core principle of IMIS is to utilize entropy as a measure of the information content a candidate feature contributes towards the target label. High information content translates to high relevance. Subsequently, redundancy between the candidate and already selected features is quantified, and high redundancy leads to a diminished feature significance. This ensures the selected feature set maximizes relevance while minimizing redundancy. Notably, IMIS incorporates incremental learning, enabling dynamic re-evaluation of feature significance upon the arrival of new data, thus empowering the detection model to adapt to evolving ransomware behaviors.

The mutual information feature selection process involves computing mutual information scores between each feature and the target variable, followed by a ranking of features based on these scores. Features exhibiting the highest scores, indicative of maximum information content, are prioritized for model training, while those with low scores are discarded, resulting in a streamlined and potentially more performant model. The culmination of this phase is the derivation of an IMIS-selected feature set, comprising the most relevant and non-redundant features, which serve as input for the detection model in Phase 3.

3.3. Phase 3: Deep Belief Network-Based Ransomware Detection Model

The feature subset extracted by IMIS in Phase 2 serves as the input for training a DBN designed for ransomware detection. A DBN comprises multiple layers of stochastic, latent variables, typically binary, organized into a directed belief network with the exception of the uppermost two layers, which constitute an undirected graphical model. This structural architecture empowers the DBN to efficiently model intricate data distributions and learn deep, hierarchical representations. DBN training is bifurcated into two primary stages: unsupervised pre-training, employing Restricted Boltzmann Machines (RBMs) to initialize network weights, and supervised fine-tuning, which refines these weights for classification. This framework mitigates the challenges inherent in directly training deep architectures via supervised learning, effectively capturing underlying data distributions and initializing the network in a parameter space proximal to the optimal solution.

In the context of ransomware detection, the DBN functions as a classifier, discerning malicious software based on the IMIS-selected feature subset. The DBN is trained on a comprehensive dataset, encompassing known ransomware samples and benign applications generated in Phase 1. Consequently, the model learns to differentiate between these categories by recognizing the deep, hierarchical patterns that distinguish ransomware activities from legitimate software operations. Upon completion of training, the DBN can classify novel, unseen software samples with a high degree of accuracy, effectively distinguishing between malicious and benign entities. The full Python implementations of the BBM-GAN data augmentation, IMIS feature selection, and UA-DES early stopping strategy can be found in Appendix B.

4. Experimental Evaluation and Analysis

This section details our multi-phase framework for early ransomware detection, designed to boost accuracy and lower false positives.

• Phase 1 uses a Bi-Gradual Minimax GAN (BGM-GAN) to generate synthetic ransomware attack patterns, overcoming data scarcity.
• Phase 2 employs the IMIS technique to dynamically select relevant features and adapt to evolving ransomware.
• Phase 3 utilizes a DBN with a UA-DES technique to optimize training and prevent overfitting.

The following sections elaborate on the methodology and results of each phase.

4.1. Dataset, Environment, and Metrics

We built a comprehensive dataset of 8152 crypto-ransomware samples from diverse families like Cerber, TeslaCrypt, CryptoWall, Petya, and WannaCry, reflecting the complexity of current threats. To train accurate detection models, we also included 1000 benign programs from informer.com. This balanced dataset, with its detailed documentation of ransomware families (see

Table 2. The ransomware families in the dataset.

Family	Samples	Family	Samples
Cryptolocker	741	Cerber	263
Cryptowall	706	Filecryptor	772
Cryrar	583	Crypt	789
Locky	567	CTB_Locker	560
Petya	593	Satana	289
Reventon	617	CryptXXX	651
Teslactypt	398	Sage	187
WannaCry	436

), ensures our research is transparent and reproducible, providing a strong foundation for developing robust ransomware detection mechanisms. Sample rows of the ransomware dataset, including extracted API call features used for model training, are illustrated in Appendix C.

For evaluation, we used Python libraries including Scikit-learn, Pandas, NumPy, and SkFeature. TensorFlow was essential for developing the GAN model’s generator and discriminator networks. We used Anaconda as our integrated development environment and Keras to streamline architecture development and model optimization. Scikit-learn, NumPy, and Pandas also handled data preprocessing and feature extraction. Complementing the malicious samples, the dataset incorporates 1000 benign programs sourced from a reputable application repository, https://www.informer.com accessed on 5 Aug 2023.

Accuracy, detection rate (DR), and false positive rate (FPR) are standard metrics for evaluating malicious program detection, including ransomware, and other machine learning tasks. While accuracy measures overall correctness, DR (also known as recall) indicates the proportion of actual attacks correctly identified. Precision shows how many predicted attacks were truly correct, and FPR quantifies benign instances incorrectly flagged. The F1-score, a balanced metric for precision and recall, is used when both false positives and false negatives are critical. Formal definitions are provided in Equations (1)–(5) below, where TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives.

$$ACC={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(1)

$$DR={\displaystyle \frac{TP}{TP+FN}}$$

(2)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(3)

$$FPR={\displaystyle \frac{FP}{FP+TN}}$$

(4)

$$F1={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(5)

4.2. Phase 1

GANs, introduced in 2014, are a major advancement in deep learning. They consist of two competing neural networks: a generator that creates data mimicking a target distribution, and a discriminator that tells real from generated data. This adversarial process iteratively refines both networks, producing increasingly realistic synthetic data. GANs are versatile, used in areas like image and video synthesis, data augmentation, anomaly detection, and are particularly relevant for ransomware detection.

In the context of adversarial attacks, GANs are employed to model the attacker’s objective. Given a sample space X, where x represents a benign sample and g(x) > 0 denotes the classification function yielding a benign result, the attacker aims to generate a malware sample $x^{*}$ such that $g\left(x^{*}\right)>0$. This objective can be formally expressed as

$$x^{*}=\mathit{arg} {\mathit{max}}_x \widehat{g}\left(x\right), s.t.d\left(x,x^{*}\right)\le d_{max}$$

(6)

The GAN reduces the loss function value $V$ during the training of both generator $\mathcal{G}$ and discriminator $\mathcal{D}$ by solving the following optimization function:

$$\underset{\mathcal{D}}{\mathit{min}} \underset{\mathcal{G}}{\mathit{max}} V\left(\mathcal{G},\mathcal{D}\right)$$

(7)

where

$${V\left(\mathcal{G},\mathcal{D}\right)=E}_x\left[\mathit{log} \left(D\left(x\right)\right)\right]+E_z\left[\mathit{log} \left(1-D\left(G\left(z\right)\right)\right)\right]$$

(8)

Here, z denotes samples drawn from the latent noise distribution. While existing GAN-based malware detection methods show promise, their efficacy in early ransomware detection is limited by the inherent scarcity of ransomware samples. This scarcity results in a poorly defined empirical estimate of the authentic ransomware data distribution, leading to substantial divergence from the generator’s output distribution. Consequently, the discriminator frequently rejects synthetic samples, hindering the generator’s ability to learn and produce realistic ransomware examples. To address this, our algorithm is structured into two distinct phases: a GAN-based data augmentation phase, comprising a generator and discriminator, followed by an early detection phase that leverages the augmented dataset.

4.2.1. GANs Generative Adversarial NWs

The GAN generates artificial pre-encryption attack patterns from real ransomware data. This addresses data insufficiency in the pre-encryption phase by adding realistic, fictional samples to the dataset.

The discriminator network evaluates input samples by propagating them through its constituent layers, thereby computing a probabilistic score that quantifies the likelihood of a given sample’s provenance—specifically, its authenticity versus its synthetic origin. Throughout the iterative training process, the generator network refines its capacity to synthesize increasingly realistic data, while the discriminator concurrently enhances its ability to discriminate between authentic and synthetic exemplars. This adversarial dynamic compels the generator to produce synthetic samples of progressively heightened fidelity, which, in turn, drives the discriminator towards improved discriminative performance.

Figure 3 illustrates the fundamental GAN architecture, comprising a generator and a discriminator engaged in an adversarial relationship. The generator synthesizes synthetic ransomware instances, emulating real attack patterns to create realistic training data. These instances are presented to the discriminator, which endeavors to classify them as authentic or artificial. The generator aims to deceive the discriminator by producing synthetic samples whose probability distribution increasingly approximates that of authentic ransomware. Conversely, the discriminator is optimized to accurately distinguish between authentic and synthetic attacks. Successful discrimination provides feedback to the generator, prompting parameter readjustment to generate synthetic instances that more closely resemble authentic ones. This iterative process drives both networks towards a Nash equilibrium (Equations (6)–(8)). Upon completion of this adversarial training, the resulting balanced dataset of authentic and high-fidelity synthetic pre-encryption ransomware samples forms the basis for training the subsequent early detection module.

Refined Minimax Loss: The Enhanced Bi-Gradual Approach

GANs rely on loss functions to measure the difference between the probability distributions of real and generated data. To improve upon the standard Minimax loss (Equation (6)), we propose the Bi-Gradual Minimax loss function. While the generator and discriminator use separate probability distributions for their respective loss calculations, both derive their final loss value from the unified formula presented in Equation (9).

$${V\left(\mathcal{G},\mathcal{D}\right)=E}_x\left[\partial \mathit{log} \left(D\left(x\right)\right)\right]+E_z\left[\left(1-\partial \right)\mathit{log} \left(1-D\left(G\left(z\right)\right)\right)\right]$$

(9)

In a GAN, the discriminator, denoted as D(x), estimates the probability that a real data instance x is authentic. The generator, G(z), produces synthetic data from a noise vector z. Consequently, D(G(z)) represents the discriminator’s assessment of the probability that a generated instance is real. The expected values Ex and Ez are taken over real data instances x and noise vectors z (or, equivalently, generated instances G(z)), respectively.

To refine the probability estimation, particularly when training a GAN-based early detection model with pre-encryption data, we introduce a gradual up-weighting coefficient, represented by ∂. This coefficient modulates the discriminator’s and generator’s estimations, and its calculation is detailed in Equation (10).

$$\partial ={\displaystyle \frac{1+a}{R}}$$

(10)

The gradual coefficient, ∂, is defined as the ratio a/R, where R represents the fixed number of real data instances and a represents the growing number of synthetic instances. Therefore, ∂ is directly proportional to a. In the early stages of training, when a is small, ∂ is also small because R is much larger than a. As the generator produces more synthetic samples, a increases, leading to a corresponding increase in ∂. This dynamic adjustment of ∂ allows the loss function to adapt, starting with a relaxed constraint and gradually tightening it as more synthetic data becomes available. This is particularly beneficial for early ransomware detection, where initial data is often limited. Consequently, the GAN discriminator’s constraints transition from lenient to stringent as the volume of generated synthetic attack patterns grows.

To dynamically adjust the upper bound of loss estimation in the Minimax loss function, we introduced the coefficient ∂. This adaptation is crucial during the pre-encryption phase of ransomware attacks, where probability distributions are highly variable. Initially, with few synthetic instances, the discriminator’s term in the loss function (Equation (6)) has minimal influence, allowing the generator’s term to dominate and produce viable attack patterns. When the discriminator’s term is minimized due to this imbalance, misclassification of synthetic patterns as real is more likely. However, as the number of generated synthetic attack patterns increases, the discriminator’s influence in Equation (6) grows, while the generator’s influence diminishes. This shift empowers the discriminator to more accurately distinguish synthetic instances from real ones.

4.2.2. Early Detection with Feature Selection

In Phase 1, the augmented dataset is used to train an LSTM-based ransomware detection model. Feature selection is performed using mutual information feature selection (MIFS), to select the most informative features, as defined in Equation (11).

$$J\left(X_k\right)=I\left(X_k;Y\right)-\beta \sum _{X_jϵS}I\left(X_j;X_k\right)+\gamma \sum _{X_jϵS}I\left(X_j;X_k|Y\right)$$

(11)

In this equation, I(X_k;Y) represents the mutual information between the candidate feature X_k and the class label Y, and I(X_j;X_k∣Y) denotes the conditional mutual information between X_k and selected features X_j (from set S), given the class label Y. The coefficients β and γ range from 0 to 1. By reducing data dimensionality through salient feature selection, MIFS helps prevent overfitting, enhances detection accuracy, minimizes false positives, and simplifies model complexity, making it suitable for real-time applications like early ransomware detection. The algorithm ranks features by entropy, selecting the top n features for input into the LSTM network.

The LSTM network was chosen for the detection module due to its inherent capacity to capture temporal dependencies, a crucial attribute for early ransomware detection. The LSTM’s cell state, with its memory mechanism for recalling past information, facilitates tracking polymorphic ransomware behavior during the pre-encryption phase, enabling effective monitoring. The LSTM network incorporates three gate types: input (Equation (12)), forget (Equation (13)), and output (Equation (14)):

$$i_t=\sigma \left({\omega }_i\left[h_{t-1},x_t+b_i\right]\right),$$

(12)

$$i_f=\sigma \left({\omega }_f\left[h_{t-1},x_t+b_f\right]\right),$$

(13)

$$i_o=\sigma \left({\omega }_o\right[h_{t-1},x_t+b_o\left]\right)$$

(14)

The variables $i_t, i_f, and i_o$ denote the input, forget, and output gates, respectively. During training, the LSTM network utilizes the selected features and data for model optimization. The model comprises input, hidden, and output layers. The number of input layer nodes corresponds to the number of features selected by MIFS. These nodes transmit data to the hidden layers by multiplying it with the input weights. The hidden layers, whose number and nodes are optimized during training via bias adjustments, process data using activation functions. The Rectified Linear Unit (ReLU) function, as defined in Equation (15), serves as the activation function for all hidden layer nodes, except for those in the layer preceding the output, where the sigmoid function from Equation (16) is employed. The output layer receives input from the final hidden layer’s sigmoid activations and classifies instances as malicious or benign based on a threshold. Instances with activation values exceeding 0.5 are classified as malicious; however, this threshold is adjustable.

$$R\left(x\right)=\mathit{max}\left(0,x\right)$$

(15)

$$\mathrm{\varnothing }\left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$$

(16)

The LSTM model employs 10-fold cross-validation, partitioning the dataset into 90% training data and 10% testing data. The training data is used to construct the model, while the testing data is used to evaluate its performance. This process is iterated 10 times, with the accuracy recorded for each iteration. The average accuracy across all iterations is then calculated to determine the model’s overall performance.

4.2.3. Performance Evaluation

This section details the facilities and setup used for our analysis, including data engineering (i.e., construction of a crypto-ransomware binary corpus), feature selection, correlation coefficient calculation, weighting factor management, and the formulation of an adjustment function.

Table 3. Performance analysis of the proposed Bi-Gradual Minimax GAN (BGM-GAN) across varying training epochs.

	Epochs	E30	E90	E150	E210
Features
5		0.890	0.934	0.942	0.926
10		0.908	0.951	0.954	0.957
15		0.898	0.937	0.950	0.961
20		0.915	0.956	0.956	0.962
25		0.910	0.941	0.971	0.969
30		0.927	0.931	0.980	0.967
35		0.936	0.938	0.979	0.981
40		0.937	0.953	0.976	0.986
45		0.939	0.946	0.981	0.992
50		0.938	0.922	0.984	0.995

and

Table 4. Training duration as a function of epoch count.

Number of Epochs	Training Time (s)	Number of Epochs	Training Time (s)
30	70 s	150	307 s
90	189 s	210	431 s

present accuracy and training time data across various epochs, while Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 provide a comparative visual representation of the results.

• Experimental Facilities

The experimental setup was conducted utilizing a suite of Python libraries. Scikit-learn, Pandas, NumPy, and SkFeature were employed for data analysis and evaluation. The GAN was implemented using TensorFlow within the Anaconda integrated development environment (IDE), with Keras facilitating streamlined architecture design and optimization. Scikit-learn, NumPy, and Pandas were used for data preprocessing and feature extraction.

• Corpus of crypto-ransomware binaries

The dataset used in this study consists of a total of 9152 binary samples, comprising 8152 crypto-ransomware binaries sourced from the public repository, VirusShare (http://www.virusshare.com, accessed on 14 March 2023) repository [60,61,62,63,64]. This corpus comprises 8152 samples, representing diverse ransomware families including Cerber, TeslaCrypt, CryptoWall, Petya, and WannaCry. These samples were collected between September 2016 and August 2017. Additionally, 1000 benign programs were downloaded from informer.com [63,64,65,66], a widely used repository for Windows-based applications. These files, commonly referred to as portable executable (PE) files, are fundamental to malware analysis. PE files serve as the standard executable format for object code and dynamic-link libraries (DLLs) in Windows operating systems. Both static and dynamic analyses of PE files are crucial for comprehending the structure, functionalities, behavior, and potential impact of ransomware. The dissection of PE files enables the extraction of valuable metadata, the identification of malicious signatures, and the understanding of ransomware behavior, including encryption methods, server communication, and persistence mechanisms. Binary analysis is essential for developing countermeasures and detection signatures, as well as for understanding ransomware tactics. In this study, dynamic analysis was employed, utilizing the PE file within a sandboxed environment.

• Data Engineering

A diverse corpus of benign applications was compiled, including compression, encryption, editing, office, developer, multimedia, driver, browser, and game software. Each application was rigorously verified using VirusTotal, a platform utilizing 56 antivirus (AV) engines for malware classification. Samples flagged as malicious by fewer than five AV engines were excluded, ensuring only applications trusted by all fifty-six engines were retained. Both ransomware and benign software were then executed and analyzed in the Cuckoo Sandbox. This environment meticulously monitored the resulting processes, generating distinct API trace files for each sample. These trace files formed the primary dataset for this study.

• Feature selection

In our previous research, we employed enhanced mutual information feature selection (EMIFS) to identify the most salient features for the pre-encryption stage of the crypto-ransomware lifecycle. We conducted experiments with varying epochs, utilizing feature counts ranging from 5 to 50, in increments of 5. The results demonstrated that accuracy fluctuated with the number of epochs. The dataset was partitioned into training and testing sets using 10-fold cross-validation. Classifier accuracy was evaluated using the testing data.

• Compilation and comparison of results

To demonstrate the efficacy of our BGM-GAN technique, we compared its performance with GAN [67], CNN [68], and DBN [69] models. Feature set sizes ranging from 5 to 50, in increments of 5, were evaluated experimentally. Table 3 shows the accuracy results based on number of epochs. Figure 4, Figure 5, Figure 6 and Figure 7 illustrate the classification accuracy results, demonstrating that BGM-GAN outperforms the other models.

Figure 4 compares the performance of our BGM-GAN against standard GAN, Convolutional Neural Network (CNN), and DBN models, trained for 30 epochs across varying feature counts. BGM-GAN demonstrates a consistent improvement in accuracy with increasing features, peaking at 0.939 with 45 features and maintaining high performance even with a minor drop to 0.938 at 50 features. The standard GAN model reaches a maximum accuracy of 0.930 at 25 features, declining from 0.891 at 5 features to 0.916 at 50 features. The CNN model peaks at 0.929 with 30 features, starting at 0.865 and ending at 0.919. The DBN model shows the least variability, peaking at 0.914 with 20 and 30 features, and ending at 0.906. Overall, BGM-GAN consistently outperforms the other models, reaching the highest accuracy of 0.939 at 45 features, as shown in Figure 4, Figure 5, Figure 6 and Figure 7.

Figure 5 compares the performance of BGM-GAN against standard GAN, CNN, and DBN models, all trained for 90 epochs and evaluated across varying feature counts. BGM-GAN demonstrates strong initial accuracy, starting at 0.934 with five features and peaking at 0.951 with ten features. Though accuracy dips to 0.931 at thirty features, it rebounds, reaching a second peak of 0.953 at forty features before a slight decline to 0.922 at fifty features. In contrast, the standard GAN model peaks at 0.927 with thirty features, starting at 0.882 and stabilizing at 0.927 at fifty features. The CNN model reaches a peak of 0.924 at forty features, starting at 0.874 and ending at 0.916. The DBN model peaks at 0.932 with thirty features, starting at 0.884 and ending at 0.916. Despite feature-dependent variations in accuracy across all models, BGM-GAN consistently outperforms the others, achieving peak accuracies of 0.951 and 0.953 at ten and forty features, respectively.

Figure 6 compares the accuracy of BGM-GAN, standard GAN, CNN, and DBN models, all trained for 150 epochs, across varying feature counts. BGM-GAN demonstrates a steady increase in accuracy, starting at 0.942 with five features and peaking at 0.984 with fifty. Though it experiences a slight dip to 0.950 at fifteen features and 0.976 at forty, BGM-GAN consistently outperforms the other models. In contrast, the standard GAN model peaks at 0.961 with twenty-five features, declining to 0.945 at fifty, with a low of 0.879 at fifteen. The CNN model peaks at 0.949 with forty-five features, ending at 0.899. The DBN model’s performance fluctuates, peaking at 0.944 with thirty-five features and ending at 0.930. BGM-GAN’s consistent upward trend and peak accuracy of 0.984 highlight its superiority.

Figure 7 compares the performance of BGM-GAN, standard GAN, CNN, and DBN models, all trained for 210 epochs, across varying feature counts. BGM-GAN demonstrates a consistent upward trend, starting at 0.926 with five features and peaking at 0.995 with fifty, highlighting its robustness. In contrast, standard GAN fluctuates, peaking at 0.979 with thirty features and ending at 0.977. The CNN model peaks at 0.959 with twenty-five features, and the DBN model at 0.950, also with twenty-five features. BGM-GAN consistently outperforms the others, achieving the highest accuracy of 0.995.

Figure 4, Figure 5, Figure 6 and Figure 7 demonstrate that our BGM-GAN model consistently achieved superior detection accuracy compared to the GAN model presented in [67] across all training epochs. Furthermore, BGM-GAN outperformed other deep learning models, including CNN and DBN. This superior performance underscores the efficacy of our Bi-Gradual Minimax loss function in accurately estimating the divergence between real and synthetic data distributions, even when confronted with limited pre-encryption ransomware attack data. These results validate our hypothesis that prior research overlooked the detrimental impact of data scarcity on generator and discriminator performance, thereby hindering the production of realistically metamorphic ransomware samples.

The comparative results depicted in Figure 4, Figure 5, Figure 6 and Figure 7 reveal that accuracy generally improves with an increase in feature count, up to a critical threshold. Beyond this threshold, accuracy plateaus or diminishes. The optimal feature count for maximizing accuracy is contingent upon the number of training epochs. For example, with 30 training epochs, peak accuracy is observed at 45 features, whereas with 90 epochs, optimal performance is attained at 20 features.

The observed accuracy declines are attributed to overfitting, a consequence of increased feature counts and training epochs. As the number of features increases, the dimensionality of the data also escalates. This heightened dimensionality adversely affects the detection model’s accuracy, particularly when handling data with insufficient ransomware behavior patterns from early phases. Moreover, as epoch counts increase, the risk of overfitting intensifies, diminishing the loss function’s effectiveness in measuring divergences between real and synthetic patterns. This phenomenon explains the accuracy decline observed beyond certain feature and epoch thresholds. Table 4 demonstrates BGM-GAN’s linear training time scaling with epochs (2–4 ms/step, ~2 s/epoch).

Figure 8 and Figure 9 show that our BGM-GAN model outperforms existing techniques in recall and false positive rate across varying feature counts and training epochs. This enhanced performance is attributed to BGM-GAN’s ability to capture early attack patterns with limited data, consistent with Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9.

In summary, this model introduces a novel approach to early-stage ransomware attack detection, addressing the critical challenge of limited and rapidly evolving attack patterns. Unlike prior studies that relied on static feature sets, which are quickly rendered obsolete by the dynamic nature of ransomware, this model leverages GANs for dynamic data augmentation. By integrating an enhanced loss function into the GAN, the model generates more accurate and representative attack patterns, significantly improving detection performance. This advancement addresses a significant gap in contemporary cybersecurity research and practice, enhancing both the speed and efficacy of ransomware detection.

4.3. Phase 2

To adapt to the evolving nature of ransomware, our model employs Incremental Mutual Information (IMI). This method dynamically recalculates feature relevance with each new data batch, ensuring the detection model remains sensitive to emerging attack patterns and maintains optimal feature selection for early detection.

To mathematically define IMI’s feature relevance update, let X represent the input feature matrix and Y represent the target class vector. Given a feature set Let $X=\{x_1,x_2,...,x_n\}$ and target variable $Y$, the mutual information between feature $x_i$ and target $Y$ is denoted as $I(x_i;Y)$.

The dynamic update process begins with calculating the initial mutual information for each feature relative to the target using

$$I(x_i;Y)\mathrm{for}i=\mathrm{1,2},...,n$$

(17)

When new data arrives, the mutual information for each feature is recalculated and updated as follows. Let X_new represent the new data. The updated mutual information is given by

$$I_{new}\left(x_i;Y\right)=\alpha \cdot I_{prev}\left(x_i;Y\right)+\left(1-\alpha \right)\cdot I\left(x_i;Y∣X_{new}\right)$$

(18)

where $I_{prev}(x_i;Y)$ is the previous mutual information value and $\alpha$ is a weighting factor $(0\le \alpha \le 1)$ that balances the impact of new data versus historical data. If new features $x_n+1,x_n+2,...,x_n+m$ are added, calculate $I(x_n+\mathrm{j};Y)$ for $j=\mathrm{1,2},...,m$.

We periodically evaluate feature relevance by updating mutual information scores and deprioritize and remove features with significantly lower scores based on the desired feature count. The model adjustment feedback loop is guided by performance metrics (e.g., accuracy, precision) that trigger feature set re-evaluation if performance deteriorates below a predefined threshold. The weighting factor (α) in dynamic feature selection balances the impact of historical and new data, determining the influence of past mutual information (MI) values relative to newly calculated MI. α is adaptively adjusted through correlation analysis that ensures an optimal balance by assessing the relationship between historical and incoming data. The mathematical formulation is next.

• Correlation Coefficient Calculation

Let $X_{hist}$ represent the historical data and $X_{new}$ represent the new data for a certain feature. Calculate the Pearson correlation coefficient, denoted as $r$, between $X_{hist}$ and $X_{new}$. The formula for Pearson correlation coefficient $\rho$ is

$$\rho ={\displaystyle \frac{\sum (X_{hist}-{X¯}_{hist})(X_{new}-{X¯}_{new})}{\sqrt{\sum {\left(X_{hist}-{X¯}_{hist}\right)}^2{\left(X_{new}-{X¯}_{new}\right)}^2}}}$$

(19)

Here, ${X¯}_{hist}$ and ${X¯}_{new}$ are the means of the historical and new data, respectively.

• Correlation-Based Adjustment of the Weighting Factor

The correlation coefficient $r$ ranges from −1 to 1, where 1 indicates a perfect positive linear relationship, −1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship.

The value of $r$ can be used to adjust $\alpha$. For example,

If ∣r∣ is high (close to 1), it implies that the new data is highly correlated with the historical data. In this case, a higher α may be appropriate, as it suggests that historical data is still very relevant.

If ∣r∣ is low (close to 0), it indicates that the new data is not well-correlated with the historical data. A lower α might be more suitable in this scenario to give more weight to the new data.

• Threshold-Based Adjustment Function for Weighting Factor Stability

A threshold mechanism is employed to prevent significant variations in α resulting from minor fluctuations in r. If the change in r is less than the threshold, α is maintained at its current value. This adaptive strategy enables α to dynamically respond to evolving data relationships, ensuring feature selection remains pertinent to contemporary trends, which is essential for ransomware attack detection.

4.3.1. IMIS-DBN Integration

The integration of IMIS with a DBN significantly enhances ransomware detection accuracy. DBNs are adept at learning intricate patterns from high-dimensional ransomware data. IMIS’s continuous evaluation and dynamic adjustment of feature selection, based on evolving attack patterns, ensures the DBN is trained on a compact, relevant feature subset. This integration not only improves detection accuracy but also optimizes computational resource utilization. The process operates in two distinct phases: initially, IMIS selects features based on their mutual information scores relative to the target class; subsequently, IMIS dynamically updates this feature subset with new data, ensuring the DBN utilizes current information. This incremental approach is efficient, scalable, and particularly beneficial for resource-constrained systems.

In the second phase, the IMIS-selected feature subset serves as input for the DBN, facilitating deep learning-based ransomware detection. The DBN’s layered architecture extracts hierarchical representations, crucial for identifying complex threats. IMIS’s adaptability prevents the DBN from being overwhelmed by data volume or misled by obsolete features, which is vital in rapidly evolving cyber threat landscapes. IMIS’s dynamic feature selection enables the detection of both known and novel threats, fostering proactive defense against evolving malware.

Algorithm 1 (below) presents the pseudocode for IMIS, which systematically selects and updates ransomware features for effective detection. This framework ensures IMIS efficiently adapts to new data patterns while maintaining computational efficiency, rendering it particularly effective for dynamic ransomware behavior.

Algorithm 1. Incremental Mutual Information Selection (IMIS) [70].

Input:

Data_Batches: Stream of data batches from devices

Target_Class: The class variable for intrusion detection (e.g., normal or attack)

Alpha: Weighting factor for balancing historical and new data (initially set)

Threshold: Threshold for significant change in mutual information

Output:

Selected_Features: Set of features selected for intrusion detection

Procedure IMIS (Data_Batches, Target_Class, Alpha, Threshold):

Initialize Historical_MI as an empty dictionary

Initialize Selected_Features as an empty set

for each Batch in Data_Batches:

Current_MI = CalculateMutualInformation(Batch, Target_Class)

Historical_MI = UpdateFeatureRelevance(Historical_MI, Current_MI, Alpha)

Selected_Features = SelectAndUpdateFeatures(Historical_MI, Selected_Features, Threshold)

Yield Selected_Features

Procedure CalculateMutualInformation(Batch, Target_Class):

return {Feature: ComputeMutualInformation(Feature, Target_Class) for Feature in Batch}

Procedure UpdateFeatureRelevance(Historical_MI, Current_MI, Alpha):

return {Feature: Alpha ∗ Historical_MI.get(Feature, 0) + (1 − Alpha) ∗ MI for Feature, MI in Current_MI.items()}

Procedure SelectAndUpdateFeatures(Historical_MI, Selected_Features, Threshold):

return {Feature for Feature, MI in Historical_MI.items() if MI > Threshold or Feature in Selected_Features}

4.3.2. Training for Adaptive Detection

This section details the architecture, configuration, and evaluation of the IMIS-DBN model, a framework designed for enhanced ransomware detection. By integrating IMIS for dynamic feature selection with the deep learning capabilities of a DBN, the model facilitates the identification of both established and emergent attack patterns. IMIS acts as the initial layer, adaptively selecting and updating relevant features from ransomware data streams. The DBN component was configured with 100 training epochs, a batch size of 64, L2 regularization of 0.0002, momentum of 0.7, and a learning rate of 0.05. Initially, IMIS employs mutual information scores to select features, reducing data dimensionality while preserving crucial information. Subsequently, as new data is introduced, IMIS dynamically updates the feature set by recalculating mutual information and adjusting feature relevance. As depicted in Figure 10, the model comprises two primary modules: the IMIS feature selection module and the DBN detection classifier. IMIS processes the dataset through a series of steps, including initial relevance estimation, weight adjustment, relevance score updates, and top-n feature selection. The resulting feature subset is then used as input for training the DBN classifier.

The DBN architecture is composed of stacked Restricted Boltzmann Machines (RBMs), each comprising visible and hidden units. This model incorporates a hierarchical structure of five hidden layers, with each successive layer reducing the unit count by 30% relative to its predecessor. The initial layer receives input processed by the IMIS module, and subsequent layers progressively extract higher-level data representations. Each RBM undergoes unsupervised pre-training, commencing with the bottom layer, wherein they learn to reconstruct inputs and model the underlying data distributions. This pre-training phase initializes network weights, facilitating subsequent supervised fine-tuning. Following pre-training, the DBN is subjected to supervised fine-tuning using labeled data, aiming to minimize classification errors and optimize detection accuracy. The efficacy of the trained IMIS-DBN model is evaluated using an independent test dataset, with performance metrics including accuracy, precision, recall, and F1-score used to assess its ability to discriminate between benign and malicious activities.

4.3.3. Performance Evaluation of IMIS

This section compares our IMIS technique against established feature selection methods like RCGU, EMRMR, MIFS, and JMI for ransomware detection [71]. We used Python 3.11.7 with libraries such as Scikit-learn, Pandas, NumPy, SKFeature, and TensorFlow, running experiments on a Windows 10 workstation. We evaluated each technique’s accuracy, FPR, DR, and F1-score across various feature set sizes. Unlike static methods, IMIS dynamically updates feature relevance, giving it a key advantage as ransomware evolves.

The evaluation rigorously assessed IMIS’s effectiveness in dynamic ransomware detection, benchmarking its real-time feature relevance against the static nature of the other techniques. Our setup allowed a direct comparison of adaptability and performance. IMIS’s unique incremental approach, continuously refining feature relevance with new data, was the central focus of this investigation.

As shown in Figure 11, IMIS consistently outperformed other methods across feature counts from 5 to 50. It achieved a strong initial accuracy of 0.949 with just five features, demonstrating its quick analytical capability. IMIS peaked at 0.979 accuracy with 25 features, showing significant performance gains in this range. Its sustained advantage, especially over RCGU, stems from its dynamic and adaptive re-evaluation of feature importance, keeping it aligned with evolving ransomware. This adaptability, a key differentiator, validates IMIS’s potential for real-time, proactive threat detection.

Figure 12 compares FPR for IMIS, RCGU, EMRMR, and MIFS (5–50 features). IMIS consistently shows lower FPR, starting at 0.175 with 5 features and reaching 0.104 at 25. While IMIS and RCGU match at 30 features (0.123), IMIS remains competitive beyond, outperforming at 40 and 45 features and maintaining a lower FPR (0.150) at 50. IMIS’s iterative feature reassessment ensures relevance to evolving ransomware, reducing false positives.

Figure 13 illustrates the DR comparison between IMIS, RCGU, EMRMR, and MIFS. IMIS achieves a high detection rate across all feature set sizes, starting at 0.913 for 5 features and peaking at 0.942 with 25 features, outperforming all other techniques. RCGU follows closely at 0.934, but IMIS maintains a consistent lead at 15, 25, 35, and 40 features. Even at 50 features, IMIS upholds a strong DR of 0.927, while other techniques see a noticeable drop in performance. This improvement comes from IMIS’s ability to balance historical and new data and enhance feature selection based on evolving attack patterns. Unlike traditional MI-based methods that use static feature importance, IMIS updates feature relevance to ensure the detection model is responsive to new and existing ransomware threats.

Figure 14 provides a detailed comparative analysis of the F1-score performance between IMIS and its competing methodologies across a comprehensive range of feature set sizes (5–50). The data unequivocally establishes IMIS’s consistent dominance, characterized by superior F1-score values throughout the entire spectrum. Commencing with a robust F1-score of 0.935 at 5 features and culminating in a peak performance of 0.950 at 20 features, IMIS demonstrates its exceptional capacity for balanced precision and recall. While a marginal convergence in F1-scores is observed as the feature count increases, IMIS steadfastly maintains its preeminence across all feature dimensions. This sustained performance superiority underscores IMIS’s adeptness at harmonizing accuracy and recall, thereby ensuring both high detection rates and minimal false positive occurrences. In stark contrast to conventional mutual information (MI)-based feature selection techniques, which operate within a static evaluation paradigm, IMIS employs a dynamic and adaptive reassessment of feature relevance based on the influx of new data. This dynamic adaptation guarantees the selection of only the most pertinent features, resulting in enhanced classification performance against the ever-evolving landscape of ransomware threats.

As shown in Figure 11, Figure 12, Figure 13 and Figure 14, F1-score consistently served as a more balanced and robust metric across varying feature sizes. This was particularly important given the dataset’s class imbalance. That ensures a fair assessment of model performance on both benign and ransomware samples.

Table 5. A comparative performance evaluation of the proposed IMIS and related techniques.

Technique	Per-Call Execution Time (s)	Per-Call Execution Time (min)	Train Time (min)
IMIS	0.01 s	3.5 m	19 m
RCGU	0.054 s	10.8 m	33 m
EMRMR	0.063 s	12.6 m	37 m
MIFS	0.03 s	6.0 m	28 m
JMI	0.07 s	4.0 m	24 m

presents the computational efficiency of IMIS compared to other techniques based on Python profiling. IMIS demonstrates significant performance improvements in terms of execution time, total runtime, and training duration.

IMIS exhibits the shortest per-call execution time (0.01 s) and lowest total runtime (3.5 min), demonstrating a substantial performance advantage over other methods. This reduction in computational overhead is attributed to IMIS’s selective feature update mechanism, which circumvents superfluous re-computation by dynamically reassessing only pertinent features upon the arrival of new data.

Table 6. The 10 most prevalent API calls utilized by ransomware, categorized by functionality.

Type	Features	Rank
Crypto APIs	CryptEncrypt	1
	CryptGenKey	3
	CryptDestroyKey	6
	BCryptGenRandom	9
File access APIs	CreateFile	2
	FindFirstFileEXA	5
	FindNextFileA	8
	DeleteFile	10
Network APIs	WinHttpConnect	4
	WinHttpOpenRequest	7

presents the top 10 API call features identified by IMIS, categorized by functionality. These features correspond to critical ransomware behaviors, including encryption, file manipulation, and network communication. These findings underscore IMIS’s capability to prioritize ransomware-specific features, thereby enhancing detection accuracy by focusing on API calls most frequently exploited in ransomware attacks.

4.4. Phase 3

In the age of digital technology, devices are commonly interconnected, which facilitates data exchange and simplifies communication for users. This connection introduces various security risks to user and business data [60]. Among these risks, malware attacks, particularly ransomware, pose significant threats by encrypting user data, and often a ransom is demanded for decryption [11,73]. However, paying the ransom does not guarantee data recovery [74]. Detecting ransomware early is crucial to mitigating its impact. Prior studies [10,16,75] emphasize the importance of early detection using machine learning to identify ransomware before encryption.

Deep learning has emerged as a powerful tool in malware detection, which improves the traditional signature-based methods. Studies highlight the advantages of deep learning in IoT security [76], Android malware defense [77], and dynamic malware analysis [78,79]. The adaptability of deep learning allows for quick analysis of attack patterns and improves detection and prevention [80,81,82,83]. However, overfitting remains a challenge, and early stopping mechanisms can help mitigate it by pausing training before performance goes down [84].

Existing early stopping techniques primarily rely on validation accuracy [85,86,87], but static criteria may lead to underfitting or missed opportunities for better generalization [88]. Dynamic stopping approaches address these limitations by adapting to the training differences [89]. However, current methods often fail to integrate uncertainty and variability, which can lead to the most suitable stopping decisions. Due to the lack of ransomware datasets, our framework contains regularization techniques, data augmentation, and active learning to enhance model performance.

4.4.1. UA-DES: Mathematical Steps and Algorithm

To optimize the accuracy and efficiency of DBN training for ransomware detection, we developed an Uncertainty-Aware Dynamic Ensemble Selection (UA-DES) approach. This methodology employs Bayesian principles to model performance uncertainty, facilitating more informed decision-making throughout the training process. Specifically, we utilize dropout as a Bayesian approximation to quantify predictive uncertainty and implement Temperature Scaling to refine probability estimates. Furthermore, an active learning framework is incorporated to prioritize the selection of high-information samples, thereby maximizing training efficiency.

The central innovation of Uncertainty-Aware Dynamic Ensemble Selection (UA-DES) resides in its dynamic stopping criterion. This criterion incorporates not only performance gains but also the model’s predictive uncertainty and predefined quality benchmarks. By integrating these multifaceted factors, we ensure robust and reliable model generalization, culminating in enhanced accuracy and efficacy in ransomware detection. Figure 15 provides a schematic representation of the UA-DES architecture.

To substantiate the implementation of UA-DES in DBN training, we have developed a rigorous mathematical formulation. This formulation precisely defines the constituent elements and procedural steps, providing a transparent and unambiguous framework for practical application.

The UA-DES technique is designed to enhance the training process of a DBN-based ransomware detection model. This framework consists of five key steps:

• Step 1. Bayesian Performance Modeling

• Performance Evaluation Based on Posterior Distribution:

Let $y$ be the actual label and $\widehat{y}$ be the label predicted by the DBN, which can be ransomware or benign. The performance metric $P$ (e.g., accuracy, precision) can be modeled as

$$P(\theta \mid D)={\displaystyle \frac{P\left(D∣\theta \right)P\left(\theta \right)}{P\left(D\right)}}$$

(20)

Within the DBN framework, θ denotes the model parameters, D represents the observed data, P (D\ θ) signifies the likelihood of the data given the parameters (model evidence), P(θ) represents the prior distribution of the parameters, and P(D) denotes the marginal likelihood of the data. Guided by empirical best practices and prior research, the DBN was configured for optimal performance with the following parameters: a batch size of 64, L2 regularization of 0.0002, momentum of 0.7, dropout rate of 0.2, and a learning rate of 0.05. To ensure the network focused on the most informative signals, API-based features were preselected using mutual information feature selection.

• Uncertainty Estimation:

Uncertainty in the performance metric P can be quantified by the variance of its posterior distribution:

$${\sigma }_P^2=Var\left[P\right(\theta \mid D\left)\right]$$

(21)

• Step 2. Bayesian Approximation via Dropout: Stochastic Inference in DBNs

Bayesian inference within the DBN was approximated using stochastic forward passes with dropout. A dropout rate of 0.2 was implemented during training to simulate sampling from the posterior distribution and mitigate overfitting. By randomly deactivating neurons, dropout forced the network to learn more generalizable features. This stochasticity was limited to training; dropout was disabled for deterministic validation and testing. Training performance was used to optimize network weights and biases, while validation and testing metrics provided a comprehensive evaluation of predictive performance.

For each pass $n$, obtain prediction $y^n$, then calculate the predictive uncertainty as

$$Uncertainty={\displaystyle \frac{1}{N}}\sum _{n-1}^N{\left({\widehat{y}}_n-\overline{\widehat{y}}\right)}^2$$

(22)

where $\overline{\widehat{y}}$ is the mean prediction over the $N$ passes.

• Step 3. Calibration for Reliable Probability Estimates

After obtaining the raw model output probabilities $p$, apply a calibration function $f$ to adjust these probabilities:

$$calibrated=f\left(p\right)$$

(23)

Temperature Scaling (TS) is a post-processing calibration technique that refines SoftMax outputs to better reflect true probabilities, without altering the model’s architecture or prediction order. By adjusting output probabilities, TS improves the model’s ability to distinguish threats, reducing false alarms. Calibrating the temperature parameter on a validation set aligns predictions with actual results, enhancing generalization and reliability, thus ensuring UA-DES-DBN’s effectiveness and maintaining predictive performance despite early stopping. The TS calibration function operates as follows. Let $z$ represent the logits, which are the raw outputs from the DBN model before applying the SoftMax activation. In other words, logits are the values produced by the neural network before conversion into probabilities. The standard SoftMax function, which transforms these logits into probabilities, is defined as

$$P\left(y=i|x\right)={\displaystyle \frac{e^{z_i}}{{\sum }_j{e}^{z_j}}}$$

(24)

where $y$ is the predicted class, $x$ is the input, $z_i$ is the logit for class $i$ and the denominator sums over all possible classes. Temperature Scaling introduces a temperature parameter $T$ to modify the SoftMax function. The adjusted SoftMax function becomes

$$P_T\left(y=i|x\right)={\displaystyle \frac{e^{z_i/T}}{{\sum }_j{e}^{z_j/T}}}$$

(25)

Here, $T$ is a scalar that adjusts the “sharpness” of the probability distribution. A higher $T$ makes the distribution softer (more uniform), while a lower $T$ makes it sharper. Calibration seeks to adjust the model’s predictive probabilities to match the true likelihood of outcomes by finding the optimal temperature. This process involves minimizing a loss function, with the Negative Log-Likelihood (NLL) being the commonly used metric, on a validation dataset:

$$NLL={\sum }_{\left(x,y\right)\ni Validation Set}\log P_T\left(y\right|x)$$

(26)

The objective is to find

$$T^{*}=\arg \underset{T}{\min } NLL\left(T\right)$$

(27)

where $T^{*}$ is the temperature that minimizes the $NLL$ over the validation data. Once $T^{*}$ is determined, the model’s final output probabilities are adjusted using this optimal temperature:

$$P_{T^{*}}\left(y=i|x\right)={\displaystyle \frac{e^{z_i/T^{*}}}{{\sum }_j{e}^{z_j/T^{*}}}}$$

(28)

This calibrated probability distribution is applied to all future predictions, helping align the model’s confidence with real outcomes. This improves reliability and enhances decision-making accuracy in ransomware detection.

The “output probabilities” in our DBN-based ransomware detection model represent the model’s predicted likelihoods of a network event belonging to either the ransomware or benign class. This results in a two-element probability vector, where the probabilities sum to one. This practice of generating class probability vectors, even in binary classification, is a common and flexible design choice in machine learning, facilitating future multi-class extensions.

• Step 4. Active Learning Framework

The selection of training samples is driven by maximizing expected information gain, a metric that quantifies the reduction in model uncertainty. Specifically, samples are chosen whose incorporation into the training set would yield the highest post-selection entropy, as evaluated by the current model state. This targeted approach prioritizes samples residing in regions of high uncertainty, thereby enabling the model to efficiently refine its predictive accuracy and robustness. By focusing on these information-rich, high-entropy samples, the model strategically addresses its knowledge gaps, leading to a more effective learning process. The information gain, $IG\left(x\right)$ for a sample $x$ can be defined as

$$IG\left(x\right)=H\left(\widehat{y}\right)-E_{p\left(\theta ∣D\right)}\left[H\right(\widehat{y}\mid x,\theta \left)\right]$$

(29)

where $H$ represents entropy, a measure of uncertainty, and $E_{p\left(\theta ∣D\right)}\left[H\left(\widehat{y}∣x,\theta \right)\right]$ denotes the expected entropy over the posterior distribution of the model parameters. To efficiently train our framework, particularly with large, unlabeled datasets, we employ an uncertainty sampling strategy. This approach selectively labels data samples where the model exhibits the highest prediction uncertainty, minimizing the need for exhaustive labeling. We quantify this uncertainty using an entropy-based measure, which guides the selection of the most informative samples.

$$H\left(y|x\right)=-{\sum }_{p}P\left(y=i|x\right)logP\left(y=i|x\right)$$

(30)

The term $H(y\mid x)$ represents the entropy of the predicted class probability distribution for a given data sample x. This entropy is calculated based on $P(y=i\mid x)$, which is the model’s estimated probability that x belongs to class i. Consequently, higher entropy values indicate that the model’s prediction is less confident.

• Step 5. Dynamic Stopping Criteria

Training termination is governed by a composite stopping criterion, incorporating an accuracy performance improvement threshold $\mathsf{\Delta }P$, predictive uncertainty ${\sigma }_{\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{d}}^2$, and calibration quality $C$. This multifaceted approach is designed to mitigate overfitting and enhance detection accuracy through early stopping. Given that accuracy is the primary objective, accuracy-related metrics are prioritized. Empirical evaluations have validated the efficacy of this methodology, demonstrating that our model surpasses those presented in related studies. Training ceases when

$$\mathsf{\Delta }P<ϵ\mathrm{and}{\sigma }_p^2<{\sigma }_{threshold}^2\mathrm{and}C>C_{min}$$

(31)

where $ϵ$ is a small positive value indicating the minimum acceptable performance improvement, ${\sigma }_{threshold}^2$ is the maximum acceptable uncertainty level, and $C_{min}$ is the minimum calibration quality threshold. The use of the AND operator in our stopping criteria is primarily intended to optimize protection against overfitting, a prominent concern in ransomware detection. This technique guarantees that the model continues training until it meets all conditions, avoiding early termination based on a single criterion, which could undermine the model’s ability to generalize. This is especially important due to the nature of ransomware early detection, which should happen as early as possible before the encryption phase takes place.

The mathematical formalization presented in Steps 1–5 establishes a rigorous foundation for the Uncertainty-Aware Deep Ensemble System (UA-DES) technique. This framework enables systematic early stopping in DBN training for ransomware detection by quantitatively evaluating performance, uncertainty, calibration, and information gain. Ultimately, this optimization enhances model generalization and accuracy. Algorithm 2 details the pseudocode for the UA-DES technique’s integration with the DBN.

The UA-DES algorithm, integrated into DBN training for ransomware detection, aims to mitigate overfitting and enhance model generalization. It initiates DBN training with predefined parameters and proceeds through epochs, employing early stopping criteria based on performance improvement, uncertainty thresholds derived from dropout simulations, and calibration quality measured by metrics like Expected Calibration Error (ECE). An active learning phase strategically selects high-value samples to optimize training efficiency. Training terminates when performance plateaus, as determined by a dynamic stopping criterion. The resulting trained DBN model is then saved for subsequent ransomware detection tasks, striving for a robust and accurate model capable of consistently identifying ransomware attacks.

Algorithm 2. UA-DES and DBN Integration Pseudocode [70].
	Initialize DBN model
	Initialize training parameters (epochs, learning rate, etc.)
	Initialize early stopping params (thrshld for perf improve, uncertainty thrshld, calibr qual. thrshld)
1.	For each epoch in training:
2.	Train DBN on training dataset
3.	Evaluate DBN on validation dataset
4.	Calculate performance metric (e.g., accuracy, F1-score)
5.	#   Dropout as Bayesian approximation for uncertainty estimation
6.	Perform dropout simulations on validation dataset
7.	Calculate mean and standard deviation of performance metric across simulations
8.	#   Calibration of probability estimates
9.	Calibrate model outputs on validation dataset
10.	Calculate calibration quality (e.g., Expected Calibration Error)
11.	# Active learning for data efficiency
12.	If epoch % active_learning_interval == 0:
13.	Identify and prioritize uncertain samples in training dataset
14.	Retrain DBN model including prioritized samples
15.	#   Dynamic stopping criterion VALIDATION based on perf improvement, uncertainty, and calibration
16.	If (performance improvement < performance improvement threshold) And
17.	(standard deviation of performance metric < uncertainty threshold) And
18.	(calibration quality > calibration quality threshold) Then
19.	Stop the training

4.4.2. Improved Detection Capability

The UA-DES mechanism refines DBN training for ransomware detection, enhancing precision and reliability. The process involves initializing the DBN with specific parameters and setting early stopping criteria based on performance, uncertainty, and calibration quality, with epoch-based training and validation set evaluation guiding stopping decisions. UA-DES incorporates uncertainty estimation via dropout simulations and calibrates predictions to ensure reliability. Active learning prioritizes informative samples, promoting training efficiency, while a dynamic stopping criterion mitigates overfitting, resulting in a robust DBN model.

For UA-DES-DBN model validation, we assembled a comprehensive dataset comprising 8152 crypto-ransomware samples from virusshare.com, representing diverse families like Cerber, TeslaCrypt, CryptoWall, Petya, and WannaCry, alongside 1000 benign programs from informer.com. Both ransomware and benign samples underwent analysis in a sandbox environment. Python libraries, including Sklearn, Pandas, NumPy, and SkFeature, facilitated data analysis and evaluation.

The API call features used as model input, a sample of which is shown in

Table 7. Sample of API call features fed into the UA-DES-DBN model [70].

Type	Features
Crypto APIs	CryptEncrypt|CryptGenKey|CryptDestroyKey|CryptGenRandom
File Access APIs	CreateFile|FindFirstFileEXA|FindNextFileA|DeleteFile
Network APIs	WinHttpConnect|WinHttpOpenRequest

, are highly indicative of ransomware behavior. Notably, cryptographic APIs like CryptEncrypt and CryptGenKey play a crucial role in file encryption. File access APIs such as CreateFile and DeleteFile indicate unauthorized file manipulation. Network APIs like WinHttpConnect and WinHttpOpenRequest suggest potential data exfiltration or command-and-control communication. These features collectively provide a robust foundation for the model’s detection capabilities.

4.4.3. Overall Results and Discussion

Figure 16 presents a comparative analysis of accuracy across varying input layer sizes, highlighting the superior performance of the UA-DES-DBN model compared to VGG16-PSO [90], DBN-IDS [91], and DBN [92]. Notably, UA-DES-DBN demonstrates a significant improvement in accuracy as the input size increases from 5 to 20, peaking at 0.986. While a slight decline is observed at larger input sizes, the model maintains robust performance. In contrast, CNN-MD exhibits lower accuracy rates, and DBN-IDS and DBN models show greater variability. At an input size of 25, UA-DES-DBN achieves 0.982, consistently outperforming the other models.

These results underscore the effectiveness of the UA-DES-DBN model, particularly in mid-range input sizes. The model’s ability to maintain high accuracy, even with slight fluctuations, demonstrates its resilience and adaptability to complex data patterns. The integration of uncertainty estimation and dynamic stopping criteria effectively mitigates overfitting, contributing to the model’s robust performance. This validation confirms the UA-DES-DBN model’s potential for reliable and accurate ransomware detection in real-world scenarios.

Despite the strong performance of UA-DES-DBN, the difference from other models is sometimes narrow. Advanced ransomware evasion techniques like polymorphic and metamorphic behaviors cover malicious intent that make early detection harder. The model focuses on the pre-encryption phase, where attack patterns are scarce and less distinct. Even slight accuracy improvements in this stage are significant because they enhance the proactive mitigation of ransomware attacks before encryption occurs. The ability to identify small anomalies early provides an advantage in cybersecurity defense.

As Figure 17 shows, UA-DES-DBN exhibits lower false positive rates (FPRs) than CNN-MD, DBN-IDS, and standard DBN models across varying input layer sizes (20–50). Notably, at an input size of 25, UA-DES-DBN achieves an FPR of 0.104, demonstrating superior performance in complex data processing. This reduction in FPR, particularly at larger input sizes, is attributed to UA-DES’s dynamic training adjustments based on uncertainty, effectively preventing overfitting and refining decision thresholds.

Figure 18 illustrates UA-DES-DBN’s consistently high DR across various input sizes, which often surpasses or closely matches other models. At an input size of 25, it achieves a DR of 0.956, emphasizing its ability to accurately identify ransomware. The UA-DES mechanism enhances detection by adjusting training based on uncertainty. This ensures a perfect balance between underfitting and overfitting. Moreover, its ability to leverage larger input sizes for improved detection is particularly useful for evolving ransomware threats.

The F-score comparison in Figure 19 further validates UA-DES-DBN’s effectiveness in balancing precision and recall for ransomware detection. At an input size of 30, it achieves an F-score of 0.962 that outperforms other models. The UA-DES mechanism’s dynamic training adjustments prevent overfitting and improve generalization. So, it ensures consistently high F-scores across different input sizes. These findings prove the importance of advanced early stopping mechanisms in developing robust and precise cybersecurity models.

Figure 20 presents the false negative rate (FNR) comparison, where UA-DES-DBN consistently shows lower FNRs and a notable reduction at an input size of 25. While the model performs well across different input sizes, slight changes suggest sensitivity to feature dimensionality. The rise in the FNR at specific feature sizes indicates potential overfitting that needs further investigation.

The UA-DES-DBN model, which significantly improves DBN training for ransomware detection by enhancing both precision and reliability, demonstrates strong specificity. As shown in Figure 21, it generally outperforms or matches competing models, achieving a peak specificity of 0.896 at an input size of 25. This ability to accurately distinguish ransomware from benign activities, thereby reducing false positives, is crucial in cybersecurity applications. The UA-DES mechanism initializes the DBN with optimized parameters and employs dynamic early stopping based on performance, uncertainty, and calibration quality, with uncertainty estimated via dropout simulations and predictions calibrated for enhanced reliability. Active learning prioritizes informative samples, maximizing training efficiency, while the dynamic stopping criterion effectively mitigates overfitting. While specificity remains strong, minor variations beyond size 25 suggest potential overfitting, highlighting areas for further model improvement.

Table 8. A summary of key experimental findings, detailing the models, metrics, and conclusions.

Fig/Tab	Phase	Comparison Details	Metrics	Value Proposition
Figure 4, Figure 5, Figure 6 and Figure 7	Phase 1	Models: BGM-GAN (Proposed) vs. standard GAN, CNN, DBN.	Accuracy	BGM-GAN consistently outperforms the other models across all training epochs, achieving a peak accuracy of 0.995 with 50 features at 210 epochs.
		Variable: Performance across 5–50 features at 30, 90, 150, and 210 training epochs.
Figure 8	Phase 1	Models: BGM-GAN (Proposed) vs. standard GAN, CNN, DBN.	Recall (TPR)	BGM-GAN demonstrates a superior recall rate compared to existing techniques across varying feature counts and epochs.
		Variable: Averaged performance across epochs.
Figure 9	Phase 1	Models: BGM-GAN (Proposed) vs. standard GAN, CNN, DBN.	False Positive Rate (FPR)	BGM-GAN achieves a lower false positive rate than comparison models, indicating higher precision.
		Variable: Averaged performance across epochs.
Figure 11, Figure 12, Figure 13 and Figure 14	Phase 2	Models: IMIS (Proposed) vs. RCGU, EMRMR, MIFS, JMI.	Accuracy, FPR, DR, F1-Score	IMIS consistently outperforms other feature selection methods, achieving a peak accuracy of 0.979 and the best balance of precision and recall (F1-score).
		Variable: Performance across 5–50 features.
Table 6	Phase 2	Models: IMIS (Proposed) Versus RCGU, EMRMR, MIFS, JMI.	Computational Efficiency	IMIS is the most computationally efficient method, with the shortest execution time and lowest total runtime.
Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21	Phase 3	Models: UA-DES-DBN (Proposed) vs. VGG16-PSO, DBN-IDS, DBN.	Accuracy, FPR, DR, F1-Score, FNR, Specificity	UA-DES-DBN model achieves superior performance, with a peak accuracy of 0.986, a lower FPR, and a higher F1-score than comparison models.
		Variable: Performance across 5–50 input features.

gives an overall summary of the experimental results.

Despite the widespread adoption of DBNs and other deep learning algorithms for ransomware detection, certain limitations warrant further investigation. A primary area for future research is model interpretability. Enhancing interpretability would enable researchers to better pinpoint detection failures and identify critical model components for targeted improvement.

Table 9. Summary of research motivation, framework components, contributions, and limitations.

Motivation	Component	Contribution	Limitation
Lack of data during pre-encryption phase	BGM-GAN	Generates synthetic ransomware attack patterns to augment early-phase data.	Effectiveness depends on the quality and diversity of initial real samples.
Static feature selection struggles to adapt to evolving threats	IMIS	Dynamically selects and updates relevant features in real-time.	May struggle with noisy or redundant features.
Risk of overfitting in deep detection models	DBN + UA-DES	Learns hierarchical patterns and employs uncertainty-aware stopping to optimize training.	Requires tuning and high computational resources.
Absence of a unified, adaptive detection approach	Integrated 3-phase framework	Combines augmentation, adaptive learning, and uncertainty-aware training for enhanced early detection.	Currently evaluated on Windows PE files; may need generalization for other environments.

provides a concise summary of the motivations, framework components, and corresponding limitations of this research.

5. Conclusions

This work explores early and accurate ransomware detection, which is crucial for mitigating its impact on individuals and organizations. Traditional detection methods often fail due to attackers’ avoidance techniques and high false positive rates. To address these challenges, this framework uses a three-phase methodology: data augmentation, feature selection, and model training and validation.

During the first phase, the framework develops a dynamic ransomware early detection model capable of predicting behavioral changes. By using GANs, the GBM-GAN model generates realistic attack patterns that address the lack of diverse ransomware data and enhance the capabilities of detection. Also, by leveraging the adversarial nature of GANs, the model adapts to polymorphic and metamorphic ransomware features, which improve early detection. The experimental results show a peak detection accuracy of 98%, a recall rate of 96%, and a false positive rate of 14%. This framework supports cybersecurity by predicting ransomware threats and mitigating them before causing damage.

During the second phase, the framework employs incremental learning to update relevant features as new data arrives dynamically. To address ransomware’s evolving nature, this work introduces the IMIS technique, which selects non-duplicative features early in an attack. This reduces computational load and improves adaptability. The results show that the IMIS-DBN model surpasses traditional methods by achieving 97.9% accuracy with 25 features and a low false positive rate of 0.104. Therefore, this highlights the importance of adaptive feature selection in ransomware detection. Future research will explore IMIS’s application in real-time and wider malware detection.

During the third phase, the proposed model employs a dynamic training regimen to mitigate overfitting and underfitting, thereby enhancing detection accuracy and reducing false positives. This phase introduces a UA-DES technique, which leverages uncertainty estimation and dropout simulations to optimize training. By integrating uncertainty and calibration quality metrics, UA-DES ensures heightened accuracy and reliability in ransomware detection.

Empirical results demonstrate that, altogether, the UA-DES-DBN model achieves a 98.6% accuracy, surpassing traditional models. Consequently, this research advances DBN-based detection, and future work will explore alternative deep learning architectures and refine uncertainty calibration to address evolving threat landscapes. While the proposed framework demonstrates strong performance, several limitations should be acknowledged. First, the dataset is limited to portable executable files on Windows, potentially affecting generalizability to other platforms such as Linux or MacOS. Second, although the GAN enhances data diversity, its output quality still depends on the initial sample distribution, which could limit detection capability in unseen ransomware variants. Third, while IMIS enables real-time feature adaptation, it may struggle in the presence of persistent noisy or redundant features. Lastly, the DBN model requires extensive training resources and tuning, which could challenge deployment in resource-constrained environments (e.g., IoT devices).

6. Future Research

Future research will focus on enhancing this framework to bolster ransomware detection efficacy and operational efficiency. Three key areas demand rigorous attention: (i) mitigating the risk of misleading samples generated by GANs, (ii) developing adaptive strategies for diverse attack behaviors in real-time, and (iii) optimizing the computational efficiency of DBNs for dynamic threat environments. The subsequent subsections explore these critical areas, outlining challenges and potential solutions.

6.1. Risk of Generating Misleading Samples

Using GANs for ransomware detection can accidentally create adversarial examples that mislead the model. While simulating real ransomware, these unreal patterns may introduce false positives or negatives, which reduce detection reliability. Since ransomware develops continuously, GAN-generated data that is incorrectly managed can lead to not fitting the model with real-world threats. Careful supervision is essential to ensure that augmented datasets accurately reflect diverse and complex ransomware behaviors that preserve detection effectiveness.

6.2. Empirical Diversity of Attack Behavior

Incremental feature selection presents challenges in real-time data processing. Unlike offline training, where preprocessing tasks like normalization and feature extraction are manageable, online approaches must handle continuous data streams efficiently. This requires rapid integration while minimizing computational overhead. The flow of new data may present noise or irrelevant features, which complicate the learning process. Also, effective strategies are needed to maintain model accuracy and adaptability without consuming too many resources.

6.3. Computational Efficiency

DBNs excel in complex feature learning, but they demand significant computational resources for both training and retraining. While continuous learning is vital for adapting to new ransomware threats, the high processing costs and slow inference times pose challenges for real-time detection. These inefficiencies may delay responses to emerging threats, leaving systems vulnerable. Optimizing DBN training and inference speeds is critical for maintaining effective ransomware detection in dynamic threat environments.

The proposed framework offers a promising advancement for cybersecurity, particularly for vendors, threat intelligence platforms, and endpoint protection services aiming to enhance their early detection capabilities. Its modular design facilitates seamless integration into current sandbox environments and real-time monitoring systems. Unlike existing industry solutions that depend on static signatures or delayed anomaly detection, our approach provides proactive defense by capturing pre-encryption behaviors and adapting to evolving attack patterns. This makes our framework a complementary and forward-looking addition, significantly strengthening existing detection pipelines.