AI-Driven Phishing Detection: Enhancing Cybersecurity with Reinforcement Learning

Abstract

Phishing remains a persistent cybersecurity threat, often bypassing traditional detection methods due to evolving attack techniques. This study presents a Reinforcement Learning (RL)-based phishing detection framework, leveraging a Deep Q-Network (DQN) to enhance detection accuracy, reduce false positives, and improve classification performance. The model was trained and evaluated using a real-world dataset comprising 5000 emails (2500 phishing and 2500 benign) and externally validated against a synthetic phishing dataset of 1000 samples simulating unseen attacks. It achieved a 95% accuracy, 96% precision, 94% recall, and a 2% false positive rate on the real-world dataset and a 93% accuracy, 94% precision, and a 4% false positive rate on the synthetic dataset. Area Under the Curve (AUC) analysis yielded a score of 0.92, confirming excellent classification separability and alignment with the model’s high accuracy and low false positive rate. This work contributes to scalable, real-world phishing defense by addressing the limitations of static detection systems and improving detection reliability.

1. Introduction

Phishing has established itself as one of the most pervasive and damaging forms of cyberattacks, responsible for a large percentage of data breaches, financial theft, and personal information compromises. Phishing attacks typically involve deceptive tactics such as fraudulent emails, websites, or messages designed to lure victims into revealing sensitive information like usernames, passwords, or financial data. The frequency and complexity of phishing attacks have escalated in recent years, challenging organizations to adopt more robust defense mechanisms. According to a 2023 report from the Anti-Phishing Working Group (APWG), phishing attacks rose by nearly 40% compared to the previous year, marking it as one of the most significant threats in the cybersecurity landscape [1]. Traditional methods of phishing detection, such as RSA-based encryption methods [2], have been largely ineffective in addressing this evolving threat landscape. These approaches include blacklist-based detection, which flags known phishing URLs, and rule-based systems that rely on predefined patterns and heuristics. However, these solutions fall short when it comes to detecting novel and sophisticated phishing attacks that evade static defenses [3]. For example, attackers have developed advanced spear-phishing techniques, where emails are carefully crafted to target specific individuals or organizations, making it much harder to detect using conventional methods [4].

Furthermore, the emergence of phishing-as-a-service (PaaS), in which cybercriminals sell ready-made phishing kits to others, has significantly lowered the barrier to entry, enabling even those with little technical expertise to launch phishing attacks [5]. Recent research has shown that ML offers a promising avenue for enhancing phishing detection. Unlike traditional rule-based systems, ML models learn from historical data, enabling them to identify patterns in phishing attacks [6].

Supervised learning algorithms, such as Support Vector Machines (SVMs), decision trees, and neural networks, have demonstrated success in phishing detection by analyzing specific features of phishing emails and websites. These models have been effective in certain scenarios but often require large amounts of labeled data to function optimally and struggle to adapt to new or unseen phishing tactics [7,8]. Moreover, machine learning (ML) models can generate high false positive rates, where legitimate emails or websites are incorrectly flagged as phishing attempts, leading to reduced trust in these systems [9]. Recent advancements in DL and RL aim to address these challenges by improving adaptability and reducing false positives in phishing detection systems [10].

This research focuses on leveraging RL as an advanced approach to optimize phishing detection systems. RL, a subset of artificial intelligence, offers a dynamic learning framework that allows systems to adapt over time based on feedback from their environment. Unlike supervised learning, which depends on labeled datasets, RL agents learn through trial and error by receiving rewards for correct decisions and penalties for mistakes. This makes RL particularly well-suited for dynamic environments like cybersecurity, where new threats are constantly emerging [11]. The idea of applying RL to phishing detection is relatively new, but promising studies have demonstrated its potential to improve both the accuracy of phishing detection systems and their ability to generalize new attack patterns. A study by [12] explored the use of Q-learning in network intrusion detection systems and found that RL-based models were able to adapt more effectively than traditional ML models in identifying anomalous behavior in network traffic [13,14].

The objective of this research was to develop an AI-driven threat-hunting system for phishing detection that is optimized using RL. Our model aims to address several key challenges: increasing detection accuracy, reducing the rate of false positives, and adapting to emerging phishing threats. This is particularly important given the growing sophistication of phishing techniques, including the use of social engineering and advanced obfuscation tactics [15,16]. To validate the effectiveness of the proposed framework, we compare its performance against state-of-the-art phishing detection models, including machine learning-based approaches. The methodology uses an RL-agent that is trained to identify phishing emails and websites by interacting with a simulated environment. Through iterative learning, the agent refines its detection strategies based on feedback, continuously improving its ability to distinguish between legitimate and phishing threats. The following list outlines our key contributions:

• Introduction of a RL-based phishing detection framework: The paper proposes a new phishing detection framework leveraging RL, specifically using DQN architecture. This approach enables dynamic learning from real-time interactions, which contrasts with static, traditional ML models.
• Enhanced adaptability to emerging phishing threats: Unlike traditional ML models that require frequent retraining with new data, the proposed RL-based model adapts continuously through trial-and-error learning. This makes it highly effective against evolving phishing techniques, including spear-phishing and PaaS attacks.
• Reduction of false positives in phishing detection: One of the key challenges in phishing detection is the high false positive rate. The paper introduces a reward–penalty mechanism within the RL model to penalize false positives, thereby improving decision-making accuracy and reducing false alarms to just 4%, compared to 10–12% in traditional models.
• Benchmarking against developed models: The proposed framework is benchmarked against state-of-the-art models, such as SVMs and Random Forests. The RL-based model achieved a 95% accuracy, outperforming traditional ML models (which achieved an 85–87% accuracy).
• Comprehensive experimental validation: The model was trained on a real-world dataset comprising 5000 emails (2500 phishing and 2500 benign), with 4000 emails used for training and 1000 emails for testing. It achieved a 95% accuracy, 96% precision, and 94% recall on the real-world dataset and maintained a 93% accuracy and 94% precision and a 4% false positive rate during external validation using a synthetic phishing dataset comprising 1000 generated samples simulating unseen attacks.

2. Background and Related Work

Phishing attacks have become one of the most common and damaging cyber threats faced by individuals and organizations worldwide. These attacks exploit social engineering tactics to deceive users into divulging sensitive information, such as passwords, financial data, and personal details. With the rapid growth of online communication, the sophistication of phishing attacks has also evolved, making it increasingly difficult for traditional detection mechanisms to keep pace. As phishing tactics continue to become more complex, cybersecurity researchers and professionals are exploring new technologies, such as AI and ML, to enhance phishing detection capabilities.

In the past decade, several studies have examined various methods for detecting phishing attacks, particularly focusing on machine learning-based solutions. These approaches rely on the analysis of specific features extracted from phishing emails, websites, or URLs. For instance, Rao et al. [17] explored the use of decision trees and support vector machines (SVMs) to classify phishing websites based on specific URL characteristics, such as domain age, HTTPS presence, and URL length. More recently, Aboud et al. [6] reviewed deep learning techniques, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs), and found that these models can significantly enhance phishing detection rates by analyzing email content. These models not only improve detection accuracy but also address the issue of false positives, which has been a major challenge in traditional ML models. While these models have shown promise, they are often constrained by the availability of labeled data and their inability to generalize to new, unseen phishing tactics.

A key limitation of traditional ML approaches in phishing detection lies in their dependence on static datasets. These models are typically trained in historical data and are unable to adapt effectively to emerging phishing strategies. For example, recent studies have explored ML approaches to phishing detection, such as SVMs and decision trees, but they still face challenges in generalizing new types of phishing attacks, especially those that use novel obfuscation techniques like URL shortening services, domain shadowing, and homograph attacks [18,19].

Recent advancements in AI have led to the development of RL techniques, which offer a promising solution to the challenges faced by traditional ML models in phishing detection. RL allows systems to learn dynamically from their environment by receiving feedback in the form of rewards or penalties.

RL achieves superior adaptability compared to traditional ML models due to its autonomous interaction with dynamic environments through Markov Decision Processes (MDPs). Unlike supervised learning, where models are trained on fixed labeled datasets, an RL agent continuously observes the current state, performs an action, and receives real-time feedback in the form of rewards. Through iterative cycles of exploration and exploitation, the agent optimizes its policy by learning from new environmental conditions, without requiring re-training on newly labeled data. The cumulative reward structure ensures that the agent gradually adjusts its decision-making to evolving phishing techniques, improving its detection capability even as attack patterns change. This intrinsic feedback loop makes RL particularly suitable for cybersecurity domains characterized by uncertainty and variability.

This adaptive learning process enables RL-based models to continuously enhance their detection capabilities, making them particularly effective in cybersecurity environments where threats are constantly evolving, as demonstrated by Kamal et al. [20] the potential of RL in real-time phishing detection by training a Q-learning model to identify phishing emails based on their content and metadata. The study showed that RL models could outperform traditional ML models in terms of accuracy and adaptability.

However, while RL has been successfully applied to areas such as network intrusion detection and malware detection, its application to phishing detection is still relatively unexplored; Kheddar et al. [21] proposed an RL-based framework for network intrusion detection that demonstrated superior performance in detecting anomalous network traffic compared to traditional ML models. Their framework relied on DQN architecture, which allowed the system to learn from raw network traffic data and make more informed decisions. This research highlights the potential of RL to enhance cybersecurity solutions, but its focus on network traffic leaves a gap in the literature regarding phishing-specific detection systems. In phishing detection, RL’s ability to adapt to new and unseen phishing attacks offers a significant advantage over static ML models; Guleria et al. [22] explored the use of deep reinforcement learning (DRL) for phishing detection, emphasizing the importance of an adaptive approach in the ever-changing threat landscape. Their research showed that DRL models could achieve higher detection accuracy and lower false positive rates compared to traditional ML approaches. However, their study focused on the theoretical application of DRL, and there remains a need for practical implementation and evaluation in real-world phishing detection systems. Figure 1 below illustrates the fundamental RL framework used in this study, depicting the continuous interaction cycle between the agent and the environment. In this RL model, the agent evaluates states (S_t) and performs actions (A_t) based on its learned policy. The environment then responds by providing a reward (R_t+1) and transitioning to a new state (S_t+1), allowing the agent to iteratively refine its decision-making process through dynamic feedback.

Another critical challenge in phishing detection is the high rate of false positives. Many ML models, including those based on supervised learning, tend to overfit specific patterns in the training data, leading to an increase in false positives when applied to new data; Alsubaei et al. [23] addressed this issue by proposing a hybrid ML model that combined multiple classifiers to reduce false positives in phishing detection. Although their model showed improved performance, it still required extensive manual tuning and lacked the ability to dynamically adapt to evolving phishing techniques.

The proposed work differs from existing research in several ways. First, we focused on developing a practical, AI-driven phishing detection framework that is optimized using RL. Unlike previous studies that have primarily explored ML approaches or theoretical RL models, our research aimed to implement a fully functional RL-based phishing detection system that can be deployed in real-world cybersecurity environments. Second, our work emphasizes the adaptive nature of RL, which allows the system to continuously improve its detection capabilities as new phishing techniques emerge. This stands in contrast to traditional ML models, which require retraining to adapt to new threats.

Additionally, we proposed an approach to reduce false positives in phishing detection by leveraging the reward and penalty structure of RL. In our framework, the RL agent is penalized for generating false positives, which incentivizes the model to improve its decision-making process and achieve a more accurate classification of phishing attempts. This approach addresses one of the key limitations of existing phishing detection systems and provides a more effective solution for real-world applications. Furthermore, our research provides a detailed comparison of the RL-based phishing detection model with state-of-the-art ML techniques. While prior studies have demonstrated the potential of RL in cybersecurity, there is limited research on its comparative effectiveness in phishing detection. By conducting extensive experiments and benchmarking our RL model against traditional ML models such as learning SVMs, decision trees, and Random Forests, we aimed to demonstrate the superiority of RL in both accuracy and adaptability. Recent studies have explored the application of RL for phishing detection. We summarized the key contributions of related work in

Table 1. Summary of phishing and intrusion detection research.

Author(s)	Year	Focus	Approach	Limitations
Routhu & Pais [24]	2019	Detecting phishing websites	ML (Random Forest and PCA-RF)	Dependency on third-party services; limited focus on evolving phishing tactics; potential feature obsolescence over time.
Sahingoz, Buber, & Kugu [25]	2024	Real-time phishing URL detection	DL (CNN, RNN, BRNN, and Attention Networks)	High computational cost; dependency on extensive training data; inability to detect URL hijacking; complexity limits adoption in low-resource environments.
Tanti [26]	2022	Phishing attack and prevention	Comprehensive analysis of phishing and mitigation strategies	Lacks implementation details for modern threats; reliance on generic prevention techniques; no real-world testing or evaluation.
Maci, Santorsola Coscia, & Iannacone [27,28]	2023	Addressing unbalanced web phishing classification	Double DQN (DDQN) with ICMDP	Significant training time; complex implementation; requires computational resources for real-time applications.
Zhang & Maple [29]	2023	Intrusion detection in IoT systems	DR-based IDS	Limited resilience to advanced AML techniques; challenges in scaling to highly diverse IoT networks; significant computational overhead.
Yan, Han, Zhu, Du, Lu, & Liu [30,31]	2024	Phishing behavior detection on blockchains	Adversarial Domain Adaptation (ADA) model	Limited generalizability to non-blockchain phishing; challenges in scaling to new, unseen blockchain platforms.

below, in phishing detection in relation to AI and RL, which highlights the focus, approach, and limitations of each study. This table helps provide a clear overview of the current research landscape and positions our work in relation to existing literature.

The research work in [32] proposed a DRL model to detect phishing websites dynamically. Their work demonstrated improved adaptability but required large training datasets and experienced slow convergence, limiting real-world deployment. Maci et al. [33] and Jabbar et al. [33] applied a Double DDQN to address unbalanced phishing datasets, achieving high detection rates but demanding extensive computational resources during training. In contrast, our work focuses on developing a lightweight DQN-based phishing detection system, optimized for real-time adaptability, reduced false positives, and lower computational requirements, making it more practical for real-world cybersecurity environments. By benchmarking against both traditional ML and advanced DL models, this research highlights the operational advantages of an RL-based approach for phishing threat hunting.

In addition to reviewing the state-of-the-art research in phishing detection, it is critical to analyze the current tools that implement these detection approaches. This comparison serves two key purposes:

• To contextualize the practical limitations of existing solutions in real-world environments;
• To highlight the specific gaps that our RL-based framework aims to address.

Understanding how commercial and open-source tools—such as PhishTank [34], OpenPhish [35], Proofpoint, and Microsoft Defender for Office 365—operate helps establish a benchmark for evaluating the effectiveness of phishing detection systems.

These tools utilize a variety of phishing detection methods, ranging from blacklist-based detection to ML and AI-driven threat hunting. However, despite their widespread adoption, most of these tools exhibit inherent limitations, particularly in terms of adaptability, scalability, and accuracy. For instance, blacklist-based tools like PhishTank rely on static indicators of compromise (IOCs), making them ineffective against zero-day phishing attacks and new obfuscation techniques. Similarly, tools that utilize traditional ML models, such as OpenPhish, often require frequent retraining to maintain accuracy and lack the capability to dynamically learn from new data in real-time, a key advantage of RL-based models. Additionally, some commercial tools, such as Proofpoint, are heavily optimized for specific attack vectors like email phishing, limiting their effectiveness in detecting more sophisticated, multi-vector phishing campaigns.

To provide a clearer comparison,

Table 2. Summary of tools utilizing phishing detection approaches and their limitations in comparison to the proposed method. All tools are presented as per their available versions as of January 2025.

Tool Name	Type	Phishing Detection Approach	Limitation of the Tool
PhishTank [34]	Open Source	Blacklist-based detection	Ineffective against new attacks; lacks adaptive learning.
OpenPhish [35]	Commercial	ML-based phishing detection	Requires frequent retraining; lacks dynamic adaptation.
PhishProtection [36]	Commercial	Supervised ML	High false positives; lacks RL’s reward-based optimization.
MailScanner [37]	Open Source	Rule-based filtering and heuristics	Limited to static rules; cannot evolve with new threats.
SpamAssassin [38]	Open Source	Heuristic spam and phishing detection	Prone to false positives; lacks real-time adaptability.
Proofpoint [39]	Commercial	RL for email phishing	Focused on emails; limited coverage for multi-vector threats.
IRONSCALES [40]	Commercial	AI-based threat hunting and phishing detection	High cost; lacks continuous real-time learning.
PhishER [41]	Commercial	ML and AI-driven analysis	Relies on manual input; lacks full automation like RL.
Sophos Email [42]	Commercial	AI-powered email protection	Limited against advanced phishing; lacks dynamic learning.
Microsoft Defender 365 [43]	Commercial	Integrated AI and machine learning	Microsoft-centric; lacks platform flexibility.
Barracuda Sentinel [44]	Commercial	AI-driven spear-phishing detection	Relies on user training; less effective without it.
Cofense PhishMe [45]	Commercial	User-based phishing simulations and analysis	Focus on training, not real-time detection.
Gophish [46]	Open Source	Open-source phishing simulation platform	Simulation-only; lacks real-time detection capabilities.
Phishing Frenzy [47]	Open Source	Automated phishing campaigns and tracking	Simulation-focused; does not adapt to live phishing threats.

below summarizes these tools, outlining their core detection methods, key limitations, and how they contrast with our proposed RL-based framework. This analysis is not only essential for identifying the shortcomings of existing solutions but also for demonstrating the practical value of our approach. The proposed RL-based phishing detection framework addresses many of these challenges by incorporating a continuous learning process that adapts in real-time, significantly reducing false positives and improving detection accuracy. Unlike static models, our RL approach can evolve autonomously with emerging phishing tactics, ensuring that the system remains resilient against ever-changing threats.

This adaptability, coupled with the ability to minimize false positives through reinforcement mechanisms, sets our model apart from existing tools and technologies. By addressing these critical gaps, our research contributes to the development of more robust, scalable, and adaptive phishing detection systems that can meet the evolving demands of modern cybersecurity environments.

3. Classification of Phishing Detection Strategies

Phishing detection strategies can be broadly classified into four primary categories: rule-based methods, heuristic approaches, ML techniques, and RL-based systems. Each category exhibits unique strengths and limitations depending on the detection context, the sophistication of phishing attacks, and the adaptability required in dynamic cybersecurity environments [48,49]. The paper will provide a brief overview of these methods and emphasize RL, highlighting its strengths in phishing detection. In the next section we will provide the theoretical foundation for work.

3.1. Rule-Based Methods

Rule-based methods [50] rely on predefined if-then conditions or static rules to identify phishing emails or websites. These rules typically include the following:

• Keyword Matching: Identifying suspicious terms (e.g., “urgent” and “verify account”);
• Blacklist Checking: Comparing URLs against known malicious databases [51,52];
• Pattern Recognition: Validating domain signatures, mismatched sender information, and invalid SSL certificates.

An example of a mathematical Representation for a simple rule-based detection function can be represented as shown below.

$$D\left(x\right)=\left\{\begin{array}{l}1 \mathrm{i}\mathrm{f} {\sum }_{i=1}^n{R}_i\left(x\right)\ge T\\ 0\hspace{1em} \mathrm{O}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\right.$$

• D(x) is the detection outcome (1 = phishing; 0 = benign).
• R_i(x) represents the rule score for feature i.
• T is the threshold score to trigger detection.
• n is the total number of rules applied.

Limitations include the following: While effective for known attack patterns, rule-based systems struggle with zero-day attacks and obfuscated phishing attempts. They lack adaptability and require constant manual updates to remain effective.

3.2. Heuristic Approaches

Heuristic methods detect phishing based on behavioral analysis and pattern recognition rather than rigid rules. These approaches consider attributes such as the following:

• URL Heuristics: Anomalous URL lengths, presence of IP addresses instead of domain names, and suspicious subdomains;
• Content Analysis: Abnormal HTML structure, deceptive anchor texts, and inconsistencies in page layout [53];
• Sender Verification: Mismatched “From” and “Reply-To” addresses;
• Example of a Heuristic Scoring Algorithm:

$$H\left(x\right)=\sum _{i=1}^m{w}_i \ast f_i\left(x\right)$$

$$D\left(x\right)=\left\{\begin{array}{c}1\hspace{1em}\mathrm{i}\mathrm{f} H\left(x\right)\ge \theta \hfill \\ 0\hspace{1em}\mathrm{O}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}\right.$$

where the following variables are used:

H(x) is the heuristic score for email/website x.

w_i represents the weight assigned to feature i.

f_i(x) is the heuristic feature function (e.g., URL length and HTML tags).

m is the total number of features analyzed.

θ is the predefined threshold determining classification as phishing (1) or benign (0).

Challenges include the following: while heuristics are more flexible than rule-based systems, they often suffer from high false positive rates because legitimate emails/websites may unintentionally trigger heuristic rules.

3.3. ML Techniques

ML techniques, particularly supervised learning models, have significantly advanced phishing detection. These models learn from labeled datasets to classify emails or URLs as malicious or benign. Common algorithms include the following:

• Decision Trees: Construct a model based on splitting data by key features;
• SVMs: Identify optimal hyperplanes that separate phishing from legitimate samples [32];
• Neural Networks: Model complex feature interactions, particularly effective in detecting sophisticated phishing attacks;

An example of a mathematical framework is shown below.

For supervised learning, the objective is to find a mapping function:

$$:X\to Y \mathrm{where}: Y=f\left(X\right) + ϵ$$

where the following variables are used:

$X$ represents the input features (e.g., URL length and sender info).

$Y$ is the output label (phishing = 1; benign = 0).

ϵ is the error term.

The model is trained to minimize the loss function, commonly the binary cross-entropy loss for classification tasks:

$$L\left(y,\widehat{y}\right)=-\left[y\ast \log \left(\widehat{y}\right)+\left(1-y\right)\ast \log \left(1-\widehat{y}\right)\right]$$

where the following variables are used:

$y$ is the true label.

$\widehat{y}$ is the predicted probability.

Drawbacks include the following:

Machine learning ML ng models using this approach typically require large, labeled datasets for training. They are prone to concept drift and have difficulty adapting to emerging phishing techniques without frequent retraining.

3.4. RL-Based Systems

RL offers a dynamic, adaptive, and self-improving approach to phishing detection. Unlike traditional supervised ML models that rely on large, labeled datasets and static training, RL systems learn from interaction with their environment, making them highly effective in combating evolving phishing tactics [54]. This continuous learning capability aligns perfectly with the dynamic nature of cybersecurity threats, where new attack vectors emerge regularly as shown in Figure 2 below.

RL components include the following:

• Environment (E): The incoming stream of emails, websites, or network data that needs analysis;
• State (s): The current representation of features extracted from emails or websites (e.g., URL length, domain age, and sender reputation);
• Action (a): The classification decision—either phishing (1) or benign (0);
• Reward (R): A feedback mechanism where the agent receives positive rewards for correct classifications and penalties for false positives/negatives.

The goal of the RL agent is to learn an optimal policy (π*), which maps states to actions to maximize cumulative rewards over time.

3.4.1. Mathematical Model: Q-Learning for Phishing Detection

The most widely used RL algorithm in phishing detection is Q-Learning, a value-based method that estimates the optimal action-selection policy using a Q-function.

3.4.2. Q-Function Definition

$$Q\left(s,a\right)=E\left[Rt∣st=s,at=a,\pi \right]$$

where the following variables are used:

Q(s,a) is the expected cumulative reward for taking action a in state s and following policy π thereafter.

Rt represents the reward at time t.

E denotes the expectation operator over possible future rewards.

3.4.3. Q-Learning Update Rule

$$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\alpha \left[r+\gamma \cdot a^{\prime }maxQ\left(s^{\prime },a^{\prime }\right)-Q\left(s,a\right)\right]$$

where the following variables are used:

α = Learning rate (0 < α\alphaα ≤ 1), controlling how much new information overrides old knowledge;

r = Immediate reward after taking action a in state s;

γ = Discount factor (0 ≤ γ < 1), determining the importance of future rewards;

a′maxQ(s′,a′) = The maximum expected future reward for the next state s′.

3.5. Mathematical Advantages of RL over Traditional ML

3.5.1. Dynamic Adaptation Without Retraining

Traditional ML models require periodic retraining to adapt to new phishing patterns, which are resource intensive. In contrast, RL agents continuously learn from new data without retraining.

$$Q\left(s,a\right)=\alpha \left[r+\gamma \cdot a^{\prime }maxQ\left(s^{\prime },a^{\prime }\right)-Q\left(s,a\right)\right]$$

This continuous update mechanism allows the model to adapt to real-time threats, improving detection performance as new attack vectors emerge.

3.5.2. Optimizing Long-Term Reward vs. Immediate Accuracy

While supervised ML models optimize for immediate classification accuracy, RL optimizes the long-term cumulative reward, balancing short-term accuracy with future performance.

$$G_t=\sum _{k=0}^{\infty }{y}^k \ast R_{t+k+1}$$

where the following variables are used:

$G_t$ is the return (cumulative future reward) starting from time t.

$y^k$ (gamma) is the discount factor (0 ≤ $y^k$ < 1) that determines the importance of future rewards.

$R_{t+k+1}$ is the reward received $k+1$ time steps after time t.

This formulation enables the agent to optimize for long-term performance rather than immediate accuracy, balancing short-term and future gains in phishing detection tasks.

3.5.3. Reduced False Positive Rate Through Reward–Penalty Mechanism

False positives in phishing detection are costly, leading to user frustration and loss of trust. In RL, we can explicitly define penalties for false positives, forcing the agent to improve its decision-making process. A reward function example is as follows:

$$R\left(s,a\right)=\left\{\begin{array}{c}+1\hspace{1em}\mathrm{i}\mathrm{f} \mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t} \mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n} \left(\mathrm{T}\mathrm{P} \mathrm{o}\mathrm{r} \mathrm{T}\mathrm{N}\right)\hfill \\ -1\hspace{1em}\mathrm{i}\mathrm{f} \mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e} \left(\mathrm{F}\mathrm{P}\right)\hfill \\ -x\hspace{1em}\mathrm{i}\mathrm{f} \mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e} \left(\mathrm{F}\mathrm{N}\right), x \ge 0\hfill \end{array}\right.$$

where the following variables are used:

$R\left(s,a\right)$ is the reward function based on state s and action a.

$\mathrm{T}\mathrm{P}$ is true positive, correctly identifying a phishing attempt.

$\mathrm{T}\mathrm{N}$ is true negative, correctly identifying benign content.

$\mathrm{F}\mathrm{P}$ is false positive, incorrectly classifying benign content as phishing.

$\mathrm{F}\mathrm{N}$ is false negative, missing an actual phishing email.

$x$ is a penalty value, with $x$ ≥ 0 applied for false negatives, ensuring heavier penalties for missing actual threats.

This differential penalty system ensures that false negatives (missing an actual phishing email) are penalized more heavily than false positives, aligning with real-world cybersecurity priorities.

3.5.4. Exploration–Exploitation Trade-Off

RL incorporates an exploration–exploitation mechanism (often using an ϵ) allowing the agent to explore new classification strategies while exploiting known optimal ones.

$$\pi \left(s\right)=\left\{\begin{array}{c}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{o}\mathrm{m} \mathrm{a}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}, \mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h} \mathrm{p}\mathrm{r}\mathrm{o}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y} \in \hfill \\ \arg {max}_a\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right), \mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h} \mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y} 1 -\in \hfill \end{array}\right.$$

where the following variables are used:

π(s) is the policy dictating action selection at state s.

$\in$ (epsilon) is the exploration rate (probability of selecting a random action).

$\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)$ represents the expected reward for taking action a in state s.

This balance helps the agent discover new phishing patterns without being confined to historical data, unlike static ML models. This formulation clearly illustrates the trade-off between exploring new possibilities (exploration) and leveraging existing knowledge (exploitation).

4. Methodology

Building on the limitations identified in traditional phishing detection strategies, including rule-based methods, heuristic approaches, and supervised ML techniques, our proposed model leverages RL to offer a dynamic and adaptive solution. As discussed in the previous section, RL outperforms traditional models by learning through continuous interaction with its environment, enabling real-time adaptation to new and sophisticated phishing attacks [54].

The core advantage of RL lies in its ability to self-improve over time without the need for manual retraining, unlike static models such as SVMs or heuristic-based filters that rely heavily on historical data [55]. This section details the methodology we employed, covering data collection, feature extraction, model architecture, and training procedures, all designed to maximize phishing detection accuracy while minimizing false positives.

4.1. Data Collection and Feature Extraction

To build and evaluate the phishing detection framework, two datasets were utilized:

• Dataset 1 (Real-World Dataset): A dataset consisting of 5000 emails, evenly distributed with 2500 phishing emails (sourced from PhishTank and OpenPhish archives) and 2500 benign emails (collected from public corporate email traffic and open datasets);
• Dataset 2 (Synthetic Phishing Dataset): A synthetic dataset containing 1000 phishing emails generated using templates based on real-world phishing attack patterns, including domain spoofing, urgent calls-to-action, credential harvesting requests, and misleading hyperlinks.

Dataset 1 was split using an 80/20 ratio (4000 for training and 1000 for testing). Dataset 2 was reserved exclusively for external validation without prior model exposure. Features extracted included domain age, presence of HTTPS, URL/IP pattern analysis, email header metadata, and text-based features derived from natural language processing. Feature extraction is a critical preprocessing step that transforms raw email data into structured information for the RL agent. We focused on extracting both content-based and header-based features, including the following:

• URL Features: Length of URLs, number of subdomains, presence of IP addresses, and suspicious domain patterns;
• HTML Structure: Presence of embedded scripts, hidden form fields, and iframe usage;
• Sender Reputation: Mismatched “From” and “Reply-To” addresses, domain age, and SPF/DKIM validation status;
• Keyword Patterns: Occurrence of phishing-related keywords like “urgent”, “verify your account”, and “password reset”.

Let F represent the feature set extracted from each email:

$$F_i=f1,f2,f3,\dots ,fn$$

where the following variables are used:

$F_i$ represents an individual feature (e.g., URL length and sender reputation score).

$n$ is the total number of extracted features.

These features were normalized to ensure uniform data distribution before being fed into the RL agent.

4.2. RL Model

Our proposed phishing detection system is based on DQN architecture, a widely adopted RL technique that combines Q-Learning with deep neural networks to handle large, complex feature spaces effectively [54].

4.2.1. Agent–Environment Interaction

In the RL framework, the phishing detection model operates as an agent interacting with an environment (the dataset). The environment presents emails with various features, and the agent’s task is to classify each as either phishing or benign. The interaction is formalized as a Markov Decision Process (MDP), defined by the tuple (S, A, R, P, γ), where the following variables are used:

S = Set of states (email features);

A = Set of actions (phishing, benign);

R = Reward function;

P = State transition probability;

γ = Discount factor for future rewards.

4.2.2. Q-Learning Algorithm

The DQN agent learns to make decisions by updating a Q-value for each state–action pair, which estimates the expected cumulative reward:

$$Q\left(s,a\right)\leftarrow Q\left(s,a\right)+\infty \left[r+\gamma \ast a^{\prime }maxQ\left(s^{\prime },a^{\prime }\right)-Q\left(s,a\right)\right]$$

where the following variables are used:

$Q\left(s,a\right)$= Q-value for state s and action a;

$\infty$ = Learning rate;

$r$ = Immediate reward after action a;

$\gamma$ = Discount factor for future rewards;

$s^{\prime }$ = the next state resulting from action a;

$a^{\prime }$ = represents the possible actions in the next state s′.

The agent receives positive rewards for correct classifications and penalties for false positives/negatives, allowing it to refine its policy over time.

4.3. Training and Evaluation

The model was trained in over 10,000 iterations to ensure convergence. The reward system was defined as follows:

$$R\left(s,a\right)=\left\{\begin{array}{c}+1\hspace{1em}\mathrm{i}\mathrm{f} \mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t} \mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n} \left(\mathrm{T}\mathrm{P}/\mathrm{T}\mathrm{N}\right)\hfill \\ -1\hspace{1em}\mathrm{i}\mathrm{f} \mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e} \left(\mathrm{F}\mathrm{P}\right)\hfill \\ -x\hspace{1em}\mathrm{i}\mathrm{f} \mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e} \mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e} \left(\mathrm{F}\mathrm{N}\right)\hfill \end{array}\right.$$

where the following variables are used:

TP (True Positive): Correctly identified phishing email;

TN (True Negative): Correctly identified legitimate email;

FP (False Positive): Legitimate email misclassified as phishing;

FN (False Negative): Phishing email misclassified as legitimate.

This differential penalty system heavily penalizes false negatives, aligning with the cybersecurity principle that failing to detect a phishing email is more dangerous than mistakenly flagging a legitimate one.

Table 3. Comparative performance metrics (training phase).

Model	Accuracy (%)	False Positive Rate (%)
RL-Based DQN	95	2
SVM	85	12
Random Forest	87	10

below summarizes the phishing detection accuracy and false positive rates during the training phase across different models.

These results highlight the adaptive advantage of the RL model, as it continuously improves without the need for retraining—a limitation in static ML models.

4.4. Linking Back to Previous Strategies

Unlike rule-based systems that rely on static thresholds [51], or heuristic models prone to high false positives [52], our RL-based approach learns from real-time feedback, dynamically adjusting its decision-making process. It also overcomes the data-dependency issues seen in supervised ML techniques [53,55], offering the following:

• Real-Time Adaptation: Continuous learning from new phishing patterns;
• Lower False Positive Rates: Optimized through tailored reward mechanisms;
• Scalability: Effective across diverse phishing attack vectors.

This methodological framework demonstrates how RL bridges the gaps left by traditional phishing detection models, providing a robust, adaptive, and scalable solution for modern cybersecurity threats.

5. Model Ongoing Training

5.1. Training Dataset and Feature Processing

Building on the methodology outlined in the previous section, the training phase was designed to optimize the DQN-based RL phishing detection system. Our approach focused on maximizing adaptability, minimizing false positives, and achieving high detection accuracy.

The training process involved multiple stages, from dataset interaction to real-time parameter adjustments, ensuring that the model could effectively learn phishing detection patterns through trial-and-error reinforcement. TensorFlow v2.13.0, a widely used open-source ML framework, was employed to construct and optimize the neural network architecture supporting the DQN model. To create a realistic phishing detection scenario, the RL agent was trained using 4000 emails (2000 phishing and 2000 benign), sampled from the full dataset of 5000 emails (2500 phishing + 2500 benign), consisting of the following:

• 2500 phishing emails (extracted from the PhishTank database and other public datasets [30]);
• 2500 benign emails (collected from real-world email traffic under controlled conditions).

Each email underwent feature extraction (as described in the methodology section), producing structured input data that was then fed into the RL model. During training, the agent continuously interacted with the dataset and updated its decision-making policy based on reward-based learning, where correct classifications received positive reinforcement, while misclassifications (false positives/false negatives) were penalized.

The training process was executed over 10,000 episodes, where the RL agent classified emails as phishing or benign while continuously refining its detection policy. At each step, the agent received rewards for correct classifications and penalties for false positives or false negatives. The training leveraged TensorFlow’s dynamic optimization to fine-tune performance in real-time. The section and

Table 4. Key training parameters used in the RL-based phishing detection model, configured to optimize performance and reduce false positives.

Parameter	Value
Total Training Episodes	10,000
Exploration Strategy	Epsilon-Greedy
Discount Factor (γ)	0.95
Learning Rate (α)	0.001
Batch Size	64
Target Network Update Frequency	Every 100 steps
Reward Function	Positive reward for correct classifications; penalties for false positives, with heavier penalties for false negatives

below summarize the key parameters used during the model training:

• Total Training Episodes: 10,000;
• Exploration Strategy: Epsilon-Greedy (balancing exploration of new decisions vs. exploiting known patterns);
• Discount Factor: 0.95 (controlling the influence of future rewards on current decisions);
• Learning Rate: 0.001 (adjusting how quickly the model updates based on new information);
• Batch Size: 64 (number of emails processed per learning step).
• Target Network Update Frequency: Every 100 steps (stabilizing training to avoid policy divergence);
• Reward Function; (phishing or benign): Rewarded
• Correct Classification (phishing or benign): Rewarded;
• False Positives: Penalized;
• False Negatives: Heavily penalized to ensure phishing threats are prioritized.

This adaptive reward mechanism improved detection accuracy while reducing false positives, a significant challenge in conventional phishing detection models.

Through this setup, TensorFlow allowed us to iterate and fine-tune the model’s behavior, ensuring that it adapted well to the phishing threats while minimizing false positives. The model achieved a final accuracy of 95% with a 2% false positive rate, significantly outperforming traditional models averaging 85–87% accuracy and higher false positive rates. The training process was executed over 10,000 episodes, where the RL agent classified emails as phishing or benign while continuously refining its detection policy. At each step, the agent received rewards for correct classifications and penalties for false positives or false negatives. The training leveraged TensorFlow’s dynamic optimization to fine-tune performance in real-time.

Through this setup, TensorFlow allowed us to iterate and fine-tune the model’s behavior, ensuring that it adapted well to phishing threats while minimizing false positives. The model achieved a final 95% accuracy with a 2% false positive rate, significantly outperforming traditional models, which averaged as an 85–87% accuracy with higher false positive rates. In addition, to assess generalization performance, the trained RL model was validated against a synthetic phishing dataset consisting of 1000 artificially generated emails, demonstrating consistent detection accuracy and robustness against unseen phishing strategies.

5.2. Model Performance and Evaluation

Using TensorFlow’s logging and visualization features (TensorBoard), we monitored key performance metrics in real-time throughout the training process. Over the course of 10,000 training iterations, the model demonstrated a significant improvement in classification accuracy and adaptability, including the RL-based model’s continuous learning from environmental feedback, formalized through Markov Decision Processes (MDPs), enabling it to adapt more effectively to novel phishing patterns without necessitating extensive retraining, unlike traditional ML models.

Table 5. Comparative performance metrics (testing phase).

Model	Accuracy (%)	False Positive Rate (%)
RL-Based DQN	95	2
SVM	85	12
Random Forest	87	10
K-Nearest Neighbors (KNN)	80	14
Naïve Bayes	78	16
Logistic Regression	83	11
CNN	96	5
BERT-Based Detection	97	3
Double [27]	95	3

summarizes the performance of various phishing detection models during the testing phase.

5.3. Computational Benchmarking Results

To further assess the computational efficiency of the proposed RL-based phishing detection framework, several benchmarks were conducted during model training and evaluation.

Table 6. Computational benchmarking results.

Model	Training Time	Peak Memory Usage	Model Size (Parameters)	Convergence Speed
RL-Based DQN	35 min	420 MB	~1.2 million	7500 episodes
CNN	2.3 h	1.6 GB	~8 million	25 epochs
BERT-Based	5.5 h	3.8 GB	~110 million	5 epochs

below summarizes the comparative results detailed as follows:

• The RL-based DQN model required approximately 35 min of total training time across 10,000 episodes, with a peak memory consumption of 420 MB.
• The CNN-based phishing detection model required approximately 2.3 h of training and 1.6 GB of memory.
• The BERT-based phishing detection model required 5.5 h of training and 3.8 GB of memory during fine-tuning.
• The RL agent achieved convergence after approximately 7500 episodes, while CNN and BERT models required multiple epochs (25+ epochs for CNN; 5+ epochs for BERT) over the entire dataset.
• These findings demonstrate that the RL-based model achieved strong phishing detection accuracy with substantially reduced computational resource requirements compared to deep learning counterparts.

5.4. Detection Latency and Performance Trade-Off

In addition to detection accuracy, we evaluated the detection latency (inference time) of the RL-based phishing detection model compared to other ML methods. After the training phase, the RL-based DQN model demonstrated a low average inference time of approximately 2.3 ms per email sample. This was comparable to traditional ML models such as SVM (2.1 ms) and significantly faster than deep learning models like CNN (7.5 ms) and BERT (21 ms). Although RL models require extensive interactions during training, their real-time inference speed is highly efficient once deployed. Thus, the trade-off between training complexity and inference efficiency makes the proposed RL-based method suitable for real-world phishing detection scenarios requiring both accuracy and timeliness.

6. Testing and Results

To comprehensively evaluate our RL-based phishing detection model, we conducted rigorous testing using a validation dataset of 1000 emails, comparing it against various machine learning, DL, and RL models. The evaluation assessed key performance metrics, including accuracy, precision, recall, false positive rates, and AUC scores, as illustrated in Figure 3, Figure 4 and Figure 5.

Figure 3 above and

Table 7. Comparative performance metrics across phishing detection models (table format).

Model	Accuracy (%)	Precision (%)	Recall (%)	False Positive Rate (%)
SVM	85	84	83	12
Random Forest	87	86	85	10
Our RL-Based Model	95	96	94	2
K-Nearest Neighbors	80	79	79	14
Naïve Bayes	78	79	79	16
Logistic Regression	83	84	83	11
Deep Learning (CNN)	96	97	96	5
XGBoost	92	93	91	5
BERT-Based Model	97	98	97	3
PCA-RF	89	87	87	8
BRNN [25]	91	90	89	7
DDQN [27]	95	94	93	3

below provides a detailed comparison of the accuracy, precision, recall, and false positive rates across various phishing detection models. Traditional ML models, including SVM (85% accuracy, 84% precision, 83% recall, and 12% false positives) and Random Forest (87% accuracy, 86% precision, 85% recall, and 10% false positives), demonstrated moderate performance but struggled with higher false positive rates.

Deep learning models such as CNN (96% accuracy, 97% precision, 96% recall, and 5% false positives) and BERT-based phishing detection (97% accuracy, 98% precision, 97% recall, and 3% false positives) outperformed traditional ML approaches but required greater computational resources.

The proposed RL-based model achieved a 95% accuracy, 96% precision, 94% recall, and a notably low 2% false positive rate, offering a strong balance between adaptability, precision, and real-time efficiency.

In Figure 4 below, the ROC curves illustrate the trade-offs between true positive rates (TPR) and false positive rates (FPR) for each phishing detection model. The XGBoost model achieved the highest AUC score (0.97), followed closely by the RL-based model (0.96), BERT-based model (0.94), CNN (0.91), and DDQN (0.90). Traditional machine learning models such as SVM (0.88), Random Forest (0.75), Logistic Regression (0.60), and Naïve Bayes (0.68) exhibited comparatively weaker performance, with lower separability between phishing and benign samples. These results underscore the superior phishing detection capabilities of reinforcement and deep learning models, which demonstrate stronger classification accuracy while minimizing false positives.

In Figure 5 below provides a comparative bar chart illustrating the F1 scores of various phishing detection models. Our RL-based model leads with the highest F1 score (0.94), followed by CNN (0.91), DDQN [27] (0.89), and Random Forest (0.87). Classical models like Naïve Bayes and PCA-RF [24] show lower F1 scores of 0.78 and 0.72, respectively, highlighting their limited effectiveness in handling complex phishing patterns. These results demonstrate the superior classification balance achieved by reinforcement learning and deep learning models.

External Validation on Synthetic Dataset

To further assess the model’s robustness against unseen phishing attacks, the trained RL-based agent was evaluated on a synthetic phishing dataset comprising 1000 samples. The model achieved a 93% accuracy, 96% precision, and a 4% false positive rate on the synthetic data. While slightly lower than the results obtained on the real-world test set, these findings highlight the model’s generalization capabilities and adaptability to emerging phishing techniques.

7. Conclusions

While RL provides strong adaptability for phishing detection, RL models can still be vulnerable to adversarial attacks. Attackers may craft specially designed phishing emails to manipulate the input feature space, causing the RL agent to misclassify phishing attempts as benign. Such adversarial perturbations have been shown to significantly degrade the performance of machine learning-based cybersecurity systems, as discussed in [56] with the context of smart grid security. Although our proposed RL-based model was not explicitly trained against adversarial examples, future extensions should consider adversarial training techniques, input sanitization methods, and robustness evaluation to mitigate this risk and strengthen the resilience of phishing detection agents against sophisticated evasion tactics.

In this research, we introduced an RL-based phishing detection model, leveraging a DQN to improve adaptability and accuracy while reducing false positives. Unlike traditional ML methods, which require frequent retraining, our approach enables continuous learning through feedback mechanisms, enhancing its resilience against evolving phishing techniques. Experimental results demonstrated a 95% accuracy, 96% precision, 94% recall, and a 2% false positive rate, significantly outperforming conventional phishing detection methods. AUC analysis (0.92) further confirmed the model’s robustness, showing competitive performance with DL models while maintaining computational efficiency. The validation results obtained from both real-world and synthetic datasets demonstrate the proposed framework’s strong potential for practical deployment in dynamic phishing detection environments. Future research should explore real-time deployment to evaluate its performance in dynamic cybersecurity environments and integrate behavioral analysis and social engineering detection to enhance robustness. Transfer learning could improve adaptability across multiple phishing attack surfaces, while hybrid AI models combining RL with natural language processing (NLP) techniques, such as BERT or GPT-based architectures, could refine phishing text analysis. Additionally, optimizing computational efficiency and exploring federated learning would enable scalable deployment in enterprise security and mobile environments, ensuring a more resilient and adaptive phishing defense system.