Hybridizing Explainable AI (XAI) for Intelligent Feature Extraction in Phishing Website Detection

Abstract

This study proposes an explainability-driven feature selection framework for phishing website detection using a large-scale, heterogeneous dataset collected from four independent sources. The combined dataset contains approximately 500,000 samples, including 300,000 phishing pages and 200,000 legitimate pages, providing a comprehensive representation of real-world web traffic. To enhance model interpretability and reduce feature redundancy, four explainable artificial intelligence (XAI) techniques—SHAP, LIME, partial dependence plots (PDPs), and permutation importance (PDI)—were applied to rank and analyze feature contributions. The union of all selected features was subsequently refined through a thresholding mechanism, forming the proposed Hybrid Explainability Random Forest Algorithm (HXRF). A Random Forest (RF) classifier was trained using the optimized feature subset and evaluated on an independently sampled set of 2000 webpages. Results demonstrate that HXRF significantly improves classification performance, achieving an accuracy of 98.2%, with balanced precision, recall, and F1 scores. The confusion matrix confirms strong generalization across both phishing and legitimate classes, with minimal false predictions. This work demonstrates that combining multi-method XAI with selective feature filtering produces a compact, interpretable, and highly discriminative feature set capable of robust phishing detection at scale.

1. Introduction

The internet has become an inseparable part of modern life, shaping how people communicate, conduct business, and access information. From social networking and e-commerce to financial transactions and entertainment, almost every aspect of daily activity now relies on online connectivity. This rapid integration has paved the way for new technological paradigms such as the Internet of Things (IoT) [1], smart homes [2] and smart cities [3] where billions of interconnected devices exchange data autonomously to enhance efficiency and improve human convenience. These advancements have transformed homes, industries, and critical infrastructures into smart, interconnected ecosystems, bringing unprecedented opportunities for automation, real-time decision-making, and global collaboration.

However, the same connectivity that drives innovation also exposes individuals and organizations to significant cybersecurity risks. Among the various cyber threats, phishing attacks remain one of the most pervasive and damaging [4]. Phishing is a fraudulent practice where attackers impersonate trusted entities to deceive users into revealing sensitive information, such as passwords, credit card details, or financial credentials. According to the Anti-Phishing Working Group (APWG), phishing attacks reached record levels in recent years, with over 4.7 million attacks reported globally in 2023, marking a 150% increase compared to 2020 [5]. Financial losses are staggering; for instance, the FBI’s Internet Crime Complaint Center reported that phishing-related crimes accounted for over 44% of all reported cyber incidents in 2023, resulting in billions of dollars in damages [6]. These statistics highlight the urgency of developing robust and intelligent solutions to detect phishing websites and protect users before sensitive information is compromised.

Traditional phishing detection methods—such as blacklists [7], heuristic rules [8], and URL-based filtering [9]—have been widely deployed but remain limited in adaptability. Blacklists can only block previously reported phishing domains, leaving users vulnerable to newly created malicious websites. Heuristic-based approaches, while more dynamic, often suffer from high false-positive rates and require constant manual updates to remain effective. To overcome these limitations, researchers have increasingly turned to machine learning (ML) and deep learning (DL) techniques, which can analyze large datasets of web features and learn complex patterns that distinguish phishing websites from legitimate ones. ML and DL methods have shown significant promise by enabling real-time detection, adaptability to evolving attack strategies, and superior accuracy compared to classical methods [10,11].

This paper proposes a novel phishing detection model called HXRF (Hybrid Explainable Random Forest) that addresses key limitations of existing approaches by combining the predictive power of machine learning with the interpretability of explainable artificial intelligence (XAI). HXRF introduces a two-layer architecture: the first layer performs feature selection using four complementary XAI algorithms—SHAP, LIME, PDP, and Permutation Importance (PDI)—to identify the most discriminative and relevant features from the dataset. These algorithms capture different perspectives of feature influence, ensuring a more stable and unbiased selection process. The second layer utilizes a Random Forest classifier trained on the optimized feature set to accurately classify websites as phishing or legitimate. To further enhance performance, the feature-ranking computations of the XAI algorithms are executed in parallel using multithreading or GPU acceleration, reducing computational overhead while maintaining high accuracy. By integrating multiple XAI techniques, HXRF not only improves the robustness of the selected features but also provides interpretable insights into the detection process, making it more transparent and trustworthy for cybersecurity analysts.

Our contribution in this work can be summarized as follows:

• Hybrid XAI-driven feature selection: A unique integration of SHAP, LIME, PDP, and PI is employed to extract and rank the most relevant features, combining multiple interpretability perspectives to improve the robustness of the final feature set.
• Enhanced detection accuracy and transparency: By leveraging the strengths of multiple XAI methods alongside a Random Forest classifier, HXRF achieves high classification accuracy while maintaining model interpretability, enabling cybersecurity experts to understand the rationale behind each prediction.
• Flexible architecture: Although Random Forest is employed as the primary classifier, the framework allows the substitution of other ML or DL models without modifying the XAI-driven feature selection layer, making the system adaptable to future phishing detection advancements.

The rest of this paper is organized as follows. Section 2 provides an overview of the background of the phishing detection and feature extraction process. Section 3 introduces the HXRF algorithm. Section 4 provides an overview of the experiment. Section 5 discusses the obtained results. We conclude this paper in Section 6.

2. Background

This section reviews feature types leveraged in the process of phishing detection. Subsequently, it reviews XAI and its algorithms. Finally, it shows the related works that have been conducted in ML and DL in the area of web phishing detection.

2.1. Features Utilized in Phishing Detection

Phishing websites evolve rapidly, making detection difficult with any single indicator. Reliable models therefore combine multiple feature types that capture structural, semantic, and behavioral patterns. These features form the foundation for machine learning and deep learning algorithms to distinguish phishing sites from legitimate ones. Figure 1 shows these categories. In refs. [12,13] the authors review and survey different features and methods utilized in phishing detection. The following list summarize these features.

• URL-Based Features

URL features describe the structure and composition of a web address. Indicators such as excessive length, suspicious subdomains, embedded IP addresses, or abnormal use of special characters often reveal deceptive intent. These features are easily extracted, and do not require the retrieving of the website or reading of its content. In ref. [13], the authors proposed a CNN model to detect website phishing utilizing features extracted from the URL only, such as, hand-crafted features, character embeddings, character-level TF-IDF, and character-level count. The authors reported an accuracy of 95% on their own constructed dataset that combined websites from four different datasets available online. Approximately 500k links have been combined in the new dataset.

• HTML/DOM Features

These features reflect the internal structure of a webpage, including suspicious tags, hidden elements, or abnormal script usage. Phishing sites frequently manipulate HTML or JavaScript to obscure true links or mimic trusted layouts. These features require reading the website code and extracting the website’s structure. In ref. [14], the authors attempted to leverage the internal structure of HTML code to generate a graph. Subsequently, a Graph neural network has been utilized to extract features from the constructed graphs. Finally, URL-based features have been added and a CNN has been used for the final classification purpose. PhishTank [15] has been used to create the dataset used with approximately 100k links.

• Content-Based Features

Content features evaluate textual and visual elements on the page. Mismatched logos, misspellings, or misleading language in forms and pop-ups can indicate fraudulent intent. These features are popular in Email and short message (SMS) phishing detection [16]. The survey in ref. [17] reviews how NLP is utilized to extract features for different Email phishing detection techniques.

• Network-Based Features

Network attributes capture the hosting environment, such as WHOIS data, DNS records, server location, or SSL certificate details. Short domain lifespans, hidden registrants, or untrusted certificate issuers are common red flags. Many statistical features can be extracted, such as traffic volume, PageRank index and Google index. The authors in ref. [18] utilized these features and combined them with other features, such as URL features to construct a web phishing detector using ANN. Two different sources of data have been used: PhishTank and the phishing dataset at UCI. An Optimal sensitive feature selection algorithm has been proposed to rank features. This index has been leveraged for feature selection.

• Visual Features

Visual similarity metrics compare the phishing page to legitimate targets using image hashing or screenshot analysis [19]. High resemblance to known brands can expose imitation attacks.

• F. Behavioral Features

Behavioral indicators arise from user interaction or site dynamics, such as abnormal redirection chains, fast-changing URLs, or inconsistent response times. In ref. [20], the authors employ real-time behavioral analysis of DNS and HTTP traffic to detect phishing domains as they evolve.

• Hybrid Features

Hybrid features combine multiple categories—such as merging URL, content, and network signals—to improve detection accuracy and robustness. This is the most popular category for features used in most of the proposed methods as shown in Table 1.

2.2. Explainable Artificial Intelligence (XAI) Algorithms

Explainable Artificial Intelligence (XAI) [21] is a field of research that aims to reduce the opacity of complex machine learning models by providing interpretable representations of their decision-making processes. Modern ML and DL algorithms are capable of modeling highly nonlinear relationships in data, but this flexibility comes at the cost of interpretability, a property that describes the degree to which a human can consistently predict a model’s output given its input. In formal terms, XAI seeks to approximate or expose the decision function $f:R^n\to R$ of a predictive model by mapping high-dimensional feature spaces into human-comprehensible structures without significantly degrading predictive accuracy. In other words, XAI attempts to explain how certain features impact the accuracy of the developed algorithm. XAI methods are typically classified along two key theoretical dimensions:

• Model specificity: whether the explanation is tied to a particular class of models, such as decision trees or linear models, or is model-agnostic and applicable to any black-box function.
• Scope of explanation: whether the explanation targets the global behavior of the model “the approximate functional form of $f\left(x\right)$ across the input space” or provides a local explanation of a single instance $x_0$ by approximating $f\left(x\right)$ in a neighborhood around $x_0$. Many XAI approaches can be viewed as constructing a surrogate function $g\left(x\right)$ that is interpretable, such as a linear or rule-based function that satisfies a fidelity constraint as in Equation (1).

$$\underset{g}{\min }{E}_{x-D}\left[L\left(f\left(x\right),g\left(x\right)\right)\right]$$

(1)

where $L$ is a loss function measuring the divergence between the predictions of the black-box model $f\left(x\right)$ and the surrogate $g\left(x\right)$. High-fidelity surrogates preserve the decision boundary of the original model while remaining simple enough for human reasoning. The following subsections describe XAI algorithms utilized in this work.

2.2.1. Model-Specific Interpretable Models

Model-specific approaches [22] leverage the internal structure of the predictive model to generate explanations. Because the interpretability is intrinsic, no external surrogate is required. Popular algorithms in this model are the Tree-based algorithms, such as RF, ExtraTrees and DT. The proposed model is constructed utilizing an RF algorithm. Interpretability in Random Forests is achieved through feature importance measures that quantify the contribution of each input variable to the ensemble’s predictive power. A common approach is Mean Decrease in Impurity (MDI), where the importance of feature $j$ is computed as the total reduction in Gini impurity or entropy across all splits involving $j$ weighted by the number of samples reaching each node. Alternatively, Permutation Feature Importance (PFI) evaluates the change in model accuracy when the values of feature $j$ are randomly permuted. Feature importance is calculated as in Equation (2).

$$Importance\left(j\right)=E\left[Accuracy\left(X\right)-Accuracy\left(x_{perm\left(j\right)}\right)\right]$$

(2)

Both metrics provide global explanations by ranking features such as URL length, number of subdomains, or presence of HTTPS according to their impact on phishing detection.

2.2.2. Local Interpretable Model-Agnostic Explanations (LIME)

LIME [23] constructs a local surrogate model around the instance of interest $x_0$ to approximate the behavior of the black-box model $f\left(x\right)$. The method samples a set of perturbed points $\left\{x_i\right\}$ in the neighborhood of $x_0$, obtains their predictions $f\left(x_i\right)$ and fits a sparse linear model $g\left(x\right)$ that minimizes Equation (3).

$$\mathcal{L}\left(f,g,{\pi }_{x_0}\right)={\displaystyle \sum }_i{\pi }_{x_0}\left(x_i\right){\left[f\left(x_i\right)-g\left(x_i\right)\right]}^2+\Omega \left(g\right)$$

(3)

where ${\pi }_{x_0}$ is a proximity kernel assigning higher weights to points closer to $x_0$ and $\Omega \left(g\right)$ is a complexity penalty. The coefficients of $g\left(x\right)$ represent feature importance in the local region, showing which URL tokens or HTML patterns drove the specific phishing prediction.

2.2.3. SHapley Additive exPlanations (SHAP)

SHAP [24] uses Shapley values from cooperative game theory to compute the marginal contribution of each feature to the model’s prediction.

Given a set of features $F$ the Shapley value for feature $j$ is defined in Equation (4).

$${\varnothing }_j={\displaystyle \sum }_{S ⊑F\left\{j\right\}}{\displaystyle \frac{\left|S\right|!\left(\left|F\right|-\left|S\right|-1\right)!}{\left|F\right|!}} \left[f_{S\cup \left\{j\right\}}\left(x_{S\cup \left\{j\right\}}\right)-f_S\left(x_s\right)\right]$$

(4)

where $f_S$ is the model prediction using only features in subset $S$. This ensures fair attribution by averaging all possible coalitions of features. SHAP values satisfy key properties such as local accuracy, consistency, and symmetry, making them theoretically grounded. In phishing detection, SHAP can precisely quantify the impact of features such as SSL status or domain age on an individual prediction.

2.2.4. Feature Interaction and PDP/ALE

Partial Dependence Plots (PDP) [25] and Accumulated Local Effects (ALE) estimate the marginal effect of a feature subset S on the prediction by computing Equation (5).

$$f_S‘\left(x_S\right)=E_{x_S^{’}}\left[f\left(x_S, x_S^{’}\right)\right]$$

(5)

where $S‘$ is the complement set. These techniques reveal non-linear dependencies and interactions, such as how URL length interacts with the presence of special symbols to elevate phishing risk.

2.2.5. Permutation Feature Importance

Permutation Importance (PDI) [26] is a model-agnostic method that measures how much a model’s predictive performance deteriorates when the relationship between a feature and the target is disrupted. After training a model $f$ on a validation set $D = \left\{\left(x_i, y_i\right)\right\}$, the baseline metric $M_{base}$ accuracy is computed. For each feature $j$, its values $x_{ij}$ are randomly permuted across samples to break any predictive association, yielding a permuted dataset $D^{{\pi }_j}$. The model is re-evaluated to obtain $M_j$, and the importance score is computed in Equation (6).

$$PI\left(j\right)=M_{base}-M_j$$

(6)

A larger drop indicates a stronger contribution of feature $j$ to the model’s decision function.

Because PI operates directly on the trained model’s outputs, it captures nonlinear interactions and can be applied to any classifier, including Random Forests and deep networks. In phishing detection, PI can rank diverse features such as URL length, SSL status, or redirection count by their true predictive value, helping analysts prioritize key indicators and refine feature selection.

2.3. ML and DL in Phishing Detection

Phishing detection has progressed from rule-based heuristics to data-driven machine learning (ML) and deep learning (DL) approaches capable of handling complex, high-dimensional data. Studies in Table 1 show that both paradigms routinely achieve more than 95% accuracy, but differ in methodology, feature handling, and scalability.

ML models such as Random Forest (RF), XGBoost, SVM, and Logistic Regression remain widely used for their efficiency and interpretability. RF, in particular [27,28,29,30,31,32], reaches up to 99.99% accuracy by leveraging ensemble learning and built-in feature importance, while feature selection methods—including fuzzy rough sets [32], explainable ranking [28], and reinforcement learning [33] remove redundancy and enhance generalization. These models excel when handcrafted features, such as URL length, domain age, HTML structure, and lexical cues, are carefully engineered and require less data and computation than DL.

DL models, by contrast, automatically learn representations from raw URLs, email text, or webpage content. Architectures such as CNNs, LSTMs, BERT, TCN, and fully connected networks achieve state-of-the-art performance, with reported accuracies up to 99.8% [34]. CNNs capture character-level URL patterns [13,35,36], while LSTMs model sequential dependencies in URLs or emails [37,38,39], and transformers like BERT exploit contextual embeddings for email content [37]. DL reduces the need for manual feature engineering but demands larger datasets and greater computational resources.

Overall, ML relies on explicit feature design and offers easier explainability, making it attractive for settings requiring transparency, whereas DL scales better with large, unstructured data and discovers complex patterns automatically. Emerging hybrid systems combine DL-based feature extraction with ML classifiers [35], seeking to merge DL’s representational power with ML’s interpretability and efficiency for robust phishing detection.

Table 1. Comparison of different phishing Detection Techniques and Their Datasets.

Ref.	Dataset Details	ML/DL	Accuracy	Selected Features	Feature Selection Method(s)
[37]	Email dataset	BERT, LSTM	99.61	Textual features extracted using NLP techniques	None
[38]	URL dataset	LSTM, XGBoost	96.04	Character-level TF-IDF features	None
[27]	URL dataset	Random Forest	96.83	Hybrid features: URL-based and hyperlink-based features	Filter-based ranking and incremental removal of less important features
[28]	Mendeley dataset	Random Forest	97.78	23 features selected from an original set of 48 features	Explainable feature selection framework
[40]	PhiUSIIL dataset	FCNN	99.3	56 features heuristic-based and statistical	None
[41]	A new dataset	none	none	111 features combined of all types	None
[42]	DARTH Email	ANN, XGBoost	99.98	Combined features from Email content and NLP	None
[33]	Email dataset	Reinforcement (RAIDER)	94.00	Reduced feature set through reinforcement learning-based feature evaluation	Reinforcement learning-based feature evaluation
[29]	Email dataset (English-Arabic)	Random Forest	97.37	Domain names, IP addresses, open ports	None
[35]	Website Phishing Detection	DNN, LSTM, CNN with Grid Search and Genetic	97.37	Combined 48 features (Tan-dataset)	none
[39]	AntiPhishStack	LSTM, XGBoost	96.05	URL and TF-IDF with 30 features	none
[30]	Website phishing	Two models XGboost, RF, SVM, LR	99.7	5 new features: Information (CN), Logo Domain (LD), Form Action Domain (FAD), Most Common Link in Domain (MCLD) and Cookie Domain	Rank the 5 features
[32]	Benchmark dataset with 14,000 website samples	Random Forest	95.00	Universal feature set selected using Fuzzy Rough Set theory	Fuzzy Rough Set feature selection
[43]	Turkish email dataset	Keras-based deep learning model	93.97	Textual and structural features	None
[31]	UCI ML phishing URL dataset	Random Forest	99.99	URL-based features	None
[36]	URL phishing detector	1D CNN	99.7	Combined features from multi sources	None
[34]	Dataset with 11,449 samples	TCN	99.8	Title, copyright information, NER, login form detection, keyword-based retrieval	None
[13]	website phishing	CNN	95	URL features	None
[44]	website phishing	Multi-stacked model with multi-ML algorithms	97	Combined features 80 features	None
[45]	website phishing	Two layers of multi-ML algorithms	96.5	Combined features 80 features	None

Table 1. Comparison of different phishing Detection Techniques and Their Datasets.

Ref.	Dataset Details	ML/DL	Accuracy	Selected Features	Feature Selection Method(s)
[37]	Email dataset	BERT, LSTM	99.61	Textual features extracted using NLP techniques	None
[38]	URL dataset	LSTM, XGBoost	96.04	Character-level TF-IDF features	None
[27]	URL dataset	Random Forest	96.83	Hybrid features: URL-based and hyperlink-based features	Filter-based ranking and incremental removal of less important features
[28]	Mendeley dataset	Random Forest	97.78	23 features selected from an original set of 48 features	Explainable feature selection framework
[40]	PhiUSIIL dataset	FCNN	99.3	56 features heuristic-based and statistical	None
[41]	A new dataset	none	none	111 features combined of all types	None
[42]	DARTH Email	ANN, XGBoost	99.98	Combined features from Email content and NLP	None
[33]	Email dataset	Reinforcement (RAIDER)	94.00	Reduced feature set through reinforcement learning-based feature evaluation	Reinforcement learning-based feature evaluation
[29]	Email dataset (English-Arabic)	Random Forest	97.37	Domain names, IP addresses, open ports	None
[35]	Website Phishing Detection	DNN, LSTM, CNN with Grid Search and Genetic	97.37	Combined 48 features (Tan-dataset)	none
[39]	AntiPhishStack	LSTM, XGBoost	96.05	URL and TF-IDF with 30 features	none
[30]	Website phishing	Two models XGboost, RF, SVM, LR	99.7	5 new features: Information (CN), Logo Domain (LD), Form Action Domain (FAD), Most Common Link in Domain (MCLD) and Cookie Domain	Rank the 5 features
[32]	Benchmark dataset with 14,000 website samples	Random Forest	95.00	Universal feature set selected using Fuzzy Rough Set theory	Fuzzy Rough Set feature selection
[43]	Turkish email dataset	Keras-based deep learning model	93.97	Textual and structural features	None
[31]	UCI ML phishing URL dataset	Random Forest	99.99	URL-based features	None
[36]	URL phishing detector	1D CNN	99.7	Combined features from multi sources	None
[34]	Dataset with 11,449 samples	TCN	99.8	Title, copyright information, NER, login form detection, keyword-based retrieval	None
[13]	website phishing	CNN	95	URL features	None
[44]	website phishing	Multi-stacked model with multi-ML algorithms	97	Combined features 80 features	None
[45]	website phishing	Two layers of multi-ML algorithms	96.5	Combined features 80 features	None

3. The Hybrid XAI-Random Forest (HXRF) Model

The HXRF (Hybrid Explainable Random Forest) model is a two-tier web-phishing detection framework that combines explainable artificial intelligence (XAI) feature selection with a high-performance machine learning classifier to accurately categorize websites as legitimate or phishing. The model is structured into two sequential layers that work in harmony to achieve both interpretability and predictive accuracy. Figure 2 shows the architecture of HXRF.

The first layer performs feature selection using four complementary XAI algorithms—SHAP, LIME, PDP and PI. Each of these algorithms evaluates the contribution of every feature to the prediction outcome on a per-sample basis, meaning that a single execution provides importance scores only for one data instance. Relying on a single XAI technique can lead to bias or the omission of important attributes because each method uses different mathematical foundations and interpretability strategies. SHAP, for example, is based on cooperative game theory and captures both global and local effects, while LIME focuses on local perturbations, PDP measures the average marginal effect of features, and PI evaluates changes in model performance when a feature is permuted. By combining the outputs of these methods, HXRF leverages their diverse perspectives to construct a more stable, comprehensive, and trustworthy feature ranking. This multi-algorithm approach reduces the risk of overfitting to the biases of any single method and ensures that subtle but important features are not overlooked.

To create a global ranking across the entire dataset, HXRF executes each XAI algorithm over all samples and aggregates the results by calculating the mean importance score of every feature. The aggregated scores are then normalized and compared to a global threshold defined as the mean of all feature means, and features whose scores are greater than or equal to this threshold are retained for each algorithm. To accelerate this computationally intensive process, HXRF employs parallelization using multithreading or GPU acceleration with CUDA, enabling SHAP, LIME, PDP, and PI to operate concurrently on different subsets of the data. After all algorithms complete their evaluations, the top-ranked features from each method are combined through a union operation, and duplicate or redundant features are removed to form a single optimized feature subset. This subset represents the most discriminative and interpretable attributes within the dataset and serves as the input to the second layer of HXRF.

The second layer applies a Random Forest (RF) classifier trained on the reduced feature space to predict whether a given website is legitimate or phished. RF is selected for its strong performance in high-dimensional spaces and its robustness to overfitting, although the architecture of HXRF is modular and can accommodate alternative machine learning or deep learning classifiers (such as XGBoost or neural networks) without altering the upstream feature selection process. This flexibility ensures that the model can be adapted to different datasets or evolving phishing strategies without redesigning the entire pipeline.

The key advantages of HXRF are multifold. First, it achieves high interpretability, as the feature selection layer provides transparent insights into which features most strongly influence classification decisions. Second, it ensures computational efficiency by employing parallelized scoring and ranking, significantly reducing processing time for large datasets. Third, it offers flexibility through its model-agnostic classification layer, allowing seamless integration of different learning algorithms. Fourth, it provides enhanced robustness and stability by combining multiple XAI methods, which minimizes the risk of feature selection bias and ensures that the final feature set reflects a balanced consensus of importance rather than the limitations of a single method. Finally, HXRF delivers enhanced predictive accuracy, as the integration of diverse XAI perspectives ensures that only the most relevant and non-redundant features are retained, producing a more discriminative and reliable feature space for phishing detection. Figure 3 shows the Pseudo code of HXRF.

4. Experiment

The experimental workflow consists of two main stages: (1) dataset construction and (2) model training and evaluation. This section presents the full methodology used to create a large-scale phishing detection dataset and describes the machine learning pipeline implemented using a Random Forest classifier.

4.1. Phishing URL Collection

The first part of the dataset was constructed by downloading approximately 300,000 phishing URLs using the abuse.ch API [46] the links has been confirmed and recently reported malicious URLs, URLHaus provides continuously updated, verified phishing and malware domains, ensuring high-quality malicious samples.

The second part of the phished links has been downloaded from Kaggle [47]. This dataset consists of 2000 links with 88 different features. The description of these features is described in [48]. This dataset is a mixture of phishing and legitimate webpages. This dataset served as both a benchmark and a reference for designing the feature extraction process for the newly collected URLs. Finally, to add more legitimate webpages, a list of websites has been downloaded using Cloudflare Radar. In the end, the list consisted of approximately 505k sites.

4.2. Feature Extraction for Collected URLs

To ensure consistency between the Kaggle dataset and the URLs gathered from URLHaus, a custom Python 3 pipeline was developed to extract the same 88 features from all additional samples. Multiple external services, APIs, and Python libraries were integrated to compute these features. Examples include:

• Google Indexing check (to evaluate whether the page is indexed by Google Search)
• Domain age and registration details via WHOIS lookup libraries
• URL and hostname lexical features (length, number of dots, entropy, etc.)
• SSL certificate attributes collected through TLS inspection
• JavaScript features, form analysis, and link statistics parsed directly from webpage HTML

This extraction pipeline ensured uniform feature representation across all data sources. After processing, the unified dataset contained, 300k phishing URLs and 200k normal URLs. Approximately 500k samples total with 88 features each.

4.3. Model Training Using Random Forest

Hardware Setup

All experiments were conducted on a workstation equipped with:

• Intel Core i7 (12th Generation)
• NVIDIA GeForce RTX 5070 GPU
• 16 GB RAM

Although Random Forest is CPU-based and does not rely on GPU acceleration, the hardware configuration ensured smooth execution of feature extraction, preprocessing, and large-scale data handling. Python has been utilized with different Libs, such as pandas, scikit-learn, matplotlib, lime, shap and dice-ml. Random Forest (RF) was selected as the primary classifier for the following reasons:

High performance on tabular data: RF is well-known for its strong accuracy and robustness when dealing with heterogeneous, structured datasets such as URL-based features.

• Resistance to overfitting: Through the use of multiple decision trees and bootstrap sampling, RF generalizes well even with large feature sets.
• Interpretability and compatibility with XAI: RF integrates smoothly with SHAP, LIME, PDP, and PDI for explainability, making it ideal for XAI-guided feature selection.
• Feature importance analysis: RF naturally provides importance scores that facilitate downstream ranking and correlation with XAI outputs.
• Scalability: RF can effectively handle datasets with hundreds of thousands of samples without requiring GPU acceleration.

Given these advantages, RF was chosen as the core predictive model and served as the foundation for evaluating the impact of different feature selection methods.

5. Results

This section begins by establishing a baseline for comparison with HXRF. It then presents the outputs of the selected explainable AI (XAI) methods applied to the dataset, followed by a description of the Random Forest (RF) classification process using the features selected by these methods. Finally, the accuracy of the proposed HXRF model is reported. To illustrate the behavior of the XAI techniques, SHAP and LIME are employed.

The complete dataset is used with 10-fold cross-validation to train five classifiers: Random Forest (RF), XGBoost, Logistic Regression (LR), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN). A paired t-test on F1-scores is subsequently conducted to assess the statistical significance of performance differences among the models. The results are summarized in

Table 2. Accuracy of Baseline Models.

	Accuracy	F1	Recall	Precision	ROC_AUC
LR	0.945669	0.945536	0.943129	0.948018	0.985428
RF	0.967104	0.967097	0.966579	0.967670	0.994000
SVM	0.958705	0.958570	0.955381	0.961821	0.990171
KNN	0.940507	0.939810	0.929140	0.950904	0.975920
XGBoost	0.969291	0.969303	0.969726	0.968933	0.994733

and

Table 3. p-Value of the Baseline Models.

Model A	Model B	p-Value
LR	RF	2.199845 $\times {10}^{-7}$
LR	SVM	2.590974$\times {10}^{-5}$
LR	KNN	6.381361$\times {10}^{-2}$
LR	XGBoost	8.996150$\times {10}^{-7}$
RF	SVN	4.165687$\times {10}^{-5}$
RF	KNN	5.770413$\times {10}^{-7}$
RF	XGBoost	1.709066 × 10−1
SVM	KNN	3.488691$\times {10}^{-6}$
SVM	XGBoost	4.657034$\times {10}^{-5}$
KNN	XGBoost	1.817466$\times {10}^{-7}$

. Tree-based ensemble models outperform both linear and distance-based classifiers, with XGBoost achieving the highest overall performance, closely followed by RF. The paired t-test analysis indicates that the performance difference between XGBoost and Random Forest is not statistically significant (p > 0.05), while both models significantly outperform LR, SVM, and KNN.

Figure 4 shows the confusion matrix of both XGBoost and RF baselines.

For XAI models, LIME offers a straightforward way to interpret a model’s prediction for an individual data sample by quantifying the contribution of each feature. Rather than providing only a final classification, LIME reveals which specific features pushed the sample toward being labeled as normal or abnormal. Positive feature values indicate that the feature supports a normal classification, increasing the likelihood that the sample is safe, whereas negative values indicate that the feature drives the prediction toward abnormal or suspicious. For instance, a website was randomly selected from the dataset, and LIME was applied to this sample. Figure 5 presents the LIME output for half of the features affecting this sample. The analysis showed that features such as nb_dollar ≤ 0, nb_tilde ≤ 0, and domain_in_brand ≤ 0 contributed positively toward a normal classification, while features like google_index ≤ 0, page_rank between 3 and 5, and low web traffic pushed the prediction toward abnormal. Overall, the positive contributions were stronger, resulting in a final classification of normal, as the safe indicators outweighed the suspicious ones.

Figure 6 illustrates the SHAP analysis for the same website sample. SHAP provides feature-level explanations by showing the contribution of each feature to the phishing prediction. For this sample, “length_hostname” had a SHAP value of –0.2 (red), pushing the model toward phishing, while “length_url” had a value of 0.2 (green), pushing the prediction toward legitimate. Red bars indicate features that increase the likelihood of phishing, and green bars indicate features that support normal behavior. In this case, the positive effect of “length_url” slightly outweighed the negative impact of “length_hostname”, so the overall prediction for the website leaned toward normal. This demonstrates how SHAP can explain the influence of individual features on a model’s decision.

Figure 5 and Figure 6 illustrate how the employed XAI algorithms explain the influence of individual features on specific samples. In our methodology, these explanation techniques are used not only for interpretability but also as a mechanism for feature selection by providing ranked importance scores for all features.

Figure 7 presents the top 20 features ranked by the four XAI selected algorithms, computed using their mean absolute contribution values across the entire dataset. As shown, the two algorithms do not produce identical rankings. For instance, “google_index” appears as the most influential feature in both SHAP and LIME. However, discrepancies emerge in the subsequent ranks: “page_rank” is identified as the second most important feature by SHAP, whereas LIME places it in the third position. It worth mentioning that extracting the top features of SHAP was fast compared with the other algorithms, taking 4 min. However, LIME and PDI required the longest time with approximately 35 min.

These differences highlight an important observation: relying solely on a single XAI method for feature ranking may introduce bias and potentially reduce the model’s overall accuracy. Using multiple XAI algorithms provides a more robust assessment of feature importance and reduces the risk of overfitting or misrepresenting the underlying data patterns.

Figure 8 presents the performance of the RF classifier when trained using the top 20 features selected by the four algorithms. To assess the contribution of each feature, the classifier was evaluated incrementally: starting with only the single most important feature, then the top two, and continuing until all twenty features were included. The results demonstrate that SHAP-based feature ranking enables the classifier to achieve higher accuracy more rapidly compared with other algorithms. In other words, SHAP identifies a more informative subset of features earlier, allowing the model to reach strong performance with fewer inputs. For instance, the accuracy curve for SHAP begins to stabilize after approximately 12 features, indicating that additional features contribute only marginal improvements. In addition, PDP conversions were faster than others with 15 features. In contrast, LIME requires nearly the full set of 20 features to reach a comparable level of accuracy. This difference suggests that SHAP and PDP provide a more efficient ordering of feature importance, while LIME distributes importance more gradually across a larger number of features. These findings further support the use of multiple XAI techniques for feature selection, as they reveal complementary insights and help avoid overlooking critical features.

The RF ML algorithm has been trained with 10-fold-cross validation utilizing the 79 features in the dataset. An 0.965% accuracy has been recorded. To enhance this value, the HXRF selects features from four different XAI algorithms. Three different methods could be utilized. First, an intersect between the top ranked features in these algorithms could be used. The top 20 features have been selected in all algorithms since 20 features are above the mean value of the importance value of features in SHAP, PDP and LIME. With the intersection, six different features are selected. The importance of each one of these features have been calculated by averaging the importance value of all algorithms. Figure 9 shows these features.

The second method of feature selection is the union of all features. Figure 10 shows 49 different features selected from the algorithms with their importance value. The third and the utilized method is to calculate the mean of the union features. Subsequently, this value is utilized as a threshold value for feature selection. Figure 11 shows 12 selected features.

To evaluate HXRF against alternative configurations, an RF model is first trained using the complete feature set, as reported in Table 2. The RF model is then retrained using subsets of top-ranked features, specifically those whose importance scores exceed the mean ranking value produced by each XAI method employed in this study. Additionally, RF models are trained using both the intersection and the union of features selected by the XAI techniques. In total, this results in seven distinct models. The performance of each model is evaluated in terms of accuracy, F1-score, precision, and recall, with the results summarized in

Table 4. Performance Comparison of HXRF.

	Accuracy	F1	Recall	Precision
HXRF	0.98204	0.984139	0.986754	0.981538
RF_All_Features	0.967104	0.967087	0.966579	0.967595
Union + RF	0.965442	0.965475	0.966404	0.964548
Intersect + RF	0.771216	0.756676	0.711461	0.808029
SHAP + RF	0.930359	0.930698	0.935258	0.926183
LIME + RF	0.923622	0.923535	0.922485	0.924588
PDP + RF	0.872966	0.878798	0.921085	0.840223
PDI + RF	0.920822	0.920691	0.919160	0.922226

. The results demonstrate a clear performance advantage of the proposed HXRF model over all other RF-based configurations. This indicates that the hybrid explainability-driven feature selection strategy effectively preserves discriminative information while reducing redundancy. This confirms that integrating multiple XAI insights in a guided manner leads to a more informative and balanced feature subset.

The Union + RF model performs comparably to the full-feature RF, indicating that aggregating features from different XAI methods retains most of the predictive power but does not provide additional gains due to residual feature redundancy. In contrast, the Intersect + RF configuration exhibits a substantial performance drop across all metrics, particularly in F1-score (0.7567) and precision (0.7115). This highlights that relying solely on features agreed upon by all XAI methods is overly restrictive, leading to the exclusion of complementary yet informative features and resulting in underfitting.

Among individual XAI-based feature selection approaches, SHAP + RF achieves the strongest performance, followed closely by LIME + RF and PDI + RF. This suggests that local and global feature attribution methods are effective at identifying relevant features, although their isolated use does not fully capture the complementary information exploited by HXRF. PDP + RF shows moderate performance, particularly in recall, indicating that partial dependence–based feature selection may overlook interaction effects critical for accurate classification.

In addition to the overall performance metrics, the stability of each model across the 10-fold cross-validation is also informative.

Table 5. Standard Deviation of Accuracy Across Folds.

Model	Standard Deviation
RF+ All Features	0.003818
HXRF	0.004155
RF-Union	005060
RF-Intersect	0.0132
RF-SHAP	0.00589
RF-LIME	0.00563
RF_PDP	0.01407
RF-PDI	0.00519

reports the standard deviation of accuracy across folds. HXRF exhibits a slightly higher standard deviation compared to the full-feature RF baseline, reflecting some variability introduced by the feature selection process. Nevertheless, this variation remains small, indicating that HXRF is generally robust across different data splits. Models based on intersected features show substantially higher standard deviations, suggesting that overly restrictive or incomplete feature sets can lead to inconsistent performance depending on the training fold.

The statistical significance analysis further supports the superiority of HXRF. A paired t-test comparing HXRF with the full-feature RF yields a t-statistic of −2.6374 and a p-value of 0.0270, indicating that the observed improvement in F1-score is statistically significant at the 5% level. This finding is corroborated by the Wilcoxon signed-rank test (statistic = 6.0, p-value = 0.0293), which confirms that the performance gains are not due to random variation.

Together, these results highlight that HXRF not only achieves higher predictive accuracy but also maintains stable and statistically significant improvements over standard RF models. The modest increase in cross-validation variability is an acceptable trade-off for the significant gains in performance, particularly when compared with unstable configurations like the intersected or partial dependence–based feature subsets. This emphasizes that carefully guided feature selection through HXRF improves both efficiency and reliability without sacrificing robustness. Figure 12 shows the confusion matrix of HXRF on the 500k dataset.

Figure 13 presents the confusion matrix obtained from testing the proposed HXRF-enhanced Random Forest model on an independent dataset of approximately 2000 samples, using the optimized feature subset derived from the thresholded Union (HXRF) selection. The model correctly classified 1122 true negatives and 1089 true positives, while producing only 35 false positives and 40 false negatives. These results indicate highly stable predictive behavior across both classes. The balance between false positives and false negatives further demonstrates that the model does not exhibit bias toward either the normal or abnormal class, maintaining consistent sensitivity and specificity. Overall, the confusion matrix confirms the strong generalization capability of the HXRF-guided feature reduction strategy, validating the previously reported performance metrics and supporting the conclusion that thresholding the multi-method XAI Union produces a compact yet highly discriminative feature set. This final experiment reinforces that HXRF not only improves accuracy, reaching 98.2% in the full evaluation, but also delivers robust and reliable predictions when tested on real, unseen data.

Finally, since HXRF employs multiple XAI models in its first layer to rank features for the selection process, it is important to assess the algorithm’s performance and computational cost in real time.

Table 6. XAI Average Single Sample Execution Time.

Model	Time (S)
SHAP	0.001342
LIME	0.272947
PDP	0.73937
PDI	0.339744

reports the average execution time for each XAI method used in the first layer to extract and rank features for a single sample. It should be noted that these methods are executed in parallel across different threads, resulting in an average increase of only 0.73 s for the feature ranking process. Importantly, this overhead does not make HXRF a slow algorithm, as XAI is applied solely during training for feature selection and is not used in the online prediction phase. Consequently, HXRF achieves faster predictions than the baseline models, requiring the extraction of only 12 features compared to 80 in the full-feature approach.

6. Conclusions

This paper introduced a hybrid explainability-driven feature selection framework designed to improve phishing website detection while maintaining interpretability and computational efficiency. A comprehensive dataset was constructed from four different sources, yielding nearly half a million samples with a realistic distribution of 300,000 phishing and 200,000 legitimate webpages. Four XAI algorithms—SHAP, LIME, PDP, and PDI—were applied to identify relevant features from the dataset. While the individual methods provided complementary insights, their direct union included redundant and low-impact attributes. To address this, we introduced the HXRF approach, which applies thresholding to the union of XAI-selected features, thereby producing a more compact and discriminative feature set.

Using this optimized feature subset, a Random Forest classifier was trained and validated on a randomly selected test set of 2000 webpages. The model demonstrated high predictive capability, achieving 98.2% accuracy and maintaining strong performance across precision, recall, and F1 metrics. The confusion matrix indicated balanced performance across both phishing and legitimate classes, confirming the robustness of the model and the effectiveness of the feature selection strategy.

Overall, the results show that combining multi-source data with multi-method XAI and thresholded feature refinement enhances both detection accuracy and interpretability. Beyond phishing detection, the HXRF framework is broadly applicable to other cybersecurity tasks, such as malware classification, network intrusion detection, and fraud detection, where feature interpretability and selection are critical. Additionally, the approach could be adapted to non-cybersecurity domains, including medical diagnosis, financial risk assessment, or any scenario where high-dimensional datasets require explainable and efficient feature selection. By providing a scalable and reliable foundation, HXRF highlights the value of explainability-guided feature engineering for both specialized and general machine learning applications. Future work may extend this approach to deep learning models, online detection systems, or adaptive feature selection in dynamic environments.