Hybridizing Explainable AI (XAI) for Intelligent Feature Extraction in Phishing Website Detection

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript proposes a Hybrid Explainability Random Forest (HXRF) framework for phishing website detection, where multiple XAI techniques (SHAP, LIME, PDP, and Permutation Importance) are jointly used for robust feature selection, followed by Random Forest (RF) classification. The study is supported by a large-scale dataset collected from multiple real-world sources (URLHaus, Kaggle, Cloudflare), which significantly strengthens the experimental foundation.

However several methodological clarifications, statistical validations, and comparative baselines are still required to elevate the work to a stronger journal contribution. My suggestions on this aim are listed below:

1-The core components of the proposed framework are all well-established techniques in machine learning and explainable AI. The current proposal appears to be primarily a system-level integration rather than the introduction of a fundamentally new algorithmic contribution. The authors must clearly articulate the scientific novelty by explicitly addressing:

-What is truly new beyond simply “using multiple XAI methods together”?

-How does the proposed HXRF framework differ in principle, not just in implementation, from existing explainability-based feature selection frameworks (e.g., Refs. [30], [42])?

2-Although the reported performance metrics (Accuracy, Precision, Recall, F1) are impressive, the evaluation currently lacks Confidence intervals, hypothesis testing, and variance or standard deviation across folds. Please include:

-Standard deviation across cross-validation folds,

-A paired t-test or Wilcoxon signed-rank test comparing HXRF against the baseline RF using all features.

3-Although the manuscript claims that: “HXRF allows other ML/DL classifiers to be substituted” no experimental evidence is provided to support this claim. At minimum, HXRF-selected features should be evaluated using XGBoost, Logistic Regression and Support Vector Machines (SVM). Without this, the model-agnostic benefit remains unverified.

4-Feature ranking appears to be performed on the entire dataset, after which the Random Forest model is trained and tested. This introduces a serious risk of feature-selection leakage, which can artificially inflate performance. Please clarify that Was XAI-based feature selection performed inside each training fold only, or on the full dataset before splitting? This issue is critical for scientific validity.

The paper mentions that LIME required approximately 35 minutes and SHAP required about 4 minutes. However, the following are not addressed:

• Total end-to-end pipeline runtime,

• Scalability to very large datasets (~500k samples),

• Memory and computational resource usage.

Please include end-to-end runtime analysis, cost of parallel execution and a practical deployability and scalability discussion.

5-The statement “Dataset available upon request” is insufficient for reproducibility. At minimum, please provide public feature extraction scripts, or anonymized feature vectors for replication.

 

Author Response

Reviewer 1: 

 

Comment 1: 

the context of cybersecurity. The proposed Hybrid Explainability Random Forest (HXRF) framework, which combines four explainability methods (SHAP, LIME, PDP, PDI) for feature selection, is conceptually compelling. The authors have collected an impressive dataset (approximately 500,000 samples). However, the model validation method raises serious statistical concerns that must be clarified before publication.   The most serious shortcoming of the paper is its model-testing procedure.   Unrepresentative test set: The authors have a set of 500,000 samples, and the evaluation is conducted on an “independently drawn set of 2,000 pages” (Abstract; Section 3). This represents only 0.4% of the entire dataset. In machine learning, a standard split for such a large dataset is, for example, 70/30 or 80/20 (which would yield 100,000-150,000 test samples). Testing on such a small sample introduces enormous variance and does not allow for a reliable assessment of the model’s generalization. The 98.2% result could be due to chance for such a small subset.

Lack of cross-validation: The paper lacks information on k-fold cross-validation. With such a complex feature selection method, it is necessary to prove that the selected features are not overfit to the specific data partition.

Response: 10-fold cross validation has been utilized in this work. The baseline models have been trained with 10-Fold cross validation and HXRF has been also trained with it. The results section shows this method. For the 2000 samples, they have been used from another only source “Public dataset” that can be found in ref [46] as shown in the dataset section. We used them only to test the model after training/validation/testing on the 500k owned dataset. And the test on this external dataset shows the performance of the model on a public dataset. In the result section we have trained/validated/tested 5 different binary classifiers as a baseline to compare them with our model. The new results have been added into the result section of the paper in read color. 

 

Comment 2: 

Cost vs. Effect: The authors propose using up to 4 XAI methods simultaneously. Methods such as SHAP (especially KernelSHAP) are extremely computationally expensive. Is running all of them justified? The authors should conduct an ablation study: how the model performs using only PDI (the fastest) vs. a hybrid of the four methods. If the difference in accuracy is, for example, 0.1%, the computational overhead of the hybrid is unjustified in real-time systems.

Response: Finally, since HXRF employs multiple XAI models in its first layer to rank features for the selection process, it is important to assess the algorithm’s performance and computational cost in real time. Table 6 reports the average execution time for each XAI method used in the first layer to extract and rank features for a single sample. It should be noted that these methods are executed in parallel across different threads, resulting in an average increase of only 0.73 seconds for the feature ranking process. Importantly, this overhead does not make HXRF a slow algorithm, as XAI is applied solely during training for feature selection and is not used in the online prediction phase. Consequently, HXRF achieves faster predictions than the baseline models, requiring the extraction of only 12 features compared to 80 in the full-feature approach.

Table 6: XAI Average Single Sample Execution Time 

Model	Time (S)
SHAP	0.001342
LIME	0.272947
PDP	0.73937

 

This has been added to the result section of the paper. Moreover, HXRF enhanced the prediction accuracy with 1.3% 

Comment 3:

Thresholding Mechanism: The feature selection threshold is defined as the “average of the average importance scores.” This is a heuristic approach. There is no theoretical or empirical justification (e.g., an “elbow” plot) for why this threshold is optimal. 

 

Response: we have compared our proposal with the union of all features and with the intersect. We selected this method after comparing it will all features between the union and the intersect. We also compared with each XAI method alone as shown in table 4.  

 

Comment 4: 

The comparison of accuracy with other methods (Table 1) is misleading because each of the cited works uses a different dataset (PhishTank, UCI).  The authors should implement 1-2 simple baseline models (e.g., a standard Random Forest without HXRF selection and an XGBoost) on the same dataset to demonstrate the actual quality gains achieved by their method.

 

Response: five different ML models have been tested with the dataset before implementing HXRF and has been added to the result section of the  paper. The comparison of these methods is shown in table 2.

Comment 4:

In anti-phishing systems, the False Positive Rate (FPR) is crucial. Blocking a legitimate bank website is significantly more costly than allowing a single phishing attack through. The authors provide “minimal false predictions,” but the reviewer expects a specific confusion matrix (CMM) and an explicit FPR. With 2,000 test samples, even a single error is 0.05%, which is a significant rate given the network's scale.

 

Response: We have added a new confusion matrix for the 500k dataset figure 12. As mentioned, 10-fold cross validation has been used in this work. The last confusion matrix of 2000 samples is another dataset used to show how the mode, HXRF, performs on a public dataset. As mentioned this dataset can be found in Ref [46].  

 

Comment 5:

• Language Quality: English requires proofreading. Example errors:
• Page 3, line 118: “These features are easy to extract” -> should be “easily extracted”.
• Page 5, line 173: “attempts to explain” -> “attempts to explain”.
• Page 6, line 238: “values ​​are randomly permuted” -> probably referring to.
• Figures: The pseudocode in Figure 3 (page 10) is too general and contains formatting errors (e.g., truncated “MethodScores-” lines, missing indentation). It should be rewritten as a formalized algorithm.

Response : These issues have been checked 

Comment 6:

Features: The authors mention 88 features, but do not list them. An appendix with a complete list of features is required for the work to be reproducible.

Response: Reference [46] has a full list and full definition of these features 

Comment 7:

Please explain the reasoning behind using a 2000-sample test set, given that you have 500,000 samples available. Please repeat the experiment on a test set that represents at least 20% of the data.

Response: this is a public dataset used only to show how HXRF performs on another dataset. All the experiments testing/training/validation used 10-fold cross validation on the 500k dataset.

1. What is the time overhead (in seconds/minutes) of generating explanations for a single sample using all four XAI methods in real time? Does this allow for online detection?

Response: yes. Since XAI algorithms are required only in construction and training the algorithm. After that they are not used. Moreover, HXRF reduced the features that should be extracted for each site from 80 to 12 features. Table 6 shows the time required to execute each sample using XAI

2. Is the test set derived from the same sources as the training set, or is it a zero-shot test on a completely new source?

Response: both. The test/validation and training utilized a single dataset with cross validation. However, 2000 samples are from another source.

3. Please provide a confusion matrix for the expanded test set.

Response: Has been added figure 12

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors This paper addresses the important topic of interpretability of machine learning (XAI) models in the context of cybersecurity. The proposed Hybrid Explainability Random Forest (HXRF) framework, which combines four explainability methods (SHAP, LIME, PDP, PDI) for feature selection, is conceptually compelling. The authors have collected an impressive dataset (approximately 500,000 samples). However, the model validation method raises serious statistical concerns that must be clarified before publication.   The most serious shortcoming of the paper is its model-testing procedure.   Unrepresentative test set: The authors have a set of 500,000 samples, and the evaluation is conducted on an “independently drawn set of 2,000 pages” (Abstract; Section 3). This represents only 0.4% of the entire dataset. In machine learning, a standard split for such a large dataset is, for example, 70/30 or 80/20 (which would yield 100,000-150,000 test samples). Testing on such a small sample introduces enormous variance and does not allow for a reliable assessment of the model’s generalization. The 98.2% result could be due to chance for such a small subset.

Lack of cross-validation: The paper lacks information on k-fold cross-validation. With such a complex feature selection method, it is necessary to prove that the selected features are not overfit to the specific data partition.

Computational Complexity and Gain (HXRF):

• Cost vs. Effect: The authors propose using up to 4 XAI methods simultaneously. Methods such as SHAP (especially KernelSHAP) are extremely computationally expensive. Is running all of them justified? The authors should conduct an ablation study: how the model performs using only PDI (the fastest) vs. a hybrid of the four methods. If the difference in accuracy is, for example, 0.1%, the computational overhead of the hybrid is unjustified in real-time systems.
• Thresholding Mechanism: The feature selection threshold is defined as the “average of the average importance scores.” This is a heuristic approach. There is no theoretical or empirical justification (e.g., an “elbow” plot) for why this threshold is optimal.

Table 1:

• The comparison of accuracy with other methods (Table 1) is misleading because each of the cited works uses a different dataset (PhishTank, UCI).  The authors should implement 1-2 simple baseline models (e.g., a standard Random Forest without HXRF selection and an XGBoost) on the same dataset to demonstrate the actual quality gains achieved by their method.
• In anti-phishing systems, the False Positive Rate (FPR) is crucial. Blocking a legitimate bank website is significantly more costly than allowing a single phishing attack through. The authors provide “minimal false predictions,” but the reviewer expects a specific confusion matrix (CMM) and an explicit FPR. With 2,000 test samples, even a single error is 0.05%, which is a significant rate given the network's scale.

 

• Language Quality: English requires proofreading. Example errors:
• Page 3, line 118: “These features are easy to extract” -> should be “easily extracted”.
• Page 5, line 173: “attempts to explain” -> “attempts to explain”.
• Page 6, line 238: “values ​​are randomly permuted” -> probably referring to.
• Figures: The pseudocode in Figure 3 (page 10) is too general and contains formatting errors (e.g., truncated “MethodScores-” lines, missing indentation). It should be rewritten as a formalized algorithm.
• Features: The authors mention 88 features, but do not list them. An appendix with a complete list of features is required for the work to be reproducible.

Questions for the authors:

1. Please explain the reasoning behind using a 2000-sample test set, given that you have 500,000 samples available. Please repeat the experiment on a test set that represents at least 20% of the data.
2. What is the time overhead (in seconds/minutes) of generating explanations for a single sample using all four XAI methods in real time? Does this allow for online detection?
3. Is the test set derived from the same sources as the training set, or is it a zero-shot test on a completely new source?
4. Please provide a confusion matrix for the expanded test set.

     

Author Response

Comment 1:

-What is truly new beyond simply “using multiple XAI methods together”?

Response: has been added in the conclusion and abstract. HXRF shows that XAI algorithms cannot be used alone for feature extraction. In addition, another technique should be proposed to select the features after the ranking process. In the results, we have compared the performance of HXRF with RF+ each one of the XAI models used in this work. We have shown that one mode without the hybrid structure is not enough. Moreover, HXRF reduced the features required from 80 to 12. This speeds up the detection process and reduces the overhead in feature extraction. 

 

Comment2: -How does the proposed HXRF framework differ in principle, not just in implementation, from existing explainability-based feature selection frameworks (e.g., Refs. [30], [42])?

Response: The hybrid structure leverages the complementary strengths of multiple XAI models, each of which ranks features differently, as illustrated in Figure 7. To capitalize on the advantages of all these methods, a hybrid combination approach is employed, distinguishing our algorithm from related works. Furthermore, a mean-value–based feature selection strategy is proposed to reduce the number of features while improving overall accuracy.

Comment 3:

Although the reported performance metrics (Accuracy, Precision, Recall, F1) are impressive, the evaluation currently lacks Confidence intervals, hypothesis testing, and variance or standard deviation across folds. Please include Standard deviation across cross-validation folds, A paired t-test or Wilcoxon signed-rank test comparing HXRF against the baseline RF using all features.

Response: Have been added to the result section as followed

In addition to the overall performance metrics, the stability of each model across the 10-fold cross-validation is also informative. Table 5 reports the standard deviation of accuracy across folds. HXRF exhibits a slightly higher standard deviation compared to the full-feature RF baseline, reflecting some variability introduced by the feature selection process. Nevertheless, this variation remains small, indicating that HXRF is generally robust across different data splits. Models based on intersected features show substantially higher standard deviations, suggesting that overly restrictive or incomplete feature sets can lead to inconsistent performance depending on the training fold.

Table 5: Standard Deviation of Accuracy Across Folds

Model	Standard Deviation
RF+ All Features	0.003818
HXRF	0.004155
RF-Union	005060
RF-Intersect	0.0132
RF-SHAP	0.00589
RF-LIME	0.00563
RF_PDP	0.01407
RF-PDI	0.00519

 

The statistical significance analysis further supports the superiority of HXRF. A paired t-test comparing HXRF with the full-feature RF yields a t-statistic of -2.6374 and a p-value of 0.0270, indicating that the observed improvement in F1-score is statistically significant at the 5% level. This finding is corroborated by the Wilcoxon signed-rank test (statistic = 6.0, p-value = 0.0293), which confirms that the performance gains are not due to random variation.

 

3-Although the manuscript claims that: “HXRF allows other ML/DL classifiers to be substituted” no experimental evidence is provided to support this claim. At minimum, HXRF-selected features should be evaluated using XGBoost, Logistic Regression and Support Vector Machines (SVM). Without this, the model-agnostic benefit remains unverified.

Response: The algorithm has two stages and they are separated from each other. The second stage is the ML stage. HXRF uses RF. any other ML models can replace RF in this stage. However, we have selected RF and this is why we called it HXRF  

4-Feature ranking appears to be performed on the entire dataset, after which the Random Forest model is trained and tested. This introduces a serious risk of feature-selection leakage, which can artificially inflate performance. Please clarify that Was XAI-based feature selection performed inside each training fold only, or on the full dataset before splitting? This issue is critical for scientific validity.

Response: On the whole dataset before splitting

The paper mentions that LIME required approximately 35 minutes and SHAP required about 4 minutes. However, the following are not addressed:

• Total end-to-end pipeline runtime,
• Scalability to very large datasets (~500k samples),
• Memory and computational resource usage.

Please include end-to-end runtime analysis, cost of parallel execution and a practical deployability and scalability discussion.

Response: As mentioned, training took a long time for this dataset. We have calculated the average time required for each sample to extract XAI models values. As shown in table 6. However , in online, real time, HXRF is fast and light weight since it requires only the extraction of 12 different features. Four different threads have been used in the training process only for the first layer and a sigle thread for the second layer. However, these are only required in training. In the online-execution of the model. The 12 features shown in the paper can be extracted in parallel using 12 different threads with “join” to wait the output of these threads and to read them to the next layer. 

5-The statement “Dataset available upon request” is insufficient for reproducibility. At minimum, please provide public feature extraction scripts, or anonymized feature vectors for replication. 

Response: After the publication of the paper, a Kaggle page will be created with the dataset and the Python Script used in this work to be cited. However, right now it is hard to make the dataset public without any citation material. 

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The manuscript proposes a feature selection framework for phishing website detection. The phishing detection model called Hybrid Explainable Random Forest addresses key limitations of existing approaches by combining machine learning with the interpretability of explainable artificial intelligence.

Please consider implementing the following suggestions for improvement:

1) Page 1: Please define the Type of paper to "Article". The first line of the manuscript still contains the part from the template: "Type of the Paper (Article, Review, Communication, etc.)".

2) The order of the references' appearance must be in order; for example, line 256 mentions references, "29", "30", and then "35", but 31-34 must be mentioned before 35. This needs to be applied to the whole manuscript.

3) Please finish the line 369: "Approximately 500k samples total with 88 features each".

4) Please explain in more detail how you split the datasets into training and validation datasets.

5) Please provide the source code of the solution.

6) Please reflect on the other use cases where your approach could be reused in the conclusion section.

Author Response

Comment 1: 

Page 1: Please define the Type of paper to "Article". The first line of the manuscript still contains the part from the template: "Type of the Paper (Article, Review, Communication, etc.)".

Response: Has been selected 

Comment 2:

The order of the references' appearance must be in order; for example, line 256 mentions references, "29", "30", and then "35", but 31-34 must be mentioned before 35. This needs to be applied to the whole manuscript.

Response: Has been done 

Comment 3:

Please explain in more detail how you split the datasets into training and validation datasets.

Response: 10-fold cross validation has been used

Comment 4: Please provide the source code of the solution.

Response: After the publication of the paper, a Kaggle page will be created with the dataset and the Python Script used in this work to be cited. However, right now it is hard to make the dataset public without any citation material. 

Comment 5: Please reflect on the other use cases where your approach could be reused in the conclusion section.

Response: added to the conclusion 

“Beyond phishing detection, the HXRF framework is broadly applicable to other cybersecurity tasks, such as malware classification, network intrusion detection, and fraud detection, where feature interpretability and selection are critical. Additionally, the approach could be adapted to non-cybersecurity domains, including medical diagnosis, financial risk assessment, or any scenario where high-dimensional datasets require explainable and efficient feature selection. By providing a scalable and reliable foundation, HXRF highlights the value of explainability-guided feature engineering for both specialized and general machine learning applications. Future work may extend this approach to deep learning models, online detection systems, or adaptive feature selection in dynamic environments.”

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

All my comments and concerns have been adequately addressed by the authors. The revisions improved the clarity and quality of the manuscript, and I am satisfied with the current version. Therefore, I recommend that the paper be accepted in its present form.