Review Reports
- Istiaque Bhuiyan 1 and
- Tanvir Bhuiyan 2,*
Reviewer 1: Dalia Simion Reviewer 2: Sayyed Abbas Reviewer 3: Anonymous
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe manuscript addresses a timely and relevant topic at the intersection of fintech, cybersecurity, phishing detection, and explainable AI. The study is well motivated and provides a clear contribution by distinguishing between generic phishing detection and finance-adapted phishing detection. The use of both email and webpage datasets offers a valuable multi-modal perspective, and the methodological design is generally robust.
The research objectives, hypotheses, datasets, preprocessing steps, model development, evaluation metrics, and statistical validation procedures are clearly presented. The inclusion of SHAP and permutation importance also strengthens the interpretability and practical relevance of the study.
The results are clearly organized and support the main conclusions. In particular, the manuscript shows that finance adaptation improves performance mainly in finance-themed email phishing detection, while the benefits are weaker and less consistent for webpage classification. The discussion of the specialization–generalization trade-off is also useful and relevant for both researchers and practitioners.
However, the manuscript would benefit from several minor revisions. Some sections, especially the literature review and discussion, are dense and somewhat repetitive, and could be made more concise. The English language should also be revised to improve clarity, flow, and readability. The authors should ensure consistency in formatting, abbreviations, citation style, and terminology throughout the manuscript.
This is a solid and relevant manuscript with a clear empirical contribution. I recommend acceptance after minor revision.Author Response
Comment: The manuscript addresses a timely and relevant topic at the intersection of fintech, cybersecurity, phishing detection, and explainable AI. The study is well motivated and provides a clear contribution by distinguishing between generic phishing detection and finance-adapted phishing detection. The use of both email and webpage datasets offers a valuable multi-modal perspective, and the methodological design is generally robust.
The research objectives, hypotheses, datasets, preprocessing steps, model development, evaluation metrics, and statistical validation procedures are clearly presented. The inclusion of SHAP and permutation importance also strengthens the interpretability and practical relevance of the study.
The results are clearly organized and support the main conclusions. In particular, the manuscript shows that finance adaptation improves performance mainly in finance-themed email phishing detection, while the benefits are weaker and less consistent for webpage classification. The discussion of the specialization–generalization trade-off is also useful and relevant for both researchers and practitioners.
However, the manuscript would benefit from several minor revisions. Some sections, especially the literature review and discussion, are dense and somewhat repetitive, and could be made more concise. The English language should also be revised to improve clarity, flow, and readability. The authors should ensure consistency in formatting, abbreviations, citation style, and terminology throughout the manuscript.
This is a solid and relevant manuscript with a clear empirical contribution. I recommend acceptance after minor revision.
Response: Thanks to the reviewer for the feedback. We have revised the literature review to make it more concise, less repetitive, and clearer in flow, while keeping the same logic and hypotheses from your uploaded section
Reviewer 2 Report
Comments and Suggestions for AuthorsComments and Suggestions for Authors
- Duplicate Table Numbering (Structural/Logical Error)
There are two separate tables both labelled as Table 23 in the manuscript. The first appears in Section 4.7 presenting the trade-off view of finance adaptation gain/loss, and the second appears in Section 4.9 presenting permutation importance results for the generic webpage phishing model. This is not merely a typographical issue — it creates a logical reference problem because Section 4.9 text explicitly directs the reader to "Table 23" for permutation importance results, yet the same number already refers to a different table earlier in the paper. All subsequent table numbers should be checked and corrected accordingly.
- Webpage FT Subset Size and Validity of H1 Testing (Logical Gap)
Table 21 reveals that the webpage dataset contains zero finance-themed instances at the dataset-wide labelling stage. However, Tables 18 and 19 subsequently report FT subset performance for webpage models with N=321 instances. The manuscript does not clearly explain how 321 FT webpage instances were obtained when the dataset-wide labelling produced zero FT records. This is a logical inconsistency that needs explicit clarification. If FT labelling was applied differently at the URL level versus the dataset-wide level, this distinction must be stated clearly in the methodology, because it directly affects the interpretation of H1 and H2 findings for the webpage modality.
- Marginal FA Random Forest Gain Used to Support H1 (Overreach)
In the webpage context, the FA Random Forest achieves an F1 gain of only +0.0018 over the generic model on the FT subset (Table 23, trade-off view). Despite this marginal difference, the manuscript uses this result as support for H1 in the webpage setting. Given that the FA models underperform the generic models for both Logistic Regression and Linear SVC on the same FT subset, concluding partial support for H1 based on a single classifier with a near-negligible margin requires more careful qualification. The authors should either provide a minimum threshold for what constitutes a practically meaningful gain, or acknowledge more explicitly that the webpage evidence for H1 is too weak to draw reliable conclusions.
- Bootstrap CIs and McNemar's Test Applied Only to Email Models (Asymmetric Validation)
Section 4.8 presents statistical validation exclusively for email FA model gains. No equivalent statistical validation is reported for webpage model comparisons. This creates a logical asymmetry: the email findings are supported by rigorous significance testing, while the webpage findings, including the partial H1 support from FA Random Forest, rest solely on point estimates. Since H2 and H3 require cross-modality comparisons, the absence of statistical validation for webpage results weakens the evidentiary basis for these hypotheses. Bootstrap CIs or McNemar's test should be applied to webpage model comparisons as well, or the authors should explicitly justify why this was not done.
- Non-Finance Ratio Parameter Fixed at 1.0 Without Sensitivity Analysis (Methodological Logic)
The FA training strategy uses a non-finance ratio of 1.0, meaning the number of sampled non-finance records equals the number of FT records. This is a consequential design choice that directly determines the degree of finance specialisation in the trained models. However, the manuscript fixes this parameter without exploring whether different ratio values would produce different trade-off patterns. Since the specialisation–generalisation trade-off is a central finding of the study, the robustness of this finding to different ratio settings is a logical question that should be addressed, even briefly, either through a sensitivity analysis or a reasoned justification for why 1.0 is the appropriate choice.
- SHAP Applied Only to FT Webpage Subset (Scope Limitation)
Section 3.11 states that SHAP values were computed on the FT webpage subset specifically. However, the permutation importance results in Tables 23 and 24 appear to reflect overall model behaviour rather than FT-specific behaviour. If SHAP was computed on the FT subset but permutation importance was computed on the full test set, the two explainability measures are not directly comparable, and the joint interpretation used to support H2 may be logically inconsistent. The authors should clarify the exact evaluation scope for each explainability method and ensure that comparisons between them are made on a consistent basis.
Author Response
In the manuscripts changes are highlighted in yellow
Comment 1. Duplicate Table Numbering (Structural/Logical Error)
There are two separate tables both labelled as Table 23 in the manuscript. The first appears in
Section 4.7 presenting the trade-off view of finance adaptation gain/loss, and the second appears
in Section 4.9 presenting permutation importance results for the generic webpage phishing
model. This is not merely a typographical issue — it creates a logical reference problem because
Section 4.9 text explicitly directs the reader to "Table 23" for permutation importance results, yet
the same number already refers to a different table earlier in the paper. All subsequent table
numbers should be checked and corrected accordingly.
Response: Thank you for pointing this out. We agree that the duplicate Table 23 numbering
caused a structural and logical referencing problem. We have corrected the numbering
accordingly. We have also reviewed all subsequent table captions and in-text table references to
ensure that the numbering is sequential and that each reference points to the correct table.
Comment 2. Webpage FT Subset Size and Validity of H1 Testing (Logical Gap)
Table 21 reveals that the webpage dataset contains zero finance-themed instances at the
dataset-wide labelling stage. However, Tables 18 and 19 subsequently report FT subset
performance for webpage models with N=321 instances. The manuscript does not clearly explain
how 321 FT webpage instances were obtained when the dataset-wide labelling produced zero FT
records. This is a logical inconsistency that needs explicit clarification. If FT labelling was applied
differently at the URL level versus the dataset-wide level, this distinction must be stated clearly
in the methodology, because it directly affects the interpretation of H1 and H2 findings for the
webpage modality.
Response: Thank you for identifying this important point. We agree that the previous version did
not clearly distinguish between the two levels of finance-theme identification used for the
webpage dataset. We have now clarified that the “zero FT webpage instances” reported in Table
21 referred to the structured-feature level, where the engineered webpage variables did not
directly encode finance-related semantic content. However, the raw URL field was retained
before model input exclusion and was used separately for URL-level finance-theme screening.
This URL-level screening identified 321 finance-themed webpage records in the held-out test set.
These records were used only for subgroup evaluation in Tables 18 and 19, while the raw URL and
finance-theme labels were excluded from model training to prevent data leakage.
We revised the both sections 3.4 (methodology) and 4.3 (result) to explain the distinction
between structured-feature-level labelling and URL-level finance-theme screening. Table 21 has
been revised to report finance-theme coverage by labelling level. We also added clarification
before the webpage FT performance tables and revised the interpretation of H1 and H2 to state
that webpage FT results are based on URL-identified finance-themed records evaluated using
structured webpage models.
Comment 3. Marginal FA Random Forest Gain Used to Support H1 (Overreach)
In the webpage context, the FA Random Forest achieves an F1 gain of only +0.0018 over the
generic model on the FT subset (Table 23, trade-off view). Despite this marginal difference, the
manuscript uses this result as support for H1 in the webpage setting. Given that the FA models
underperform the generic models for both Logistic Regression and Linear SVC on the same FT
subset, concluding partial support for H1 based on a single classifier with a near-negligible
margin requires more careful qualification. The authors should either provide a minimum
threshold for what constitutes a practically meaningful gain or acknowledge more explicitly that
the webpage evidence for H1 is too weak to draw reliable conclusions.
Response: We agree with the reviewer that the previous wording overstated the webpage
evidence for H1. The FA Random Forest model produced only a small absolute F1 gain of +0.0018
over the generic Random Forest, corresponding to a relative gain of approximately +0.18%. While
small improvements can be operationally meaningful in large-scale fraud or phishing detection
systems, the present result is based on a relatively small URL-identified FT webpage subset and
is not consistent across classifier families. FA Logistic Regression and FA Linear SVC both
underperformed their generic counterparts. We have therefore revised the manuscript to state
that H1 is strongly supported in the email modality but not reliably supported in the webpage
modality. The webpage FA Random Forest result is now described as marginal, model-specific,
and insufficient for drawing a general conclusion. Please see the revised interpretation in section
4.4.
Comment 4. Bootstrap CIs and McNemar's Test Applied Only to Email Models (Asymmetric Validation)
Section 4.8 presents statistical validation exclusively for email FA model gains. No equivalent
statistical validation is reported for webpage model comparisons. This creates a logical
asymmetry: the email findings are supported by rigorous significance testing, while the webpage
findings, including the partial H1 support from FA Random Forest, rest solely on point estimates.
Since H2 and H3 require cross-modality comparisons, the absence of statistical validation for
webpage results weakens the evidentiary basis for these hypotheses. Bootstrap CIs or
McNemar's test should be applied to webpage model comparisons as well, or the authors should
explicitly justify why this was not done.
Response: Thank you for identifying this asymmetry in the validation strategy. We agree that the
previous version applied bootstrap confidence intervals and McNemar’s test only to the email
model comparisons, while webpage comparisons were interpreted mainly from point estimates.
To address this, we have extended the same statistical validation framework to the webpage
models. Specifically, we computed bootstrap confidence intervals for ΔF1 and ΔMCC and
applied McNemar’s test to paired predictions for each generic–FA webpage model comparison
on the URL-identified FT subset and the NFT subset. Please see the highlighted parts in section
4.8 reporting statistical validation for webpage model comparisons.
Comment 5. Non-Finance Ratio Parameter Fixed at 1.0 Without Sensitivity Analysis (Methodological
Logic)
The FA training strategy uses a non-finance ratio of 1.0, meaning the number of sampled non
finance records equals the number of FT records. This is a consequential design choice that
directly determines the degree of finance specialisation in the trained models. However, the
manuscript fixes this parameter without exploring whether different ratio values would produce
different trade-off patterns. Since the specialisation–generalisation trade-off is a central finding
of the study, the robustness of this finding to different ratio settings is a logical question that
should be addressed, even briefly, either through a sensitivity analysis or a reasoned justification
for why 1.0 the appropriate choice is.
Response: Thank you for raising this important methodological point. We agree that the non
finance ratio is a consequential design parameter because it determines the degree of finance
specialisation in the FA training set. In the original version, we used a ratio of 1.0 to create a
balanced FA setting, where the model was exposed to all finance-themed records and an
equivalent number of sampled non-finance records. However, we agree that the robustness of
the specialisation–generalisation trade-off should be examined more explicitly. We have added a
sensitivity analysis using alternative non-finance ratio values of 0.5 and 2.0. Please see the
highlighted section 4.10.
Comment 6. SHAP Applied Only to FT Webpage Subset (Scope Limitation)
Section 3.11 states that SHAP values were computed on the FT webpage subset specifically.
However, the permutation importance results in Tables 23 and 24 appear to reflect overall model
behaviour rather than FT-specific behaviour. If SHAP was computed on the FT subset but
permutation importance was computed on the full test set, the two explainability measures are
not directly comparable, and the joint interpretation used to support H2 may be logically
inconsistent. The authors should clarify the exact evaluation scope for each explainability
method and ensure that comparisons between them are made on a consistent basis.
Response: Thank you for raising this point. We have checked the analysis code and clarified the
evaluation scope in the manuscript. Both SHAP and permutation importance were computed on
the same URL-identified finance-themed webpage test subset containing 321 records. The raw
URL field was used only to identify the FT webpage subgroup and was excluded from model input.
Therefore, the SHAP and permutation importance results are directly comparable as FT-subset
explanations of the Random Forest webpage models. Please see the highlighted parts of section
4.9.
Author Response File:
Author Response.pdf
Reviewer 3 Report
Comments and Suggestions for AuthorsSee the report.
Comments for author File:
Comments.pdf
Author Response
Comment 1. The finance-theme labelling strategy is central to the contribution and should be more
transparent. Because the FT/NFT label determines the finance-adapted training set, subgroup
evaluation, and interpretation of domain-specific gains, readers need enough detail to assess this
construction. The authors should provide the full dictionary or appendix-level details, examples of FT
and NFT records, and a small manual validation exercise or sensitivity check around the threshold and
weights. It would also be useful to clarify whether the finance terms used for labelling overlap with the
TF-IDF features used by the email classifiers.
Response: Thank you for this important comment. We agree that the finance-theme labelling strategy
is central to the study and have revised the manuscript to improve transparency and reproducibility.
Full dictionary added:
We added the complete finance-theme dictionary in Appendix A1 including core finance terms, finance
brands/institutions, finance-context phishing phrases, generic phishing action terms, and their scoring
weights.
FT/NFT examples added:
We added examples of finance-themed and non-finance-themed records to illustrate how the rule-based
labelling procedure assigns FT/NFT labels. (See Appendix A2)
Manual validation added:
We added a small stratified manual validation exercise using email records and webpage URL records
to assess whether the rule-based FT/NFT labels align with manual judgement. See Appendix A3 and
revised section 4.10 (highlighted in green).
TF-IDF overlap clarified:
We clarified that some overlap between dictionary terms and TF-IDF tokens is unavoidable because
both are derived from the original email subject/body text. However, the derived finance-theme
variables, including finance score, FT/NFT label, and dictionary hit counts, were excluded from model
input. The dictionary was used only for subgroup construction and finance-adapted sampling, while the
classifiers learned from the full TF-IDF text representation. Please see the explanation in revised section
3.7 (highlighted in green).
Comment 2. Several sample-count and subgroup definitions need to be reconciled. The data section
describes the email dataset as 82,486 observations, while the abstract and Table 21 refer to 164,972
emails in stances. If this reflects pooling, duplication, augmentation, or preprocessing, the explanation
should be explicit. The webpage analysis also needs clarification: Table 21 reports zero finance-themed
webpage observations, yet Tables 18 and 19 report FT webpage results with N = 321. The bootstrap
gains in Figure 5 also appear larger than the corresponding F1 differences in Tables 16 and 17. A
compact audit table with sample counts and gain calculations by modality would help.
Response:
• Thank you for identifying these important consistency issues. The reference to 164,972 email
instances in the abstract/Table was caused by a reporting inconsistency and has been corrected.
• Thank you for identifying this important point. We agree that “The webpage analysis also needs
clarification: Table 21 reports zero finance-themed webpage observations, yet Tables 18 and 19
report FT webpage results with N = 321.” We have now clarified that the “zero FT webpage
instances” reported in Table 21 referred to the structured-feature level, where the engineered
webpage variables did not directly encode finance-related semantic content. However, the raw
URL field was retained before model input exclusion and was used separately for URL-level
finance-theme screening. This URL-level screening identified 321 finance-themed webpage
records in the held-out test set. These records were used only for subgroup evaluation in Tables
18 and 19, while the raw URL and finance-theme labels were excluded from model training to
prevent data leakage. We revised (see highlighted in yellow) the both sections 3.4 (methodology) and 4.3 (result) to
explain the distinction between structured-feature-level labelling and URL-level finance-theme
screening. Table 21 has been revised to report finance-theme coverage by labelling level. We
also added clarification before the webpage FT performance tables and revised the
interpretation of H1 and H2 to state that webpage FT results are based on URL-identified
finance-themed records evaluated using structured webpage models.
• We thank the reviewer for identifying the inconsistency in Figure 5. After checking the analysis
workflow, we found that the discrepancy was caused by an error in the Python input used to
generate Figure 5. We have corrected the Python input and regenerated Figure 5.
• We have added an audit table Appendix A4.
Comment 3. The interpretation should be framed as promising but conditional. The evidence is
strongest for email models, while the webpage results appear to rely mainly on broad structural and
reputation features. The XAI and practical discussion should therefore avoid implying strong financespecific webpage learning unless the FT webpage subset is clearly established. The discussion may also
benefit from briefly situating the fintech platform context. For example, Bao and Huang (2021) may be
useful for positioning fintech platforms as digital financial-intermediation settings where risk
management and model deployment choices have practical significance.
Response:
Thank you for this helpful comment. We agree that the interpretation should be framed as promising
but conditional rather than as uniformly strong evidence for finance-specific learning across both
modalities. In the revised manuscript, we have sharpened this distinction throughout the Abstract,
summary of findings, Results and Conclusion (See highlighted texts). We now state more explicitly that
the strongest evidence is obtained from the email modality, where finance-related semantic cues are
directly available in subject/body text and where finance-adapted models consistently outperform
generic models on the finance-themed subset.Finally, we have strengthened the fintech platform context by adding a new section 5.3 Implications for Fintech Platforms situating the study within digital financial intermediation settings. In this regard, weadded Bao and Huang (2021) to support the argument that fintech platforms are risk-bearing digital
financial-intermediation environments where model deployment, fraud control, and operational risk
management have practical significance.
Author Response File:
Author Response.pdf