LLM-Based Multimodal Feature Extraction and Hierarchical Fusion for Phishing Email Detection

Abstract

Phishing emails continue to evade conventional detection systems due to their increasingly sophisticated, multi-faceted social engineering tactics. To address the limitations of single-modality or rule-based approaches, we propose SAHF-PD, a novel phishing detection framework that integrates multi-modal feature extraction with semantic-aware hierarchical fusion, based on large language models (LLMs). Our method leverages modality-specialized large models, each guided by domain-specific prompts and constrained to a standardized output schema, to extract structured feature representations from four complementary sources associated with each phishing email: email body text; open-source intelligence (OSINT) derived from the key embedded URL; screenshot of the landing page; and the corresponding HTML/JavaScript source code. This design mitigates the unstructured and stochastic nature of raw generative outputs, yielding consistent, interpretable, and machine-readable features. These features are then integrated through our Semantic-Aware Hierarchical Fusion (SAHF) mechanism, which organizes them into core, auxiliary, and weakly associated layers according to their semantic relevance to phishing intent. This layered architecture enables dynamic weighting and redundancy reduction based on semantic relevance, which in turn highlights the most discriminative signals across modalities and enhances model interpretability. We also introduce PhishMMF, a publicly released multimodal feature dataset for phishing detection, comprising 11,672 human-verified samples with meticulously extracted structured features from all four modalities. Experiments with eight diverse classifiers demonstrate that the SAHF-PD framework enables exceptional performance. For instance, XGBoost equipped with SAHF attains an AUC of 0.99927 and an F1-score of 0.98728, outperforming the same model using the original feature representation. Moreover, SAHF compresses the original 228-dimensional feature space into a compact 56-dimensional representation (a 75.4% reduction), reducing the average training time across all eight classifiers by 43.7% while maintaining comparable detection accuracy. Ablation studies confirm the unique contribution of each modality. Our work establishes a transparent, efficient, and high-performance foundation for next-generation anti-phishing systems.

1. Introduction

Email continues to serve as a fundamental pillar of modern communication and business operations. Nevertheless, its ubiquity has rendered it a primary vector for cyberattacks, with phishing constituting one of the most prevalent and consequential threats. These attacks exploit human cognitive vulnerabilities through social engineering tactics, frequently employing techniques such as identity spoofing, malicious hyperlinks, and direct solicitation of sensitive information. Such methods can culminate in severe security incidents, including financial fraud and large-scale data breaches. The phishing threat landscape is undergoing rapid evolution: a report by Qi An Xin [1] indicates a tenfold increase in the volume of AI-generated phishing emails in 2023, highlighting a paradigm shift toward highly sophisticated and personalized attack methodologies. Adversaries are now leveraging generative artificial intelligence (AI) tools to produce highly convincing, contextually tailored content that closely mimics legitimate communications. This advancement significantly enhances attack efficacy and poses considerable difficulties for detection via conventional security mechanisms.

Consequently, traditional phishing detection methods, which rely heavily on keyword matching, rule-based engines, or shallow machine learning models, are increasingly unable to keep pace with the sophistication and adaptability of modern phishing campaigns. These approaches suffer from several critical limitations. First, they are brittle and exhibit poor generalization, struggling to adapt to the dynamic and constantly evolving tactics employed by attackers. Second, they typically depend on handcrafted, surface-level features that fail to capture deeper semantic cues and contextual deception strategies—such as emotional manipulation, urgency induction, or impersonation techniques. Although pre-trained language models like BERT offer more nuanced semantic representations, they still fall short in comprehensively modeling the multi-stage, intent-driven logic that characterizes advanced phishing operations.

To enhance detection robustness, a consensus has emerged that integrating multi-modal information analysis is imperative [2]. However, prevailing multi-modal fusion strategies encounter substantial limitations. Early fusion approaches, which concatenate features from disparate modalities, often fail to adequately capture complex intra-modal dynamics and inter-modal interactions. Late fusion techniques, which aggregate decisions from modality-specific models, risk neglecting the synergistic potential of complementary information across modalities, potentially resulting in suboptimal performance. More critically, these methods frequently treat the fusion process as a “black box,” exhibiting limited transparency regarding the contribution of individual features to the final decision. This opacity undermines model interpretability, erodes trust, and impedes security analysts’ capacity to validate, diagnose, and refine detection mechanisms.

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and logical reasoning, making them compelling candidates for advancing phishing detection systems. Their ability to discern subtle linguistic manipulations and interpret complex HTML/JavaScript structures holds significant promise for uncovering sophisticated phishing tactics. However, their practical deployment in this domain faces a fundamental challenge: LLMs typically generate unstructured, free-form textual outputs, which are ill-suited for integration into downstream machine learning pipelines that require quantifiable, structured feature vectors. This mismatch hinders the effective incorporation of LLM-derived semantic insights into systematic, multi-modal detection frameworks. Moreover, current LLM-based approaches tend to focus either on end-to-end classification or post-hoc interpretability, often neglecting the crucial step of structured feature engineering. As a result, despite recent progress in LLM-driven and multimodal phishing detection, three critical gaps persist:

(1)

The unstructured nature of LLM outputs impedes their compatibility with multimodal pipelines, complicating the alignment of semantic features across heterogeneous data sources;

(2)

Existing feature fusion strategies treat all inputs as a flat collection, disregarding their varying degrees of relevance to phishing intent—leading to redundancy and reduced interpretability;

(3)

There is a lack of standardized benchmarks that provide structured, multimodal feature representations, which limits reproducible evaluation and fair comparison across methods.

To address these challenges, this paper proposes SAHF-PD (Semantic-Aware Hierarchical Fusion for Phishing Detection), a novel framework for multi-modal phishing detection that redefines the paradigm of feature extraction and fusion. Our principal contributions are summarized as follows:

1.

Schema-Constrained LLM Feature Extraction: We propose a modality-specialized, prompt-driven pipeline that transforms LLMs into deterministic feature extractors via a unified UnifiedPhishingFeatures schema. This solves the core bottleneck of unstructured LLM outputs in prior work, enabling consistent, interpretable feature integration across text, URL OSINT, image, and code modalities.

2.

Semantic-Aware Hierarchical Fusion: We design a SAHF mechanism that stratifies features into core, auxiliary and weakly-associated layers based on mutual information with phishing intent, applies intra-modal PCA compression, and dynamically integrates features with weighted training. This goes beyond blind concatenation or black-box fusion in existing work, balancing accuracy, efficiency, and interpretability.

3.

A Benchmark Dataset for Structured Multi-Modal Phishing Detection: We release a human-verified dataset PhishMMF (Phishing Multi-modal Features) with structured features across four modalities, coupled with a modality presence mask to model missing data. This addresses the lack of standardized evaluation resources in prior multimodal phishing detection. The dataset is openly accessible at: https://github.com/12345677876/PhishMMF (accessed on 2 December 2025).

The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 details the proposed SAHF-PD framework. Section 4 presents the experimental setup and results. Section 5 concludes the paper and discusses future directions.

2. Related Work

The growing sophistication of phishing attacks has spurred extensive research into detection methodologies. This section categorizes and reviews the most relevant prior work, highlighting their strengths, limitations, and the research gaps that motivate our approach.

2.1. Rule-Based and Shallow Machine Learning Methods

Early phishing detection systems primarily relied on rule-based heuristics and shallow machine learning models. These approaches typically extract handcrafted features from URLs, email headers, or webpage content, such as lexical characteristics (e.g., domain length, presence of IP addresses) [3], blacklists/whitelists [4], or syntactic patterns in HTML [5]. While effective against simple attacks, these methods are inherently brittle. They require constant manual updates to keep pace with evolving attack tactics and struggle to generalize to novel or obfuscated phishing techniques. Their reliance on shallow features limits their ability to understand deep semantic context and sophisticated social engineering cues.

2.2. Deep Learning and Multi-Modal Fusion Approaches

To overcome the limitations of shallow methods, researchers have turned to deep learning for more powerful feature representation. Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) have been applied to analyze text content [6] and website screenshots [7]. More recently, multi-modal fusion has become a dominant paradigm, aiming to integrate complementary signals from different sources (e.g., URL, text, visual layout) to improve robustness. Common fusion strategies include early fusion, which concatenates raw or embedded features before input to a classifier [8], and late fusion, which combines predictions from separate single-modality models [9]. Hybrid fusion schemes have also been explored [10]. Despite their improved performance, these methods often treat modalities as isolated inputs and fail to explicitly model the complex interactions and dependencies between them. Furthermore, the fusion process is frequently opaque, making it difficult to interpret which specific features or modalities drove the final decision—a critical drawback for security applications requiring transparency and auditability.

2.3. Knowledge-Enhanced and LLM-Based Detection

Incorporating external knowledge, such as domain reputation databases (e.g., VirusTotal, Google Safe Browsing) or WHOIS information, has proven effective in enhancing detection accuracy [11]. These knowledge-enhanced methods provide valuable contextual intelligence but are reactive in nature and may be ineffective against newly registered malicious domains (zero-day threats).

The advent of Large Language Models (LLMs) has opened new avenues for phishing detection. Some studies leverage pre-trained language models like BERT for text classification tasks, achieving state-of-the-art results on textual phishing datasets [12]. More advanced approaches use LLMs for end-to-end classification [13] or to generate explanations for model decisions [14]. Beyond general semantic understanding, recent benchmarks emphasize that LLMs deployed in security-critical contexts must exhibit secure reasoning—the ability to recognize and avoid endorsing content that exhibits known adversarial patterns, such as deceptive urgency, brand impersonation, or credential solicitation [15]. In this light, LLMs can serve as powerful semantic analyzers capable of detecting subtle social engineering tactics that evade traditional lexical or syntactic checks.

However, the integration of LLMs into security pipelines introduces new risks. A recent survey on backdoor attacks in LLMs reveals that models can be compromised during fine-tuning via data poisoning (e.g., COVERT, TROJANPUZZLE) or manipulated at inference time through adversarial prompts or contaminated retrieval corpora [16]. In the context of phishing detection, an attacker could potentially poison the training data or external knowledge sources used by an LLM to systematically misclassify malicious messages as benign. This vulnerability underscores a critical limitation of current LLM-based detectors: they often treat the model as a monolithic black box, offering neither transparency nor mechanisms to verify the integrity of its reasoning.

Compounding these security risks, a significant technical gap remains: most existing LLM-based methods do not fully exploit the potential of LLMs for systematic feature engineering. They either use LLMs as opaque decision-making units or focus on generating natural language explanations, neglecting the crucial step of transforming rich LLM-generated insights into structured, quantifiable, and reusable feature vectors. This limitation prevents the seamless combination of LLM’s deep semantic understanding with other modalities like vision or code analysis within a unified framework.

2.4. Research Gap and Our Positioning

Overall, while the aforementioned approaches have made significant progress, several key challenges persist. First, there is a lack of systematic methods to extract structured, interpretable, and semantically rich features from multi-modal data, especially by leveraging the full analytical power of LLMs. Second, existing fusion mechanisms often sacrifice interpretability for performance, creating models that are difficult to trust in operational security settings. Third, the absence of standardized, high-quality benchmark datasets with structured multi-modal features hinders fair comparison and reproducible research. Our work directly addresses these gaps. We propose a framework that uses specialized LLMs guided by custom prompts to perform deep, modality-specific analysis and output features in a unified, structured format (UnifiedPhishingFeatures). This bridges the gap between LLMs’ unstructured outputs and the need for structured feature vectors. Furthermore, our semantic-aware hierarchical fusion mechanism (SAHF-PD) enhances both accuracy and interpretability. Finally, we contribute PhishMMF, a benchmark dataset designed to standardize evaluation in this domain.

3. Proposed Method

Modern phishing attacks employ a sophisticated, multi-modal strategy, combining deceptive text, URL obfuscation, visual spoofing, and malicious code to maximize their success rate. Traditional detection methods, constrained by shallow feature engineering and inadequate fusion mechanisms, struggle to counter these coordinated threats. To address this, we propose SAHF-PD (Semantic-Aware Hierarchical Fusion for Phishing Detection), a comprehensive framework that rethinks the entire detection pipeline. As illustrated in Figure 1, SAHF-PD consists of two core components: (1) an LLM-driven structured multi-modal feature extraction system, and (2) a semantic-aware hierarchical fusion mechanism. This design enables deep semantic analysis, generates interpretable features, and achieves high detection accuracy with enhanced model transparency.

3.1. Overview

The SAHF-PD framework operates in five sequential stages: (1) Multi-Modal Data Acquisition, (2) Modality-Specific LLM Processing, (3) Structured Feature Fusion and Vectorization, (4) Hierarchical Feature Selection and Fusion, and (5) Classification. The first three stages constitute our LLM-driven feature extraction pipeline, transforming raw, heterogeneous inputs into a standardized, high-dimensional feature vector. The final two stages implement the SAHF-PD fusion strategy, which refines this vector for efficient and interpretable classification.

3.2. LLM-Driven Structured Multi-Modal Feature Extraction

The primary goal of this component is to overcome the non-structured output limitation of LLMs and generate a unified, machine-readable feature representation from diverse data sources. We achieve this through a three-stage process.

3.2.1. Multi-Modal Data Acquisition

For a given phishing email, we automatically collect four key modalities:

(1)

Email Text: The subject line, sender address, and body content are parsed to preserve their original semantic structure.

(2)

URL Intelligence: For each URL extracted from the email, we query DNS and WHOIS records to obtain domain registration details, historical data, and reputation scores.

(3)

Website Screenshot: Using a headless browser (Puppeteer), we render the target webpage and capture a full-page screenshot to analyze its visual layout.

(4)

Website Code: We crawl the complete HTML source code and JavaScript, preserving the DOM tree structure and dynamic script behaviors. All data collection is performed in a secure sandbox environment to prevent any potential harm from malicious payloads.

3.2.2. Modality-Specific LLM Processing with Unified Template

The core of our feature extraction system lies in transforming large language models from general-purpose tools into specialized, deterministic feature extractors. This is achieved through a tightly coupled strategy combining specialized LLM variants, custom-designed prompts, and a unified output schema. Specifically, we assign dedicated LLMs—Qwen-Max for text and OSINT analysis, Qwen-VL for visual interpretation of screenshots, and Qwen-Coder-Plus for parsing website code—each optimized for its respective modality. To ensure deterministic output and reproducibility of feature extraction results, we strictly configure the model parameters as shown in

Table 1. Configuration Parameters of Specialized LLMs for Multi-Modal Feature Extraction.

Modality	Specific LLM Version	Temperature	Random Seed
Text	Qwen3-max	0.0	42
URL	Qwen3-max	0.0	42
Image	qwen3-vl-235b-a22b-thinking	0.0	42
Code	qwen3-coder-plus	0.0	42

Temperature is set to 0.0 to maximize output determinism and avoid stochastic variations. Consistent random seed (42) is used across all LLMs to ensure reproducible feature extraction results. All models are accessed via the official API with default timeout settings (30 s) and an automated retry mechanism (max 2 retries for failed requests).

. These LLMs are guided by meticulously crafted prompts that enforce strict adherence to the UnifiedPhishingFeatures schema, a hierarchical JSON structure encompassing 85 atomic features across four domains: text_features, url_intelligence_features, image_features, and website_features. The prompts instruct each LLM to assume the role of a “Phishing Email Analyst,” analyze its assigned input, and generate a structured JSON output containing only the predefined features, with missing values filled as “unknown”, default values applied for numerical and boolean fields, and empty arrays for list-type entries. For instance, the text-processing LLM populates attributes such as urgency level and sentiment score within text_features, while the image LLM identifies the presence of login forms or calculates visual similarity scores in image_features. This constrained generation process ensures consistency and interpretability, converting the raw, unstructured outputs of LLMs into a standardized, machine-readable feature object that is then fused into a single comprehensive representation. The complete UnifiedPhishingFeatures schema is provided in Appendix A and the example of prompt is shown in Appendix B.

3.2.3. Implementation Robustness and Quality Control

To ensure the reliability and reproducibility of our LLM-based feature extraction pipeline, we implement a suite of robustness mechanisms addressing service instability, inter-model inconsistency, and hallucination risks.

First, a prompt retry mechanism is employed to handle transient LLM failures or malformed outputs. After each inference, the response is validated against the UnifiedPhishingFeatures JSON schema using the jsonschema library. If validation fails, the system retries up to two additional times per modality, with a total timeout of 30 s per sample.

Second, to resolve cross-model disagreements, defined as inconsistent feature values for semantically equivalent attributes across modalities, we apply modality-specific priority rules grounded in cybersecurity domain knowledge. For example, the text modality is given precedence for semantic features such as phishing intent and urgency, while the code/HTML modality is prioritized for technical features including obfuscated scripts and suspicious URLs. Unresolvable edge cases are escalated to human experts for adjudication, and their decisions override model outputs.

Third, to mitigate hallucination and label noise, all prompts are explicitly designed to constrain the LLMs to extract only verifiable signals and refrain from speculative or generative behavior. Additionally, we implement post-hoc quality control through manual auditing of a randomly selected 5% subset of extracted features. All retry events, disagreement resolutions, and instances flagged as potential hallucinations are systematically logged to ensure full traceability. Together, these measures enhance the reliability and trustworthiness of the extracted feature set.

3.2.4. Feature Vectorization and Modality Completeness Modeling

The structured UnifiedPhishingFeatures object, rich in semantic information but heterogeneous in data types, is transformed into a fixed-length numerical vector suitable for machine learning. This vectorization process employs tailored encoding strategies: boolean features are directly mapped to binary values (0 or 1); categorical features such as top-level domains or SSL certificate status are converted via one-hot encoding, with low-frequency categories grouped to manage dimensionality; numerical features undergo Min-Max normalization, with a log-transform applied beforehand for those exhibiting long-tailed distributions like domain age. For string-valued and list-type features—such as lists of suspicious keywords extracted from text or images—each entry is first encoded into a 384-dimensional sentence embedding using a pre-trained SBERT model. The embeddings for list fields are averaged to produce a single representative vector, and Principal Component Analysis (PCA) is subsequently applied to reduce dimensionality while preserving at least 95% of the variance. The resulting encoded features from all modalities are concatenated into a base feature vector of 228 dimensions. To ensure robustness in real-world scenarios where certain modalities may be unavailable (e.g., a missing URL or an inaccessible webpage), a 4-dimensional modality presence mask is appended, with a value of 1 indicating a missing modality (and its features zero-padded) and 0 indicating its presence, as shown in Equation (1). This results in a final 232-dimensional input vector that is both comprehensive and resilient to incomplete data.

3.3. Semantic-Aware Hierarchical Fusion Module

The raw 232-dimensional vector, while rich in information, may contain redundancy and noise. The SAHF-PD mechanism refines this vector through a series of steps to enhance both performance and interpretability.

3.3.1. Semantic Feature Space Mapping and Modality Marking

We first establish a semantic mapping between each dimension of the 228-dimensional feature vector and its corresponding meaning in UnifiedPhishingFeatures. This creates a structured semantic feature space. The 4 appended dimensions serve as the modality presence mask, denoted as:

$$\mathbf{M}=[m_{\mathrm{text}},m_{\mathrm{url}},m_{\mathrm{image}},m_{\mathrm{code}}]\in {\{0,1\}}^4$$

(1)

where each $m_i$ indicates the presence (1) or absence (0) of a specific modality.

3.3.2. Feature Importance Assessment and Hierarchical Partitioning

We assess the importance of each original semantic feature (e.g., urgency_level, domain_length) using Mutual Information (MI) between the feature $f_j$ and the phishing label y:

$$\mathrm{MI}(f_j;y)=\sum _{f_j\in \mathcal{F}}\sum _{y\in \{0,1\}}p(f_j,y)log\left({\displaystyle \frac{p(f_j,y)}{p\left(f_j\right)p\left(y\right)}}\right)$$

(2)

Features are then partitioned into three layers based on their MI scores:

• Core Layer (${\mathcal{F}}_c$): Features with importance ≥ 90th percentile ($Q_{90}$). These are deemed essential for detection.
• Auxiliary Layer (${\mathcal{F}}_a$): Features with importance between the 70th ($Q_{70}$) and 90th percentile. These provide supplementary evidence.
• Weakly-Associated Layer (${\mathcal{F}}_w$): Features below the 70th percentile. These are considered noisy or redundant and are discarded.

Let $\mathcal{F}={\mathcal{F}}_c\cup {\mathcal{F}}_a\cup {\mathcal{F}}_w$ denote the full set of semantic features.

3.3.3. Intra-Modal Redundancy Compression

Even within the core and auxiliary layers, features from the same modality may be correlated. To reduce intra-modal redundancy, we apply Principal Component Analysis (PCA) separately to the features of each modality within ${\mathcal{F}}_c$ and ${\mathcal{F}}_a$. For each modality $M_k$, we compute:

$${\mathbf{Z}}_k={\mathrm{PCA}}_{\ge 95\%}\left({\mathbf{X}}_{M_k}\right)$$

(3)

where ${\mathbf{X}}_{M_k}$ is the submatrix of features from modality $M_k$, and ${\mathbf{Z}}_k$ contains the principal components that explain at least 95% of the variance.

3.3.4. Dynamic Feature Integration

We first train a preliminary model using only the fused core features (${\mathbf{X}}_{\mathrm{fused\_core}}$). If the validation AUC is below a threshold $\tau =0.98$, we dynamically integrate the auxiliary features (${\mathbf{X}}_{\mathrm{fused\_aux}}$) to boost performance:

$${\mathbf{X}}_{\mathrm{integrated}}=\left\{\begin{array}{cc}{\mathbf{X}}_{\mathrm{fused\_core}},\hfill & \mathrm{if}{\mathrm{AUC}}_{\mathrm{val}}\ge \tau ,\hfill \\ {\mathbf{X}}_{\mathrm{fused\_core}},{\mathbf{X}}_{\mathrm{fused\_aux}}],\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

This “simple-first, enhance-on-demand” strategy ensures efficiency without sacrificing accuracy.

3.3.5. Hierarchical Weighted Training

Finally, we train the primary classifier on the integrated feature set. To emphasize the importance of core features, we apply a feature weighting scheme:

$$w_j=\left\{\begin{array}{cc}1.2,\hfill & \mathrm{if}{f}_j\in {\mathcal{F}}_c,\hfill \\ 1.0,\hfill & \mathrm{if}{f}_j\in {\mathcal{F}}_a.\hfill \end{array}\right.$$

(5)

Additionally, we apply sample weighting to address residual class imbalance:

$$v_i=\left\{\begin{array}{cc}1.2,\hfill & \mathrm{if}{y}_i=1\left(\mathrm{phishing}\right),\hfill \\ 1.0,\hfill & \mathrm{if}{y}_i=0\left(\mathrm{benign}\right).\hfill \end{array}\right.$$

(6)

This final step ensures the model prioritizes the most discriminative signals during training. The detailed pseudocode of the SAHF-PD algorithm is provided in Appendix C.

4. Experimental Design and Results Analysis

4.1. Experimental Objectives

This study aims to systematically evaluate the effectiveness of the proposed Semantic-Aware Hierarchical Fusion framework for Phishing Detection (SAHF-PD). The specific objectives are: (1) to validate that SAHF-PD can significantly reduce model complexity and training time while maintaining or even improving detection accuracy; (2) to conduct ablation studies to deeply analyze the contribution and complementarity of each modality; and (3) to comprehensively assess the model’s interpretability, providing decision support for security operations.

4.2. Experimental Setup

To ensure fair comparison and reproducibility, all experiments were conducted on a high-performance server with the hardware and software configurations detailed in

Table 2. Hardware and Software Configuration of the Experimental Platform.

Component	Specifications
CPU	Intel Xeon Platinum 8369B (32 cores/64 threads) @ 2.70 GHz
RAM	125 GB DDR4
GPU	2 × NVIDIA Tesla T4 (16 GB GDDR6 VRAM each)
Operating System	Ubuntu 22.04 LTS
Programming Language	Python 3.10.16
GPU Drivers & CUDA	NVIDIA Driver 535.230.02, CUDA Toolkit 12.2
Key Libraries	scikit-learn 1.2.2, XGBoost 3.0.5, pandas 2.2.3, NumPy 1.26.4

.

All models were trained using identical random seeds for reproducibility, and hyperparameter tuning was performed via grid search on the validation set.

4.3. Dataset Construction

To support the development and evaluation of end-to-end multimodal phishing detection models, we constructed a high-quality, structured multimodal feature dataset, named PhishMMF (Phishing Multimodal Features). The dataset was built upon authoritative open-source corpora and processed through a systematic pipeline of feature extraction, quality verification, and post-processing to ensure its representativeness, consistency, and usability.

(1)

Data Sourcing and Initial Pool: The initial dataset comprised 12,000 email samples, evenly split between 6000 phishing emails and 6000 legitimate emails, ensuring balanced class distribution from the outset. The phishing samples were sourced from the DataCon 2023 and PhishPot platforms, covering a wide range of typical attack scenarios. The legitimate emails were drawn from the CEAS 2008 and SpamAssassin corpora and were manually sampled and verified to be free of malicious content.

(2)

LLM-Based Feature Extraction and Quality Control: Each email sample underwent multimodal feature extraction using the LLM-driven framework described in Section 3.2. To guarantee the quality and structural integrity of the extracted features, a rigorous validation process was implemented:

• Structural Validation: All extracted feature data were validated against a predefined JSON Schema to ensure correct data types, value ranges, and mandatory field presence.
• Automated Retry Mechanism: Samples with formatting errors or missing fields were automatically flagged and subjected to a re-extraction process. Those that failed after retries were marked for manual review. This quality control phase resulted in the removal of 164 samples with complete feature extraction failure, leaving 11,836 samples for vectorization.

(3)

Vectorization, Missingness Analysis, and Final Curation: After vectorization, a detailed analysis of modality completeness was conducted. The results revealed significant differences in extraction success rates across modalities. Text features exhibited the highest stability due to the universal presence of textual content in emails. In contrast, URL, image, and website code modalities showed higher missingness rates. This is primarily attributed to three factors: (i) structural absence (e.g., emails without URLs), (ii) dependency on external resources (e.g., OSINT queries, image loading, code crawling), which are susceptible to network and availability issues, and (iii) the intentional design of phishing attacks, where malicious links are often quickly taken down or blocked, rendering them inaccessible.

To ensure data quality, the following final curation steps were applied:

• High-Noise Sample Removal: 13 extreme cases (0.11%) with three or more modalities as zero vectors were identified and removed.
• Label Balancing: Undersampling was performed on the phishing class to achieve perfect class balance.
• Modality Masking: A 4-bit binary Modality Mask was appended to each sample to explicitly indicate the presence or absence of valid features for each modality, enabling missingness-aware modeling.

The final PhishMMF dataset contains 11,672 high-quality samples (5836 phishing and 5836 legitimate emails), with a feature vector dimensionality of 232. The feature values exhibit a healthy distribution (mean = 0.065, std = 0.309, no NaN or infinite values), confirming the stability and reliability of the entire pipeline. This dataset, characterized by its high quality, consistency, and reproducibility, provides a solid foundation for training and evaluating our proposed framework.

4.4. Evaluation Metrics

The following metrics were used for comprehensive evaluation:

• AUC (Area Under the ROC Curve): Measures the overall discriminative capability across varying classification thresholds. A value closer to 1 indicates superior performance.
• Precision: Proportion of correctly identified phishing emails among all predicted positives.

  $$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

  (7)
• Recall (True Positive Rate): Proportion of actual phishing emails correctly detected.

  $$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

  (8)
• F1-Score: Harmonic mean of precision and recall, providing a balanced assessment of model performance.

  $$F_1=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

  (9)
• Confusion Matrix: Provides a detailed breakdown of classification outcomes, where $TP$, $TN$, $FP$, and $FN$ denote true positives, true negatives, false positives, and false negatives, respectively:

  $$\mathrm{CM}=\left[\begin{array}{cc}TN& FP\\ FN& TP\end{array}\right]$$

  (10)
• Training Time: Wall-clock time from initialization to convergence (in seconds), used to assess computational efficiency.
• Number of Features Used: The dimensionality of the final input space, reflecting model compactness and interpretability.

The dataset was randomly partitioned into an 80% training set and a 20% test set. To ensure result stability, a 5-fold cross-validation strategy was employed:

$$\begin{array}{cc}\hfill {\mu }_{\mathrm{metric}}& ={\displaystyle \frac{1}{5}}\sum _{i=1}^5{\mathrm{Metric}}_i\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\sigma }_{\mathrm{metric}}& =\sqrt{{\displaystyle \frac{1}{5}}\sum _{i=1}^5{({\mathrm{Metric}}_i-{\mu }_{\mathrm{metric}})}^2}\hfill \end{array}$$

(12)

where ${\mathrm{Metric}}_i$ denotes the evaluation score (e.g., accuracy, F1-score) obtained on the i-th fold. The mean ${\mu }_{\mathrm{metric}}$ reflects the overall performance of the model across all folds, while the standard deviation ${\sigma }_{\mathrm{metric}}$ quantifies the variability of the results, providing insight into the robustness and consistency of the model’s behavior under different data partitions.

4.5. Results and Analysis

4.5.1. Experimental Evaluation on PhishMMF

The PhishMMF dataset comprises multimodal features extracted from phishing emails using our proposed feature engineering pipeline, prior to the Semantic-Aware Hierarchical Fusion (SAHF) step. This design enables a fair ablation study: we evaluate standard machine learning models both on the raw multimodal features (baseline) and on the fused feature representation produced by SAHF.

Detection Performance: As shown in

Table 3. Comparison of model performance and training efficiency with and without Semantic-Aware Hierarchical Fusion (SAHF) across eight classifiers on PhishMMF.

Model	AUC	F1 Score	Precision	Recall	Train Time (s)
XGBoost	0.99921	0.98642	0.97729	0.99572	0.24
XGBoost+SAHF	0.99927	0.98728	0.97733	0.99743	0.19 ↓
LightGBM	0.99942	0.98684	0.97811	0.99572	0.27
LightGBM+SAHF	0.99886	0.98220	0.97150	0.99314	0.17 ↓
Gradient Boosting	0.99889	0.98220	0.9715	0.99314	12.59
Gradient Boosting+SAHF	0.99841	0.98266	0.96995	0.99572	11.72 ↓
SVM	0.99773	0.98096	0.96906	0.99314	11.76
SVM+SAHF	0.99654	0.97419	0.96237	0.98629	5.43 ↓
Random Forest	0.99537	0.96471	0.94641	0.98372	0.79
Random Forest+SAHF	0.99746	0.97016	0.95215	0.98886	0.60 ↓
Decision Tree	0.97620	0.95917	0.96207	0.95630	0.18
Decision Tree+SAHF	0.98797	0.96823	0.95729	0.97943	0.14 ↓
KNN	0.99116	0.96998	0.95743	0.98286	0.34
KNN+SAHF	0.98959	0.96839	0.95274	0.98458	0.35
Logistic Regression	0.99428	0.97210	0.95913	0.98543	4.18
Logistic Regression+SAHF	0.99070	0.96284	0.94921	0.97686	1.78 ↓

“+SAHF” denotes the inclusion of the Semantic-Aware Hierarchical Fusion component (i.e., the full SAHF-PD pipeline); models without this suffix use raw multimodal features without semantic-aware fusion. ↓ indicates reduced training time. Boldface values indicate improved performance when SAHF is applied.

, gradient boosting decision tree (GBDT) models—particularly XGBoost, LightGBM, and Gradient Boosting—achieve consistently strong performance on PhishMMF, with AUC scores exceeding 0.99 and F1 scores above 0.98 across most configurations. This confirms that the extracted multimodal features are highly discriminative and well-suited for structured learning algorithms. When equipped with the SAHF module, certain models exhibit modest but consistent improvements. For instance, XGBoost+SAHF achieves a marginal yet measurable gain over its baseline, with AUC increasing from 0.99921 to 0.99927 and F1 score rising from 0.98642 to 0.98728. Similarly, both Random Forest and Decision Tree show improvements across all reported metrics—AUC, F1 score, Precision, and Recall—after SAHF fusion. In contrast, LightGBM experiences slight performance degradation when SAHF is applied, with reductions in AUC, F1 score, and Recall, despite a notable reduction in training time (37%). This variation highlights that the impact of SAHF is model-dependent. Specifically, while some classifiers effectively leverage the semantic structure encoded in the fused features, others may be less aligned with the inductive bias introduced by the semantic-aware fusion process. Here, inductive bias refers to the inherent assumptions a learning algorithm makes to generalize from limited data—for example, decision trees prefer sparse, interpretable splits, whereas gradient boosting methods implicitly prioritize feature interactions. The SAHF module, by design, emphasizes semantically coherent cross-modal patterns (e.g., aligning suspicious URL signals with phishing-related textual cues) and suppresses redundant or noisy dimensions. Models whose internal optimization mechanisms naturally resonate with this structured representation—such as XGBoost—tend to benefit more, whereas those relying heavily on fine-grained raw statistics may lose discriminative signals during fusion.

Computational Efficiency: A key advantage of SAHF is its substantial reduction in computational overhead. By hierarchically fusing and compressing the original 228-dimensional feature space into a compact 56-dimensional representation (a 75.4% reduction, as detailed in

Table 4. Statistics of feature fusion across different layers.

Layer	Modality	Num. Features	Dim.	Fused Dim.
Core Layer	text_features	8	48	28
	url_intelligence_features	1	1	1
Auxiliary Layer	text_features	8	8	5
	url_intelligence_features	8	21	3
	image_features	1	32	19
Weak Layer	all	59	118	0
Total	—	85	228	56
Compression Rate	—	—	—	75.4%

Note: “Num. Features” = Number of Features; “Dim.” = Feature Dimension; “Fused Dim.” = Feature Dimension After Fusion.

), SAHF alleviates the computational burden of training without modifying the base classifier architectures. Consequently, the average training time across all eight classifiers is reduced by 43.7%, with Logistic Regression achieving the largest speedup (57.4% reduction, from 4.18 s to 1.78 s). Importantly, this efficiency gain comes with minimal performance degradation. For instance, SVM and Logistic Regression experience only minor drops in AUC (<0.004) and F1 score (<1%), offering an excellent cost-performance trade-off for real-world deployment where training latency matters. KNN, as expected, shows negligible change in both performance and training time due to its inherent insensitivity to feature dimensionality.

Overall, the improved performance of XGBoost and other models under SAHF indicates that the fusion process effectively preserves or amplifies discriminative signals while suppressing noise. Notably, all models that benefit from SAHF are non-linear, ensemble, or hierarchical learners, whereas linear or distance-based models tend to degrade—hinting at a compatibility between SAHF’s semantic fusion and models capable of capturing complex feature interactions.

4.5.2. In-Domain Comparison with Modern End-to-End Baselines

To validate the effectiveness of our LLM-derived structured feature representation, we compare SAHF-PD against strong end-to-end deep learning baselines that operate directly on raw inputs—without any handcrafted or LLM-guided feature engineering.

Specifically, we implement two representative modern architectures:

• DeBERTa-v3: Fine-tuned on raw email body text from PhishMMF for binary classification.
• CLIP (ViT-L/14): Fine-tuned on paired email screenshots and associated textual prompts from PhishMMF, using contrastive learning followed by a linear probe for classification.

All models are trained and evaluated under identical data splits within PhishMMF. As shown in

Table 5. End-to-End Deep Learning Baselines vs. Structured LLM Features on PhishMMF.

Model	Input Modality	AUC	F1	Precision	Recall
SAHF-PD (Ours)	Structured LLM features	0.99927	0.98728	0.97733	0.99743
DeBERTa-v3	Raw email text	0.99510	0.97240	0.96810	0.97670
CLIP (ViT-L/14)	Screenshot + text prompt	0.99130	0.95870	0.94250	0.97520

All models use the same train/validation/test splits.

, SAHF-PD (XGBoost on structured features) achieves an AUC of 0.99927 and F1-score of 0.98728, outperforming both DeBERTa-v3 and CLIP. This demonstrates that the semantic abstraction provided by our hierarchical LLM-guided feature pipeline yields more discriminative and noise-resilient signals than direct modeling of raw modalities, even when using powerful foundation models.

4.5.3. External Generalization on Public Email Datasets

To rigorously assess the generalizability of SAHF-PD beyond the PhishMMF dataset, we perform external evaluation using publicly available email corpora. Specifically, we construct an external test set that includes 300 phishing emails from the Nazario corpus, 300 phishing emails from the TREC-2006 Spam Corpus, and 600 legitimate emails from TREC-2006. This composition reflects a realistic distribution of phishing and benign messages while encompassing diverse origins and temporal contexts.

The model adopted for this evaluation corresponds to our best-performing configuration, which was trained exclusively on PhishMMF using the optimal XGBoost classifier. Without any fine-tuning, retraining, or domain adaptation, this setup supports a rigorous zero-shot evaluation of cross-dataset generalization capability.

As reported in

Table 6. In-Domain and Out-of-Domain Performance of the SAHF-PD Model Trained on PhishMMF.

Dataset	AUC	F1	Precision	Recall
PhishMMF (in-domain)	0.99927	0.98728	0.97733	0.99743
External (Nazario + TREC-06)	0.99743	0.97890	0.99145	0.96667

, the model achieves an AUC of 0.99743 and an F1-score of 0.97890 on the external test set, with precision of 0.99145 and recall of 0.96667. Although a slight performance degradation is observed relative to the in-domain PhishMMF results, the model maintains high detection efficacy across unseen data sources. These results confirm that the semantic representation learned through our LLM-guided hierarchical feature extraction framework exhibits strong transferability and robustness in real-world settings.

4.5.4. Ablation Study on Modality Contributions in SAHF-PD

The SAHF-PD framework integrates four complementary modalities—Text, Url, Image, and Code—as the foundation of its multi-modal feature extraction pipeline. To validate the necessity of this specific modality design, we conduct a comprehensive ablation study across eight classifiers, systematically removing each modality while keeping all other components of SAHF-PD unchanged.

As shown in Figure 2, Figure 3, Figure 4, and

Table 7. Ablation study results: AUC and F1-score across different modality combinations for all eight models.

Model	Full		−Text		−URL		−Image		−Code
	AUC	F1	AUC	F1	AUC	F1	AUC	F1	AUC	F1
Logistic Regression	0.99487	0.97129	0.96024	0.91765	0.96940	0.93620	0.99321	0.96341	0.99471	0.97050
Decision Tree	0.97961	0.96620	0.96157	0.91799	0.96222	0.93982	0.97445	0.96109	0.97804	0.96498
Random Forest	0.99502	0.96381	0.98702	0.95503	0.98038	0.93113	0.99394	0.95708	0.99520	0.96481
Gradient Boosting	0.99896	0.98686	0.99350	0.97558	0.98979	0.95958	0.99886	0.98771	0.99889	0.98712
KNN	0.98778	0.96679	0.97798	0.95350	0.97662	0.95254	0.98739	0.96826	0.98741	0.96716
SVM	0.99739	0.97954	0.97986	0.94842	0.98611	0.95332	0.99598	0.97367	0.99727	0.97912
XGBoost	0.99912	0.98839	0.99368	0.97462	0.99037	0.96033	0.99900	0.98839	0.99914	0.98908
LightGBM	0.99912	0.98874	0.99369	0.97541	0.99086	0.96196	0.99905	0.98873	0.99904	0.98934

The Full column represents the performance with all four modalities. The best AUC and F1-score for each model (row) are shown in bold.

, the full four-modality configuration consistently achieves the highest or near-highest performance for nearly every model. Critically, removing any single modality generally leads to a consistent decline in AUC, with the average drop ranging from 0.0012 (Code) to 0.0062 (Text) across models, demonstrating that each channel contributes unique and non-redundant signals to phishing detection. For example, omitting the Text modality reduces XGBoost’s AUC by 0.00544, highlighting its role in capturing semantic deception cues; removing Image features impairs visual spoofing detection; excluding URL features weakens external reputation awareness; and discarding Code features limits insight into malicious web payloads.

Notably, a few exceptions exist: Random Forest and XGBoost exhibit marginally higher AUC when the Code modality is removed. This suggests that for certain high-capacity models, the raw code features may introduce minor noise or redundancy that slightly offsets their discriminative value—a phenomenon also observed in other multimodal systems [17]. Nevertheless, no ablated configuration consistently outperforms the full multimodal setting across all models, as visually confirmed by the heatmap in Figure 4.

Collectively, these results demonstrate that the four modalities are not redundant but form a mutually complementary sensing system—precisely what enables SAHF-PD to achieve both high accuracy and robustness. The removal of any component creates a partial “perception blind spot,” weakening the overall robustness of phishing detection.

4.5.5. Interpretability Analysis of SAHF-PD

SAHF-PD enhances model interpretability without compromising performance. Its semantic-aware hierarchical fusion mechanism categorizes features by discriminative power, offering transparency into the decision-making process.

To ensure this interpretability is built on a sound feature foundation, we validate the UnifiedPhishingFeatures schema at the atomic level. As shown in Figure 5, the top 30 atomic features exhibit high mutual information with the phishing label, confirming their strong discriminative capability. Moreover, the correlation matrix in Figure 6 reveals that most pairwise correlations are weak, indicating minimal redundancy among the 85 atomic features. This confirms that our schema is both informative and non-redundant, providing a robust basis for downstream modeling.

Building on this validated set, SAHF-PD applies hierarchical fusion to produce semantically meaningful predictions. Figure 7 shows that the final model is dominated by text- and URL-based features. For instance, contains_phishing_call_to_action and sender_reputation_score act as primary decision drivers, while auxiliary features like sentiment_score and domain_similarity_to_known_brands provide contextual support in ambiguous cases. Although Figure 5 and Figure 7 highlight similar top features, they reflect different stages: the former measures pre-fusion intrinsic informativeness, while the latter measures captures post-fusion model contributions.

This multi-layered analysis, spanning from atomic validation to high-level interpretation, ensures that SAHF-PD is both accurate and interpretable. Security analysts can trace predictions to specific, interpretable features, enabling effective auditing and forensic investigation.

Ablation studies further confirm the dominance of text and URL modalities: removing either reduces the XGBoost model’s AUC by 0.00544 and F1-score by 0.01377 (Table 7). This aligns with Figure 7, where text- and URL-related features constitute over 90% of the top 20 most discriminative features.

By explicitly linking predictions to semantically meaningful evidence, SAHF-PD transforms from a “black box” into a transparent and accountable component of security operations, where explainability is essential for forensic analysis and real-world adoption.

4.5.6. Sensitivity Analysis of MI-Based Feature Partitioning and PCA Variance Retention

To rigorously justify our choice of the 70th and 90th percentiles for partitioning semantic features into Core, Auxiliary, and Weakly-Associated layers as detailed in Section 3.3.2, we conduct a comprehensive sensitivity analysis across multiple threshold configurations. In our hierarchical feature encoding pipeline, features assigned to the Core and Auxiliary layers are compressed and encoded into the final representation, while those in the Weakly-Associated layer are discarded. Consequently, different threshold pairs directly influence both the discriminative capacity and the dimensionality of the resulting feature vector.

We evaluated nine representative threshold combinations by varying the lower bound from Q60 to Q80 and the upper bound from Q80 to Q95, and further reported the corresponding phishing detection performance metrics as well as the dimensionality compression ratio. As summarized in

Table 8. Detection performance and dimensionality compression under different MI-based feature partitioning thresholds.

Threshold (Upper/Lower)	AUC	F1	Precision	Recall	Compression Ratio
$Q_{95}$/$Q_{80}$	0.99634	0.97497	0.96555	0.98458	42.0%
$Q_{90}$/$Q_{80}$	0.99267	0.96102	0.95054	0.97172	38.6%
$Q_{95}$/$Q_{70}$	0.99846	0.98391	0.97238	0.99572	51.8%
$Q_{90}$/$Q_{70}$	0.99927	0.98728	0.97733	0.99743	75.4%
$Q_{85}$/$Q_{70}$	0.99883	0.98515	0.97563	0.99486	49.1%
$Q_{95}$/$Q_{60}$	0.99851	0.98557	0.97645	0.99486	50.8%
$Q_{90}$/$Q_{60}$	0.99911	0.98643	0.97649	0.99657	47.5%
$Q_{85}$/$Q_{60}$	0.99873	0.98515	0.97563	0.99486	49.2%
$Q_{80}$/$Q_{60}$	0.99607	0.97128	0.95754	0.98543	49.2%

Note: Compression ratio is defined as the dimensionality of the final encoded feature vector divided by that of the original full feature vector, expressed as a percentage. The $Q_{90}$/$Q_{70}$ configuration corresponds to our original design.

, the Q90/Q70 configuration achieves the best overall performance, yielding an F1-score of 0.9873, an AUC of 0.9993, and a compression ratio of 75.4%. This result demonstrates that retaining features with mutual information above the 70th percentile while designating those at or above the 90th percentile as core features effectively balances model expressiveness against noise suppression.

Having established the soundness of our mutual information–based feature partitioning strategy, we proceed to examine intra-modal redundancy compression via PCA, the subsequent stage in our hierarchical encoding pipeline. In SAHF, intra-modal PCA aims to eliminate redundant signals within each modality while preserving discriminative information critical for phishing detection. The variance retention threshold thus governs the trade-off between compression efficiency and detection performance. To validate the choice of a 95% cutoff, we conduct a systematic sensitivity analysis across three representative thresholds: 90%, 95%, and 99%.

We tested each cutoff configuration on the PhishMMF dataset using the XGBoost classifier (consistent with the primary experimental setup), and reported key metrics including AUC, F1-score, Precision, Recall, and feature dimensionality compression ratio. The results are summarized in

Table 9. Detection performance and dimensionality compression under different PCA variance cutoffs.

PCA Variance Cutoff	AUC	F1-Score	Precision	Recall	Compression Ratio
90%	0.99919	0.98388	0.97397	0.99400	64.5%
95%	0.99927	0.98728	0.97733	0.99743	75.4%
99%	0.99897	0.98516	0.97483	0.99572	26.4%

Note: The 95% cutoff corresponds to our original design, balancing signal preservation and compression efficiency.

.

As shown in Table 9, the 95% PCA variance cutoff achieves the optimal balance between detection performance and compression efficiency, it yields the highest AUC, F1-score, Precision, and Recall among the three configurations, while attaining a substantial compression ratio of 75.4%. Compared to the 90% cutoff, it preserves critical discriminative signals to avoid performance loss; compared to the 99% cutoff, it effectively eliminates redundant noise and achieves much higher compression efficiency without sacrificing accuracy. This fully validates that the 95% variance cutoff is a rigorously justified choice, not an ad hoc setting, as it precisely meets the dual requirements of high detection accuracy and computational efficiency for phishing detection systems.

5. Discussion

The results in Section 4 indicate that the proposed SAHF-PD framework achieves high performance in phishing email detection, demonstrating strong accuracy, improved computational efficiency, and enhanced model interpretability. This section discusses the implications of these findings in the context of existing research, the role of the introduced dataset, and outlines the current limitations and potential future directions.

Reconciling Performance, Efficiency, and Interpretability: A persistent challenge in AI-based cybersecurity is the balance between detection accuracy, computational cost, and model transparency. Many high-accuracy models, particularly deep neural networks, operate as opaque systems and can be resource-intensive. The SAHF-PD framework aims to address this challenge by integrating a semantic-aware hierarchical fusion mechanism. This mechanism functions by identifying and prioritizing features deemed most relevant to the phishing detection task, while de-emphasizing less informative or redundant ones. This process contributes to the observed performance gains by focusing the model on salient signals, reduces computational load through effective feature dimensionality management, and improves interpretability by providing a structured view of feature contributions. This integrated approach suggests a viable path towards developing detection systems that are both effective and amenable to scrutiny.

Advancing the State of the Art in Multi-Modal Detection: The design of SAHF-PD responds to limitations observed in prior work. Compared to rule-based systems or traditional machine learning models that rely on shallow syntactic features, SAHF-PD leverages Large Language Models (LLMs) to perform deeper semantic analysis of email content, enabling the identification of complex social engineering cues that are difficult to capture with simple pattern matching. Furthermore, the hierarchical fusion strategy contrasts with common concatenation or black-box fusion methods by explicitly modeling the relative importance of different features and modalities. This provides a more transparent decision process. A key aspect of our methodology is the use of LLMs not as end-to-end classifiers, but as specialized components for structured feature extraction. This “LLM-as-a-feature-extractor” approach seeks to harness the reasoning capabilities of LLMs while maintaining compatibility with traditional, interpretable machine learning classifiers, promoting a modular system architecture.

A Resource for the Community: The research community has identified a need for standardized, high-quality datasets to advance multi-modal phishing detection. To support reproducible research, we constructed and publicly released the PhishMMF dataset. It consists of 11,672 phishing and legitimate email samples, with a focus on providing structured, multi-modal features that have been systematically extracted and human-verified. The dataset includes features derived from email text, OSINT data, website screenshots, and HTML source code, all mapped to a unified schema (UnifiedPhishingFeatures). By making PhishMMF available, we aim to provide a common benchmark that can facilitate more consistent evaluation and comparison of future detection algorithms, thereby reducing the effort required for data collection and preprocessing in subsequent studies.

Limitations and Future Work: Several limitations of the current study should be acknowledged. First, the framework depends on external services (e.g., VirusTotal) and webpage rendering to extract certain features, which introduces non-negligible latency and poses challenges for deployment in real-time email filtering systems. In particular, these dependencies are incompatible with the sub-second response requirements of high-throughput email gateways. To address this limitation, we propose a two-stage deployment strategy: (1) a lightweight triage stage using only textual and header features to fast-track benign emails; and (2) a deep inspection stage that activates the full multi-modal pipeline only for emails that, during the triage stage, are identified as containing lightweight yet high-risk indicators based solely on local analysis of email text and headers. These indicators include the presence of external URLs, domain mismatches between sender and recipient, urgent or action-oriented phrasing, and matches against a curated set of known phishing keywords.Critically, our SAHF architecture is designed to operate robustly under missing modalities—when OSINT or screenshots are unavailable, it falls back to core textual features without catastrophic performance loss. Future work will formalize this trade-off between coverage and throughput and explore caching, asynchronous execution, and browserless HTML analysis to further improve deployability. Second, while leveraging specialized large language models (LLMs) for semantic feature extraction proves effective, it incurs non-negligible computational overhead. Investigating model compression, knowledge distillation, or quantization techniques to develop more efficient LLM-based extractors represents a promising direction. Third, the UnifiedPhishingFeatures schema is currently static, designed based on known phishing tactics. To ensure long-term relevance, future efforts should focus on building an adaptive schema capable of evolving through continuous learning as new attack vectors emerge. Fourth, although the Semantic-Aware Hierarchical Fusion (SAHF) module generally enhances classification robustness, its effectiveness varies across different base learners—e.g., yielding significant gains for XGBoost but modest or even slightly negative impacts for LightGBM. This suggests that fusion strategies could be better optimized by tailoring them to the inductive biases of specific classifier architectures. Finally, the generalizability of the SAHF-PD framework beyond email-based phishing detection remains an open question. Extending it to other cyber-threat domains—such as malicious content detection in social media posts or SMS messages—would be a valuable avenue for future research.

6. Conclusions

This paper has presented SAHF-PD, a framework for phishing email detection that combines Large Language Models (LLMs) with a semantic-aware hierarchical fusion mechanism. To address the non-structured nature of LLM outputs, we propose a strategy involving specialized LLMs, customized prompts, and a unified template to generate structured and interpretable feature vectors from four modalities: email text, url-derived OSINT, screenshots, and HTML code.

A hierarchical fusion architecture is introduced to organize these features into core, auxiliary, and weakly-associated categories based on their semantic relevance. This structure enables dynamic feature weighting and redundancy reduction, contributing to the framework’s high detection performance and improved interpretability. To support this research and promote reproducibility, we developed the PhishMMF dataset, which contains 11,672 samples with structured, multi-modal features that have been human-verified.

Experiments with eight classifiers show that SAHF-PD achieves exceptional performance: XGBoost with SAHF attains an AUC of 0.99927 and F1-score of 0.98728—outperforming the same model on the original features—while compressing the 228-dimensional feature space to just 56 dimensions (75.4% reduction), cutting average training time by 43.7% with negligible accuracy loss.

In summary, SAHF-PD presents a method for integrating deep semantic analysis with interpretable machine learning for phishing detection. By using large language models as structured feature extractors and incorporating a semantic-aware hierarchical fusion mechanism, our framework offers a modular and extensible architecture for multi-modal threat analysis. The public release of the PhishMMF dataset further supports reproducible research and community-driven advances in this domain. Future work will focus on optimizing the pipeline for real-time deployment and exploring its application to other cyber threat detection tasks.