A Fusion-Based Approach with Bayes and DeBERTa for Efficient and Robust Spam Detection

Abstract

Spam emails pose ongoing risks to digital security, including data breaches, privacy violations, and financial losses. Addressing the limitations of traditional detection systems in terms of accuracy, adaptability, and resilience remains a significant challenge. In this paper, we propose a hybrid spam detection framework that integrates a classical multinomial naive Bayes classifier with a pre-trained large language model, DeBERTa. The framework employs a weighted probability fusion strategy to combine the strengths of both models—lexical pattern recognition and deep semantic understanding—into a unified decision process. We evaluate the proposed method on a widely used spam dataset. Experimental results demonstrate that the hybrid model achieves superior performance in terms of accuracy and robustness when compared with other classifiers. The findings support the effectiveness of hybrid modeling in advancing spam detection techniques.

1. Introduction

Spam emails have been a persistent and evolving challenge in electronic communication [1,2,3,4,5], posing severe risks such as personal privacy breaches, phishing attacks targeting financial credentials, corporate data leaks, and the spread of malicious links and malware. According to recent reports, billions of spam emails are sent daily, accounting for more than 45% of global email traffic, severely degrading user experience and imposing significant costs on organizations in terms of cybersecurity countermeasures [6]. Consequently, robust and accurate spam detection systems have become an indispensable component of modern email security infrastructures [7,8].

Conventionally, the problem of spam detection has been predominantly addressed using statistical and shallow machine learning techniques. Early approaches relied on rule-based systems and keyword filtering, which, while simple, proved brittle in the face of evolving spam tactics [9,10]. More advanced methods adopted probabilistic models such as naive Bayes (NB) classifiers, which leverage the Bayes theorem to estimate the likelihood of a message being spam based on word frequency distributions. For example, ref. [11] demonstrated one of the earliest successful applications of NB to spam filtering. Support vector machines (SVMs), known for their ability to handle high-dimensional sparse features, also gained popularity in text classification tasks, including spam detection, due to their margin-maximization principle. A common pipeline involved converting raw email texts into numerical features using term frequency-inverse document frequency (TF-IDF) representations [12,13,14], followed by classification through these statistical models. Despite their ease of deployment and low computational cost, such methods often struggle with evolving spam strategies, such as obfuscated language, image-based spam, or contextually ambiguous messages, primarily because they fail to capture deeper semantic relationships or sequential dependencies within text data.

In recent years, natural language processing (NLP) has undergone a paradigm shift with the advent of deep learning [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31], particularly with the development of large language models (LLMs). Transformer-based architectures, introduced by [32], have revolutionized NLP by enabling models to attend to long-range dependencies within sequences [33,34,35,36,37,38,39,40,41,42]. Landmark models like BERT [43] introduced bidirectional contextual representations, excelling in a wide range of language understanding tasks such as sentiment analysis, question answering, and named entity recognition. Following this trend, GPT-3 [44], with its 175 billion parameters, demonstrated impressive few-shot learning capabilities, allowing models to generalize to unseen tasks with minimal supervision. DeBERTa [45] further enhanced these capabilities by disentangling content and positional encoding, resulting in improved language modeling performance.

While LLMs have achieved state-of-the-art performance on many academic benchmarks, their application in operational spam detection systems remains relatively limited. This is primarily due to two factors. First, the deployment of LLMs such as GPT-3 or DeBERTa introduces substantial computational overhead, requiring high memory footprints and specialized hardware (e.g., GPUs), which may not be feasible for latency-sensitive applications like email filtering [46,47]. Second, spam detection datasets, such as the SMS Spam Collection or Enron Email Corpus, are typically small-scale and feature relatively simplistic linguistic structures compared to complex tasks like open-domain question answering or reasoning. As a result, organizations often default to lightweight traditional models to balance performance and efficiency [48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65]. Nonetheless, recent research suggests that LLMs, when fine-tuned appropriately, can offer significant improvements in robustness, particularly in detecting adversarial spam, spam with domain shifts, or messages employing sophisticated evasion techniques.

In this work, we explore the complementary strengths of classical machine learning models and LLMs for spam classification. We design a two-branch hybrid architecture where the first branch employs a lightweight multinomial naive Bayes classifier trained on TF-IDF features, efficiently capturing surface-level lexical patterns and common spam keywords (Figure 1). In parallel, the second branch leverages DeBERTa, a state-of-the-art LLM known for its disentangled attention mechanism, to extract deeper semantic and contextual features from messages. To effectively balance computational efficiency with classification accuracy, we adopt a weighted probability fusion strategy that combines the prediction outputs from both models into a final decision.

We validate our approach on the SMS Spam Collection dataset, a widely used spam detection benchmark. Experimental results show that our hybrid model outperforms both individual components, achieving higher accuracy, precision, and recall compared to standalone naive Bayes and DeBERTa models. Moreover, the proposed method demonstrates strong generalization while maintaining practical inference speed, making it a promising solution for real-world spam filtering applications where both performance and efficiency are critical.

Our main contributions are as follows:

• We propose a hybrid spam detection framework that combines a classical multinomial naive Bayes model and DeBERTa through a weighted probability fusion strategy.
• This fusion strategy effectively balances computational efficiency and classification accuracy, enabling the system to perform well in both simple and complex cases.
• Extensive experiments on the SMS Spam Collection dataset demonstrate that our method outperforms both traditional machine learning baselines and standalone LLMs, achieving superior performance in accuracy, precision, and recall. Additionally, the method maintains practical inference efficiency, making it suitable for real-world deployment.

2. Related Work

2.1. Traditional Machine Learning for Spam Detection

Spam detection has been a well-studied and critical problem in the fields of natural language processing (NLP) and cybersecurity, as it directly impacts digital security and user experience. Early works on spam detection predominantly focused on rule-based systems and keyword filtering, where simple heuristics such as identifying specific keywords or phrases were used to filter out spam messages [66]. These systems were relatively straightforward but lacked the adaptability to address the evolving and increasingly sophisticated tactics employed by spammers. As a result, such methods proved to be brittle, especially when spam messages began to incorporate obfuscated language or unconventional formatting. With the advancement of machine learning techniques, more robust and statistically sound approaches began to gain popularity, particularly for their ability to adapt to evolving spam tactics. Among these, naive Bayes classifiers emerged as one of the most widely used baselines for spam classification. Their simplicity, interpretability, and relatively good performance on short texts such as emails and SMS messages made them particularly attractive [11]. Despite their effectiveness, these classifiers are still limited by their assumption of feature independence, which may not hold in the context of natural language, where word relationships and context are often important. In addition to naive Bayes, other shallow machine learning models, such as support vector machines (SVMs) [67] and logistic regression [68], were commonly adopted. These models, often used in conjunction with text representation methods like bag-of-words or TF-IDF, proved to be useful in capturing relevant patterns for spam detection. SVMs, in particular, gained popularity due to their ability to handle high-dimensional and sparse feature spaces effectively, making them well suited for text classification tasks.

2.2. LLMs for Spam Detection and Text Classification

Pre-trained language models (PLMs) have been widely used for various natural language processing (NLP) tasks [69], but their application to spam detection remains relatively under-explored. Early spam detection methods relied on simpler models, but as NLP advanced, more powerful models like BERT gained attention for their ability to capture complex language patterns. Recent research has fine-tuned BERT for SMS and email spam classification, showing significant improvements over traditional models such as naive Bayes and support vector machines [11]. However, the generalization of large models like BERT is limited by the small scale and simplicity of spam datasets, which are often not diverse enough to effectively fine-tune the models [66]. Without data augmentation or domain-specific tuning, these models can struggle to handle the varied nature of real-world spam.

The emergence of large language models (LLMs), such as GPT-3 [44], PaLM [70], and DeBERTa [45], has opened up new avenues for enhancing spam detection. These models are not only capable of understanding complex linguistic structures but also excel at reasoning and learning contextual relationships within text [67]. Their advanced attention mechanisms allow them to process long-range dependencies in text, making them highly effective at detecting subtle patterns often present in sophisticated spam messages, including those that may employ adversarial tactics like obfuscation, synonym replacement, or hidden meaning. Despite these promising capabilities, the application of LLMs in real-world spam detection systems faces significant challenges, particularly due to their high computational cost. Deploying such models in production systems requires substantial hardware resources, such as GPUs, and incurs high memory and processing costs. Additionally, the large size of these models can lead to latency issues, which is a critical concern in time-sensitive applications such as email filtering [68].

To overcome these challenges, hybrid strategies that combine the strengths of LLMs with the efficiency of traditional lightweight models, such as naive Bayes or support vector machines, have been explored in related domains. For example, in phishing detection [31] and toxic comment classification [71], hybrid approaches have shown promise by leveraging the fast, low-cost inference of traditional models while also incorporating the powerful contextual understanding of LLMs. These hybrid models aim to balance the trade-off between performance and computational efficiency, providing a scalable solution that can be deployed in real-world applications. However, despite the success of hybrid strategies in other domains, their use in spam detection has been limited and remains an area that is still under-explored.

Our work contributes to this line of research by proposing a hybrid spam detection system that effectively combines the speed and simplicity of the naive Bayes classifier with the deep semantic understanding of the DeBERTa model. By leveraging the strengths of both models, our approach achieves an effective balance between high classification performance and computational efficiency, making it a promising solution for real-world spam filtering tasks.

3. Methodology

In this section, we introduce our hybrid spam detection framework, which integrates a classical multinomial naive Bayes (MNB) classifier with a large language model (LLM) based on DeBERTa. The framework combines the fast inference speed of MNB and the strong semantic understanding of LLMs through a weighted probability fusion strategy Algorithm 1.

Algorithm 1: Hybrid spam detection pipeline.

Require: Input message x Ensure: Predicted label $\widehat{y}\in \{\mathrm{spam},\mathrm{ham}\}$   1: // Stage 1: Preprocessing   2: $x_{\mathrm{clean}}\leftarrow$ CleanText $\left(x\right)$                      ▹ Lowercase, remove punctuation, stopwords   3: $\mathbf{v}\leftarrow$ TFIDF_Vectorize $\left(x_{\mathrm{clean}}\right)$   4: $\mathbf{t}\leftarrow$ Tokenize $\left(x\right)$                                                                         ▹ For DeBERTa input   5: // Stage 2: Classification   6: $P_{\mathrm{NB}}\leftarrow$ MNB.predict_proba $\left(\mathbf{v}\right)$   7: $P_{\mathrm{DeBERTa}}\leftarrow$ DeBERTa.predict_proba $\left(\mathbf{t}\right)$   8: // Stage 3: Weighted Fusion   9: $\alpha \leftarrow$ predefined weight factor (e.g., 0.5) 10: $P_{\mathrm{fused}}=\alpha ·P_{\mathrm{NB}}+(1-\alpha )·P_{\mathrm{DeBERTa}}$ 11: // Stage 4: Final Decision 12: $\widehat{y}\leftarrow argmax{P}_{\mathrm{fused}}$ 13: return  $\widehat{y}$

3.1. Hybrid Architecture Overview

As shown in Figure 2, the overall architecture consists of two parallel branches:

• Naive Bayes Branch: This branch utilizes a lightweight multinomial naive Bayes (MNB) classifier, which is trained on term frequency-inverse document frequency (TF-IDF) features extracted from the input text. The naive Bayes classifier focuses on identifying statistical patterns within the text, particularly the frequency of specific keywords or word combinations that are indicative of spam. By leveraging the probabilistic framework of naive Bayes, this model computes the likelihood that a given message belongs to a particular class (spam or ham) based on the occurrence of these features. This method is highly efficient and interpretable, making it suitable for quick classification tasks, especially when computational resources are limited.
• DeBERTa Branch: In this branch, we employ a fine-tuned DeBERTa model, which is a transformer-based model designed to capture deep contextual representations in text. DeBERTa enhances traditional BERT-like architectures by utilizing disentangled attention mechanisms, which separate the content and position information of tokens within a sequence. This allows the model to better understand the relationships and dependencies between words, even in complex contexts or long-range sequences. Fine-tuning the DeBERTa model on spam datasets enables it to learn nuanced patterns and semantic information that are crucial for distinguishing between spam and legitimate messages, especially in cases where the text is obfuscated or contains sophisticated language. This branch excels in capturing subtle and intricate contextual clues that simpler models may miss.

The outputs from these two branches are combined via weighted probability fusion to produce the final classification.

3.2. Naive Bayes Classifier

Given an input message x, we first extract its term frequency-inverse document frequency (TF-IDF) feature vector $\mathbf{v}=[v_1,v_2,\dots ,v_n]$, where each $v_i$ corresponds to the TF-IDF value of the i-th word in the input message. The TF-IDF representation captures the importance of each word in the context of the entire corpus, allowing the classifier to focus on words that are unique or rare, which are often indicative of spam content. The probability of the message x being classified as spam using the multinomial naive Bayes (MNB) classifier [11] is computed as follows:

$$log{P}_{\mathrm{NB}}(y\mid x)\propto logP\left(y\right)+\sum _{i=1}^n{v}_i·logP(w_i\mid y)$$

(1)

where $w_i$ represents the i-th word in the vocabulary, $v_i$ is the TF-IDF value of that word, and Z is a normalization constant to ensure the probabilities sum to one. This equation applies Bayes’ theorem, where the prior $P(y=\mathrm{spam})$ is the probability of a message being spam, and $P(w_i\mid y=\mathrm{spam})$ is the conditional probability of the word $w_i$ given that the message is spam. By multiplying the probabilities of all words raised to the power of their corresponding TF-IDF values, this classifier focuses on how specific words contribute to the likelihood of a message being spam based on observed word distributions.

3.3. DeBERTa Classifier

The second branch of the hybrid architecture uses the DeBERTa model to directly process the raw text input x. Unlike the MNB classifier, which operates on discrete word frequencies, DeBERTa utilizes deep contextual representations derived from the relationships between words in the text. This enables DeBERTa to capture complex semantics and contextual cues, making it highly effective for detecting spam messages that employ subtle obfuscation or ambiguous phrasing.

The model outputs a logits vector $\mathbf{l}=[l_{\mathrm{ham}},l_{\mathrm{spam}}]$, where each logit $l_k$ corresponds to the unnormalized score for class k, with $k\in \{\mathrm{ham},\mathrm{spam}\}$. These logits are converted into probabilities using the softmax function:

$$P_{\mathrm{DeBERTa}}(y\mid x)={\displaystyle \frac{e^{l_y}}{{\sum }_{k\in \{\mathrm{ham},\mathrm{spam}\}}{e}^{l_k}}}$$

(2)

Here, $P_{\mathrm{DeBERTa}}(y\mid x)$ denotes the predicted probability of class y. The softmax ensures that the probabilities across all classes sum to one, making the output interpretable as a confidence score. This probabilistic formulation is standard in neural classification tasks using transformer-based models such as BERT [43] and DeBERTa [45].

DeBERTa further improves language modeling by disentangling content and positional encodings, which enhances its ability to capture semantic dependencies in text. This property makes it particularly robust in identifying spam messages that may bypass traditional filters.

3.4. Weighted Probability Fusion

To balance efficiency and accuracy, we propose a weighted fusion of both classifiers. This fusion combines the probabilities from the naive Bayes (MNB) classifier and the DeBERTa model, ensuring that the strengths of each model are utilized in the final decision. The combined probability of the message x being classified as spam is given by the following equation:

$$\begin{array}{cc}\hfill P_{\mathrm{final}}(y=\mathrm{spam}\mid x)=& \alpha ·P_{\mathrm{NB}}(y=\mathrm{spam}\mid x)\hfill \\ \hfill & +(1-\alpha )·P_{\mathrm{DeBERTa}}(y=\mathrm{spam}\mid x)\hfill \end{array}$$

(3)

Here, $\alpha \in [0,1]$ is a parameter that controls the weight distribution between the two models. The value of $\alpha$ is determined through validation experiments to find the optimal balance that maximizes both performance and efficiency.

The final prediction $\widehat{y}$ is made based on a threshold at $0.5$. If the final combined probability for the message being spam is greater than or equal to $0.5$, the message is classified as spam; otherwise, it is classified as ham. The decision rule is expressed as follows:

$$\widehat{y}=\left\{\begin{array}{cc}\mathrm{spam},\hfill & \mathrm{if}{P}_{\mathrm{final}}\ge 0.5\hfill \\ \mathrm{ham},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

3.5. Overall Inference Algorithm

The complete inference process, from input message to final prediction, is summarized in the algorithm below. The algorithm outlines the sequential steps taken by the hybrid spam detection model to process an input message x, including feature extraction, classification using both the naive Bayes and DeBERTa models, and combining the results to produce the final classification decision. Each step is critical for ensuring that both models contribute their strengths to the overall prediction process.

The algorithm begins by extracting the TF-IDF features from the input message, which allows the naive Bayes classifier to evaluate the message based on its keyword frequencies. Simultaneously, the DeBERTa model processes the raw text input, leveraging its deep learning capabilities to understand the contextual and semantic relationships between words. The probabilities generated by both models are then fused using a weighted strategy, where the weight $\alpha$ determines the contribution of each model based on their relative strengths. This fusion allows the model to maintain high accuracy for complex messages, where DeBERTa excels, while also ensuring fast inference for simpler cases, where the naive Bayes model is more efficient.

This design ensures that the hybrid system achieves both high accuracy in complex cases, where understanding the context of the text is essential, and fast inference in simpler scenarios, where keyword-based classification can suffice. The naive Bayes model provides a quick and efficient classification, making it particularly suitable for processing large volumes of data in real time. Meanwhile, DeBERTa’s robust contextual understanding improves performance in more ambiguous or sophisticated spam messages.

The balance between the two models allows the hybrid approach to be both computationally efficient and highly effective, making it ideal for real-world spam filtering applications. This dual approach of combining a lightweight model for speed and a deep model for accuracy provides a scalable solution that can adapt to varying types of spam messages encountered in practice, where both performance and speed are critical.

4. Experiment

In this section, we present our experimental setup, evaluation metrics, baseline comparisons, and results to validate the effectiveness of our proposed hybrid spam detection framework. The experiments are designed to comprehensively evaluate the performance of our model under real-world conditions, using standard datasets and comparing our method with established baselines.

4.1. Dataset

We evaluate our method on the widely-used SMS Spam Collection dataset [11], a well-known benchmark in the spam detection community. This dataset consists of 5574 SMS messages that are labeled as either ham (legitimate) or spam. The dataset contains 4827 ham messages and 747 spam messages, leading to a class imbalance that mirrors real-world spam distributions, where spam messages are typically much fewer than legitimate ones. The imbalance presents a challenge for classification models, making it an ideal dataset for testing the effectiveness of our hybrid approach.

Each message in the dataset is labeled as “ham” if it is a legitimate message or “spam” if it is unsolicited or unwanted. The class imbalance is a common issue in spam detection tasks, as spam messages typically make up a small percentage of overall communication. This class imbalance can affect model performance, especially in terms of recall, and is important for evaluating how well our model generalizes to real-world scenarios, where spam detection systems must be able to identify the relatively rare spam messages among a larger set of legitimate messages.

We split the dataset into training and test sets, ensuring that the test set contains a representative distribution of both ham and spam messages, which allows us to evaluate the generalization capability of our hybrid spam detection framework.

4.2. Evaluation Metrics

To comprehensively assess model performance, we adopt four standard evaluation metrics, which are commonly used in classification tasks [72]. Each metric provides unique insight into the model’s strengths and weaknesses, helping to evaluate its effectiveness in different aspects of spam detection.

• Accuracy: Accuracy measures the overall percentage of correctly classified messages. It is defined as the ratio of the number of correct predictions (both spam and ham) to the total number of messages:

  $$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

  (5)

  where $TP$ is the number of true positives (correctly classified spam messages), $TN$ is the number of true negatives (correctly classified ham messages), $FP$ is the number of false positives (ham messages incorrectly classified as spam), and $FN$ is the number of false negatives (spam messages incorrectly classified as ham).
• Precision: Precision measures the proportion of true spam messages among all messages classified as spam. It is defined as follows:

  $$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

  (6)
  Precision focuses on the quality of spam predictions, and a high precision value indicates that the model does not misclassify ham messages as spam frequently.
• Recall: Recall (also known as sensitivity or true positive rate) measures the proportion of correctly identified spam messages among all actual spam messages. It is defined as follows:

  $$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

  (7)
  Recall highlights the model’s ability to identify all spam messages, and a high recall value indicates that the model minimizes false negatives, capturing as many spam messages as possible.
• F1-score: The F1-score is the harmonic mean of precision and recall, providing a balanced measure that considers both false positives and false negatives. It is defined as follows:

  $$\mathrm{F}1\mathrm{-score}=2·{\displaystyle \frac{\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

  (8)
  The F1-score is especially useful when there is an imbalance between precision and recall, as it combines both metrics into a single score that penalizes large discrepancies between them.

We also report the AUC (area under the ROC curve) score, which provides an overall assessment of the classifier’s ability to distinguish between spam and ham messages. The ROC curve plots the true positive rate (recall) against the false positive rate (1 − specificity), and the area under the curve (AUC) quantifies the classifier’s discriminative ability. The AUC score is defined as follows:

$$\mathrm{AUC}={\int }_0^1\mathrm{TPR}\left(f\right)d\mathrm{FPR}\left(f\right)$$

(9)

where $\mathrm{TPR}\left(f\right)$ is the true positive rate (recall) as a function of the decision threshold f, and $\mathrm{FPR}\left(f\right)$ is the false positive rate. A higher AUC value indicates better performance, with a value of 1 indicating perfect classification and a value of 0.5 indicating random guessing.

4.3. Baselines

We compare our hybrid approach against the following baselines to assess its performance in relation to standard methods in spam detection:

• Multinomial Naive Bayes (MNB): This baseline model uses the traditional multinomial naive Bayes classifier, which is trained on TF-IDF features. As one of the most popular and widely used algorithms for text classification tasks, naive Bayes operates by calculating the likelihood of a message belonging to the spam class based on the frequency of words. Despite its simplicity, MNB is highly efficient and interpretable, making it a strong baseline for comparison. However, it may struggle with more complex linguistic patterns or semantic relationships within the text, as it relies heavily on word frequencies and assumes feature independence.
• DeBERTa-base: DeBERTa (decoding-enhanced BERT with disentangled attention) is a more advanced deep learning model for natural language processing, built upon the transformer architecture. In this baseline, we use the pre-trained DeBERTa-base model, which has been fine-tuned on the same SMS spam dataset. DeBERTa improves upon BERT by incorporating disentangled attention mechanisms, which separate content and positional information for more accurate context understanding. This allows DeBERTa to capture deeper semantic relationships in text, making it particularly effective at detecting nuanced spam messages. However, DeBERTa requires more computational resources and may not be as efficient as simpler models in terms of inference time, especially when deployed in real-world spam detection systems.
• Our Hybrid Model: This is the proposed model, which combines the strengths of both MNB and DeBERTa through weighted probability fusion. The hybrid model integrates the fast and lightweight naive Bayes classifier with the powerful, context-aware DeBERTa model, offering a balance between computational efficiency and accuracy. By using a weighted fusion strategy, the hybrid model combines the predictions of the two classifiers to produce a final classification, leveraging MNB’s efficiency for simpler cases and DeBERTa’s robust contextual understanding for more complex scenarios. This fusion strategy ensures that the model performs well in both real-time applications with large message volumes and in cases where deeper semantic analysis is required to accurately classify spam messages.

4.4. Implementation Details

For the multinomial naive Bayes (MNB) classifier, we use the CountVectorizer and TfidfTransformer from scikit-learn to extract features from the input messages. The CountVectorizer converts the raw text into a term frequency (TF) representation, counting the occurrences of each word in the document, while the TfidfTransformer transforms the raw term frequencies into the term frequency-inverse document frequency (TF-IDF) representation, which scales the term frequencies based on the importance of words across the entire corpus. These two components are used together with the default settings, which makes the setup simple and efficient. This feature extraction process is ideal for the naive Bayes classifier, as it relies on the frequency of words to compute the probabilities for each class.

For the DeBERTa branch, we use the transformers library, which provides pre-trained models and tools for fine-tuning transformer-based architectures. Specifically, we initialize the model with the microsoft/deberta-base checkpoint, which is a pre-trained version of DeBERTa fine-tuned on a large corpus. DeBERTa introduces disentangled attention mechanisms, which allow the model to better capture the relationships between words and their positions within a sentence. We fine-tune the model on the SMS Spam Collection dataset for 15 epochs, using a batch size of 16. The AdamW optimizer, which is designed for transformer models, is used to minimize the loss function during training. The learning rate is adjusted using the default settings, ensuring effective training without overfitting.

The fusion weight $\alpha$ is set to 0.3 based on tuning experiments conducted on the validation set. This value gives a higher weight to the LLM (DeBERTa) branch, allowing it to contribute more to the final decision. At the same time, the naive Bayes classifier retains a significant role, preserving its efficiency for simpler cases. The choice of $\alpha =0.3$ was found to provide an optimal balance between the computational speed of naive Bayes and the contextual accuracy of DeBERTa, ensuring that the hybrid model performs well in both speed and accuracy across various types of spam messages.

To address the class imbalance in the SMS Spam Collection dataset, we applied class weighting during training. For the naive Bayes classifier, class priors were computed from the empirical label distribution in the training set. For the DeBERTa model, we employed a weighted cross-entropy loss, assigning a higher weight to the minority (spam) class to reduce its under-representation in the gradient updates. This approach helped to balance the learning process without altering the original data distribution.

4.5. Results and Discussion

Table 1. Performance comparison on SMS Spam Collection dataset.

Method	Accuracy	Precision	Recall	F1	AUC
Logistic Regression (TF-IDF)	95.8%	94.2%	92.5%	93.3%	97.9%
MNB (TF-IDF)	96.1%	93.7%	95.6%	94.6%	98.3%
SVM (TF-IDF)	96.5%	95.5%	94.1%	94.8%	98.5%
DistilBERT	96.9%	95.7%	94.8%	95.2%	98.5%
BERT-base	97.0%	95.9%	95.1%	95.5%	98.6%
RoBERTa-base	96.9%	96.0%	95.3%	95.6%	98.7%
DeBERTa	97.2%	96.1%	95.4%	95.7%	98.7%
Hybrid (Ours)	97.5%	96.8%	96.2%	96.5%	99.1%

summarizes the performance of all methods on the test set, and we can see the experimental results intuitively in Figure 3 and Figure 4.

Our hybrid model outperforms both the multinomial naive Bayes (MNB) and DeBERTa models across all evaluation metrics, as shown in Table 1. Specifically, the hybrid model achieves the highest accuracy at 97.5%, surpassing MNB (96.1%) and DeBERTa (97.2%). This improvement reflects the model’s ability to integrate the strengths of both classifiers, providing a more reliable and comprehensive solution for spam detection.

In terms of precision, our hybrid approach also leads with 96.8%, outperforming MNB (93.7%) and DeBERTa (96.1%). This indicates that the model is highly effective at correctly classifying spam messages, with fewer false positives compared to MNB. Furthermore, the hybrid model achieves a recall rate of 96.2%, which is the highest among all methods, significantly improving the identification of spam messages. The ability to capture more spam messages is critical for spam filtering applications, as it reduces false negatives and ensures that fewer spam messages are missed.

When considering the F1-score, our hybrid model achieves an impressive 96.5%, which is higher than both MNB (94.6%) and DeBERTa (95.7%). The F1-score is a balanced measure of precision and recall, and the superior performance of our hybrid model highlights its ability to strike an optimal balance between minimizing both false positives and false negatives.

Additionally, the AUC score, which measures the model’s ability to distinguish between spam and ham messages, is highest for our hybrid model at 99.1%. This demonstrates that our model is highly effective at differentiating between the two classes, with a much better ability to separate spam from legitimate messages compared to the other methods, including MNB (98.3%) and DeBERTa (98.7%).

Inference experiments further demonstrate that our hybrid model retains the computational efficiency of MNB for simpler cases, where spam messages have clear, identifiable patterns. This allows the hybrid model to maintain fast prediction speeds even when processing large volumes of data. At the same time, the DeBERTa branch enhances the robustness of the model in more complex or context-sensitive cases, where spam messages may contain subtle linguistic variations or obfuscations. The integration of both classifiers ensures that our hybrid model performs well across a wide range of spam detection tasks, from straightforward to more sophisticated cases.

In conclusion, the hybrid model not only improves performance across all evaluation metrics but also strikes an ideal balance between accuracy, efficiency, and robustness, making it a highly effective solution for real-world spam filtering tasks.

4.6. Sensitivity Analysis of the Fusion Weight

This experiment aims to investigate the impact of different fusion weights, $\alpha$, on the final performance of our hybrid spam detection model. By systematically varying the value of $\alpha$, we seek to identify the optimal weight distribution between the naive Bayes (MNB) and DeBERTa branches and understand how each branch contributes to the overall model’s performance. The fusion weight $\alpha$ ranges from 0.0 to 1.0, with increments of 0.1. A value of $\alpha =0.0$ signifies complete reliance on the DeBERTa model, while $\alpha =1.0$ indicates complete reliance on the multinomial naive Bayes model.

For each $\alpha$ value, we evaluated the model’s performance on the validation set, recording key metrics including accuracy, precision, recall, and F1-score. The results, summarized in

Table 2. Model performance under different fusion weights ($\alpha$).

$\mathit{\alpha }$ Value	Accuracy	Precision	Recall	F1-Score
0.0 (DeBERTa Only)	97.2%	96.1%	95.4%	95.7%
0.1	97.3%	96.4%	95.8%	96.1%
0.2	97.4%	96.6%	96.0%	96.3%
0.3 (Proposed)	97.5%	96.8%	96.2%	96.5%
0.4	97.3%	96.5%	95.9%	96.2%
0.5	97.1%	96.0%	95.7%	95.8%
0.7	96.6%	94.8%	95.6%	95.2%
1.0 (MNB Only)	96.1%	93.7%	95.6%	94.6%

and

Table 3. Sensitivity analysis of decision threshold in the hybrid model.

Threshold	Precision	Recall	F1	Note
0.30	94.6%	97.3%	95.9%	High recall
0.40	95.7%	96.7%	96.2%
0.45	96.8%	96.2%	96.5%	F1-max
0.50	97.2%	94.9%	96.0%	Default
0.60	98.1%	92.3%	95.1%	High precision

, reveal that the model’s performance exhibits a non-linear trend as $\alpha$ changes. When $\alpha$ is set too low (close to 0.0) or too high (close to 1.0), the performance closely mirrors that of the individual baseline models (DeBERTa or MNB), which reflects the loss of the hybrid model’s advantages. Specifically, when $\alpha =0.0$, the model’s performance is largely driven by the DeBERTa branch, and when $\alpha =1.0$, the model is dominated by the MNB branch.

The experiment results indicate that the optimal balance is achieved when $\alpha \approx 0.3$, where the hybrid model delivers the best overall performance across all evaluation metrics. At $\alpha =0.3$, the model achieves an accuracy of 97.5%, precision of 96.8%, recall of 96.2%, and an F1-score of 96.5%, surpassing both baseline models. This optimal value demonstrates that the effective combination of the naive Bayes and DeBERTa models allows them to compensate for each other’s weaknesses. The naive Bayes model contributes efficiency in simple cases, while the DeBERTa model provides robustness in more complex or context-sensitive scenarios.

The results validate our choice of fusion weight $\alpha =0.3$, confirming that the hybrid approach leverages the strengths of both models while minimizing their individual shortcomings, leading to enhanced overall performance in spam detection tasks.

4.7. Robustness Test on Different Types of Spam

To evaluate the model’s robustness against evolving spam tactics, we created a sub-test set containing different types of adversarial attacks, designed to challenge the model’s ability to handle obfuscated or manipulated spam messages. These attacks simulate common evasion techniques used by spammers to bypass traditional spam detection systems. The sub-test set consists of 100 manually selected spam samples from the original test set, which we modified into the following three types of adversarial attacks:

1.

Spelling Obfuscation: In this type of attack, common spam keywords such as “free” and “winner” are replaced with visually similar variants, like “fr3e” and “w1nner”. This tactic is designed to confuse models that rely on exact keyword matching, making it more difficult for them to identify spam based solely on word frequencies.

2.

Zero-Width Character Insertion: This attack involves inserting invisible zero-width spaces between characters of spam keywords, such as “fr[ZWSP]ee” or “w[ZWSP]inner”. The goal is to deceive tokenizers, which may fail to recognize the original words due to the inserted invisible characters. This technique is commonly used to bypass simple text-based classifiers.

3.

Synonym Replacement: In this attack, common spam keywords are replaced with less frequent but semantically similar words. For example, the word “free” might be replaced with “complimentary”, or “winner” could be replaced with “champion”. While this does not alter the meaning of the message, it challenges models that rely on exact keyword matching and may affect their ability to classify the message correctly.

As shown in

Table 4. Performance on adversarial samples (F1-score).

Model	Standard Spam	Spelling Obfuscation	Zero-Width Chars	Synonym Replacement
MNB (TF-IDF)	94.6%	65.2%	71.5%	78.4%
DeBERTa	95.7%	92.1%	93.5%	92.8%
Hybrid (Ours)	96.5%	92.5%	94.0%	93.1%

, the performance of the MNB model degrades sharply when handling these obfuscated messages. The naive Bayes classifier heavily relies on keyword frequencies, and any manipulation of the words (such as through spelling obfuscation or synonym replacement) significantly affects its ability to correctly classify the messages. For example, in the case of spelling obfuscation, the F1-score drops from 94.6% for standard spam to just 65.2%. Similarly, the MNB model’s performance on zero-width characters and synonym replacement is also significantly impacted, as the obfuscated words do not match the keywords it relies on.

In contrast, the DeBERTa model, which uses contextual understanding rather than simple keyword matching, demonstrates much better robustness against these attacks. The model is able to handle spelling obfuscation, zero-width characters, and synonym replacement with minimal performance degradation, maintaining F1-scores of 92.1%, 93.5%, and 92.8%, respectively. This resilience is due to DeBERTa’s ability to capture deeper semantic and contextual relationships between words, allowing it to recognize spam even when the message’s wording is altered.

Our hybrid model, while experiencing some performance drop, shows a significantly smaller decline than the pure MNB model and outperforms the pure DeBERTa model in terms of F1-score. For instance, the hybrid model achieves an F1-score of 92.5% on spelling obfuscation, 94.0% on zero-width characters, and 93.1% on synonym replacement. This indicates that the DeBERTa branch greatly enhances the model’s resilience to semantic variations, while the MNB branch continues to work efficiently on unaffected samples, leading to superior overall robustness.

In summary, the hybrid model combines the strengths of both naive Bayes and DeBERTa, ensuring high robustness against various adversarial attacks while maintaining performance on standard spam detection tasks. The results underscore the effectiveness of the hybrid approach in handling diverse types of spam manipulation, making it a highly reliable solution for real-world spam filtering applications.

4.8. Error Analysis

To gain insight into the limitations of our hybrid model, we conducted a brief analysis of the misclassified samples (

Table 5. Average inference time per sample for different models.

Model	Avg. Inference Time (ms)
Naive Bayes (TF-IDF)	0.45
DistilBERT	13.2
RoBERTa-base	29.5
DeBERTa-base	27.8
Hybrid (Ours)	28.0

). We observed that false positives were often triggered by benign messages containing words frequently associated with spam, such as “free” or “win”, leading to overconfident predictions by the naive Bayes component. On the other hand, false negatives were typically short or obfuscated spam messages that lacked strong lexical cues, posing challenges even for the contextual understanding of DeBERTa. These observations highlight the need for improved handling of ambiguous and adversarial inputs, which we consider as directions for future work.

5. Ethical Considerations

In developing and evaluating our spam detection framework, we acknowledge several ethical dimensions that warrant attention for responsible research and real-world deployment.

5.1. User Privacy

Our model is trained and evaluated on the publicly available SMS Spam Collection dataset, which contains anonymized text messages without personally identifiable information. However, in practical applications, spam detection systems are often applied to private communication content. Therefore, privacy-preserving measures such as on-device inference, data anonymization, and compliance with regulations like GDPR should be considered before deployment.

5.2. Bias and Fairness

The dataset used in this study exhibits a class imbalance, with a significantly higher proportion of ham messages compared to spam. This imbalance may lead to biased predictions if not properly addressed. While we have applied class weighting and evaluated performance using metrics suitable for imbalanced data, further steps such as fairness audits and bias mitigation strategies are recommended, especially when the system is extended to diverse linguistic or cultural settings.

5.3. Responsible Deployment

Spam detection systems can influence message deliverability and user experience. Overly aggressive filters may lead to legitimate messages being blocked (false positives), while overly permissive systems may allow harmful content to pass through (false negatives). We recommend that such systems be deployed with human-in-the-loop mechanisms, configurable thresholds, and ongoing monitoring. Additionally, transparency through interpretability techniques can enhance trust and accountability in automated decisions.

6. Conclusions

In this paper, we presented a hybrid spam detection framework that effectively combines a traditional naive Bayes classifier and a large language model, DeBERTa, through a simple yet effective weighted probability fusion mechanism. Our approach achieves a favorable balance between computational efficiency and classification accuracy. Experimental results on the SMS Spam Collection dataset demonstrate that the proposed hybrid model consistently outperforms both individual baselines, achieving up to 97.5% accuracy and 96.5% F1-score. This demonstrates the potential of combining classical statistical models and LLMs for practical, high-performance spam filtering.

In future work, we plan to extend this approach to other security-related text classification tasks such as phishing detection and toxic comment filtering. Additionally, we aim to explore more adaptive fusion strategies and investigate the effectiveness of lightweight LLM variants to further optimize the trade-off between performance and computational cost.