A Multi-Agent Framework for Arabic Scam Analysis: Dataset-Specific mT5 Classification and Cultural Annotations ^†

Abstract

This study presents a multi-agent framework for Arabic scam analysis, with roles for classification, behavioural interpretation and educational guidance. A total of 3000 messages were translated and enriched from English-language collections associated with the Short Message Service (SMS) Spam Collection, Enron and SmishTank. The experiments examine whether dataset-specific multilingual Text-to-Text Transfer Transformer (mT5) training improves classification and whether cultural annotations and an additional cultural training objective improve performance. The training, validation and test partitions contained 2400, 300 and 300 messages. Across ten random seeds, dataset-specific classifiers achieved 88.17% mean test accuracy and 88.07% macro-averaged F1 score (macro-F1); the 95% confidence interval for mean accuracy was 87.36–88.97%. Accuracy exceeded combined-data mT5 by 1.90 percentage points (Holm-adjusted $p=0.0250$). Mean accuracies were 98.50%, 92.90% and 73.10% for SMS, email and SmishTank-derived messages. In exploratory cultural comparisons, supplying annotations increased encoder–decoder accuracy by 1.80 points, while adding the cultural objective increased it by 1.23 points; neither difference was statistically significant after adjustment. The 150-message agent assessment yielded 84.7% mean integrated threat accuracy. The results support dataset-specific classification within the evaluated collections and show limited benefits from the tested cultural formulation. Independent Arabic data and user studies are priorities for extending the framework.

1. Introduction

Phishing and scam attacks use social-engineering strategies to influence users and obtain information, credentials, or money [1,2]. Controlled English-language studies have reported phishing-detection accuracy above 99% [3,4]; however, strong performance on one dataset does not establish cross-dataset or cross-language robustness [5]. Phishing research and benchmark resources remain predominantly English-language, while Arabic phishing corpora and language-specific evaluations are comparatively limited [6,7].

Arabic scam detection presents additional modelling challenges because Arabic has rich morphology, substantial dialectal variation, and orthographic characteristics such as character joining and optional diacritics [8,9,10]. Modern Standard Arabic and regional varieties differ in vocabulary and grammar, so a detector evaluated on one variety cannot be assumed to cover the others. These linguistic properties motivate Arabic-specific preprocessing and evaluation; their effects on classification require direct assessment.

Social engineering also depends on persuasion, trust, perceived authority, and situational context [1,11]. Religious references, appeals to familial relationships, misuse of authority and community trust are therefore examined through predefined annotation categories. These categories describe the forms of manipulation represented in the study dataset. Existing Arabic phishing studies commonly emphasise message content, uniform resource locators (URLs), or platform-specific classification [12,13,14], leaving explicit category-level cultural reporting comparatively under-examined.

Recent Arabic-language studies examine short message service (SMS) phishing, SMS scam classification and spam detection [12,14,15,16]. Differences in labels, data preparation and evaluation methods motivate comparisons under a shared experimental design. The present study compares classifiers on identical test messages to assess the value of dataset-specific training. It also examines cultural annotations and an additional cultural training objective through matched comparisons. These analyses establish how the tested modelling choices affect classification performance within the proposed multi-agent framework.

Multilingual and Arabic pretrained language models improve cross-lingual and Arabic text representation [17,18,19,20,21]. Task-specific evaluation is needed to assess how these representations support the cultural categories considered here.

Multi-agent systems have been studied in cybersecurity as a way to combine specialised analytical components [22], while cultural modelling and Arabic support remain comparatively limited. Research on people and culture in cybersecurity also indicates that these factors require explicit, context-sensitive treatment [23]. Recent Arabic scam detectors commonly report dataset-specific performance [12,14,15]. This study evaluates classification across three processed Arabic datasets and examines the contribution of cultural annotations within a multi-agent framework.

Our prior published work [24] introduced a human-centred security framework incorporating behavioural insights and Arabic translation in different experimental settings. The present study evaluates classification and cultural interpretation within a multi-agent framework. The framework assigns three complementary tasks to specialised agents. The classification agent distinguishes spam from legitimate messages. The behavioural agent interprets persuasion and manipulation within the message, while the educational agent provides guidance on responding to suspicious content. This division connects the classification decision with an interpretation of the message and advice for the recipient. Cultural annotations describe contextual features examined in the classification analysis. Each component therefore has a defined purpose within the framework, with classification performance assessed separately from behavioural interpretation and educational guidance. The quantitative evaluation addresses two research questions:

RQ1:

How does training mT5 separately on each dataset compare with training on the combined datasets and with alternative classifiers?

RQ2:

To what extent do cultural annotations and an additional cultural training objective improve classification performance?

This work makes three contributions. First, it specifies a multi-agent framework that assigns distinct roles to classification, behavioural interpretation and educational guidance. Second, it compares dataset-specific and combined-data mT5 classifiers with alternative methods using identical test messages and ten random seeds. Third, it examines supplied cultural annotations, an additional cultural training objective and prediction of cultural indicators from text. The evaluation concerns three processed Arabic datasets; assessment using independently collected Arabic messages and evaluation of user outcomes are directions for future work.

Section 2 reviews related work. Section 3 describes the framework, data and experiments. Section 4 reports classification comparisons and cultural analyses. Section 5 answers the research questions and examines limitations, and Section 6 concludes with future work.

2. Related Work

2.1. Evolution of Scam and Phishing Detection Techniques

Phishing detection has progressed from blacklists and rule-based filtering to statistical machine learning, deep learning, transformer models, and ensemble methods [2]. Traditional classifiers such as Naive Bayes, support vector machines, and random forests learn from labelled features, while deep neural models can learn representations directly from text, URLs, and related signals [25,26,27].

Transformer models such as Bidirectional Encoder Representations from Transformers (BERT) provide contextual language representations [28]. Large language models have also been evaluated directly for phishing-email classification and explanation; for example, Koide et al. report strong GPT-4 performance on a balanced controlled dataset [4]. Performance depends on the language, message type and attack distribution used for evaluation.

Recent research shows that near-perfect results on one phishing dataset may fall substantially when models are evaluated on different or more current datasets [5]. Detection research must therefore distinguish in-dataset accuracy from cross-dataset robustness. Human and cultural factors are also relevant to cybersecurity behaviour and phishing susceptibility, but their effects depend on population and context [11,23]. These observations motivate explicit reporting of language, communication type, and behavioural and cultural scope rather than assuming generalisation from a single benchmark.

2.2. Multilingual and Arabic Natural Language Processing Challenges in Cybersecurity

Arabic phishing resources have been expanded through translated and parallel corpora [6]. These resources improve language coverage, but results obtained from translated or same-source data do not by themselves establish performance on independently collected Arabic scams. Modern multilingual models, including Cross-lingual Language Model RoBERTa (XLM-R) [17] and multilingual Text-to-Text Transfer Transformer (mT5) [18], share representations across languages and support cross-lingual transfer. The present study examines an additional objective for the task-specific cultural categories.

Arabic presents modelling challenges because of its morphological variety, dialectal diversity, and orthographic complexity [8,9]. Modern Standard Arabic and regional dialects differ in vocabulary and grammatical usage, while character joining and optional diacritics complicate preprocessing and normalisation [10].

Arabic-specific language models such as AraBERT [19], CAMeLBERT [20], and MARBERT [21] improve general Arabic language representation. The cultural annotations considered here provide a complementary task-specific evaluation.

2.3. Cultural Context and Social Engineering in Cybersecurity

Phishing and social engineering involve both technical artefacts and human responses to persuasion, trust, urgency, and authority [1,29]. Protection Motivation Theory describes threat appraisal through perceived severity and vulnerability and coping appraisal through response efficacy and self-efficacy [30]. Recent cybersecurity studies continue to use these constructs when examining protective behaviour [31,32].

Research in a Middle Eastern, non-English-speaking setting also shows that trust and security knowledge can affect spear-phishing susceptibility [11]. These findings motivate context-sensitive analysis. The predefined categories in this study describe religious references, familial relationships, authority misuse and community trust as represented in the processed messages. Their contribution to classification is examined through the cultural comparisons.

2.4. Multi-Agent Systems in Cybersecurity

Multi-agent cybersecurity architectures decompose analysis into specialised components and coordinate their outputs. Hmimou et al. demonstrate this pattern for threat detection and correlation using large language models [22]. The architectural value is modularity and inspectable component outputs; robustness still depends on the data, agents, fusion procedure, and validation design.

The present framework uses this decomposition to distinguish message classification from behavioural interpretation and educational guidance. Its empirical contribution concerns the classifier and the cultural comparisons specified by RQ1 and RQ2.

2.5. Recent Advances and Research Gaps in Arabic Phishing Detection

Recent Arabic-language studies address related but distinct detection tasks. Kaddoura et al. and Alhamdan et al. classify Arabic Twitter spam [33,34]. Alsufyani and Alajmani examine Arabic SMS phishing using URL and text-content features [12,15]. Zawari et al. evaluate Arabic SMS scam classification [14], while Kanaan et al. study Arabic email spam [35]. Differences in labels, datasets, message types and reported metrics prevent direct comparisons of performance.

Three gaps motivate this work. First, most cited systems evaluate one dataset, leaving cross-source robustness unresolved. Second, cultural and behavioural categories are often not reported as explicit outputs, although sociocultural modelling is beginning to appear in phishing research [36]. Third, practical use benefits from explanations that expose the basis of a prediction, a broader concern identified in explainable-cybersecurity research [37]. These observations motivate the proposed multi-agent organisation and the two experimental questions. Comparisons on shared data address classifier performance, while the cultural analysis tests a specific annotation-based formulation.

3. Materials and Methods

This study evaluates a multi-agent framework through binary message classification, agent outputs and cultural analysis. RQ1 compares text classifiers on a common 300-message test partition. RQ2 examines cultural inputs and an additional training objective using an encoder–decoder formulation on 300 validation messages. Agent outputs are assessed on 150 messages stratified by threat category and risk level. Each analysis uses the task labels, sample and metrics specified below.

3.1. Multi-Agent Framework

The framework comprises a threat classification agent, a behavioural analysis agent and an educational guidance agent. The threat and behavioural agents analyse the message in parallel. Their outputs and cultural-risk indicators inform a weighted decision, after which the educational agent provides Arabic or bilingual guidance. The binary mT5 classifier assigns spam or legitimate labels, while the agent assessment concerns threat categories, risk levels and behavioural indicators. Figure 1 shows the information flow.

Figure 1. High-level architecture of the culturally aware three-agent Arabic scam detection framework. The threat and behavioural agents analyse messages in parallel. The mT5 classifier receives Arabic message text alone; the cultural-risk contribution enters the combined assessment separately. The combined decision then passes to the educational guidance agent, which adds culturally appropriate guidance; the final system output combines that guidance with the ensemble decision and explanation. The dashed arrow represents the cultural-risk input; solid arrows represent message and decision flow. mT5: multilingual Text-to-Text Transfer Transformer; SMS: short message service.

The behavioural and educational agents use structured GPT-3.5-turbo prompting (OpenAI, San Francisco, CA, USA). Behavioural analysis identifies persuasion, urgency, authority claims and requests for sensitive information. Educational guidance uses the combined decision and contextual interpretation to explain possible responses, with language and explanatory detail adapted to the user. Guidance is generated after the decision and is assessed through review of religious sensitivity, contextual appropriateness, language register and community-oriented messaging.

3.2. Agent Assessment and Combination of Outputs

The agent assessment uses 50 messages from each processed source dataset, giving 150 messages per agent, stratified by threat category and risk level. Threat-category and risk-assessment accuracy refer to agreement with the corresponding annotations. Behavioural assessment concerns social engineering, manipulation and cultural exploitation. These task-specific scores describe agent outputs; binary classifier comparisons use the common 300-message test partition in Section 3.8.

Social-engineering detection is measured by the mean Jaccard similarity between the predicted and annotated tactic sets, expressed as a percentage. For each message, similarity is the number of shared tactics divided by the number of distinct tactics in either set; two empty sets receive a score of one, and an unsuccessful analysis receives zero. Manipulation accuracy measures agreement on the annotated intensity level. Agent F1 is calculated for threat categories and averaged in proportion to the number of reference examples in each category.

The risk score combines threat, behavioural and cultural-risk contributions according to Equation (1):

$$R=0.60R_{\mathrm{threat}}+0.25R_{\mathrm{behavioural}}+0.15R_{\mathrm{cultural}}.$$

(1)

The fixed heuristic weights give the largest contribution to the threat score. Integrated-system results characterise this particular combination. The cultural contribution is an input to the combined decision. Guidance follows the decision and does not contribute to its accuracy. Reported confidence describes the assigned decision score; it is distinct from accuracy.

3.3. Dataset Construction and Classification Labels

This study uses 3000 processed Arabic messages organised into three collections associated with the SMS Spam Collection [38], the Enron email corpus [39] and Smishing Dataset I from SmishTank [40]. Each processed collection contains 1000 records. The selected English-language material underwent quality filtering, translation into Arabic, linguistic and cultural enrichment, and structured annotation. The resulting study collections have the class distributions shown in Table 1. Throughout this paper, SmishTank denotes the processed study subset associated with that source.

Table 1. Composition and partition sizes of the processed Arabic datasets.

Dataset	Total	Legitimate	Spam	Train/Validation/Test
SMS	1000	500	500	800/100/100
Email	1000	593	407	800/100/100
SmishTank	1000	531	469	800/100/100
Total	3000	1624	1376	2400/300/300

The Arabic text was produced through translation and enrichment. Bilingual annotators adapted lexical choices, colloquial expressions and regional phrasing across five predefined Arabic-variety categories, while preserving message intent and security-relevant content such as URLs and numerical information. These categories describe enrichment targets. The evaluation population therefore consists of processed Arabic messages derived from English-language material.

The primary classification target is the stored spam or legitimate label. Spam, phishing and the individual scam categories represent different annotation tasks: a spam prediction is evaluated against the binary spam label. Cultural, behavioural, risk and threat-type annotations describe additional aspects of a message. The text classifier receives the Arabic message alone, and its accuracy is calculated against the binary target.

The three datasets contribute equally in size, with 1000 messages each. Legitimate messages account for 50.0% of the SMS collection, 59.3% of the email collection and 53.1% of the SmishTank collection. Each dataset contributes 800 training, 100 validation and 100 test messages, as shown in Table 1. Table 2 provides illustrative English-to-Arabic records.

Table 2. Illustrative English-to-Arabic records from the processed dataset.

Record	English Source Text	Processed Arabic Text	Class	Variety
0005	Good Morning plz call me sir		Ham	Egyptian
0000	My fri ah… Okie lor, goin 4 my drivin den go shoppin after tt…		Ham	Gulf

Note: The examples reproduce the stored English and processed Arabic text. Ham denotes a legitimate message.

Before partitioning, text was normalised using Unicode NFKC, consistent line endings and removal of surrounding whitespace. Class labels and Boolean annotations were standardised, while message wording and binary labels were retained. Fourteen groups of exact duplicate text were assigned wholly to one partition; all 3000 records remained in the dataset. The resulting training, validation and test sets contain 800, 100 and 100 messages per source, respectively, with no exact text shared across partitions. The partition assignment used the fixed random seed 20260909 to initialise the random-number generator. The same partitions were used for all models and training seeds.

Both the validation and test partitions contain 162 legitimate and 138 spam messages. Validation performance determines training duration and model selection. RQ1 is evaluated on the common 300-message test partition after configuration selection. The cultural comparisons are exploratory analyses of the 300-message validation partition. Repeated training changes model initialisation and data ordering, while the evaluated messages remain fixed.

3.4. Annotation and Expert Review

Two bilingual Arabic–English annotators with cybersecurity expertise completed a 50-record pilot to align the annotation guidelines. They reviewed semantic equivalence, preservation of persuasive cues, cultural clarity and the integrity of security-relevant details. Both then independently reviewed the same stratified 500-message sample. Ninety messages had at least one discrepancy across the combined review criteria; disagreements were resolved by discussion and consensus. This count describes the overall review process. Cohen’s $\kappa =0.82$ summarises agreement on the 13 threat-category labels. The remaining 2500 messages were annotated using the finalised protocol and quality-checked by the research team.

Cohen’s $\kappa$ measures agreement beyond chance between two annotators. It is calculated from their labels, rather than selected as a training parameter. The value of 0.82 concerns the threat-category task. Cultural, behavioural and emotional annotations are study-specific descriptive labels whose reliability requires separate consideration when interpreting those analyses.

The cultural annotations describe religious references, exploitation of familial relationships, misuse of authority, exploitation of community trust and Islamic-finance deception within individual messages. These categories organise the forms of persuasive content examined in this study. The contribution of the annotation set is assessed through comparisons with and without the supplied annotations, while a separate comparison examines the additional cultural training objective. Arabic-script presence is an additional binary text property. The six values form the reference vector used by the cultural training objective. Unassigned Boolean values are encoded as absence. The substantive category analysis focuses on the four indicators with positive validation examples; their definitions and counts appear in Table 3.

Table 3. Definitions and validation counts for the four substantive cultural indicators. Each row contains 300 messages.

Indicator	Annotation Meaning	Positive	Negative
Religious references	Religious concepts used in the message’s persuasive content	21	279
Familial relationships	Appeals to relationships or familial obligations	7	293
Authority misuse	Appeals to or impersonation of an authority	63	237
Community trust	Appeals to social bonds or community credibility	15	285

3.5. Text Classification with mT5

The classification agent uses the encoder of pretrained mT5-small [18]. Arabic message text is tokenised to at most 512 subword tokens. The mean of the encoder representations over non-padding tokens is passed to a linear layer with two outputs. Softmax gives the probabilities of legitimate and spam labels, and the larger probability determines the prediction. Cross-entropy trains the encoder and classification layer. Cultural annotation values are used in the cultural analysis in Section 3.7.

For a message x containing T non-padding tokens, the classifier is

$$h\left(x\right)={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{H}_t\left(x\right),p(y\mid x)=softmax(Wh\left(x\right)+b),$$

(2)

where $H_t\left(x\right)$ is the mT5 encoder representation, and W and b are learned parameters for the two classes.

For dataset-specific training, a separate mT5 classifier is fitted to the 800 training messages from each source. Each classifier is selected using the 100 validation messages from that source and applied to its corresponding 100 test messages. Their predictions are combined to calculate the overall 300-message result. Source identity determines which classifier is used. The comparison model is trained on all 2400 training messages and selected using all 300 validation messages. Both approaches use the same encoder, optimisation settings, maximum number of epochs and decision rule. Dataset-specific training maintains three models, each fitted to fewer messages.

The mT5 encoder classifiers use AdamW with learning rate ${10}^{-4}$, weight decay 0.01 and no added dropout. Training uses batches of eight messages with gradients accumulated over two batches, giving an effective batch size of 16. The learning rate increases during the first 25 updates and then decreases linearly. The maximum gradient norm is 1.0. Training runs for at most 20 epochs and stops after four epochs without an improvement in validation macro-F1; the parameters with the highest validation macro-F1 are retained.

3.6. Alternative Classifiers

Four alternatives provide complementary comparisons: a linear support vector machine (SVM), a character-based convolutional neural network (CNN), AraBERT and mT5 without task-specific training. The trained alternatives use all 2400 training messages. Feature extraction is fitted on the training data, and validation macro-F1 determines the selected training epoch. All methods use the same test messages and binary labels.

The SVM uses character sequences of lengths three to five within word boundaries, represented by term frequency–inverse document frequency (TF–IDF). At most 30,000 features are retained, with minimum document frequency two, logarithmically scaled term frequency and unit-length normalisation. A linear decision function is trained with hinge loss and AdamW using learning rate 0.01, weight decay ${10}^{-4}$, batch size 256 and at most 100 epochs. Training stops after 12 epochs without validation improvement. A score of zero or greater predicts spam.

The CNN processes at most 512 Unicode characters per message. Its character vocabulary is derived from the training text. It uses 64-dimensional character embeddings, 128 convolutional filters at each of three widths (three, four and five characters), global maximum pooling, dropout 0.5 and a two-class output layer. AdamW minimises cross-entropy with learning rate 0.001, weight decay ${10}^{-4}$ and batch size 64 for at most 30 epochs. Training stops after five epochs without an improvement in validation macro-F1.

AraBERT uses the pretrained AraBERTv2 base model [19] and its associated tokeniser, with inputs truncated to 256 subword tokens. The two-class model is fine-tuned using AdamW with learning rate $2\times {10}^{-5}$, weight decay 0.01, dropout 0.1 and batch size 16. The learning rate increases over the first 10% of updates and then decreases linearly. Training lasts for at most eight epochs and stops after two epochs without an improvement in validation macro-F1. The maximum gradient norm is 1.0. Predictions use the larger class score.

The mT5 reference without task-specific training uses the pretrained mT5-small encoder–decoder. The input instruction is “classify Arabic scam message:” followed by the Arabic text. The candidate responses are “classification: ham” and “classification: spam”. The candidate with the larger mean token log-probability determines the prediction. Input length is limited to 512 tokens. This deterministic comparison uses one evaluation.

3.7. Cultural Inputs and Additional Training Objective

RQ2 is examined using an mT5-small encoder–decoder trained on all 2400 training messages. The model learns a structured text response containing the binary class and, for spam messages with a mapped threat annotation, the threat category. During binary evaluation, the decoder receives the shared classification prefix. The scores of the first token that differs between the legitimate and spam responses are normalised over those two alternatives, and the larger score determines the class. A separate linear layer predicts six binary annotation values from the mean encoder representation.

The training objective combines sequence cross-entropy with an additional binary cross-entropy term:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{sequence}}+\lambda {\displaystyle \frac{1}{6}}{\displaystyle \sum _{j=1}^6}{\mathcal{L}}_{\mathrm{binary}}(c_j,{\widehat{c}}_j),$$

(3)

where $c_j$ is the reference value of annotation j, ${\widehat{c}}_j$ is its predicted probability and $\lambda =0.1$ when the cultural objective is included.

Three variants isolate two changes. The first receives the message and its six reference annotation values and includes the additional objective. The second receives the same inputs with $\lambda =0$. The third receives message text alone while retaining $\lambda =0.1$. In the first two variants, annotation values are supplied at both training and evaluation. Their classification results therefore describe performance conditional on annotated context. The third variant assesses prediction of the indicators from text.

All three cultural variants use identical training and validation membership and the same ten random seeds. AdamW uses learning rate $5\times {10}^{-5}$ and weight decay 0.01, with 100 initial learning-rate warm-up updates. Inputs are limited to 512 tokens, and target responses to 128 tokens. Gradients are accumulated over eight batches of two messages, giving an effective batch size of 16. Training lasts at most six epochs and stops after two epochs without an improvement in validation macro-F1. The maximum gradient norm is 1.0, and the parameters with the highest validation macro-F1 are retained. These settings and the encoder–decoder formulation differ from the primary text classifier.

For prediction of cultural indicators from text, a probability of at least 0.5 is treated as positive. Precision, recall and F1 are calculated for each indicator, together with the area under the receiver operating characteristic curve (ROC-AUC). Precision is assigned zero when no positive prediction is made. The Islamic-finance indicator has zero positive validation examples, so positive-class performance is undefined. Arabic-script presence occurs in 297 of 300 validation messages and describes a text property rather than successful recognition of persuasion.

3.8. Evaluation and Statistical Analysis

Accuracy and macro-F1 are the principal classification metrics. Macro-F1 is the unweighted mean of the F1 scores for the legitimate and spam classes. Spam precision and recall describe false-positive and false-negative behaviour. The ten training seeds are 42, 123, 256, 512, 1024, 2048, 3141, 4096, 5555 and 9999. Mean metrics and two-sided 95% Student-t confidence intervals are calculated across seeds using nine degrees of freedom. These intervals quantify variation from repeated training on fixed data. Seed 42 also contributed to the initial formulation selection.

For RQ1, differences are calculated between dataset-specific mT5 and each of four trained comparators at the same seed. Two-sided paired t tests assess the mean differences. Holm adjustment is applied separately to the four accuracy comparisons and the four macro-F1 comparisons. For RQ2, Holm adjustment covers four tests jointly: accuracy and macro-F1 for each of the two cultural comparisons. The significance level is 0.05. Difference intervals use the paired differences across seeds; they are unadjusted 95% intervals. These analyses concern training variability on the evaluated messages.

3.9. Sensitivity to the Proportion of Spam

An analytical sensitivity analysis examines how predictive values change with the proportion of spam. Let $\pi$ be that proportion, s the sensitivity and q the specificity. At a fixed sensitivity and specificity,

$$\mathrm{Accuracy}\left(\pi \right)=\pi s+(1-\pi )q,\mathrm{PPV}\left(\pi \right)={\displaystyle \frac{\pi s}{\pi s+(1-\pi )(1-q)}},$$

(4)

$$\mathrm{NPV}\left(\pi \right)={\displaystyle \frac{(1-\pi )q}{(1-\pi )q+\pi (1-s)}}.$$

(5)

Positive predictive value (PPV) is the proportion of spam predictions that are correct; negative predictive value (NPV) is the proportion of legitimate predictions that are correct. Sensitivity and specificity are calculated from the combined validation predictions for each seed and averaged across the ten seeds before calculating accuracy and predictive values at each assumed spam proportion. Scenarios use spam proportions of 1%, 5%, 10%, 25%, 46% and 50%. The results are conditional projections at the same decision rule.

3.10. Computing Environment

Training was performed with PyTorch 2.9.1 and CUDA 12.8 on an NVIDIA GeForce RTX 5060 (NVIDIA, Santa Clara, CA, USA) with 8 GB of graphics memory. Neural model training used bfloat16 mixed precision. These experiments concern offline classification and cultural analysis; assessment of application response time, computational cost and user interaction is part of future deployment evaluation.

4. Results

The classifier comparisons address RQ1. Agent results describe the framework’s threat, risk and behavioural outputs, and the cultural analysis addresses RQ2. Classifier and cultural means include all ten specified training seeds. Tables and captions identify the task and evaluation sample.

4.1. Classifier Comparisons on the Test Set

Dataset-specific mT5 achieved 88.17% mean accuracy and 88.07% macro-F1 on the 300 test messages. The 95% confidence intervals across seeds were 87.36–88.97% and 87.22–88.92%, respectively. The sample standard deviation of accuracy was 1.12 percentage points. Mean spam precision was 87.16%, recall 87.46% and F1 87.13%. Table 4 compares the six methods under the same test membership. Figure 2 shows the mean accuracy and variation across training seeds for the trained methods.

Table 4. Test classification performance. Values are percentages; intervals describe the mean across training seeds. CI: confidence interval; CNN: convolutional neural network; TF–IDF: term frequency–inverse document frequency; SVM: support vector machine.

Method	Runs	Accuracy (95% CI)	Macro-F1 (95% CI)
Dataset-specific mT5	10	88.17 (87.36–88.97)	88.07 (87.22–88.92)
AraBERT	10	87.00 (86.12–87.88)	86.84 (85.94–87.74)
Combined-data mT5	10	86.27 (85.31–87.22)	86.19 (85.22–87.15)
Character CNN	10	86.17 (85.29–87.05)	86.01 (85.14–86.88)
Character TF–IDF SVM	10	85.87 (85.19–86.54)	85.77 (85.09–86.45)
mT5 without task training	1	49.67	43.23

Each evaluation uses the same 300 messages. The mT5 reference without task training is deterministic.

Figure 2. mT5 classification and performance comparisons. (a) Test accuracy for the ten seeds listed in Section 3.8, in the stated order; the dashed line shows the 88.17% mean. (b) Mean accuracy and spam precision, recall and F1 by dataset. (c) Mean ROC curves obtained by interpolating each run at common false-positive rates; legends show mean per-run AUC. (d) Mean test accuracy and 95% intervals across training seeds. The mT5 reference without task-specific training is deterministic. Each classifier run uses the same 300 test messages, with 100 per source.

Dataset-specific training improved mean accuracy by 1.90 percentage points over combined-data mT5. It also exceeded AraBERT by 1.17 points, the CNN by 2.00 points and the SVM by 2.30 points. The four accuracy differences remained significant after Holm adjustment (Table 5). The corresponding macro-F1 improvements were 1.89, 1.23, 2.06 and 2.31 points, respectively, with adjusted p values of 0.0272, 0.0272, 0.0030 and 0.0041. These differences describe the selected configurations and evaluated data.

Table 5. Paired accuracy differences on the test messages: dataset-specific mT5 minus each comparator. Differences and intervals are percentage points. The p-values use Holm’s method for the four comparisons.

Comparator	Difference	95% CI	p-Value
Combined-data mT5	1.90	0.52–3.28	0.0250
AraBERT	1.17	0.29–2.04	0.0250
Character CNN	2.00	1.07–2.93	0.0037
Character TF–IDF SVM	2.30	1.21–3.39	0.0037

4.2. Performance Across the Three Datasets

Mean accuracy was 98.50% for SMS-, 92.90% for email- and 73.10% for SmishTank-derived messages (Table 6). The overall accuracy therefore combines substantially different levels of difficulty. Each source contributes 100 test messages, so the overall accuracy weights the three sources equally.

Table 6. Dataset-specific mT5 accuracy by source on the test set.

Dataset	Messages per Run	Mean Accuracy (%)	95% CI (%)
SmishTank	100	73.10	70.54–75.66
Email	100	92.90	91.10–94.70
SMS	100	98.50	97.89–99.11

4.3. Agent and Integrated-System Performance

The threat classification agent achieved threat-category accuracies of 84.0% for SMS, 88.0% for email and 92.0% for SmishTank (Table 7). The behavioural agent’s mean tactic-similarity scores ranged from 72.0% to 78.0%, and cultural-exploitation scores from 82.0% to 86.0% (Table 8). Figure 3 summarises the agent measurements and integrated outputs.

Table 7. Threat classification agent performance.

Dataset	Threat Acc.	Risk Acc.	Weighted F1	Confidence
SMS	84.0%	78.0%	83.0%	86.0%
Email	88.0%	84.0%	87.0%	88.0%
SmishTank	92.0%	80.0%	81.0%	84.0%

Table 8. Behavioural analysis agent performance.

Dataset	Tactic Similarity	Manipulation	Cultural Exploitation
SMS	72.0%	76.0%	82.0%
Email	78.0%	82.0%	86.0%
SmishTank	75.0%	80.0%	84.0%

Figure 3. Agent assessment across the three processed datasets, using 50 messages per source. Panels show (a) threat category and risk accuracy, (b) mean tactic similarity, manipulation accuracy and cultural-exploitation scores, (c) integrated threat and risk accuracy, and (d) reported decision confidence. Confidence is a score assigned to the decision and is not an accuracy measure.

The integrated system achieved reported threat-category accuracies of 82.0% for SMS, 88.0% for email and 84.0% for SmishTank, giving an equally weighted mean of 84.7% across the three datasets. Mean risk-assessment accuracy was 80.0%, and the mean of the dataset-specific weighted F1 scores was 84.3%. Mean decision confidence was 82.3%; this represents the confidence assigned to decisions (Table 9).

Table 9. Integrated system performance.

Dataset	Threat Acc.	Risk Acc.	Weighted F1	Confidence
SMS	82.0%	76.0%	83.0%	76.0%
Email	88.0%	84.0%	87.0%	87.0%
SmishTank	84.0%	80.0%	83.0%	84.0%
Average	84.7%	80.0%	84.3%	82.3%

Note: Threat and risk scores describe the fixed combined configuration on the 150-message agent assessment. Educational guidance follows the combined decision.

4.4. Stability Across Training Seeds

The ten-seed classifier results in Figure 4 show greater variation for SmishTank-derived messages than for SMS or email. The sample standard deviations were 3.57, 0.85 and 2.51 percentage points, respectively. Together with the lower SmishTank mean, this variation identifies source-specific generalisation as a priority for further evaluation.

Figure 4. Dataset-specific mT5 accuracy across ten random seeds on 100 test messages per source. Dashed lines mark mean accuracies of 98.50% for SMS, 92.90% for email and 73.10% for SmishTank. The corresponding sample standard deviations are 0.85, 2.51 and 3.57 percentage points.

4.5. Cultural Comparisons on the Validation Set

The encoder–decoder receiving annotated context and the additional cultural objective achieved 75.60% accuracy and 75.41% macro-F1. Removing the additional objective gave 74.37% accuracy and 74.29% macro-F1. Removing annotated context while retaining the additional objective gave 73.80% accuracy and 73.29% macro-F1 (Table 10).

Table 10. Exploratory encoder–decoder results on 300 validation messages across ten seeds. Values are percentages.

Variant	Accuracy (95% CI)	Macro-F1 (95% CI)
Annotated context + cultural objective	75.60 (74.91–76.29)	75.41 (74.77–76.05)
Annotated context only	74.37 (71.25–77.48)	74.29 (71.17–77.40)
Text + cultural objective	73.80 (71.53–76.07)	73.29 (70.95–75.62)

Supplying cultural annotations increased mean accuracy by 1.80 points (95% CI, $-0.55$ to 4.15) and macro-F1 by 2.12 points ($-0.39$ to 4.63). Adding the cultural objective, conditional on annotated input, increased accuracy by 1.23 points ($-2.14$ to 4.61) and macro-F1 by 1.12 points ($-2.12$ to 4.37). Adjusted p values were 0.3521 for both annotation-input comparisons and 0.8604 for both additional-objective comparisons. Neither tested modification produced a statistically significant improvement. Figure 5 shows both mean differences and their uncertainty relative to zero.

Figure 5. Paired changes in validation accuracy across ten seeds. Supplying annotations compares annotated input with text input while retaining the cultural objective. Adding the cultural objective compares its inclusion and omission with annotated input. Horizontal lines are unadjusted 95% intervals; displayed p values are Holm-adjusted. Both intervals include zero.

4.6. Prediction of Cultural Indicators from Text

At the 0.5 probability threshold, the text-input variant predicted none of the four substantive cultural indicators as positive in any training seed. Recall and F1 were therefore zero for all four indicators; precision was zero under the convention defined in Section 3.7. The intervals across seeds for these threshold-based metrics were also zero to zero, reflecting identical outcomes on the fixed validation messages. Table 11 reports the positive counts and ROC-AUC values. The ROC-AUC values indicate some ranking information for authority misuse and community trust, but this did not translate into positive detections at the specified threshold.

Table 11. Cultural indicator prediction from text on the validation set. Precision, recall, F1 and ROC-AUC are percentages.

Indicator	Positive/Total	Precision/Recall/F1	ROC-AUC (95% CI)
Religious references	21/300	0/0/0	55.63 (49.07–62.19)
Familial relationships	7/300	0/0/0	52.80 (48.91–56.70)
Authority misuse	63/300	0/0/0	61.07 (57.93–64.21)
Community trust	15/300	0/0/0	61.00 (53.79–68.21)

Note: At the probability threshold of 0.5, none of these indicators were predicted as present in any of the ten runs. Recall and F1 are zero; precision is recorded as zero by convention when there are no positive predictions.

Arabic-script presence had precision 99.00%, recall 100% and F1 99.50%, with the same values in every seed. Its prevalence was 99%, making it a near-constant text property. The absence of positive Islamic-finance examples and the low counts for several substantive categories limit category-specific interpretation.

4.7. Predictive Values at Different Spam Proportions

The validation sensitivity and specificity of dataset-specific mT5 averaged 88.55% and 88.09%, respectively. Under the assumptions in Section 3.9, expected accuracy remained approximately 88% across the six spam proportions, while positive predictive value changed markedly (Table 12). At 1% spam, expected PPV was 6.98% and NPV was 99.87%; at 50% spam, they were 88.14% and 88.50%. A high accuracy on a nearly balanced collection can therefore coexist with a low proportion of correct alerts when spam is rare. Figure 6 shows how predictive values change across the six scenarios.

Table 12. Analytical projections at fixed validation sensitivity, specificity and decision rule. All values are percentages. PPV: positive predictive value; NPV: negative predictive value.

Spam Proportion	Expected Accuracy	PPV	NPV
1	88.09	6.98	99.87
5	88.11	28.12	99.32
10	88.13	45.23	98.58
25	88.20	71.24	95.85
46	88.30	86.36	90.03
50	88.32	88.14	88.50

Figure 6. Expected accuracy and predictive values at six assumed spam proportions using validation sensitivity and specificity at a fixed decision rule. Symbols identify the calculated scenarios; connecting lines guide the eye. These are analytical projections.

5. Discussion

5.1. Answer to RQ1

Training mT5 separately on each dataset improved mean test accuracy by 1.90 percentage points over combined-data training under the specified settings. The improvement also appeared in macro-F1 and remained significant after adjustment across the paired comparisons. Dataset-specific mT5 exceeded the selected SVM, CNN and AraBERT configurations on the common test partition. These results support source-dependent classification within the three processed collections.

A separate classifier can adapt to the vocabulary and message structure of its dataset. However, the three-model approach requires the source to be known and maintains more model parameters overall than a single combined classifier. Its performance also remains uneven: SmishTank-derived messages were substantially harder to classify than SMS or email. Evaluation on a previously unseen source is a distinct generalisation question.

5.2. Answer to RQ2

The tested cultural annotations and additional cultural objective produced small positive mean differences, but neither yielded a statistically significant improvement after adjustment. The intervals include both decreases and increases in performance. The results therefore indicate limited benefits for this particular formulation, rather than a general advantage from cultural information.

The supplied annotations describe context available to the model for the first two cultural variants. Their results concern classification with annotated messages. Automatic prediction from text remains a limitation: the model detected none of the four substantive cultural indicators in the validation messages at the specified threshold. This distinction is important for a multi-agent application: contextual labels supplied by an annotator and contextual labels inferred automatically have different practical roles and error characteristics.

5.3. Implications for the Multi-Agent Framework

The results in Section 4.3 describe the threat-category and risk-assessment tasks within the multi-agent framework. The framework combines classification, behavioural interpretation and educational guidance to provide a decision together with contextual analysis and suggested responses. The mT5 classifier’s 88.17% accuracy concerns binary spam classification, a separate task from the integrated threat-category assessment. Future user studies will assess the accuracy and comprehensibility of explanations and their effect on recipients’ responses to suspicious messages.

For practical use, the proportion of spam and the cost of false alerts matter alongside accuracy. The analytical projections show that alert precision can decline sharply when spam is rare. Deployment evaluation should therefore specify the target population, decision threshold and consequences of each error type. Response time and cost should be measured for the complete interaction, including any external language-model requests.

5.4. Limitations

This study evaluates 3000 translated and enriched messages from three processed collections. Arabic-variety enrichment targets do not represent naturally sampled regional or dialectal populations. Translation and enrichment are part of data construction, rather than independently varied experimental factors. Further work should quantify translation-related changes in meaning and persuasive content through independent bilingual assessment of matched source and Arabic messages. Applicability to naturally occurring Arabic scams and evolving attack patterns requires independent evaluation.

The test set contains 300 messages, including only 100 from each source. Training-seed intervals describe variation on these fixed records and do not account for uncertainty from sampling a different population. One seed contributed to initial formulation selection, and model choices were based on the validation results. Exact duplicate text was kept within a single partition, but this does not establish separation of all paraphrases or related attack campaigns.

The cultural comparisons are exploratory validation analyses using a different architecture, target format and training duration from the primary classifier. Their absolute scores therefore cannot isolate the cost or benefit of cultural information relative to the primary model. Adding annotation text can also change how much message text fits within the input-length limit. The observed differences apply to the specified formulation and annotation availability.

Cultural categories are predefined labels for the study data rather than universal descriptions of Arabic-speaking communities. Their unequal frequencies, sparse positive examples and dependence on annotation quality limit interpretation. The expert agreement statistic concerns the 13 threat categories, while the cultural analysis uses descriptive annotations. The evaluation uses one resolved annotation set; sensitivity to alternative annotation decisions is a further methodological question.

The agent assessment includes 150 messages and uses a fixed heuristic combination of scores. Its interpretation is specific to the assessed threat and risk tasks. Sensitivity to alternative weights and evaluation on independently collected messages are needed to assess the stability of the combined decision. User studies are needed to assess whether the explanations and guidance improve responses to suspicious messages.

6. Conclusions

This study presents a multi-agent framework for Arabic scam analysis and evaluates two questions concerning its classification methods. Dataset-specific mT5 classifiers achieved 88.17% mean accuracy and 88.07% macro-F1 on 300 test messages across ten training seeds. Training separately on each dataset improved accuracy by 1.90 percentage points over combined-data mT5 under the specified settings. Performance differed substantially across the three datasets. In the exploratory cultural analysis, supplied annotations and an additional cultural objective produced small, statistically non-significant gains.

Future work will examine independent Arabic messages spanning wider dialects, regions and attack types; adaptation to new sources; experiments varying the training class proportions; and decision thresholds appropriate to the target spam proportion. For cultural prediction, future work should examine greater representation of positive examples and thresholds selected on validation data, followed by evaluation on independently annotated data. Further evaluation of the multi-agent framework should compare classification alone with classification plus interpretation and guidance, measuring explanation quality, user decisions, response time and cost. These studies will assess the practical value of combining classification, interpretation and guidance.