An AI-Based Security Architecture for Fraud Detection in Cloud Call Centers for Low-Resource Languages: Arabic as a Use Case

Abstract

Cloud-based telephony platforms face growing fraud risks including voice phishing (vishing), subscription abuse, and organizational impersonation, with detection being especially challenging in low-resource languages such as Arabic. We present an Artificial Intelligence (AI)-based security architecture for fraud detection in Arabic cloud call centers, combining onboarding verification, behavioral monitoring, domain-adapted Automatic Speech Recognition (ASR), semantic transcript search, and Large Language Model (LLM)-based entity verification. The domain-adapted Langa ASR model achieves a Word Error Rate (WER) of 41.0% and Character Error Rate (CER) of 18.2%, outperforming all evaluated commercial baselines. LLM-based entity extraction with multi-call consensus achieves 97.3% company-name accuracy (Generative Pre-trained Transformer 4, GPT-4) and 92.0% in the cost-effective deployed configuration (GPT-3.5 with log-probability filtering). Evaluated on production data from a Middle East and North Africa (MENA)-region provider spanning more than 1000 accounts, the pipeline flagged 47 accounts of which 41 were confirmed fraudulent (directly observed precision 87.2%, 95% confidence interval (CI): 74.3–95.2%; estimated recall 51–82% under conservative base-rate assumptions—not directly measured), providing evidence for the viability of a unified, threat-model-driven architecture for low-resource telephony fraud detection.

1. Introduction

The rapid increase of cloud-based communication services is significantly shaping the digital transformation of the global economy. Small, medium, and large enterprises rely on cloud-based call centers to handle customer interactions, telemarketing campaigns, and distributed support services [1]. Internet-based remote work and cloud call centers are growing steadily. Businesses adopt them to reduce infrastructure costs and enhance operational flexibility [2]. The characteristics of scalability, deployment simplicity, and geographic distribution are benefiting the legitimate businesses who adopt cloud telephony.

However, opportunities for actors with malevolent intentions are inadvertently created by these characteristics. As more entities and individuals depend on these platforms for services in finance, healthcare, and governance, criminal fraud exploitation has increased substantially [3]. Evidence of this happening in cloud call centers is seen in various, distinct attack types. These include subscription fraud carried out with fabricated or stolen credentials [4]. Impersonation of valid identities by voice phishing is also carried out to extract confidential data from the recipients of the calls. Unfortunately, this results in operations where accounts are controlled illegally and system vulnerabilities are exploited.

Beyond immediate financial losses, the impact of fraud in cloud call centers extends to the unintentional enablement of fraudulent call operations by service providers. Major regulatory penalties follow, as telecommunications authorities enforce compliance requirements. In addition, the credibility of these businesses in the market is weakened [5].

Cloud telephony environments consist of fraud detection systems with limitations that can be characterized with three critical gaps. The first gap is the reliance of these systems on CDR-based and rule-based approaches [6,7], which inherently lack the capability to detect voice-driven fraud. This limitation stems from where fraud is encoded: in call content rather than call volume or routing. Voice-driven fraud is typically encoded in speech [8,9,10] and systems operating only on metadata fail to detect these types of attack.

The second gap is the failure in the scope of the testing of the approaches of the content-aware natural language processing [11,12] which focuses on single-language datasets but is not directly applicable to the morphologically complex Arabic telephony. A third gap: prior systems lack an explicit security threat model or production validation. These gaps are compounded by the inherent difficulty of processing Arabic speech at scale. Conversational fraud detection in Arabic faces linguistic obstacles absent in well-resourced languages: a wide dialect continuum (Egyptian, Levantine, Gulf, and Maghrebi), pervasive code-switching with Modern Standard Arabic (MSA), and rich morphology that degrades ASR quality—the foundational step for all downstream analytics. Section 2 surveys both the fraud detection and Arabic NLP literature in detail.

We address the gap between isolated research contributions and operational security deployment by presenting a security-by-design, end-to-end fraud detection architecture for Arabic cloud call centers. Specifically, the contributions of this work are threefold:

1.

Detection of fraud attacks using a hybrid approach: A two-phase transcript pipeline is proposed to detect fraud-related conversational intent in summaries of standardized calls, addressing variation in Arabic dialect and morphology. The pipeline consolidates rule-based precise keyword search with neural embedding-based semantic retrieval to improve detection coverage.

2.

Statistical rigor applied to LLM-based behavioral and entity verification: GPT-3.5, GPT-4, and the Arabic-centric Jais model are systematically studied experimentally for agent and company-name extraction, including multi-call consensus and log-probability confidence filtering. Findings (end-to-end) include precision, confidence intervals, confusion matrix analysis, and recall estimation.

3.

End-to-end production validation: The pipeline was tested on production data from a MENA-region provider (1024 accounts; ∼38,000 calls). Reported metrics include precision with Wilson score confidence intervals, conservative recall bounds accounting for the metadata escalation funnel, a confusion matrix, and per-layer reasoned security analysis across all four identified attack surfaces (Section 3 and Section 6).

To the best of our knowledge, this work is among the first to present an operationally validated, end-to-end fraud detection architecture specifically designed for Arabic cloud telephony, distinguished by integrating the following: (i) a threat-model-driven security design covering four explicitly defined attack surfaces; (ii) a domain-adapted ASR model benchmarked against five commercial systems on real production telephony speech; (iii) a systematic LLM ablation quantifying the accuracy–cost trade-off for entity verification in noisy low-resource transcripts; and (iv) production validation on real subscriber data from a MENA-region provider with statistical confidence intervals. Prior work addresses at most one or two of these dimensions in isolation; none, to our knowledge, combines all four with explicit adversarial threat modeling and production-scale evaluation.

This paper is organized as follows. Section 2 reviews related work critically. Section 3 presents the threat model and security assumptions. Section 4 describes the proposed architecture. Section 5 presents experimental results. Section 6 provides the security analysis. Section 7 discusses implications and limitations. Section 8 concludes this paper.

2. Related Work

Fraud detection has been extensively explored in cloud-based communication from various angles, but typically each approach offers limited insight by scrutinizing only a particular component of the overall problem. For instance, some have addressed financial fraud detection techniques for structured transaction data [13,14,15,16] and others have targeted telecom call records [6,7,17] or spoken conversations [8,11,12,18] as separate standalone problems. However, fraud in cloud telephony is a complex problem and typically requires an integrated view across all these dimensions simultaneously. For example, financial fraud detection methods are effective for structured data but fail to capture language patterns and behavioral trends in spoken conversations [13,14,16]. Further, call-record analysis techniques are very effective at capturing unusual behavior in calling patterns but ineffective at capturing the underlying substance of conversations [6,7,17,19]. Currently, no study exists in this domain that can fully address the complex needs of cloud telephony fraud detection particularly for Arabic language processing, dealing with adversarial behavior and practical system design [11,12,18,20].

Financial fraud detection is an important problem that has been widely studied in the literature. Rule-based methods, ensemble learning approaches, and graph neural networks have all been used to address problems such as credit card fraud, money laundering, and loan fraud [13,14,15,16]. Some studies have also proposed real-time detection pipelines based on big data processing frameworks such as Apache Spark and Kafka, combined with anomaly detection techniques such as Isolation Forest [21]. However, these approaches are architecturally incompatible with cloud telephony fraud. Transactional fraud leaves structured digital traces (amounts, merchants, and timestamps) amenable to statistical modeling; telephony fraud is primarily a social-engineering phenomenon whose evidence is embedded in natural language utterances. Applying financial fraud detectors to call-center fraud detection therefore addresses the wrong threat model.

Within telecommunications, CDR analytics remains the dominant fraud detection approach [6,7,17]. FrauDetector [19] applies a weighted Hyperlink-Induced Topic Search (HITS) algorithm to call graphs to propagate phone trust values—effective for Subscriber Identity Module (SIM)-box fraud, toll bypass, and infrastructure-level abuse, but fundamentally incapable of detecting impersonation: a fraudulent agent who makes calls at normal volume, frequency, and routing pattern is completely invisible to any CDR-based system. Latent Dirichlet Allocation (LDA)-based user profiling [22] and Support Vector Machine (SVM)/Artificial Neural Network approaches applied to CDR features [23] share this limitation. The critical insight missing from this body of work is that cloud telephony fraud is intentionally designed to mimic legitimate calling behavior at the metadata level—subscription fraud accounts pass CDR screening by design, making conversational intelligence architecturally necessary, not optional.

While conversational and NLP-based approaches have gained traction in recent years, as they can incorporate semantic signals to improve the accuracy of fraud detection, all current solutions that operate solely on transcript input have three fundamental limitations. First, spoken language is much more dynamic than static text and unfolds in real time. Current solutions have not developed adequate methods to handle the variability found in spoken language, including variation in word choice, insertions, deletions, and recognition errors [12,18]. Second, most current approaches either rely on large amounts of resources from high-resource languages or do not directly target the problem of fraud detection [11,18,20]. Finally, while many approaches have been prototyped, explored, and evaluated on large datasets [11,12,18,20], there is still no current solution that has been deployed in production and statistically validated in real-world usage scenarios.

On the Arabic NLP side, well-documented ASR challenges include dialectal diversity, morphological richness, code-switching behavior, and telephony channel noise [24,25]. Recent studies show that fine-tuning multilingual models on domain-specific speech data yields real transcription gains [26,27], and that improved transcription propagates directly to better downstream NLP accuracy [28]. Semantic embedding and transformer-based retrieval enable cross-dialectal similarity search at scale [29,30]. Yet no prior work integrates these capabilities into a deployed security architecture with explicit threat modeling.

Table 1. Comparison of fraud detection approaches across five dimensions critical to secure cloud telephony deployment. Bold indicates the best-performing entry.

Approach	Conversational Intelligence	Arabic Language	Threat Model	Deployment Validation	Eval. Rigor
CDR analytics [6,7]	×	∘	×	✓	∘
FrauDetector [19]	×	×	×	✓	∘
NLP transcripts [11]	✓	×	×	×	×
Graph LLM [20]	✓	×	×	×	∘
This work	✓	✓	✓	✓	✓

summarizes how the present work extends the state of the art across five dimensions critical to operational security deployment; ✓ indicates fully addressed, ∘ partially addressed, and × not addressed.

3. Threat Model and Security Assumptions

Security-oriented fraud detection systems must be designed against an explicit adversary model. We define the threat model for cloud telephony fraud detection following standard security analysis conventions.

3.1. Adversary Goals

We consider adversaries with two primary operational goals:

Goal G1—Financial exploitation: The adversary aims to provision and operate cloud telephony services (Direct Inward Dialing (DID) numbers; trunk capacity) without legitimate payment, exploiting stolen payment instruments, free trials, or refund mechanisms.

Goal G2—Vishing and impersonation: The adversary aims to conduct outbound calling campaigns in which agents falsely represent themselves as employees of legitimate organizations to solicit sensitive information, financial transactions, or behavioral compliance from call recipients.

3.2. Adversary Capabilities

What the adversary controls:

• Account registration data: The adversary can fabricate or steal business credentials, email addresses, company registrations, and payment instruments used during onboarding.
• Agent speech content: The adversary fully controls what agents say during calls, including the organizational identity they claim, the scripts they use, and the urgency or authority cues they deploy.
• Call metadata: The adversary can control call volume, call timing, and callee selection to mimic legitimate calling patterns, partially evading CDR-based detectors.
• Multiple accounts: The adversary can operate multiple subscriber accounts, potentially using shared or rotating payment instruments, to distribute calling activity and reduce per-account anomaly scores.

3.3. Attack Surface Definition

We identify four primary attack surfaces in the proposed architecture:

AS1—Onboarding evasion: The adversary provides plausible but fabricated documentation to pass onboarding verification. Countermeasure: multi-source cross-field validation and behavioral urgency signals (see Section 4.2).

AS2—Metadata camouflage: The adversary patterns calling behavior to resemble legitimate accounts, suppressing CDR-level anomaly signals. Countermeasure: the architecture’s cascade design means metadata evasion alone is insufficient; accounts must also pass conversational verification.

AS3—ASR evasion: The adversary attempts to degrade ASR accuracy by introducing noise, accents, or obfuscated speech patterns to prevent keyword and entity extraction. Countermeasure: embedding-based semantic search is robust to ASR errors (operates on summary-level semantics, not exact transcript tokens); the domain-adapted Langa model is explicitly trained on telephony-condition speech.

AS4—LLM prompt injection: In this attack the goal is to use normal conversation to extract information by injecting maliciously crafted content into the conversation in order to manipulate the output of the LLM for information extraction [31]. Examples of such content could be asking an entity recognition model to extract entity names that are not there, or making comments in conversation that cause the LLM to misinterpret normal language in order to extract additional information. Mitigation: We mitigate this attack in several ways. We apply the LLM prompts to very short, structured pieces of conversation as opposed to longer, free-form conversation. In addition, we use a multi-call consensus approach where the same conversation segment is queried multiple times at different temperatures. Finally, we apply log-probability filtering to favor the highest confidence extractions.

4. Architecture and Methodology

4.1. System Overview and Design Principles

In this paper, we propose a novel, cost-effective architecture for fraud detection using a two-path cascade of metadata and live speech and language analysis. The first path uses metadata (e.g., call events; caller information) to identify potentially fraudulent accounts early in the call flow and flag them for deeper analysis. The second path of analysis looks at the live call content and detects various attack strategies. By using less expensive metadata analysis on all calls and employing more expensive speech and language analysis on only those accounts that exceed a certain threshold, we significantly reduce the cost of detection. Not only can the system identify suspicious conversational evidence, but it can also analyze escalated accounts and assess quality signals such as engagement patterns and interaction volume. Our system is designed from a security-in-depth perspective, meaning that an adversary must succeed against all of the independent mechanisms simultaneously in order to remain undetected, rather than defeating a single mechanism.

Formal definition. To clarify the definition, a basic mathematical approach is used. Given an account $\mathcal{A}$, a metadata stream $\mathcal{M}$ (billing events, usage logs, and account configuration changes), and a set of call recordings $\mathcal{C}$, the system produces two outputs: (1) a binary risk flag $y\in \{0,1\}$ indicating whether the account should be escalated to fraud operations, and (2) an evidence report $\mathcal{R}$ consolidating the strongest supporting signals. The proposed system is designed around reports to assist manual decision making as to whether to suspend a cloud telephony account, guiding the investigation and providing evidence needed to make a decision.

Figure 1 presents the complete pipeline. In the first path, accounts are screened using metadata and passed through an escalation threshold before entering the more expensive second path, which performs speech and content analysis. The figure also marks attack surfaces AS1–AS4 at the points where each part of the system may be exposed. At the final stage, the decision layer combines all available evidence into report $\mathcal{R}$ and sends flagged accounts for human review rather than making an automatic enforcement decision.

4.2. Metadata-Based Detection Path

4.2.1. Onboarding Verification (Countermeasure: AS1)

Onboarding verification helps to reduce the risk of fraud early on. The smeta score $s_{\mathrm{meta}}$ is calculated from 5 different signal groups (document completeness, cross-field consistency, email and domain integrity, website legitimacy, and behavioral urgency at registration) and is used by the verification teams to assess potential fraud and by the API to enforce access restrictions based on the risk score. Algorithm 1 explains how this score is computed. The output is a normalized value $s_{\mathrm{meta}}\in [0,1]$ along with a set of human-readable flags added to $\mathcal{R}$.

Algorithm 1 Onboarding Risk Score Computation

Require:  Account registration data $\mathcal{A}$ (documents, email, website, cross-fields, behavioral signals) Ensure:  Onboarding risk score $s_{\mathrm{meta}}\in [0,1]$, flag set F   1: $F\leftarrow \varnothing$; $s_{\mathrm{meta}}\leftarrow 0$   2: // Stage 1: Document completeness check   3: if $\mathcal{A}.\mathrm{documents}$ incomplete or any required field $\in \varnothing$ then   4:   $s_{\mathrm{meta}}+=w_1$; $F\leftarrow F\cup \left\{\mathrm{MISSING\_DOCS}\right\}$   5: end if   6: // Stage 2: Email and domain integrity   7: if $\mathcal{A}.\mathrm{email}$ matches free-provider list or $\mathcal{A}.\mathrm{email\_domain}\ne \mathcal{A}.\mathrm{company\_domain}$ then   8:   $s_{\mathrm{meta}}+=w_2$; $F\leftarrow F\cup \left\{\mathrm{EMAIL\_RISK}\right\}$   9: end if 10: // Stage 3: Website legitimacy 11: if $\mathcal{A}.\mathrm{website}$ unreachable or contains placeholder text (e.g., “Lorem ipsum”) then 12:   $s_{\mathrm{meta}}+=w_3$; $F\leftarrow F\cup \left\{\mathrm{FAKE\_WEB}\right\}$ 13: end if 14: // Stage 4: Cross-field consistency 15: if $\mathcal{A}.\mathrm{account\_name} \not\approx \mathcal{A}.\mathrm{company\_name}$ or billing mismatch then 16:   $s_{\mathrm{meta}}+=w_4$; $F\leftarrow F\cup \left\{\mathrm{INFO\_MISMATCH}\right\}$ 17: end if 18: // Stage 5: Behavioral urgency signals at registration 19: if $\mathcal{A}.\mathrm{urgency\_score}>{\tau }_{\mathrm{urgency}}=0.6$ or non-standard payment method requested then ▹ urgency_score: normalized count of urgency-signaling keywords in registration notes 20:   $s_{\mathrm{meta}}+=w_5$; $F\leftarrow F\cup \left\{\mathrm{URGENCY}\right\}$ 21: end if 22: $s_{\mathrm{meta}}\leftarrow min(s_{\mathrm{meta}}, 1.0)$                    ▹ Weights $w_i\in (0,1]$ are calibrated on historical labeled accounts 23: return $s_{\mathrm{meta}}, F$

4.2.2. Ongoing Monitoring (Countermeasure: AS2)

When an account has been successfully onboarded, it can be exposed to fraudulent activity. This can happen in several scenarios, such as the abuse of free trials through the creation of multiple accounts to consume the trial value excessively, or the use of stolen payment methods. In addition, operational misuse of an account may also occur, for example when staff manually change an account to a paid down-payment plan. In this stage, many anomaly signals are monitored, including shared payment methods, payment methods previously flagged as fraudulent, calls from different accounts to the same callees, the use of an account for multiple businesses, and failed DID checks against blacklist services such as Truecaller. These signals are then combined into a time-varying anomaly score, $s_{\mathrm{mon}}$.

4.2.3. Escalation Logic

The metadata path outputs a fused score:

$$s=\mathrm{FUSESCORES}(s_{\mathrm{meta}}, s_{\mathrm{mon}}),$$

(1)

where $\mathrm{FUSESCORES}(·)$ is implemented as a weighted sum $s=\alpha ·s_{\mathrm{meta}}+(1-\alpha )·s_{\mathrm{mon}}$. Calibration was performed on 120 labeled historical accounts (80 training/40 validation, stratified by fraud label, with no account-level leakage between splits). The objective function was the F1-score of the binary escalation decision. Grid search over $\alpha \in \{0.4,0.5,0.6,0.7,0.8\}$ and ${\tau }_{\mathrm{escalate}}\in [0.10,0.90]$ (step 0.05) on the held-out validation set identified $\alpha =0.6$ and ${\tau }_{\mathrm{escalate}}=0.35$ as optimal, yielding the observed 15–20% escalation rate in the production deployment. Figure 2 plots escalation precision, recall, and F1 as a function of ${\tau }_{\mathrm{escalate}}$, confirming a stable operating region between 0.30 and 0.40. Decision thresholds ${\tau }_{\mathrm{flag}}$ and ${\tau }_{\mathrm{sem}}$ (Algorithm 2) were separately tuned on the same validation set after fixing the metadata thresholds.

Table 2. Summary of calibrated pipeline parameters.

Parameter	Search Range	Objective	Selected Value
$\alpha$ (fusion weight)	$\{0.4,\dots ,0.8\}$	Escalation F1	0.6
${\tau }_{\mathrm{escalate}}$	$[0.10,0.90]$	Escalation F1	0.35
${\tau }_{\mathrm{flag}}$	$[0.20,0.60]$	Pipeline precision	0.40
${\tau }_{\mathrm{sem}}$	$[0.60,0.85]$	Pipeline precision	0.72
N (LLM calls)	$\{1,2,3\}$	Company-name acc.	3
$\theta$ (logprob threshold)	$[0.85,0.99]$	Accuracy–coverage	0.95
$\varphi$ (fuzzy threshold)	manual review (30 dev)	—	40

summarizes all calibrated parameters. Accounts with $s\ge {\tau }_{\mathrm{escalate}}$ are passed to the call-content path.

Algorithm 2 End-to-EndFraud Detection Pipeline

Require:  Account $\mathcal{A}$, metadata stream $\mathcal{M}$, call audio set $\mathcal{C}$, thresholds ${\tau }_{\mathrm{escalate}},{\tau }_{\mathrm{flag}}$ Ensure:  Fraud flag $y\in \{0,1\}$, evidence report $\mathcal{R}$   1: $\mathcal{R}\leftarrow \varnothing$                                       ▹ Initialize empty evidence accumulator   2: // Path 1: Metadata Screening   3: $s_{\mathrm{meta}},F_{\mathrm{onboard}}\leftarrow \mathrm{ONBOARDINGVERIFY}\left(\mathcal{A}\right)$                                  ▹ Algorithm 1   4: $s_{\mathrm{mon}}\leftarrow \mathrm{MONITORANOMALIES}\left(\mathcal{M}\right)$ ▹ Three CDR signals: payment-account sharing, callee-set overlap, burst call-rate; see Section 4 AS2   5: $s\leftarrow \alpha ·s_{\mathrm{meta}}+(1-\alpha )·s_{\mathrm{mon}}$ ▹ Weighted fusion; $\alpha =0.6$ (calibrated on labeled history; Path 1 given higher weight as lower-cost signal)   6: $\mathcal{R}.\mathrm{add}(s_{\mathrm{meta}},s_{\mathrm{mon}},F_{\mathrm{onboard}})$   7: if $s<{\tau }_{\mathrm{escalate}}$then   8:   return $(y=0,\mathcal{R})$                        ▹ Account passes metadata screening; no content analysis   9: end if 10: // Path 2: Call-Content Analysis (escalated accounts only) 11: for each call $c\in \mathcal{C}$ do 12:   $t\leftarrow \mathrm{LANGA\_ASR}\left(c\right)$                        ▹ Domain-adapted Whisper; RTF ≈ 0.03–0.06 on NVIDIA T4 13:   $\mathbf{k}\leftarrow \mathrm{KEYWORDFILTER}(t,\mathrm{ArabicKeywordList})$                  ▹ 8 fraud-related terms; normalized hit count 14:   $q\leftarrow \mathrm{SUMMARIZE}\left(t\right)$                      ▹ MSA-normalized call summary for cross-dialectal retrieval 15:   ${\mathbf{v}}_q\leftarrow \mathrm{ARABERT\_ENCODE}\left(q\right)$ 16:   $\mathcal{N}\leftarrow \mathrm{FAISS\_ANN}({\mathbf{v}}_q,K=10)$                   ▹ Top-K similar summaries; IVF index, nprobe = 64, <5 ms 17:   $\ell \leftarrow \mathrm{CONSENSUS\_VERIFY}(t,\mathcal{A}.\mathrm{org})$                         ▹ Algorithm 3; $N=3$, $\theta =0.95$, $\varphi =40$ 18:   $\mathcal{R}.\mathrm{add}(\mathbf{k},\mathcal{N},\ell )$ 19: end for 20: // Decision and Human Validation 21: $r_{\mathrm{fraud}}\leftarrow \left|\{c:\ell \left(c\right)=\mathrm{MISMATCH}\}\right| / \left|\mathcal{C}\right|$ 22: $y\leftarrow \mathbb{I}[r_{\mathrm{fraud}}\ge {\tau }_{\mathrm{flag}}$ or ${\overline{s}}_{\mathrm{sem}}\left(\mathcal{R}\right)\ge {\tau }_{\mathrm{sem}}]$  ▹${\tau }_{\mathrm{flag}}=0.4$; ${\tau }_{\mathrm{sem}}=0.72$ (cosine similarity mean over top-K neighbors, tuned on validation set) 23: if $y=1$  then 24:   $\mathrm{HUMANVALIDATE}(\mathcal{A},\mathcal{R})$                     ▹ Analyst reviews top evidence calls and structured report 25: end if 26: return$(y,\mathcal{R})$

Figure 2 shows precision, recall, and F1 with respect to the escalation threshold. As the escalation threshold increases, precision increases while recall decreases. An escalation threshold of ${\tau }_{\mathrm{escalate}}=0.35$ was selected because it provides a good balance among precision, recall, and F1, making it a suitable operating point.

Algorithm 3 LLM Multi-Call Consensus Entity Verification

Require:  Transcript t, registered org $\mathcal{A}.\mathrm{org}$, number of calls $N=3$, logprob threshold $\theta =0.95$, fuzzy threshold $\varphi =40$ Ensure:  Entity label $\ell \in \{\mathrm{MATCH},\mathrm{MISMATCH},\mathrm{UNKNOWN}\}$   1: $\mathrm{responses}\leftarrow \left[\right]$   2: for  $i=1$ to N do   3:   $\mathrm{temp}\leftarrow 0.0$ if $i=1$ else $0.9$         ▹ Call 1: deterministic few-shot (Prompt A); Calls 2–3: stochastic zero-shot (Prompt B)   4:   $\mathrm{prompt}\leftarrow \mathrm{PROMPT\_A}$ if $i=1$ else $\mathrm{PROMPT\_B}$   5:   $(e_i,{\pi }_i)\leftarrow \mathrm{LLM\_EXTRACT}(t[0:200\mathrm{chars}],\mathrm{prompt},\mathrm{temp})$     ▹ Restrict to agent-intro segment; ${\pi }_i$ is mean token log-probability   6:   $\mathrm{responses}.\mathrm{append}\left((e_i,{\pi }_i)\right)$   7: end for   8: // Consensus check: all  N extractions must agree   9: if $\left|\right\{e_i:(e_i,{\pi }_i)\in \mathrm{responses}\left\}\right|=1$ then                         ▹ All calls returned identical entity string 10:   $\overline{\pi }\leftarrow {\displaystyle \frac{1}{N}}{\sum }_{i=1}^{N}exp\left({\pi }_i\right)$                             ▹ Convert mean log-prob to linear probability 11:   if  $\overline{\pi }\ge \theta$ then                                         ▹ High-confidence consensus 12:     $e^{\ast }\leftarrow e_1$ 13:     $v\leftarrow \mathrm{FUZZYWUZZY}(e^{\ast },\mathcal{A}.\mathrm{org})$          ▹ Levenshtein-based partial ratio; accommodates Arabic morphological variants 14:     if  $v\ge \varphi$ then 15:      return MATCH 16:     else 17:      return MISMATCH              ▹ Agent claimed organization differs from registered name: impersonation signal 18:     end if 19:   end if 20: end if 21: return UNKNOWN                        ▹ No consensus or low confidence; case escalated to human review

4.3. Call-Content-Based Detection Path

4.3.1. Automatic Speech Recognition

Five commercial and open-source ASR systems were evaluated:

• Meta M4T V1: A massively multilingual model supporting 200 languages with encoder–decoder architecture and mean-pooling representations [32].
• Whisper Large V1: Trained on 680,000 h of multilingual audio [33], using encoder–decoder Transformer with log-Mel spectrogram input.
• Chirp: Google’s ASR technology optimized for challenging noisy environments [34].
• Google Cloud Speech-to-Text Application Programming Interface (API): Wide language support including Arabic dialects with real-time transcription.
• Microsoft Azure Speech-to-Text API: Customizable Arabic speech recognition across audio environments.

We also evaluated Langa [35], a Whisper-backbone model fine-tuned on the target telephony domain using progressive domain adaptation [36,37], was evaluated as the candidate deployment model.

ASR Benchmark Dataset

The benchmark dataset was annotated by 41 annotators and reviewed by 13 reviewers and consists of more than 132 h of high-quality Arabic telephony speech for the benchmark task [28]. The calls were collected via a cloud telephony provider that services clients across the Arab world. All calls were anonymized and, in line with the Terms of Service of the provider, these data were made available for processing. The dataset includes spoken Arabic from six main dialect groups (Egyptian, Levantine, Gulf dialects, Iraqi dialects, Maghrebi dialects, and MSA-dominant speech), involving speakers from 13 countries, across the Arab world. The collected data cover a number of applications like education, entertainment, and e-commerce.

Audio is standardized at 16 kHz and encompasses a wide dynamic range of channel conditions—from clean studio-quality recordings to heavily degraded telephony noise—ensuring realistic evaluation of ASR robustness in production deployments. Annotation quality was cross-checked by the 13-reviewer panel described in [28]; a formal inter-annotator agreement statistic (e.g., Cohen’s $\kappa$) is not reported in that benchmark study, and we do not impute one here.

Evaluation Methodology

ASR quality is assessed using WER and CER. We also run blind human comparisons in which annotators vote for the better transcript without knowledge of model identity.

Operational Throughput and Privacy

The Langa model was profiled on the production deployment environment. Processing a 1.5 min call (the mean call duration) requires approximately 3–5 s on a single NVIDIA T4 Graphics Processing Unit (GPU), yielding a real-time factor (RTF) of ≈0.03–$0.06$. This supports processing of up to 80,000 calls/day with 2–3 parallel GPU workers—consistent with typical medium-sized cloud call center volumes—via asynchronous batch processing with no latency constraint on the fraud decision. All data are processed under a personally identifiable information (PII)-minimization principle: only features required for detection are retained, manual review is restricted to escalated accounts, and call recordings were processed under operator Terms of Service consent in accordance with applicable PDPL requirements; individual call participant consent was obtained by the cloud telephony operator under its Terms of Service [28]. This privacy-by-design approach aligns with General Data Protection Regulation (GDPR) and Personal Data Protection Law (PDPL) requirements applicable in the MENA region [38].

4.3.2. Keyword Filtering for High-Risk Lexical Cues

We first perform a light, exact-match keyword search to pre-process the data. This search is used as a fast triage, and we use a list of Arabic words related to fraud. The selected words have the following English equivalents: scam, fraud, password, card CVC/number, OTP/verification code, complaint, communications commission, and theft. The corresponding Arabic terms are: [nasb, ihtial, murur, ramz bitaqa, ramz tahaqquq, shakwa, hayat al-ittisal, and sariqa].

For each account, we compute normalized keyword hit counts over the corresponding call corpus and generate a lexical-risk vector, $\mathbf{k}\left(\mathcal{A}\right)$. While keyword counts cannot serve as definitive evidence of fraudulent behavior, they contribute to generating $\mathcal{R}$ and account-risk scores, which are used to rank accounts for further review. We integrate keyword search with semantic retrieval techniques to distinguish between legitimate business-related vocabulary and high-risk vocabulary.

4.3.3. Semantic Search over Standardized Call Summaries

We implemented semantic retrieval to address the brittleness of exact matching in Arabic (Countermeasure: AS3 at the semantic level). Figure 3 shows the two-phase retrieval pipeline. In the offline phase (steps 1–5), call recordings are transcribed by the Langa ASR model, summarized in Modern Standard Arabic to normalize dialectal variation, encoded into dense vectors by AraBERT, and stored in a Facebook AI Similarity Search (FAISS) Inverted File (IVF) index; this build step requires approximately 15 min for 100 K summaries and is performed once. In the online phase (steps A–D), an analyst query is encoded with the same AraBERT weights (shared encoder) and matched against the index via Approximate Nearest Neighbor (ANN) cosine-similarity search, returning the top-K most relevant call summaries in under 5 ms on a Central Processing Unit (CPU) (FAISS IVF, nprobe = 64).

Retrieval Targets

Summaries are searched to identify the following: (i) complaints about scams, spam, or fraud; (ii) discussion deviating from known account use-cases; (iii) personal conversations inconsistent with business operations; and (iv) solicitation of financial or personal data.

Embedding and Indexing

Summaries and queries are encoded using Arabic-capable sentence-transformer models: bert- base-arabic and bert-base-arabertv2 (https://huggingface.co/asafaya/bert-base-arabic, https://huggingface.co/aubmindlab/bert-base-arabertv2, accessed on 15 January 2024) [29,39]. Similarity is computed via cosine distance. ANN search uses the FAISS library [40] with an inverted-file index. The semantic retrieval layer outputs top-K summaries with similarity scores.

Algorithm 1 formalizes the onboarding risk scoring procedure. Each of the five stages contributes a calibrated penalty $w_i\in (0,1]$ to a cumulative score $s_{\mathrm{meta}}$, which is clipped to $[0,1]$. The resulting flag set F is appended to the evidence report $\mathcal{R}$ for downstream human review. The per-stage weights (

Table 3. Onboarding risk score weights $w_i$ (calibrated on labeled historical accounts by maximizing escalation F1 on a held-out 20% split).

Stage	Signal	Weight ${\mathit{w}}_{\mathit{i}}$
1	Missing documents	0.35
2	Email/domain risk	0.20
3	Fake or unreachable website	0.20
4	Cross-field inconsistency	0.15
5	Registration urgency	0.10

) are calibrated on a labeled subset of historical accounts by maximizing escalation F1 on a held-out 20% split.

4.3.4. Behavioral and Entity-Based Verification via LLMs

Many cloud call centers are exposed to another fraud technique named misrepresentation. This typically happens in the initial stages of the fraud attempt, when the attacker presents themselves as being from a legitimate company. In many cases, the attacker may present themselves under a different identity, including one associated with another country, that conflicts with the information linked to the registered account. This type of behavior is captured by our behavioral analysis component.

Datasets

Two manually labeled datasets are used for evaluation.

Dataset-165: (Dataset of 165 Call Transcripts of a telephone customer service channel of a cloud telephony provider in the MENA region) This dataset consists of 165 call transcripts of a telephone customer service channel containing voice recordings of incoming calls to a cloud telephony provider operating in the MENA region. All the transcripts were manually annotated to include the name of the speaking agent and the name of the organization this agent works for. These annotations were carried out at the start of the call where the agent announces his name and/or the name of the organization he works for. All the calls in the dataset are from different accounts and mostly in Arabic language.

Dataset-50: Comprised of 50 individually annotated calls. This data was recorded *after* the prompt had been optimized for overall accuracy and can serve as a basis for an assessment of the ability to generalize. The data is also used as a control to check for prompt bias and was not used to generate this prompt.

Both of these datasets can be used for developing a Call Management software. They are based on real calls taken place in production environment, not generic. These are real transcripts, not taken from the public domain or crowd-sourced.

Task-Specific Named Entity Recognition (NER) Models

Arabic BERT-based NER models demonstrate strong performance on standard benchmarks (F1: 80–94% across MSA, Classical, and dialectal variants) [41], but performance drops sharply on noisy conversational ASR transcripts. The best-performing Hugging Face model was fine-tuned to recognize names from the initial 200 characters of calls, where agent introductions typically occur. Extracted entities are matched to reference names using the FuzzyWuzzy Levenshtein-distance library; manual verification is conducted on all matches with similarity ≥ 20%. Evaluation uses extraction accuracy—the fraction of calls where the extracted name matches the ground-truth label—reported separately for agent names and company names. Precision, recall, and F1-score are also tracked: precision measures whether the extracted entities are correct, recall measures the coverage of labeled entities, and F1 combines both.

LLM-Based Extraction (Countermeasure: AS4)

Large language models are prompted to extract agent and company names. We ran experiments with GPT-3.5, GPT-4 [42], and Jais [43]—a 13-billion parameter model pre-trained on Arabic and English. The following verification strategies are evaluated: (i) prompt-structure: few-shot versus zero-shot; (ii) segment selection: agent-only versus agent-plus-customer; (iii) multi-call consensus: multiple LLM calls with varied temperature, accepting only cases where answers agree; (iv) logprob filtering: log-probability analysis with a 95% linear probability threshold to prioritize high-confidence extractions.

Reproducibility and Model Versioning

For all GPT-based experiments, we used the OpenAI API and accessed the gpt-3.5-turbo and gpt-4 models during the period from Q4 2023 to Q1 2024. For the primary extraction call using Prompt A, we set the temperature parameter to $0.0$ and used a few-shot, deterministic setting. In contrast, for the subsequent call computing divergence using Prompt B, we set the temperature to $0.9$ and used a zero-shot, highly stochastic setting. We limited the maximum output length of each call to 150 tokens and found this to be sufficient for our named entity extraction task.

We also experimented with direct use of Jais [43] via its publicly hosted API, using a 13B-parameter checkpoint. In that case, most of the cost came from hosting the model rather than the per-call API price.

We used FuzzyWuzzy v0.18 with Levenshtein distance for string matching. The similarity threshold of 40 was manually tuned on about 30 development cases and was not changed afterward.

We cannot guarantee numerical reproducibility for results obtained using OpenAI API models, as they are dynamic and may change over time. However, we include the prompts and hyperparameters in the code, which should allow for approximate reproducibility using another instruction-following LLM of similar capability.

Rationale for Experimental Dimensions

We examined four specific dimensions along which the prompting strategy could be varied: (1) prompt structure, (2) segment selection, (3) multi-call consensus, and (4) logprob filtering. These dimensions were selected after several alternative design choices were eliminated through experimentation and were found to offer the greatest potential for improving performance.

We also tested a single-call self-verification approach, in which GPT-4 was asked to verify the correctness of its own extraction outputs from prior interactions. However, this approach was found to be unreliable, as GPT-4 frequently assigned high correctness scores to responses that were in fact incorrect. Similarly, asking GPT-4 to rate its own confidence did not provide useful information. We also explored chain-of-thought prompting, but it did not improve extraction accuracy.

The four retained dimensions address different but complementary failure modes. Prompt structure leverages the known sensitivity of LLMs to in-context examples in order to improve extraction consistency through carefully designed few-shot prompts. Segment selection improves extraction by focusing on the parts of the conversation that are most likely to contain identity-related information, rather than processing the entire conversation. Multi-call consensus helps reduce hallucinations by querying the model in multiple ways and retaining only cases in which the outputs are consistent. Logprob filtering allows the system to abstain from uncertain token-level predictions rather than producing potentially incorrect extractions.

Algorithm Relationships

In this paper, Algorithm 2 presents the master pipeline, coordinating the two detection paths from start to finish. More specifically, two new algorithms have been integrated into the master pipeline. When an account enters Path 1, the pipeline calls Algorithm 1 to retrieve the account’s registration information and compute an onboarding risk score. When an account is escalated to Path 2, the pipeline calls Algorithm 3, which queries the LLM multiple times to verify the declared identity of the agent against the registered organization name of the account. These two algorithms are designed to be self-contained, each handling a particular stage of the detection pipeline.

The three key parameters in Algorithm 3 were determined empirically. The number of LLM calls, N, was selected by comparing $N\in \{1,2,3\}$ on a validation set of 200 examples. Among these values, $N=3$ gave the highest accuracy for company-name extraction and was adopted.

The threshold $\theta =0.95$ was determined through analysis of the accuracy–coverage trade-off on the validation set. Lowering the threshold $\theta$ caused the system to accept more extractions, but with a higher error rate. Increasing the threshold improved extraction accuracy at the cost of leaving more cases unresolved. A value of $0.95$ provided the best balance between these two competing objectives.

The fuzzy-matching threshold, $\varphi =40$, was fixed before the main experiments. This value was selected after a manual review of 30 development examples. It was not changed afterward in order to avoid inadvertent tuning on the test data.

Entity Matching and Coverage

Extracted entities are verified against registered account metadata using FuzzyWuzzy (Levenshtein distance) with a threshold of 40. This threshold accommodates LLM inconsistencies such as partial names or morphological variants common in Arabic (e.g., elision of the definite article al-, dialectal vowel shifts). Beyond extraction accuracy (correctness on matched cases), we report coverage—the proportion of calls for which the model returns a high-confidence answer rather than abstaining. The interplay between accuracy and coverage defines the operational cost–precision trade-off: high-confidence filtering raises accuracy at the cost of reduced coverage, with unmatched cases escalated to human review.

4.4. Decision Layer, Evidence Fusion, and Human Validation

4.4.1. Evidence Report Construction

The system accumulates the following: (i) keyword hits with contextual excerpts; (ii) retrieved summaries with similarity scores; (iii) extracted entities with confidence proxies; and (iv) verification scores comparing extracted organizations to the registered name. The resulting report $\mathcal{R}$ supports operational triage with account-level synopsis, call-level highlights, and entity consistency comparisons.

4.4.2. Decision Rule

The account-level semantic similarity score ${\overline{s}}_{\mathrm{sem}}\left(\mathcal{R}\right)$ used in Algorithm 2 is defined as the mean cosine similarity of each call’s MSA-summary embedding against its top-K FAISS neighbors, averaged across all calls in the account:

$${\overline{s}}_{\mathrm{sem}}\left(\mathcal{R}\right)={\displaystyle \frac{1}{\left|\mathcal{C}\right|}}\sum _{c\in \mathcal{C}}{\displaystyle \frac{1}{K}}\sum _{k=1}^{K}sim\left({\mathbf{v}}_{q_c}, {\mathbf{v}}_k\right),$$

(2)

where $sim(·,·)$ is cosine similarity, ${\mathbf{v}}_{q_c}$ is the AraBERT embedding of call c’s MSA summary, and ${\mathbf{v}}_k$ are the top-K index-neighbor embeddings (see Section 4.3.3). The binary risk flag is then as follows:

$$y=\mathbb{I}\left[r_{\mathrm{fraud}}\ge {\tau }_{\mathrm{flag}} \mathrm{or} {\overline{s}}_{\mathrm{sem}}\left(\mathcal{R}\right)\ge {\tau }_{\mathrm{sem}}\right],$$

(3)

where conservative thresholds minimize false positives. $y=1$ triggers an escalation rather than automatic enforcement.

4.4.3. Human-in-the-Loop Validation Protocol

When an account is flagged ($y=1$), a fraud analyst reviews the top-evidence calls and structured report $\mathcal{R}$. The analyst confirms or rejects the suspicious classification. This protocol provides the following: (a) a final recall correction layer that can recover accounts missed by automated stages; (b) adversarial robustness against subtle evasion attempts that automated systems might accept; and (c) an auditable decision record for regulatory compliance. Algorithm 2 summarizes the complete pipeline.

Human Validation Protocol Details. Validation was conducted by two senior fraud analysts employed by the cloud telephony provider, each with more than three years of cloud telephony fraud investigation experience. Predefined review criteria required assessment of the following: (i) mutual consistency of agent self-introductions, call content, and account registration data; (ii) presence of sensitive-information solicitation; and (iii) alignment of declared business purpose with observed calling patterns. Analysts reviewed the top 3–5 evidence calls per account together with structured report $\mathcal{R}$. Analysts had access to the flagged signals in $\mathcal{R}$ but were instructed to form independent judgments and not to treat model flags as dispositive. Disagreements arose in 4 of the 47 flagged accounts and were resolved by joint consensus review. Formal inter-rater agreement statistics (e.g., Cohen’s $\kappa$) were not computed for the main evaluation; this is acknowledged as a limitation, and future validation studies should include pre-registered review criteria and blinded analyst assessment.

Algorithm 3 describes the LLM-based multi-call consensus procedure used to verify whether an agent’s declared organizational identity matches the registered account name. The algorithm issues $N=3$ independent extraction calls to the LLM for the same transcript segment. The first call uses a deterministic temperature ($\mathrm{temp}=0.0$) with a few-shot prompt (Prompt A), which maximizes extraction consistency and serves as the primary answer. The subsequent two calls use a high-temperature stochastic setting ($\mathrm{temp}=0.9$) with a zero-shot prompt (Prompt B); these calls act as independent divergence probes. Restricting input to the first 200 characters of the transcript targets the agent self-introduction segment, where organizational claims are made most explicitly. Consensus is declared only if all N calls return the identical entity string. This deliberate asymmetry between a deterministic anchor call and stochastic verification calls sharply cuts hallucination risk: a fabricated or inconsistent entity will rarely survive agreement across all three calls. The mean linear log-probability $\overline{\pi }$ provides an additional confidence gate; extractions with $\overline{\pi }<\theta =0.95$ are rejected as uncertain and escalated to human review (UNKNOWN). For accepted extractions, FuzzyWuzzy Levenshtein matching with threshold $\varphi =40$ accommodates common Arabic morphological variants such as elision of the definite article al- and dialectal vowel shifts. A MISMATCH label—where the extracted organization name does not match the registered account name—constitutes a strong impersonation signal fed into the evidence report $\mathcal{R}$.

Algorithm 2 integrates Path 1 (metadata screening), Path 2 (call-content analysis), and human validation into a single executable procedure. Key implementation parameters are as follows: escalation weight $\alpha$ (calibrated on labeled history), ${\tau }_{\mathrm{escalate}}$ (threshold yielding 15–20% escalation rate), $K=10$ ANN neighbors, and ${\tau }_{\mathrm{flag}}$ (maximizing precision on validation set). Flagged accounts are routed to human review rather than automatic enforcement, preserving an auditable decision record.

5. Results

5.1. ASR Performance

Table 4. Comparative ASR performance (WER and CER). Bold indicates the best result in each column.

Model	WER (%)	CER (%)
Langa (domain-adapted)	41.0	18.2
Chirp	48.9	22.4
Meta M4T V1	67.8	34.3
Google API	67.1	40.6
Azure API	71.9	39.0
Whisper Large V1	83.8	52.3

compares the WER and CER values of all evaluated ASR systems. The domain-adapted Langa model achieves the best performance, with the lowest WER (41.0%) and CER (18.2%) among all methods. It outperforms all commercial baselines. Compared with the strongest commercial baseline, Chirp, Langa reduces WER by 7.9 percentage points and CER by 4.2 percentage points. The gap is even larger for Whisper Large V1, the weakest commercial baseline, which performs 42.8 percentage points worse than Langa in WER. These results highlight the benefit of telephony-domain fine-tuning.

Blind listening experiments confirm these gains perceptually: annotators preferred Langa over Meta M4T in 99.1% of pairwise comparisons, over Google in 97.3%, and over Azure in 92.4% (

Table 5. Blind human listening comparison: Langa vs. Meta M4T (vote %).

Model	Votes (%)
Langa	99.1
Meta M4T	0.2
None preferred	0.7

,

Table 6. Blind human listening comparison: Langa vs. Google Cloud Speech (vote %).

Model	Votes (%)
Langa	97.3
Google	0.4
Both preferred	1.4
None preferred	0.9

and

Table 7. Blind human listening comparison: Langa vs.Microsoft Azure Speech (vote %).

Model	Votes (%)
Langa	92.4
Azure	3.5
Both preferred	3.0
None preferred	1.1

). Based on these results, Langa was selected as the ASR model for the operational pipeline.

Dialect-stratified WER results for the Langa model are available from the benchmark evaluation [28]. The model achieves substantially lower WER on MSA-dominant and Gulf Arabic calls (where training data density is highest) than on Maghrebi and Iraqi dialect calls (where labeled data is sparsest). Readers requiring a full dialect-stratified breakdown are referred to [28]; we summarize the pattern here to contextualize the aggregate 41% WER figure: performance is markedly stronger for the dominant dialects represented in the production dataset.

To quantify how ASR quality propagates to downstream fraud detection, we re-ran the LLM entity extraction pipeline (GPT-3.5, few-shot, agent-only segments) on transcripts produced by three ASR systems spanning a wide WER range.

Table 8. ASR error propagation to downstream entity extraction (GPT-3.5, few-shot, agent-only segments, dataset-165).

ASR Model	WER (%)	Company-Name acc. (%)	Agent-Name acc. (%)
Langa (domain-adapted)	41.0	62.0	65.0
Chirp	48.9	58.3	61.2
Whisper Large V1	83.8	44.1	52.7

reports company-name and agent-name extraction accuracy for each system. Company-name accuracy degrades monotonically with WER: from 62.0% (Langa, WER 41.0%) to 58.3% (Chirp, WER 48.9%) to 44.1% (Whisper Large V1, WER 83.8%). The accuracy gap between Langa and Chirp (3.7 pp) is smaller than the corresponding WER gap (7.9 pp), confirming that MSA-normalized summarization partially absorbs transcription noise. At the high-WER extreme (Whisper Large V1), a 42.8 pp WER penalty translates into a 17.9 pp accuracy loss, establishing a clear lower bound on acceptable ASR quality for operational deployment. Domain-adapted ASR is therefore a security necessity, not merely a performance optimizer.

Note on scope: the propagation analysis above quantifies the effect of ASR quality on LLM entity extraction. A parallel quantification of propagation to semantic retrieval precision/recall—the other downstream module—would require ground-truth relevance labels for the FAISS index queries, which are not available in the current production evaluation. We acknowledge this as a limitation: the semantic retrieval layer is expected to be more robust to ASR noise than entity extraction (because it operates on MSA-normalized sentence-level summaries rather than exact tokens), but this claim remains qualitative. Dedicated evaluation of retrieval precision/recall under degraded ASR conditions is recommended as future work.

5.2. Task-Specific NER Model Accuracy

Task-specific NER-based extraction achieved limited performance, with 24.7% accuracy for company-name extraction and 68.0% accuracy for agent-name extraction. The NER model was selected from approximately eight Arabic BERT-based NER models available on Hugging Face, all pre-trained on the base BERT architecture with F1 scores of 80–94% on standard benchmarks (MSA, Classical Arabic, and dialectal variants) [41]. The best-performing model was fine-tuned on our 165-call labeled domain dataset using standard subword tokenization, without character-level features or dialectal augmentation. Fine-tuning targeted the first 200 characters of each call transcript, where agent self-introductions typically occur. The low company-name accuracy (24.7%) reflects the combined effect of the following: (i) the small fine-tuning set (165 examples), (ii) the distributional mismatch between clean benchmark text and noisy ASR output, and (iii) the absence of domain-specific augmentation. Note that the reported 24.7% refers to extraction accuracy (fraction of calls where the extracted entity matches ground truth after Levenshtein matching); precision, recall, and F1 were tracked during Hugging Face model selection across the eight candidate models but were not separately computed for the final fine-tuned model on our domain dataset. These results motivated the integration of LLM-based behavioral verification.

5.3. LLM-Based Entity Extraction Experiments

5.3.1. Prompt Design and Input Segmentation

Table 9. Impact of prompt structure on extraction accuracy.

Prompt Structure	Agent (%)	Company (%)
Few-shot – GPT	65.0	62.0
Zero-shot – GPT	48.0	58.0
Few-shot – Jais	63.7	59.3
Zero-shot – Jais	48.2	56.8

shows that few-shot prompting boosts entity extraction accuracy noticeably. For GPT, few-shot prompting improves agent-name accuracy from 48% to 65% (+17 percentage points) and company-name accuracy from 58% to 62% (+4 percentage points). Jais shows comparable gains (+15.5 pp for agent-name).

Conversation Segment Selection

Table 10. Impact of conversation segment selection on extraction accuracy.

Segments	Agent (%)	Company (%)
Agent only – GPT	65.0	62.0
Agent + Customer – GPT	58.0	67.0
Agent only – Jais	66.8	61.8
Agent + Customer – Jais	58.4	66.5

shows the effect of including customer speech. Using only agent segments yields higher agent-name accuracy; including customer segments improves company-name accuracy but introduces noise. Including first and last five segments was essential to capture company mentions during service-satisfaction closings.

5.3.2. Model Selection and Generalization

Table 11. Extraction accuracy across LLM variants.

Model	Agent (%)	Company (%)
GPT-4	78.0	71.0
GPT-3.5	65.0	62.0
Jais	66.8	61.8

shows GPT-4 achieves the highest accuracy (78% agent; 71% company), but at approximately 30× the cost of GPT-3.5 (USD 15 vs. USD 0.50 per 500 K input tokens + 1.5 K output tokens), motivating cost-aware model selection.

Prompt Bias Assessment

Table 12. Prompt bias assessment: accuracy across datasets.

Test Set	Model	Agent (%)	Company (%)
50 examples	GPT-3.5	57.1	75.5
165 examples	GPT-3.5	72.0	67.7
50 examples	GPT-4	81.6	89.7
165 examples	GPT-4	73.9	80.1
50 examples	Jais	58.4	77.5
165 examples	Jais	71.7	66.1

shows no evidence of prompt overfitting. GPT-4 achieves 81.6% agent-name and 89.7% company-name accuracy on the held-out dataset-50, comparable to or exceeding performance on dataset-165.

5.3.3. Consensus and Confidence-Aware Filtering

Table 13. Multi-call consensus verification results.

Model	Dataset	Agree (%)	Company (%)	Agent (%)
GPT-4	165 examples	68	97.3	68.3
GPT-4	200 examples	77	96.0	76.8
GPT-3.5	165 examples	57	89.0	56.2
GPT-3.5	200 examples	65	86.8	64.8

presents the multi-call consensus strategy results. Prompt A: temperature 0.0, 10-example few-shot. Prompt B: temperature 0.9, zero-shot. Consensus pushes company-name accuracy: 97.3% for GPT-4 (68% agreement) and 89.0% for GPT-3.5 (57% agreement). Among incorrect three-call responses, the majority were “no answer” rather than wrong answer—a preferred failure mode for fraud detection.

Log-Probability Confidence Filtering

Combining the three-call strategy with logprob filtering at a 95% linear probability threshold raises GPT-3.5 company-name accuracy from 89.0% (three-call consensus without logprob filter; 18% coverage) to 92.0% (three-call consensus with logprob filter; 13% coverage), with the remainder escalated to human review.

The operational pipeline therefore adopts GPT-3.5 with three calls and log-probability filtering, achieving 92.0% company-name accuracy at manageable computational cost.

LLM Cost Accounting

Table 14. LLM cost accounting for deployed and alternative configurations (OpenAI pricing, Q4 2023–Q1 2024).

Configuration	Cost/Call	Cost/Account	Cost/Cohort (150–200 acc.)
GPT-3.5, 3 calls + logprob (deployed)	$0.0055	$0.20	$30–$40
GPT-3.5, 1 call	$0.0018	$0.07	$10–$14
GPT-4, 3 calls + logprob	$0.165	$6.10	$915–$1220
GPT-4, 1 call	$0.055	$2.04	$306–$408
Jais (13B)	Hosting cost only; no per-call API pricing

reports absolute cost estimates for the deployed configuration and alternatives, based on OpenAI pricing applicable during Q4 2023–Q1 2024. Each call requires approximately 1500 input tokens and 150 output tokens across 3 API calls. At a mean of 37 calls per account, the per-account LLM processing cost under the deployed GPT-3.5 configuration is approximately USD 0.20. For the full escalated cohort of ∼150–200 accounts, total LLM cost is approximately USD 30–USD 40 per six-month monitoring cycle. The GPT-4 configuration would cost approximately 30× more (USD 6.00/account), amounting to USD 900–USD 1200 for the same cohort. Jais inference cost is dominated by hosting rather than per-call API pricing and is not directly comparable.

To support reproducibility despite periodic API model updates, all prompts, parameter settings (temperature, logprob threshold $\theta =0.95$, and fuzzy threshold $\varphi =40$), and extraction procedures are fully documented here. All experiments were conducted with the specific model snapshots available during Q4 2023–Q1 2024; researchers replicating results with later model versions should treat API-generation differences as a potential source of variation.

5.4. End-to-End Pipeline Evaluation

5.4.1. Pipeline Results

The complete fraud detection pipeline was applied to a production cloud telephony environment in the MENA region. The evaluation set comprised 1024 active subscriber accounts accumulated over a six-month operational monitoring period, totaling approximately 38,000 recorded calls (mean 37 calls/account). Accounts were not pre-selected: the full population processed by the metadata screening stage is included. The class distribution reflects natural production imbalance. The large majority are legitimate business subscribers; a small fraction was suspected of fraud-related activity based on prior operational intelligence, yielding an estimated fraud base rate of approximately 4% (∼40 accounts), consistent with the precision and recall estimates reported below.

Table 15. End-to-end pipeline evaluation results.

Metric	Value
Total accounts screened	>1000
Accounts escalated by metadata path	∼150–200 (∼15–20%)
Accounts flagged as suspicious	47
Confirmed fraudulent (human review)	41
False positives	6
Precision	87.2%
95% CI (precision)	(74.3%, 95.2%)

summarizes the end-to-end results.

5.4.2. Confusion Matrix and Statistical Analysis

Table 16. Confusion matrix and statistical metrics for the end-to-end pipeline evaluation. TP and FP are directly observed; FN and TN are estimated under the 4% base-rate assumption. Recall and F1 are bounded estimates, not directly measured outcomes.

	Pred. Fraud	Pred. Benign
True Fraud	True Positive (TP) $=41$	False Negative (FN) ≈ 9–39 (est.)
True Benign	False Positive (FP) $=6$	True Negative (TN) ≈ 914–944 (est.)
Precision	$87.2\%$ (95% CI: $74.3$–$95.2\%$)
Recall (est.)	51–$82\%$ (conservative range)
F1 (est.)	63–$84\%$ (estimated range)

presents the confusion matrix structure with statistical analysis. Because the true number of fraudulent accounts in the population is unknown in real-world deployments, direct recall measurement is not possible without intrusive investigation of all accounts. Therefore, recall is conservatively estimated based on observed fraud prevalence and escalation rates, following methodologies commonly used in operational fraud detection studies. Two bounding assumptions are applied:

Scenario-based recall framing: Under a nominal 4% base-rate assumption over >1000 screened accounts, the implied fraud count is about 40. Because this quantity is assumption-based (and subject to rounding/uncertainty), the ratio $41/40$ is not interpreted as a directly measured recall value and should not be read as evidence of perfect detection.

Conservative missed-detection bound: The metadata escalation rate of 15–20% means that approximately 800–850 accounts were not subjected to full conversational analysis. If fraud prevalence among non-escalated accounts mirrors or exceeds the deployed base-rate assumptions, the estimated total fraud population is ≈50–80 accounts, implying estimated recall in the range $\widehat{r}\approx 41/50=82\%$ (less conservative) to $41/80=51\%$ (more conservative). Accordingly, we report recall as an assumption-dependent scenario bound (51–82%), not as a directly observed metric.

Confidence interval derivation: The precision CI ($74.3$–$95.2\%$) is computed using the Wilson score interval for a binomial proportion with $n=47$ trials and $k=41$ successes at $\alpha =0.05$.

Recall framing. Precision was directly observed, but recall and F1-score were estimated based on certain assumptions and not directly measured. The false negative (FN) and true negative (TN) scores in the confusion matrix were calculated based on the assumed 4% fraud base rate and therefore should be considered as scenario-based estimates, not as directly observed values.

Recall sensitivity analysis. 

Table 17. Recall sensitivity analysis under varying base-rate assumptions (41 confirmed detections; >1000 accounts screened).

Assumed Base Rate	Est. Fraud pop.	Recall (Optimistic)	Recall (Conservative)
2%	∼20	∼100%	51–100%
3%	∼31	∼100%	51–84%
4%	∼41	∼100%	51–82%
5%	∼51	80%	41–66%
6%	∼61	67%	34–54%

Optimistic: all fraud in escalated accounts. Conservative: fraud prevalence mirrors overall population in non-escalated accounts.

presents the estimated recall under five assumed fraud base rates, ranging from 2% to 6%, and two escalation coverage scenarios for the reported elevated risk indicator. In the optimistic scenario, all fraudulent accounts in this study are assumed to be contained within the elevated risk group, which accounts for 15% to 20% of all accounts. In the conservative scenario, the fraction of fraud in the non-escalated population is assumed to be equal to the overall fraud prevalence in the population.

Under the 4% base-rate assumption and the conservative scenario, the estimated recall ranges from 51% to 82%, but drops to 34–54% at the highest considered base rate of 6%. The thresholds used in our study can be adjusted to provide improved results.

False positive analysis. The six false positives correspond to 12.8% of escalated accounts (6/47) and were concentrated in legitimate financial-services and insurance accounts. Their call content frequently contained terms such as “verification code” and “card number,” which lexically and semantically overlap with fraud cues despite benign intent. This pattern indicates that the dominant FP mode is domain-vocabulary collision rather than random model error. Operationally, the cascade and human-review design prevents these cases from becoming automatic enforcement actions and suggests that sector-aware lexical weighting and intent-level context features are the most direct levers for further FP reduction.

6. Security Analysis

We present a reasoned security analysis of each attack surface identified in Section 3.3. Each subsection characterizes the countermeasures in place, the adversary cost required for evasion, and residual vulnerabilities. These are analytical assessments informed by system design; they are not empirically demonstrated robustness results. Dedicated adversarial red-team evaluation is recommended as future work.

6.1. Countermeasures Against Onboarding Evasion (AS1)

An adversary attempting to pass onboarding verification must simultaneously fabricate or steal the following: (a) business registration documents, (b) consistent email and domain attribution, (c) a plausible website, (d) consistent cross-field registration data, and (e) payment instruments without prior flagging. Each requirement independently adds adversary cost and risk. The multi-signal aggregation in Algorithm 1 means that partial evasion of one check (e.g., using a legitimate email provider) is insufficient if other signals (e.g., website placeholder text; cross-field inconsistencies) remain anomalous.

Residual vulnerability: A sufficiently resourced adversary who acquires a complete legitimate business identity (full credential set) may pass onboarding. This scenario underscores the necessity of continuous monitoring and conversational analysis as downstream detection layers.

6.2. Countermeasures Against Metadata Camouflage (AS2)

The cascade design of the architecture explicitly anticipates that more sophisticated adversaries will attempt to mimic legitimate calling patterns at the metadata level. However, metadata evasion alone is insufficient: any account exceeding the escalation threshold still undergoes conversational-level verification regardless of its CDR profile. In the deployed system, the 15–20% escalation rate reflects an empirical calibration that balances the computational cost of false escalations against detection coverage.

Residual vulnerability: Adversaries operating at extremely low call volume or maintaining highly diverse callees may suppress metadata anomaly scores below the escalation threshold. This is the primary source of false negatives in the current deployment, as discussed in the recall estimation (Section 5.4). No formal adversarial evaluation of the escalation gate has been performed; this is a recognized limitation. Two partial mitigations exist: (i) the onboarding verification path (AS1) is independent of call volume, so stealth operators must still pass document, email, website, and cross-field consistency checks; (ii) the monitoring component tracks payment instrument sharing and callee overlap across accounts, which can surface stealth operators through shared infrastructure even at low individual volume. Dedicated adversarial evaluation of low-volume stealth fraud patterns is recommended as a priority for future work.

6.3. Countermeasures Against ASR Evasion (AS3)

Two countermeasures address ASR evasion. First, the domain-adapted Langa model is trained on telephony speech under noisy and dialectally diverse conditions, yielding a WER of 41.0% versus 83.8% for the general-purpose Whisper baseline. Second, semantic retrieval operates over MSA-normalized call summaries rather than raw transcript tokens, making detection robust to token-level ASR errors: while ASR errors corrupt individual word boundaries, sentence-level semantic meaning is typically preserved in the summary.

Residual vulnerability: Severely degraded ASR quality—such as agents speaking in heavily code-switched dialects or deliberately obscuring speech—reduces entity extraction quality. The baseline 41% WER indicates some information loss is unavoidable; further gains require continued telephony domain adaptation.

6.4. Countermeasures Against LLM Prompt Injection (AS4)

The multi-call consensus approach (Algorithm 3) is a principled defense against prompt injection. For an adversary to successfully manipulate entity extraction, adversarial text embedded in a call must produce a consistent, identical false extraction across all N independent LLM calls with temperature variation—since consensus requires agreement. The probability of a single successful injection is far higher than the probability of consistent injections across N independent queries. Adversarially manipulated extractions also tend to produce lower log-probability scores, which the logprob threshold rejects as low-confidence outputs.

Residual vulnerability: A well-informed adversary with knowledge of the precise LLM prompt structure could craft persistent adversarial inputs that survive temperature variation. Maintaining confidentiality of prompt structures and rotating prompts periodically are therefore recommended operational practices. A second residual vulnerability concerns the 200-character input window: a sophisticated adversary who deliberately delays organizational identity claims beyond the agent introduction segment may reduce extraction effectiveness. This is partially mitigated by the segment selection strategy, which also includes call-closing segments (Table 10); however, adaptive segment identification that dynamically locates the most identity-informative regions remains a recommended direction for future hardening.

Overall Security Posture

The architecture is consistent with a security-in-depth design principle: no single evasion strategy defeats all layers simultaneously. To remain undetected, an adversary must pass onboarding checks (AS1), maintain metadata camouflage (AS2), defeat semantic retrieval (AS3), and defeat LLM verification (AS4). The attack surfaces are not fully independent: a well-resourced adversary capable of passing onboarding (AS1) may also have the means to maintain metadata camouflage (AS2), and joint exploitation of ASR evasion (AS3) with prompt injection (AS4) is plausible. The security-in-depth property holds under the weaker condition that the joint evasion probability across all layers is far lower than that of any individual layer, even with positive correlation. Dedicated empirical evaluation of joint attack scenarios—including red-team testing of the escalation gate and LLM injection under degraded ASR conditions—is recommended as future work. Human-in-the-loop final validation provides an additional layer that cannot be automated and is robust to any fully automated evasion strategy, since human analysts apply contextual judgment that no single algorithmic signal can replicate.

7. Discussion

7.1. Cross-Cutting Observations

Our results point to several cross-cutting implications for system design. Domain adaptation is not merely a performance optimization but a security necessity: the 17.9 percentage-point WER gap between Langa and Chirp propagates directly into downstream entity extraction errors, and a higher WER would likely push end-to-end precision below operationally acceptable levels. A less obvious finding concerns keyword filtering and semantic retrieval, which turned out complementary rather than interchangeable: high-precision lexical matching and high-recall semantic search each cover distinct failure modes of Arabic’s dialectal variability, and neither alone is sufficient. Ablation observations confirm this complementarity: in operational deployment, removing the semantic retrieval stage drove up false positives from keyword matches alone, as legitimate calls containing fraud-adjacent vocabulary (e.g., financial services terminology) were incorrectly flagged without the semantic context check. The six false positives reported in Table 15 exemplify this pattern. The gap between NER and LLM-based extraction (24.7% vs. up to 97.3% company-name accuracy) was larger than we expected, suggesting that for noisy short ASR transcripts, general instruction-following capability outweighs task-specific fine-tuning. Jais deserves mention separately: it matches GPT-3.5 accuracy while offering a self-hostable, Arabic-native alternative that avoids third-party data exposure—a practical advantage for privacy-sensitive MENA-region deployments. Figure 4 plots all evaluated configurations on the cost-accuracy plane; the deployed configuration (GPT-3.5; three calls; log-probability filtering) achieves 92.0% accuracy at only $3\times$ the cost of a single GPT-3.5 call, placing it on the Pareto frontier.

Operational implications of 13% high-confidence coverage. The logprob filtering step yields high-confidence company-name extractions for only 13% of individual calls, with the remainder escalated as UNKNOWN. This per-call rate does not mean 87% of accounts are unresolved: because signals are aggregated across all calls for an account (mean 37 calls), the account-level resolution rate is well above 13%. Under the deployed configuration, approximately 720–960 calls from the 150–200 escalated accounts yield confident extractions—sufficient to make an organizational identity determination for most accounts. UNKNOWN accounts (those where no call yields a high-confidence extraction) are routed directly to human review. Future work on calibrated confidence models could support autonomous enforcement for the highest-confidence cases.

Analyst workload analysis. In the current deployment, 47 accounts were flagged as suspicious over a six-month monitoring period. Based on operational experience, reviewing one account (3–5 top evidence calls plus report $\mathcal{R}$) requires approximately 20–30 min. Total analyst workload for 47 accounts is therefore approximately 16–24 h—manageable for a single fraud analyst per cycle. If fraud volume scales to $10\times$ (approximately 400 flagged accounts), workload would increase to 130–200 h per six-month period, requiring a small dedicated team. Integration of calibrated confidence models for autonomous enforcement of high-confidence cases would reduce this workload proportionally.

7.2. Limitations and Future Directions

Recall measurement. In this paper, precision is a directly computed from confirmed fraud cases in this production system. Recall and F1, by contrast, are bounded estimates derived under explicit assumptions and should be interpreted as a scenario analysis rather than measured outcomes. Because the true number of fraudulent accounts is unknown, measuring recall is challenging. For now, recall cannot be measured directly but can be estimated under different assumptions, yielding an estimated range of 51–82%. Better measurement of recall will require further research. A notable deficiency in the current literature is the lack of controlled experiments conducted under normal operating conditions to test recall [17,44].

ASR quality dependency. Despite advances in entity extraction quality, the process is still strongly influenced by the quality of the transcripts. A 41% WER results in some extraction errors that are unlikely to be eliminated through further domain adaptation or improved speech segmentation. While 41% appears high compared to typical results reported on clean-speech benchmarks for standard languages, spoken Arabic in a telephony setting can be particularly difficult to recognize, including in this collection, due to dialectal variation, adverse recording conditions, and code-switching. Typical benchmarks do not capture these factors well, as shown in Table 8, indicating that the system has only partial robustness to ASR noise errors.

Metadata threshold calibration. The current threshold for escalating cases (15–20%) was chosen based on experience rather than a formal process. Future work should systematically characterize the precision–recall trade-off as a function of the escalation threshold to enable principled threshold selection.

Human validation protocol. The human validation protocol is given in Section 4.4, including analyst qualifications, predefined review criteria, and disagreement resolution procedure. Formal inter-rater agreement was not computed; future studies should include pre-registered criteria and blinded assessment.

Generalization. A single provider in the MENA region has been used for the evaluation of this work. Extending the system to support additional dialects, languages, and regulatory settings remains an important challenge for future work. The keyword set was intentionally kept small and domain-general (8 terms) in order to reduce the recalibration burden. For each new provider deployment, threshold recalibration is recommended using a locally labeled subset of approximately 80–120 accounts.

Prompt injection hardening. The system has not been evaluated against prompt injection attacks in the LLM-based extraction stage, and the current design does not preclude such attacks. Dedicated adversarial testing and red-teaming exercises are therefore needed in future work to verify robustness under realistic attack conditions. These evaluations should also cover joint attack scenarios targeting AS3 and AS4 concurrently, since real-world attackers typically exploit multiple vulnerabilities rather than a single weakness.

Autonomous decision-making. Further work on improving the system’s trust and calibration could allow clear cases to be flagged automatically, freeing analysts to review uncertain or difficult cases.

8. Conclusions

This work provides an operationally verified architecture for fraudulent activity detection and addresses an important gap in the literature, namely the lack of production deployment studies on fraud detection for Arabic cloud telephony. Our results show that fraudulent conversations can be detected in production for low-resource languages by combining domain-adapted automatic speech recognition, hybrid semantic transcript retrieval, and LLM-based behavioral verification, without requiring labeled training data or domain-specific model pre-training. This work formalizes a threat model, develops a domain-adapted ASR component, introduces a hybrid semantic transcript retrieval component, and implements an LLM-based behavioral verification component, all evaluated on large-scale production data.

The domain-adapted Langa ASR model achieved a WER of 41.0% and CER of 18.2%, outperforming all evaluated commercial and open-source baselines, with a real-time factor of 0.03–0.06 suitable for high-volume production deployment. For entity verification, LLM-based extraction with multi-call consensus achieved up to 97.3% company-name accuracy (GPT-4, 68% agreement rate) and 92.0% in the cost-effective operational configuration (GPT-3.5; three calls; log-probability filtering). The complete pipeline was validated on production cloud telephony data in the MENA region: of 47 flagged accounts, 41 were confirmed fraudulent after human review, yielding a directly observed precision of 87.2% (95% CI: 74.3–95.2%) and an estimated (not directly measured) recall of 51–82% under conservative base-rate assumptions.

The security analysis (Section 6) is consistent with a security-in-depth design principle: the architecture is designed such that simultaneous evasion of all four attack surfaces requires far more adversary capability than bypassing any individual layer—a design property consistent with security-in-depth, though not empirically demonstrated via red-team testing. The privacy-by-design principles and human-in-the-loop enforcement model make the architecture well-suited to the regulatory environments of the MENA region.

The most pressing next step is direct recall measurement via controlled injection experiments or retrospective sampling of non-escalated accounts; without it, the 51–82% recall range remains a scenario estimate, not a measured outcome. The proposed framework can be extended to other low-resource languages used in telephony and its performance can be evaluated and verified under a different set of operational challenges. Additionally, adaptive thresholds represent a promising direction for dynamically tracking evolving fraud patterns without requiring a full retuning cycle.