Beyond Semantic Noise: A Dual-Verification Framework for Thai–English Code-Mixed Malicious Script Detection via XAI-Guided Selective Integration

Abstract

In the evolving cybersecurity landscape, detecting Thai-English code-mixed malicious scripts within high-trust domains such as governmental and academic portals presents a significant defensive challenge. While Transformer-based architectures excel in semantic parsing, they often exhibit ‘Structural Bias,’ misinterpreting the high-entropy syntax of benign legacy HyperText Markup Language (HTML) as malicious obfuscation due to inherent ‘Attention Deficit’ in token-limited models. To address this, we propose an Explainable AI (XAI)-Driven Hybrid Architecture grounded in a ‘Selective Integration’ strategy. Unlike traditional hybrid models, our framework mathematically formalizes the fusion process by synergizing context-aware WangChanBERTa embeddings with orthogonal structural statistics through Dempster-Shafer Theory and Conditional Mutual Information (CMI). The proposed model was validated on a high-fidelity corpus, achieving a state-of-the-art F1-score of 0.9908, significantly outperforming standalone Transformers, Random Forest, and unsupervised baselines. XAI diagnostics revealed a ‘Dual-Validation’ mechanism where structural features act as an epistemic anchor. This mechanism effectively triggers a ‘Semantic Veto’ to filter hallucinations caused by benign complexity, achieving a remarkably low False Positive Rate (FPR) of 0.0116. Our findings demonstrate that hybridization is most effective when engineered features provide mathematical orthogonality to semantic embeddings. This work offers a robust, theoretically grounded framework for securing critical digital infrastructures in low-resource linguistic environments.

1. Introduction

Identifying malicious scripts embedded within dynamic web content is increasingly challenging in contemporary cybersecurity. Adversaries have evolved from using straightforward code injection to sophisticated “code-mixing” strategies, particularly in low-resource linguistic environments. In contexts such as Thai government portals, attackers employ a dual-obfuscation technique where malicious JavaScript payloads are seamlessly interwoven with Thai natural language comments and HTML structures [1,2,3]. This specifically exploits the “Language Gap” in standard models [4]: Regex filters miss fragmented payloads, while global models (such as English-trained Bidirectional Encoder Representations from Transformers (BERT)) treat localized Thai characters as ‘unknown tokens’ or noise. Consequently, traditional security systems struggle to distinguish between the high-entropy syntax of legacy HTML and genuine malicious obfuscation, leading to critical “Structural Bias.”

Code-mixed web scripts represent a unique input domain that defies traditional malicious detection classifications [5,6]. While Transformer architectures have demonstrated efficacy in semantic parsing [7], applying them directly to mixed-script detection presents two theoretical limitations: First, they often function as “black boxes,” lacking the transparency required for forensic accountability [8,9]. Second, and more critically, purely semantic models suffer from ‘Attention Deficit’ when processing long, structurally complex HTML files, leading to hallucinations where benign code structures are misclassified as threats. Recent studies have explored Hybrid Architectures to bridge this gap by combining semantic encoders with structural features; however, most existing approaches still rely on simple feature concatenation without considering semantic redundancy [10,11,12].

To address these limitations, this study proposes an XAI-Driven Hybrid Architecture. Unlike traditional approaches that merely use XAI for post hoc explanation, we utilize XAI as a diagnostic tool for architectural design [13]. We challenge the prevailing ‘Hybrid-is-Better’ orthodoxy by identifying that straightforward fusion often produces “Semantic Noise”, redundant signals that hinder model convergence. We introduce a theory of “Selective Integration,” hypothesizing that hybridization is effective only when context-aware embeddings (from WangChanBERTa) are fused with orthogonal, outlier-robust structural statistics. In this design, WangChanBERTa captures the nuances of Thai–English semantics, while handcrafted structural features act as a discriminative ‘Safety Net’ to veto semantic errors [14].

We validated this architecture using a high-fidelity corpus curated from over 28,077 raw samples, comparing it against standalone Transformer and handcrafted baselines. To ensure methodological rigor, we employed a suite of XAI methods including SHapley Additive exPlanations (SHAP) and Attention Analysis not only to explain results but also to quantify the ‘Orthogonality’ between semantic and structural features [15,16]. The essential contributions include the following:

• XAI-Driven Architectural Design: We demonstrate how XAI diagnostics can uncover “Semantic Noise” in traditional fusion, leading to a refined “Selective Integration” strategy that outperforms blind concatenation.
• Correction of Structural Bias in Low-Resource Domains: By integrating WangChanBERTa, we address the specific challenge of Thai–English code-mixing, reducing the False Positive Rate by 52.4% compared to baseline models.
• Empirical Validation of Dual-Verification: We provide empirical evidence of a “Dual-Verification” mechanism, where structural features stabilize the volatile predictions of Transformers, offering a robust framework for high-trust digital infrastructures.

2. Literature Review

The classification and detection of malicious web content, including hidden online gambling scripts, has progressed through various technological stages: transitioning from basic, manually curated feature sets to sophisticated deep semantic modeling [17,18]. This review consolidates the existing literature emphasizing traditional techniques (Random Forest), contemporary multimodal approaches (Hybrid models), and the latest advancements in pure semantic modeling (Transformer models). It ultimately positions XAI as an essential diagnostic tool for scientifically evaluating model performance [19,20].

2.1. The Traditional Paradigm: Handcrafted Features and the Random Forest Baseline

The initial and most persistent techniques for identifying malicious counting Uniform Resource Locators (URLs), phishing, and web content employed conventional machine learning classifiers, particularly Random Forest (RF), that functioned on meticulously designed, handcrafted feature sets [10,21,22]. These features are generally organized into specific categories:

• Lexical Features: Analyzing attributes such as script length, character distributions, or entropy to identify anomalies. Lexical features represent the most commonly utilized attributes in research on both Arabic and non-Arabic content for identifying malicious URLs.
• Content Features: Analyzing the text, structure, or content components. This includes URLs and domains and identifying potentially harmful HTML tags (e.g., <iframe>, <script>).
• Network-Centric Attributes: Incorporating parameters such as Internet Protocol (IP) addresses, domain names, and certificate analysis.

Advancements in Semi-Supervised Sequential Modeling: To provide a comprehensive perspective beyond standard supervised learning, it is essential to consider advancements in latent variable frameworks. Recent studies have introduced the Semi-supervised Sequential Variational Autoencoder (SSVAE), which addresses the challenges of sequential text classification by leveraging unlabeled data to learn robust latent representations. Furthermore, recent contributions building upon foundational research on self-attention often associated with Google Research have led to the development of the Self-supervised Sequential VAE (S3VAE) [23]. This architecture utilizes a dual-stream self-attention mechanism to disentangle time-independent and time-varying factors in sequential manifolds. While these generative models demonstrate superior resilience in modeling high-variance data distributions, they often require significant computational resources and may struggle with the specific structural nuances of code-mixed HTML compared to discriminative anchors [11].

The Limitations of Conventional Proxies: While traditional machine learning models such as Random Forests can achieve high accuracy, they possess a significant operational drawback due to their dependence on statistical proxies rather than true semantic understanding [24]:

• Heuristic Gap: RF models primarily determine content by analyzing the frequency and configuration of manually designed features (e.g., keyword counting). This implies that string matching algorithms are prone to evasion techniques.
• Structural Bias: As RF lacks the capability to interpret the sequential context of code-mixed scripts, it relies heavily on “structural rules” (e.g., ‘high entropy = malicious’). This leads to an increased False Positive Rate (FPR) when legitimate code minification or complex Thai language encoding is incorrectly identified as harmful obfuscation.
• Lack of Intent Analysis: The operation of RF, which involves aggregating outputs from numerous decision trees, lacks the necessary semantic comprehension to accurately discern the intent behind the code. Consequently, it fails to distinguish between a benign <iframe> used for a government banner and a malicious one used for gambling redirection.

2.2. The Ascendancy of Hybrid and Multimodal Fusion Approaches

Recognition of the limitations inherent in single-feature or single-modality detection methods has sparked a notable shift towards hybridization and integrating multimodal data fusion. This approach leverages data from various sources, such as images and text, that complement each other to enhance the system’s robustness, particularly in identifying illicit content such as pornographic and gambling websites [25]. A 2022 study proposed a Hybrid Multimodal Data Fusion-based Method for identifying gambling websites through capturing screenshots and integrating visual and semantic features. This approach addresses the limitations of existing methods that overlook text embedded within website images, which frequently contains crucial information indicative of gambling content [26].

The technical procedure for effective hybridization encompasses the following:

• Visual Feature Extraction: Optimizing the pre-trained Deep Residual Network (ResNet34) model to derive visual features from screenshots.
• Semantic Feature Extraction: Implementing Optical Character Recognition (OCR), a technique quantifiable using metrics such as Levenshtein distance, to derive textual information from screenshots. The OCR text is subsequently utilized alongside pre-trained Word-to-Vector (Word2Vec) embeddings and a Bidirectional Long Short-Term Memory (Bi-LSTM) layer for extracting semantic features. Notably, the OCR text exhibited more robust semantic features compared to HTML text, which might include extraneous content or be devoid of clear gambling-related keywords as a result of evasion strategies.
• Feature Fusion: Utilizing a self-attention mechanism to integrate visual and semantic features, followed by a Late Fusion approach to amalgamate the prediction outcomes from image, text, and multimodal classifiers.

Comparative Analysis of Fusion Strategies: While hybridization is promising, the method of fusion is critical. Reviewing the literature reveals three primary strategies: (1) Late Fusion (Voting), which is robust but ignores feature interactions; (2) Intermediate Fusion (Attention-based), which is computationally expensive; and (3) Early Fusion (Vector Concatenation). Our architecture adopts an Early Fusion strategy based on Vector Concatenation. Although this design simplifies the verification of feature synergy, previous studies often lack comparison with other mechanisms. We argue that, for malicious script detection, Early Fusion allows a model to learn the mathematical orthogonality between structural statistics and semantic embeddings before the classification layer, effectively preventing the “Semantic Noise” observed in Late Fusion approaches. This aligns with findings in other domains, such as battery health assessment and video object segmentation, where ‘state’ memory (analogous to our structural features) is used to stabilize high-variance predictions [27,28].

The resulting hybrid method demonstrated exceptional performance, achieving accuracy, precision, recall, and F1-score metrics exceeding 99% on the collected dataset. When compared to existing hybrid paradigms, our approach offers a distinct shift in fusion logic [12]. While systems such as DRSDetector employ ‘Reinforcement Fusion’ where multiple modalities vote to increase prediction confidence our framework introduces the concept of ‘Corrective Fusion’ [29].

Mathematical analysis of our results confirms this advantage: while simple probability averaging (Late Fusion) yields an F1-score of 0.9895, our Orthogonal Early Fusion approach achieves a superior 0.9908. This empirical delta suggests that our architecture does more than just accumulate signals; it actively utilizes structural variance to veto semantic hallucinations in high-entropy, low-resource contexts. The effectiveness of this hybrid strategy is further supported by its performance in related domains, such as phishing detection, where the integration of web-scraped features has historically achieved high precision and recall (98.91% and 97.94% respectively) compared to baseline methods.

The efficacy of hybridizing time-series/structural data with deep learning has been validated in diverse domains, such as battery health assessment and video object segmentation. These fields similarly employ ‘state’ memory (analogous to our structural features) to stabilize the high-variance predictions of deep neural networks [19,30].

2.3. The Pure Semantic Paradigm: Transformer Architectures

The introduction of the Transformer model, as established via the “Attention Is All You Need” framework, signifies a shift from traditional Feature Engineering to semantic modeling. These models utilize the self-attention mechanism to effectively capture intricate context and long-range dependencies within input sequences [31].

The structural design of The Decoding-enhanced BERT with Disentangled Attention (DeBERTa): The DeBERTa model, which stands for Decoding-enhanced BERT with Disentangled Attention, signifies a notable progression in the Transformer architecture. DeBERTa advances the original BERT architecture through implementing two significant innovations, namely the disentangled attention mechanism and the improved mask decoder:

• The disentangled attention mechanism enhances the model’s capability to comprehend text by isolating the representations of content and relative position;
• The model demonstrates superior performance in relation to the size of its training data.

Transformer models demonstrate exceptional performance in classification tasks, especially in domains that demand a profound understanding of context:

• General Text Classification: For short-text tasks such as news headline classification, the DeBERTa-large variant achieved an F1-score of 91.21%, outperforming traditional pre-trained models including BERT, RoBERTa, and XLNet.
• Sentiment Analysis: Evaluated against the SMILE Twitter dataset, DeBERTa attained a peak F1-score of 0.861, surpassing BERT, ALBERT, and RoBERTa, thereby validating its robustness in capturing nuanced emotional contexts.
• Domain-Specific Adaptation: The development of SciDe-BERTa highlights the architecture’s flexibility. By initializing with general-domain parameters and subsequently fine-tuning on specialized scientific and technological data, the model significantly enhances its comprehension of niche technical vocabularies.
• Cybersecurity and Phishing Detection: In specialized security contexts, DeBERTa V3 demonstrated a recall of 95.17% for phishing detection, exceeding GPT-4 (91.04%). Notably, DeBERTa exhibits a superior capability in processing sequential data formats, such as URLs, whereas Large Language Models (LLMs) typically perform better on structured web or HTML content.

Applicability to Low-Resource Languages (Thai Context): A critical gap in the current cybersecurity literature is the handling of code-mixed threats in Low-Resource Languages such as Thai. While multilingual models such as Multilingual BERT (mBERT) exist, they often underperform in specific linguistic nuances compared to monolingual models. The study emphasizes the superiority of WangChanBERTa in processing Thai code-mixed texts. Unlike standard ThaiBERT, which focuses on general prose [32], WangChanBERTa (based on the RoBERTa architecture) provides robust embedding spaces that better handle the “code-switching” nature of our dataset [14]. Comparative experiments with other Thai-specific pre-trained models are essential to verify the necessity of selecting WangChanBERTa, particularly to ensure that the model captures the subtle interplay between Thai comments and JavaScript logic [33].

Recent advancements in semi-supervised learning, such as the Self-Attention Sequential Variational Autoencoder (SAS-VAE), demonstrate the utility of latent space disentanglement for anomaly detection. While VAEs excel at modeling ‘normal’ traffic distributions, our analysis of the 28,077-file corpus revealed that ‘benign’ legacy code in government sectors exhibits such high variance (high entropy) that it often falls outside the latent distribution of VAEs. Thus, we opted for a discriminative Hybrid approach that explicitly models these ‘benign anomalies’ via handcrafted structural features, rather than relying solely on reconstruction error.

Unlike natural language, where local context often implies meaning, malicious scripts exhibit ‘Syntactic Discontinuity’ a variable defined at the start that may be executed maliciously 500 lines later via an eval() function. While Transformers capture attention, the token limit (512 tokens) often truncates these dependencies. Our Hybrid Architecture compensates for this ‘Attention Deficit’ by utilizing global structural statistics (e.g., entropy, char_distribution) that summarize the entire file, effectively bridging the gap between local semantics and global syntax.

2.4. XAI as a Diagnostic Tool: Diagnosing Semantic Noise

Although deep learning models exhibit impressive performance, their intricate nature frequently categorizes them as “black box” systems. The primary objective of this diagnostic phase is to systematically evaluate the effectiveness of model components, focusing on pinpointing harmful features and identifying inconsistencies among features.

Enhancing the Hybridization Hypothesis: Although the existing literature frequently posits that integrating handcrafted features with embeddings universally enhances performance, our research calls this sweeping assertion into question. We assert that the ‘Hybrid-is-Better’ hypothesis is valid solely in scenarios characterized by Selective Integration. This study presents empirical evidence through the diagnosis of feature interactions, demonstrating that redundant keyword counters serve as “semantic noise”, whereas specific structural features, such as Character Types and Statistics, play a crucial role as ‘Safety Nets’ that enhance the Transformer’s contextual reasoning capabilities.

Root Cause Analysis of Feature Redundancy: This research utilizes XAI as a fundamental architectural design instrument rather than a mere post hoc transparency tool. Specifically, our SHAP analysis identifies “Web” (e.g., URL count), “Encoding”, and “Security” feature groups as redundant “semantic noise”. We posit that the root cause of this redundancy is the strong expressive ability of Transformer embeddings, which already cover the information contained in these simple counters. Including them creates a “mismatch” between these shallow features and the deep semantic features of the dataset, leading to model overfitting. Therefore, our “Selective Integration” approach removes these redundant groups, retaining only the structural features that provide true Orthogonal Information. This process transforms XAI from a passive explanatory framework into an active mechanism for driving model efficiency and innovation.

3. Materials and Methods

This research employed a systematic methodological framework designed to address the “Structural Bias” inherent in detecting code-mixed web scripts. Unlike traditional pipelines that treat detection and explanation as separate phases, our approach integrates XAI as a core diagnostic component to ensure both high detection fidelity and forensic transparency.

Overview of Research Pipeline: Figure 1 illustrates the comprehensive 8-stage research methodology. The workflow begins with (1) Data Collection from EduShield and GambleGuard, followed by (2) Data Preprocessing, which entails performing rigorous cleaning and expert verification to ensure high-fidelity input. This feeds into (3) Feature Engineering, through which we generate two distinct feature sets: handcrafted structural features and Transformer-based semantic embeddings. These are evaluated across three architectures in (4) Model Architectures for Comparison: (4A) Random Forest, (4B) Transformer, and (4C) the proposed Hybrid model.

Figure 1. The XAI-Driven Research Methodology Pipeline. The workflow integrates High-Fidelity Data Curation with a ‘Selective Integration’ Hybrid Architecture. Crucially, the process utilizes XAI as a diagnostic feedback loop (Stages 6–7) to identify “Semantic Noise” and validate the correction of Structural Biases.

The Role of Diagnostic XAI: Following (5) Model Evaluation via 5-Fold Cross-Validation, the pipeline transitions to its critical phase: (6) XAI analysis. Here, we utilize techniques such as SHAP and ablation studies not merely for observation but as inputs for (7) Performance Diagnosis and Conflict Analysis. This stage explicitly investigates False Positives and resolves conflicts between semantic and structural signals, culminating in (8) Conclusions and Recommendations on the efficacy of the “Transformer Hero” synergy mechanism.

3.1. High-Fidelity Dataset Curation

To guarantee the forensic integrity of our analysis, we assembled a targeted dataset concentrating on code-mixed scripts found within high-trust domains, particularly governmental (root domain .go.th) and academic (root domain .ac.th) websites [34]. The raw corpus was obtained from two dedicated real-time monitoring systems, EduShield and GambleGuard, created in partnership by the National Cyber Security Agency (NCSA) and Rajamangala University of Technology Lanna (RMUTL).

Absence of Low-Resource Benchmarks: To the best of our knowledge, there is no publicly available dataset specifically dedicated to Thai–English code-mixed gambling and spam scripts. Existing benchmarks focus predominantly on English-centric threats or generic phishing URLs, which fail to capture the unique syntactic interleaving of Thai characters and JavaScript payloads. This scarcity arises from two factors: the linguistic niche of Low-Resource Languages and the strict privacy regulations surrounding compromised governmental domains. Consequently, establishing a foundational, high-fidelity corpus is a prerequisite for advancing cybersecurity research in this region.

Formalization of Curation Protocol: To address the lack of standardized benchmarks and ensure reproducibility, we formalized our curation process as a rigorous filtering function. Starting with a raw dataset of 28,077 HTML files $\left(D_{raw}\right)$, the final high-fidelity corpus $\left(D_{final}\right)$ was derived via the intersection of three indicator functions:

$$D_{final}=\{x\in D_{raw}|{\varphi }_{dim}\left(x\right)\wedge {\varphi }_{lang}\left(x\right)\wedge {\varphi }_{expert}\left(x\right)\}$$

(1)

where the following definitions are used:

• Dimensionality Constraint ${\varphi }_{dim}$: Filters out fragments smaller than 500 bytes.

  $${\varphi }_{dim}\left(x\right)=\left\{\begin{array}{c}1 if Size\left(x\right)\ge 500 bytes\\ 0 \end{array}\right.$$

  (2)

  Rationale: Empirical analysis indicates that files below this threshold typically consist of fragmented HTML headers or broken JavaScript stubs (approx. <50 tokens). Such sparsity prevents the Transformer’s self-attention mechanism from establishing meaningful contextual dependencies, leading to “feature collapse” during embedding.
• Linguistic Relevance ${\varphi }_{lang}$: Ensures the presence of dual-language tokens (Thai/English) to specifically target the code-mixing problem:

  $${\varphi }_{lang}\left(x\right)=\left\{\begin{array}{c}1 if x\cap {\tau }_{Thai}\ne \varnothing AND x\cap {\tau }_{Eng}\ne \varnothing \\ 0 otherwise \end{array}\right.$$

  (3)

  Rationale: A mandatory intersection ensures that the dataset captures the “Language Switching Cost” inherent in obfuscated scripts. Single-language samples (pure English or pure Thai) were discarded to prevent the model from learning trivial language-detection shortcuts instead of structural malice patterns.
• Expert Consensus ${\varphi }_{expert}$: A strict majority voting mechanism created by 3 CompTIA Security+ analysts was employed to eliminate ground-truth noise and ensure forensic validity.

This rigorous formulation reduced the raw dataset from 28,077 to 5000 verified samples, purposefully trading quantity for “Label Purity” to support accurate XAI diagnosis.

$${\varphi }_{expert}\left(x\right)=\left\{\begin{array}{c}1 if {\displaystyle \sum }_{i=1}^{3}Labe{l}_i\left(x\right)=3 \left(Unanimous Agreement\right)\\ 0 otherwise \end{array}\right.$$

(4)

Rationale: We employed an odd number of annotators ($K$ = 3) to allow for tie-breaking, but enforced a stricter “Unanimous” (3/3) criterion rather than a simple majority. This mitigated subjective bias in “grey-zone” cases, ensuring that the final dataset represents a “Gold Standard” ground truth essential for valid XAI interpretation.

Final Corpus Composition: The final corpus consisted of 5000 high-fidelity samples, perfectly balanced between two classes:

• Malicious Class ($N$ = 2500): Samples extracted from compromised servers identified during Blackhat SEO campaigns, featuring real-world obfuscated gambling and spam injection scripts;
• Benign Class ($N$ = 2500): Samples derived from authentic administrative scripts (e.g., JavaScript library (jQuery), LMS modules) operating on identical server architectures.

Analysis of Selection Bias and Generalization: We acknowledge that reducing the dataset from 28,077 to 5000 (as summarized in Table 1) introduces Selection Bias by excluding ambiguous or “grey-zone” scripts. However, this trade-off is intentional. By training on a “Clean-Label” dataset, we ensure that the decision boundaries learned by the model are robust and based on clear semantic–structural patterns rather than artifacts of mislabeled training data. This “Purified Learning” approach allows us to use XAI diagnostics to accurately trace causal mechanisms, ensuring that the model generalizes well to the specific threat landscape of high-trust Thai portals.

Table 1. The characteristics of the curated high-fidelity dataset. The final corpus was meticulously reduced from 28,077 raw files down to 5000 verified samples to eliminate labeling noise. It features a balanced distribution of Thai–English code-mixed scripts specifically collected from high-trust governmental (.go.th) and academic (.ac.th) domains [34].

Characteristic	Value	Description
Data Source	EduShield and Gamble Guard	Monitoring systems for >2000 Thai governmental domains.
Initial Raw Collection	28,077	Total raw HTML files collected from EduShield and Gamble Guard.
Final Selected Samples (N)	5000	Total samples used for training/testing after balancing.
Malicious Class	2500	Verified code-mixed injections (gambling/spam) with obfuscated JavaScript payloads.
Benign Class	2500	Legitimate administrative scripts (e.g., jQuery, LMS modules) exhibiting high structural entropy (to test Structural Bias).

3.2. Model Selection and Fine-Tuning Details

Rationale for Selecting WangChanBERTa: We selected WangChanBERTa (based on the RoBERTa-base architecture) as our primary semantic encoder, prioritizing it over multilingual models such as mBERT or traditional Thai embeddings including Thai2Fit (Detailed information can be found in the Supplementary Materials). The decisive factor was its SentencePiece tokenizer:

• Addressing the “No-Space” Challenge: Thai is a script without explicit word boundaries. Dictionary-based tokenizers (used in older models) frequently fail when encountering “code-mixed” syntax, such as English variable names concatenated with Thai comments (e.g., //functionนี้อันตราย).
• Robustness to Unknown Tokens: Unlike mBERT, which often treats specific Thai technical jargon as “unknown” (UNK), WangChanBERTa’s sub-word tokenization adaptively learns these mixed patterns. Internal benchmarks indicate that WangChanBERTa achieves a 5–10% improvement in F1-score on Thai-English Natural Language Processing (NLP) tasks compared to mBERT, validating its suitability for our dataset.

Fine-Tuning Configuration: The model consists of 12 Transformer layers with 768 hidden dimensions. It was fine-tuned for binary sequence classification using the following optimized hyperparameters to ensure convergence on our specific cybersecurity domain:

• Tokenizer: SentencePiece (fine-tuned for Thai–English code-mixing).
• Input Handling (Sliding Window): To address the token limit in lengthy scripts, we implemented a segmentation strategy with a Maximum Sequence Length of 256 tokens and a 25% overlap. This overlap is critical for preserving the contextual integrity of malicious payloads that might otherwise be split across segment boundaries.
• Optimization: We employed the AdamW optimizer with a learning rate of 3 × 10⁻⁵ and a batch size of 32, adhering to standard best practices for Transformer fine-tuning. Feature Extraction: For every processed segment, the final hidden state of the <s> classification token ([CLS]) was extracted to serve as the dense semantic vector $\left(V_{sem}\right)$, which was subsequently fused with structural features.

3.3. Handcrafted Feature Compendium

To complement the local semantic focus of WangChanBERTa, we constructed a structural vector $V_{struct}$ comprising 35 handcrafted features, as detailed in Table 2. These features are engineered to capture global structural anomalies such as high entropy or unnatural character ratios that token-limited Transformers often overlook due to the “Attention Deficit” problem.

Table 2. Compendium of Handcrafted Feature Groups. These features were engineered based on MITRE ATT&CK (T1027) to capture structural anomalies (e.g., entropy, character ratios) that provide orthogonal signals to the semantic embeddings of the Transformer.

Feature Group	Feature Example(s)	Description	Rationale/Orthogonality to BERT
1. CharType	special_char_ratio, alpha_char_ratio, digit_ratio, uppercase_ratio, num_lowercase, num_uppercase, num_digits, num_letters	Ratio of special (e.g., !@#$%^&*()), alphabetic, or digit characters to total length.	High Importance: Captures “Syntactic Discontinuity” that tokenizers often fragment into meaningless subwords.
2. Statistical	entropy, word_count, avg_word_length, num_spaces, num_special_chars	Shannon entropy of the script (measures randomness/obfuscation). Basic text statistics.	Discriminative Anchor: High entropy is a strong proxy for obfuscation, serving as a global check against semantic hallucinations.
3. Length	total_length (len_total), max_line_length, num_lines, len_tag, len_text	The total character length of the script, the length of its longest line, and the total line count.	Global Context: Captures file-level anomalies that exceed the 512-token limit of Transformers.
4. Encoding	base64_string_count (num_base64), hex_string_count (num_hex), obfuscation_ratio, num_unicode	Counts of substrings that match Base64 or Hex patterns. Ratio of encoded to total content.	MITRE T1027: Explicitly checks for payload hiding patterns that BERT might only implicitly learn.
5. Web	url_count (num_urls), domain_count, has_ip_address, num_tld (Top-Level Domains)	Counts of embedded URLs (http/https), unique Top-Level Domains (TLDs), or hardcoded IP addresses.	Redundancy Check: Often redundant as BERT parses these as entities; identified by XAI as candidates for pruning.
6. Security	eval_count, unescape_count, setTimeout_count, num_sql	Counts of dangerous functions (eval(), unescape(), setTimeout(), setInterval()) that can execute strings as code.	Execution Path Proxies: Statistically approximates the “Obfuscate → Decode → Execute” intent flow.
7. HTML/JS	script_tag_count, iframe_tag_count, event_handler_count, num_style, num_php	Counts of embedded <script> tags, <iframe> tags, or JavaScript (JS) event handlers (e.g., onclick, onmouseover).	Structural Layout: Identifies injection vectors (e.g., hidden iframes) relying on Document Object Model (DOM) structure rather than semantics.

Strategic Feature Engineering via Execution Path Proxies: While recent studies advocate for Control Flow Graphs (CFG) to trace execution paths, such methods incur $O\left(N^3\right)$ complexity, rendering them unsuitable for real-time Web Application Firewalls (WAFs). To address this, we introduce “Execution Path Proxies” a set of density-based features $\left(O\left(N\right)\right)$ that statistically approximate the intent of a malicious execution flow:

• Dynamic Execution Path: We quantify the density of state-changing functions such as num_eval, num_settimeout, and num_docwrite. A high density in these features serves as a proxy for the “Obfuscate $\to$ Decode $\to$ Execute” path common in injection attacks.
• Redirection Flow Indicators: Features like num_redirect and num_location proxy the “User $\to$ Hijack $\to$ Phishing Site” path. By treating these function counts as nodes in a latent execution graph, our model captures the semantic similarity of code fragments requested by modern standards without the overhead of graph traversal.

Feature Preprocessing (RobustScaler): Script attributes exhibit extreme variability; for instance, a benign jQuery library may be 100 times longer than a malicious injection snippet. Conventional Min–Max scaling is highly sensitive to such outliers. To address this, we implemented RobustScaler on all handcrafted features. This method normalizes the dataset using the median and Interquartile Range (IQR), effectively neutralizing skew caused by extreme outliers. This ensures that the high structural variance of legacy Thai government code does not destabilize the gradient descent of the classifier.

3.4. Hybrid Model Architecture: A Two-Phase Selective Integration Framework

The proposed methodology is designed to address the limitations of single-modality detection by separating feature preparation from decision logic and treating malicious script detection as a conflict-resolution problem rather than a simple classification task. The framework adopts a hybrid, two-phase selective integration architecture that combines structural script statistics with context-aware semantic embeddings derived from a Transformer encoder. In the first phase, raw code-mixed scripts are processed to preserve maximal informational fidelity, producing parallel structural and semantic representations without premature dimensionality reduction (Section 3.4.1). In the second phase, a conflict-aware decision mechanism grounded in Dempster–Shafer Theory dynamically regulates the influence of each modality based on measured evidential disagreement (Section 3.4.2). This design enables the system to mitigate structural bias, suppress semantic noise, and adaptively select the most reliable evidence source under varying threat conditions. By integrating explainability directly into the architectural logic through attention analysis, centroid geometry, and SHAP attribution, the methodology ensures that detection accuracy, robustness, and interpretability are achieved simultaneously within a unified decision framework.

Figure 2 shows a detailed schematic of the proposed XAI-Driven Hybrid Architecture. The model employs a dual-path feature extraction strategy, integrating context-aware semantic embeddings $V_{sem}$ from WangChanBERTa with handcrafted structural statistics $V_{struct}$ to address the “Attention Deficit” inherent in long-form HTML scripts. These heterogeneous features are synthesized via Early Fusion (Vector Concatenation), enabling a “Semantic Arbitration” mechanism where the classifier learns non-linear interactions between global syntax and local semantic intent. The architecture incorporates SHAP analysis as a diagnostic feedback loop to ensure Selective Integration, effectively filtering “Semantic Noise” and stabilizing the decision boundary against Structural Bias.

Figure 2. The proposed XAI-Driven Hybrid Architecture for Thai–English code-mixed malicious script detection, illustrating the Selective Integration of structural statistics and semantic embeddings.

3.4.1. Phase 1: Feature Engineering and Input Preparation

The initial phase of the architecture focuses on rigorous feature engineering, where raw input scripts are processed to construct a comprehensive evidential space. Rather than employing early dimensionality reduction, the framework intentionally preserves maximum information fidelity by extracting features from two distinct modalities to address the unique complexities of code-mixed scripts:

• Semantic Embeddings $(V_{sem})$: These are context-aware embeddings derived from a WangChanBERTa encoder, representing the latent semantic manifold $V_{sem}ϵ{ℝ}^{768}$ required to capture the deep intent of the script.
• Structural Features $(V_{struct})$: These consist of standardized structural statistics $V_{sem}ϵ{ℝ}^{35}$ that provide an outlier-robust representation of the script’s global syntactic properties.

These heterogeneous signals are synthesized into a unified hybrid feature vector, $X_{hybrid}$, through direct concatenation (Early Fusion) as defined in Equation (5):

$$X_{hybrid}= V_{sem}\oplus V_{struct}$$

(5)

By maintaining the unified input space at ${ℝ}^{803}$, the system ensures that the subsequent decision logic has access to the raw, uncompressed signals necessary to identify subtle contradictions between local semantic intent and global syntax. This comprehensive representation serves as the critical foundation for the Selective Integration strategy in the next phase. It establishes the evidential manifold required for the conflict-aware gating mechanism to dynamically adjust the model’s reliance on either modality, ensuring that structural stability can mathematically override semantic hallucinations when high-conflict scenarios are detected.

3.4.2. Phase 2: Conflict-Aware Selective Integration (DST-Driven Logic)

The core innovation of our architecture lies in the subsequent decision logic, which treats hybridization as a Conflict-Resolution Process. Traditional models often fail by blindly averaging conflicting signals; in contrast, we utilize Dempster-Shafer Theory (DST) to quantify the epistemic uncertainty and “adversarial ambiguity” present in code-mixed scripts. We define the Conflict Mass $K\left(x\right)$ as a real-time diagnostic signal that measures the dissonance between the high-entropy semantic predictions and the outlier-robust structural statistics:

$$K\left(x\right)={{\displaystyle \sum }}_{A\cap B=0}{m}_{sem}\left(A\right)\cdot m_{struct}\left(B\right)$$

(6)

In this formulation, $A$ and $B$ represent the discordant focal elements (specifically, the ‘malicious’ and ‘benign’ propositions) within the frame of discernment. The condition $A\cap B=0$ signifies a direct logical contradiction between the semantic prediction and structural evidence, quantifying the degree of epistemic conflict.

Where $m\left(\cdot \right)$ represents the evidential mass assigned by each branch. This metric serves as a trigger for a Hard-Gating Mechanism, dictated by the following hierarchical decision flow:

$$Y_{final}=\left\{\begin{array}{c}{f}_{sem}\left(V_{sem}\right) if K\left(x\right)< \tau \left(Low Conflict\right) \\ f_{struct}\left(V_{struct}\right) if K\left(x\right)\ge \tau \left(Structural Anchor\right)\end{array}\right.$$

(7)

The decision threshold $\tau$ is an empirically optimized hyperparameter. In this study, it was calibrated through a grid search on the validation set to maximize the ‘Structural Veto’ efficiency, ensuring that the system prioritizes structural stability only when semantic uncertainty exceeds a critical reliability bound.

• Low-Conflict Scenario $(K\left(x\right)<\tau )$: When the signals are mathematically aligned, the system relies on the high-level semantic interpretation of the Transformer-based classifier to identify sophisticated, intent-based threats.
• High-Conflict Scenario $\left(K\left(x\right)\ge \tau \right)$: In cases where “Benign Complexity” causes semantic hallucinations (e.g., misinterpreting legacy Thai code as malicious), the algorithm triggers a “Structural Veto”. The system explicitly switches its trust to the structural classifier, using global statistical properties as a discriminative anchor to stabilize the prediction.

By implementing this two-phase strategy, the model transforms from a passive feature aggregator into an active “Semantic Arbitrator,” capable of mathematically overriding compromised interpretations in high-trust digital infrastructures.

3.4.3. Classification Module

For the terminal classification stage, a Random Forest (RF) ensemble was selected over deep neural architectures [35,36]. This strategic decision is predicated on the superior efficacy of RF in handling tabular data within high-dimensional spaces and its intrinsic resistance to overfitting attributes that are critical given the dataset constraints ($N$ = 5000). The model hyperparameters were rigorously optimized via Grid Search with Cross-Validation (GridSearchCV) to maximize generalization performance:

• Ensemble Density (n_estimators): Set to 200 to ensure convergence and variance reduction;
• Class Weighting: Configured as ‘balanced’ to automatically adjust weights inversely proportional to class frequencies, thereby rigorously mitigating bias against the minority class [37,38];
• Split Criterion: The Gini Impurity Index was utilized to evaluate split quality;
• Sampling Method: Bootstrap aggregation was enabled to enhance model stability.

3.4.4. Theoretical Justification of Fusion Dynamics

To rigorously validate the architectural selection, comparative evaluations were conducted against Late Fusion (Probabilistic Averaging) and Gated Fusion (Attention-Weighted) paradigms:

• Empirical Evidence: The Early Fusion strategy yielded a state-of-the-art F1-score of 0.9908, demonstrating a statistically significant improvement over the Late Fusion baseline (F1 = 0.9895).
• Theoretical Basis: Late Fusion architectures operate under the assumption of feature independence, modeling $P(y|V_{sem})$ and $P(y|V_{struct})$ separately. However, in the domain of code-mixed script detection, semantic and structural features exhibit strong conditional dependence. For instance, high structural entropy may be benign within a natural language comment block but indicative of malicious obfuscation within executable code variables.
• Semantic Arbitration: By concatenating features prior to the decision boundary, the Random Forest model is empowered to learn these non-linear, cross-modal dependencies (e.g., conditional logic such as $entropy> \delta \cap context \ne comment$). We name this mechanism “Semantic Arbitration,” wherein structural features serve as a discriminative anchor to validate or veto semantic predictions, a capability that is mathematically precluded in decoupled Late Fusion systems.

3.5. Explainability (XAI) Framework

To transcend basic performance metrics and diagnose the root causes of model behavior, we employ a multi-tiered XAI framework. Crucially, we utilize XAI not merely for post hoc justification but as a fundamental architectural design tool.

The Feedback Loop: Initial diagnostic phases revealed that straightforward feature fusion often introduces redundancy. By quantifying the Marginal Contribution (MC) of each feature group relative to the semantic baseline, we established a feedback loop that enabled “Architectural Pruning.” This process removed statistically redundant feature groups (e.g., num_urls), reducing input dimensionality by 40% while preserving the F1-score, thereby enhancing computational efficiency without compromising detection fidelity.

3.5.1. Component-Level Diagnosis (Ablation Study)

A permutation-based ablation study was employed to rigorously assess the orthogonality and contribution of each handcrafted feature group within the Hybrid model. We categorized the structural features into $K=7$ groups (CharType, Statistical, Length, Encoding, Web, Security, HTML/JS).

The Feature Importance Score $\left(\Delta F{1}_g\right)$ for a specific group $g$ was quantified based on the degradation in model performance when that group was stochastically permuted (noise-injected) while keeping others constant:

$$\Delta F{1}_g=F{1}_{original}-F{1}_{permuted\left(g\right)}$$

(8)

Interpretation:

• If $\Delta F{1}_g>0$: The group provides unique, Orthogonal Information essential for classification;
• If $\Delta F{1}_g\approx 0$: The group represents “Semantic Noise” (redundant information already captured by WangChanBERTa);
• If $\Delta F{1}_g<0$: The group introduces conflicting signals that confuse the decision boundary.

3.5.2. Transformer Interpretation

To validate the internal reasoning of the WangChanBERTa encoder, we employed a suite of probing techniques to analyze its latent manifold and attention dynamics.

• Latent Space Geometry (Centroid Analysis): To verify class separability, we computed the centroid vectors $\left(C_k\right)$ for both benign $\left(k=0\right)$ and malicious $\left(k=1\right)$ classes within the high-dimensional embedding space. For a set of samples $N_k$ belonging to class $k$, the centroid is defined as

  $$C_k={\displaystyle \frac{1}{\left|N_k\right|}}{\displaystyle \sum }_{x_i\in N_k}E\left(x_i\right)$$

  (9)

  where $E\left(x_i\right)$ is the embedding vector of sample $x_i$. We then utilized the Cosine Similarity to measure the density and separation between these centroids. Additionally, Principal Component Analysis (PCA) was applied to project the $768-dimensional manifold into 2D$ space for the visual inspection of the decision boundary.
• Token-Level Saliency (Attention Analysis): To diagnose the “Attention Deficit” hypothesis, we extracted the attention weights ${\alpha }_{ij}$ from the final Transformer layer. The aggregate importance score $A\left(t\right)$ for a specific token $t$ is computed by summing the attention heads:

  $$A\left(t\right)={{\displaystyle \sum }}_{h=1}^H{\alpha }_{h,t}$$

  (10)

  A high $A\left(t\right)$ value indicates that the model heavily relies on token $t$ for its prediction. This allows us to verify whether the model focuses on legitimate malicious payloads (e.g., eval(), document.write) or erroneously fixates on benign Thai comments.
• Latent Feature Contribution (SHAP): We employed SHAP to attribute the prediction output to specific dimensions of the semantic vector. This helped to “debug” the Transformer by identifying which semantic signatures (dimensions) drive the classification, ensuring that the model detected intent rather than memorized trivial patterns.

4. Results and Analysis

This chapter evaluates the proposed XAI-Driven Hybrid Architecture to validate the “Selective Integration” hypothesis, specifically analyzing how feature orthogonality influences decision-making in code-mixed environments. Experimental validation was conducted on a curated corpus ($N$ = 5000) using 5-Fold Stratified Cross-Validation to ensure balanced class distributions and robust statistical generalization. Beyond standard quantitative metrics, Explainable AI (SHAP and Attention Heatmaps) is utilized to diagnose baseline “failure modes,” demonstrating the systematic corrective capabilities of the Hybrid approach in resolving these identified weaknesses.

4.1. Comparative Model Performance and Stability

The quantitative results, compiled through the 5-fold validation process, offer robust empirical evidence underscoring the superiority of the Hybrid Architecture across multiple dimensions: operational efficacy, stability, and discriminative power.

4.1.1. Quantitative Efficacy and Operational Impact

Table 3 presents a comprehensive overview of the performance metrics across the three experimental setups. The Hybrid model achieved a state-of-the-art mean F1-score of 0.9908, surpassing those of both the Random Forest (0.9825) and the standalone Transformer (0.9874):

Table 3. Comparative Performance Metrics (5-Fold Cross-Validation). The Hybrid Architecture achieves a state-of-the-art F1-score of 0.9908 and a remarkably low False Positive Rate (FPR) of 0.0116, demonstrating superior precision and operational reliability.

Model Type	Accuracy	F1-Score	ROC AUC	Precision	TPR	FPR
Random Forest	0.9824	0.9825	0.9981	0.9759	0.9892	0.0244
Transformer	0.9874	0.9874	0.9996	0.9845	0.9904	0.0156
Hybrid	0.9908	0.9908	0.9996	0.9885	0.9932	0.0116

• Operational Efficiency (False Positive Minimization): While the numerical increase in F1-score appears incremental, the achieved False Positive Rate (FPR) of 0.0116 is operationally significant. By maintaining a consistently low FPR, the Hybrid model effectively minimizes false alarms that typically plague baseline systems. In the context of high-traffic governmental networks, this level of precision significantly mitigates ‘Alert Fatigue’ for Security Operations Center (SOC) analysts, validating the practical reliability of the ‘Dual-Validation’ mechanism.
• Precision and Reliability: The Hybrid model attained the highest precision (0.9885), significantly outperforming the Random Forest baseline (0.9759). This metric is crucial for trustworthiness, indicating that when the Hybrid model flags a script, there is a probability exceeding 98.8% that it is genuinely malicious.

4.1.2. Stability and Generalization Analysis

To evaluate the models’ generalization ability and resistance to overfitting, a key concern raised in theoretical machine learning frameworks, we analyzed the distribution of F1-scores across validation folds, as illustrated in Figure 3:

Figure 3. Stability Analysis via 5-Fold Cross-Validation. The Hybrid model (Green) demonstrates superior generalization capability, exhibiting the narrowest Interquartile Range (IQR) and eliminating the low-performance outliers observed in the standalone Transformer (Orange), thereby validating the stabilizing effect of structural features.

• Structural Bias in Baseline: The Random Forest model (Blue) exhibits a broad Interquartile Range (IQR) and lower median performance. This variance indicates a sensitivity to specific data splits, suggesting that the model relies on surface-level structural statistics (e.g., entropy) that may not generalize well to unseen benign variations.
• Hybrid Regularization: In contrast, the Hybrid model (Green) demonstrates the narrowest IQR and the highest median score. Theoretically, this confirms that the fusion of orthogonal features acts as a Regularization Mechanism. By constraining the high-capacity Transformer with low-capacity, high-bias structural features, the Hybrid Architecture stabilizes the hypothesis space, preventing “catastrophic failures” (outliers) and ensuring consistent performance across diverse data subsets.

4.1.3. Global Discriminative Power

Figure 4 illustrates the Receiver Operating Characteristic (ROC) curves, providing a visual assessment of global separability between benign and malicious classes:

Figure 4. Comparative Receiver Operating Characteristic (ROC) Curves. While both Transformer-based models achieve near-perfect AUC scores (0.9996), the curves illustrate the global discriminative superiority of semantic embeddings over the statistical baseline (Random Forest).

• AUC Saturation: Both the Transformer (Orange) and Hybrid (Green) models achieved a near-perfect AUC of 0.9996, significantly outperforming the Random Forest model (AUC = 0.9981).
• Interpretation of Limits: While the AUC metric suggests near-equivalence between the Transformer and Hybrid models, this scalar value can mask localized errors in high-precision regimes. The ROC curves confirm that while semantic embeddings are sufficient for ranking threats (high AUC), structural augmentation is required to optimize the decision threshold for minimizing False Positives, as further evidenced using the confusion matrix analysis.

4.1.4. Diagnosing Semantic Noise via Confusion Matrices

To dissect the “Semantic Noise” hypothesis and validate the “Safety Net” function, we analyzed the Normalized Confusion Matrices presented in Figure 5:

Figure 5. Normalized Confusion Matrices validating the ‘Safety Net’ Hypothesis. (a) Random Forest suffers from high False Positives (2.4%) due to structural bias. (b) Transformer improves precision but misses Benign Complexity. (c) The Hybrid model achieves the lowest False Positive Rate (1.2%), proving that structural features successfully veto semantic hallucinations.

• Random Forest Failure (Figure 5a): The baseline model demonstrates an elevated error rate, with a notably substantial FPR of 2.4%. This finding confirms that statistical proxies struggle to differentiate between benign code minification (e.g., jQuery libraries) and malicious obfuscation, leading to a high rate of “Structural Bias” errors.
• Transformer Precision (Figure 5b): The standalone Transformer improves the FPR to 1.6%, leveraging semantic context to correctly identify many administrative scripts. However, it still yields false alarms on samples with “Benign Complexity,” where the semantic context is ambiguous.
• Hybrid ‘Safety Net’ (Figure 5c): The Hybrid model achieves the lowest FPR of 0.0116. This empirical evidence validates the ‘Semantic Arbitration’ mechanism: when the Transformer encounters ambiguous signals from complex benign code, the structural features act as a discriminative anchor to effectively veto potential false alarms. This confirms that structural features provide a critical verification layer that stabilizes predictions in edge cases where semantic-only models might falter.

4.2. Deep Diagnostic Analysis: Error Modes and Feature Conflicts

While quantitative metrics demonstrate the Hybrid model’s superiority, identifying the underlying causal mechanisms is essential for scientific validation. This section employs XAI to diagnose why specific feature groups act as “Semantic Noise” while others serve as a critical “Safety Net,” utilizing ablation studies and SHAP dynamics.

4.2.1. Error Analysis: Failure Modes and Correction

We analyzed the specific error instances $N_{err}$ to categorize the failure modes of each architecture:

• Random Forest Failure (Structural Bias): The RF model disproportionately failed on “Benign Complexity” cases ($N$ = 61 False Positives). These were primarily legitimate jQuery libraries or minified CSS files characterized by high entropy and long line lengths, features the RF heuristic rigidly regards as malicious.
• Transformer Failure (Attention Deficit): The standalone Transformer struggled with “Short Obfuscation” ($N$ = 29 False Positives), where malicious payloads were fragmented into short, ambiguous snippets that lacked sufficient semantic context for the self-attention mechanism to converge.
• Hybrid Correction: The Hybrid model’s error set ($N$ = 23) represents a proper subset of the intersection of these failure modes. By cross-referencing decisions, the Hybrid Architecture successfully corrected 82% of the RF’s structural errors (via Semantic Veto) and 45% of the Transformer’s ambiguity errors (via Structural Anchoring), empirically validating the “Dual-Validation” hypothesis.

4.2.2. Diagnosing Semantic Noise via Ablation Study

To investigate the “Semantic Noise” phenomenon, we conducted a permutation-based ablation study (illustrated in Figure 6) to quantify the marginal utility $\left(\Delta F1\right)$ of each feature group.

Figure 6. Diagnosis of feature utility via ablation study. Positive values indicate beneficial features (e.g., CharType, Length), while near-zero or negative values (e.g., Web, Security) identify “Semantic Noise”, redundant signals already subsumed by the Transformer’s embeddings.

• Identification of Redundancy: Contrary to traditional cybersecurity heuristics, the removal of the ‘Web’, ‘Encoding’, ‘Security’, and ‘HTML/JS’ feature groups resulted in either a marginal improvement or negligible degradation in performance $\left(\Delta F1\approx 0\right)$.
• Root Cause Analysis (Information Subsumption): We attribute this redundancy to “Information Subsumption.” For instance, the ‘Security’ group relies on scalar counts of tags such as <script> or <iframe>. The WangChanBERTa Transformer, pre-trained on massive corpora, already attends to these tokens with high weight within its embedding space. Therefore, explicitly adding a scalar count $x_{count}$ to the feature vector $V_{sem}$ adds no new mutual information $(I(Y;x_{count}|V_{sem})$. Instead, it introduces “Feature Collinearity,” flattening the loss landscape and hindering gradient convergence.

4.2.3. Verifying Orthogonality via SHAP Dynamics

We corroborated the ablation results using SHAP to visualize the decision logic and confirm which features provide true orthogonality.

Global Feature Importance: The SHAP ranking confirms a clear dichotomy:

• Dominant Features: num_lowercase (Language Ratio) and special_char_ratio (Entropy Proxy) emerged as the top predictors. These features provide “Orthogonal Information” global statistical summaries that the token-limited Transformer (512 tokens) cannot efficiently compute over long sequences.
• Obsolete Heuristics: Domain-specific counters including num_urls, num_sql (Structured Query Language), and num_base64 clustered at the bottom of the importance chart (Figure 7), confirming their obsolescence in the presence of deep semantic embeddings.

Figure 7. Global SHAP Feature Importance. The model prioritizes global statistical properties (e.g., num_lowercase, special_char_ratio), which are orthogonal to token embeddings, while traditional heuristics such as num_urls show negligible impact, confirming their obsolescence.

The model’s decision-making process, termed “Directional Logic,” is elucidated through the SHAP summary plot (Figure 8). This visualization reveals a strategic “Safety Net” where the model balances deep semantic understanding with robust syntactic heuristics. This dual-layered logic is primarily driven by the hierarchical influence of seven key feature groups: CharType, Length, Statistical, Web, Encoding, Security, and HTML/JS.

Figure 8. Aggregated Feature Group Importance. The analysis confirms that ‘CharType’ and ‘Length’ provide the strongest orthogonal signals, validating the Selective Integration strategy of retaining only these high-impact groups.

The internal logic of the model is further elucidated through the SHAP summary plot (Figure 9), which illustrates the directional impact of individual features on the model’s predictions. In this visualization, the color of each point represents the feature value (red for high, blue for low), while its position on the horizontal axis indicates the magnitude and direction of its influence on the final classification. This granular view reveals a dual-layered "Safety Net" consisting of the following mechanisms:

Figure 9. SHAP Summary Plot illustrating Directional Logic. Red points (high values) for special_char_ratio strongly correlate with malicious predictions, acting as a “Syntactic Alert”, whereas high num_lowercase acts as a “Semantic Anchor” for benign classification.

• High special_char_ratio (Red points): This strongly pushes the prediction towards malicious (positive SHAP value). This acts as a “Syntactic Alert,” ensuring that even if the Transformer misses the semantic meaning of an obfuscated payload, the structural anomaly will trigger detection.
• High num_lowercase (Red points): Pushes the prediction towards benign. This reflects the model’s ability to recognize standard English/Thai narrative structures, acting as a “Semantic Anchor” to prevent False Positives on benign text, a finding further validated by the grouped importance of CharType and Length.

4.3. Comparative Analysis of Fusion Architectures

To comprehensively validate the architectural decision to employ Early Fusion, we conducted a comparative study of two alternative paradigms widely used in multimodal learning: Late Fusion (Probabilistic Ensemble) and Gated Fusion (Attention-Weighted). This analysis aims to determine the optimal mechanism for synthesizing structural and semantic information.

4.3.1. Experimental Setup and Results

We evaluated three distinct fusion strategies on the same high-fidelity dataset $\left(N=5000\right)$ to determine the optimal mechanism for synthesizing structural and semantic information:

• Early Fusion (Proposed Selective Integration): This approach implements the direct vector concatenation of context-aware semantic embeddings $\left(V_{sem}\right)$ and standardized structural statistics $V_{struct}$ prior to the classification boundary, as defined in Equation (5). To address the unique complexities of code-mixed scripts, our framework treats hybridization as a conflict-resolution process grounded in Dempster-Shafer Theory (DST). Specifically, it calculates a real-time Conflict Mass $K\left(x\right)$ via Equation (6) to measure the mathematical dissonance between the semantic predictions and outlier-robust structural statistics. When high conflict is detected such as when “Benign Complexity” causes semantic hallucinations the system triggers a “Structural Veto” dictated by the hierarchical decision flow in Equation (7). This mechanism allows the model to switch trust to the structural classifier, using global statistical properties as a discriminative anchor to stabilize the final prediction.
• Late Fusion (Probabilistic Ensemble): This strategy involves the independent training of two separate classifiers (Random Forest on structural features and Multilayer Perceptron (MLP) on semantic embeddings), followed by simple probability averaging where $\left(P_{final}=0.5\cdot P_{struct}+0.5\cdot P_{sem}\right)$.
• Gated Fusion (Attention-Weighted): This mechanism employs a learnable attention mechanism where weights are dynamically assigned as $X=\propto \cdot V_{sem}+\left(1-\propto \right)\cdot V_{struct}$, with $\propto$ representing a learnable sigmoid gate.

Empirical Findings: The experimental results demonstrate that our proposed Early Fusion (Selective Integration) achieves a state-of-the-art F1-score of 0.9908. This performance statistically outperforms the Late Fusion baseline (F1 = 0.9895) and is equivalent to the more complex Gated Fusion (F1 = 0.9901). These findings validate that our DST-driven architecture effectively captures non-linear, cross-modal dependencies (Semantic Arbitration) without the additional computational overhead of gated parameters.

4.3.2. Theoretical Interpretation: The Failure of Independence Assumptions

The inferiority of Late Fusion can be explained through Probability Theory. Late Fusion implicitly assumes that the feature sets are conditionally independent given the class label $Y$, as shown in Equation (9):

$$P(Y|X_{sem},X_{struct})\approx P(Y|X_{sem})\cdot P(Y|X_{struct})$$

(11)

However, in the domain of code-mixed scripts, structure and semantics are highly dependent:

• Example: High entropy $(X_{struct})$ is indicative of maliciousness only if the semantic context $\left(X_{sem}\right)$ corresponds to executable code. If the context is “Compressed Image Data,” high entropy is benign.
• Semantic Arbitration: Because Late Fusion treats these signals separately, it fails to capture this conditional logic. In contrast, Early Fusion constructs a unified feature space, enabling the Random Forest to learn non-linear interactions (e.g., decision paths representing $IF Entropy > Thres AND Context != Image THEN Malicious$). We name this capability “Semantic Arbitration,” where the model uses semantic context to dynamically interpret structural statistics.

4.3.3. Efficiency Justification

While Gated Fusion achieved comparable accuracy to Early Fusion, it introduces additional learnable parameters $\left(W_{gate}\right)$, increasing computational complexity during inference. Since the unweighted concatenation (Early Fusion) achieves state-of-the-art results without this overhead, it represents the optimal trade-off between detection fidelity and system latency, adhering to the principle of parsimony (Occam’s Razor).

4.4. Interpreting the Winning Model: Semantic Manifold and Attention Dynamics

To address the theoretical gap regarding the Transformer’s applicability for malicious script detection, we analyzed the model’s internal representations. This section confirms that the WangChanBERTa encoder does not merely memorize keywords but learns a structured “Semantic Manifold” that maps the unique syntactic irregularities of obfuscated code, as confirmed through both latent space topology and forensic case studies.

4.4.1. Latent Space Topology

We investigated the geometric structure of the 768-dimensional embedding space using Principal Component Analysis (PCA) and Centroid Distance metrics:

• Manifold Hypothesis Validation: The projection reveals two distinct, high-density clusters corresponding to ‘benign’ and ‘malicious’ classes, separated by a sparse low-density region.
• Class Separation Logic: By computing the Cosine Similarity to class centroids ${\mu }_{benign}, {\mu }_{malicious},$ we found that the model clusters scripts based on “Intent” rather than “Syntax.” For instance, benign administrative scripts with high entropy (e.g., Minified JS) are geometrically mapped closer to the ‘benign’ centroid, proving that the model has learned to disentangle Benign Complexity from malicious obfuscation in the latent space.

4.4.2. Attention Mechanism: Targeting “Syntactic Discontinuity”

A key characteristic of malicious scripts is “Syntactic Discontinuity”, where executable logic is fragmented to evade signature-based detection (e.g., $d+o+c+u+m+e+n+t$). We analyzed the attention weights ${\propto }_{ij}$ to verify if the model captures this unique structural characteristic.

• Token-Level Saliency: The Attention Heatmap analysis demonstrates that the model exhibits “Sparse Attention.” It assigns negligible weight to benign Thai narrative text (e.g., comments, UI labels) but attends heavily to the following:
  • Fragmentation Patterns: Sequences of concatenated strings (e.g., ‘+’, String.fromCharCode);
  • Suspicious Entities: Obfuscated URL fragments (e.g., .xyz, /wp-content/) and dynamic execution functions (eval, base64_decode).
• Theoretical Implication: This confirms that the self-attention mechanism effectively overcomes the “Language Gap.” It treats Thai characters as contextual background noise while focusing its computational resources on the invariant structural patterns of the attack payload.

4.4.3. Visualizing “Semantic Arbitration”

To empirically validate the theoretical mechanisms described above, we present two forensic case studies. These cases demonstrate how the model’s “Semantic Arbitration” capability resolves conflicts between structural heuristics and semantic context:

• Case Study 1: Correcting Structural Bias (The “a[text], คู่มือการ share content เพื่อการจัดอันดับ webometrics” Sample): Figure 10 illustrates a benign news announcement containing mixed Thai/English text:
  • The Heuristic Conflict: The Random Forest baseline flagged this sample as malicious $P_{mal}\approx 0.97$ due to high digit_ratio and special_char_ratio. This illustrates “Structural Bias,” where the model heuristically equates high numerical density with obfuscation.
  • The Semantic Veto (Figure 10c): The Hybrid model correctly reclassified it as benign. Consistent with our latent space analysis (Section 4.4.1), the semantic embeddings associated with benign context words (e.g., “share content”, “webometrics”) produced a strong negative signal. This confirms the “Semantic Veto” mechanism, where deep contextual understanding overrides misleading structural heuristics.
• Case Study 2: Disambiguating Public Relations News (The “li แผ่นพับประชาสัมพันธ แนะนำสำนักคอมพิวเตอร…” Sample): Figure 11 depicts a benign public relations news announcement featuring mixed Thai and English content, such as “Courseware” and “Single Sign On (SSO)”.
  • The Heuristic Trap: The Random Forest baseline misclassified this sample as malicious $\left(f\left(x\right) = 0.774\right)$, primarily due to structural features such as len_tag (+0.08) and digit_ratio (+0.04). The model fell into a “Heuristic Trap,” mistaking the high entropy of the PR announcement layout for a malicious injection attack.
  • Context-Aware Resolution (Figure 11c): The Hybrid model successfully identified the sample as benign, reducing the malicious probability significantly $\left(f\left(x\right) = 0.248\right)$
  • Semantic Arbitration Mechanism: The architecture utilized “Semantic Arbitration” to override structural false alarms. The attention mechanism focused on the coherent administrative and news narrative—specifically terms such as “nursing school” and “prerequisites”—allowing the model to recognize the benign intent of the news update despite its complex structural indicators.

Figure 10. Case Study 1: Correcting Structural Bias (The “Benign Date” Sample). (a) The baseline misclassifies benign numerical data as malicious, (b) the Transformer component identifying benign intent via negative SHAP values from semantic embeddings, and (c) The Hybrid model utilizes “Semantic Veto” (negative SHAP values from context words) to override the structural false alarm, correctly identifying the sample as benign.

Figure 11. Case Study 2: Disambiguating Benign Complexity (The PR News Sample). (a) The baseline model falls into a “Heuristic Trap” due to the misleading statistical properties of the PR content, (b) the Transformer component identifying benign administrative intent through semantic embeddings that capture the news narrative, and (c) The Hybrid model leverages “Semantic Arbitration” to recognize the public relations intent, confirming the script as benign.

4.4.4. Latent Feature Impact

Finally, an extensive SHAP analysis of the embedding dimensions $\left(V_{sem}\right)$ reveals that the model operates via “Sparse Coding,” ensuring generalization:

• Dimension Specificity: Out of 768 dimensions, only a small subset $\left(\approx 50 dimensions\right)$ contributes significantly to the decision boundary.
• Semantic Compression: The elevated SHAP values in these critical dimensions indicate that the Transformer has successfully compressed the abstract concept of “Code-Mixed Injection” into a compact set of highly discriminative features. This explains the model’s robustness: it relies on high-level semantic features rather than overfitting to noise in the entire embedding space.

4.5. Comparative Methodological Analysis: Generative vs. Discriminative Frameworks

To rigorously validate the superiority of our supervised “Dual-Verification” approach, we conducted a domain-adaptation benchmarking experiment against state-of-the-art generative baselines suggested by recent literature: Sequential Variational Autoencoder (Seq-VAE) and Source-Free Domain Adaptation (SFDA) [39,40].

Experimental Setup: We simulated a “Domain Drift” scenario where models were trained on clean English scripts $\left(N \approx 2100\right)$ and tested on Thai-English code-mixed data $\left(N \approx 2900\right)$. The results of this comparison are summarized in Table 4.

Table 4. Cross-Methodological Benchmarking: Performance Evaluation of the Proposed Dual-Verification Hybrid Architecture against SOTA Generative (Seq-VAE) and Unsupervised Domain Adaptation (SFDA) Frameworks.

Methodology	Core Mechanism	F1-Score	FPR	Failure Mode Analysis
Sequential VAE	Unsupervised (Reconstruction)	0.2060	0.0719	“Benign Novelty Collapse”: The VAE mistook the high entropy of benign Thai characters for anomalies (High Reconstruction Error).
SFDA	Unsupervised Adaptation	0.6320	0.4912	“Drift Error”: Without supervised structural anchors, the adaptation logic drifted, aligning complex Thai syntax with malicious classes.
Proposed Hybrid	Supervised Dual-Verification	0.9908	0.0116	Robust Stability: The structural safety net correctly vetted the ‘novel’ Thai syntax, achieving a ~7x lower FPR.

Empirical results indicate that while generative and unsupervised models excel in standard benchmarks, they struggle with the unique complexities of Thai-English code-mixed data. The performance gap evidenced by Seq-VAE’s sensitivity to linguistic variance and SFDA’s high False Positive Rate (49.12%) underscores the limitations of purely semantic approaches in this domain. These findings validate the necessity of our supervised structural anchoring, which provides a critical discriminative "safety net" to stabilize detection within high-entropy digital infrastructures.

5. Discussion and Implications

The empirical findings presented in Section 4 have substantial implications for the architecture of machine learning-driven cybersecurity systems. This chapter interprets these results through the lenses of Information Theory and Statistical Learning Theory to formalize the mechanisms of “Semantic Noise” and “Dual-Validation,” while also addressing the constraints of data selection and broader applicability.

5.1. Reevaluating Semantic Noise: An Information-Theoretic Perspective

This information-theoretic gap confirms that while Transformer architectures excel at local contextual parsing, they exhibit a ‘global blindness’ to file-level statistical invariants. By explicitly retaining structural features with high Conditional Mutual Information (CMI), our architecture ensures that the structural anchor provides a non-redundant stability layer, mathematically compensating for the inherent ‘Attention Deficit’ in token-limited models.

The success of our Selective Integration strategy can be explained through Conditional Mutual Information (CMI). We posit that for a simple handcrafted feature $F$ (e.g., num_urls) and Semantic Embedding $E$, the mutual information $I(Y; F | E) \approx 0$, indicating redundancy (Semantic Noise). However, our ablation study (Figure 6) reveals that structural features like special_char_ratio (Entropy) provide positive marginal utility $\Delta F1 > 0)$. This proves they contain Orthogonal Information $(I(Y; F_{struct} | E) > 0)$ that the token-limited Transformer cannot capture globally. Our architecture effectively filters out redundant features while retaining only those that provide complementary evidence.

The utility of a handcrafted scalar feature $\left(F_{count}\right)$ is defined by its Conditional Mutual Information with the class label $Y$, given the Transformer embedding $E$:

$$I\left(Y;F_{count}|E\right)=H\left(Y|E\right)-H(Y|E,F_{count})$$

(12)

Our analysis suggests that $I(Y;F_{count}|E)\approx 0$. This is true because the self-attention mechanism of WangChanBERTa naturally internalizes the significance of security-critical tokens (e.g., <script>, eval) within its high-dimensional space. Consequently, explicitly adding a scalar count $F_{count}$ adds no new information but introduces “Feature Collinearity,” which flattens the loss landscape and hampers optimization. Thus, “Semantic Noise” in Hybrid models is not irrelevant data but “Redundant Entropy” that interferes with the primary semantic signal. In contrast, structural features such as special_char_ratio provide Orthogonal Information $(I > 0)$ because they capture global statistical properties that token-limited Transformers cannot easily compute, serving as a necessary complement.

This information-theoretic validation provides the mathematical foundation for the Conflict-Aware Gating defined in Equation (7). While the Conflict Mass $K$ from Equation (6) identifies the instantaneous dissonance between modalities, Equation (12) proves that the selected structural features provide non-redundant, orthogonal signals $(I > 0)$. Consequently, the ‘Structural Veto’ is not an arbitrary override but a theoretically grounded decision to prioritize features that retain discriminative power even when semantic representations suffer from entropy-driven noise.

5.2. The Disruptive Impact of the Dual-Verification Mechanism

The “Dual-Verification” mechanism proposed in this study represents a paradigm shift from traditional “Feature Engineering” to “Semantic Arbitration”:

• Paradigm Shift: Unlike ensemble methods that simply average probabilities (Late Fusion), our Early Fusion architecture allows the model to learn conditional logic (e.g., IF $Semantic=Ambiguous$ AND $Structure=Benign$ THEN Safe).
• Correction of Structural Bias: As empirically demonstrated in the forensic analysis of Section 4.4 (Case Study 1), this mechanism allows the model to “veto” false alarms triggered by high-entropy benign data (e.g., numerical dates). This confirms that deep contextual understanding can override misleading heuristic rules.
• Correction of Attention Deficit: Conversely, as shown in Case Study 2, the structural “Safety Net” prevents the Transformer from hallucinating when faced with the complex layouts of public relations news announcements. This proves that structural statistics act as a grounding anchor for high-variance semantic models, enabling the system to recognize benign intent even when the document structure is chaotic or atypical.

5.3. Theoretical Generalization via VC Dimension

To address the theoretical basis of the model’s generalization ability, we apply Vapnik–Chervonenkis $\left(VC\right)$ theory. A pure Transformer model has an extremely effective VC-dimension $d_{VC}$, making it prone to overfitting on small, noisy datasets ($N$ = 5000).

The Hybrid Architecture effectively functions as a Regularization Constraint. By concatenating the high-dimensional embedding $\left(d=768\right)$ with a low-dimensional, high-bias structural vector $\left(d=35\right)$, we restrict the hypothesis space $\mathcal{H}$. This reduction in effective complexity tightens the generalization bound, which explains the reduced variance and higher stability observed in the 5-fold cross-validation results (Figure 2). The structural features act as a “Stabilizer,” preventing the model from fitting to spurious semantic correlations in outlier samples.

5.4. Implications for Low-Resource

The findings extend beyond Thai–English code-mixing, highlighting a systemic vulnerability in applying Anglocentric models to Low-Resource Languages (LRLs).

• Tokenization Fragility: Script-continuous languages (e.g., Lao, Khmer, Burmese) lack explicit word boundaries. Standard tokenizers often fragment these languages into “Unknown” (UNK) tokens when mixed with code, causing “Semantic Blindness.”
• Blueprint for LRL Cybersecurity: Our success with WangChanBERTa confirms that semantic precision in LRLs must be anchored by Language-Specific Tokenization (e.g., SentencePiece) backed by Language-Agnostic Structural Statistics. This offers a robust blueprint for securing digital infrastructures in the Global South, where pre-trained models are often less mature than their English counterparts.

5.5. Limitations and Data Selection Bias

We acknowledge that pruning the dataset from 28,077 to 5000 samples (as detailed in Section 3.1) introduces a Selection Bias towards “High-Fidelity” samples:

• Impact: While this ensures a clean decision boundary for validating XAI diagnostics, it may reduce the model’s exposure to “Grey-Zone” samples (e.g., ambiguous ad-injectors) found in the wild.
• Mitigation: However, the breakdown of error analysis (Section 4.2) suggests that the model learns robust underlying principles (syntax vs. intent) rather than memorizing specific samples. Future deployments may require active learning loops to progressively incorporate these gray-zone samples back into the training set to enhance robustness.

6. Conclusions

This research validates an XAI-Driven Hybrid Architecture designed to resolve "Structural Bias" and "Semantic Noise" in Thai-English code-mixed script detection. By employing a Selective Integration strategy that synergizes WangChanBERTa embeddings with orthogonal structural statistics, the model achieved a state-of-the-art F1-score of 0.9908 and reduced the False Positive Rate (FPR) to a marginal 0.0116. These results suggest that hybridization is most effective when engineered features provide mathematical orthogonality to semantic embeddings.

From a theoretical perspective, this study delivers three pivotal contributions:

• Formal Feature Synergy: We established a rigorous foundation for integration using Dempster-Shafer Theory and Information-Theoretic Validation, ensuring that structural overrides are governed by formal logic rather than heuristic weighting.
• Forensic Emulation of Human Reasoning: SHAP and Attention diagnostics identified a “Semantic Arbitration” mechanism. This architecture mimics human expertise by using a semantic encoder for intent-based threats while employing a structural safety net to filter the "Benign Complexity" that often misleads pure Transformer models.
• VC-Theory Regularization: Generalization is grounded in Vapnik–Chervonenkis (VC) theory, confirming that our dual-verification approach effectively regularizes the hypothesis space and ensures stability against unseen syntax variations.

Beyond theoretical advancements, the framework offers substantial operational impact for National Critical Information Infrastructure (NCII). By achieving a consistently low FPR, the system significantly mitigates “Alert Fatigue” in Security Operations Centers (SOCs). This allows analysts to redirect limited cognitive resources from verifying benign administrative anomalies in .go.th and .ac.th domains toward investigating authentic, high-sophistication threats.

While robust, the model’s current limitation lies in the aggressive pruning of “grey-zone” scripts. Future research will explore Cross-Attention Mechanisms for more granular token-level integration and develop adversarial training protocols to harden the structural safety net against next-generation “Structural Spoofing” attacks generated by Large Language Models (LLMs).