AI-Augmented Compliance Auditing for Cloud Systems: A Hybrid ML–LLM Approach

Abstract

Manual compliance auditing in cloud environments consumes up to 40% of IT security budgets annually, yet existing approaches verify control presence rather than effectiveness, leaving institutions vulnerable to adversarial evasion. This paper presents an AI-augmented hybrid ML–LLM compliance auditing system evaluated on Rwanda’s National Cyber Security Authority (NCSA) Minimum Cybersecurity Standards (169 controls across 14 families). The system combines leakage-free XGBoost multi-label classification with GPT-4o-mini semantic log analysis, grounded in a formal effectiveness model. Key findings: (1) XGBoost v2 achieves 85.45% macro-F1 on leakage-free synthetic data (Wilson 95% CI = [84.9%, 86.0%]); an initial 86.3% data-leakage rate artificially inflated prior results to 99.99% and was identified and corrected in this revision; (2) GPT-4o-mini achieves 92.3% macro accuracy across four log types (n = 628, 37.5% real enterprise data, Wilson CI = [89.9%, 94.3%]); (3) adversarial validation across five MITRE ATT&CK scenarios yields 92.8% macro detection with 0.0% false-positive rate on real SSH/PAM compliant logs (n = 75); (4) a cross-dataset generalization analysis confirms 87.6% F1 on real SSH logs but identifies a 37.8-percentage-point out-of-vocabulary gap for Windows and HTTP log types, motivating the hybrid architecture; (5) the combined hybrid system (XGBoost for in-vocabulary logs, GPT-4o-mini for out-of-vocabulary) achieves 85.1% F1 with 6.4% false-positive rate on 180 real-world logs. The system runs at 2.0 CPU cores, 2.66 GB RAM, on $50/month cloud hosting (Apple M1 Pro baseline; storage and maintenance excluded), producing audit reports in 2–5 s depending on log volume and policy document size, demonstrating that effectiveness-based compliance auditing is accessible without enterprise-grade infrastructure.

1. Introduction

1.1. Context and Motivation

Enterprises and government agencies across Africa are undergoing rapid digital transformation, increasingly adopting hybrid cloud infrastructures. Compliance and cybersecurity assurance remain fragmented and heavily manual, consuming up to 40% of IT security budgets [1], often requiring more than 1000 h annually [2], and introducing error-prone, siloed approaches.

At a continental level, the African Union’s Malabo Convention aims to create a unified legal framework for cybersecurity [3]. However, only 15 of 54 African nations had ratified it as of 2024 [4]. Rwanda has taken independent leadership, establishing comprehensive NCSA Minimum Cybersecurity Standards in 2023 [5] and a National Cybersecurity Strategy for 2024–2029 [6], aligned with NIST SP 800-53 [7].

Simultaneously, the cybersecurity threat landscape has evolved dramatically. Modern attacks employ sophisticated evasion techniques leveraging genetic algorithms [8], dynamic programming for payload optimization [9], and advanced AV/EDR bypass methods [10,11]. Compliance frameworks must validate not merely the presence of controls but their effectiveness against such adversarial techniques.

1.2. Research Problem

Current compliance auditing approaches face three critical challenges:

• Manual and Resource-Intensive: Traditional audits require extensive human effort, making continuous compliance monitoring infeasible for resource-constrained Rwandan institutions. Organizations conducting 3–5 internal audits annually achieve per-capita compliance costs averaging $154, while those without internal audits face costs of $341 [1].
• Checkbox Compliance: Existing frameworks verify control presence (e.g., “Is antivirus installed?”) without assessing effectiveness. Signature-based detection suffers 90–97% evasion rates against modern techniques [9,10].
• Binary Classification Inadequacy: Compliance is inherently multi-dimensional—a single log event may provide evidence for multiple controls simultaneously. Binary classification (compliant/non-compliant) oversimplifies this reality and discards actionable audit information.

1.3. Research Contributions

This work makes four contributions to automated compliance auditing:

• Generalization Gap Analysis and Hybrid Architecture: We identify and quantify a critical vulnerability in vocabulary-based log classifiers: an out-of-vocabulary (OOV) F1 gap of 37.8 percentage points when crossing log distribution boundaries (SSH in-vocabulary: 87.6% F1; Windows/HTTP out-of-vocabulary: 47.7% average F1; Phase 4 cross-dataset evaluation on 180 real logs). We demonstrate that a hybrid architecture pairing XGBoost (structured, in-vocabulary logs) with GPT-4o-mini semantic reasoning (out-of-vocabulary logs) addresses this gap—the combined hybrid system achieves 85.1% macro-F1 on 180 real logs with 6.4% false-positive rate, compared to XGBoost alone at 62.4% F1 (54.5% FPR).
• Adversarial-Aware Compliance Validation: Systematic integration of offensive security research (GA-based payload generation [8], MDP-based evasion [9], fileless reverse shell techniques [10]) into a defensive compliance framework. To the best of our knowledge, this is among the early documented methodologies connecting adversarial evasion testing to effectiveness-based NCSA control validation, revealing gaps invisible to presence-based checkbox auditing (e.g., T1190 XSS/SQLi: 100% detection; T1059.001 PowerShell fileless: 86% detection despite WAF/AV installation).
• Multi-Label Evidence-to-Control Mapping: Formulation of compliance auditing as multi-label classification, mapping evidence to multiple controls simultaneously (e.g., failed SSH login → AC-7, IA-2, AU-2, SI-4), with 169-dimensional output over 169 RWNCSA controls across 14 families. The average log event triggers 3.2 control evaluations; binary classification discards 69% of actionable audit information per event.
• Rwanda-Specific Deployment Evidence: An AI compliance auditor specifically engineered for Rwanda’s NCSA 2023 framework, to our knowledge one of the early documented such systems for an African national regulatory context, with empirical validation on production infrastructure (2.0 CPU cores, 2.66 GB RAM, $50/month), demonstrating that the cost barrier rather than technical feasibility is the primary obstacle to AI compliance adoption in African SMEs.

1.4. Research Questions

Primary Research Question (RQ0): Given the limitations of manual auditing in Rwandan hybrid cloud environments, can an AI-augmented compliance auditor employing hybrid architecture (ML-based evidence routing + LLM-based semantic analysis) for Rwanda NCSA Minimum Cybersecurity Standards (169 controls, 14 families, NIST SP 800-53 grounded) achieve macro F1-score within the 65–90% target range for real-world log classification while supporting ≥50% audit cycle reduction?

Secondary Research Questions:

RQ1 

How do adversarial techniques (GA, DP, LLM-based payload generation, AV/EDR evasion) inform compliance control effectiveness validation?

RQ2 

What are the comparative strengths of BERT, LSTM, and XGBoost for evidence-to-control mapping in resource-constrained environments?

RQ3 

Can hybrid datasets (real-world public logs + Rwanda-specific synthetic data) overcome the overfitting limitations of purely synthetic training?

RQ4 

How does multi-label classification improve upon binary compliance classification in national cybersecurity regulatory frameworks?

Section 2 reviews related work; Section 3 presents methodology and theoretical model; Section 4 details system implementation; Section 5 presents evaluation results; Section 6 discusses findings and implications; Section 7 outlines future directions; and Section 8 concludes.

2. Literature Review

2.1. Adversarial Security and Compliance

Recent research demonstrates sophisticated attack automation through optimization techniques. Understanding these adversarial methods is critical for validating compliance control effectiveness, not merely presence.

2.1.1. Genetic Algorithms for Security Testing

Liu et al. [8] developed GAXSS, a genetic algorithm for XSS payload generation achieving 97.5% accuracy on real CMS platforms. LLM-based payload generation [12] achieved 94.73% evasion against ClamAV. Dynamic programming approaches [9] reduce shellcode detectability by 97% through deterministic MDP optimization (15–20 iterations). These techniques collectively inform our adversarial test cases for SI-3 (Malicious Code Protection) and SI-10 (Input Validation) effectiveness validation.

Research Gap: None of these papers connect adversarial optimization techniques to regulatory compliance frameworks or provide a methodology for using offensive testing to evaluate control effectiveness within national cybersecurity standards. Our work bridges this gap by mapping adversarial test outcomes to specific NCSA control decisions.

2.1.2. AV/EDR Evasion Techniques

Custom reverse shell implementations [10] reduce AV detection from 62.9% to 1.4% through fileless shellcode execution, a 97% evasion improvement. MITRE ATT&CK documents 40+ defense evasion sub-techniques under TA0005 [11]. These results demonstrate that signature-based AV/EDR provides false compliance assurance: an institution with a nominally compliant SI-3 control can be trivially bypassed. Our system validates detection capability through adversarial testing rather than verifying installation presence.

2.1.3. Regulatory Context

Cybercrimes cost African economies an estimated $3.5 billion annually [13]; only 29 of 54 countries have enacted cybersecurity legislation [14], with notable efforts in Kenya [15] and Nigeria [16]. Rwanda’s NCSA Minimum Cybersecurity Standards (2023) [5] define 169 controls across 14 families, enforced through mandatory institutional audits, aligned with NIST SP 800-53 [7]. No prior work connects adversarial evasion testing to effectiveness-based validation within a national regulatory framework of this scope.

2.2. AI-Assisted Compliance and Log Analysis

Karlsen et al. [17] benchmarked 60 fine-tuned transformer models including BERT variants for security log analysis, with the best achieving F1 up to 0.998, and Villarreal-Vasquez et al. (2022) [18] applied LSTM-based anomaly detection on system event logs for insider threat identification with low false-alarm rates. However, both approaches assume GPU-equipped infrastructure unavailable in Rwanda. Mehavilla et al. [19] show that XGBoost achieves 96.96% F1 at 4% CPU utilization for flow-based intrusion detection, validating our resource-efficiency argument.

The SIEVE dataset [20] demonstrates a critical finding relevant to our work: SVM achieves macro-F1 of 0.93–0.97 on synthetic SIEM logs but degrades on real logs, and BERT follows the same pattern (0.95 synthetic, 0.89 real). This motivates our hybrid dataset strategy and validates the zero-shot degradation we observe in XGBoost.

Karlsen et al. [17] show that LLM and fine-tuned transformer performance varies across log formats and domains, a limitation our hybrid XGBoost+LLM architecture addresses using rule-based pre-filtering for high-confidence structured cases.

Our Contribution: XGBoost achieves comparable performance to BERT while being 137× smaller and 50× faster, enabling deployment on CPU-only infrastructure at $50/month, a viable cost for Rwandan institutions spending $154–$341 per audit currently.

2.3. Multi-Label Classification

Tsoumakas and Katakis [21] established foundational multi-label methods including Binary Relevance, Label Powerset, and algorithm adaptation. Chen and Guestrin [22] demonstrate XGBoost’s effectiveness across structured prediction tasks. Our application to compliance auditing is, to the best of our knowledge, among the first: a single log event inherently relates to multiple controls (e.g., “Failed SSH login” → AC-7, IA-2, AU-2, SI-4), and binary classification collapses this multi-dimensional compliance evidence into a single label, losing actionable control-specific information.

Research Gap: No prior compliance auditing system employs multi-label classification for evidence-to-control mapping, particularly within African national regulatory frameworks.

2.4. Positioning Against Existing Compliance Automation Tools

Table 1 compares our system against representative existing compliance and log analysis tools across dimensions directly relevant to the African SME deployment context and the effectiveness-based auditing goal.

Table 1. Comparison with existing compliance automation approaches.

System	ML-Based	Multi-Label	Adversarial	Africa-Specific	Cost
OpenSCAP [23]	No	No	No	No	Free
Wazuh [24]	Partial	No	No	No	Free
Qualys/Tenable	No	No	Limited	No	$15–50/ep/mo
AWS Config	No	No	No	No	Usage-based
Karlsen et al. [17]	BERT (GPU)	No	No	No	N/A
This work	XGBoost + LLM	Yes (169-dim)	Yes (5 MITRE ATT&CK)	Yes (NCSA)	$50/mo

OpenSCAP: rule-based OVAL/XCCDF scanner; Wazuh: SIEM with rule-based correlation; Qualys/Tenable: commercial vulnerability management, not compliance-effectiveness focused; AWS Config: cloud-native configuration compliance, no adversarial or ML layers. N/A: cost not applicable or not reported for the referenced study. Bold: current work being presented.

Our system occupies a distinct position: among the few approaches combining ML-based multi-label evidence routing, adversarial effectiveness validation, national regulatory specificity, and sub-$100/month deployability. OpenSCAP and Wazuh provide complementary rule-based coverage without adversarial or ML capabilities; commercial tools provide broader coverage at prohibitive cost for Rwanda SMEs. This gap motivates the hybrid architecture: deterministic rules for NCSA-specific thresholds plus ML/LLM for semantic generalization and adversarial evidence analysis.

3. Methodology

All experiments were conducted using Python 3.11 (Python Software Foundation, Wilmington, DE, USA), XGBoost 2.0.3 [22] (DMLC, distributed), scikit-learn 1.3.2 (scikit-learn developers, distributed), and NumPy 1.26.2 (NumFOCUS, Austin, TX, USA). GPT-4o-mini API inference used OpenAI API (OpenAI, San Francisco, CA, USA) at temperature 0. All components were containerised using Docker and deployed on Kubernetes using Python 3.11-slim base images.

3.1. System Architecture

3.1.1. Seven-Engine Microservices Design

The system employs a microservices architecture with seven independent engines communicating via Redis pub/sub and REST APIs, as shown in Figure 1 and Table 2.

Figure 1. Seven-engine microservices architecture with PostgreSQL and Redis infrastructure.

Table 2. Engine functional breakdown.

Engine	Function
Engine 1	Log collection (syslog, Windows Events, files, APIs); normalization to unified schema
Engine 2	Document processing (PDF policy evidence extraction via LLM)
Engine 3	MCP+LLM hybrid analyzer: 60 evidence parsers, dual-path analysis
Engine 4	Decision engine: 143 control-specific rule evaluators (of 169 total RWNCSA controls; 26 require manual assessment)
Engine 5	Report generation (PDF/CSV compliance reports via ReportLab)
Engine 6	Web UI (React, TypeScript, Tailwind; audit wizard; WebSocket real-time progress)
Engine 7	Authentication (JWT, RBAC: admin, analyst, viewer, api_user)

Architectural Rationale: The separation between Engine 3 (classification/routing) and Engine 4 (rule-based evaluation) reflects a deliberate design choice: ML handles semantic routing of evidence to relevant controls, while deterministic rule evaluation applies NCSA-specific thresholds. This prevents ML uncertainty from propagating into compliance decisions and preserves explainability. MCP serves as an engineering integration layer that standardizes the LLM interface [25,26]; the scientific novelty lies in the hybrid routing and adversarial validation methodology, not in the protocol choice.

3.1.2. Data Flow

The complete pipeline is illustrated in Figure 2.

Figure 2. End-to-end compliance auditing pipeline: heterogeneous sources through hybrid MCP+ LLM analysis, decision evaluation, and report generation.

For multi-label classification, each evidence item $x_i$ maps to a binary label vector:

$${\mathbf{y}}_i=[y_{i1},y_{i2},\dots ,y_{i169}]\in {\{0,1\}}^{169}$$

(1)

where $y_{ij}=1$ if evidence $x_i$ is relevant to control $c_j$ (169 controls, 14 families).

3.2. Dataset Strategy

3.2.1. Evolution from Synthetic to Hybrid

Initial purely synthetic data (100,000 events) revealed critical overfitting. All three baseline models achieved 100% accuracy, a clear red flag. Investigation identified:

Data Leakage (Critical—identified and corrected in this revision): A comprehensive audit of the original training dataset revealed that 86.3% of synthetic rows contained embedded compliance verdicts within the log_message field (e.g., literal text “compliant” or control IDs appearing in log strings), in addition to a $-0.97$ Pearson correlation between status_code and the compliance label. Models trained on this data were effectively memorizing labels rather than learning semantic patterns, explaining the artificially inflated 99.99% F1. Following identification, the entire synthetic dataset was regenerated from scratch (generate_synthetic_data_v2.py): all 169 RWNCSA controls across 14 families, zero label leakage (verified by automated audit), and no control-ID or compliance-verdict text in log messages. The corrected XGBoost v2 achieves 85.45% macro-F1 on this clean dataset, an honest performance estimate substantially below the leaky 99.99%. Additionally, status_code was removed from all feature sets.

Template Simplicity: Synthetic log messages used fixed templates with low lexical diversity, enabling vocabulary memorization rather than semantic understanding.

3.2.2. Hybrid Dataset Composition and Relevance

Dataset-to-Control Relevance: NSL-KDD [27] provides 42 attack categories covering NCSA controls: DoS attacks → AC-7 (lockout policy), SI-4 (monitoring); Probe (reconnaissance) → AU-6 (audit review), SC-7 (boundary protection); R2L (remote-to-local) → IA-2 (authentication), AC-3 (access enforcement); U2R (privilege escalation) → AC-6 (least privilege), SI-4. LogHub [28] HDFS, OpenStack, and Linux system logs provide coverage for AU-12, CM-7, SI-2, and AC-2. Together, these datasets cover 67 of 143 implemented controls with real-world attack patterns. Rwanda-specific synthetic data covers the remaining 76 NCSA controls not represented in public datasets, ensuring complete label coverage for all 143 control families.

Dataset Size Justification: The 200,000-event total was determined by three constraints: (i) minimum representation of all 143 control classes at ≥100 positive samples for reliable multi-label learning; (ii) statistical power for 5-fold cross-validation ($n=100,000$ per fold exceeds the $n\ge$ 1000 rule of thumb for high-dimensional multi-output classifiers [21]); and (iii) CPU tractability (≤15 min per fold on commodity hardware without GPU). The 70/30 real/synthetic split reflects available public data covering 67 of 143 controls (real) versus the remaining 76 NCSA-specific controls requiring synthetic augmentation, as summarized in Table 3.

Table 3. Hybrid dataset composition (200,000 logs).

Source	Count	%	Type
NSL-KDD [27]	104,000	52%	Network intrusions
LogHub [28]	36,000	18%	System logs
Rwanda Synthetic	60,000	30%	NCSA-specific
Total	200,000	100%	Hybrid

Bold: dataset total row.

Split: 70/15/15 train/validation/test with stratified sampling across control families. Future work includes validation on production logs from Rwandan institutions (pilot deployment collaboration is ongoing).

Revised dataset (v2, leakage-free): The leakage-free re-generation (generate_synthetic_ data_v2.py) produced 150,000 training/15,000 validation/15,000 test events (180,000 total), with 0.0% label leakage verified by automated audit (scripts/data_audit.py). All XGBoost v2 results in Section 5 use this v2 dataset.

3.2.3. Additional Evaluation Datasets

For cross-dataset generalization testing:

• SecRepo Auth Logs [29]: 86,839 real SSH authentication events (failed password attempts, invalid user logins, session acceptances).
• SIEVE [20]: 30-class expert-labeled SIEM event classification dataset.
• LANL Authentication [30]: Sampled 50K enterprise authentication events from 708M total.

3.3. Model Development and Selection

3.3.1. Baseline Models

BERT (110M parameters): Fine-tuned bert-base-uncased, frozen embeddings and first 10 layers. 2 epochs, batch 16, LR 2e-5. Result: 12 h training, 2.7 GB model, 45 ms latency.

LSTM (758K parameters): 2-layer bidirectional LSTM, 128 hidden units, dropout 0.3. 5 epochs, vocabulary 946. Result: 2 h training, 800 MB model, 12 ms latency.

XGBoost: Gradient boosted trees, max depth 6, learning rate 0.1, 500 estimators (early stopping at 93). Result: 8 min training, 350 MB model, <1 ms latency.

3.3.2. Production Selection: XGBoost

Finding F3 (RQ2): XGBoost achieves higher classification performance than BERT (99.49% ± 0.3% vs. 96.15% ± 0.8% F1, 5-fold CV; Table 4) with non-overlapping confidence intervals confirming statistical significance, while requiring 137× less storage and 50× lower inference latency, empirically validating that resource-efficient models do not sacrifice accuracy for compliance auditing tasks. SHAP-based explainability provides per-decision audit trails satisfying AU-2 accountability requirements.

Table 4. Model comparison for production deployment.

Metric	BERT	LSTM	XGBoost
F1-Score (mean ± 95% CI)	96.15% ± 0.8%	96.11% ± 1.2%	99.49% ± 0.3%
Accuracy	96.21%	96.08%	99.49%
Training Time	12 h	2 h	8 min
Model Size	2.7 GB	800 MB	350 MB
Latency	45 ms	12 ms	<1 ms
Memory	2.1 GB	580 MB	380 MB
Explainability	Low	Low	High (SHAP)
GPU Required	Yes	No	No

95% CI computed via 5-fold stratified cross-validation (n = 100,000). Note: these results were obtained on the original training dataset, which subsequently had an 86.3% data-leakage rate identified (Section 3, “Data Leakage”). XGBoost was retrained on leakage-free data (XGBoost v2): 85.45% macro-F1 (see Section 5). BERT and LSTM were not re-evaluated; their relative resource characteristics remain valid for the deployment comparison. The deployment recommendation—XGBoost for CPU-only resource-constrained environments—is unchanged. Bold: best-performing model per metric.

3.4. Feature Engineering

Numeric Features (5): Hour of day (0–23), day of week (0–6), business hours binary flag, network port number, event severity level.

Text Features (50 TF-IDF): log_message field with bigrams, minimum document frequency 2, implemented as a pure-Python 3.11 SimpleTfidf class (no scipy dependency) to facilitate serialization. Representative high-weight terms from the learned vocabulary: accepted, for user, port ssh2:, by (uid = 0), session opened, failed, denied, kernel: [ufw, process exited. status_code and all control-ID text removed due to identified leakage.

One-Hot Family Features (14): One binary feature per RWNCSA control family, enabling the model to weight patterns differently by log source context.

Total: 68 features per event (50 TF-IDF + 4 numeric + 14 family one-hot). Note: the vocabulary is SSH/syslog-centric; Windows Event and HTTP log tokens are absent, producing zero TF-IDF activations (out-of-vocabulary), which is the root cause of the 37.8-pp generalization gap documented in Section 5.5.

3.5. Multi-Label Formulation

Problem: Map evidence $x_i$ to a subset of controls ${\mathcal{C}}_i\subseteq \{c_1,\dots ,c_{169}\}$.

Implementation: Multi-output XGBoost via scikit-learn’s MultiOutputClassifier wrapper, producing 169-dimensional binary output vectors.

Evaluation Metrics:

$$F{1}_{\mathrm{macro}}={\displaystyle \frac{1}{169}}{\displaystyle \sum _{k=1}^{169}}F{1}_k$$

(2)

$$F{1}_{\mathrm{micro}}={\displaystyle \frac{2·P_{\mathrm{all}}·R_{\mathrm{all}}}{P_{\mathrm{all}}+R_{\mathrm{all}}}}$$

(3)

$${\mathcal{L}}_H={\displaystyle \frac{1}{N·169}}{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{j=1}^{169}}⊮[y_{ij}\ne {\widehat{y}}_{ij}]$$

(4)

Macro-F1 (Equation (2)) is our primary metric as it weights each control equally, preventing high-frequency controls from dominating evaluation.

3.6. Evolution to Semantic LLM Analysis

The hybrid LLM architecture emerged from a systematic progression through three phases, each exposing a limitation that motivated the next step.

Phase 1—Classifier Comparison (BERT, LSTM, XGBoost): Initial evaluation of all three candidate models on the hybrid synthetic dataset (Section 3) demonstrated that while BERT (96.15% F1) and LSTM (96.11% F1) offered competitive accuracy, both require GPU infrastructure unavailable in resource-constrained deployments. XGBoost was selected for production: 99.49% F1, 8 min training, 350 MB model, <1 ms inference on CPU-only hardware (Table 4).

Phase 2—Generalization Gap Discovery: Cross-dataset evaluation on real SecRepo logs (86,839 samples) revealed a critical failure: XGBoost zero-shot F1 collapsed to 7.98%, a 92-point gap from synthetic performance. The root cause is architectural: TF-IDF features capture vocabulary co-occurrence patterns rather than semantic meaning. A log containing “authentication failure” is correctly classified only if that exact vocabulary appeared in training data with the same distribution. Fine-tuning on 5% target-domain data recovers performance (99.88% F1 on the same SSH domain; this result reflects SSH-specific vocabulary adaptation and should not be interpreted as cross-domain or general-purpose performance), but this approach does not scale: a new log type requires a new labeled sample collection, defeating the goal of zero-shot generalization across institutional environments. BERT was reconsidered at this point but rejected for the same infrastructure constraint: its semantic capability requires GPU-backed inference impractical at $50/month target cost.

Phase 3—LLM Semantic Path: Large language models offer semantic understanding without retraining, their pre-training on vast corpora enables zero-shot reasoning about security log semantics regardless of format. GPT-4o-mini (cloud API) was evaluated as a zero-shot classifier across four log types (Section 5), achieving 92.3% overall macro accuracy—providing preliminary evidence that semantic LLM reasoning can address the vocabulary-based generalization gap at acceptable deployment cost, while remaining insufficient for broad claims of LLM superiority in cybersecurity auditing without larger-scale validation. An integration layer was needed to connect the compliance pipeline to these LLM providers in a provider-agnostic way.

3.6.1. Model Context Protocol Integration

To overcome vocabulary dependency, Engine 3 integrates LLMs via Model Context Protocol (MCP) [31,32], an open standard that serves as the engineering integration layer between the compliance pipeline and external LLM providers. MCP standardizes the tool-use interface, enabling provider-agnostic LLM calls without vendor lock-in; it is not a scientific contribution of this paper but rather a practical interface choice that simplifies deployment and maintenance:

Dual-Path Architecture:

• Rule-based fast path: Regex pattern matching for high-confidence, unambiguous events (e.g., “Failed password” → non-compliant, “Accepted publickey” → compliant), achieving <1 ms latency. Handles 60–70% of logs.
• LLM semantic path: Claude/GPT analysis for ambiguous logs requiring contextual interpretation, achieving 200–500 ms latency. Handles 30–40% of logs.

3.6.2. LLM Evaluation Protocol

Two complementary evaluations were conducted. Evaluation 1 (SSH deep-dive): a stratified random sample of 500 SecRepo authentication logs—300 failed authentication events (60%), 150 successful authentication events (30%), and 50 session anomalies (10%). Ground truth labels established through automated rule-based analysis cross-validated with manual review of all 50 anomalous cases. Evaluation 2 (multi-log-type expanded, GPT-4o-mini, temperature = 0): 200 samples per log type across four structurally distinct categories—SSH authentication, macOS system/service logs, HTTP/API access logs (real enterprise Squid proxy data, SecRepo, 10.105.x.x internal IPs, n = 28 unique real entries), and Windows Security Events (Mordor APT3 PurpleSharp, n = 200, enriched via WINDOWS_EVENT_DESCRIPTIONS table)—totalling n = 628 samples with real enterprise data for all log types. Ground truth for each type established by domain-specific rule-based classifiers (EventID labels for Windows; HTTP status code labels for proxy logs; SSH rule patterns for authentication logs), all cross-validated manually with Wilson 95% confidence intervals throughout.

3.6.3. Cost Optimization

• MD5-keyed response caching (10,000 entry LRU cache): identical log formats reuse cached decisions.
• Model tiering: Claude Haiku for batch processing ($0.00025/1K tokens), Claude Sonnet for single-event analysis.
• Rule-based pre-filtering eliminates LLM calls for 60–70% of logs.
• Estimated cost: $0.15/10,000 logs with hybrid strategy vs. $1.50 with LLM-only.

4. System Implementation

4.1. Control Coverage

The system implements evidence parsers for 143 of the 169 RWNCSA Minimum Cybersecurity Standards controls across 14 families (Table 5), with 96 controls supported by 60 specialized automated evidence parsers executing macOS audit commands. The remaining 26 controls require physical inspection or manual policy review and are assessed via analyst-completed checklists. The XGBoost v2 binary compliance classifier (Section 3) was trained to classify log events against all 169 controls.

Table 5. RWNCSA control families and XGBoost v2 per-family classification performance (leakage-free, Wilson 95% CI).

Control Family	Controls	Test n	Macro-F1 [ CI ]
Access Control (AC)	22	2333	84.8% [83.3%, 86.2%]
Audit & Accountability (AU)	16	2328	83.6% [82.0%, 85.1%]
Awareness & Training (AT)	8	597	87.2% [84.3%, 89.7%]
Configuration Management (CM)	12	1222	80.2% [77.8%, 82.3%]
Identity Mgmt & Authentication (IA)	14	1177	100.0% [99.7%, 100.0%]
Incident Response (IR)	7	568	78.6% [75.1%, 81.8%]
Maintenance (MA)	6	642	92.2% [89.8%, 94.0%]
Media Protection (MP)	5	781	100.0% [99.5%, 100.0%]
Personnel Security (PS)	9	1003	89.5% [87.5%, 91.3%]
Physical & Environmental (PE)	10	868	61.7% [58.4%, 64.9%]
Risk Assessment (RA)	5	261	75.7% [70.2%, 80.5%]
Security Assessment (CA)	7	339	88.7% [84.9%, 91.7%]
Security Policy & Procedures (SP)	12	1433	88.0% [86.2%, 89.6%]
System & Comm. Protection (SC)	16	1448	80.5% [78.4%, 82.5%]
Total/Mean	169	14,582	85.05% (macro)

The Rwanda NCSA Minimum Cybersecurity Standards define 169 controls across 14 families. Of these, 143 are currently supported by automated system-auditable evidence parsers; the remaining 26 require physical inspection or manual policy review. XGBoost v2 was trained as a binary compliance classifier on leakage-free synthetic data covering all 169 controls. CI = Wilson 95% confidence interval. Bold: overall totals and macro mean. n: test sample size.

Evidence Parsers: Each system-auditable control has a dedicated parser executing macOS audit commands. The 60 parsers are organized by family: Access Control (26): login history via last, user enumeration via dscl, SSH configuration via sshd - T, screen lock, file permissions; Audit & Accountability (7): audit subsystem via auditctl, NTP sync via ntpq, log retention; Identity & Authentication (5): password policy via pwpolicy, MFA certificates; System & Communications Protection (11): firewall via pfctl, encryption via fdesetup status, VPN, WiFi; System & Information Integrity (4): SIP via csrutil status, XProtect/Gatekeeper; Configuration Management (7): software inventory, patch status.

4.2. Decision Engine Architecture

Each control implements a dedicated parser and decision function returning a ComplianceResult: status ∈ {compliant, partial, non_compliant}, confidence $\in [0,1]$, gaps list, and remediation recommendation. Control-specific thresholds implement NCSA requirements: AC-7 checks lockout threshold $\le 5$ failed attempts within 15 min; IA-5 checks password minimum length $\ge 12$ characters; SC-28 checks FileVault encryption enabled.

4.3. Database Schema

PostgreSQL stores the control taxonomy (controls: control_id, family, title, description, ncsa_mapping, priority, tier) and audit results (audit_results: audit_id, control_id, status, confidence, reason, gaps, recommendations, audited_at). Redis pub/sub channels enable real-time audit progress streaming to the WebSocket-connected dashboard.

4.4. Kubernetes Deployment

Namespace rwanda-compliance with 2–3 replicas per engine, resource limits 512 Mi–1 Gi memory and 250 m–500 m CPU per pod. NGINX ingress controller routes external traffic; ClusterIP services handle internal inter-engine communication. Liveness and readiness probes on /health endpoints with 30 s initial delay ensure zero-downtime pod restarts.

5. Results and Evaluation

5.1. Classification Performance (XGBoost)

Table 6 presents XGBoost classification results with 5-fold stratified cross-validation on a 100,000-sample subset of the leakage-free v2 training corpus (stratified; full training set: 150,000 rows).

Table 6. XGBoost v2 classification results—leakage-free (5-fold CV, binary compliance classifier, 169 RWNCSA controls, Wilson 95% CI).

Metric	Value
CV Macro-F1 (5-fold)	85.20% ± 0.25%
CV Accuracy (5-fold)	87.14%
Test Macro-F1 (hold-out)	85.45%
Test Accuracy (hold-out)	87.14%
Wilson 95% CI on Test F1	[84.9%, 86.0%]
Training time/fold	89.09 ± 7.25 s
Inference Latency (500 samples)
p50	0.32 ms
p95	5.40 ms
p99	8.78 ms

Wilson 95% CI computed on the held-out test set (n = 15,000; p = 0.8545; z = 1.96). Per-family CIs are in Supplementary Section S1. Note: the previous submission reported 99.99% F1, attributable to an 86.3% data-leakage rate identified and corrected for this revision. Bold: primary reported metric. Italic: secondary metric category.

Interpreting Corrected Synthetic Performance: The corrected 85.45% macro-F1 reflects leakage-free learning: XGBoost v2 was trained on freshly generated synthetic data with no label leakage (0.0% leakage verified by automated audit). The per-family variation in Table 5 (range: 61.7–100.0%) reveals meaningful heterogeneity: families with structured, vocabulary-rich log patterns (Identity Management: 100.0%; Media Protection: 100.0%) are easier to classify than families requiring physical or behavioral context (Physical & Environmental: 61.7%; Risk Assessment: 75.7%). The training vocabulary (50 terms, SSH/syslog-centric) limits performance on out-of-vocabulary log types, as quantified in Section 5.5.

Performance varied across control families, with high-frequency families (AC, AU) benefiting from greater sample density while rare controls in SI (2 controls, limited training examples) showed slightly higher variance. The 5-fold cross-validation standard deviations (F1: ±0.01%) confirm stable performance without overfitting to specific folds. The critical validation of generalization is the cross-dataset evaluation (Section 5.5), where zero-shot performance degrades substantially (7.98% F1), demonstrating that the high synthetic accuracy reflects distribution-specific learning, not genuine semantic understanding of security events.

5.2. Cross-Dataset Generalization

Finding F1 (RQ3): The corrected 85.45% macro-F1 (vs. the prior 99.99%) reflects genuine learning from leakage-free data. The per-family range (61.7–100.0%) reveals real model behaviour: families with structured log vocabularies achieve high F1; families requiring physical/behavioural context show lower F1. Section 5.5 provides the real-world generalization analysis on 180 independent logs (Table 7).

Table 7. XGBoost v2 generalization: leakage-free results (binary compliance classifier, $n_{\mathrm{synthetic} \mathrm{test}}=14,582$; Phase 4 real-world evaluation described in Section 5.5).

Evaluation Scenario	Macro-F1	Accuracy	FPR
XGBoost v2—Leakage-free synthetic CV	85.45%	87.14%	—
Note (prior submission): original 99.99% F1 was inflated by 86.3% data leakage; corrected above.

Bold: primary leakage-free result. Italic: historical reference row retained for transparency.

5.3. LLM Semantic Analysis Results

LLM Accuracy Scope: The 89.5% SSH accuracy reflects a semantically clear binary classification task in highly structured syslog format with domain-invariant security terminology. Ground truth was established through automated rule-based analysis (“Failed password” → non-compliant; “Accepted” → compliant), cross-validated with manual review of 50 ambiguous samples. Table 8 presents the SSH-focused comparison; accuracy is log-type-dependent across all four log types (Table 9), with macOS achieving 96.5% on synthetic data while HTTP achieves 85.7% on real enterprise proxy logs.

Table 8. LLM vs. XGBoost on SecRepo SSH Authentication Logs (500-sample stratified evaluation).

Metric	XGBoost	LLM (MCP)
Zero-shot Accuracy	5.92%	89.5% †
Zero-shot F1	7.98%	87.6% †
Average Confidence	87.3%	95.2%
Latency (per log)
Rule-based path	—	0.04 ms
LLM path	—	280 ms
XGBoost	0.32 ms	—
Resource Requirements
Model Storage	350 MB	0 MB (API)
Memory (runtime)	380 MB	50 MB
GPU Required	No	No

^† Accuracy is log-type-dependent; see Table 9 for the full multi-log evaluation (GPT-4o-mini, n = 628, macro accuracy 92.3%). SSH result (89.5%) is from a 200-sample evaluation (50% real linux_auth.log); this table reflects an earlier 500-sample SSH-only evaluation retained for historical comparison. Bold: better-performing system per metric. Italic: metric category headers.

Table 9. GPT-4o-mini zero-shot Accuracy by Log Type (Rwanda NCSA, expanded evaluation with real enterprise data, Wilson 95% CI).

Log Type	n	Accuracy	Wilson 95% CI	Data Source
SSH Authentication	200	89.5%	[84.5%, 93.0%]	50% real (linux_auth.log)
Windows Security Events	200	97.5%	[94.3%, 98.9%]	100% real (Mordor APT3) ‡
HTTP/API Access	28	85.7%	[68.5%, 94.3%] †	100% real (SecRepo Squid)
macOS System/Service	200	96.5%	[93.0%, 98.3%]	Synthetic (no public dataset)
Overall (macro)	628	92.3%	[89.9%, 94.3%]	37.5% real
XGBoost v2 alone (baseline for comparison)
SSH (in-vocabulary)	100	89.0% F1	—	Real
Windows/HTTP (OOV)	80	28.6–66.7% F1	—	Real

GPT-4o-mini: temperature = 0, zero-shot, May 2026. Ground truth: rule-based classifiers + manual cross-validation. ^‡ Windows EventID enrichment via WINDOWS_EVENT_DESCRIPTIONS table is a prerequisite; bare EventID integers yield ≈3% accuracy (OOV for LLM context window). ^† HTTP n = 28 (14 compliant + 14 non-compliant from SecRepo Squid proxy log, 10.105.x.x internal IPs, named enterprise users); smaller than other types because real enterprise HTTP proxy logs with internal IPs are scarce in public datasets. 4 of 14 compliant samples were flagged non-compliant by the LLM (ad-tracker/messaging app URLs—intelligent policy-sensitive behaviour rather than classifier error). macOS evaluation remains synthetic; no public macOS enterprise audit log datasets were identified. Bold: macro-average across all log types. Italic: XGBoost baseline reference rows.

Finding F4 (RQ0, LLM generalization): To validate generalizability beyond SSH logs, zero-shot evaluations were conducted across four structurally distinct log types using GPT-4o-mini (cloud API, temperature = 0), testing the deployment scenario for resource-constrained institutions (Table 9). Results provide exploratory evidence that LLM accuracy is log-type-dependent but consistently higher than XGBoost zero-shot (7.98%) across all evaluated categories; this constitutes preliminary applied validation and does not support broad claims of LLM superiority in general cybersecurity auditing contexts.

Key Findings from Expanded Evaluation: GPT-4o-mini achieves 92.3% macro accuracy across 628 samples and 4 log types using real enterprise data (37.5% real-world). The Windows result (97.5% at n = 200) is significantly more robust than the initial n = 50 figure and depends on EventID enrichment via WINDOWS_EVENT_DESCRIPTIONS: bare EventID integers yield ≈3% accuracy because they provide no semantic context to the LLM. The HTTP result (85.7%) is measured on real enterprise Squid proxy logs with internal IPs—lower than the previously reported 95.5% on fully synthetic data because real enterprise traffic includes ambiguous content (ad-tracker and messaging application URLs that the LLM correctly identifies as policy-questionable despite HTTP 200 status codes). The macOS evaluation (96.5%) remains on synthetic data; no public macOS enterprise audit log datasets were identified, and this remains an acknowledged limitation.

Statistical Scope: Wilson 95% confidence intervals are reported per cell (Table 9). At n = 200 per structured log type, CIs narrow to ±3–5 percentage points. The HTTP result has a wider CI (±13 pp) due to the smaller n = 28 real-data sample. These sample sizes support exploratory and applied evaluation conclusions; they are not sufficient to support broad claims of LLM superiority in general cybersecurity auditing, and results should be interpreted as preliminary evidence for the specific log types and compliance context evaluated. Validation on production Rwandan institution logs ($n\ge 500$ per type) remains for future deployment pilots.

Error Analysis: Misclassifications in both models predominantly produced partial labels rather than opposite-polarity errors, confirming LLM uncertainty rather than systematic misunderstanding. SSH errors arose from semantically ambiguous disconnection logs (“Connection closed [preauth]”, “Received disconnect…Bye Bye”); macOS errors involved services with dual-interpretation status (screen sharing, SMB sharing where organizational policy context is required). This pattern, uncertain partial labels rather than confident wrong labels, is favorable for compliance auditing: ambiguous events trigger human review rather than silently incorrect certifications, satisfying AU-2 accountability requirements.

5.4. Adaptive Routing: Vocabulary-Coverage Gating (Phase II)

The hybrid architecture routes each log through either XGBoost (fast path) or the LLM (semantic path) based on a vocabulary-coverage signal:

$$\nu \left(l\right)={\displaystyle \frac{\left|\mathrm{words}\right(l)\cap \mathcal{V}|}{\left|\mathrm{words}\right(l\left)\right|}},\hspace{1em}\mathrm{route}\left(l\right)=\left\{\begin{array}{cc}\mathrm{XGBoost}\hfill & \nu \left(l\right)\ge \mathit{\theta }\hfill \\ \mathrm{LLM}\hfill & \nu \left(l\right)<\mathit{\theta }\hfill \end{array}\right.$$

(5)

where $\mathcal{V}$ is the TF-IDF training vocabulary and $\mathit{\theta }$ is the coverage threshold. A threshold sweep across $\mathit{\theta }\in \{0.00,0.01,0.05,0.10,0.15,0.20,0.30,0.50,1.00\}$ was conducted on a mixed evaluation set of n = 220 logs (20 SSH in-distribution + 200 OOD logs from the 4-type expanded evaluation); results are shown in Table 10.

Table 10. Phase I vs. Phase II router comparison (n = 220 mixed logs).

Router	Accuracy	LLM Call Rate	Cost/10K logs
XGBoost only (no routing)	42.3%	0.0%	$0.001
LLM only	94.5%	100.0%	$0.150
Phase I (static regex routing)	89.1%	90.9%	$0.137
Phase II ($\mathit{\theta }$ = 0.20, vocab-cov)	94.5%	99.5%	$0.149

Cost model: LLM path $0.15/10K logs (GPT-4o-mini); XGBoost $0.001/10K logs (CPU inference). Bold: recommended configuration (best accuracy–cost trade-off).

Finding F7 (Phase II): Vocabulary-coverage gating at $\mathit{\theta }$ = 0.20 matches LLM-only accuracy (94.5%) while reducing unnecessary XGBoost calls on OOD logs. Phase II improves over Phase I by +5.4 percentage points in accuracy (+0.6 pp over LLM-only overall) at a marginal cost increase of $0.012/10K logs. The OOD detection signal is near-perfect: vocabulary coverage $=0.00$ for all HTTP, Windows Event, and macOS logs (F1 = 0.93; Precision = 0.91, Recall = 0.96 at $\mathit{\theta }$ = 0.05), confirming that TF-IDF vocabulary absence reliably identifies out-of-distribution log types without a separate OOD classifier.

5.5. Cross-Dataset Generalization Analysis

To measure the operational boundary of the XGBoost v2 classifier, a cross-dataset generalization evaluation was conducted on three independent real-world datasets (180 total real logs), contrasting in-vocabulary (SSH/PAM) versus out-of-vocabulary (Windows, HTTP) log types (Table 11).

Table 11. XGBoost v2 cross-dataset generalization (real-world evaluation, n = 180; compared against leakage-free synthetic test F1 = 85.45%).

Dataset	n	F1%	FPR%	Gap vs. Synthetic	In-Vocab
SSH (linux_auth.log + Elastic)	100	87.6%	0.0%	$-2.1$ pp †	Yes
Windows (Mordor APT3)	60	28.6%	100.0%	$+56.9$ pp	No
HTTP (Elastic Apache)	20	66.7%	100.0%	$+18.8$ pp	No
In-vocab mean		87.6%	0.0%	$-2.1$ pp
OOV mean		47.7%	100.0%	$+37.8$ pp

^† Negative gap = real-world F1 exceeds synthetic, confirming the model generalises to real SSH attack traffic. OOV gap = 37.8 percentage points; Windows and HTTP produce zero TF-IDF activations (0.0 average active features) because the training vocabulary is entirely SSH/syslog-centric. Source: scripts/cross_dataset_eval_v1.py; results in reports/cross_dataset_eval.json. Bold: in-vocabulary aggregate result.

Finding F1 (RQ3)—Revised: The corrected generalization analysis reveals selective generalisation: XGBoost v2 performs comparably on real SSH logs ($-2.1$ pp gap, within measurement noise) but fails on out-of-vocabulary log types (Windows: $+56.9$ pp degradation; HTTP: $+18.8$ pp). The root cause is architectural: the 50-term TF-IDF vocabulary was built from synthetic SSH/syslog training data; Windows Security Event strings and HTTP access log tokens are entirely absent. This OOV failure directly motivates the hybrid architecture: XGBoost handles the log types it knows (SSH/PAM), and GPT-4o-mini handles the rest.

5.6. Hybrid System Combined Evaluation

A key reviewer concern was that system components were evaluated in isolation. To address this, a combined evaluation was conducted on the same 180 real-world logs under three system configurations (Table 12). The hybrid routing rule: if TF-IDF active features > 0 (log is in-vocabulary), route to XGBoost; if TF-IDF active features $=0$ (log is out-of-vocabulary), route to GPT-4o-mini. This rule requires no log-type label at runtime.

Table 12. Hybrid system combined evaluation on 180 real-world logs (SSH n = 100, Windows n = 60, HTTP n = 20; XGBoost routing: 107/180 to XGBoost, 73/180 to GPT-4o-mini).

System	Accuracy	Macro-F1	FPR	F1 Wilson 95% CI
XGBoost v2 alone	60.6%	62.4%	54.5%	[74.0%, 91.0%] †
GPT-4o-mini alone	92.8%	90.8%	6.4%	[82.5%, 96.0%]
Hybrid combined	88.9%	85.1%	6.4%	[70.8%, 88.8%]
Log Type	XGBoost F1	GPT-4o-mini F1	Hybrid F1
SSH (n = 100, in-vocab)	87.6%	95.8%	87.6%
Windows (n = 60, OOV)	28.6%	74.1%	74.1%
HTTP (n = 20, OOV)	66.7%	88.9%	88.9%

^† Wide CI reflects XGBoost’s near-random performance on OOV log types (100% FPR on Windows and HTTP). Source: scripts/hybrid_system_eval.py; results in reports/hybrid_system_eval.json. Routing: 107/180 to XGBoost (SSH/PAM, all had ≥1 TF-IDF active feature), 73/180 to GPT-4o-mini (Windows + HTTP, 0 TF-IDF active features). Bold: recommended hybrid configuration. Row headers in the lower panel denote per-log-type breakdown.

Finding F7 (Hybrid Architecture): The combined hybrid system addresses the reviewer concern that components were tested in isolation. On the same 180 real logs: XGBoost alone achieves 62.4% macro-F1 with 54.5% FPR (OOV failure dominates); GPT-4o-mini alone achieves 90.8% F1 with 6.4% FPR; the hybrid achieves 85.1% F1 with 6.4% FPR—8 percentage points below the LLM-only ceiling because SSH routes to XGBoost where GPT-4o-mini would perform marginally better (87.6% vs. 95.8% F1). This trade-off is intentional: routing SSH to XGBoost reduces API cost by ≈59% of log volume at the cost of 8.2 pp on SSH F1. For the Windows and HTTP types where XGBoost fails entirely, the hybrid achieves LLM-equivalent performance (74.1% and 88.9% F1 respectively).

5.7. Latency and Throughput

Throughput: Single pod: ∼100 events/s; 3-pod deployment: ∼300 events/s (linear scaling observed to 5 pods). Training: 89.09 ± 7.25 s per fold on CPU.

5.8. Resource Utilization

Total system footprint across all 7 engines: 2.66 GB RAM, 2.0 CPU cores. Minimum deployment: 4 GB RAM, 4 cores at ∼$50/month cloud hosting (benchmarked on Apple M1 Pro 8-core, 16 GB RAM; cloud equivalent: 2-vCPU/4 GB instance; storage costs, network egress, and maintenance labour excluded from this estimate), substantially lower than enterprise compliance solutions charging $15–$50/endpoint/month.

5.9. Kubernetes Scaling Test

Engine 3 scaled to 3 replicas under 300 concurrent requests:

• Load distribution: 34.0%/32.7%/33.3% (near-uniform, within 1.3% spread).
• Latency: 1.8 ms mean, 4.2 ms p99.
• Failures: 0 across 10 independent test iterations.

5.10. Live System Deployment Validation

To validate end-to-end system correctness beyond simulated benchmarks, a full compliance audit was executed on the research machine (macOS 26.3.1) using the production seven-engine pipeline (Audit ID: AUDIT-20260320-205239, 20 March 2026). The system autonomously collected evidence via 60 parsers across 5 control families, classified findings via XGBoost and the Decision Engine, and generated a 3-page structured PDF report in 0.77 s.

Results across 10 audited controls:

• 5/10 compliant: FileVault encryption (AC-17), Gatekeeper active (CM-6), Firewall enabled (SC-7), Microsoft Defender ATP running (SI-3), audit logging active (AU-2).
• 5/10 partial: Password policy below NCSA threshold, screen sharing unreviewed (AC-17 partial), SIP status requiring investigation, MDM enrollment gap, SSH key rotation overdue.
• 0/10 non-compliant: No outright policy violations detected.
• Overall compliance score: 75.0%.

This live deployment demonstrates that the system produces actionable, granular audit findings on real production infrastructure. The 75% score reflects realistic partial compliance, validating the system’s discriminative capability in production conditions. Report generation time (0.77 s on a 10-control macOS research environment) is faster than the 2–5 s target in Table 13; production full-scale audits (169 controls) are expected to fall within the 2–5 s range depending on log volume and policy document size.

Table 13. System performance (500 inference samples).

Operation	Latency
XGBoost Inference (p50)	0.32 ms
XGBoost Inference (p95)	5.40 ms
XGBoost Inference (p99)	8.78 ms
Decision Engine (per control)	5–10 ms
End-to-End Pipeline (single log)	10–15 ms
Report Generation	2–5 s

5.11. Adversarial Validation

Adversarial validation was expanded to five MITRE ATT&CK v14 scenarios (up from two in the initial submission), using a combination of real-world attack datasets and synthesised scenarios executed in an isolated virtual machine (Windows 10 21H2, network-isolated) and a real Linux honeypot environment. The compliant baseline (false-positive rate measurement) used n = 75 real SSH/PAM log entries exclusively (Elastic AWS EC2 auth.log + LogPai Linux logs), replacing the synthetic baseline that produced inflated FPR estimates in the prior submission. HTTP and Windows log types are excluded from the compliant baseline because XGBoost produces 100% FPR on these OOV types by design (vocabulary absence, not security policy violation). Results are summarised in Table 14.

Table 14. Adversarial validation—five MITRE ATT&CK scenarios (Wilson 95% CI).

MITRE ID	Scenario	Detection%	Evasion%	Wilson 95% CI
T1059.001	PowerShell Fileless Shell (SI-3, SI-4)	86.0%	14.0%	[73.8%, 93.0%]
T1190	XSS/SQLi WAF Bypass (SI-10, AC-3)	100.0%	0.0%	[92.9%, 100.0%]
T1110	SSH Brute Force (AC-7, IA-2; real data)	100.0%	0.0%	[92.9%, 100.0%]
T1078	Lateral Movement—Valid Accounts (AC-17, AU-2; real data)	100.0%	0.0%	[92.7%, 100.0%]
T1562.001	Impair Defenses: Disable Tools (SI-3)	78.0%	22.0%	[64.8%, 87.2%]
Macro	All five scenarios	92.8%	7.2%	—
Compliant baseline false-positive rate (real SSH+PAM logs, n = 75)
FPR = 0.0%			(0/75 compliant logs misclassified)

T1110 and T1078 use real Mordor APT3 + linux_auth.log data; T1059.001 and T1190 and T1562.001 use synthesised attack scenarios in isolated VM. “Detection” = system correctly flags as non-compliant or partial. FPR measured on real SSH Accepted + PAM session logs—confirmed 0% FPR because these token patterns are in-vocabulary. Source: scripts/adversarial_validation_v2.py; results in reports/adversarial_validation_v2.json. Bold: aggregate detection and evasion rates across all five scenarios; bold 0.0% FPR denotes zero false positives on real compliant logs. Italic: baseline false-positive rate measurement row.

Test 1 (T1059.001): Fileless Reverse Shell (SI-3, SI-4, SC-7 effectiveness): A custom reverse shell implemented via VirtualAlloc+CreateThread without disk writes, evading 43/44 AV engines tested. 86.0% detection (EDR behavioural alert triggered; signature AV failed).

Test 2 (T1190): GA-Generated XSS Payloads (SI-10, AC-3 effectiveness): 50 GAXSS-inspired XSS payload variants. 34/50 blocked (68%); 16/50 bypassed (32%). SI-10 marked non-compliant. A checkbox audit (“WAF installed?”) would return compliant—demonstrating the effectiveness gap our system captures.

Tests 3–5 (T1110, T1078, T1562.001): SSH brute force (100% detection, real linux_auth.log data), lateral movement via valid accounts (100% detection, Mordor APT3), and security tool disablement (78.0% detection, synthesised).

Finding F5 (RQ1): Adversarial testing across five MITRE ATT&CK scenarios reveals effectiveness gaps invisible to presence-based checkbox auditing. Macro detection rate: 92.8%; compliant baseline FPR: 0.0% on real SSH/PAM logs. The lower detection rates for T1059.001 (86.0%) and T1562.001 (78.0%) identify specific control weaknesses for remediation. This directly answers RQ1: adversarial techniques are necessary to distinguish control installation from control effectiveness.

5.12. Research Questions Summary

Table 15 consolidates the experimental evidence for each of the four research questions addressed in this study.

Table 15. Research questions and experimental evidence.

RQ	Question Focus	Evidence
RQ0	Achieve 65–90% macro F1 with ≥50% cycle reduction	XGBoost v2: 85.45% F1 (leakage-free); Hybrid: 85.1% F1 on real logs; 2–5 s live audit (varies by log volume and policy size) vs. 1000 h manual
RQ1	Adversarial techniques inform compliance	5 MITRE ATT&CK scenarios: 92.8% macro detection; T1059.001 86.0%, T1190 100.0%, T1110 100.0%, T1078 100.0%, T1562.001 78.0%; 0.0% FPR on real compliant logs
RQ2	Model comparison in resource-constrained env.	XGBoost: 137× smaller, 50× faster than BERT; GPT-4o-mini: 92.3% zero-shot (API)
RQ3	Hybrid dataset overcomes synthetic overfitting	Zero-shot: 7.98%; 5% fine-tune: 99.88% (SSH-domain fine-tune only); vocab-cov router: 94.5%
RQ4	Multi-label vs. binary classification	3.2 controls avg./event; binary loses per-control evidence
—	LLM generalization across log types	GPT-4o-mini: 92.3% (n = 628, 4 types, 37.5% real); 85.1% hybrid on 180 real logs

6. Discussion

6.1. Integration of Adversarial Security Research (RQ1)

Our methodology connects offensive security research to compliance validation through three concrete mappings:

• Adversarial Evidence Generation: GA and LLM techniques generate diverse attack scenarios for training data augmentation. The 50 GAXSS payload variants used in Test 2 could be scaled to thousands of variants covering MITRE ATT&CK sub-techniques, providing realistic SI-10 training examples absent from public datasets.
• Control Effectiveness Quantification: Evasion results provide measurable effectiveness metrics. SI-3 effectiveness against the fileless reverse shell: $E_{SI\text{-}3}=\mathrm{detected}/(\mathrm{detected}+\mathrm{bypassed})=1/5=0.20$ (EDR detected one of five execution attempts). This maps directly to our theoretical effectiveness model (Section 6.3).
• Risk-Calibrated Scoring: Knowledge that 97% evasion rates are achievable [10] informs control weight assignments. Institutions relying solely on signature-based AV for SI-3 compliance receive risk-adjusted scores reflecting demonstrated real-world ineffectiveness rather than nominal installation.

6.2. Multi-Label vs. Binary Classification (RQ4)

The pivot from binary to multi-label classification addresses a fundamental compliance modeling mismatch.

Binary (inadequate): $f: \mathrm{Log}\to \{\mathrm{compliant}, \mathrm{non}\text{-}\mathrm{compliant}\}$

Multi-Label (correct): $f: \mathrm{Log}\to \{c_j:c_j\in \mathcal{C}\}$

Example: “Failed SSH login from 203.0.113.15” maps to four controls simultaneously, AC-7 (lockout threshold exceeded?), IA-2 (authentication bypass attempted?), AU-2 (event audited?), SI-4 (monitoring detecting intrusion?), each evaluated independently. In our evaluation, the average log event triggered 3.2 compliance control evaluations, meaning binary classification discards 69% of actionable audit information per event.

Finding F6 (RQ4): Multi-label formulation is not merely a modeling convenience, it is the correct representation of compliance semantics. A single security event simultaneously provides evidence for multiple controls, and collapsing this to a binary label destroys information that is legally and operationally significant under NCSA reporting requirements.

6.3. Theoretical Contributions

6.3.1. Effectiveness-Based Compliance Model

Prior compliance frameworks treat control assessment as binary presence detection: $C_{\mathrm{presence}}\left(k\right)\in \{0,1\}$, a control either exists or it does not. This model empirically demonstrates insufficient performance: our adversarial tests show that a control can satisfy $C_{\mathrm{presence}}\left(k\right)=1$ while providing near-zero actual security benefit ($C_{\mathrm{presence}}\left(\mathrm{SI}\text{-}3\right)=1$ with fileless shell detection rate $=0.20$).

We propose replacing binary presence checking with a continuous effectiveness function:

$$C\left(k\right)=\alpha ·D_k+\beta ·V_k+\gamma ·R_k,\hspace{1em}\alpha +\beta +\gamma =1$$

(6)

where $D_k\in [0,1]$ is the detection rate (fraction of attack instances targeting control k’s scope that are detected), $V_k\in [0,1]$ is coverage (fraction of log sources monitored for control k events), $R_k\in [0,1]$ is evasion resistance ($1-{\mathrm{evasion}\text{\_}\mathrm{rate}}_k$), and $\alpha ,\beta ,\gamma$ are configurable institutional weights (default: $\alpha =0.5,\beta =0.3,\gamma =0.2$, reflecting a detection-priority weighting appropriate for resource-constrained institutions).

This formulation satisfies three formal properties: (i) Monotonicity, improving any component cannot decrease $C\left(k\right)$; (ii) Boundedness, $C\left(k\right)\in [0,1]$ enabling consistent cross-control and cross-institution comparison; (iii) Decomposability, each component is independently measurable and improvable, enabling targeted remediation rather than pass/fail verdicts.

Relation to Prior Compliance Models: The NIST SP 800-53 control assessment approach implicitly models compliance as $C_{\mathrm{NIST}}\left(k\right)\in \{\mathrm{satisfied},\mathrm{other} \mathrm{than} \mathrm{satisfied}\}$, a binary model. The CIS Controls framework introduces prioritization tiers but maintains categorical rather than continuous scoring. ISO 27001 [33] Annex A similarly uses not-implemented/partially implemented/implemented categories. Our model is more expressive: $C_{\mathrm{presence}}\left(k\right)=1$ is equivalent to $C\left(k\right)=\alpha ·1+\beta ·V_k+\gamma ·1$ only when $D_k=1$ and $R_k=1$, a condition our adversarial tests show is frequently violated in practice.

Experimental Grounding: For Test 1, $D_{SI\text{-}3}=0.20$, $V_{SI\text{-}3}=1.0$, $R_{SI\text{-}3}=0.014$, yielding $C\left(\mathrm{SI}\text{-}3\right)=0.5\left(0.20\right)+0.3\left(1.0\right)+0.2\left(0.014\right)=0.403$, consistent with the “partial” compliance determination and distinct from the $C=1.0$ a presence-based audit would assign. This 0.597-point gap represents the effectiveness deficit invisible to conventional auditing.

6.3.2. Multi-Label Compliance Framework

The formulation $\mathbf{y}\in {\{0,1\}}^K$ for K controls provides a transferable framework applicable to any hierarchical taxonomy (NCSA [5], ISO 27001 [33], NIST SP 800-53 [7], CIS Controls [34]). Control taxonomies differ in their K and label semantics, but the multi-output classification architecture requires no structural changes, only re-labeling of training data according to the target taxonomy’s mapping logic. This framework-agnostic property is significant for the African context, where different nations may enforce different standards (Rwanda NCSA, Kenya CMCA, Nigeria NITDA) while sharing the same infrastructure.

6.3.3. Adversarial-Informed Validation Theory

We argue that compliance validation without adversarial testing provides systematically inflated assurance, not merely incomplete assurance. This distinction matters: inflated assurance creates a false sense of security that may discourage investment in genuine defensive capability. The mechanism for inflation is precisely captured by the gap between $C_{\mathrm{presence}}\left(k\right)=1$ and $C\left(k\right)=0.403$ in our SI-3 example.

This connects to game-theoretic security models [35] where defender effectiveness must be measured against adaptive adversaries, not fixed attack signatures. A compliance framework that does not model adversary behavior cannot distinguish a control that blocks 99% of attacks from one that blocks 1%, both satisfy $C_{\mathrm{presence}}\left(k\right)=1$. Our contribution is to instantiate this theoretical gap with concrete empirical measurements within a nationally-defined regulatory framework.

6.4. Theoretical Synthesis

Findings F1–F6 collectively validate the central theoretical claim: compliance effectiveness cannot be assessed through presence-based binary checking but requires three independently measurable components formalized in Equation (6): detection rate $D_k$, coverage $V_k$, and evasion resistance $R_k$.

F1 shows that $D_k$ collapses to near-zero when measured against out-of-distribution logs, because vocabulary-based models conflate detection capability with training-distribution membership. F2 shows that $D_k$ is recoverable with minimal target-domain exposure, making $V_k$ the dominant practical constraint. F5 directly instantiates $R_k$: SI-3’s evasion resistance is $R_k=0.014$, yielding $C\left(\mathrm{SI}\text{-}3\right)=0.403$ rather than the $C=1.0$ that presence-based auditing assigns. F4 shows that $D_k$ is log-type-dependent even for LLMs. F6 shows that the correct output space is $\mathbf{y}\in {\{0,1\}}^K$. F3 shows that the resource cost of achieving high $D_k$ and $V_k$ is not a barrier in the resource-constrained African context. Together, these six findings provide empirical grounding for a compliance model that is more expressive, more honest, and more actionable than existing approaches.

6.5. Resource Efficiency for African Contexts (RQ2)

XGBoost’s 137× size reduction and CPU-only execution enable deployment that BERT makes infeasible for Rwandan SMEs. With minimum requirements of 1 GB RAM and 2 CPU cores, the system aligns with available infrastructure. The cloud deployment cost ($50/month) compares favorably to current manual audit costs ($154–$341 per-capita). The MCP+LLM path adds ∼$0.15 per 10,000 logs, negligible for typical institutional audit volumes.

6.6. Ethical Considerations

Automation Bias: Organizations may over-trust ML compliance outputs. Mitigation: SHAP explanations accompany all classification decisions; Decision Engine outputs require human sign-off for non-compliant findings.

Accountability: Incorrect compliance determinations carry institutional and legal implications. Design positions ML as advisory; final compliance certifications require authorized human reviewer signature.

Privacy: Centralized log collection creates surveillance risks. Mitigation: data minimization (only compliance-relevant features stored); raw logs processed in-memory and not persisted.

Fairness: Institutions with less structured logging infrastructure may receive lower compliance scores due to parsing limitations rather than genuine non-compliance. Mitigation: System provides logging improvement recommendations rather than penalizing format variations.

6.7. Limitations

• Control Coverage: 143 of 169 RWNCSA controls (85%) are system-auditable via automated evidence parsers; 26 require physical inspection or manual policy review.
• Synthetic Training Data: The XGBoost v2 classifier (85.45% F1) was trained on leakage-free synthetic data covering all 169 RWNCSA controls. An 86.3% data leakage rate in the original training data was identified and corrected in this revision. The 50-term TF-IDF vocabulary is SSH/syslog-centric; Windows and HTTP log tokens are absent, causing 37.8-pp out-of-vocabulary F1 degradation (Section 5.5)—motivating the hybrid routing to GPT-4o-mini for OOV log types.
• LLM Evaluation Scope: GPT-4o-mini achieves 92.3% macro accuracy on n = 628 real enterprise logs (Wilson 95% CI: [89.9%, 94.3%]). The macOS evaluation remains on synthetic data (no public macOS enterprise audit log datasets identified). The HTTP evaluation (n = 28 real Squid proxy logs) has wide CI ([68.5%, 94.3%]) due to sample size. Validation on production Rwandan institution logs ($n\ge 500$ per type) remains for deployment pilots.
• Real-World Institution Validation: Experiments use public datasets (linux_auth.log honeypot, Mordor APT3, SecRepo Squid proxy, Elastic Apache logs); validation on production Rwandan institution infrastructure logs remains for ongoing pilot deployment.
• Adversarial Coverage: Validation covers 5 MITRE ATT&CK scenarios (T1059.001, T1190, T1110, T1078, T1562.001). Comprehensive coverage of all relevant techniques requires a continuous adversarial red-teaming pipeline.
• LLM Cost at Scale: Processing >10,000 logs/hour with the LLM path at current API costs is impractical without the rule-based fast path. On-premise LLM deployment (Llama, Mistral) is needed for high-volume deployments; the vocabulary-coverage gating router (Section 5.4) limits LLM calls to ≈40% of log volume.

7. Future Work

Short Term:

• Evaluate LLM analysis on heterogeneous log types: Windows Event logs, API access logs, multi-line application logs.
• Conduct pilot deployments at 2–3 Rwandan institutions to validate on production logs.
• Optimize LLM prompt engineering for partial compliance edge cases.
• Add per-control F1 breakdown to classify model performance by control family.

Medium Term:

• Complete remaining 26 NCSA controls requiring physical/manual assessment.
• Cross-framework mapping: NCSA ↔ ISO 27001 ↔ NIST SP 800-53 ↔ CIS Controls.
• Fine-tune open-source LLMs (Llama-3.1-8B, Mistral-7B-Instruct) for on-premise deployment eliminating API dependency.
• Integrate open-source EDR (Wazuh, OSSEC) for cost-effective SI-3/SI-4 validation.
• Evaluate cross-regulatory transfer: Kenya CMCA, Nigeria NITDA, South Africa POPIA.

Long Term:

• Pan-African compliance platform supporting Malabo Convention.
• Federated learning for cross-institution model improvement preserving data privacy.
• Continuous adversarial red-teaming pipeline integrated with compliance validation cycles.
• Blockchain-based audit trail for evidence integrity and non-repudiation.

8. Conclusions

This research presents an AI-augmented compliance auditor evaluated on Rwanda’s NCSA Minimum Cybersecurity Standards as a representative national framework, integrating adversarial security research with multi-label ML–LLM classification for automated compliance validation.

Key results: (1) 169 RWNCSA controls across 14 families, with 143 system-auditable via 60 specialized evidence parsers; (2) XGBoost v2 achieves 85.45% macro-F1 on leakage-free synthetic data (corrected from 99.99%—an 86.3% data-leakage rate was identified and fixed for this revision); (3) a cross-dataset generalization analysis on 180 real logs confirms 87.6% F1 for in-vocabulary SSH logs but a 37.8-pp out-of-vocabulary gap for Windows/HTTP, quantifying the operational boundary of the vocabulary-based classifier; (4) GPT-4o-mini achieves 92.3% macro accuracy across 4 log types (n = 628, 37.5% real enterprise data, Wilson CI: [89.9%, 94.3%])—SSH (89.5%), macOS (96.5%), HTTP (85.7% real Squid data), Windows (97.5% enriched); (5) the combined hybrid system on 180 real logs achieves 85.1% macro-F1 with 6.4% FPR, compared to XGBoost alone at 62.4% F1 (54.5% FPR); (6) adversarial validation across five MITRE ATT&CK scenarios (T1059.001, T1190, T1110, T1078, T1562.001) achieves 92.8% macro detection with 0.0% FPR on real SSH/PAM compliant logs (n = 75); (7) vocabulary-coverage gating (Phase II router, $\mathit{\theta }$ = 0.20) achieves 94.5% accuracy at $0.149/10K logs; (8) total resource footprint: 2.0 CPU cores, 2.66 GB RAM, $50/month cloud hosting (Apple M1 Pro baseline; storage and maintenance excluded).

The effectiveness-based compliance model $C\left(k\right)=\alpha D_k+\beta V_k+\gamma R_k$ formalizes control quality as a continuous, measurable function of detection capability, log coverage, and adversarial resistance, replacing binary presence checking with actionable posture scoring. For Rwandan institutions, the system suggests potential efficiency gains in audit cycle time—based on 2–5 s automated report generation (varying by log volume and policy document size) versus estimated manual review—along with continuous monitoring capability and cost-accessible deployment at ≤$50/month. Beyond Rwanda, this demonstrates a potentially replicable approach for automated compliance auditing in African nations pursuing cybersecurity maturity under the Malabo Convention; full continental portability requires validation on additional regulatory frameworks.