GTP-AEGIS: A Selective Heterogeneous Ensemble for GTP Intrusion Detection Under Data Scarcity

Abstract

Mobile networks have become targets of sophisticated cyber attacks. Critical vulnerabilities persist in the General Packet Radio Service Tunneling Protocol (GTP). Signature-based Intrusion Detection Systems (IDS) are inadequate against zero day exploits and novel attack patterns, necessitating more adaptive approaches. We propose GTP-AEGIS (Adaptive Ensemble with Gated Input Selection), a hybrid IDS that integrates signature-based detection with a CatBoost gradient boosting classifier via a Selective Heterogeneous Ensemble (SHE) framework. An Input-Dependent Confidence Gate (IDCG) applies a per-sample priority rule over CatBoost, a NearestCentroid Rule Engine (NCRE), and a signature pathway. A real Suricata engine detects attacks with 100% precision but only 69.2% binary recall when run standalone; within the ensemble, the signature role is played by an idealized Signature-Detection Surrogate (SDS), so the reported ensemble gains are upper bounds. On the evaluated GTP-U dataset, GTP-AEGIS reaches accuracy above 90% with 10% of the training data and raises recall for the rare invalid-TEID class from 48.9% to 64.4%; this improvement comes from the signature pathway rather than the NCRE, and the aggregate accuracy gain is not statistically significant after correction for multiple comparisons. All accuracies are obtained under a packet-level split, which a group-aware comparison shows to be optimistic by approximately 19 percentage points. The model flags 81 to 100% of packets from unseen attack families as non-normal, although this does not constitute unknown-class recognition. We report the limits of signature-only detection and of packet-level evaluation alongside the gains.

1. Introduction

Mobile networks are an important part of modern communication. In recent years, they have become targets of cyber attacks [1]. These networks attract malicious actors to exploit vulnerabilities for financial gain, espionage, and large-scale data breaches. One of the most significant attack surfaces is the General Packet Radio Service Tunneling Protocol (GTP) [2,3]. Vulnerabilities in GTP can expose sensitive subscriber information such as the International Mobile Subscriber Identity (IMSI) and critical core network elements including the Home Location Register (HLR) and Visitor Location Register (VLR). In April 2025, attackers compromised SK Telecom’s core network, including the Home Subscriber Server, exposing subscriber data for roughly 25 million customers [4]. This is a direct example of a mobile-core element being compromised, and such elements are exactly the assets that GTP-layer weaknesses can expose, for instance through IMSI and location disclosure. Earlier large-scale operator breaches, such as the 2023 T-Mobile incident affecting about 76 million subscribers [5], illustrate the sensitivity of the subscriber data at stake, although that incident was not GTP-specific.

Intrusion Detection Systems (IDS) have become a key defense layer against cyber threats [6]. When combined with firewalls and Security Information and Event Management (SIEM) systems, IDS contributes to a broader layered security architecture [7]. In general, there are two types of IDS: based on a signature and based on an anomaly. Signature-based systems detect attacks by matching traffic to predefined signatures. They are quick and accurate for known threats [8,9,10]. But because they depend on predefined signatures, they cannot detect zero day or novel attacks. Anomaly-based IDSs detect attacks by identifying deviations from normal behavior, which makes it possible to identify attacks that have never been seen before [11,12,13]. However, this flexibility can sometimes lead to more false positives, especially in mobile networks where traffic patterns and settings change quickly. This is why using one IDS method to protect modern mobile networks is often not enough. This shows that we need a hybrid detection system that is more flexible and uses different methods to give better protection.

To enhance the security of mobile communication networks, researchers have applied Artificial Intelligence (AI)-based solutions [14,15]. They leverage deep neural networks, autoencoders, and reinforcement learning to detect anomalous traffic [14]. One study proposed AI-based IDSs as core security components of 5G networks, capable of analyzing both IP and non-IP IoT traffic via transformer encoder models [16]. Multi-tier hybrid architectures that combine signature-based and anomaly-based detection have also been shown to improve accuracy alongside reduced false positive rates [17]. Despite these advances, important gaps remain. Most IDS research focuses on IP layer attacks, while protocol-specific vulnerabilities in GTP, both GTP-C and GTP-U, receive comparatively less attention. Widely used public datasets such as 5G-NIDD [18], LTE dataset [19], network operator dataset [20], and CICIDS 2017/2018 [21] lack representative GTP traffic patterns. Crucially, most existing works assume abundant labeled training data, whereas real deployments face severe data scarcity: novel GTP attack captures are expensive to collect, legally restricted, and cannot be exhaustively pre-cataloged. These limitations constrain the development of effective GTP-specific detection mechanisms.

To address these challenges, we proposed GTP-AEGIS (Adaptive Ensemble with Gated Input Selection), a hybrid IDS tailored to securing the GTP protocol under realistic data scarcity conditions. GTP-AEGIS employs the Selective Heterogeneous Ensemble (SHE) strategy to integrate two components: CatBoost gradient boosting, enhanced by a NearestCentroid Rule Engine (NCRE) for centroid proximity correction of rare attack classes; and a signature based pathway. We evaluate this pathway with a real Suricata engine, and we additionally model it with the Signature-Detection Surrogate (SDS), an idealized upper bound used to study behavior under partial signature availability. The surrogate, and not the real engine, occupies the signature role in every ensemble result reported in this paper. An Input-Dependent Confidence Gate (IDCG) unifies these two components by applying a per-sample priority rule at inference time. The main contributions of this paper are as follows:

• An application of selective ensembling to GTP intrusion detection under data scarcity (SHE-CB): We apply three established ideas to the specific problem of GTP-U intrusion detection when labeled attack data and attack signatures are both scarce. These are gradient boosting on tabular protocol features, signature-based detection, and confidence-based routing in the mixture-of-experts and selective-prediction traditions. We do not claim methodological novelty for any of the three components or for their combination. The contribution is the application to a protocol that existing IDS research has largely neglected, together with an empirical characterization, including negative results, of what the combination does and does not deliver in this setting. SHE-CB improves over standalone CatBoost by up to +1.77 percentage points in accuracy and raises invalid-TEID recall from 48.9% to 64.4%. A component ablation localizes the entire effect in the signature pathway: CatBoost with the NCRE is identical to CatBoost alone at every evaluated level and for every class, so the NCRE contributes nothing at these data scales. After Holm–Bonferroni correction across the tested data levels, the aggregate accuracy gain is not statistically significant; the rare-class recall improvement is the more consistent effect. CatBoost alone at 10% training data already attains accuracy close to that at 20%.
• Component analysis and a real signature-based baseline: We provide a four-way component ablation (CatBoost, CatBoost+NCRE, CatBoost+SDS, and full SHE-CB) that isolates the source of the ensemble gain, and we evaluate a real Suricata engine on the captured GTP-U traffic. The real Suricata baseline detects attacks with high precision and no false positives on normal traffic but with limited recall, which quantifies the limits of signature-only detection and motivates the hybrid design. We report the outcome of the ablation as a negative result: the NCRE changes no test prediction at any evaluated data scale, so it is retained in the architecture and in the released code for reproducibility, but no rare-class benefit is claimed on its behalf.
• GTP Attack Dataset and Simulation Environment: We present a publicly available GTP-U traffic dataset generated in a simulated 5G core network using Open5GS and UERANSIM, covering six traffic classes across five GTP attack types with 18 packet-level features. To the best of our knowledge, this is the first publicly available GTP dataset with multi-class packet level labeling, enabling reproducible evaluation of GTP intrusion detection under data scarcity conditions. We further use a broader set of nine attack families to evaluate the behavior of the trained model on attacks absent from training, through leave-one-attack-out and closed-set flagging tests. These tests measure the rate at which unseen attacks receive a non-normal label; they do not establish unknown-class recognition, which would require a rejection mechanism that the proposed system does not contain.

The rest of this paper is organized as follows. Section 1 is the introduction. Section 2 elaborates related works. Section 3 describes the GTP communication cyber threat model. Section 4 describes the methodology for the proposed hybrid IDS. Section 5 presents the experimental results and analysis. Finally, Section 6 presents the conclusions and future works.

2. Related Works

IDS for mobile and 5G networks have attracted considerable research attention given the increasing complexity and attack surface of modern cellular infrastructures. Other studies primarily concentrated on traditional machine learning techniques, including K-Nearest Neighbor (KNN), Decision Trees (DT), and ensemble methods, utilized for flow-level traffic characteristics such as packet size, header flags, and source–destination data. Kim et al. showed that feature selection methods can cut down on the amount of computing power needed while still keeping the accuracy of detection for IoT-based DDoS attacks in 5G core networks. This allows the system to operate nearly in real time [22]. Radoglou et al. utilized DT and XGBoost to identify volumetric attacks from flow statistics, resulting in enhanced performance; however, their analysis was limited to IP layer traffic and did not consider protocol-specific vulnerabilities [23].

Deep learning and representation learning have since been applied to improve generalization and detection resilience. Transformer-based models such as BERT and RoBERTa are well suited to network traffic analysis given their ability to capture long range sequential dependencies. Sheikhi et al. proposed a self-supervised framework using contrastive learning with augmented traffic pairs to distinguish normal from anomalous patterns [24]. While effective, the approach relied on payload and flow-level features and lacked any signature-based component.

Federated Learning (FL) has been proposed to mitigate privacy and scalability limitations in centralized Intrusion Detection System (IDS) implementations. Sheikhi and Kostakos created an unsupervised FL framework that used autoencoders to find GTP attacks in 5G core networks [25]. This work specifically focuses on GTP traffic but exclusively utilizes anomaly-based detection, neglecting signature-based mechanisms that are effective against established attack patterns. Table 1 shows the results of the survey by detection method, feature set, and GTP protocol support. The table shows that most prior work does not cover GTP and relies on a single detection paradigm, which reduces effectiveness against both known and novel attacks.

Table 1. Comparison with related work on IDS for mobile networks, extended with dataset, coverage, evaluation-split, and unseen-attack dimensions.

Ref	Year	Method	GTP-C/U	5G SA	Label Level	Public Data	Unseen-Attack Eval.
[22]	2022	KNN, DT	No	No	Flow	No	No
[23]	2023	DT, XGBoost	No	No	Flow	No	No
[24]	2024	BERT, RoBERTa	GTP-U	No	Flow	No	No
[25]	2023	Autoencoder	GTP-U	Partial	Flow	No	No
This work	2026	SHE-CB: CatBoost + Suricata + NCRE via IDCG	GTP-U (GTP-C in open-set)	Yes	Packet	Yes	Yes (LOAO + open-set)

KNN: K-Nearest Neighbor; DT: Decision Tree; BERT: Bidirectional Encoder Representations from Transformers; RoBERTa: Robustly Optimized BERT; SHE-CB: Selective Heterogeneous Ensemble with CatBoost; NCRE: NearestCentroid Rule Engine; IDCG: Input-Dependent Confidence Gate; GTP: General Packet Radio Service Tunneling Protocol; GTP-C: GTP control plane; GTP-U: GTP user plane; SA: Standalone; LOAO: leave-one-attack-out.

Feature engineering remains very important for IDS performance. It has been shown that combining traffic statistics, temporal characteristics, and protocol-specific attributes is necessary for accurate detection [26]. Weller-Fahy et al. provided a comprehensive taxonomy of distance and similarity measures for Network Intrusion Anomaly Detection (NIAD), highlighting their efficacy for clustering and classification [27]. These contributions enhance classification performance overall; however, they are not tailored for GTP-specific traffic characteristics or mobile core network semantics.

Among gradient boosting frameworks, CatBoost [28] is notable for its ordered boosting technique that prevents target leakage in categorical features and its symmetric (oblivious) decision trees structure that enables efficient and regularized inference. Ring et al. [29] showed that gradient boosted tree ensembles consistently outperform linear models and deep learning on tabular packet level features, primarily because their axis aligned split inductive bias efficiently exploits the discrete and sparse nature of network protocol fields such as GTP TEIDs, message types, and payload entropy values.

Prior studies have shown that ensembles methods are usually more effective when the base learners are different from one another rather than built on the same learning principle [30]. Dietterich [31] explained this in terms of error diversity: models that make different mistakes can compensate for one another when combined properly. This idea is developed further in Mixture of Experts models [32], where a gating mechanism decides which expert should handle a given input. Instead of relying on one fixed combination rule for all samples, the model can adapt its decision to the characteristics of each case. A related idea appears in selective prediction [33], where a model defers a decision when its confidence is too low and passes the sample to another decision mechanism.

GTP-AEGIS addresses the gaps identified above: it is a Selective Heterogeneous Ensemble IDS designed specifically for GTP traffic in mobile networks under data scarcity. By combining CatBoost gradient boosting with a NearestCentroid Rule Engine and Suricata-based signature detection via an Input-Dependent Confidence Gate, GTP-AEGIS moves beyond the fixed weight fusion of prior work and instead routes each sample to the most reliable expert at inference time, enabling precise corrections for rare attack classes without degrading performance on well-characterized samples.

3. GTP Communication Cyber Threat Model

Mobile telecommunication networks are often called walled gardens, under the assumption that core network elements are isolated from direct internet access and therefore inherently secure [2]. This idea was true to some extent in the first generations of cellular networks, but it is becoming less and less true in modern LTE and 5G networks. Modern mobile core networks have many interfaces that connect to other networks, such as roaming partners, inter operator signaling networks, and services that are available on the Internet. These exposure points greatly increase the attack surface and protocol level weaknesses, especially in the GTP, are a major security issue.

Figure 1 presents the architectural differences between 4G LTE and 5G Standalone (SA) core networks. In LTE architectures, GTP-C is responsible for control plane signaling between the Mobility Management Entity (MME), Serving Gateway (SGW), and Packet Data Network Gateway (PGW), while GTP-U transports user plane traffic. Although 5G SA introduces a service based architecture that replaces the MME with the Access and Mobility Management Function (AMF) and Session Management Function (SMF), GTP-U remains the user-plane transport protocol. It is carried over the N3 interface between the gNB and the User Plane Function (UPF) and over the N9 interface between UPFs. The N4 interface between the SMF and the UPF carries PFCP session-management signaling rather than GTP-U, and is therefore outside the scope of the GTP-U detection considered in this work. As a result, despite architectural evolution, GTP still plays an important part in managing sessions, supporting mobility, and sending data between mobile generations.

Figure 1. LTE and 5G SA Core Network Architecture.

From a threat modeling perspective, GTP is a high-value target: it makes implicit trust assumptions, lacks strong authentication, and is deployed across inter domain interfaces. GTP traffic traverses several critical trust boundaries: (i) the S5/S8 interface between SGW and PGW in LTE, (ii) the N3 interface between the gNB and the UPF and the N9 interface between UPFs in 5G SA, and (iii) roaming interconnection points such as S8 in LTE and N9 in 5G. The N4 interface between the SMF and the UPF is also security-relevant, but it carries PFCP rather than GTP and is not considered here. These interfaces connect to external operator networks via the GRX/IPX, making them potential entry points for attackers with access to signaling infrastructure or compromised roaming partners.

Threat modeling and attack vectors in mobile systems have been studied extensively [34,35] and security assessments from GSMA FS.20 describe vulnerabilities related to inter-operator signaling and GTP-based communication [36]. These studies show that GTP does not provide built in protections such as message integrity, mutual authentication, or encryption. Instead, it depends on perimeter defenses and trust assumptions between connected parties. As a result, attackers who are able to inject, replay or alter GTP messages may exploit these weaknesses to disrupt availability, expose confidential information or manipulate billing processes.

Based on existing literature and operational threat reports [36,37], the most common GTP-related attack methods can be summarized as follows:

• Denial of Service: In this scenario, an attacker floods the PGW or UPF with a high volume of Create Session Request messages sent through the GRX/IPX network. The request is similar to a legitimate session setup attempt, which causes the core network to allocate IP addresses and maintain session state until resources are exhausted. Without validation or rate limiting, these repeated requests can exhaust the DHCP address pool or overload the session tables. When those resources run out, real subscribers cannot start data sessions anymore, which causes major service problems and outages. We include this as control-plane (GTP-C) context that motivates the broader problem; the controlled experiments in this paper target the GTP-U user plane (UDP port 2152), where the corresponding data-plane manifestation is a GTP-U TPDU flood. Control-plane (GTP-C) attack families are evaluated in the open-set setting of Section 5.
• Subscriber Location Tracking: GTP messages carry User Location Information (ULI), particularly within Mobile Station Info Change Notification procedures exchanged between the Visited Public Land Mobile Network (VPLMN) and the Home Public Land Mobile Network (HPLMN). An attacker with access to GTP-U traffic can extract these messages to infer subscriber movements and locations. This creates a significant privacy concern, particularly for roaming users, because it can be abused to monitor a subscriber’s whereabouts over time.
• Billing Fraud and Identity Impersonation: Billing fraud can arise from the limited identity verification found in GTP procedures. By using a forged or stolen IMSI in messages such as Create Session Request or Create PDP Context Request, an attacker may be able to open a data session under another subscriber’s identity. In this situation, the resulting network usage is charged to the victim, while the attacker gains access to services without paying for them.

To connect the threat model to the experiments, Table 2 maps each modeled threat to the corresponding evaluated dataset class and to the Suricata rule that targets it.

Table 2. Mapping from modeled GTP threats to the five evaluated dataset attack classes.

Threat	Dataset Class	Plane/Signature
Denial of Service (volumetric)	attack_flood	GTP-U; rate threshold
Malformed/abusive signaling	attack_malformed	GTP-U; bad version/non-G-PDU type
Fragmentation-based evasion	attack_fragmented	IP fragmentation
Forged tunnel identifiers	attack_invalid_teid	GTP-U; TEID = 0
Source-address forgery	attack_spoofing	GTP-U; inner-IP (no signature)
Subscriber location tracking; billing fraud	control-plane (open-set)	GTP-C; evaluated as unseen

Production security tools still rely on rule-based and signature-based detection. While effective for known attacks, this approach struggles against zero day attacks, subtle GTP misuse, and malicious signaling crafted to resemble legitimate traffic. GTP-aware security systems must therefore move beyond static signatures to incorporate protocol semantics, behavioral analysis, and cross-plane correlation. This threat landscape motivates the adaptive detection approach proposed in this work.

4. Hybrid Intrusion Detection for Mobile Communication Networks

This section discusses the approach of the proposed hybrid intrusion detection system, GTP-AEGIS. The SHE approach comprises two elements: CatBoost gradient boosting (augmented by the NCRE for centroid proximity adjustment of infrequent attack classes) and a signature-based pathway, evaluated with a real Suricata engine and modeled under data scarcity by the SDS. The IDCG integrates these two elements by implementing per-sample routing during inference. The subsequent subsections address the simulation model, IDS architecture, feature extraction, data preparation, and each element of the SHE-CB framework.

4.1. Simulation Model

The network traffic is simulated using OPEN5GS v2.7.6 [38] and UERANSIM v3.2.7 [39], as illustrated in Figure 2. The experimental environment consists of two virtual machines (VMs). VM1 runs UERANSIM, which emulates both the User Equipment (UE) and the gNodeB (gNB), enabling the generation of realistic user plane and control plane traffic toward the 5G core network. VM2 runs the Open5GS core network components, including the AMF, SMF, and UPF, along with MongoDB for storing subscriber profiles and session information.

Figure 2. Simulation using OPEN5GS and UERANSIM.

When the UE attaches to the gNB, signaling procedures are initiated over the control plane between the UE and the AMF. The SMF coordinates session establishment, and GTP-U carries user plane data traffic through the UPF.

During the experiments, Wireshark captured all traffic at the monitored interfaces and stored it in PCAP format. For normal traffic, several attack scenarios were generated to ensure the dataset covers the full range of expected GTP behavior. These scenarios were selected based on known weaknesses in GTP and were designed to reflect threat patterns that may also appear in operational networks.

The dataset includes both normal and malicious GTP traffic, making it suitable for the development and evaluation of intrusion detection models aimed at securing the mobile core network. In addition, the source code and configuration files of the proposed GTP-AEGIS framework have been made publicly available [40].

4.2. Intrusion Detection System Architecture

The proposed IDS employs a hybrid two-component architecture that integrates gradient boosting-based detection (CatBoost, with centroid proximity correction via NCRE) and signature-based detection (Suricata) at the sample level through an Input Dependent Confidence Gate. Figure 3 illustrates the overall SHE-CB framework. The detection process begins with traffic capture at the monitored network interface, from which 18 packet-level features are extracted to characterize each GTP packet. These features serve as input to the SHE-CB detection module.

Figure 3. SHE-CB System Architecture: Two-component selective ensemble (CatBoost and Suricata) unified by an Input-Dependent Confidence Gate.

The SHE-CB module performs multi-class detection across six traffic classes (normal plus five GTP attack types). The two ensemble components operate in parallel: CatBoost delivers ML-based detection with the NCRE providing targeted centroid proximity correction for rare classes, while the SDS delivers rule-based signal for known attack PCAPs. The IDCG applies a priority decision rule on a per-sample basis to select the more reliable component and produce the final classification. This confidence-aware approach enables detection of rare and previously unseen GTP attack variants while preserving CatBoost’s accurate predictions on well-characterized samples.

The signature-based component is evaluated in two stages, and the two must not be conflated. First, we run a real Suricata 7.0.3 engine in offline mode on the raw PCAP captures with a purpose-built GTP rule set and report the measured per-class detection performance (Section 5.4). This is a standalone baseline: the real engine is never placed inside the Input-Dependent Confidence Gate. Second, to study behavior under data scarcity, where only a subset of attack signatures would realistically be available, the signature pathway inside the gate is played by the Signature-Detection Surrogate (SDS), a modeling device that projects an idealized signature contribution onto each training-data percentage through a Bernoulli signature-availability model. Every SHE-CB result reported in this paper is obtained with the SDS inside the gate, with no execution of the Suricata engine at inference time. The word “Suricata” is used in the remainder of this manuscript exclusively for that real engine. Consequently the system evaluated here is a hybrid of a gradient-boosting classifier and an idealized signature pathway; an end-to-end integration of a rule engine into the gate has not been evaluated and remains the principal item of future work.

4.3. Feature Extraction

Feature extraction is used to convert captured traffic into numerical feature vectors for machine learning algorithms. This transformation captures behavioral characteristics of traffic patterns. We use 18 features that are summarized in Table 3.

Table 3. Extracted Features.

Category	Features
Packet Parameters	(1) packet length, (2) IP total length, (3) time to live (TTL), (4) IP protocol number, (5) IP flags, (6) fragmentation offset, (7) fragmentation indicator
Transport Layer	(8) UDP source port, (9) UDP destination port, (10) UDP length
GTP Protocol	(11) Tunnel Endpoint Identifier (TEID), (12) GTP message type, (13) sequence number
Payload	(14) payload length, (15) Shannon entropy of payload bytes, (16) zero-byte count
Statistical	(17) GTP presence indicator, (18) inter arrival time

TTL: Time to Live; UDP: User Datagram Protocol; GTP: General Packet Radio Service Tunneling Protocol; TEID: Tunnel Endpoint Identifier.

We define the less obvious features and their handling explicitly, for reproducibility. The payload entropy (feature 15) is the base-2 Shannon entropy $-{\sum }_{b}p\left(b\right){log}_{2}p\left(b\right)$ over the byte-value histogram $p\left(b\right)$ of the GTP payload (range 0 to 8 bits per byte); it is high for encrypted or random payloads and low for structured or zero-filled ones. The inter-arrival time (feature 18) is the capture-timestamp difference to the previous packet of the same 5-tuple flow on the monitored interface. The TEID (feature 11) and GTP message type (feature 12) are present only for GTP packets; for non-GTP packets, they take a sentinel value and the GTP-presence indicator (feature 17) is set to 0, so non-GTP packets seen on the monitored interface are retained rather than filtered out. CatBoost consumes the identifier-like fields (TEID, message type) categorically; their treatment in the distance-based NCRE is discussed in Section 4 below.

4.4. Data Preprocessing

After feature extraction, we apply RobustScaler normalization, which transforms each feature using its median and interquartile range (IQR). For a feature X, the scaled value is computed as:

$$X_{\mathrm{scaled}}[i,j]={\displaystyle \frac{x_{ij}-{\mathrm{median}}_i}{{\mathrm{IQR}}_i}}$$

(1)

where ${\mathrm{median}}_i$ is the median of feature i across all training samples, and ${\mathrm{IQR}}_i=Q_3-Q_1$ is the interquartile range. RobustScaler is preferred over Z-score standardization because GTP features such as TEID values and inter arrival times often contain extreme outliers introduced by flood and spoofing attacks; using median and IQR ensures that outliers in the training data do not distort the normalization. These scaling parameters are computed solely on the training subsample and applied consistently to all test samples. Scaling is applied to the components that are sensitive to feature scale, namely the distance-based NCRE and the linear and neural baselines. Tree-based models, including CatBoost, are invariant to monotone per-feature transformations and do not require scaling; we verified that toggling RobustScaler changes CatBoost accuracy by $0.00$ percentage points at both 10% and 20% training data, so the scaler does not drive the tree-model results.

4.5. CatBoost Gradient Boosting Classifier

CatBoost [28] is a gradient boosting framework that introduces two key innovations over earlier methods such as XGBoost and LightGBM.

4.5.1. Ordered Boosting

CatBoost employs Ordered Boosting (Ordered TS) to prevent target leakage in training statistics. For each training sample, statistics are computed only from preceding samples in a random permutation, eliminating the conditional shift that occurs when a model computes target statistics using the same sample that will later be used for training. This anti-leakage mechanism is particularly important for network traffic features such as flow statistics that are computed over multiple packets.

4.5.2. Symmetric Decision Trees

CatBoost builds symmetric (oblivious) decision trees where each node at the same tree depth uses the same split condition. This produces balanced trees with exactly $2^d$ leaves for depth d, which provides natural regularization against overfitting and enables efficient vectorized evaluation during inference. For a tree of depth $d=6$, this yields 64 leaf nodes, balancing model capacity and generalization for the 1% to 20% training subsets used in our data scarcity experiments.

The CatBoost classifier is configured with $T=300$ iterations, depth $d=6$, and learning rate $\eta =0.05$. These values are chosen for the data-scarcity regime: depth-6 symmetric trees give $2^6=64$ leaves, enough to separate six classes while limiting overfitting on the 1% to 20% subsets, and a low learning rate with a moderate iteration count keeps variance down when data are scarce. A 27-configuration sensitivity grid (depth $\in \{4,6,8\}$, learning rate $\in \{0.03,0.05,0.1\}$, iterations $\in \{200,300,500\}$) at 20% data yields accuracies spanning only $89.53$ to $90.14\%$ (a $0.61$ pp spread), with the chosen configuration near the top, so the reported results are not an artifact of fragile tuning. The same tuning budget was applied to the other gradient-boosting baselines. The multi-class softmax cross entropy loss function is minimized using symmetric Newton boosting. For each test sample x, CatBoost produces a calibrated probability vector: $p_{\mathrm{CB}}\left(x\right)=\mathrm{softmax}\left(F_T\left(x\right)\right)$, where $F_T\left(x\right)$ is the sum of T weak learners. CatBoost’s certainty for sample x is:

$$\mathrm{cb}\text{\_}\mathrm{conf}\left(x\right)=\underset{c}{\max }{p}_{\mathrm{CB}}\left(x\right)\left[c\right]$$

(2)

4.6. NearestCentroid Rule Engine (NCRE)

The NCRE is a lightweight expert that computes per-class centroid proximity probabilities from the training subsample. The key insight is that samples lying close to an attack class centroid in feature space are likely members of that class, even when CatBoost’s probabilistic boundary is uncertain. Because Euclidean distance is not meaningful for identifier-like fields, the NCRE computes distances over the continuous feature subset only (packet and payload length, TTL, UDP length, payload entropy, zero-byte count, inter-arrival time); the TEID is represented by a categorical indicator (TEID = 0) rather than by its raw integer value, and the GTP message type is excluded from the L2 distance. An ablation confirms that excluding the raw identifier fields from the centroid space leaves the rare-class recall unchanged while removing the artificial distance relationships these fields would otherwise introduce.

4.6.1. Centroid Computation

For each class $c\in \{0,1,\dots ,C-1\}$, the class centroid is computed as:

$${\mathit{\mu }}_c={\displaystyle \frac{1}{|{\mathcal{S}}_c|}}\sum _{i\in {\mathcal{S}}_c}{\mathbf{x}}_i$$

(3)

where ${\mathcal{S}}_c=\{i:y_i=c\}$ is the set of training sample indices belonging to class c.

4.6.2. Softmax Temperature Calibration

The softmax temperature $\sigma$ is set to the median intra class L2 distance, calibrating the sharpness of the proximity distribution:

$$\sigma =\mathrm{median}\left(\bigcup _c\left\{\parallel {\mathbf{x}}_i-{\mathit{\mu }}_c{\parallel }_2:i\in {\mathcal{S}}_c\right\}\right)$$

(4)

4.6.3. NCRE Soft Probabilities

For each test sample $\mathbf{x}$, the NCRE computes class proximity logits and converts them to soft probabilities via temperature-scaled softmax:

$${\mathcal{\ell }}_c\left(\mathbf{x}\right)=-{\displaystyle \frac{\parallel \mathbf{x}-{\mathit{\mu }}_c{\parallel }_2}{\sigma }}$$

(5)

$$p_{\mathrm{NC}}\left(\mathbf{x}\right)\left[c\right]={\displaystyle \frac{\exp \left({\mathcal{\ell }}_c\left(\mathbf{x}\right)\right)}{{\sum }_{c^{\prime }}\exp \left({\mathcal{\ell }}_{c^{\prime }}\left(\mathbf{x}\right)\right)}}$$

(6)

The NCRE confidence for sample $\mathbf{x}$ is:

$$\mathrm{nc}\text{\_}\mathrm{conf}\left(\mathbf{x}\right)=\underset{c}{\max }{p}_{\mathrm{NC}}\left(\mathbf{x}\right)\left[c\right]$$

(7)

A high $\mathrm{nc}\text{\_}\mathrm{conf}\left(\mathbf{x}\right)$ indicates that $\mathbf{x}$ lies close to one class centroid in feature space. The design intention is that, as the training data percentage increases, centroid estimates become more accurate and a larger fraction of test samples exceeds the confidence threshold $\theta$, producing more corrections. We should note that this intended mechanism is not observed in our experiments: the ablation of Section 5.7 shows that the NCRE override condition never both fires and changes a prediction at any evaluated data level, so the growth of SHE-CB’s advantage over CatBoost is attributable to the signature pathway alone and not to centroid accuracy.

Algorithm 1 formalizes the complete NCRE computation.

Algorithm 1: NearestCentroid Rule Engine (NCRE)

4.7. Signature-Detection Surrogate (SDS)

The SDS models an idealized signature contribution under partial signature availability using an independent Bernoulli trial per attack capture. For a training data percentage $k\%$, each of the five attack captures is independently available for rule generation with probability $k/100$. We state clearly that this coupling between signature availability and labeled-data availability has no operational justification and is not offered as one. It is a modeling convenience that keeps the scarcity sweep one-dimensional, motivated by the general consideration that signatures and labels are both produced by the same scarce activity of analyzing captured attack traffic. That intuition is plausible, but it is not evidence, and we do not present it as such. Because the coupling is arbitrary, we recommend a reading of the results that does not depend on it. Since SHE-CB is identical to CatBoost whenever no signature is available, the surrogate’s contribution depends only on how many attack classes are covered, whose expectation is $5k/100$ and which ranges from $0.05$ covered classes at $k=1$ to $1.0$ covered class at $k=20$. The SHE-CB results of Section 5.3 may therefore be read as a sweep over signature coverage from zero to one class out of five, and the mapping to the training-data percentage should be regarded as a labeling of the horizontal axis rather than as a claim about how signature availability behaves in practice.

$$\mathrm{known}\left(\mathrm{pcap}\right)\sim \mathrm{Bernoulli}(k/100)$$

(8)

The expected number of known attack PCAPs at level k is $\mathbb{E}[\#\mathrm{known}]=5\times k/100$. At $k=1\%$, $\mathbb{E}=0.05$ (SDS effectively never fires); at $k=20\%$, $\mathbb{E}=1.0$ (on average one attack class is fully identified by Suricata rules).

For each test sample ${\mathbf{x}}_i$ with source PCAP $s_i$:

$$p_{\mathrm{SDS}}\left({\mathbf{x}}_i\right)\left[c\right]=\left\{\begin{array}{cc}0.92\hfill & \mathrm{if}{s}_i\in \mathrm{known}\mathrm{and}c=\mathrm{attack}\text{\_}\mathrm{class}\left(s_i\right)\hfill \\ 0.08/(C-1)\hfill & \mathrm{if}{s}_i\in \mathrm{known}\mathrm{and}c\ne \mathrm{attack}\text{\_}\mathrm{class}\left(s_i\right)\hfill \\ 0.80\hfill & \mathrm{if}{s}_i\notin \mathrm{known}\mathrm{and}c=\mathrm{normal}\hfill \\ 0.20/(C-1)\hfill & \mathrm{if}{s}_i\notin \mathrm{known}\mathrm{and}c\ne \mathrm{normal}\hfill \end{array}\right.$$

(9)

The high confidence value (0.92) when SDS fires reflects the near-deterministic precision of a matching signature for a known attack. When SDS does not fire (no known PCAP), the soft output gently biases toward normal.

What the SDS assumes, and why it is not realizable. The surrogate is oracle-informed and we state its assumptions explicitly. For a test packet, it consults the identity of the source capture and the mapping from that capture to its attack class, and when the capture is drawn as available, it assigns probability $0.92$ to the correct class. Neither operation is available to a deployed detector: a rule engine evaluates packet content against rules and has no access to capture provenance or to ground-truth labels. Two consequences follow, and both are material to how the results of this paper should be read. First, within a covered class, the surrogate cannot commit a per-packet detection error, so it has no false negatives of the kind that limit any real rule set; the measured engine of Section 5.4, by contrast, achieves $0\%$ per-class recall on three of the five attack classes (malformed, flood, and spoofing). Second, the surrogate cannot be implemented, so no deployment could reproduce the SHE-CB results reported here. The surrogate is therefore an upper bound on the contribution of any signature pathway with the same coverage, and it is used for that purpose alone. Together with the measured engine, it brackets the achievable contribution: the real signature pathway attains $76.7\%$ multi-class accuracy and $69.2\%$ binary recall standalone, while the idealized pathway is what produces the SHE-CB figures in Section 5.3. The true value of an integrated rule engine lies between these, and we do not claim to have measured it.

4.8. Input Dependent Confidence Gate (IDCG)

The IDCG is a hand-specified priority rule, not a learned combiner. It applies a fixed sequence of conditions to each test sample, using the weights ${\beta }_{\mathrm{sds}}=0.85$ and ${\beta }_{\mathrm{nc}}=0.70$ and two thresholds selected by grid search on out-of-fold probabilities. Its design intention is to apply different expert combinations to different samples, so that corrections can be made in specific regions of feature space (near cluster boundaries or matching known attack patterns) without disturbing samples on which CatBoost is already reliable. We describe below the extent to which that intention is realized in our experiments, which is considerably less than the design suggests.

4.8.1. Priority Decision Rules

For each test sample $\mathbf{x}$, the IDCG applies the priority rules formalized in Algorithm 2:

Algorithm 2: Input-Dependent Confidence Gate (IDCG)

    Priority 1 gives precedence to the signature decision when a rule matches a known attack PCAP: when SDS fires (${\beta }_{\mathrm{sds}}=0.85$), the ensemble output is dominated by the signature-based signal. Priority 2 reduces CatBoost’s false negatives for samples near attack cluster boundaries: when NCRE is highly confident ($\mathrm{nc}\text{\_}\mathrm{conf}\left(x\right)>\theta$) and CatBoost is uncertain ($\mathrm{cb}\text{\_}\mathrm{conf}\left(x\right)<\tau$), the centroid proximity expert overrides the gradient boosting decision. Priority 3 preserves CatBoost’s accurate predictions for the majority of samples, avoiding NCRE or SDS interference where CatBoost is already reliable.

4.8.2. Relationship to Simpler Fusion Rules

At the data scales evaluated in this paper, the gate is equivalent to the single rule “if a signature fires, output the signaled class; otherwise output the CatBoost prediction”. The equivalence is exact and follows from the parameters rather than from measurement. When the surrogate fires, the signaled class receives at least ${\beta }_{\mathrm{sds}}\times 0.92=0.782$, whereas any other class receives at most ${\beta }_{\mathrm{sds}}\times 0.08/(C-1)+(1-{\beta }_{\mathrm{sds}})\times 1=0.164$; the signaled class therefore attains the argument maximum for every possible CatBoost output, so Priority 1 is a hard override and the value of ${\beta }_{\mathrm{sds}}$ does not affect any decision. Priority 2, the only rule that mixes probabilities from two different models, never changes a prediction at any evaluated level (Section 5.7). Priority 3 returns CatBoost unmodified. Three consequences follow. First, we do not claim that input-dependent gating outperforms fixed-weight fusion; no result in this paper supports that claim. Second, cross-model probability calibration does not affect any reported decision, since no reported decision compares probabilities from different models on a common scale; we have accordingly not reported reliability diagrams, expected calibration error or Brier scores, and we record their absence as a limitation of the gate in its general form, in which Priority 2 would be active. Third, an empirical comparison against direct rule override would by construction return a difference of exactly zero at these scales; comparisons against probability averaging, stacking, and calibrated confidence fusion remain to be performed and are identified as future work. The value of the gate, if it has one, lies in providing a principled place for a rare-class expert to intervene; this paper does not demonstrate that such intervention occurs.

4.8.3. Threshold Calibration

The thresholds $\theta$ and $\tau$ are jointly calibrated by grid search over $\theta \in \{0.55,0.65,0.70,0.75,0.80,0.85\}$ and $\tau \in \{0.50,0.55,0.60,0.65,0.70\}$, selecting the pair that maximizes accuracy on out-of-fold (OOF) CatBoost probabilities obtained by 5-fold cross-validation within the training subsample, so that each sample’s probability is produced by a model that did not train on it. This avoids the optimistic bias of in-sample probabilities. The calibration adds one cross-validation pass over the training subsample and no change to the final fitted model.

4.9. Dataset

The GTP-AEGIS dataset comprises 18 packet level features extracted from PCAP captures using Scapy. It is built from 12 PCAP captures (7 normal traffic scenarios totaling 70,000 packets, and 5 data-plane attack captures totaling 23,734 packets), i.e., 93,734 labeled packets before balancing. Normal traffic, the dominant class in the raw captures, was downsampled while all attack packets were retained, yielding a class-rebalanced set of 27,734 packets across six classes (Table 4). The result is markedly less imbalanced than the raw capture but is still uneven across attack classes; we therefore describe it as class-rebalanced rather than balanced. Normal traffic constitutes $21.6\%$ of this set. That prevalence is an artefact of the construction and does not resemble operational traffic, in which normal packets are overwhelmingly dominant. The consequence for the metrics reported in this paper should be understood clearly. Per-class recall and the false-positive rate on normal traffic are computed within a class and are therefore invariant to the class mixture; they carry over to a different prevalence unchanged. Overall accuracy, weighted F1, and precision all depend on the mixture and do not carry over. Readers seeking an operational interpretation should therefore consult the per-class recalls reported in Section 5.5 and the normal false-positive rate, and should treat the accuracy figures throughout this paper as comparative quantities between methods evaluated on an identical artificial mixture, not as estimates of field performance or of the alert burden a deployment would face. We do not re-weight the results to an assumed operational prevalence, since no defensible estimate of that prevalence is available to us for GTP-U traffic. Each packet is labeled by its source capture, and each of the five attack classes is represented by exactly one capture. This design has a consequence that must be stated before any result is presented: class identity and capture identity are perfectly collinear in the primary dataset. Packets from one capture share the generation script, the topology, the timing conditions, and the protocol parameterization, so features such as the TEID, the sequence number, the TTL, the UDP ports, the packet size, the inter-arrival time, and the payload entropy may encode which capture a packet came from rather than which attack it represents. A model can therefore attain high accuracy by recognizing captures, and no measurement made on this dataset can distinguish that behavior from genuine attack recognition. The asymmetry in feature handling deserves note in this connection: the NCRE excludes the raw identifier fields from its distance computation, but CatBoost consumes the TEID and the message type as categorical features and thus retains access to the fields most capable of identifying a capture. The primary train/test protocol is an 80/20 packet-level split; the effect of packet-level versus PCAP-level (group-aware) splitting is analyzed in Section 5.10, and that analysis should be read as a measurement of this confounding rather than as a comparison of splitting conventions.

Table 4. GTP-AEGIS class-rebalanced dataset: per-class packet count and share (27,734 packets total).

Class	Packets	Share (%)
Normal	6000	21.6
attack_flood	6000	21.6
attack_fragmented	3000	10.8
attack_invalid_teid	734	2.6
attack_malformed	6000	21.6
attack_spoofing	6000	21.6
Total	27,734	100.0

Bold denotes the total row (the column sums over the six classes).

The attack_invalid_teid class (2.6%) is the rarest and most challenging, making it the critical stress test for data scarcity robustness.

5. Results and Discussion

5.1. Experimental Setup

Experiments were conducted in a controlled environment emulating 5G core network operations. Open5GS served as the 5G core network, while UERANSIM emulated the UE and radio access network to generate realistic GTP-U traffic. All experiments were run on an Ubuntu 24.04 platform with Intel Xeon^® E5-2680 v4 processors and 64 GB RAM. CatBoost was implemented in Python (version 3.13.12) via the CatBoost library; the signature pathway was evaluated both with a real Suricata 7.0.3 engine (Section 5.4) and, for the data-scarcity sweep, with the Signature-Detection Surrogate (SDS). To avoid any ambiguity, the real engine is evaluated only as a standalone baseline and is never placed inside the confidence gate: every SHE-CB number reported in this section is produced with the SDS surrogate occupying the signature role. Five GTP attacks were emulated: flood, fragmented, invalid TEID, malformed, and spoofing. Data scarcity conditions were evaluated across 11 training data levels (1, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20%), with 10 random seeds per level (mean ± std reported); all methods use the identical set of 10 seeds at every level. An 80/20 stratified split (seed = 42) was applied to the class-rebalanced dataset.

5.2. Standalone Machine Learning Baselines

Table 5 compares 11 machine learning models trained on 1% to 20% of the available training data, with CatBoost achieving the best standalone performance across all levels.

Table 5. ML model accuracy (%) under data scarcity (mean over 10 seeds). Packet-level split; optimistic relative to a leakage-free PCAP-level protocol (Section 5.10).

Model	1%	4%	8%	10%	20%
Logistic Regression	72.85	73.22	73.00	73.00	71.58
SVM (RBF)	75.44	73.47	73.83	73.83	72.49
MLP	73.80	74.23	74.20	74.20	73.52
1D CNN	73.57	75.07	76.23	76.23	75.72
BiLSTM	56.24	71.06	69.86	69.86	70.42
Decision Tree	87.06	88.20	88.34	88.34	88.59
Random Forest	86.91	88.07	88.14	88.14	88.56
Extra Trees	86.80	87.93	87.98	87.98	88.41
LightGBM	86.94	89.32	89.43	89.43	89.65
XGBoost	86.20	89.02	89.48	89.52	89.82
CatBoost	88.00	89.30	89.58	89.72	89.83

SVM: Support Vector Machine (RBF kernel); MLP: Multi-Layer Perceptron; CNN: Convolutional Neural Network; BiLSTM: Bidirectional Long Short Term Memory. Column headers indicate percentage of training data. Values are mean accuracy (%) over 10 seeds. Bold marks the best-performing model at each data level (CatBoost).

Among the evaluated baselines, linear models (Logistic Regression, SVM) and deep learning architectures (MLP, CNN, BiLSTM) exhibit limited performance (56 to 76%), showing clear data dependency and inadequate capacity for capturing the discrete, protocol level feature distributions of GTP traffic. Tree ensemble models (Decision Tree, Random Forest, Extra Trees, LightGBM, XGBoost, CatBoost) are markedly more resilient, achieving 86 to 90% even at 1% training data. Among gradient boosting variants, XGBoost and LightGBM perform comparably, while CatBoost consistently leads at every data level, confirming that its ordered boosting and symmetric tree architecture provides the best inductive bias for tabular GTP features. However, even CatBoost exhibits a performance plateau: from 10% to 20% training data, accuracy only improves by 0.11 pp (89.72% → 89.83%), motivating the SHE hybrid architecture.

5.3. SHE-CB Hybrid Performance

Before presenting the headline results, we restate their scope. Every accuracy in this subsection is obtained under a packet-level stratified split of a dataset that contains one capture per attack class. As quantified in Section 5.10, this protocol is optimistic relative to a leakage-free PCAP-level split, and the numbers below should be read with that qualification attached. Table 6 presents the SHE-CB performance across all evaluated data levels, alongside CatBoost standalone and SDS surrogate reference values. For reference, the SDS surrogate attains 62.00% accuracy at 10% signature availability and 65.75% at 20%, while CatBoost reaches 89.72% and 89.83% at the same levels. These surrogate values vary with the data percentage only because the surrogate’s signature-availability parameter is tied to it; they are not measurements of a rule engine, and they should not be compared with the real Suricata result of Section 5.4, which is by construction independent of the training-data percentage.

Table 6. SHE-CB overall performance vs. baselines (mean ± std over 10 seeds). Packet-level split; optimistic relative to a leakage-free PCAP-level protocol (Section 5.10). The expected number of attack classes covered by the signature surrogate is $5k/100$ at data level k, ranging from $0.05$ at 1% to $1.0$ at 20%.

% Data	SHE-CB Acc (%)	CB Acc (%)	F1-W	$\mathbf{\Delta }$ (pp)
1%	$88.00\pm 0.99$	$88.00\pm 0.99$	0.8744	$+0.00$
2%	$88.73\pm 0.66$	$88.73\pm 0.66$	0.8824	$+0.00$
4%	$89.30\pm 0.24$	$89.30\pm 0.24$	0.8902	$+0.00$
6%	$89.46\pm 0.29$	$89.46\pm 0.29$	0.8919	$+0.00$
8%	$90.02\pm 0.65$	$89.58\pm 0.21$	0.8982	$+0.44$
10%	$90.15\pm 0.62$	$89.72\pm 0.30$	0.8996	$+0.43$
12%	$90.86\pm 1.88$	$89.67\pm 0.19$	0.9067	$+1.19$
14%	$90.97\pm 1.91$	$89.74\pm 0.16$	0.9078	$+1.22$
16%	$91.03\pm 1.90$	$89.79\pm 0.24$	0.9085	$+1.24$
18%	$91.43\pm 2.21$	$89.75\pm 0.26$	0.9124	$+1.68$
20%	$\mathbf{91.60}\pm \mathbf{2.23}$	$89.83\pm 0.19$	0.9141	$+\mathbf{1.77}$

SHE-CB: Selective Heterogeneous Ensemble with CatBoost (proposed); CB: CatBoost standalone; F1-W: weighted F1 score; $\Delta$ (pp): accuracy gain of SHE-CB over CB in percentage points. Values are mean ± std over 10 random seeds. The bold row marks the 20% training-data level, at which the SHE-CB gain over CB is largest and which is the headline configuration discussed in the text.

The SHE-CB advantage activates at 8% training data (where the expected number of known Suricata PCAPs reaches $\mathbb{E}[\#\mathrm{known}]=5\times 0.08=0.4$) and grows with data availability. At 1 to 6% data, the gain is zero because the signature pathway rarely fires. From 8% onwards, the SDS signature pathway contributes signal for known attack PCAPs, and the gain reaches $+1.77$ pp at 20% training data.

Because the standard deviations of the two systems overlap, we tested the gain for statistical significance. For each data level we compared the 10 paired per-seed accuracies of SHE-CB and CatBoost with a paired t-test and a Wilcoxon signed-rank test, and computed the 95% confidence interval of the mean difference (Table 7). Uncorrected, the paired t-test gives $p<0.05$ at 18% and 20% training data. These tests are performed at multiple data levels, so a correction for multiple comparisons is required. Applying the Holm–Bonferroni procedure to the five reported p-values gives adjusted values of $0.175$ at 18% and 20% and $0.222$ at 8%, 10%, and 12%; correcting over all seven levels at which the gain is non-zero would be more conservative still. No level is significant at $\alpha =0.05$, and the Wilcoxon signed-rank test was already non-significant at $p=0.063$. We therefore do not describe the aggregate accuracy gain as statistically significant anywhere in this paper. A further consideration argues against relying on this test in either direction. The seed-to-seed variability of SHE-CB (≈$\pm 1.9$ to $2.2$ percentage points, against ≈$\pm 0.2$ for CatBoost) originates in the Bernoulli draw that determines which attack captures are covered by the surrogate, not in any resampling of traffic. The paired test across seeds therefore measures the variability of a modeling device rather than sampling variability of the underlying data, and the resulting p-values do not estimate the quantity that a significance test is intended to estimate. The consistent effect in these experiments is the rare-class recall improvement of Section 5.5, which is present at every level at which a signature is available and does not depend on this test.

Table 7. Statistical significance of the SHE-CB accuracy gain over CatBoost (paired across 10 seeds), with correction for multiple comparisons.

% Data	SHE-CB	CB	$\mathbf{\Delta }$ (pp)	95% CI (pp)	t-Test p	Holm-adj. p	Wilcoxon p
8	$90.02$	$89.58$	$+0.44$	$[-0.07,+0.95]$	$0.081$	$0.222$	$0.250$
10	$90.15$	$89.72$	$+0.43$	$[-0.07,+0.93]$	$0.081$	$0.222$	$0.250$
12	$90.86$	$89.67$	$+1.19$	$[-0.14,+2.53]$	$0.074$	$0.222$	$0.125$
18	$91.43$	$89.75$	$+1.68$	$[+0.15,+3.21]$	$0.035$	$0.175$	$0.063$
20	$91.60$	$89.83$	$+1.77$	$[+0.13,+3.41]$	$0.037$	$0.175$	$0.063$

No level is significant at $\alpha =0.05$ after Holm–Bonferroni correction across the five reported levels. Correction across all seven levels at which the gain is non-zero would be more conservative. The seed-to-seed variance of SHE-CB originates in the Bernoulli signature-availability draw of the surrogate rather than in resampling of traffic, so these tests characterize the variability of the modeling device rather than of the data. Levels 1 to 6% are omitted because the gain is exactly zero. CB: CatBoost; CI: confidence interval; pp: percentage points.

5.4. Real Suricata Evaluation

We ran a real Suricata 7.0.3 engine in offline mode on the captured PCAPs with a purpose-built GTP rule set. Two properties of this evaluation should be noted before the results. First, it is a standalone baseline: the engine is not placed inside the Input-Dependent Confidence Gate, and no SHE-CB result in this paper involves executing it. Second, its performance is independent of the training-data percentage, since a rule engine does not train. The single set of values reported here therefore applies at every data level, in contrast to the surrogate values of Table 6, which vary with the data percentage only because the surrogate’s availability parameter is tied to it by the modeling choice described in Section 4.

The engine was run with a purpose-built GTP rule set (invalid-TEID content match, malformed bad-version and non-G-PDU message-type checks, IP fragmentation, and a volumetric-flood rate threshold). The measured per-packet results are summarized in Table 8. Real Suricata is high-precision but low-recall: it flags attacks with $100\%$ precision and a $0\%$ false-positive rate on normal traffic, but its binary attack-detection recall is $69.2\%$ and its multi-class macro-F1 is only $0.235$. The flood attack is not detected because the captured rate (≤72 packets/s) is below the threshold, and inner-IP spoofing has no GTP-U header signature. Real Suricata is therefore substantially weaker than the SDS surrogate; the comparison confirms the well-known limitation of signature-only detection (coverage is bounded by the rule set) and motivates the ML hybrid, and we report it in place of the earlier SDS-based “Suricata” comparison so that the baseline is genuine rather than idealized.

Table 8. Real Suricata 7.0.3 per-packet detection on the captured GTP-U PCAPs.

Metric	Real Suricata
Binary attack-vs-normal recall	$69.2\%$
Binary attack precision	$100.0\%$
Normal false-positive rate	$0.0\%$
Multi-class accuracy	$76.7\%$
Multi-class macro-F1	$0.235$

Per-class recall: Normal $100\%$, invalid-TEID $52.3\%$, fragmented $50\%$, malformed $0\%$, flood $0\%$ (rate below threshold), spoofing $0\%$ (no signature).

The full per-packet confusion matrix is shown in Figure 4.

Figure 4. Real Suricata per-packet confusion matrix (row-normalized). Suricata is precise on normal traffic and on the TEID-zero signature but cannot discriminate attacks that share low-level structure, and does not detect flood or spoofing.

5.5. Per Class Detection Performance

Table 9 presents per class recall for SHE-CB, CatBoost, and real Suricata at 20% training data. The largest improvement is observed for attack_invalid_teid, where the signature pathway raises recall from 48.16% (CatBoost) to 64.35% (+16.19 pp). For attack_flood, SHE-CB achieves 87.99% recall (+4.62 pp over CatBoost). All three methods achieve near perfect recall for normal traffic, attack_fragmented, and attack_malformed. The macro-averaged F1 (which weights the rare class equally) improves from $0.8696$ (CatBoost) to $0.8987$ (SHE-CB) at 20% training data, and from $0.8656$ to $0.8857$ at 10%; the normal false-positive rate is 0 in all cases. A 64.35% recall on attack_invalid_teid still leaves about 35.6% of these attacks undetected, so we present this as an improvement rather than a solution.

Table 9. Per-class recall at 20% Training Data. Packet-level split; optimistic relative to a leakage-free PCAP-level protocol (Section 5.10).

Class	Real Suricata	CatBoost	SHE-CB	$\mathbf{\Delta }$ (pp)
Normal	100.00	99.98	100.00	$+0.02$
Flood	0.00	83.37	87.99	$+4.62$
Fragmented	50.00	100.00	99.97	$-0.03$
Inv. TEID	52.30	48.16	64.35	$+\mathbf{16.19}$
Malformed	0.00	99.72	99.77	$+0.05$
Spoofing	0.00	75.98	77.77	$+1.79$

SHE-CB: Selective Heterogeneous Ensemble with CatBoost; pp: percentage points; all values at 20% training data. Real Suricata values are measured per-packet (Section 5.4). Flood = attack_flood; Fragmented = attack_fragmented; Inv. TEID = attack_invalid_teid; Malformed = attack_malformed; Spoofing = attack_spoofing. Bold values in the SHE-CB column mark the proposed method; the bold invalid-TEID row label and its $\Delta$ mark the rarest class, where the recall change is largest.

The attack_invalid_teid pattern deserves particular attention. CatBoost achieves near perfect precision (99.7%) but only 48.2% recall for this class, indicating that it is overly conservative: when GTP TEID features are ambiguous, CatBoost defaults to normal classification. The signature pathway corrects this for the test packets whose source PCAP is recognized, raising SHE-CB’s recall from 48.16% to 64.35% while maintaining 100% precision on this class and improving per-class F1 from 0.6493 to 0.7608. A component ablation (Section 5.7) shows that this rare-class correction is delivered by the signature (SDS) pathway rather than by the NCRE in the current configuration.

Table 10 summarizes per class F1 scores at 10% and 20% training data.

Table 10. Per-class F1 score: SHE-CB vs. CatBoost at 10% and 20% training data. Packet-level split; optimistic relative to a leakage-free PCAP-level protocol (Section 5.10).

	10% Training Data			20% Training Data
Class	CatBoost	SHE-CB	ΔF1	CatBoost	SHE-CB	ΔF1
Normal	0.9997	1.0000	$+0.0003$	0.9999	1.0000	$+0.0001$
Flood	0.7960	0.8042	$+0.0082$	0.8023	0.8376	$+0.0353$
Fragmented	0.9975	0.9982	$+0.0007$	0.9975	0.9981	$+0.0006$
Inv. TEID	0.6280	0.7482	$+\mathbf{0.1202}$	0.6493	0.7608	$+\mathbf{0.1115}$
Malformed	0.9777	0.9789	$+0.0012$	0.9779	0.9815	$+0.0036$
Spoofing	0.7829	0.7846	$+0.0017$	0.7818	0.8142	$+0.0324$
Weighted F1	0.8949	0.8996	$+0.0047$	0.8962	0.9141	$+0.0179$

SHE-CB: Selective Heterogeneous Ensemble with CatBoost; ΔF1: per-class F1 improvement of SHE-CB over CatBoost; Weighted F1: macro weighted average. Class names follow the same abbreviations as Table 9. Bold values mark the proposed method’s results for the rare invalid-TEID class and the weighted F1 average, where the improvements over CatBoost are largest.

5.6. Summary Comparison: All Methods

Table 11 provides a comprehensive comparison of all methods at 10% and 20% training data.

Table 11. Overall accuracy (%) of all methods at 10% and 20% training data. Packet-level split; optimistic relative to a leakage-free PCAP-level protocol (Section 5.10).

Method	Type	Acc @10%	Acc @20%
SDS surrogate (idealized)	Signature surrogate	62.00	65.75
Real Suricata 7.0.3	Rule-based	76.70	76.70
Logistic Regression	Linear ML	73.00	71.58
SVM (RBF)	Kernel ML	73.83	72.49
MLP	Neural Net	74.20	73.52
1D CNN	Deep Learning	76.23	75.72
BiLSTM	Deep Learning	69.86	70.42
Decision Tree	Tree ML	88.34	88.59
Random Forest	Ensemble	88.14	88.56
Extra Trees	Ensemble	87.98	88.41
LightGBM	Gradient Boost	89.43	89.65
XGBoost	Gradient Boost	89.52	89.82
CatBoost	Gradient Boost	89.72	89.83
SHE-CB (proposed)	Selective Ensemble	90.15	91.60

SHE-CB: Selective Heterogeneous Ensemble with CatBoost (proposed); SVM: Support Vector Machine; MLP: Multi Layer Perceptron; CNN: Convolutional Neural Network; BiLSTM: Bidirectional Long Short Term Memory; Acc: overall classification accuracy (%); @10%, @20%: evaluated at 10% and 20% training data, respectively. The SDS surrogate row is a modeling device whose value changes with the data percentage only because its signature-availability parameter is tied to that percentage; the real Suricata row is a measurement and does not depend on the training-data percentage. All learned-model accuracies are obtained under a packet-level split and are optimistic relative to a leakage-free PCAP-level protocol (Section 5.10). Bold rows mark the best standalone baseline (CatBoost) and the proposed method (SHE-CB).

SHE-CB attains the highest accuracy at 20% training data among the evaluated methods, with a gain of $+1.77$ pp over the best standalone baseline (CatBoost). We describe this improvement as incremental. It is not statistically significant after correction for multiple comparisons (Table 7), it is obtained under a packet-level split that Section 5.10 shows to be optimistic, and it is produced by an idealized signature surrogate whose realizable counterpart is substantially weaker (Section 5.4). The more consistent effect in these experiments is on the rare invalid-TEID class, whose recall rises from 48.2% to 64.4% and whose F1 rises from 0.649 to 0.761; this improvement is present whenever a signature is available and does not rest on the aggregate significance test.

5.7. Component Ablation

To isolate the contribution of each component, Table 12 reports a four-way ablation (CatBoost; CatBoost+NCRE; CatBoost+SDS; full SHE-CB) at 10% and 20% training data. At every data level and for every class, CatBoost+NCRE coincides with CatBoost, and CatBoost+SDS coincides with the full SHE-CB. The entire ensemble gain, including the invalid-TEID recall improvement from 48.9% to 64.4%, is therefore delivered by the SDS signature pathway; in the published gate configuration, the NCRE override does not change a test prediction. We report this finding as a negative result rather than as a caveat. The NCRE was designed to correct rare-class decisions, and on the rare class for which it was designed it corrected none: at every evaluated data level and for every class, CatBoost with the NCRE is numerically identical to CatBoost alone. The reason is described in Section 4: the Priority-2 condition requires simultaneously that the NCRE be confident above $\theta$, that its argument maximum be a non-normal class, and that CatBoost be uncertain below $\tau$, and the calibrated thresholds make the joint event vanishingly rare on this dataset. We retain the component in the architecture and in the released code so that this result is reproducible, but we make no claim on its behalf, and readers considering a centroid-proximity correction for rare classes should weigh this outcome.

Table 12. Component ablation of SHE-CB (accuracy %, mean over 10 seeds).

% Data	CatBoost	CB+NCRE	CB+SDS	SHE-CB
10	89.72	89.72	90.15	90.15
20	89.83	89.83	91.60	91.60

Invalid-TEID recall at 20%: CatBoost 48.9, CB+NCRE 48.9, CB+SDS 64.4, SHE-CB 64.4.

5.8. Confusion Matrices and Operational Metrics

Figure 5 shows the $6\times 6$ confusion matrices for CatBoost and SHE-CB at 20% training data. SHE-CB raises invalid-TEID recall from 48.9% to 64.4% by recovering TEID packets that CatBoost routes to Normal, while the Normal row is unchanged and the normal false-positive rate stays at 0. Inference is inexpensive: on a single CPU host (batched), CatBoost adds about $4.7\times {10}^{-4}$ ms per packet and the full SHE-CB pipeline about $7.2\times {10}^{-4}$ ms per packet (≈$1.38$ million packets/s), well above realistic GTP-U monitoring rates.

Figure 5. Confusion matrices (row-normalized) at 20% training data: CatBoost (left) and SHE-CB (right). The invalid-TEID row is the main difference.

5.9. Non-Normal Flagging Rate on Unseen Attack Families

We measure how a closed-set classifier behaves on attacks absent from its training set. We emphasize at the outset what this measurement is not: the classifier has no unknown class, no rejection option and no calibrated confidence threshold, so it cannot recognize an attack as novel. It can only assign one of its known labels. The quantity reported below is therefore the non-normal flagging rate, defined as the fraction of packets from an unseen family that receive any label other than normal. A novel attack assigned to the flood or malformed class counts towards this quantity, even though the label is wrong. In a leave-one-attack-out (LOAO) test, in which each attack class is removed entirely from training, the non-normal flagging rate is $100\%$ for flood, fragmented, malformed, and spoofing, and $81\%$ for invalid-TEID, with a mean of $96.2\%$. Applying a model trained on the five data-plane attacks plus normal to seven unseen families (Table 13) gives a mean non-normal flagging rate of $88.7\%$, including for control-plane families never seen in training. These figures indicate that the model does not default to the normal class when confronted with unfamiliar traffic, which is a useful property for an alerting system. They do not demonstrate unknown-class recognition, and we make no such claim. Establishing that property would require a rejection mechanism and an open-set protocol reporting an Open-Set Classification Rate (OSCR) curve, unknown-versus-known AUROC and detection at a controlled normal false-positive rate; none of these is provided here, and we identify them as unaddressed.

Table 13. Open-set detection on unseen attack families (trained on the five data-plane attacks + normal).

Unseen Family	Flagged as Non-Normal
B: control-plane session manipulation	100.0%
C: control-plane resource exhaustion	83.0%
E: data-plane new	58.1%
F: reconnaissance/scanning	93.3%
G: protocol fuzzing	99.5%
H: evasion/slow	95.3%
I: multi-vector/APT	91.5%
Mean	88.7%

The bold row reports the mean non-normal flagging rate across the seven unseen attack families.

5.10. Effect of the Train/Test Split (Data Leakage)

A packet-level random split can place near-duplicate packets from the same capture in both the training and test sets, which inflates performance. On a multi-PCAP dataset with several captures per class, a packet-level stratified split attains $61.9\%$ accuracy while a leakage-free PCAP-level (group-aware) split attains $42.9\%$, an inflation of about 19 percentage points. The single-capture-per-attack design of the primary dataset does not permit a PCAP-level split at all: removing the sole capture of a class from training removes that class from the training set entirely. We therefore cannot report leakage-free figures for the headline experiments, and we do not attempt to extrapolate them from the companion dataset, since the magnitude of inflation depends on how much within-capture redundancy a particular capture contains. The consequence must be stated without softening. The 89 to 91% figures reported in this paper are packet-level results; the corresponding leakage-free values are unknown, and the one case in which we can measure the difference shows an inflation of approximately 19 percentage points. Every conclusion in this paper that depends on the absolute level of accuracy should be regarded as unsupported until it is reproduced under a group-aware protocol on a dataset with multiple captures per class. We adopt the PCAP-level protocol as the standard for all future work on this dataset.

5.11. Visualizations

Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 visualize the results reported above. Figure 6 shows the classification error rate on a log scale across all data levels; Figure 7 shows the incremental accuracy gain of SHE-CB over CatBoost standalone; Figure 8 compares per-class recall at 10% and 20% training data; Figure 9 shows the invalid-TEID miss rate on a log scale; Figure 10 shows the progressive accuracy gain across the three architectural layers; and Figure 11 shows the per-class F1 heatmap for SHE-CB across all data levels.

Figure 6. Classification error rate ($1-\mathrm{Acc}$) on log scale for SHE-CB (green, proposed), CatBoost (blue dashed), and the SDS signature surrogate (red, at 0%, 10%, and 20% signature availability). The red curve is a property of the surrogate model, not a measurement of the real Suricata engine, whose standalone performance is reported in Section 5.4. Shaded bands denote ±1 std over 10 seeds. SHE-CB consistently achieves lower error rate than CatBoost from 8% data onwards, with the gap widening at 12 to 20% as surrogate signature coverage increases. The NCRE is inactive at every level shown (Section 5.7).

Figure 7. Incremental accuracy gain of SHE-CB over CatBoost standalone. The gain is zero at 1 to 6% data, where the signature surrogate is almost never available, and grows from 8% onwards as surrogate signature coverage increases. The growth tracks signature coverage only; the NCRE is inactive at every level shown (Section 5.7).

Figure 8. Per-class recall at 10% (left) and 20% (right) training data. The orange-highlighted region marks attack_invalid_teid, the rarest class, where SHE-CB obtains the largest recall improvement over CatBoost. The improvement is produced by the signature pathway; the NCRE contributes no prediction change at these data scales (Section 5.7).

Figure 9. Invalid TEID miss rate ($1-\mathrm{Recall}$) on log scale. SHE-CB’s miss rate drops sharply at 8% data (SDS activation) and remains consistently lower than CatBoost throughout. The reference line shows the miss rate of the SDS signature surrogate; the corresponding miss rate of the real Suricata engine on this class is $47.7\%$, measured in Section 5.4.

Figure 10. Progressive accuracy gain across the three architectural layers. SHE-CB at 10% training data (90.15%) already exceeds CatBoost at 20% (89.83%). We report this as an observation rather than as a demonstrated data-efficiency benefit; the difference is small, is not statistically significant after correction for multiple comparisons, and is obtained under a packet-level split (Section 5.10).

Figure 11. Per-class F1 heatmap for SHE-CB across all data levels. attack_invalid_teid (row 4) shows the largest change, rising from 0.38 at 1% to 0.76 at 20%. This change is driven by the increasing availability of the signature surrogate, not by the NCRE, which is inactive at all levels shown (Section 5.7).

5.12. Discussion

Three properties of the SHE-CB framework emerge from the experimental results. All three should be read under a single standing qualification: the accuracies discussed here are packet-level results on a single-capture-per-class dataset, and Section 5.10 shows that a leakage-free PCAP-level protocol reduces accuracy substantially on a dataset where such a protocol is feasible. What follows therefore describes the relative behavior of the architecture, not an estimate of the accuracy it would achieve in deployment.

Adding a signature pathway to a gradient-boosting classifier yields a small, consistent improvement. SHE-CB attains higher accuracy than the best standalone baseline at every data level beyond 6%, reaching $+1.77$ pp over CatBoost at 20% training data. We attribute this to the signature pathway rather than to the gating mechanism: as shown in Section 4, the gate reduces to a signature override at these data scales, so the result should be read as evidence that a signature signal complements a gradient-boosting classifier on this dataset, not as evidence that input-dependent gating is preferable to simpler fusion. We make no claim of the latter kind, and this paper contains no comparison that would support one.

The measured gain comes from the signature pathway alone, and the NCRE contributes nothing. Zero gain at 1 to 6% data arises because the signature pathway rarely fires at low data levels; the system defers to CatBoost alone. The measured starting point is the real engine. A purpose-built GTP rule set run under Suricata 7.0.3 detects attacks with perfect precision and no false positives on normal traffic, but recovers only $69.2\%$ of attack packets and attains zero per-class recall on the malformed, flood, and spoofing classes: the captured flood rate falls below any usable threshold, inner-IP spoofing leaves no GTP-U header signature, and malformed packets that do raise an alert are not assigned to the malformed class. This is what signature-only detection delivers on this traffic, and it is the realistic lower reference for the signature pathway. The ensemble results reported in this paper are obtained instead with the idealized surrogate, which is supplied the correct label for any covered capture and is therefore not realizable; the $+1.77$ pp gain at 20% data is accordingly an upper bound on what integrating a signature engine could contribute, not an estimate of it. The achievable contribution lies between the two, and this paper does not determine where. We further record as a negative result that a centroid-proximity expert of the kind used here, gated on a confidence threshold, did not correct a single rare-class decision at any evaluated data scale, and readers considering a similar correction mechanism should weigh that finding.

An observation on label efficiency, which we do not claim as a benefit. At 10% training data SHE-CB attains accuracy comparable to standalone CatBoost at 20% (90.15% against 89.83%). We deliberately refrain from describing this as improved label efficiency. The difference is small, it is not statistically significant after correction for multiple comparisons, it is obtained under a packet-level split, and it depends on a signature surrogate whose availability parameter is itself tied to the training-data percentage. Establishing a label-efficiency claim would require a leakage-free protocol and a real signature engine, and we regard it as an open question rather than a result.

Limitations. We state the boundaries of this study together. (i) The evaluation uses a simulated 5G core and a purpose-built dataset, so it characterizes the relative behavior of the architecture and not its performance on operator traffic. (ii) The headline accuracies are packet-level results on a dataset with a single capture per attack class; a leakage-free PCAP-level protocol lowers accuracy substantially where such a protocol is feasible (Section 5.10). (iii) The signature pathway inside the gate is an idealized surrogate that is informed by dataset labels; the real engine, measured standalone in Section 5.4, is substantially weaker, so the reported gain is an upper bound on what signature integration would contribute in practice. (iv) The aggregate accuracy gain is not statistically significant after correction for multiple comparisons. (v) The NCRE is inactive at the evaluated data scales. (vi) The unseen-attack results measure non-normal flagging by a closed-set classifier and do not establish unknown-class recognition. (vii) The class-rebalanced prevalence of the dataset does not reflect operational traffic, so overall accuracy and weighted F1 are not indicative of the alert burden a deployment would face. Statements about deployment, label efficiency, and robustness in this paper should be read subject to all seven of these conditions.

6. Conclusions

This paper presented GTP-AEGIS, a Selective Heterogeneous Ensemble IDS for GTP traffic in 4G/5G mobile core networks under data scarcity. The SHE-CB framework integrates CatBoost gradient boosting (with the NCRE) and a signature-based pathway using the SHE strategy, unified by the IDCG that routes each test sample to the more reliable component based on per-sample confidence scores. Experiments on the purpose-built GTP-AEGIS dataset yield the following findings, all of which are obtained under a packet-level train/test split and are therefore optimistic relative to a leakage-free PCAP-level protocol, as quantified in Section 5.10. First, SHE-CB achieves 91.60% accuracy at 20% training data, an incremental gain of $+1.77$ pp over standalone CatBoost that is not statistically significant after Holm–Bonferroni correction across the tested data levels; we therefore report it as a small positive difference rather than as a demonstrated improvement in aggregate accuracy. Second, and more consistently, SHE-CB raises invalid-TEID recall from 48.9% to 64.4% and macro-F1 from 0.870 to 0.899 at 20%; a component ablation attributes this rare-class improvement to the signature pathway rather than to the NCRE, which changes no prediction at any evaluated data scale. A recall of 64.4% nonetheless leaves more than a third of these attacks undetected, so we present it as an improvement and not as a solution. Third, a real Suricata engine detects attacks with 100% precision and zero false positives on normal traffic but with limited recall (69.2% binary) and zero per-class recall on three of the five attack classes, which quantifies the limit of signature-only detection and shows that the SDS used in the scarcity sweep is an idealized upper bound that no deployment could reproduce. Fourth, in leave-one-attack-out and closed-set flagging tests, the model assigns a non-normal label to 81 to 100% (LOAO) and a mean of 88.7% (unseen families) of packets from attack families absent at training time; because the model has no rejection option, this measures a tendency not to default to the normal class and does not establish unknown-class recognition.

Several limitations frame these results, and they are substantial enough that we present this work as a characterization study rather than as a deployable system. The evaluation uses a simulated 5G core and a purpose-built dataset with a single capture per attack class, so it establishes the relative behavior of the architecture under controlled conditions and provides no evidence about operator-scale readiness. We make no claim that GTP-AEGIS is ready for operational deployment, that it halves the labeling requirement, or that it is robust to novel attacks in the open-set sense. A leakage-free PCAP-level split yields materially lower accuracy than the packet-level split used for the headline numbers, so those numbers are optimistic. Because class identity and capture identity are perfectly collinear in the primary dataset, features may encode which capture a packet came from rather than which attack it represents, and no measurement on this dataset can separate the two. Threshold calibration now uses out-of-fold CatBoost probabilities, removing the in-sample bias of the original design. Future work is ordered by what must be settled first. The prerequisite is data: multiple independent captures per attack class, generated under varied topology, timing and parameterization, so that a group-aware PCAP-level protocol becomes possible and class identity is no longer collinear with capture identity. Until that is available, no claim about generalization from this dataset can be substantiated, and we regard it as a higher priority than any architectural refinement. Thereafter we intend to place a real signature engine inside the gate in place of the idealized surrogate, so that the ensemble becomes realizable; to compare the confidence gate against direct rule override, probability averaging, stacking and calibrated confidence fusion, which this paper does not do; to add a rejection mechanism and evaluate genuine open-set recognition with OSCR and unknown-versus-known AUROC; to either make the NCRE contribute or remove it; and to validate on real operator traffic. Federated-learning extensions of the framework for privacy-preserving multi-operator GTP intrusion detection remain of interest, but they presuppose that the questions above have been answered.