Cost-Sensitive Threshold Optimization for Network Intrusion Detection: A Per-Class Approach with XGBoost

Abstract

Machine learning-based Network Intrusion Detection Systems (NIDSs) typically optimize uniform metrics such as accuracy and F1-score, overlooking the asymmetric cost structure of real-world security operations, where a missed attack (False Negative (FN)) far outweighs a false alarm (False Positive (FP)). We propose a cost-sensitive threshold optimization framework based on XGBoost, using a 10:1 FN-to-FP cost ratio derived from established cost models. We first demonstrate that the default threshold of 0.5 is suboptimal and that a globally optimized threshold of 0.08 substantially reduces total cost. However, a single global threshold cannot accommodate the heterogeneous detection characteristics of diverse attack types. We therefore introduce Per-Class Thresholding, which assigns independently optimized thresholds to each attack class. Evaluated on CIC-IDS2018 and UNSW-NB15 across five independent random seeds, our method achieves a 28.19% cost reduction over the Random Forest baseline on CIC-IDS2018, demonstrating that attack classes undetectable under the global threshold—including DDoS attack-LOIC-UDP (100%), DoS attacks-SlowHTTPTest (99.79%), and FTP-BruteForce (98.16%)—can achieve near-complete cost elimination through individual per-class threshold search. Cross-dataset validation on UNSW-NB15 further confirms that per-class thresholding consistently improves class-level detection, with cost reductions of 74.10% for Reconnaissance, 69.06% for Backdoor, and 54.42% for Analysis attacks. These results confirm that class-specific threshold calibration is essential for cost-effective intrusion detection.

1. Introduction

The scale and complexity of modern network environments are increasing rapidly due to the proliferation of cloud computing and IoT, and cyber attacks are becoming increasingly sophisticated. In particular, Advanced Persistent Threats (APTs) and large-scale Distributed Denial of Service (DDoS) attacks cause immense financial and social losses to enterprises and institutions. Consequently, research on Network Intrusion Detection Systems (NIDSs) utilizing Machine Learning (ML) and Deep Learning technologies has been actively conducted [1,2].

However, existing NIDS studies tend to focus mostly on improving uniform performance metrics such as Accuracy and F1-Score [1]. This approach has a critical limitation in that it overlooks the specific cost structure of actual security operation centers. In a security operation environment, a False Positive (FP), which mistakes benign traffic for an attack, merely wastes the analysis time of security personnel; however, a False Negative (FN), which misclassifies an actual attack as normal and misses it, leads to severe consequences such as data leakage or system destruction. In other words, the cost of False Negatives ($C_{FN}$) is overwhelmingly higher than the cost of False Positives ($C_{FP}$) [3,4].

Most machine learning classification algorithms set the threshold for class classification to a default value of 0.5. This assumes that the costs of FN and FP are equal, which causes an unnecessary increase in Total Cost in the intrusion detection domain, which has an asymmetric cost structure. For example, if the probability of an attack is 0.49, it is classified as ‘Benign’ based on the 0.5 threshold, resulting in a miss (FN). However, in reality, it may be necessary to detect this as an ‘Attack’ for risk management purposes.

To address these problems, this study proposes a Cost-Sensitive Threshold Optimization methodology based on the XGBoost model. Specifically, we define operational cost by setting the cost ratio of FN to FP at 10:1 and explore the optimal threshold that minimizes this cost. Furthermore, to overcome the structural limitations that arise when applying a single Global Threshold, we establish a Per-Class Thresholding strategy that applies different optimal thresholds for each Attack Type and verify its effectiveness.

2. Related Work

2.1. Evolution of Network Intrusion Detection Systems

Early NIDSs, such as Snort developed by Roesch [5], relied on Misuse Detection techniques based on predefined rules. However, these were structurally vulnerable to variant attacks or zero-day attacks. To overcome this, research on Anomaly Detection utilizing statistical techniques and machine learning has been actively performed.

Moustafa and Slay [6] emphasized the importance of ML-based detection by proposing the UNSW-NB15 dataset, which reflects the complex characteristics of modern attack traffic. Sharafaldin et al. [7] released the CIC-IDS2018 dataset containing traffic patterns similar to real operational environments, providing a standard benchmark for NIDS research. Regarding algorithms, tree-based Ensemble models such as Random Forest proposed by Breiman [8] and XGBoost by Chen and Guestrin [9] have become mainstream, surpassing traditional models like SVM and k-NN. According to Zhou [10], ensemble techniques have superior generalization performance compared to single models and are effective in preventing Overfitting. Ferrag et al. [11] showed that Deep Learning models such as CNN and RNN are excellent for learning time-series traffic patterns, but lightweight tree-based models still tend to be preferred in security control systems requiring real-time performance due to high computational costs and training times. More recently, Le et al. [12] further validated XGBoost’s effectiveness in multiclass IIoT intrusion detection under severe class imbalance, reinforcing its practical advantage in security-critical environments.

2.2. Class Imbalance and Data-Level Solutions

Real-world network traffic shows severe Class Imbalance, where benign packets are overwhelmingly numerous and attack packets are very few. Provost [13] pointed out that models trained on such imbalanced data are biased towards increasing accuracy for the majority class (Benign), resulting in poor detection of the minority class (Attack), which is critical for security. As a data-level approach to solve this, Chawla et al. [14] proposed SMOTE (Synthetic Minority Over-sampling Technique), which generates minority class data via interpolation. However, limitations have been pointed out that SMOTE risks propagating noise data and can distort the distribution in high-dimensional data. Therefore, recent studies are shifting towards cost-sensitive learning that handles the imbalance problem at the algorithm level rather than artificially manipulating data.

2.3. Cost-Sensitive Learning and Threshold Optimization

Cost-Sensitive Learning starts from the premise that the risk cost due to misclassification differs for each class. Elkan [3] mathematically established optimal decision criteria reflecting the cost ratio of False Negative and False Positive based on Bayesian decision theory. Stolfo et al. [4] applied this Cost-based Modeling to actual intrusion detection and fraud detection systems, proving early on that cost efficiency is a more important indicator than simple accuracy in security operations. This aligns exactly with the practical requirement that the cost of missed attacks is far higher than the cost of responding to false alarms.

Sheng and Ling [15] presented a methodology to find the optimal threshold by analyzing the Cost Curve post hoc without modifying the trained model. However, existing studies have mostly focused on Binary Classification or remained at Global Optimization, treating all attacks with the same cost. Although Gu et al. [16] measured intrusion detection capability through an information-theoretic approach, research on dynamically optimizing thresholds for each attack type (Per-Class Thresholding) by reflecting the individual risks in a Multi-class environment is still lacking. Class-specific threshold optimization is well established in multi-label classification, where per-label threshold tuning has been shown to significantly improve over global thresholding by accommodating varying class prevalences and asymmetric misclassification costs [17,18]. We adapt this principle to the multi-class intrusion detection setting, where each class-conditional score distribution is exploited to minimize an asymmetric FN/FP cost function. This study proposes a framework that minimizes total operational cost by deriving optimal thresholds per attack type to overcome these limitations.

3. Proposed Methodology

The proposed framework consists of four sequential stages forming an end-to-end data processing and optimization pipeline, as illustrated in Figure 1. In Stage 1 (Data Preprocessing), the raw CIC-IDS2018 dataset undergoes cleaning (removal of NaN and Inf values), Label Encoding of categorical labels, and Stratified splitting into train/validation/test sets (60/20/20 ratio) to preserve class distribution. In Stage 2 (Cost Function Definition), an asymmetric cost model is established with a 10:1 ratio between FN and FP costs, providing the objective function for subsequent optimization. In Stage 3 (Model Training and Global Threshold Optimization), the XGBoost classifier is trained, and a single cost-sensitive threshold (${\tau }_{cs}$) is derived by exhaustive search over [0.01, 0.99] on the validation set. In Stage 4 (Per-Class Threshold Optimization), the global threshold limitation is addressed by computing independent optimal thresholds (${\tau }_k$) for each attack class k, minimizing class-specific costs.

Figure 1. Overview of the proposed cost-sensitive Per-Class Threshold Optimization framework. The dataset is partitioned into Train (60%), Validation (20%), and Test (20%) sets. Threshold optimization is performed exclusively on the Validation set, and final performance is reported on the held-out Test set only.

3.1. Cost Function Definition and Operational Cost Model

The core goal of this study is not simply to improve classification accuracy, but to minimize the total intrusion detection cost representing business risk. For this, the Cost Function is defined as follows:

$$Total Cost = {\displaystyle \sum }_{i=1}^N\left( F{N}_i{C}_{FN}+ F{P}_i{C}_{FP}\right)$$

(1)

• $N$: Total number of samples;
• $FN$: Number of False Negatives;
• $FP$: Number of False Positives;
• $C_{FN}$: Cost per unit of False Negative;
• $C_{FP}$: Cost per unit of False Positive.

In this study, the cost ratio of False Negatives to False Positives was set to $C_{FN}$ = 10 and $C_{FP}$ = 1. This is based on the cost model of KDD Cup 99, a standard benchmark in the intrusion detection field, and Elkan’s cost-sensitive learning theory. According to the evaluation criteria of KDD Cup 99 proposed by Stolfo et al. [4], a cost weight of up to 4 times that of normal traffic was assigned depending on the attack type (R2L). This study adopted a conservatively increased weight of 10 times, reflecting the increased damage scale and risk of modern cyber threats. While the exact cost ratio is inherently context-dependent, the 10:1 ratio serves as a representative asymmetric setting that is consistent with values commonly adopted in cost-sensitive intrusion detection literature [3]. The sensitivity of the results to this cost ratio is explicitly examined in Section 4.5.

In Elkan’s study [3], the theoretical optimal threshold in binary classification is defined as $C_{FP}$/($C_{FP}$ + $C_{FN}$). Substituting the 10:1 ratio of this study into the formula yields a theoretical threshold of approximately 0.09 (1/11). As will be validated in Section 4.2, the experimental results confirm that the single optimal classification threshold for the baseline (Random Forest) is 0.09, consistent with this theoretical value, and the optimal threshold for the proposed model (XGBoost) is 0.08. This demonstrates that the cost ratio setting of this study aligns with the intrinsic characteristics of the dataset and mathematical theory.

3.2. Threshold Definition

In general binary classification, the threshold is set to τ = 0.5. That is, if the probability of an attack exceeds 50%, it is considered an “Attack”. However, τ = 0.5 implicitly assumes the following:

• (Assumption A) The risk (cost) of False Negatives and False Positives is treated equally.
• (Assumption B) The prior probability of Attack/Benign classes and the operational purpose are close to accuracy-centric.

In security operation environments, these assumptions usually do not hold. In particular, the mistake of missing an attack (FN) can lead to intrusion spread, data leakage, and service interruption, resulting in high damage costs. On the other hand, False Positives cause analysis personnel consumption and alarm fatigue, but their cost structure is generally evaluated differently from FN. Therefore, the mechanical criterion of τ = 0.5 can be inefficient from a security operations perspective.

Therefore, this study defines a cost-sensitive optimal threshold ${\tau }_{cs}$, which is distinct from the default threshold τ = 0.5, as the value that minimizes the Total Cost function defined in Section 3.1. Formally, ${\tau }_{cs}$ = argmin (Total Cost(τ)) over the range τ ∈ [0.01, 0.99]. This study applies ${\tau }_{cs}$ instead of a fixed threshold to minimize total cost. In the common security operations scenario where $C_{FN}$ > $C_{FP}$, the resulting ${\tau }_{cs}$ tends to be lower than 0.5; however, ${\tau }_{cs}$ is not restricted to values below 0.5 and depends entirely on the operational cost structure. Specifically, with $C_{FN}$ > $C_{FP}$ in this study, ${\tau }_{cs}$ = 0.08 for XGBoost. This lowered threshold makes the model more sensitive to attacks by classifying more traffic as potential threats. While this may partially increase FPs, it can substantially reduce the high-cost FNs, yielding a net reduction in total operational cost. For instance, setting ${\tau }_{cs}$ = 0.1 means that even a 0.1 (10%) probability raises suspicion and targets it for detection (reducing FN), which can be more advantageous from a total security cost perspective. The key is not to view detection performance solely as simple accuracy, but to optimize the threshold according to the cost structure ($C_{FN}$, $C_{FP}$) of the operating environment.

3.3. XGBoost-Based Threat Detection and Threshold Optimization

This study uses the XGBoost (eXtreme Gradient Boosting) algorithm, which offers excellent predictive performance and computational efficiency, as the classification model. The optimization process proceeds in two main stages:

• Global Threshold Optimization: For all traffic, a single threshold (${\tau }_{global}$) is searched exhaustively from 0.01 to 0.99 in increments of 0.01 on the validation set to ensure global optimality. The Cost Curve is analyzed to find the point where the total cost is minimized. Although the search spans the full probability range, as discussed in Section 3.2, the cost-sensitive optimal threshold ${\tau }_{cs}$ converges below 0.5 when $C_{FN}$ > $C_{FP}$
• Per-Class Threshold Optimization: We focus on the fact that data distribution and detection difficulty differ for each attack type. After global optimization, independent cost-minimizing thresholds (${\tau }_k$) are recalculated and applied for each attack class ($k$). The final decision is determined by whether the probability $P(y=k|x)$ for each class exceeds ${\tau }_k$.

XGBoost is trained with the multi:softprob objective, which outputs a probability distribution over all K classes for each sample, satisfying $\sum P(y=k|x)$ = 1. This multi-class probability output enables two distinct threshold strategies. In Global Threshold Optimization, the aggregate attack probability is defined as 1 − P(Benign|x), collapsing all attack classes into a single score against which a single threshold ${\tau }_{cs}$ is applied. In Per-Class Threshold Optimization, each class-specific probability P(y = k|x) is used directly and compared against its own independently optimized threshold ${\tau }_k$. This distinction is critical: the global approach treats all attack types uniformly, whereas the per-class approach exploits the fact that different attack types exhibit characteristically different probability ranges, enabling class-level sensitivity tuning that the global approach cannot achieve. The Formal Decision rule for the per-class prediction is presented in Algorithm 1.

Algorithm 1: Per-class threshold decision rule
Input: Sample x    P[k] = P(y = k|x) for each class k    Per-class thresholds τk (optimized on validation set) Output: Binary label: Attack or Benign
1	for each attack class k (k ≠ Benign) do
2	if P[k] ≥ τk then
3	flagk ← 1
4	else
5	flagk ← 0
6	end if
7	end for
8	if ∃k: flagk = 1 then
9	return Attack    ▷ OR-combination: any class trigger raises alert
10	else
11	return Benign    ▷ fallback when no class exceeds its threshold
12	end if

4. Experiments and Results

4.1. Experimental Environment and Dataset

The experiment was conducted using the CIC-IDS2018 dataset. For data preprocessing, missing values (NaN) and infinite values (Inf) were replaced with 0, and categorical labels were quantified through Label Encoding. Specifically, infinity values in flow-rate features (e.g., Flow Bytes/s, Flow Pkts/s) arise from division by zero when a flow’s duration is zero, representing instantaneous flows with no measurable rate; replacing these values with zero is therefore semantically appropriate, as such flows contribute no flow-rate signal to the classifier. The affected samples constitute less than 0.60% of the full dataset (95,760 out of 16,232,943 rows), confirming that this preprocessing choice has a negligible impact on the overall results. The entire data was split into training, validation, and test sets at a 60/20/20 ratio using stratified sampling to preserve class distribution. Thresholds are optimized exclusively on the validation set, and final performance is reported on the held-out test set only. All experiments are repeated with five independent random seeds {42, 0, 7, 13, 99}, and the results are reported as mean ± standard deviation. Naive Bayes (GaussianNB, default parameters), Decision Tree (max_depth = 10), and Random Forest (n_estimators = 100, max_depth = 20) were selected as comparison models, and all experiments were performed using Scikit-learn and XGBoost libraries in a Python (v3.13.2) environment. The XGBoost classifier was configured with the following hyperparameters: n_estimators = 500, learning_rate = 0.1, max_depth = 10, objective = multi:softprob. The implementation code and random seeds used in this study are publicly available at https://github.com/987sng/NIDS_Cost_Sensitive_TH (accessed on 23 March 2026).

4.2. Experiment 1: Baseline Comparison and Global Threshold Analysis

In the first experiment, we compared the performance when using the existing method with a fixed threshold (0.5) versus applying the Optimal Threshold derived for cost minimization.

Figure 2 shows the change curve of total cost according to threshold changes. Except for Naive Bayes, tree-based models (Decision Tree, Random Forest, XGBoost) showed a tendency for cost to be minimized at a point much lower than 0.5 (approx. 0.08). This suggests that since $C_{FN}$ is 10 times higher than $C_{FP}$, it is cost-efficient to lower the threshold so that the model detects attacks more sensitively.

Figure 2. Total cost curve by threshold (Validation set, CIC-IDS2018, seed = 42). Star markers indicate the cost-minimizing threshold for each model.

According to the experimental results in Table 1, each model exhibits a distinct optimal threshold: Naive Bayes converges at 0.99 (reflecting its fundamentally different probability calibration; no explicit calibration was applied, as XGBoost’s softmax probabilities were used directly and threshold optimization on the validation set implicitly compensates for miscalibration), Decision Tree at 0.09, Random Forest at 0.09, and XGBoost at 0.08. This variation arises from differences in how each model calibrates its predicted probabilities. The widely used Random Forest model incurred a mean cost of 318,011.6 (±479.5) when using the 0.5 threshold, but the cost decreased by approximately 22.5% to 246,521.8 (±1296.6) when the optimal threshold (0.09) was applied. XGBoost also showed a 27.7% cost reduction compared to the fixed τ = 0.5 baseline, demonstrating that threshold optimization is essential regardless of the model type.

Table 1. Performance comparison of algorithms: Fixed vs. optimal threshold. All metrics are reported as mean across 5 seeds; cost is additionally reported with ± std.

Model	Metric	Fixed (τ = 0.5)	Optimal (Best)	Improvement
Naive Bayes	Threshold	0.50	0.99	-
	Total Cost	2,634,294.6 ± 2406.1	2,632,796.6 ± 2584.0	0.06%
	Accuracy	0.1924	0.1930	-
	Recall	0.9977	0.9976	-
Decision Tree	Threshold	0.50	0.09	-
	Total Cost	325,776.2 ± 3843.5	262,111.2 ± 4300.5	19.54%
	Accuracy	0.9897	0.9799	-
	Recall	0.9411	0.9600	-
Random Forest	Threshold	0.50	0.09	-
	Total Cost	318,011.6 ± 479.5	246,521.8 ± 1296.6	22.48%
	Accuracy	0.9896	0.9737	-
	Recall	0.9424	0.9671	-
XGBoost	Threshold	0.50	0.08	-
	Total Cost	315,717.0 ± 1064.8	228,402.2 ± 288.5	27.65%
	Accuracy	0.9898	0.9740	-
	Recall	0.9429	0.9713	-

Notably, this single global threshold introduces a structural tradeoff: attack types whose predicted probabilities are predominantly below 0.08 may be systematically missed (increasing FNs), while benign traffic slightly above 0.08 may generate excessive false alarms. This observation motivates the Per-Class Thresholding approach evaluated in the following experiment.

4.3. Experiment 2: Performance Evaluation of Proposed Method (Per-Class Thresholding)

In the second experiment, we evaluated the effectiveness of the Per-Class Threshold Optimization technique proposed in this study. Figure 3 compares the final operational costs of baseline models and the proposed technique. Random Forest (Fixed 0.5) recorded a mean cost of 318,011.6 (±479.5), whereas the proposed XGBoost (Per-Class Opt) model recorded a mean cost of 228,368.6 (±292.2), achieving a cost reduction of approximately 28.19%. The aggregate cost difference between Per-Class Opt (228,368.6 ± 292.2) and Global Opt (228,402.2 ± 288.5) is modest (0.01%), which is attributable to the dominance of a single attack class (Infiltration) that accounts for over 99% of the total cost and whose optimal threshold already coincides with the global optimum. However, the practical value of Per-Class Thresholding becomes evident when examining individual attack types.

Figure 3. Total operational cost comparison (mean over 5 seeds, CIC-IDS2018). The proposed XGBoost Per-Class method achieves a 28.2% cost reduction relative to the Random Forest fixed-threshold baseline.

According to the detailed analysis in Table 2, for DDoS attack-LOIC-UDP, the cost decreased by an additional 100.00% when applying the individual optimal threshold (0.01) compared to applying the global optimal threshold (0.08). This means that this attack tends to be predicted with low probability values by the model, resulting in FNs under the global threshold of 0.08. Conversely, DoS attacks-GoldenEye showed no additional cost reduction under per-class optimization, as its Global Cost (3) is already negligible and its optimal threshold (0.02) coincides closely with the global optimum, suggesting the model handles this class well under any threshold in the tested range.

Table 2. Per-class performance analysis.

Attack Type	Global Cost (τ = 0.08)	Per-Class Cost	Reduction (%)	Optimal Threshold
DDoS attack-LOIC-UDP	50	0	100.00%	0.01
DoS attacks-SlowHTTPTest	3303	7	99.79%	0.01
FTP-BruteForce	434	8	98.16%	0.01
DDoS attack-LOIC-HTTP	178	87	51.12%	0.01
Brute Force -XSS	107	99	7.48%	0.06
Brute Force -Web	494	479	3.04%	0.10
Bot	985	982	0.30%	0.09
DoS attacks-GoldenEye	3	3	0.00%	0.02
Infiltration	226,756	226,756	0.00%	0.08
SSH-Bruteforce	110	110	0.00%	0.16
DoS attacks-Slowloris	458	468	−2.18%	0.11
SQL Injection	22	23	−4.55%	0.02
DDoS attack-HOIC	37	40	−8.11%	0.01
DoS attacks-Hulk	1	2	−100.00%	0.02

In summary, the optimal threshold (0.08) of the Infiltration attack, which accounts for the majority of the total cost, had a Dominant Influence on the entire optimization process. Consequently, attack types with completely different traffic characteristics, such as DDoS attack-LOIC-UDP or DDoS attack-LOIC-HTTP, were forced to apply the uniform standard of 0.08. This confirms the structural contradiction of a single optimal threshold applied globally and corroborates the necessity of dynamic thresholds by attack type. Notably, some classes (e.g., DoS attacks-Slowloris, DDOS attack-HOIC) exhibit slight cost increases under per-class optimization, indicating that the global threshold already represents a local optimum for those classes and that per-class tuning does not universally guarantee improvement. Furthermore, the Infiltration class represents a fundamentally different challenge: its predicted probabilities consistently fall below any effective threshold regardless of tuning, suggesting that post hoc threshold optimization alone is insufficient for this class. Complementary approaches such as cost-sensitive training, dedicated anomaly detectors, or ensemble strategies may be required.

The key insight of per-class threshold optimization is that each attack class has a distinct probability distribution and detection sensitivity, and a single global threshold inevitably sacrifices detection quality for some classes to accommodate others. By independently tuning ${\tau }_k$ per class, the framework recovers attacks systematically missed under the global threshold—as demonstrated by DDoS attack-LOIC-UDP (100% cost reduction) and FTP-BruteForce (98.16%)—while adjusting the threshold for other classes according to their individual detection characteristics. However, individual class improvements do not always translate to a net reduction in total operational cost. Lowering ${\tau }_k$ for a specific class reduces FNs for that class but may increase FPs across classes with overlapping probability distributions, potentially offsetting the gains elsewhere. This explains why some classes show negative reductions in Table 2 and why the aggregate improvement over global optimization remains modest (0.01% on CIC-IDS2018). The practical value of per-class thresholding therefore lies not in its aggregate cost advantage, but in its ability to provide tailored, class-specific control—particularly critical for high-priority or hard-to-detect attack categories where the cost of a miss far exceeds the cost of a false alarm.

Figure 4 presents the confusion matrix of the proposed XGBoost Per-Class model on the CIC-IDS2018 test set (seed = 42). The model correctly detects 533,505 out of 549,647 attack instances (recall = 97.07%), suppressing FNs to 16,142 cases. The 67,308 FP cases reflect the intentional lowering of per-class thresholds to prioritize recall under the 10:1 cost asymmetry.

Figure 4. Confusion Matrix of XGBoost Per-Class Threshold Optimization (CIC-IDS2018, seed = 42). The model correctly detects 533,505 out of 549,647 attack instances (recall = 97.07%), while 16,142 attacks are missed (FN) and 67,308 benign flows are flagged (FP).

4.4. Generalizability Validation on UNSW-NB15

To assess the generalizability of the proposed framework, experiments were repeated on the UNSW-NB15 dataset using the same 60/20/20 train/validation/test protocol and five independent random seeds. As shown in Table 3, XGBoost with global threshold optimization achieved a mean cost of 5398.8 (±65.7), and per-class optimization achieved 5420.0 (±64.9), representing 66.85% and 66.72% cost reductions, respectively, relative to the Random Forest fixed baseline (16,284.0 ± 229.9). Unlike CIC-IDS2018, the difference between global and per-class optimization on UNSW-NB15 is statistically negligible (within one standard deviation), which may reflect the smaller per-class sample sizes in this dataset, making per-class threshold estimates less stable. Nevertheless, per-class optimization yields a higher recall (0.9989 vs. 0.9979), which is advantageous in recall-critical deployment scenarios. The per-class breakdown (Table 4) reveals that all attack categories benefited from individualized thresholds, with Reconnaissance (74.10%), Backdoor (69.06%), and Analysis (54.42%) showing the largest improvements, confirming the cross-dataset utility of the proposed approach.

Table 3. UNSW-NB15—Generalizability validation (mean ± std, 5 seeds). Fixed τ = 0.5 baseline applied to all models; XGBoost thresholds optimized on the validation set.

Model	Global Threshold	Fixed Cost (Mean ± Std)	Optimal Cost (Mean ± Std)	Optimal Recall
Naive Bayes	0.03	18,719.0 ± 217.5	13,835.4 ± 230.5	0.9927
Decision Tree	0.01	26,042.8 ± 2727.9	5555.6 ± 15.3	1.0000
Random Forest	0.09	16,284.0 ± 229.9	5441.2 ± 46.5	0.9994
XGBoost (Global)	0.03	15,803.0 ± 351.7	5398.8 ± 65.7	0.9979
XGBoost (Per-Class)	—	—	5420.0 ± 64.9	0.9989

Table 4. Per-class cost breakdown—XGBoost on UNSW-NB15 (seed = 42, global τ = 0.03).

Attack Type (UNSW-NB15)	Global Cost (τ = 0.03)	Per-Class Cost	Reduction	Optimal τ
Reconnaissance	1965	509	74.10%	0.01
Backdoor	2240	693	69.06%	0.01
Analysis	1867	851	54.42%	0.01
DoS	2279	1377	39.58%	0.01
Shellcode	368	269	26.90%	0.01
Generic	1480	1096	25.95%	0.01
Worms	62	52	16.13%	0.01
Fuzzers	5857	4985	14.89%	0.01
Exploits	1125	1070	4.89%	0.02

4.5. Cost Ratio Sensitivity Analysis

To examine the sensitivity of results to the assumed cost ratio, experiments were conducted under FN:FP ratios of 5:1, 10:1, and 20:1 on CIC-IDS2018. As shown in Table 5, as the cost ratio increases, the globally optimal threshold decreases (0.17 → 0.08 → 0.04), consistent with Elkan’s theoretical prediction that the optimal threshold equals $C_{FP}$/($C_{FP}$ + $C_{FN}$). Across all three ratios, the per-class and global approaches yield comparable total costs (differences within ±0.13%), confirming that the dominance of the Infiltration class is robust to cost ratio variation. These results demonstrate that the proposed threshold optimization framework generalizes reliably across a range of operationally plausible cost assumptions. Figure 5 illustrates this monotonic decrease in optimal threshold as the FN:FP cost ratio increases, corroborating Elkan’s theoretical prediction.

Table 5. Cost ratio sensitivity—XGBoost, CIC-IDS2018 (seed = 42). Optimal Global Threshold (τ*) decreases as FN:FP ratio increases, consistent with Elkan’s theoretical prediction τ* = $C_{FP}$/($C_{FP}$ + $C_{FN}$).

FN:FP Ratio	τ*	Global Cost	Per-Class Cost	Difference (%)
5:1	0.17	134,359	134,530	−0.13%
10:1	0.08	228,756	228,728	+0.01%
20:1	0.04	358,933	358,840	+0.03%

Figure 5. Optimal Global Threshold (τ*) vs. FN:FP Cost Ratio (XGBoost, CIC-IDS2018, seed = 42). τ* decreases monotonically from 0.17 (5:1) to 0.08 (10:1) to 0.04 (20:1), consistent with Elkan’s theoretical prediction τ* = $C_{FP}$/($C_{FP}$ + $C_{FN}$).

5. Conclusions

This study proposed an XGBoost-based Per-Class Threshold Optimization methodology to solve the asymmetric misclassification cost problem that has been underexplored in Network Intrusion Detection Systems. Experimental results demonstrated that the conventionally used threshold of 0.5 is highly suboptimal in security operation environments and that the threshold should be significantly lowered to the level of 0.08 in consideration of False Negative costs.

Furthermore, by identifying that the optimal detection sensitivity differs for each attack type and applying the Per-Class Dynamic Thresholding technique reflecting this, we achieved an operational cost reduction effect of approximately 28.19% compared to the baseline (Random Forest, Fixed τ = 0.5), across five independent random seeds on the CIC-IDS2018 dataset, with further validation on the UNSW-NB15 dataset achieving a 66.72% cost reduction. In particular, assigning individually low thresholds (0.01) to attacks that are difficult to detect, such as FTP-BruteForce (98.16%) and DoS attacks-SlowHTTPTest (99.79%), to increase defense rates is a practical contribution of this study. This finding is consistent with recent work by Howe et al. [2], who demonstrated that cost-sensitive approaches significantly improve operational efficiency in real-world security environments. However, for attack classes such as Infiltration whose predicted probabilities consistently fall below any effective threshold, post hoc threshold optimization alone is insufficient. Future work should explore complementary approaches, including cost-sensitive training, dedicated anomaly detectors, and adaptive thresholding mechanisms, for such hard-to-detect classes.

Nevertheless, this study has several limitations. First, while experiments were conducted on two benchmark datasets (CIC-IDS2018 and UNSW-NB15), further validation on additional diverse network environments would strengthen generalizability. Second, although a sensitivity analysis across cost ratios (5:1, 10:1, 20:1) was conducted, and the results remained consistent, the optimal FN-to-FP ratio remains inherently context-dependent and may vary across organizations. Third, this study focuses specifically on post hoc threshold optimization as a cost minimization strategy, rather than modifying the model training process itself. Investigating the complementary effects of combining threshold optimization with training-time cost-sensitive techniques (e.g., class weight adjustment or resampling methods) remains an avenue for future work. Finally, the thresholds are optimized statically on historical data and do not adapt to evolving attack patterns in real time.

To address remaining limitations, future research plans should aim to validate the framework on further benchmark datasets such as NSL-KDD [1], investigate cost-sensitive training strategies that complement post hoc threshold optimization, and expand to a Reinforcement Learning-based system that dynamically updates thresholds (Adaptive Thresholding) in accordance with changing attack patterns in real-time traffic environments.