IID-DAKD: An Incremental Intrusion Detection Method for Encrypted Traffic Based on Dual Augmentation and Fusion Knowledge Distillation

Abstract

To address the pronounced degradation in detection accuracy of encrypted traffic intrusion models after incremental updates, which is primarily caused by catastrophic forgetting and task-level overfitting during incremental learning, this paper proposes a novel incremental intrusion detection method for encrypted traffic based on dual augmentation and fusion knowledge distillation, termed IID-DAKD. First, both known and previously unseen attacks identified during detection are leveraged to update a representative sample set. An encrypted traffic representative sample augmentation strategy based on Gaussian noise is then devised to reduce storage requirements and classifier bias, thereby effectively mitigating catastrophic forgetting. Second, a self-supervised learning framework driven by encrypted traffic class augmentation is constructed to alleviate representation bias and suppress task-level overfitting. Finally, three complementary knowledge distillation strategies are jointly employed to extract and transfer attack classification knowledge from the old model to the updated model, further improving detection accuracy and robustness while enhancing training efficiency. Extensive experimental results demonstrate that the proposed IID-DAKD approach alleviates catastrophic forgetting and task-level overfitting while maintaining symmetrical knowledge transfer during incremental learning, enabling efficient model updates and high detection accuracy for encrypted traffic intrusion detection.

1. Introduction

In real-world network environments, which are inherently open and highly dynamic, encrypted traffic patterns and attack strategies continuously evolve, leading to the persistent emergence of novel attack categories [1,2]. As a result, existing network intrusion detection models often exhibit a substantial degradation in performance, primarily because they lack the capability to rapidly adapt to the growing prevalence of previously unseen attack types [3]. This challenge underscores the urgent need to investigate lightweight and efficient incremental model update mechanisms for unknown attack detection. The fundamental objective of such methods is to enable intrusion detection models to continuously adapt to and accurately identify newly emerging network attacks throughout the incremental learning process, while simultaneously maintaining a high detection performance for previously learned, known attacks.

Deep learning-based intrusion detection models for encrypted traffic typically rely on large-scale and comprehensive training datasets. However, retaining excessive volumes of historical data not only incurs prohibitive computational and storage costs but may also be infeasible due to privacy constraints [4]. Under these practical limitations, incremental learning for encrypted traffic intrusion detection faces two major challenges. First, catastrophic forgetting of previously learned attack categories remains a critical issue [5]. Each incremental update conventionally requires retraining the model using both new and historical attack data, which leads to inefficient resource utilization and increased model update complexity. Given storage limitations and privacy concerns, it is impractical to preserve large amounts of historical attack data over long periods. Consequently, when samples from old attack categories are insufficient, the model tends to bias its learning toward newly introduced attack types, gradually losing previously acquired knowledge. This phenomenon results in a pronounced decline in detection accuracy for known attack categories. Second, task-level overfitting with respect to current attack types poses another significant challenge [6]. During incremental training, deep learning-based intrusion detection models often focus excessively on the features of the attack data encountered in the current task, which hampers the extraction of generalized representations shared across different attack categories. This limitation significantly restricts the model’s generalization capability and its ability to incrementally detect unknown attacks, thereby undermining its ability to effectively adapt to the continuously evolving network threat landscape.

To address the aforementioned challenges, this paper proposes an incremental intrusion detection method for encrypted traffic based on dual augmentation and fusion knowledge distillation, referred to as IID-DAKD. The proposed approach effectively mitigates catastrophic forgetting and task-level overfitting during model updates by leveraging only a limited amount of historical attack data, thereby enabling accurate identification of previously unseen encrypted malicious traffic. The main contributions of this work are summarized as follows:

• A Gaussian noise-based encrypted traffic representative sample augmentation strategy is developed to substantially reduce storage requirements and classifier bias, while alleviating catastrophic forgetting in incremental intrusion.
• A self-supervised learning framework based on encrypted traffic class augmentation is constructed to address representation bias and suppress task-level overfitting during incremental updates.
• A fusion knowledge distillation method that integrates attention mechanisms, sample relationships and network responses (FKD-ARR) is proposed, enhancing the effectiveness of symmetrical knowledge transfer and overall model performance.
• Finally, the complete IID-DAKD framework is implemented and systematically evaluated against state-of-the-art incremental intrusion detection approaches and classical incremental learning models. Experimental results demonstrate that IID-DAKD markedly reduces prior knowledge forgetting and overfitting to current attack types while requiring only a minimal amount of historical attack data. Moreover, the proposed method substantially improves the incremental detection accuracy of newly emerging attacks, highlighting its practical potential for deployment in network intrusion detection systems operating in open and dynamic network environments.

2. Related Work

2.1. Class-Incremental Learning Methods

An ideal machine learning model should be capable of continuously acquiring new classes from streaming data, thereby progressively enhancing its discriminative capability. This learning paradigm, known as class-incremental learning, has emerged as a prominent research topic in the machine learning community in recent years [7]. Existing studies on class-incremental learning methods primarily focus on three complementary levels, namely data, parameters, and algorithms, as summarized in Table 1.

Table 1. Classification of class incremental learning methods.

Classification Level	Subtype	Representative Methods	Characteristics
Data level	Data replay	SR [8], GR [9]	Easy to operate, prone to overfitting.
	Data constraints	GEM [10], A-GEM [11]	High optimization complexity, dependent on assumptions.
Parameter level	Parameter regularization	EWC [12], SI [13], IADM [14]	The storage of the constraint matrix is costly.
	Dynamic structure	DEN [15], DER [16]	The number of model parameters increases linearly with the data.
Algorithmic level	Knowledge distillation	LwF [17], iCaRL [18]	Constraints are intuitive and easy to implement.
	Lagged adjustment	Rebalancing [19], BiC [20], WA [21]	Can eliminate model bias.

Data-level methods focus on the effective utilization, storage, and generation of data to mitigate catastrophic forgetting. These approaches are generally categorized into data replay and data constraint strategies. Data replay methods maintain a subset of samples from previously learned categories to prevent the model from forgetting historical knowledge while learning new classes. Representative techniques include reservoir sampling to preserve sample diversity [8], the integration of recent class-center classifiers to address concept drift [22], and the incorporation of meta-learning mechanisms to anchor old class knowledge [23]. In addition, generative models have been employed to dynamically synthesize samples from previous classes, thereby facilitating continual learning without explicit data storage [9,24]. In contrast, data constraint methods aim to restrict model updates using old category data so as to prevent catastrophic forgetting. Notable approaches such as GEM [10] constrain parameter updates by solving quadratic programming problems, whereas A-GEM [11] improves training efficiency by imposing sample-based constraints on randomly selected batches. Other methods, including orthogonal parameter updates [25] and gradient decoupling techniques [26], further ensure that parameter updates for new categories do not degrade the performance on previously learned classes.

Parameter-level methods focus on controlling and adapting model parameters in response to continuously arriving incremental data and can be broadly divided into parameter regularization and dynamic structure approaches. Parameter regularization methods, such as EWC [12] and SI [13], estimate the importance of model parameters and constrain critical parameters to prevent excessive drift during new task learning. Related studies incorporate Fisher information and information gain [27], as well as IADM [14], to further enhance parameter stability. Dynamic structure methods, including model replication [28,29] and pruning strategies [15,30], adapt the network architecture by either duplicating relevant model components or eliminating redundant parameters to accommodate new tasks. Notably, DER [16] achieves state-of-the-art class-incremental learning performance on multiple benchmark datasets by jointly leveraging model replication and masking. Nevertheless, both parameter regularization and dynamic structure approaches often incur increased storage and computational overhead, which may limit their applicability in resource-constrained real-world scenarios.

Algorithm-level methods seek to optimize incremental learning by designing efficient model update mechanisms or exploiting inductive biases inherent in the learning process. These methods can be divided into knowledge distillation-based and logit adjustment-based strategies. Knowledge distillation approaches focus on transferring knowledge from previously learned tasks to new ones, as exemplified by LwF [17] and iCarl [18]. Subsequent studies have enhanced distillation effectiveness by incorporating unlabeled data [31], attention-based distillation mechanisms [32], class-wise Gaussian distribution modeling [33], and feature-space distillation techniques [34]. Conversely, logit adjustment methods exploit biases in model outputs and weight distributions to correct class imbalance effects during incremental learning [19]. Typical strategies include adjusting linear classifiers using uniformly sampled exemplar sets [35] and retraining models with replayed samples to improve online incremental learning performance [36]. Moreover, several studies have observed that models tend to assign higher confidence scores to newly introduced classes, motivating corrective mechanisms such as dual memory storage [37], linear bias correction layers [20], and feature residual weighting functions [38]. The WA strategy [21] further normalizes classifier weights after each task to alleviate prediction bias. Despite their effectiveness, knowledge distillation methods may impose constraints on model flexibility, potentially hindering the learning of new classes, whereas logit adjustment strategies often introduce additional statistical assumptions, increasing algorithmic complexity and storage overhead.

2.2. Encrypted Traffic Intrusion Detection Methods

The core of network intrusion detection lies in the rapid identification of malicious behaviors through the analysis of network traffic. Traditional traffic detection techniques based on port monitoring or deep packet inspection are not suitable for encrypted traffic. Although the content of encrypted traffic is inaccessible, machine learning can still detect potential network attacks by relying solely on packet header information and network flow statistics. In recent years, significant research efforts have focused on intrusion detection for encrypted traffic. Based on whether traffic decryption is performed, encrypted traffic intrusion detection methods can be broadly categorized into active detection and passive detection [39,40].

Active detection methods for encrypted traffic intrusion emphasize the application of security mechanisms such as trusted execution environments and controllable transport protocols. Depending on the degree of decryption, active detection can be further divided into detection based on plaintext-accessible techniques and detection based on searchable encryption techniques. Ref. [41] first proposed a MitM (Man-in-the-Middle) plaintext detection strategy, which decrypts encrypted enterprise traffic via security appliance proxies and leverages deep packet inspection to obtain full traffic information. However, the openness of MitM raises concerns regarding the security of middlebox devices. To address this, Ref. [42] introduced the LightBox system assisted by Intel SGX, incorporating virtual network devices to access fully protected packets, thereby ensuring traffic security. Nevertheless, LightBox still exhibits limitations in protecting virtualized environments. Subsequently, Ref. [43] proposed a modular virtual security architecture, S-Blocks, which utilizes SGX to create protected execution environments that safeguard sensitive data from external attacks and interference, thereby enhancing the accuracy of encrypted traffic detection. With the development of cloud computing, the cloud-based deployment of middlebox devices has become an emerging trend. Nevertheless, privacy protection and the security of middlebox environments remain critical challenges for active detection methods in practical applications. Ref. [44] introduced BlindBox, a relatively mature system for detecting encrypted traffic based on searchable encryption. BlindBox can detect predefined keywords in encrypted traffic through middleboxes while ensuring that no other traffic information is disclosed. However, its practical efficiency is limited due to significant delays incurred during the setup of inspection channels. To mitigate this limitation, Ref. [45] proposed the PrivDPI system, which introduces a reusable obfuscation rule generation technique, substantially improving the overall efficiency of BlindBox and making encrypted traffic intrusion detection more feasible in real-world scenarios. Further addressing the efficiency and security limitations of BlindBox and PrivDPI, Ref. [46] proposed the P2DPI system, which provides the same level of security and privacy as BlindBox while offering faster configuration and encryption capabilities, outperforming PrivDPI in terms of performance. Despite the notable advantages of searchable encryption in privacy preservation, its application still faces challenges regarding key management, encryption strategies, and analysis efficiency [39].

Passive detection methods for encrypted traffic intrusion identify malicious flows without performing any encryption or decryption operations. These methods primarily rely on machine learning or deep learning algorithms, leveraging model architecture design and parameter optimization strategies to achieve optimal performance [39,40]. For machine learning-based approaches, the first step involves extracting features from encrypted traffic, followed by classification using algorithms such as SVM, Random Forest, and XGBoost. Ref. [47] employed packet-level, protocol-level, and certificate-level features in conjunction with an SVM classifier to detect encrypted malicious traffic. Ref. [48] extracted protocol-related and protocol-independent features and applied a Random Forest multi-classification model for detection. Ref. [49] utilized connection, session, and certificate features, applying SVM, Random Forest, and XGBoost models to differentiate between benign and encrypted malicious traffic. Compared with traditional machine learning methods, deep learning approaches can more readily discover non-intuitive relationships among traffic features that are difficult for humans to identify. Currently, deep learning-based detection methods for encrypted traffic intrusion focus primarily on the performance of CNN, LSTM, and Auto-Encoder models [50]. For instance, Ref. [51] converted SSL/TLS features of encrypted traffic into images and trained CNN models on datasets enhanced with Perlin noise, improving model generalization. Ref. [52] processed encrypted sessions at both the field and packet levels, applying BiLSTM and LSTM to capture packet-level and flow-level semantics, which were then fused to represent encrypted traffic for detection purposes. Ref. [53] proposed a neural network method based on stacked autoencoders to identify encrypted traffic intrusion in software-defined networks. Given that malicious encrypted communications often exhibit forward and backward dependencies, researchers have explored hybrid models combining CNN and RNN architectures to capture both spatial and temporal correlations among encrypted traffic packets, thereby improving detection accuracy. Refs. [54,55] employed 1D-CNN and BiGRU to extract spatial and temporal features, respectively, and utilized dual-attention mechanisms to highlight critical malicious features, enhancing detection accuracy. Overall, deep learning-based encrypted traffic intrusion detection models are better suited for uncovering intrinsic relationships among high-dimensional features and typically achieve superior performance.

2.3. Incremental Intrusion Detection Approaches

Current research on the incremental updates of network intrusion detection models predominantly centers on incremental learning paradigms. These approaches leverage newly observed and previously unknown attack samples to continuously update detection models, thereby progressively accumulating attack knowledge. Existing incremental intrusion detection methods can be broadly classified into machine learning-based and deep learning-based approaches.

Machine learning-based incremental intrusion detection methods have been extensively explored with the aim of improving detection efficiency. For instance, Ref. [56] combines SVM with self-organizing incremental neural networks to enhance incremental learning accuracy. However, this approach performs suboptimally on large-scale datasets. Ref. [57] dynamically adjusts relevance vector machines using performance metrics as thresholds to preserve prior knowledge in an integrated incremental intrusion detection model, although this strategy incurs substantial computational overhead during classifier training and maintenance. Refs. [58,59] integrate Hoeffding trees with machine learning algorithms to construct ensemble incremental intrusion detection frameworks; however, their effectiveness is highly sensitive to hyperparameter configurations. Ref. [60] combines multiple machine learning techniques and adaptively updates model parameters to address concept drift during incremental detection, achieving notable performance improvements over single-model approaches. Incremental intrusion detection methods based on clustering algorithms have also attracted considerable attention. These approaches generally demonstrate strong capability in handling nonlinearly correlated data and supporting incremental model updates. For example, Ref. [61] integrates density-based clustering with threshold strategies to achieve accurate intrusion detection and incremental model updates. Ref. [62] proposes an edge-based intrusion detection framework using multi-center incremental clustering, where DBSCAN is employed to process new data and identify unknown intrusions. Ref. [63] presents an online incremental intrusion detection model based on robust clustering that can identify concept drift locations within network flows without requiring labeled data, thereby reducing performance degradation by discarding outdated information. Furthermore, Ref. [64] utilizes semantic embedding-based clustering to uncover potential unknown attacks and updates the classifier using an incremental nearest cluster center strategy. Despite their advantages, clustering-based incremental intrusion detection methods are sensitive to initialization parameters, data distribution, and noise, which may limit their effectiveness in detecting complex attack patterns.

Deep learning-based incremental intrusion detection methods have emerged as a major research focus in recent years. Ref. [65] introduces an incremental intrusion detection approach based on long short-term memory (LSTM) networks, which incrementally learns dynamic changes in network traffic to effectively identify novel attack types. In addition, several hybrid frameworks that combine deep learning with traditional machine learning techniques have been proposed. For example, Ref. [66] presents an incremental intrusion detection framework that integrates open-set recognition with clustering algorithms, consisting of four stages: open-set recognition, clustering or post-training, supervised labeling, and incremental updating. Ref. [67] develops an incremental intrusion detection system that combines a deep neural network with the k-nearest neighbors algorithm, employing a query strategy and a sliding-window mechanism to enhance unknown attack detection while continuously updating the DNN model to adapt to emerging attack patterns. Ref. [68] proposes a DDoS incremental open-set recognition framework that integrates Bi-LSTM and GMM, using GMM to identify unknown traffic and subsequently feeding labeled samples back into both models for incremental learning.

Meanwhile, replay-based incremental learning strategies have also been explored. Ref. [69] implements an incremental intrusion detection system based on a gradient episodic memory algorithm, which stores a subset of samples in auxiliary memory and replays them during training to mitigate forgetting of prior tasks. Ref. [70] employs a super-resolution model to compress and replay original images in conjunction with an anomaly detection module, enabling continuous anomaly detection in environments characterized by sequential task arrival. Overall, deep learning-based incremental intrusion detection methods exhibit the capability to model complex attack behaviors, particularly excelling in the analysis of sequential network traffic. Through incremental learning, these models can continuously assimilate new attack patterns while progressively optimizing detection performance.

In summary, within complex and dynamic open network environments, deep learning-based incremental intrusion detection methods demonstrate significant advantages. Nevertheless, two critical challenges remain. First, catastrophic forgetting of previously learned attack categories persists, causing models to gradually lose detection capability for old attacks during successive incremental updates and leading to substantial performance degradation. Second, existing methods are prone to task-level overfitting when incorporating newly introduced attack types, which hampers the extraction of generalized representations shared across diverse attacks and limits the detection of novel and unknown threats. To address the pronounced accuracy degradation observed after incremental updates, this work proposes a model update strategy based on dual augmentation and fusion knowledge distillation. By leveraging only a limited amount of historical attack data, the proposed approach effectively mitigates catastrophic forgetting and task-level overfitting, thereby substantially improving post-update detection performance in incremental intrusion detection models.

3. Proposed Method

3.1. Overall Architecture

To enhance the incremental update capability of the attack detection model while minimizing the consumption of storage and computational resources during training and updating processes, we present improvements and optimizations to the data augmentation and knowledge distillation algorithms. Specifically, we introduce an incremental intrusion detection method for encrypted traffic based on dual augmentation and fusion knowledge distillation (IID-DAKD). The proposed model combines attack representative sample set augmentation with fusion knowledge distillation to preserve the decision boundaries established in previous tasks. In addition, a self-supervised learning mechanism is employed to extract more generalizable encrypted traffic features, thereby improving incremental detection performance for both known and unknown attacks. The overall architecture of the IID-DAKD model is illustrated in Figure 1.

Figure 1. Architecture of incremental intrusion detection model for encrypted traffic.

As illustrated in Figure 1, the IID-DAKD method integrates representative sample augmentation, class-level self-supervised augmentation, and fusion knowledge distillation to enhance incremental learning for encrypted traffic intrusion detection. The incremental intrusion detection process for encrypted traffic is delineated as follows:

(1) Construction and augmentation of the representative sample set for encrypted traffic: To alleviate storage constraints and reduce classifier bias, the model identifies and preserves a representative sample set for each previously learned attack class in the deep feature space. During the incremental learning phase for new and unknown attacks, Gaussian noise is applied to the representative samples of the existing classes, and these augmented samples are processed by the joint classifier. This strategy not only helps retain classification knowledge of known attacks but also preserves the decision boundaries established in prior tasks. Consequently, it effectively mitigates catastrophic forgetting, enabling continuous learning and incremental optimization. The model stores m representative samples for each learned class, resulting in a total storage complexity of O(mC). Adding Gaussian noise to representative samples is an element-wise linear operation with a complexity of O(md). The augmented samples are trained together with current task samples, slightly increasing the data volume per training epoch, but introducing no additional network parameters and requiring no changes to the model architecture.

(2) Design of attack class augmentation: To promote more generalizable representation learning across various attack classes and to counteract representation bias, a series of class augmentation strategies is employed, including rotational transformations and interpolation techniques applied to the labels of the current task. These techniques generate new augmented samples with diverse class labels, thereby increasing data diversity and expanding the model’s learning perspective. Incorporating these augmented samples into the feature extractor during training exposes the model to a broader range of class information, effectively alleviating task-level overfitting. Rotation and interpolation operations are linear data transformations with a complexity of approximately O(N). If the augmentation ratio is denoted as $\alpha$, the training sample size increases to approximately $\alpha N$, and the overall training time grows nearly linearly: O(αNF). This module does not increase network depth or introduce additional parameters; therefore, the computational complexity during inference remains unchanged.

(3) Development of a fusion knowledge distillation algorithm: To maximize symmetrical knowledge transfer from the previous model, a fusion knowledge distillation strategy is employed that integrates attention-based, relational, and response-based distillation. This approach extends the focus beyond individual sample features to include structural relationships among multiple samples and the model’s logit outputs. By capturing both relational and response-level knowledge, this comprehensive distillation framework reduces the loss of previously acquired knowledge about old classes, thereby enabling the updated model to maintain robust performance while effectively detecting unknown attacks. The fused distillation consists of response distillation, attention distillation, and relation distillation. Response distillation computes KL divergence with a complexity of O(NC). Attention distillation operates on intermediate feature representations with a complexity of O(Nd). Relation distillation computes pairwise relationships within a mini-batch with a complexity of O(B²d). The fused distillation process only introduces additional loss terms and does not add extra trainable parameters.

In practical intrusion detection deployments, unknown attacks are first identified during the detection phase and subsequently labeled by human experts to determine their specific attack types, which are then further utilized for incremental model updates.

3.2. Dual Augmentation Design for Encrypted Traffic

In incremental learning scenarios, deep learning-based encrypted traffic intrusion detection models often suffer from catastrophic forgetting when learning new attack classes, primarily due to representation bias and classifier bias [33]. Although some incremental learning approaches attempt to mitigate this issue by reusing data from prior tasks or employing complex generative models, these methods are frequently limited by memory constraints or computational inefficiencies. Building on the augmentation strategy proposed in Ref. [71], we introduce a dual augmentation strategy for encrypted traffic to simultaneously address both representation bias and classifier bias. This strategy comprises two key components: (1) the construction and augmentation of representative encrypted traffic samples, and (2) attack class augmentation via self-supervised learning. By eliminating the need to store and replay large volumes of historical training data, this approach effectively mitigates catastrophic forgetting in the incremental detection of encrypted malicious attacks.

3.2.1. Construction and Augmentation of the Encrypted Traffic Representative Sample Set

During incremental learning, the encrypted traffic intrusion detection model constructs and maintains a representative sample set for each known attack type. These samples are subsequently augmented during training to reinforce the model’s retention of classification knowledge for known attacks and to preserve the decision boundaries established in previous tasks.

• Construction of an encrypted traffic representative sample set

Given limited storage resources, a small but representative subset of samples is retained for each known category during the incremental learning process. This strategy ensures that the model can effectively leverage these samples to support incremental training while minimizing storage overhead. Although the total storage requirement increases with the number of attack categories, this growth remains manageable, as only a minimal number of representative samples are preserved per category.

For the sample set $X_k$, its class mean ${\mu }_k$ is used to characterize the overall features of class $c_k$. Specifically, ${\mu }_k$ is computed as the sum of the feature representations of all samples in the class, divided by the number of samples. Mathematically, this is expressed as Equation (1).

$${\mu }_k={\displaystyle \frac{1}{N_k}}{\displaystyle {\sum }_{x\in X_k}f(x,\theta )}$$

(1)

where $f(x,\theta )$ denotes the feature representation of sample $x$ and $N_k$ is the total number of samples in attack class $c_k$.

In incremental update tasks, when a new attack type is introduced to the current model, the most representative subset of samples is selected and stored in memory. The selection is based on the distance of samples in the deep feature space: samples closer to the class mean are considered more representative. This mechanism allows new samples to be incorporated sequentially while minimizing the deviation between the mean of the selected samples and the mean of all available samples. Specifically, for each attack class $c_k$, the class mean ${\mu }_k$ is first calculated according to Equation (1). Then, the $m$ most representative samples are sequentially selected from all samples of the class. The procedure for constructing the representative sample set of encrypted traffic based on the class mean is summarized in Algorithm 1.

Algorithm 1. The method for constructing encrypted traffic representative sample set.

Input: the sample set of attack class: $X_k=\{x_1,x_2,\cdots ,x_n\}$, number of class representative samples $m$

Output: class representative sample set $P_k=\{p_1,p_2,\cdots ,p_m\}$

1. Initialize $P_k=\varphi$

2. Calculate the class mean ${\mu }_k$ for $X_k$ according to Formula (1).

3. For each $i\in \{1,2,\cdots ,m\}$ do

4.  $p_k=\underset{x\in X_k}{\arg \min }‖{\mu }_k-{\displaystyle \frac{1}{i}}[f(x,\theta )+{\displaystyle {\sum }_{j=1}^{i-1}f(p_j,\theta )}]‖$

5.  $P_k=P_k\cup p_k$

6. End For

7. Return: Class representative sample set $P_k$

2.

Augmentation of encrypted traffic representative sample set

When learning a new task, the encrypted traffic that enhances memory through noise represents a sample, referred to as ETSAug. That is, Gaussian noise is added to the samples in each class representative sample set, resulting in additional pseudo-label samples. For example, for an encrypted traffic type $c_k$ in the t − 1 phase, this process represents a sample enhancement method as expressed in Equation (2).

$$F_{k,t-1}=p_{k,t-1}+e\cdot r$$

(2)

where $p_{k,t-1}$ denotes the sample in the t − 1 phase type $c_k$, $e\sim (0,1)$ is Gaussian noise, and $r$ is a hyperparameter used to control the scale representing the enhanced uncertainty of the sample. $r$ can be predefined as the mean variance of class representations, as shown in Equation (3).

$$r_t=\sqrt{{\displaystyle \frac{1}{K_{t-1}+K_t}}(K_{t-1}\cdot r_{t-1}^2+{\displaystyle {\sum }_{k=1}^{K_t}\frac{Tr({\mathrm{\Sigma }}_{t,k})}{D}})}$$

(3)

where $K_{t-1}$ and $K_t$ represent the number of categories in the old and new classes at the t-th incremental stage, respectively. $D$ denotes the feature dimension; ${\mathrm{\Sigma }}_{t,k}$ is the covariance matrix of the sample features in the t-th incremental stage for class $c_k$; and $Tr(\cdot )$ calculates the trace of the matrix.

During training with current data, new class attack samples are fed to the joint classifier along with representative sample enhancements from old classes to maintain differentiation and balance between previously learned and new classes. Specifically, at the t-th increment, $D_t=\{X_t,Y_t\}={\{x_{t,j},y_{t,j}\}}_{j=1}^{N_t}$ denotes the dataset of newly detected attacks, where $N_t$ is the number of samples. The model’s training objective is to minimize the prediction loss on the new dataset without degrading, and potentially improving, the loss of previously learned classes:

$$\{{\theta }_t,{\varphi }_t\}=\underset{{\theta }_t,{\varphi }_t}{\arg \min }\{L_t(G(F(X_t;{\theta }_t);{\varphi }_t),Y_t)+{\displaystyle {\sum }_{i=1}^{t-1}L(G(F_i;{\varphi }_t),Y_i)}\}$$

(4)

where ${\theta }_t$ is the feature extractor parameter of the t-stage attack detection model, and ${\varphi }_t$ is the classifier parameter of the t-stage model. $L_t(G(F(X_t;{\theta }_t);{\varphi }_t),Y_t)$ denotes the loss of the stage t-phase model on the new dataset $D_t$, and $F_i$ represents the enhancement of the phase i old class representative sample. $L(G(F_i;{\varphi }_t),Y_i)$ represents the loss of the t-stage model on the enhanced dataset $F_i$, where the old class corresponds to the original sample in the i-th phase.

3.2.2. Network Traffic Class Augmentation Self-Supervised Task Design

Refs. [33,72] have demonstrated the effectiveness of self-supervised learning in the fields of encrypted traffic classification and incremental learning, respectively. On this basis, this subsection uses the self-supervised learning strategy to enrich the current class to ensure that the model can effectively overcome the representation bias in the incremental update process, thereby alleviating the task-level overfitting problem.

• Encrypted traffic self-supervised task design

Traditional deep learning attack detection models usually assume that the number of attack types is fixed and the same during the training and inference phases, which results in the model showing strong memory ability for normal and known attack types, and lacking generalization ability for unknown attack types. To overcome this limitation, inspired by Refs. [33,72], we propose an encrypted traffic self-supervised learning framework, as illustrated in Figure 2. The framework introduces a class augmentation mechanism, termed ETCAug, which expands the training label space by constructing auxiliary classes derived from original encrypted traffic samples.

Figure 2. Encrypted traffic self-supervised learning framework.

Specifically, instead of training the model solely on the original $k$ traffic categories, ETCAug generates additional $m$ helper classes through structured data transformations. These auxiliary classes are incorporated only during the training phase, thereby expanding the optimization objective from a $k$-class classification problem to a $k+m$ class learning task. By exposing the model to a broader spectrum of encrypted traffic variations during training, the proposed framework enables the feature extractor to learn richer and more transferable representations. This augmentation-driven supervision encourages the model to capture fine-grained structural characteristics of encrypted traffic flows, improving inter-class separability and enhancing robustness against distribution shifts. During inference, the classifier is evaluated only on the original $k$ classes, while the auxiliary classes serve exclusively as representation-enhancing supervision. In this way, the framework strengthens the model’s ability to generalize to unknown attack types and provides a more stable feature space for subsequent incremental learning stages.

As shown in Figure 2, in order to effectively learn the initial $k$ types (including 1 normal type and $k-1$ attack type) and $m$ auxiliary classes, the attack detection model uses a classifier capable of processing $k+m$ classes to directly extend the class nodes of the final layer, and performs SoftMax operations on all $k+m$ classes. The output probability that sample $x$ belongs to each category is shown in Equation (5).

$$\begin{array}{l}SoftMa{x}_{k+m}:p(y|x)={\displaystyle \frac{\exp (d_y(x))}{{\displaystyle {\sum }_{j=1}^{k+m}\exp (d_j(x))}}},\\ {\forall }_y\in \{1,\cdots ,k,k+1,\cdots ,k+m\}\end{array}$$

(5)

In this case, the sum of the output probabilities of the $k+m$ categories is 1, i.e., ${\displaystyle {\sum }_{y=1}^{k+m}p(y|x)=1}$. Therefore, ${\displaystyle {\sum }_{y=1}^{k}p(y|x)<1}$, the sum of the output probabilities of the initial $k$ types is less than 1. From the point of view of the classification problem, the assumption of a closed world is broken. Finally, the training objectives of the class augmentation attack detection model are as follows:

$$\underset{\theta ,{\{w_j,b_j\}}_{j=1}^{k+m}}{\min }({\mathbb{Z}}_{(x,y)\sim D_{trn}\cup D_{aug}}[-\log p(y|x)])$$

(6)

where $D_{trn}$ represents the initial K-class training set, $D_{trn}={\{x_i,y_i\}}_{i=1}^{N_1},y_i\in \{1,2,\cdots ,k\}$; $D_{aug}$ represents m auxiliary datasets of the augmentation class, $D_{aug}={\{x_i^{\prime },y_i^{\prime }\}}_{i=1}^{N_2},y_i^{\prime }\in \{k+1,k+2,\cdots ,k+m\}$. $\theta$ represents the feature extractor parameter of the k + m class of joint learning; ${\{w_j,b_j\}}_{j=1}^{k+m}$ is the weight and bias parameter; $p(y|x)$ is defined by Equation (5).

In the training stage, the SoftMax function was applied to the initial k categories and the additional extended m categories at the same time to construct a k + m class classification framework. The attack detection model can learn richer feature representations to enhance the generalization ability of the model and the identification of potential attack patterns. However, in the inference stage, the attack detection model only focuses on the original k categories and limits the application scope of the SoftMax function to these k outputs to ensure the accuracy and reliability of the model output.

2.

Two encrypted traffic class augmentation methods

In order to effectively construct more new classes from a limited number of known types of encrypted traffic samples, two efficient category enhancement strategies have been designed and integrated: encrypted traffic interclass interpolation and encrypted traffic image rotation.

(1) Synthesize encrypted traffic Class with Between-Class Interpolation

The operation process of interpolation to synthesize a new class consists of randomly selecting a sample from two different types of encrypted traffic attacks and using the linear interpolation method to fuse the features of the two samples to generate samples representing a new type. Figure 3 visually illustrates the process of interpolating between classes to generate samples of new types.

Figure 3. Synthesize class with between-class interpolation.

Specifically, set up two real cyber encryption attack types, A and B. A random encrypted traffic sample is selected from type A, and the vector-transformed image sample is labeled as $x_a$, while another encrypted traffic sample is randomly selected from type B, and the vector-transformed image sample is labeled $x_b$. Next, the two samples are linearly interpolated using Equation (7) to synthesize a new sample of a new type, labeled as $x_{ab}^{new}$.

$$x_{ab}^{new}=\lambda \cdot x_a+(1-\lambda )\cdot x_b$$

(7)

where $\lambda$ denotes the random interpolation coefficient, whose value range is constrained to the interval [0, 1]. To ensure that the synthesized samples of the new class are positioned within the low-confidence region between the two base classes, and to reduce the potential overlap between the newly generated enriched class and the original class, the sampling range of $\lambda$ is deliberately restricted to [0.4, 0.6].

(2) Synthesize encrypted traffic class with image rotation

In addition to the method of interpolating between classes to create new classes for encrypted traffic, most traditional data augmentation techniques tend to generate samples that are highly similar to the original data. Therefore, it would be inappropriate to treat these expanded samples simply as entirely new categories. However, studies in Ref. [71] have shown that the robustness of the model can be significantly improved by predicting the type of image rotation. In light of this, this section employs an image rotation strategy to generate image samples that represent the new category.

Specifically, as shown in Figure 4, different rotation versions can be synthesized by applying different angles of rotation (e.g., 90 degrees, 180 degrees, 270 degrees, etc.) to the training data of each known attack category. Each spinning version is treated as a new, separate category and assigned a unique new label, as shown in Equation (8).

$$x_a^{new}=rotate(x_a,\theta ),\theta \in \{90,180,270\}$$

(8)

where $\theta$ denotes the degree of rotation. In this paper, the sample is rotated at 90 degrees, 180 degrees, and 270 degrees to obtain 3 different rotation versions. Next, the enhanced samples of image rotation synthesis are used for class augmentation, and the original k-class classification problem is effectively extended into a new 4k-class classification problem with more classes.

Figure 4. Synthesize encrypted traffic class with image rotation.

3.3. Fusion Knowledge Distillation Method for Encrypted Traffic

Knowledge distillation aims to improve performance by migrating knowledge from the source model to the target model. In incremental learning scenarios, knowledge distillation can help overcome catastrophic forgetting [73]. However, in the field of intrusion detection, although some models use simple response distillation or feature distillation methods, which are effective in maintaining stability against old class attacks, they may limit the plasticity of the model when faced with new, unknown attacks. According to Ref. [74], it is more effective to use the relationship between features as a means of symmetrical knowledge transfer than to learn sample features alone. This method can capture the structural information of encrypted traffic data, thereby improving the performance and generalization ability of the model. In order to make the new model more effective in transferring the classification knowledge of known attacks from the old model, a fusion knowledge distillation method (FKD-ARR) is designed in this paper that combines the attention mechanism, sample relationship, and network response. As shown in Figure 5, the framework of FKD-ARR can effectively transfer knowledge from the logit output of attention diagrams, structural information, and models, thereby enhancing the effect of knowledge distillation and model performance.

Figure 5. Structure of fusion knowledge distillation method (FKD-ARR).

3.3.1. Attention-Based Knowledge Distillation for Encrypted Traffic

In the network attack detection model, the convolutional layer can extract rich feature information from encrypted traffic samples for subsequent classification and detection tasks. This feature information is usually represented in the form of a feature map, such as $A\in {ℝ}^{C,H,W}$, where C, H, and W represent the number of channels, height, and width, respectively. This section combines the spatial attention map and the channel attention map to form a hybrid attention mechanism, which helps the student model pay attention to the spatial dimension and the channel dimension of the feature map at the same time, and then captures the importance of different regions, locations and features in the input data, so as to improve the feature capture ability of the front-end convolutional layer for the gray map of encrypted traffic.

• Spatial Attention graph and channel attention graph

In the attack detection model, the spatial attention diagram can guide the student model to focus on the specific area or location in the network traffic that may hide the attack behavior, and then improve the detection ability of the student model to abnormal traffic or attack behavior. The spatial attention map is seen as a mapping function that maps different regions or locations in the input data to attention weights, highlighting specific areas that are closely associated with the attack behavior while suppressing other areas that are less correlated. Construct a spatial attention graph $G^s$ by summing the absolute values in each channel dimension, as in Equation (9).

$$G^s(A)={\displaystyle \frac{1}{C}}{\displaystyle {\sum }_C^{k=1}\left|A_{k,\cdot ,\cdot }\right|}$$

(9)

where k represents the k-th channel slice of A in the channel dimension.

In the attack detection model, the channel attention diagram can help the student model to understand and pay attention to the specific characteristics that are decisive for distinguishing between normal traffic and attack traffic, and then improve the model’s ability to detect abnormal traffic or attack behavior. The channel attention map can also be thought of as a mapping function that maps different features in the input data to attention weights, thereby highlighting features that are closely related to attack behavior while suppressing other features that are less correlated. $G^c$ is constructed by summing the absolute values of the width and height dimensions, as in Equation (10).

$$G^c(A)={\displaystyle \frac{1}{HW}}{\displaystyle {\sum }_H^{i=1}{\displaystyle {\sum }_W^{j=1}\left|A_{\cdot ,i,j}\right|}}$$

(10)

where i and j represent the tile indexes of A in the height dimension and width dimension, respectively.

2.

Loss of attention distillation

The combination of spatial and channel attention distillation allows the student model to focus on both the spatial position and channel features in the input data, so as to extract and understand the feature information of the input data more comprehensively. Attention distillation loss $L_{akd}$ consists of two parts: attention transfer loss $L_{at}$ and attention mask loss $L_{am}$. namely

$$L_{akd}=L_{at}+L_{am}$$

(11)

Among them, $L_{at}$ is used to encourage the student model to mimic the attention of the teacher model in the spatial and channel dimensions, and the loss is calculated by comparing the difference between the attention weights of the student model and the teacher model in each channel or position to measure the difference between the two. $L_{at}$ is calculated as shown in Equation (12). $L_{am}$ is used to encourage the student model to mimic the features of the teacher model by the spatial attention mask $M^s$ and the channel attention mask $M^c$, calculating the loss by comparing the difference between the predicted mask and the true mask. $L_{am}$ is calculated as shown in Equation (13).

$$L_{at}=L_2(G^s(A^s),G^s(A^t))+L_2(G^c(A^s),G^c(A^t))$$

(12)

$$L_{am}={({\displaystyle {\sum }_{k=1}^C{\displaystyle {\sum }_{i=1}^H{\displaystyle {\sum }_{j=1}^W(A_{k,i,j}^t-}}}{A}_{k,i,j}^s)}^2\cdot M_{i,j}^s\cdot M_k^c{)}^{1/2}$$

(13)

where $M^s$ and $M^c$ are the embodiment of two core attention mechanisms, which focus on the spatial position of the feature map and the importance of channel features, respectively. They are calculated by summing the attention maps from the teacher and student detectors, as shown in Equations (14) and (15).

$$M^s=HW\cdot SoftMax((G^s(A^s)+G^s(A^t))/Te)$$

(14)

$$M^c=C\cdot SoftMax((G^c(A^s)+G^c(A^t))/Te)$$

(15)

where superscripts $s$ and $t$ are used to distinguish between the student model and the teacher model. $Te$ is a hyperparameter-temperature introduced into the SoftMax function to adjust the distribution of each element in the attention mask.

3.3.2. Sample Relational-Based Knowledge Distillation for Encrypted Traffic

In classification tasks, information about the relationships between different objects is essential for knowledge extraction [74]. However, knowledge extraction based on the relationship between samples has not been fully explored in attack detection tasks. In order to dig deeper into the knowledge of the implicit relationship between encrypted traffic and enable the student network to better inherit the structural information and high-level semantics of the teacher network, a knowledge distillation method based on the sample relationship is introduced in this section. This method focuses on the interrelationship between the samples of the transferred encrypted traffic and punishes the structural differences in the relationship by introducing the distance-based second-order distillation loss and the angle-based third-order distillation loss, which enriches the symmetrical knowledge transfer of binary distance and ternary angle in the sample relationship, so as to provide better guidance for the student model.

• Distance relationship distillation

The distance between the two pairs of m encrypted traffic samples in each batch is calculated to obtain a distance matrix of size m × m, which is output as a relational structure. By learning the structural output of the teacher model, the student model realizes the distillation learning of the distance relationship between samples. Specifically, for any pair of training samples $t_i$ and $t_j$, the distance function ${\psi }_D$ is used to calculate the Euclidean distance between the two samples in the feature space, as shown in Equations (16) and (17).

$${\psi }_D(t_i,t_j)={\displaystyle \frac{1}{\mu }}{‖t_i-t_j‖}_2$$

(16)

$$\mu ={\displaystyle \frac{1}{\left|X^2\right|}}{{\displaystyle {\sum }_{(x_i,x_j)\in X^2}‖t_i-t_j‖}}_2$$

(17)

where $\mu$ is the normalization factor for distance, set to the average distance between all sample pairs from $X^2$ in the batch.

Distance distillation loss is used to match the difference between the current model and the previous model in terms of the distance between two samples. The distance relationship between samples for distillation loss is defined as follows:

$$L_{RKD-D}={\displaystyle {\sum }_{(x_i,x_j)\in X^2}{l}_{\delta }}({\psi }_D(t_i,t_j),{\psi }_D(s_i,s_j))$$

(18)

where $\left(x_i,x_j\right)$ is the selected binary in a batch, $(t_i,t_j)$ is its output in the old model, and $(s_i,s_j)$ is its output in the new model. The loss function $l_{\delta }(\cdot ,\cdot )$ uses the Huber loss calculation method, which is defined in Equation (19).

$$l_{\delta }(x,y)=\left\{\begin{array}{l}{\displaystyle \frac{1}{2}}{(x-y)}^2,for\left|x-y\right|\le 1\\ \left|x-y\right|-{\displaystyle \frac{1}{2}},otherwise\end{array}\right.$$

(19)

2.

Angular relationship distillation

A series of triples was constructed from m samples in each batch, two direction vectors were constructed for each triplet and the angle between the two vectors was calculated. An angular matrix of size m × m × m is generated and used as the output of a relational structure. By learning the structural output of the teacher model, the student model realizes the distillation learning of the angular relationship between samples.

Specifically, for three samples ($t_i,t_j,t_k$) of any group, two lines are constructed by connecting these three points, and then two direction vectors are obtained. The angle function is used to calculate ${\psi }_A$ angle formed by three samples in the feature space, which reflects the relative position of the three samples in the feature space. The calculation is shown in Equation (20).

$${\psi }_A(t_i,t_j,t_k)=\cos \angle t_i{t}_j{t}_k=⟨e^{ij},e^{kj}⟩$$

(20)

$$e^{ij}={\displaystyle \frac{t_i-t_j}{{‖t_i-t_j‖}_2}}$$

(21)

$$e^{kj}={\displaystyle \frac{t_k-t_j}{{‖t_k-t_j‖}_2}}$$

(22)

where $e^{ij}$ is the direction vector of $t_i$ and $t_j$, as in Equation (21); $e^{kj}$ is the direction vector of $t_j$ and $t_k$, as in Equation (22). ${\psi }_A$ is the angle between the two direction vectors $e^{ij}$ and $e^{kj}$.

Angular distillation losses are used to match the differences between the current and previous models in terms of angles between the three samples. The ante-sample angular relationship distillation loss is defined as follows:

$$L_{RKD-A}={\displaystyle {\sum }_{(x_i,x_j,x_k)\in X^3}{l}_{\delta }}({\psi }_A(t_i,t_j,t_k),{\psi }_A(s_i,s_j,s_k))$$

(23)

3.3.3. Network Response-Based Knowledge Distillation for Encrypted Traffic

Given the input encrypted traffic sample $x$, the output logit value obtained by the attack detection model on category $c_k$ is defined as $l_k(x)$, then sample $x$ is predicted as the probability distribution $p_k\in {ℝ}^M$ belonging to each class can be calculated by Equation (24):

$$p_k(x;\theta ,\tau )={\displaystyle \frac{\exp (l_k(x)/\tau )}{{\displaystyle {\sum }_{k=1}^K\exp (l_k(x)/\tau )}}}$$

(24)

where $\theta$ is the network parameter of the model, $k$ is the index of the attack class, and $\tau$ is the temperature factor used to adjust the softening degree of the output probability distribution.

Response knowledge distillation achieves efficient knowledge transfer by minimizing the KL (Kullback–Leibler) divergence bias of the prediction distribution between the old and new models. The response knowledge distillation loss is as follows:

$$L_{t,kd}=-{\tau }^2{{\displaystyle {\sum }_{x\in X}{\displaystyle {\sum }_{k=1}^{K_{t-1}}p}}}_k(x;{\theta }_{t-1},\tau )\log (p_k(x;{\theta }_t,\tau ))$$

(25)

where ${\theta }_{t-1}$ and ${\theta }_t$ represent the parameters of the old model and the current model, respectively. $X$ represents the input data for a batch. K_t−1 represents the number of classes that have already been learned in the old model. $p_k(x;{\theta }_{t-1},\tau )$ and $p_k(x;{\theta }_t,\tau )$ represent the predicted probability distribution of sample $x$ for the k-th category under the old model and the current model, respectively.

3.4. Incremental Updates to the Attack Detection Model

The detection model incremental update problem can be understood as the detection sample $X_t$ of a new unknown attack detected and collected at stage t as part of the training set. Given the existing detection model $f_{t-1}(x,{\theta }_{t-1},{\phi }_{t-1})$, a new detection model $f_t(x,{\theta }_t,{\phi }_t)$ is trained using the combination of the class-enhanced sample set of $X_t$ and the enhanced sample set of the representative sample set of the old class $P_t$.

3.4.1. Incremental Update of Representative Sample Set

During the operation of the attack detection model, new unknown attack samples will be continuously detected. After the model is trained based on the current training data, it needs to construct a representative sample set for the new attack type and update the overall representative sample set. Specifically, for a new attack type, the representative sample set of the type is first constructed by Algorithm 1, and then the overall representative sample set is updated incrementally by Algorithm 2; that is, the newly constructed class representative sample set is integrated into the existing overall representative sample set.

Algorithm 2. Representative sample set incremental update method.

Input: Representative sample set of old known class encrypted traffic $P_{t-1}=\left\{P_k\right|P_k=\{p_1^k,p_2^k,\cdots ,p_m^k\},1\le k\le N_{t-1}\}$

New class encrypted traffic sample set $X_t=\{X_k,y_k\},N_{t-1}+1\le k\le N_{t-1}+N_{new}\}$

Current attack detection model: $f(x,{\theta }_t)$

Output: Representative sample set after incremental update $P_t$

Initialize $P_t=P_{t-1}$   For each $k\in (N_{t-1}+1,N_{t-1}+N_{new})$ do     Calculate the representative sample set $P_k$ of the new attack class $X_k$ based on Algorithm 1.     Update the saved representative sample set $P_t\leftarrow P_t\cup P_k$   End For   Return: $P_t$

In Algorithm 2, the number of known attack types before the incremental update is $N_{t-1}$, and the total number of attack types after the incremental update is $N_{t-1}+N_{new}$. In the third step, a representative sample set is calculated for each emerging and previously unknown attack type. Step 4: The representative sample set of the old attack type is merged with the representative sample set of the new class, and the merged sample set is the incrementally updated overall class representative sample set.

3.4.2. Model Incremental Learning Loss Function Construction

The design of the loss function is a very important part of deep learning, and different loss functions need to be selected for different tasks. For the multi-classification task of encrypted traffic attack detection, the commonly used cross-entropy discriminant loss is used as the basic loss function. However, in order to achieve the goal of incremental learning, representative sample enhancement loss and fusion knowledge distillation loss are further introduced. The combination of these loss functions aims to ensure that the attack detection model absorbs the knowledge of new types of attacks while not forgetting the knowledge of old types of attacks, so as to achieve effective transfer and accumulation of knowledge. The overall incremental training loss function is shown in Equation (26).

$$L_{t,total}=L_{t,cb}+{\lambda }_{ra}\cdot L_{t,repAug}+{\lambda }_{akd}\cdot L_{t,akd}+{\lambda }_{rkd}\cdot L_{t,rkd}+{\lambda }_{kd}\cdot L_{t,kd}$$

(26)

where $L_{t,cb}$ represents the discriminant loss of the model in learning a new class task at the current stage. $L_{t,regAug}$ represents the augmentation loss of the representative sample of the old category attack, and ${\lambda }_{ra}$ is the weight parameter that adjusts the augmentation loss of the representative sample. $L_{t,akd}$ is the attention distillation loss, and ${\lambda }_{akd}$ is the weight parameter for adjusting the attention distillation loss. $L_{t,rkd}$ is the relational distillation loss, and ${\lambda }_{rkd}$ is the weight parameter to adjust the relational distillation loss. $L_{t,kd}$ represents the response knowledge distillation loss, and ${\lambda }_{kd}$ is the weight parameter that adjusts the response knowledge distillation loss.

Considering the imbalance of the intrusion detection dataset, $L_{t,cb}$ uses the class equilibrium cross-entropy loss function to improve the model’s attention to tail class samples, as shown in Equation (27). $L_{t,regAug}$ is shown in Equation (28).

$$L_{t,cb}=-{\displaystyle \frac{1-\beta }{1-{\beta }^n}}\log \left({\displaystyle \frac{\exp \left(\frac{l_i(x)}{\tau }\right)}{{\displaystyle {\sum }_{k=1}^K\exp \left(\frac{l_k(x)}{\tau }\right)}}}\right)$$

(27)

$$L_{t,regAug}={\displaystyle {\sum }_{i=1}^{t-1}{L}_{t,cb}(G(F_i;{\varphi }_t),Y_i)}$$

(28)

where $L_{t,rkd}$ combines distance-related distillation loss $L_{t,rkd-d}$ and angle-related distillation loss $L_{t,rkd-a}$, as shown in Equation (29). Among them, $L_{t,rkd-d}$ focuses on the relative distance relationship between samples, while $L_{t,rkd-a}$ focuses on the preservation of the angular relationship between samples.

$$L_{t,rkd}=L_{t,rkd-d}+L_{t,rkd-a}$$

(29)

In the process of incremental update of the IID-DAKD model, the five knowledge synergies of the old model are fully utilized to guide the new model. Through this collaborative guidance mechanism, the new model can not only effectively transfer the feature information of a single sample, but also deeply learn the relational information between samples and the direct information contained in the output of the old model, so that the new model can better fit the learning ability of the old model.

3.4.3. Attack Detection Model Training and Parameter Update

For the training of the attack detection model, the method in this paper adopts an incremental learning strategy, which allows the model to process different classes in batches. When new types of unknown attack data emerge, the model performs an update to accommodate these new data categories. In order to understand the knowledge transfer mechanism between the old attack detection model and the new attack detection model more clearly, Algorithm 3 describes in detail the incremental update training steps of the attack detection model in the incremental stage t. In this method, the parameters of the model are adjusted, and the representative sample set is updated by using the enhanced set of the old type of encrypted traffic representative samples that have been saved and the enriched set of the newly identified new type of encrypted traffic attack samples.

Algorithm 3. Model incremental update method based on dual augmentation and fusion knowledge distillation.

Input: Representative sample set of old known class encrypted traffic $P_{t-1}=\left\{P_k\right|P_k=\{p_1^k,p_2^k,\cdots ,p_m^k\},1\le k\le N_{t-1}\}$,

New class encrypted traffic sample set $X_t=\{X_k,y_k\},N_{t-1}+1\le k\le N_{t-1}+N_{new}\}$,

Feature extractor parameters before incremental updates ${\theta }_{s,t-1}$,

Corresponds to the old class output layer parameters ${\theta }_{o,t-1}$

Output: Representative sample set before incremental update $P_t$,

Feature extractor parameters ${\theta }_{s,t}$,

Corresponds to the old class output layer parameters ${\theta }_{o,t}$, Corresponds to the new class output layer parameters ${\theta }_{n,t}$

Initialize incremental learning training set $X_{train}=\varphi$     Initialize the network model parameters ${\theta }_{s,t}={\theta }_{s,t-1}$ ${\theta }_{o,t}={\theta }_{o,t-1}$, ${\theta }_{n,t}=RandInit(\left|{\theta }_{n,t}\right|)$    $X_{t,aug}={\{X_k^{\prime },y_k\}}_{k=N_{t-1}+1}^{N_{t-1}+N_{new}+N_{aug}}$     Update $X_{train}\leftarrow X_t\cup X_{t,aug}$    $P_{t-1,aug}={\{P_k^{\prime },y_k\}}_{k=1}^{N_{t-1}}$     Update $X_{train}\leftarrow X_{train}\cup P_{t-1,aug}$     If t = 1 then      train the model parameters ${\theta }_{s,t}$, ${\theta }_{o,t}$ and ${\theta }_{n,t}$ with the loss function $L_{t,ce}$     Else      train the model parameters ${\theta }_{s,t}$, ${\theta }_{o,t}$ and ${\theta }_{n,t}$ with the loss function $L_{t,total}$     Calculate the representative sample set for each class in $X_t$ based on Algorithm 1     Update the representative sample set $P_t$ based on Algorithm 2     Return: the representative sample set $P_t$, parameters ${\theta }_{s,t}$, ${\theta }_{o,t}$ and ${\theta }_{n,t}$

In Algorithm 3, steps 1 and 2 initialize the parameters of the training dataset and model. Steps 3 and 4 perform class enhancement processing on the new type of attack samples, effectively expanding and integrating them into the training data set, so as to enhance the model’s ability to identify new unknown attacks. Steps 5 and 6 augmented the representative sample sets of the old types of attacks and added them to the training dataset to further enhance the detection performance of the model against known attack types. Steps 7 to 10 use the enhanced training set to train the model and adjust the model parameters ${\theta }_{s,t}$, ${\theta }_{o,t}$ and ${\theta }_{n,t}$ to ensure that the model can accurately identify and handle various attack types. Step 11 and Step 12 update the representative sample set $P_t$ based on the training results, which helps to keep the sample set up-to-date and representative, and ensures that the model can still quickly adapt and make accurate judgments in the face of new attacks in the future.

4. Experiments and Analysis

This section presents a comprehensive evaluation of our proposed model’s performance in network-encrypted traffic attacks detection. We begin by providing a thorough description of the datasets used and the evaluation metrics utilized for our experiments. Subsequently, we also conducted ablation experiments to evaluate the effectiveness of the self-supervised task design method and the knowledge distillation-based fusion design approach, as well as their impact on the IID-DAKD method.

4.1. Dataset and Metrics

In order to verify the effectiveness of the incremental intrusion detection method for encrypted traffic proposed in this paper, MCFP and CICIDS2017 were used as the benchmark datasets. Among them, MCFP is integrated by three public data sources, Stratosphere IPS [75], MTA [76], and CTU-13 [77], and the attack samples selected the encrypted traffic capture of 9 kinds of malware, such as Zeus and Hancitor. CICIDS2017 [78] is a real network traffic dataset with encrypted traffic, finally determines 10 representative categories for experiments, in which the Web Attack label data is composed of Web Attack-SQL Injection, Web Attack-Brute Force, and Web Attack-XSS data. Nine attack traffic captures, such as DoS-Hulk and DDoS, were selected for the attack samples. Regarding the preprocessing methods for the two datasets, for MCFP, the corresponding traffic PCAP files are downloaded, and the four-tuple connections and key features are extracted. For CICIDS2017, the directly available CSV data files are downloaded first, followed by cleaning and transformation operations, and then the encrypted traffic samples are converted into grayscale images. Both datasets cover normal traffic and nine different types of attack traffic, and this paper reorganizes the datasets according to specific settings, and the specific sample distribution of each category of the two datasets is shown in Table 2.

Table 2. The quantitative information of experimental data sets.

No.	Types	MCFP		CICIDS2017
		Label	Session Number	Label	Session Number
1	Known normal	Benign	186,332	Benign	841,138
2	Known attacks	Zeus	19,010	DoS-Hulk	14,108
3		Hancitor	16,584	DDoS	45,392
4		Neris	19,481	DoS-GoldenEye	7458
5	Unknown attacks	Dridex	39,555	DoS-slowloris	3873
6		RBot	31,527	DoS-Slowhttptest	4216
7		HTBot	9380	Web Attack	2000
8		Wannacry	15,938	SSH-Patator	2979
9		Miuref	27,819	PortScan	158,676
10		Sathurbot	29,964	FTP-Patator	3991

Specifically, the normal traffic in the same dataset and the randomly selected 3 types of attack traffic together form 4 types of known old class samples, while the remaining 6 types of attack traffic are set as new unknown attacks. In order to simulate the scenario in which new unknown attacks continue to arrive and the detection model learns incrementally in the real open network environment, the training set composed of traffic samples of the known category is first used as the initial training set to train the initial attack detection model. Then, in each incremental learning stage, new class attack samples are added sequentially, and incremental learning of new class attack samples is realized on the trained model.

We evaluate the advantages and disadvantages of the algorithm based on the detection effect of the data set in the experimental results. Therefore, this experiment employs commonly used machine learning evaluation indices such as Accuracy, Precision, Recall and F1_score to measure the detection performance of the incremental intrusion detection model. The calculation formula is as follows:

$$Accuracy=\left(TP+TN\right)/\left(TP+TN+FN+FP\right)$$

(30)

$$Precision=TP/\left(TP+FP\right)$$

(31)

$$Recall=TP/\left(TP+FN\right)$$

(32)

$$F_1\_Score=2\cdot Precision\cdot Recall/\left(Precision+Recall\right)$$

(33)

where TP represents the number of samples correctly identified as attack traffic by the detection model; FP represents the number of normal traffic samples incorrectly identified as attack traffic by the detection model; TN represents the number of samples correctly identified as normal traffic by the detection model; FN represents the number of normal traffic samples incorrectly identified by the detection model.

4.2. Comparative Experiment of Detection Effect Based on the IID-DAKD Method

This section visually demonstrates the feature space distribution learned by the proposed incremental intrusion detection model (IID-DAKD) on network-encrypted traffic data sets before and after updating. Then, the incremental attack detection experiments across two datasets are designed in both intra-domain and inter-domain scenarios to further validate the strong adaptability of the model in various incremental attack scenarios.

4.2.1. Visual Analysis

In order to visually demonstrate the improvement effect of the IID-DAKD method on catastrophic forgetting of old tasks when fitting new tasks, the t-SNE method was used to visualize the data distribution in the feature space. As shown in Figure 6, points of different colors represent different types of data. In this experiment, four classes of encrypted traffic data from the MCFP dataset were used as the initial categories to train the model. Subsequently, two new encrypted traffic categories were added as incremental tasks on the trained model. Figure 6a shows the distribution of the initial four classes of data learned by the model before incremental update, while Figure 6b shows the redistribution of these four classes of data learned by the model after incremental update. Furthermore, Figure 6c shows the comprehensive distribution of all six classes of data learned by the model after incremental update (including the initial four classes and the two newly added classes) in the feature space. In the figure, class0 to class3 represent the initial four classes of data, while class4 and class5 represent the subsequently added incremental categories.

Figure 6. The learned representations of IID-DAKD before and after incremental update.

From the visualization results of Figure 6a,b, it can be seen that whether before or after incremental updates, the classification performance of the attack detection model on the initial dataset remains consistent. This means that when updating model parameters, the model successfully retains memory of known data, ensuring stable detection capabilities when facing attacks from known categories. Figure 6c also shows that the new model has clearly defined classification boundaries for initial type attacks and new attacks in the feature space. This indicates that through incremental updates, the model not only retains memory of knowledge of old category attacks, but also effectively learns the classification knowledge of new category attacks, enabling it to adapt and accurately identify new unknown attack types. This has important practical application value for the constantly evolving network threat environment in the real world.

4.2.2. Verification Experiment of Incremental Intrusion Detection Based on IID-DAKD

In this experiment, two new attack categories are gradually introduced into the model in each incremental phase as incremental tasks to simulate the scenario where new unknown attacks gradually appear in a real environment. Table 3 shows the classification performance of the IID-DAKD method against initial known attack types in two datasets. On the MCFP dataset, the overall attack detection accuracy of the model reached 99.52%, demonstrating excellent performance. The detection accuracy for the Zeus and Neris attack types reached 100%, while for the Benign and Hancitor attack types, the detection accuracy was also as high as 99.13% and 98.93% respectively. On the CICIDS2017 dataset, the model also performed well, with an overall attack detection accuracy of 99.12%. The detection accuracy for the Benign, Dos-Hulk, Web Attack, and BruteForce-FTP attack types were 99.02%, 99.70%, 98.85%, and 99.18% respectively. Since new attack categories were not introduced in the initial phase, these results indicate that the IID-DAKD model has excellent accuracy and reliability in traditional closed-set attack detection scenarios.

Table 3. Detection performance of IID-DAKD on base classes of encrypted attack.

Dataset	Types	Pre	Rec	F1	Acc
MCFP	Benign	99.13	98.94	99.03	99.52
	Zeus	100	100	100
	Hancitor	98.93	99.13	99.03
	Neris	100	100	100
CICIDS2017	Benign	99.02	99.90	99.46	99.12
	Dos-Hulk	99.70	96.80	98.22
	DDoS	98.85	95.35	97.07
	DoS-GoldenEye	99.18	99.51	99.34

Next, incremental learning of new class attack samples is implemented on the attack detection model. After completing the incremental update of the model, bar charts are plotted for the detection performance of each type of attack in each incremental learning stage, as shown in Figure 7. Specifically, Figure 7a shows the detection accuracy of the model for initial type attacks during the incremental learning process. Figure 7b focuses on the detection accuracy of the model when facing new type attacks. The horizontal axis in the figure represents different incremental stages, while the vertical axis represents the detection accuracy of each type of attack. In particular, stage 1 is the initial stage, reflecting the state of the model after training only on initial known type data. Subsequent stages 2, 3, and 4 represent the performance changes after gradually introducing 2 new types of attacks into the model each time. For example, in stage 2, the model learns the new attacks Dridex and RBot; in stage 3, the model further includes HTBot and Wannacry attack types; and in stage 4, the model continues to expand its recognition capabilities, including the new attack types Miuref and Sathurbot.

Figure 7. Precision of each incremental detection stage on MCFP.

The experimental results shown in Figure 7 indicate that in stage 1, the initial stage, the model exhibited high detection accuracy for normal traffic and initial type attacks, demonstrating good initial performance. Moving into stage 2, the model was further trained on 2 additional types of attack data on top of the initial training. At this point, there was a certain degree of decrease in detection accuracy for old-type attacks, reflecting the impact of the additional data on the model’s classification performance. With the continuous addition and training of more new attack type samples in subsequent stages, the recognition accuracy for old type attacks showed a gradual decrease. This is mainly because the incremental training of new attack data inevitably leads to the model forgetting the classification knowledge of old type attacks, resulting in a decrease in its classification accuracy. However, it is worth noting that despite this forgetting phenomenon, the model is still able to maintain a certain level of detection capability for old-type attacks. From Figure 7, it is also evident that after three stages of incremental training, the model can effectively identify new attacks and maintain high detection accuracy, while still maintaining relatively high detection accuracy for old-type attacks. This strongly demonstrates the effectiveness and practicality of the incremental update method for detection models, indicating that this incremental update method allows detection models to continuously learn and adapt to new attack types in a constantly changing network environment, while maintaining the ability to recognize known attack types.

4.2.3. Comparative Experiment of Intra-Domain and Inter-Domain Incremental Attack Detection Tasks

In the real world, new types of attacks and basic types of attacks do not always come from the same domain. In this example, tasks in which both the old type of attack and the new type of incremental update attack come from the same dataset are defined as intra-domain incremental attack detection, while the task of incremental update and attack detection on different datasets is defined as cross-domain incremental attack detection. This experiment focuses on verifying the classification performance of IID-DAKD and other comparison methods in the face of incremental tasks from the same or across domains.

Figure 8 shows the performance comparison and decreasing trend between the IID-DAKD method proposed in this paper and several comparison methods, including EWC, CIL-A-NIDS [69], CL-based A-NIDS, and SPCIL, in the same domain incremental attack detection task. Specifically, Figure 8a,b reflect the performance of the domain incremental intrusion detection task executed on the MCFP and CICIDS2017 datasets, respectively. In the figure, the horizontal axis represents different incremental stages, while the vertical axis represents the overall accuracy on the test set. According to the experimental results in Figure 8, it can be observed that in the domain incremental intrusion detection tasks based on the two datasets, IID-DAKD demonstrates certain advantages at each incremental intrusion detection stage, achieving superior performance compared to other comparison algorithms. Notably, after the third stage of incremental updates, the overall accuracy of IID-DAKD remains above 92%, which fully validates its effectiveness and robustness.

Figure 8. Comparison of the detection performance of incremental attacks in the same domain.

Figure 9 shows the average detection accuracy of newly added types of attacks at each incremental stage in the same-domain incremental intrusion detection task on the MCFP dataset. From Figure 9, it can be seen that IID-DAKD has a higher detection accuracy for newly added types of attacks at each stage compared to the methods in CIL-A-NIDS, CL-based A-NIDS, and SPCIL methods, and is slightly higher than the EWC method. Combining the experimental results from Figure 8 and Figure 9, it can be concluded that the EWC method demonstrates a high detection capability when the model faces newly added types of attacks; however, its overall attack detection accuracy is the lowest. This is because the core advantage of this method lies in not needing to store old data, which effectively alleviates the pressure of data storage and privacy protection. However, during the incremental learning process, it is prone to causing the model to catastrophically forget the classification knowledge of previously learned old attack types, leading to a decline in the ability to recognize old types of attacks, thereby affecting the overall accuracy.

Figure 9. Detection accuracy of new types of attacks in each incremental stage on MCFP.

Figure 10 presents the performance differences and trends of the proposed IID-DAKD method compared to several baseline methods at each incremental stage in the cross-domain incremental intrusion detection task. In this experiment, four types of data from the MCFP dataset were used as the initial base class, while attack type data from the CICIDS2017 dataset served as the data for subsequent incremental tasks. According to the experimental results in Figure 10, it can be observed that when facing cross-domain incremental attacks, IID-DAKD demonstrates superior stability and detection performance compared to other methods, with a more gradual decline trend. Notably, after the incremental update in the third stage, the overall accuracy of this method still maintains around 90%. This indicates that the method can adapt well to the differences between different domains and has good generalization performance.

Figure 10. Comparison of the overall accuracy of cross-domain incremental attack detection at each stage.

Combining the experimental results from Figure 9 and Figure 10, it can be seen that IID-DAKD not only performs excellently in incremental updates and attack detection in the same domain for incremental intrusion detection, but also adapts well to the differences that exist between different domains, demonstrating outstanding generalization performance. This is attributed to IID-DAKD enhancing the model’s generalization ability and alleviating catastrophic forgetting of old attack classification knowledge through the application of category-enhanced self-supervision and mixed knowledge distillation. In Phase 0, since there have been no incremental updates yet, all algorithms have the same overall accuracy in the intrusion detection task. From Phase 1 to Phase 3, as the number of incremental updates to the intrusion detection model increases, the phenomenon of forgetting old classification knowledge gradually becomes apparent during the incremental training process. This forgetting leads to a decrease in accuracy when the model identifies a high proportion of old attack types and normal traffic, affecting the overall accuracy. By comparing with other methods, it was found that EWC has the lowest overall accuracy because it does not store old data, which can easily lead to catastrophic forgetting of knowledge about old attack types, reducing the ability to recognize old class attacks and thus affecting overall accuracy. The SPCIL method is somewhat competitive as it alleviates the forgetting of old class knowledge by storing a small number of samples in a cache. However, as the number of increments continues to increase, the limited capacity of the cache gradually reduces the number of samples that can be retained for each old class, which may lead to gradual forgetting of old class knowledge. The CL-based A-NIDS retains a class-balanced memory bank to balance the class distribution in continual learning, but when the number of samples is limited, it can lead to a decline in generalization ability for new classes. The method CIL-A-NIDS uses the GEM algorithm, which relies on a small-scale exemplar set, but due to the constraints of multiple gradient information during the optimization process, it is prone to overfitting issues, resulting in poor performance in practical incremental intrusion detection.

The IID-DAKD method proposed in this paper uses a simple technique of class augmentation, allowing the model to encounter more classes during the training process, thereby achieving effective transfer learning of inter-class representations. This enables the model to better adapt to incremental samples of different attack types and demonstrates a greater competitive advantage in complex incremental learning tasks.

4.2.4. Comparative Experiment of Incremental Intrusion Detection Model Under Different Number of Unknown Attacks

In the previous experiments, it was assumed that each incremental update introduced two new types of attacks. In reality, the update frequency can be adjusted according to actual needs, meaning the model can be updated at a higher or lower frequency. To systematically evaluate the impact of different update frequencies on the performance of the detection model, this section explores a strategy of performing a unified incremental update after discovering one or more new types of attacks, and presents the experimental results based on the MCFP dataset in Figure 11.

Figure 11. Comparison of accuracy when the update includes different numbers of unknown attack categories.

Among them, Figure 11a depicts a scenario where each incremental update contains only one new type of attack, requiring six independent incremental training sessions to fully adapt to six new types of attacks; while Figure 11b shows the case where each incremental update includes three new types of attacks, allowing for the adaptation to all new types of attacks to be completed in just two incremental training sessions.

By comparing and analyzing Figure 11a,b with the previous Figure 8a, a clear trend can be observed: increasing the number of new types of attacks in a single incremental update can effectively improve the accuracy of the intrusion detection model. This is because, under the premise that the initial types of attacks remain consistent and the total number of new attacks does not change, the number of model updates is reduced. Each update is accompanied by forgetting known attacks, so reducing the update frequency helps to mitigate the cumulative effect of this knowledge forgetting, thereby enhancing the overall detection performance of the model. However, the increase in the number of new types of attacks introduced in each incremental update also means that the model’s update frequency decreases, which in turn slows down the response speed to new threats. Therefore, when formulating the incremental update strategy for the detection model, it is important to consider both the model’s detection performance and its response speed to new threats.

4.2.5. Comparative Experiments in Different Base Class of Incremental Attack Scenarios

The above experiments all use the normal traffic in the data and the randomly selected three types of attack data as the initial category data. In order to comprehensively evaluate the performance of the detection model under different initial class numbers, three incremental attack scenarios are designed for experiments. These scenarios are based on different initial known categories and are designed to simulate a diverse threat landscape in a real-world cyber environment. In Scenario 1, four categories of normal traffic and randomly selected three types of attack data are set as known categories, and the remaining six attack categories are used as new categories. In the incremental learning process, two new categories are introduced in each stage, and a total of three incremental updates are performed. Experimental Scenario 2 increases the number of initial categories, including normal traffic and randomly selected five types of attack data, a total of six categories, and the remaining four attack categories become new unknown categories. In the incremental learning process, two new categories are introduced at each stage, and only two incremental updates are required. Experimental Scenario 3 further expands the scope of the initial categories, including normal traffic and randomly selected seven types of attack data, a total of eight categories, and the remaining two attack categories are added as unknown categories. During incremental learning, a single incremental update can cover all new categories. Table 4 shows the changes in the overall attack detection accuracy of different incremental learning methods in these three incremental attack detection scenarios. The bold number indicates the optimal result.

Table 4. Attack detection accuracy in different stages of different methods in three scenarios.

	Scenario	MCFP			CICIDS2017
Method		S1	S2	S3	S1	S2	S3
EWC		61.36	78.28	91.25	55.41	75.32	90.71
CIL-A-NIDS		79.17	88.32	93.13	72.22	83.61	92.17
CL based A-NIDS		82.36	90.24	95.08	76.26	85.48	93.08
SPCIL		90.25	93.53	95.62	86.23	90.53	94.16
IID-DAKD		92.73	95.65	97.17	91.06	92.95	95.21

As can be seen from Table 4, compared with other comparison algorithms, the IID-DAKD method proposed in this paper shows advantages and has the highest overall accuracy in the three incremental attack detection scenarios, which strongly verifies the effectiveness and superiority of the IID-DAKD method. At the same time, with the changes in scenario 1, scenario 2, and scenario 3, the overall accuracy of the attack detection model shows a trend of increasing sequentially. This is mainly due to two factors: first, the increase in the number of initial attack types, which allows the model to be exposed to more categories and learn richer and generalizable features during the training process; The second is the reduction in the number of incremental updates, which reduces the forgetting of known attack classification knowledge that may be brought about by each update, thereby improving the overall classification accuracy of the model.

4.3. Self-Supervised Method Experimental Results and Analysis

In the incremental update method, IID-DAKD, for the intrusion detection model studied in this paper, a self-supervised learning method based on enhanced encrypted traffic categories is effectively introduced. ETCAug improves the model’s ability to recognize new class attacks by integrating inter-class interpolation synthesis and rotation enhancement techniques. To comprehensively evaluate the effectiveness of the self-supervised learning method based on ETCAug, this section conducts both within-domain and cross-domain experiments on the MCFP and CICIDS2017 datasets, specifically performing incremental updates and attack detection on the same dataset, as well as on different datasets [79]. During the experimental process, the other structures of IID-DAKD remain unchanged, focusing on comparing the different experimental results with and without self-supervised learning, while also exploring the impact of using different self-supervised learning methods on the overall intrusion detection performance. Figure 12 and Figure 13 respectively show the comparison results of the overall accuracy of the incremental intrusion detection models using self-supervised learning methods based on four different category enhancement algorithms (including Mixup [80], ETCAug, Rotation [81], and BCI [71]), as well as without self-supervised learning (None), in scenario 1 of the MCFP dataset and scenario 2 of the CICIDS2017 dataset during within-domain incremental intrusion detection.

Figure 12. The result of scenario 1 on MCFP.

Figure 13. The result of scenario 2 on CICIDS2017.

Figure 14 shows the comparison of the overall accuracy of the incremental attack detection model in the complex context of cross-domain and incremental scenario 3, when four different synthetic class augmentation algorithms are used and no class augmentation is used.

Figure 14. Results of cross-domain scenario 3.

Figure 12, Figure 13 and Figure 14 are the average values of 10 repeated experiments to eliminate the impact of random factors. From the experimental results, it can be seen that the overall accuracy of the ETCAug algorithm is higher than that of the non-class augmentation in the same domain and cross-domain incremental attack detection tasks. This result highlights the key role of using self-supervised tasks in improving the performance of the model in detecting cyberattack behaviors, and also strongly verifies the applicability and effectiveness of the ETCAug method in the incremental update of the detection model. Creating and expanding new categories through self-supervised learning can not only effectively increase the diversity of data and reduce the interference of unreal samples but also help the model learn the internal rules and structure of the data more deeply, so as to achieve better generalization capabilities on new categories.

The ETCAug method presented in this paper stands out among all the algorithms and shows the best performance. This is due to the fact that ETCAug comprehensively considers the distribution of the data itself and the similarities between different categories and introduces new perspectives and variations through interpolation and rotation. This method not only retains the characteristics of spinning and synthesizing new classes to improve the robustness of the model, but also successfully introduces obfuscation information through interpolation synthesis of new classes, which promotes more general and transferable features of network learning. This strategy overcomes the representation bias to the greatest extent, improves the generalization performance of the model, and effectively solves the task-level overfitting problem in incremental learning. This is critical for tasks such as attack detection in cybersecurity, as it can help models adapt more flexibly to evolving real-world scenarios to more effectively respond to increasingly sophisticated cyber threats.

4.4. Fusion Knowledge Distillation Method Experimental Results and Analysis

In the incremental update method, IID-DAKD, for the intrusion detection model studied in this paper, an effective knowledge distillation method called FKD-ARR is employed, which integrates three techniques: attention distillation, relation distillation, and response distillation. To comprehensively evaluate the effectiveness of FKD-ARR, this section conducts both within-domain and cross-domain experiments on the MCFP and CICIDS2017 datasets. During the experimental process, the other structures of IID-DAKD remain unchanged, focusing on comparing the experimental results with and without knowledge distillation, while also exploring the impact of different knowledge distillation methods on the overall attack detection performance of the model. Figure 15 and Figure 16 respectively show the comparison results of the overall accuracy of the incremental intrusion detection model using five different knowledge distillation methods (including the traditional monolithic feature knowledge distillation method HKD, FKD-ARR, distance relation knowledge distillation method RKD-D [74], angle relation knowledge distillation method RKD-A [82], and distance and angle relation knowledge distillation method RKD-DA [74]) and the case without knowledge distillation, in scenario 1 of the MCFP dataset and scenario 2 of the CICIDS2017 dataset during within-domain incremental intrusion detection.

Figure 15. The result of scenario 1 on MCFP.

Figure 16. The result of scenario 2 on CICIDS2017.

Figure 17 shows the comparison of the overall accuracy of the incremental attack detection model in the complex context of cross-domain and incremental scenario 3, when five different knowledge distillation methods are used and when no knowledge distillation is used.

Figure 17. Results of cross-domain scenario 3.

From the experimental results of Figure 15, Figure 16 and Figure 17, it can be seen that the overall accuracy of the model using the knowledge distillation method is better than that of the non-knowledge distillation method in the same domain and cross-domain incremental attack detection tasks. This result not only highlights the key role of knowledge distillation technology in improving the performance of the model in detecting network attack behaviors but also provides strong evidence for the applicability and effectiveness of the FKD-ARR method in incremental attack detection models. Through the knowledge distillation technology, the knowledge of known attack classification contained in the old model is effectively refined and imported into the new detection model. This not only improves the performance and generalization ability of the new model in incremental attack detection but also effectively alleviates the overfitting problem in incremental learning.

Comparing the five knowledge distillation methods, it is found that the HKD method has certain advantages over RKD-DA, RKD-D and RKD-A in terms of performance, while the effect of the RKD-A method is relatively weak. However, the FKD-ARR method used in this paper stands out among all the methods and exhibits the best performance. This is mainly due to the fact that the FKD-ARR method integrates three complementary techniques: attention distillation, relational distillation, and response distillation. In the process of symmetrical knowledge transfer, FKD-ARR more comprehensively considers the attention distribution, distance and angle relationships between samples, and the logit output of the model. This comprehensive consideration enables the attack detection model to retain the memory of the old category attack knowledge to the greatest extent in the incremental learning process and effectively improve the adaptability to the new category attack. By alleviating the model’s forgetfulness of the old category attack knowledge, the conflict between improving the plasticity of the attack detection model and maintaining stability is successfully balanced. This enables the model to detect and defend against new attacks more quickly and accurately, improving the accuracy and efficiency of attack detection.

5. Conclusions

This paper studies the incremental update challenges faced by deep learning intrusion detection models when dealing with new types of attacks that continuously emerge in real network environments. To address the issues of catastrophic forgetting of old knowledge and task-level overfitting to current attack types during the incremental update process of intrusion detection models, a method for incremental updating of intrusion detection models based on dual enhancement and fused knowledge distillation, called IID-DAKD, is proposed.

First, a sample enhancement strategy for encrypted traffic is designed to reduce storage pressure and classifier bias, and to alleviate catastrophic forgetting in incremental learning; second, self-supervised learning based on enhanced categories of encrypted traffic effectively overcomes representation bias and mitigates task-level overfitting in incremental learning; finally, by utilizing a knowledge distillation method that integrates attention maps, structural information, and Logit outputs, old model knowledge is efficiently transferred, improving the effectiveness of knowledge distillation and the overall performance of the new model. Comparisons with experimental results from contrasting methods (EWC, CIL-A-NIDS, CL-based A-NIDS, and SPCIL) show that the proposed IID-DAKD method effectively alleviates the forgetting of knowledge related to old class attacks while relying on only a small amount of old class attack data and maintaining low storage costs. It also avoids overfitting to new types of attacks that arise. While continuously engaging in incremental learning to adapt to new network attack samples and categories, the model still maintains a high recognition rate for old attack samples and types without the need to relearn old knowledge. This advantage gives the proposed IID-DAKD method practical application potential in network intrusion detection systems within open dynamic network environments.