A Lightweight Multi-Classification Model for Identifying Network Application Traffic Using Knowledge Distillation

Abstract

To address the limitations of insufficient feature representation, large model size, and high deployment cost in network traffic classification, a lightweight classification framework based on multi-teacher knowledge distillation is proposed. The framework consists of two heterogeneous teacher networks and a compact student network to enable end-to-end traffic classification under constrained computational resources. The teacher networks incorporate complementary spatio-temporal modeling strategies, including a bidirectional temporal convolutional network (BiTCN) enhanced with attention mechanisms and convolutional neural network (CNN), and a parallel spatio-temporal fusion architecture integrating bidirectional long short-term memory (BiLSTM) and CNN. Knowledge from the teacher ensemble is distilled into a lightweight CNN-based student network through soft-target supervision, leading to improved generalization capability with significantly reduced model complexity. Experimental results demonstrate that effective knowledge transfer is achieved while reducing model parameters by more than 80%, and performance gains of about 1–3% are obtained compared with baseline methods. These results indicate strong potential for practical deployment in resource-constrained network environments.

1. Introduction

As the demand for data security and privacy continues to grow, there has been a notable increase in the volume and complexity of Internet traffic. This trend reflects a heightened awareness of the importance of safeguarding sensitive information against unauthorized access and cyber threats. Network protocols play a crucial role in protecting data transmitted over networks, ensuring that only authorized parties can access the information [1]. However, the growing diversity of public and private network protocols introduces significant challenges for the identification and classification of network traffic. Traditional methods of traffic analysis often struggle to keep pace with the rapid evolution of these protocols, which can vary widely in their implementation and functionality. Each protocol may utilize different encryption algorithms, communication patterns, and obfuscation techniques, complicating the task of accurately identifying and classifying traffic flows. For example, some malicious network activities, such as botnets, also utilize proprietary protocols to evade detection and enhance their concealment. Hence, accurate classification of network traffic is critical for network management and security.

The increasing complexity and diversity of network traffic, together with the widespread use of proprietary protocols, have significantly reduced the effectiveness of traditional traffic identification methods, including signature-based classification, anomaly-based classification and behavioral-based classification [2]. Each method of classification has its own limitations. Signature-based classification only detects known threats and requires constant updates to its signature database, making it prone to false negatives for variants of existing malware and resource-intensive due to its reliance on large databases. Anomaly-based classification, while useful, often suffers from high false positive rates for legitimate behavior and requires substantial amounts of normal-behavior data for effective training. Behavioral-based classification is context-sensitive, as behavior can vary across different environments, and continuous monitoring can consume significant resources. It also has limitations in detecting threats that do not exhibit noticeable behavioral changes, and attackers can sometimes mimic normal behavior to evade detection.

To address the aforementioned issues, a network traffic classification framework based on many-to-one knowledge distillation is proposed. The knowledge distillation framework consists of a multi-teacher model and a lightweight student model. The multi-teacher model integrates two complementary components: one is a bidirectional temporal convolutional network with attention mechanisms (BiTCN) and a convolutional neural network (CNN), and the other is a spatio-temporal parallel fusion of bidirectional long short-term memory (BiLSTM) and a CNN. Flexible weighting between the two teacher components allows different propotional combinations to be constructed. The student network is designed as a shallow two-layer CNN. This choice is motivated by three considerations: lightweight deployment requirements, structural consistency with the shared spatial extractor in the teacher models, and the relatively compact input representation after preprocessing. In this way, the student can preserve efficient local spatial feature extraction while relying on distilled teacher knowledge to compensate for the removed temporal modeling modules. Experimental results demonstrate that this design achieves effective knowledge transfer and maintains competitive classification performance with substantially reduced model complexity.

The primary contributions of this paper are as follows:

• A heterogeneous multi-teacher distillation framework is proposed for network traffic classification. Unlike conventional single-teacher or homogeneous multi-teacher settings, the proposed framework combines CNN-BiTCN and CNN-BiLSTM to provide complementary supervision for different temporal dependency patterns in traffic data.
• A lightweight student architecture is designed by retaining only the shared CNN-based spatial extractor of the teacher models and removing explicit temporal modeling modules. This design makes the contribution of teacher knowledge more interpretable while substantially reducing model size and computational overhead.
• Extensive experiments are conducted on the ISCX-VPN2016, USTC-TFC2016, ISCX-Tor2016, and CIC-IoT2022 datasets. The results verify that the proposed framework achieves competitive multi-class classification performance while maintaining strong compression efficiency, demonstrating its practical value for deployment in resource-constrained environments.

The remainder of this paper is organized as follows: Section 2 reviews related work and research methods about network traffic classification and knowledge distillation. Section 3 presents our proposed knowledge distillation model and the corresponding algorithms. Section 4 provides a comprehensive overview of the experimental results and comparisons with other models. Section 5 summarizes the conclusions and presents the future work.

2. Related Work

This section provides a comprehensive review of existing network traffic classification algorithms and models, with an emphasis on their strengths and limitations. Subsequently, the concepts of model compression and knowledge distillation are introduced, followed by a survey of recent studies employing knowledge distillation techniques.

2.1. Network Traffic Classification and Identification

Traffic classification is essential for application identification and network management. Early methods, such as port-based classification, were simple but suffered from limited accuracy. Deep packet inspection improved accuracy but faced challenges with encrypted traffic and high computational complexity. With the widespread adoption of encryption, machine learning techniques based on statistical and behavioral features became mainstream [1]. Recently, deep learning has gained considerable traction owing to its ability to automate feature extraction and provide enhanced accuracy and portability. This positions deep learning as a highly promising solution for encryption and malicious network traffic classification.

Various deep learning architectures have been explored for network traffic classification. In recent years, classical deep learning techniques such as CNN, Recurrent Neural Networks (RNN), Autoencoders, and Graph Neural Networks (GNN) have demonstrated strong performance in traffic classification tasks. For instance, Wang et al. [3] proposed a 1D-CNN–based end-to-end encrypted traffic classification method that directly processes raw traffic data. In Ref. [4], FlowPic converts encrypted traffic into visual images, allowing basic deep learning models to handle application traffic identification and classification. Lin et al. [5] introduced ET-BERT, a bidirectional transformer-based model that employs multi-head self-attention to classify encrypted traffic. Liu et al. [6] proposed ATVITSC, which integrates bidirectional LSTM and attention mechanisms for spatiotemporal feature fusion, effectively capturing complex temporal dependencies in encrypted traffic. Li et al. [7] proposed SAT-Net, a GNN-based model that formulates traffic classification as a graph learning problem by constructing packet byte graphs.

Lightweight models have also attracted increasing attention due to their deployment efficiency. Ref. [8] combined LSTM and CNN with Squeeze-and-Excitation mechanisms to enhance spatio-temporal feature selection. Wang et al. [9] developed a lightweight fully CNN for malicious traffic classification, optimizing both shallow and deep layers to enhance efficiency without sacrificing accuracy. Zhao et al. [10] proposed YaTC, an MAE-based traffic transformer with multi-level flow representation and two-stage training, achieveing strong rubustness across multiple real-world datasets. Zhan et al. [11] proposed EAPT, which employs adversarial pre-trained transformers with disentangled attention and a replaced burst detection pre-training task. Yu et al. [12] developed a BERT-based encrypted traffic classification model leveraging header field tokens to achieve accurate and privacy-preserving classification. Gwak et al. [13] introduced a lightweight Transformer-based model with optimized self-attention to reduce model complexity while maintaining competitive performance. Moreover, the L2-BiTCN-CNN model in Ref. [14] combines BiTCN and CNN architectures with dual-level attention mechanisms. Its parallel fusion design facilitates comprehensive feature extraction across byte and packet levels.

Beyond model architecture optimization, recent studies have also explored deployment-oriented and real-time classification strategies. Nunez-Agurto et al. [15] proposed a deep learning–based framework within a software-defined networking architecture for centralized traffic analysis. Xi et al. [16] introduced SwiftSession, an incremental classification framework that leverages early packet features and confidence-based decision mechanisms to achieve rapid traffic identification. Despite the significant progress in traffic classification, existing methods still face notable limitations. In particular, achieving an effective balance among classification accuracy, computational efficiency, and deployment adaptability remains a persistent challenge, especially in resource-constrained network environments.

2.2. Knowledge Distillation and Its Application of Traffic Classification

The balance between the model accuracy and the training efficiency has been a key issue for researchers. C. Bucilua et al. [17] introduced the early concept of model compression, providing a seminal foundation for subsequent teacher-student learning and knowledge distillation studies. Knowledge distillation, introduced by G. Hinton et al. [18], has gained prominence as a model compression technique. It transfers knowledge from a large model to a smaller, more efficient one by retaining essential information while discarding less critical parts.

In the field of traffic classification with knowledge distillation, substantial progress has been made toward teacher-student model design. Wu et al. [19] proposed a distillation-based encrypted traffic classification framework in which both teacher and student models adopted LSTM structures, and an adaptive temperature mechanism was introduced to dynamically adjust label softness during training. Zhu et al. [20] developed LKD-STNN, combining depthwise separable convolutions and BiLSTM within a distilled student model to capture multi-dimensional features under resource constraints. Lu et al. [21] further explored compression strategies by integrating self-distillation with knowledge distillation, achieving additional parameter reduction through a two-stage compression process. Zhao et al. [22] introduced KDTC, transferring knowledge from a Transformer-based teacher to a MobileNet student via an improved distillation loss, thereby enhancing both efficiency and classification accuracy.

Beyond conventional lightweight classification frameworks, recent studies have extended knowledge distillation to broader traffic analysis scenarios. Yang et al. [23] systematically reviewed data and knowledge condensation techniques for traffic classification and summarized deployment challenges in resource-limited environments. Wang et al. [24] proposed a spatio-temporal distillation framework for traffic prediction to preserve temporal modeling capability while improving efficiency. Tran et al. [25] introduced KD-PFL-TC, a knowledge distillation–based personalized federated learning framework that enables adaptive cross-domain knowledge transfer with reduced communication overhead. Yang et al. [26] developed a growth-adaptive distillation fusion model for IoT traffic identification, employing staged structural adjustment and feature-based transfer to support lightweight deployment in cloud–edge collaborative environments.

As knowledge distillation methods have evolved, researchers have explored diverse applications. To address the limitation of the single-teacher model, researchers have proposed some multi-teacher knowledge distillation models. Lin et al. [27] introduced ATMKD, which incorporates a dynamic, learnable temperature to adaptively regulate knowledge complexity. Ge et al. [28] proposed a novel framework for decentralized federated learning, named PFL-DKD, which employs decoupled knowledge distillation to facilitate personalized local model adaptation.

In summary, existing knowledge distillation studies have demonstrated the value of teacher-student learning for efficient traffic analysis, but most of them rely on single-teacher transfer or do not explicitly exploit heterogeneous temporal modeling complementarities. This limitation motivates the present work, in which CNN-BiTCN and CNN-BiLSTM are jointly employed as complementary teachers and their knowledge is transferred to a lightweight CNN student. The proposed design therefore differs from prior approaches by emphasizing heterogeneous multi-teacher supervision for lightweight multi-class traffic classification.

3. Our Model

3.1. Overview

The proposed knowledge distillation model consists of two teacher models and one student model. As illustrated in Figure 1, the figure presents both the overall architecture and the training process, offering a clear comparison of the structure and size between the teacher and student models.

The first teacher model is CNN-BiTCN, which is based on dual attention mechanisms and the parallel fusion of spatio-temporal features. This model captures critical traffic characteristics at both the byte and packet levels. The second teacher model is CNN-BiLSTM, which also leverages attention mechanisms and spatio-temporal feature fusion. However, the key difference lies in the inclusion of LSTM, which makes this model particularly effective at recognizing long-sequence features within network traffic. The combination of CNN for spatial feature extraction and bidirectional LSTM for temporal sequence modeling enables the CNN-BiLSTM model to provide a complementary perspective to the CNN-BiTCN model, further enriching the student’s learning process. The student model, distilled from both teacher models, adopts a simplified deep learning architecture. It incorporates the shared two-layer CNN structure from the teacher models and learns temporal features as distilled knowledge. Unlike the teacher models, the student model retains and simplifies only the CNN layers, thereby significantly reducing computational complexity while preserving key performance attributes.

In the knowledge distillation process, the preprocessed data is also input into the student model. The pre-trained teacher model is typically used to generate output predictions directly. The optimal balance between the outputs of the two teacher models is determined by adjusting the parameter $\beta$. These soft labels are paired with the student model’s predictions, and the temperature coefficient $\tau$ is adjusted to compute the soft label loss. Simultaneously, the hard label loss is calculated based on the ground truth. The total loss is obtained by combining the soft and hard label losses using parameter $\alpha$. By iteratively updating the student model based on the total loss, the algorithm enables the student to effectively mimic the teacher’s behavior while improving its generalization ability.

3.2. Traffic Preprocessing

Network traffic is composed of a series of packets, each carrying distinct identifying data. A flow is characterized as a sequence of packets that share a common five-tuple, which includes the source IP address, source port, destination IP address, destination port, and the transport layer protocol. These packets are ordered chronologically. Given the bidirectional nature of both TCP and UDP protocols, network flows can be categorized as either unidirectional or bidirectional [14]. The preprocessing framework involves multiple stages, including traffic segmentation, anonymization detection, data cleansing, and format conversion. As depicted in Figure 2, this procedure transforms raw traffic packets in pcap format into both a one-dimensional traffic byte sequence and a two-dimensional numerical matrix.

To segment the raw traffic data, the publicly available SplitCap tool is utilized, which separates the data into discrete network flows represented as bidirectional flows. During the anonymization stage, IP address fields are removed from the packets to minimize their influence on feature extraction during classification, as IP addresses do not necessarily reflect the intrinsic content or behavioral characteristics of the traffic. Moreover, since IP addresses vary across different network environments, including them may hinder the model’s ability to generalize. The anonymized traffic data is further processed to remove duplicate packets and those without payloads. Each packet is normalized to a fixed length T, which is set to 784 bytes in this study. Packets exceeding this length are truncated, while shorter packets are padded with zeros. This preprocessing step ensures that all samples have a consistent representation, thereby simplifying batch processing and improving the comparability of results across different datasets.

The choice of 784 bytes is motivated by both prior practice and the design requirements of our framework. Recent studies have shown that traffic samples are often standardized to a fixed byte length before being used for deep learning, and 784-byte representations have also been adopted in recent encrypted traffic classification research [29]. Moreover, transforming byte sequences into square matrices such as $28\times 28$ has been used to facilitate two-dimensional convolutional feature extraction [30]. In the proposed framework, 784 bytes provide a practical trade-off between retaining leading payload information and controlling computational overhead, while also allowing the input to be reshaped into a $28\times 28$ matrix for CNN-based spatial modeling. This image-like structure allows convolutional filters to extract local spatial dependencies, thereby improving feature representation. After normalization, the processed network flow data is initially converted into a one-dimensional byte sequence, transformed into hexadecimal format, and labeled for both models. It is then further organized into a two-dimensional numerical matrix, where the first T bytes of the binary operation packet from each session flow are extracted. The final dataset is structured as a two-dimensional matrix of dimensions $(N,T)$, and two .idx files are generated: one containing the one-dimensional byte sequences and another containing the corresponding two-dimensional numerical matrices.

3.3. Distillation Model Structure

This framework employs two teacher models to provide complementary supervision for the student model: CNN-BiTCN and CNN-BiLSTM. The CNN-BiTCN teacher guides the extraction of byte- and packet-level spatio-temporal features, while the CNN-BiLSTM teacher enhances long-sequence temporal modeling. Through joint distillation, the student acquires a more comprehensive representation of network traffic. The student adopts a simplified two-layer CNN structure for three reasons. First, the target deployment scenario requires a lightweight architecture with low storage and computation overhead. Second, both teacher models share the same CNN-based spatial feature extractor, and retaining this part in the student enables a fairer structural comparison while making the contribution of distillation more interpretable. Third, after preprocessing, each traffic sample is transformed into a relatively compact two-dimensional representation, for which a shallow CNN is sufficient to capture local spatial correlations without introducing unnecessary complexity.

3.3.1. Multi-Teacher Model

The first teacher model employed is the CNN-BiTCN, which captures critical temporal features at both byte and packet levels via a dual attention mechanism. The detailed discussion of spatial feature processing will be presented in the student model section, while the current section primarily focuses on illustrating the methodology for temporal feature processing. For temporal feature extraction, this model operates at both byte and packet levels.

Byte level processing: Raw traffic data is preprocessed into 1D byte sequences and embedded as vectors:

$$X_i=\{x_{i1},x_{i2},\dots ,x_{iT}\}$$

(1)

These vectors are fed into the BiTCN module to capture bidirectional temporal dependencies, as shown in Figure 3.

The BiTCN utilizes dilated convolutions to expand the receptive field without increasing computational complexity. This capability proves critical for capturing long-range temporal patterns in traffic sequences. For forward feature extraction, the convolution at time step t integrates past information through spaced inputs determined by dilation rate $\eta$. The receptive field grows as:

$$\mathrm{RF}=(m-1)\eta +1$$

(2)

where m is the filter size. To mitigate gradient vanishing, residual connections are incorporated followed by nonlinear transformation via LeakyReLU:

$$\overrightarrow{h_{it}}={\mathrm{LeakyReLU}}^f(\omega ·C_{it}+b_1)$$

(3)

Here, $C_{it}$ denotes the convolved output, and $\omega ,b_1$ are learnable parameters. Backward features $\overleftarrow{h_{it}}$ are extracted identically after sequence reversal. These bidirectional representations are concatenated to preserve both forward and backward temporal contexts:

At the packet level, similar to the byte-level design, a parallel dilated convolutional structure is employed to capture inter-packet dependencies, extracting bidirectional temporal features that complement the global temporal information less effectively represented at the byte level. The CNN-BiLSTM model also extracts spatial and temporal features from network traffic. The CNN component processes payload information, while the BiLSTM captures sequential dependencies from packet length series. The CNN component follows an architecture similar to that described earlier, consisting of two convolutional layers each followed by max pooling operations to extract meaningful spatial features from traffic flows. As in previous models, the CNN output is flattened into a vector $f_2$, which will be concatenated with the features from the BiLSTM network.

The BiLSTM component focuses on processing packet length sequences, crucial for capturing temporal dynamics of network traffic. Its input is a 1D byte sequence, with each layer containing 256 hidden units. The network structure appears in Figure 4.

At the byte level, the BiLSTM generates hidden states for each packet i and byte t through sequential processing:

$$h_{it}=\mathrm{LSTM}(x_{it},h_{it-1})$$

(4)

Being bidirectional, it processes both forward and backward sequences to capture relationships between distant packets, producing combined states $h_t$ that integrate forward $h_f$ and reverse $h_b$ information. An attention mechanism enhances performance by computing weights for each packet in the sequence:

$${\omega }_t={\displaystyle \frac{e^{(W·h_t)}}{{\sum }_{k=1}^T{e}^{(W·h_k)}}}$$

(5)

where W is the attention weight matrix. These weights produce a weighted sum of hidden states, forming the final temporal feature vector $f_1^{\prime }$. The spatial features $f_2$ and temporal features $f_1^{\prime }$ are concatenated into a joint feature vector f, which passes through a fully connected layer for classification. The model is trained using cross-entropy loss, with weights updated via gradient descent using standard backpropagation. Through this combination of CNN for spatial feature extraction and BiLSTM for temporal sequence analysis, the model can effectively capture both the spatial and temporal aspects of network traffic.

3.3.2. Student Model

The student model utilized in this study is designed as a lightweight architecture that specifically focuses on processing spatial features. This distinguishes it from the two teacher models, which integrate both spatial and temporal features through feature fusion. The primary objective of the student model is to achieve a balance between computational efficiency and classification performance. This is accomplished by utilizing a simplified architecture that minimizes computational overhead while maintaining robust classification capabilities. Specifically, the student model employs a two-layer convolutional neural network structure, which mirrors the CNN module of the teacher models. The CNN module in this structure is composed of two sets of convolutional and pooling layers, followed by a flattening layer and a fully connected layer. The pooling layers use max pooling, and the fully connected layer concatenates the extracted features. The choice of a two-layer CNN is deliberate that it keeps the model compact, maintains architectural consistency with the teacher networks, and allows the performance gain brought by distillation to be observed more directly without being confounded by a deeper student backbone. The workflow of the CNN module is shown in Figure 5.

The network traffic data is preprocessed into a two-dimensional matrix, denoted as $(N,T)$, which serves as the input to the CNN model. Each element $f_M$ represents a feature of an individual flow data, and this structure is shown as:

$$f_M=\left(\begin{array}{cccc}{f}_{11}& f_{12}& \dots & f_{1T}\\ f_{21}& f_{22}& \dots & f_{2T}\\ ⋮& ⋮& \ddots & ⋮\\ f_{N1}& f_{N2}& \dots & f_{NT}\end{array}\right)$$

(6)

where $f_M$ represents the traffic data matrix containing binary information. The data is then processed through a series of convolutional layers. In these layers, the activation function used is LeakyReLU, which introduces non-linearity into the network. The first convolutional layer uses a $5\times 5$ kernel, and the second layer uses a $1\times 3$ kernel. The convolution operation is expressed as:

$$Z^{\left(1\right)}=f(W^{\left(1\right)}\ast X+b^{\left(1\right)})$$

(7)

where X is the input data, $W^{\left(1\right)}$ is the convolutional kernel, $b^{\left(1\right)}$ is the bias, ∗ represents the convolution operation, and f is the LeakyReLU activation function. Subsequently, max pooling is introduced after each convolutional layer to further reduce the feature dimension. The feature maps after pooling are flattened into one-dimensional vectors and then concatenated during the feature fusion stage to form the spatial feature vector f2 of the student model. Afterward, gradient descent and parameter updates are performed. By combining the temporal and spatial features obtained, the final feature vector f is formed for classification. The loss is calculated using the cross-entropy function, which is expressed as:

$$L=-\sum _{c=1}^C{f}_{c}ln{p}_c$$

(8)

where $f_c$ represents the true label and $p_c$ represents the predicted probability. For parameter optimization, Stochastic Gradient Descent (SGD) is employed to update the model parameters, which iteratively minimizes the total loss to ensure stable convergence.

This design enables evaluation of classification performance alongside computational efficiency within a lightweight structure. By incorporating different temporal modeling strategies in the teacher networks, the framework clarifies the impact of temporal feature processing on student performance. The analysis highlights the importance of temporal representations and elucidates the trade-offs involved in lightweight model design under resource constraints.

3.4. Knowledge Distillation Algorithm

The integration of teacher and student models forms the core of the proposed framework. During training, the student model is optimized under the supervision of the teacher’s soft outputs, enabling more effective parameter learning and improved final performance. Algorithm 1 provides the recursive implementation of knowledge distillation.

Algorithm 1 Knowledge Distillation

1: Input: Teacher Models $T_1$, $T_2$, Student Model S, Training Dataset D, Temperature $\tau$, Distillation Loss Weight $\alpha$, Teacher Weight $\beta$ 2: Output: Trained Student Model S 3: function TrainModel(D, S) 4:       if D is empty then 5:             return S 6:       end if 7:       for each minibatch $(x,y)$ in D do 8:             $z_{T_1}\leftarrow T_1\left(x\right)$ 9:             $z_{T_2}\leftarrow T_2\left(x\right)$ 10:           $z_S\leftarrow S\left(x\right)$ 11:           $q_{T_1}\leftarrow \mathrm{Softmax}(z_{T_1}/\tau )$ 12:           $q_{T_2}\leftarrow \mathrm{Softmax}(z_{T_2}/\tau )$ 13:           $q_S\leftarrow \mathrm{Softmax}(z_S/\tau )$ 14:           $q_T\leftarrow \beta q_{T_1}+(1-\beta )q_{T_2}$ 15:           $L_{\mathrm{CE}}\leftarrow \mathrm{CrossEntropy}(y,q_S)$ 16:           $L_{\mathrm{D}}\leftarrow \mathrm{KL}(q_T\Vert q_S)$ 17:           $L\leftarrow (1-\alpha )L_{\mathrm{CE}}+\alpha {\tau }^2{L}_{\mathrm{D}}$ 18:           Update S using loss L 19:       end for 20:       return S 21: end function 22: return TrainModel(D, S)

The student model is optimized under the supervision of the teacher model to facilitate effective knowledge transfer. In the distillation process, both hard labels and teacher-generated logits are incorporated into the loss function, where the latter act as soft targets.

During multi-teacher distillation, the logits from the various teacher models are combined to generate the soft labels for the student model. The combined teacher output is typically calculated as a weighted average of the logits from the different teacher models. This process is defined as:

$$z_T^{\prime }=\sum _{i=1}{\beta }_i{z}_{T,i}^{\prime }$$

(9)

where the weight ${\beta }_i$ controls the contribution of each teacher model to the generation of the soft labels, $z_T^{\prime }$ represents the combined logits from all teacher models, and $z_{T,i}^{\prime }$ is the logits of the i-th teacher model.

The temperature parameter $\tau$ is a significant parameter, used to smooth the distribution of probabilities. When $\tau$ is equal to 1, the softmax function simplifies to its standard form; when $\tau$ is greater than 1, the probability distribution becomes smoother; when $\tau$ is less than 1, the distribution becomes sharper. The soft labels are generated by the teacher model through the softmax function, and the probability distribution is calculated as:

$${\widehat{y}}_{T,i}={\displaystyle \frac{e^{(z_{T,i}/\tau )}}{{\sum }_j{e}^{(z_{T,j}/\tau )}}}$$

(10)

where $z_T$ denotes the logits from the teacher model, representing the unnormalized log probabilities prior to the application of the softmax function, and $z_{T,i}$ is the logit value for the i-th class generated by the teacher model. Similarly, the softmax output expression for the student model is given by:

$${\widehat{y}}_{S,i}={\displaystyle \frac{e^{(z_{S,i}/\tau )}}{{\sum }_j{e}^{(z_{S,j}/\tau )}}}$$

(11)

where ${\widehat{y}}_{S,i}$ represents the predicted probability of the i-th class, and $z_{S,i}$ denotes the logit value of the i-th class. By tuning the temperature parameter, the student model can better learn from the knowledge of the teacher model.

To incorporate soft labels into student training, a distillation loss is defined. The overall objective consists of two components: the cross-entropy (CE) loss computed with ground-truth labels and the Kullback–Leibler (KL) divergence between the teacher and student soft outputs. The CE loss with the true labels is formulated as follows:

$$L_{CE}(y,{\widehat{y}}_S)=-\sum _i{y}_{i}log\left({\widehat{y}}_{S,i}\right)$$

(12)

where $y_i$ is the true label of the i-th class, and ${\widehat{y}}_{S,i}$ is the predicted probability of the i-th class. Subsequently, the KL divergence between the teacher and student soft outputs is computed as follows:

$$KL({\widehat{y}}_T,{\widehat{y}}_S)=\sum _i{\widehat{y}}_{T,i}log\left({\displaystyle \frac{{\widehat{y}}_{T,i}}{{\widehat{y}}_{S,i}}}\right)$$

(13)

The total distillation objective is defined by integrating the CE and KL divergence terms, and is given by:

$$L_D=(1-\alpha )L_{CE}(y,{\widehat{y}}_S)+\alpha {\tau }^{2}KL({\widehat{y}}_T,{\widehat{y}}_S)$$

(14)

where $\alpha$ is a balancing factor to control the weights between the CE loss with the true labels and the KL divergence with the soft labels, and ${\widehat{y}}_T$ represents the soft labels generated by the combined logits of all teacher models.

The balancing factor $\alpha$ and temperature parameter $\tau$ regulate the contribution of the distillation term, while the weights ${\beta }_k$ control the relative influence of each teacher model. Proper adjustment of these parameters enables effective knowledge transfer from multiple teachers to the student model. The multi-teacher framework provides complementary representations, particularly when teacher models differ in architecture or training data.

4. Performance Evaluation and Experimental Result Analysis

4.1. Experimental Environment Settings

This experiment was conducted on a Windows 64-bit operating system with 128 GB of RAM, equipped with an Intel(R) Xeon(R) W-2223 CPU @ 3.60 GHz and an NVIDIA T1000 GPU. Python 3.9 was used as the programming environment, and PyTorch 2.4.1 Four public datasets are used to validate the performance of the distillation model: ISCX-VPN2016, USTC-TFC2016, ISCX-Tor2016, and CIC-IoT2022. The detailed descriptions for these datasets can be found in

Table 1. Dataset description.

Dataset	Labels	Training Set	Test Set
ISCX-VPN	12	35,501	3945
USTC-TFC	24	379,812	42,201
ISCX-Tor	8	34,608	3844
CIC-IoT	10	24,641	2740

.

The ISCX-VPN dataset is specifically designed for the study of VPN-based communications, containing traffic traces from diverse applications such as web browsing, chat, email, and streaming, with a total of 12 VPN and nonVPN traffic types. The USTC-TFC dataset provides a comprehensive benchmark for distinguishing between benign and malicious network traffic. It encompasses multiple common application protocols, including HTTPS, Email, FTP, and P2P, and is balanced with 12 benign and 12 malicious categories. The ISCX-Tor dataset contains both Tor-encrypted and non-Tor traffic, covering 8 application types. It serves as a valuable benchmark for evaluating classification methods in anonymized communication scenarios. The CIC-IoT dataset captures traffic from a wide range of IoT devices and incorporates simulated attacks including DDoS, malware, and data exfiltration, covering 10 categories of malicious traffic. The selected datasets are representative public benchmark datasets in the field of network traffic classification, and are of great value for comparison with previous studies. Furthermore, they collectively cover VPN, Tor, malicious traffic, and IoT attack traffic, providing a fairly diverse evaluation environment for the proposed framework.

4.2. Performance Evaluation

This experimental section utilizes several evaluation metrics, including accuracy, precision, recall, $F_1$-score and time, to evaluate the efficiency of the knowledge distillation model in traffic classification. Detailed information is provided in

Table 2. Evaluation metric.

Metric	Formula
Accuracy	$A={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$
Precision	$P={\displaystyle \frac{TP}{TP+FP}}$
Recall	$R={\displaystyle \frac{TP}{TP+FN}}$
$F_1$-score	$F_1={\displaystyle \frac{2\times P\times R}{P+R}}$
Time	$T=T_{end}-T_{start}$

.

As summarized in Table 2, Accuracy measures the overall proportion of correctly classified samples, Precision and Recall reflect the false-positive and false-negative characteristics of the model, respectively, and the $F_1$-score provides a balanced evaluation of Precision and Recall. In addition, Time denotes the average training time per epoch and is used to reflect computational efficiency under the same hardware conditions.

4.2.1. Performance Evaluation Under Various Parameters

To comprehensively evaluate the proposed model, experiments were conducted under different distillation settings and backbone architectures. The main experimental settings are summarized in

Table 3. Experimental parameter settings.

Parameter	Setting
Batch size	50
Dropout rate	0.1
Learning rate	0.003
$\tau$	4
$\alpha$	0.4
$\beta$	0.4

.

Considering the differences in dataset scale, the ISCX-VPN2016 dataset was trained for 200 epochs, whereas the other three datasets stabilized at around 40 epochs. Training was terminated once the validation accuracy converged. The key tunable parameters include the temperature $\tau$, distillation loss weight $\alpha$, and teacher weight coefficient $\beta$, whose effects are further analyzed on the four datasets. To further examine the stability of the proposed framework, the main comparison experiments were repeated five times. The observed fluctuations were small across all four datasets, indicating that the proposed method yields stable results under repeated runs.

The experimental results at various temperatures are shown in Figure 6. We set seven different values for $\tau$, ranging from 1 to 7 with an interval of 1, in order to conduct a comparative analysis of the results. The experimental results indicate that as the number of training epochs increases, the curves gradually converge and eventually stabilize at a relatively consistent value. With $\tau$ set to 4, 5, and 6, the accuracy on ISCX-VPN and USTC-TFC reaches over 99.35% and 99.75%, respectively, while for ISCX-Tor and CIC-IoT it exceeds 97.85% and 99.35% when $\tau$ is 3, 4, and 5. It is worth noting that as the temperature increases from an initial value of 1, the model’s performance improves, peaking at a temperature of 4, after which it begins to decline. This is primarily because a low temperature sharpens the probability distribution, which prevents the student model from acquiring sufficient knowledge. Conversely, a high temperature results in the loss of the original characteristics. According to a series of experimental tests, the optimal temperature is determined to be 4.

Additionally, the parameters $\alpha$ and $\beta$ control the balance between soft and hard labels as well as the combination of the two teacher models, respectively. If these coefficients are set too high or too low, the model’s performance may be compromised, as either component could dominate the influence on the overall training process, resulting in an imbalance. To address this issue, $\alpha$ and $\beta$ were varied within the range of 0.4 to 0.6, and three configurations were evaluated to assess their influence on model performance. The comparative results are summarized in Figure 7.

As shown in Figure 7, heatmaps are used to visualize performance across four datasets under different combinations of $\alpha$ and $\beta$. As illustrated, the configuration $\alpha =0.4$ and $\beta =0.4$ yields consistently strong performance across all datasets, with accuracy values of 0.985 on ISCX-VPN, 0.992 on USTC-TFC, 0.988 on ISCX-Tor, and 0.993 on CIC-IoT. Although this setting is not the absolute best on the USTC-TFC dataset, the margin compared to the optimal configuration is negligible. Overall, this choice achieves a favorable trade-off between robustness and accuracy and is therefore adopted in our subsequent experiments.

4.2.2. Performance Comparisons with Other Models

The proposed model was evaluated against its variants as well as several state-of-the-art (SOTA) methods, including ET-BERT [5], ATVITSC [6], YaTC [10], and EAPT [11]. These baseline models were selected to assess classification robustness under multi-class settings. The comparative results are summarized in

Table 4. Performance comparison of different models across four datasets.

Models	ISCX-VPN				USTC-TFC				ISCX-Tor				CIC-IoT
	Accuracy	Precision	Recall	${\mathit{F}}_{\mathbf{1}}$-Score	Accuracy	Precision	Recall	${\mathit{F}}_{\mathbf{1}}$-Score	Accuracy	Precision	Recall	${\mathit{F}}_{\mathbf{1}}$-Score	Accuracy	Precision	Recall	${\mathit{F}}_{\mathbf{1}}$-Score
ET-BERT	96.029	96.060	96.036	96.046	98.705	98.779	98.778	98.778	94.330	95.136	93.434	94.123	95.834	96.162	95.485	95.632
ATVITSC	96.401	96.630	96.046	96.332	99.661	99.669	99.676	99.671	95.340	95.347	95.340	95.345	96.787	96.755	96.821	96.622
YaTC	96.107	96.046	96.180	96.079	98.839	98.859	98.820	98.839	95.731	95.721	95.740	95.731	96.584	96.543	96.620	96.576
EAPT	95.447	96.011	94.683	95.291	99.222	99.220	98.869	99.310	94.960	95.064	94.850	94.899	95.346	95.754	94.884	94.690
Our Model	96.603	97.209	96.448	96.811	99.796	99.668	99.723	99.695	95.650	95.754	95.519	95.616	97.925	97.708	97.881	97.728

.

As shown in Table 4, the proposed model consistently outperforms across all four datasets compared with SOTA models. On ISCX-VPN and USTC-TFC, it achieves the highest accuracy with consistent gains in precision, recall, and $F_1$-score. For ISCX-Tor, competitive performance is maintained with noticeable improvements in recall compared to several baselines. The most pronounced advantage is observed on the CIC-IoT dataset, where the proposed model demonstrates clear improvements across all evaluation metrics.

To evaluate run-to-run stability, we further repeated the main comparison experiments with different random seeds. The standard deviations of the proposed model on ISCX-VPN, USTC-TFC, ISCX-Tor, and CIC-IoT were 0.09, 0.02, 0.07 and 0.06, respectively. The relatively small fluctuations indicate that the proposed framework maintains stable classification performance in repeated runs. Overall, the results indicate strong robustness and generalization capability in diverse multi-class traffic classification scenarios.

In addition, model size, parameter count, and FLOPs are key factors in evaluating the effectiveness of model compression and deployment efficiency. The sizes, parameter counts, and computational costs of the teacher and student models are presented in

Table 5. Comparison of model size across different datasets.

Models	ISCX-VPN			USTC-TFC			ISCX-Tor			CIC-IoT
	Size	Params	FLOPs	Size	Params	FLOPs	Size	Params	FLOPs	Size	Params	FLOPs
ET-BERT	114.28 MB	29,930,105	116,779,195	115.63 MB	30,282,952	116,846,791	114.91 MB	30,094,533	116,756,663	115.47 MB	30,241,361	116,767,929
ATVITSC	105.18 MB	27,551,984	107,465,229	106.42 MB	27,876,385	108,730,732	105.73 MB	27,695,629	108,025,700	106.08 MB	27,787,310	108,383,298
YaTC	41.12 MB	10,769,865	33,429,370	41.64 MB	10,911,249	33,867,986	40.80 MB	10,722,737	33,282,854	41.00 MB	10,746,301	33,355,995
EAPT	24.21 MB	6,341,304	19,416,064	24.47 MB	6,408,912	19,489,792	24.01 MB	6,318,768	19,391,488	24.10 MB	6,330,036	19,403,776
Our Model	5.03 MB	1,314,452	3,700,992	5.16 MB	1,348,256	3,734,784	4.98 MB	1,303,184	3,689,728	5.01 MB	1,308,818	3,695,360

. The two complex baseline models, ET-BERT and ATVITSC, require approximately 115 MB and 106 MB of storage, with around 30 M and 27 M parameters, respectively, and both exhibit FLOPs exceeding 100 M across the four datasets. In contrast, the lightweight baselines YaTC and EAPT have model sizes of about 41 MB and 24 MB, corresponding to approximately 10.7 M and 6.3 M parameters, with FLOPs of about 33 M and 19 M, respectively.

Compared with these models, the proposed model maintains a compact structure of approximately 5 MB with about 1.3 M parameters across all four datasets, while requiring only about 3.7 M FLOPs. This represents a substantial reduction in both model scale and computational cost, achieving nearly an order-of-magnitude compression relative to large transformer-based models and significant reductions compared with other lightweight architectures. These results demonstrate that the proposed approach effectively lowers model complexity, storage overhead, and computational burden while preserving competitive classification performance, thereby further supporting its suitability for deployment in resource-constrained environments.

4.2.3. Ablation Experiments

The performance of the teacher and student models is evaluated in terms of classification accuracy and training time. The results show that the student model learns “knowledge” from the teachers.

The comparison of training time between the student and teacher models is illustrated in Figure 8. GPU training time is analyzed to assess computational efficiency. These substantial differences in training time demonstrate the advantage of the lightweight student architecture. The spatial feature extraction in all three models employs the same CNN structure, meaning the primary differences in training time arise from the temporal feature extraction modules.

As shown in Figure 8, the CNN-BiTCN model incurs the highest training cost due to its bidirectional TCN and attention mechanisms, resulting in the longest training duration. The CNN-BiLSTM model ranks second, reflecting the computational overhead introduced by bidirectional LSTM-based temporal modeling. In contrast, the student model, which relies solely on a CNN architecture, demonstrates the lowest training cost across all datasets. These findings underscore the computational efficiency of the student model, demonstrating its suitability for deployment in resource-constrained environments.

Figure 9 provides a comparative analysis of evaluation metrics for both the student model and teacher models. The proposed model is compared with several related approaches, including a standalone CNN, the first teacher model (CNN-BiTCN), the second teacher model (CNN-BiLSTM), a CNN distilled from CNN-BiTCN, and a CNN distilled from CNN-BiLSTM.

As shown in Figure 9, the proposed model achieves the best overall performance across all datasets. On ISCX-VPN and USTC-TFC, accuracy exceeds 96% and 99%, respectively, with consistent improvements in precision, recall, and $F_1$-score. Competitive and stable performance is also observed on ISCX-Tor and CIC-IoT, where accuracy remains above 97%. Furthermore, student models distilled from CNN-BiLSTM and CNN-BiTCN demonstrate consistent improvements over their standalone counterparts. On ISCX-VPN and USTC-TFC, gains of at least 0.2% are observed, while on ISCX-Tor and CIC-IoT the improvements exceed 1.0%, indicating that temporal feature guidance from the teacher models enhances classification effectiveness. Given the substantial reductions in time costs and computational resource consumption, these findings validate the feasibility and potential of the proposed teacher-student framework in handling traffic classification tasks effectively.

To conclude, the results emphasize the strength of the teacher-student model. It boosts classification performance and remains resource-efficient. The improved metrics and notable reductions in training time confirm its practicality. This approach is highly suitable for network traffic classification in heterogeneous network environments.

5. Conclusions and Future Work

This paper proposes a novel multiclass traffic classification model for network traffic, leveraging multi-teacher knowledge distillation and spatio-temporal feature fusion techniques. The proposed method integrates multi-teacher knowledge distillation with spatio-temporal feature fusion to improve traffic classification performance. By employing a lightweight student network, the framework substantially reduces computational cost while preserving competitive accuracy. Experimental results demonstrate that the proposed model significantly improves accuracy and speed compared to existing methods, outperforming other SOTA models.

Future work will focus on analyzing the underlying causes of reduced classification performance for specific traffic types and further optimizing the model structure and hyperparameters. Although the adopted public benchmarks cover several representative traffic scenarios, some of them were collected relatively early and may not fully reflect the latest characteristics of modern network traffic. Therefore, future research will further validate the proposed framework on more recent large-scale datasets and real-world traffic traces. In addition, adaptive model selection mechanisms will be investigated to dynamically adjust the framework according to varying network conditions and computational constraints. Further exploration of knowledge distillation in practical network environments is also anticipated, particularly in addressing challenges such as training overhead and network load balancing through efficient resource allocation.