Transformer–DDQN-Based Explainable and Active Intrusion Detection Architecture for Network Traffic Analysis

Abstract

This study proposes a novel intrusion detection and response architecture that formulates network traffic analysis as a sequential decision-making problem rather than a static classification task. The architecture integrates a Transformer Encoder for temporal feature extraction with a Dueling Double Deep Q-Network (DDQN) to enable autonomous and risk-aware security decisions. Network flows are modeled within a Markov Decision Process, where the agent learns an optimal policy over a hierarchical action space consisting of IGNORE, LOG, ESCALATE, and BLOCK actions. To evaluate generalization capability, a transfer learning-based cross-domain adaptation strategy was employed. The CICIDS2018 and CICIoT2023 datasets were re-partitioned using a stratified 70/15/15 train/validation/test split. The proposed model achieved high detection performance on these datasets with F1-scores of 99.48% and 99.13%, respectively. After transfer learning to the AWID3 dataset, the model preserved strong generalization capability with F1-scores of 96.76% and 96.61%, demonstrating its robustness across wired, IoT, and wireless network environments. A risk-aware reward function is designed to balance detection accuracy and operational cost, while Integrated Gradients-based explainability is incorporated to analyze decision behavior. Experimental results further show that the proposed Transformer–DDQN framework achieves more stable learning, lower optimization loss, and more consistent action policies compared to alternative reinforcement learning-based approaches. The model operates with high computational efficiency while maintaining real-time processing capability in high-throughput network environments.

1. Introduction

Network security systems have undergone a rapid transformation in recent years due to increasing connection density and the growing complexity of attack vectors. The acceleration of digitalization has also expanded the economic impact of cybercrime, with global economic losses projected to reach 10.5 trillion dollars annually by 2025 [1]. Although signature-based IDS approaches remain computationally efficient and highly effective for known threats, relying solely on predefined patterns limits their ability to detect evolving and zero-day attacks. Therefore, modern cybersecurity environments increasingly require hybrid IDS frameworks that combine traditional detection mechanisms with adaptive learning-based approaches to improve detection robustness and reduce false alarms [2]. Similarly, static machine learning approaches often fail to reach expected detection performance as they cannot sufficiently model the volatile nature and temporal evolution of network traffic. These limitations reinforce the view in the literature that transitioning from passive defense to active strategies—which produce real-time and autonomous responses to threats—is a necessity rather than an option [3].

In the field of cybersecurity, Transformer-based models capable of modeling long-range dependencies in sequential data have accelerated a significant shift through self-attention mechanisms [4,5,6]. Existing studies suggest that network traffic exhibits sequential and contextual dependencies that can be effectively modeled through such architectures and self-attention mechanisms [7,8,9]. From this perspective, the semantic relationships between packets can be represented and analyzed at a higher level of granularity thanks to the Transformer architecture. The fact that attacks in complex network traffic are often spread over time highlights the boundaries of models that perform only passive detection. This increases the demand for systems capable of autonomously learning defense policies through interaction with their environment. Within this framework, Reinforcement Learning (RL), particularly the approach of modeling cyber environments as a Markov Decision Process (MDP) using Deep Q-Networks (DQN), has become prominent [10]. To mitigate the well-known overestimation problem in standard DQN, Dueling and Double DQN (DDQN) architectures make learning more stable and enhance the reliability of autonomous defense mechanisms [11,12,13]. Although Transformer architectures effectively capture contextual dependencies in network traffic, they primarily function as representation-learning models and do not inherently determine optimal security actions. Conversely, reinforcement learning agents can learn adaptive decision policies from environmental feedback, though their performance may be limited when operating directly on high-dimensional raw traffic representations. Therefore, integrating Transformer and DDQN creates a complementary framework in which the Transformer encoder generates contextual state representations while DDQN learns autonomous defense policies over these representations. In this architecture, Transformer serves as a contextual feature extraction component, whereas DDQN functions as an adaptive decision-making mechanism responsible for generating operational actions, including IGNORE, LOG, ESCALATE, and BLOCK.

In this study, we present a hybrid active defense framework that combines a Transformer Encoder-based feature extractor with a Dueling Double DQN agent to analyze behavioral patterns in sequential network traffic windows of 128 packets. Within the architecture, the Transformer module generates high-dimensional contextual representations that serve as input for the DDQN agent. The agent learns the optimal policy within a hierarchical action space including IGNORE, LOG, ESCALATE, and BLOCK. Unlike similar approaches in the literature, this method utilizes a risk-aware reward function [5,12,14]. This function establishes a dynamic balance between the cost of missing an attack and the operational burden of unnecessary intervention, thereby optimizing the defense policy more consistently. In this context, a two-pronged transfer learning strategy was implemented to demonstrate the model’s resilience. General threat patterns were learned from the CICIDS2018 dataset, while contemporary IoT attack characteristics were captured using the state-of-the-art CICIoT2023 dataset. In both scenarios, the learned representations were transferred to the AWID3 dataset, and the cross-protocol generalization performance on wireless traffic was evaluated through extensive experiments. Furthermore, since model transparency and interpretability have become fundamental requirements in modern cybersecurity systems [15,16,17], the reasoning behind decisions was examined using Integrated Gradients (IG) based explainable artificial intelligence (XAI) methods. The analysis shows that the model ignores misleading variables like IP addresses and focuses instead on features that directly reflect attack characteristics, such as flow duration, packet variance, and Inter Arrival Time (IAT) [6,10,18].

The main contributions of this study to the literature are as follows:

• A hybrid active intrusion detection and response architecture is proposed. The architecture models network flows as an MDP and combines a Transformer Encoder-based feature extractor with a Dueling Double DQN agent to learn autonomous security policies over IGNORE, LOG, ESCALATE, and BLOCK actions.
• A risk-aware reward function is designed to balance the cost of missed attacks against the operational burden of unnecessary intervention while reducing false alarm rates.
• A transfer learning-based domain adaptation is established by migrating knowledge from both CICIDS2018 and CICIoT2023 datasets to the AWID3 dataset, with experimental results validating the model’s generalizability across wired, wireless, and IoT networks and diverse protocols. Furthermore, IG and self-attention analyses demonstrate that decisions are rooted in behavioral features like flow duration, packet variance, and IAT rather than misleading cues such as IP addresses.

The remainder of this paper is organized as follows. Section 2 reviews the related work. Section 3 presents the proposed methodology in detail. Section 4 describes the experimental study and analyzes the results. Section 5 discusses the findings and their implications. Finally, Section 6 concludes the paper.

2. Related Work

In cybersecurity, intrusion detection and prevention approaches have shifted over time from traditional machine learning to deep learning models that generate stronger representations. Recently, this line has expanded from systems that perform only detection toward reinforcement learning-based autonomous structures capable of generating state-dependent actions. In this section, sequential modeling and hybrid deep learning approaches, modeling of long context in Transformer-based IDSs, autonomous defense strategies with deep reinforcement learning, as well as XAI and transfer learning studies are addressed.

2.1. Sequential Modeling and Hybrid Deep Learning Approaches

The time-series characteristic inherent in network traffic has led Recurrent Neural Networks (RNNs) and hybrid architectures to be accepted as the standard in the literature for a long time. When studies in the literature are examined, Shaikh et al. [11] supported CNN and LSTM models with DQN and PPO algorithms, focusing on IoMT networks. Rasel and Peter [19] successfully applied LSTM-based anomaly detection in multi-cloud scenarios. Cao et al. [18] present a hierarchical classification model through the integration of 1D-CNN and Bidirectional GRU (BiGRU). However, as pointed out by Prasanga et al. [20], these traditional sequential architectures face the ‘vanishing gradient’ problem in very long sequences and remain weak in representing long-term dependencies. At this point, Anis et al. [1] make a critical observation, arguing that the sequential (serial) processing nature of RNN-based models increases training costs, while Transformer architectures overcome this bottleneck with the advantage of parallelization.

2.2. Transformer-Based IDS and Long-Context Modeling

The quest to overcome the chronic limitations of RNN structures has enabled Transformer architectures to become dominant in this field in recent years. Ma et al. [21] demonstrated that the self-attention mechanism is a new reference point in anomaly detection and that inter-packet relationships can be modeled independently of distance. RTIDS [5], proposed by Wu et al., focuses on noise resilience, while Rajkumardheivanayahi et al. [6] succeeded in capturing long-term behavioral patterns in network flows with the FlowTransformer framework. AboulEla and Kashef [7], instead of generating an autonomous actions against detected threats, keep the system at the level of a passive alarm mechanism. Our study aims to address this deficiency by preserving 128-step context windows directly on the raw data, without resorting to dimensionality reduction methods observed in similar studies such as Hasan et al. [22], which can compromise temporal integrity.

2.3. Deep Reinforcement Learning and Autonomous Defense

The necessity of developing active defense against threats has moved Deep Reinforcement Learning (DRL)-based solutions to the center of research. In this direction, Wang et al. [23] introduced the Dueling DQN architecture to stabilize policy learning by decomposing the network into separate state-value and advantage functions. However, the tendency to overestimate action values observed in standard DQN structures, which leads to unstable policies, limits the reliability of these models. The Double-DQN (DDQN) and ‘Prioritized Experience Replay’ (PER) mechanisms [13], proposed to overcome the aforementioned problem, offer a more stable framework, especially for critical security applications with low fault tolerance. In studies focusing on the synergy between Transformer and RL. Lin [8] proposed the TransRL framework, integrating self-supervised learning, Transformer-based feature modeling, and reinforcement learning for adaptive network security detection. Bhutto et al. [24] constructed the Reinforced Transformer structure. Bakhshad et al. [25] aimed to increase DRL performance through feature selection, and Danasabe et al. [26] through quantum optimization techniques. Junbo [27] modeled long-term dependencies in complex network traffic by combining Transformer-based feature extraction with the DQN structure via the BFQ-GN algorithm. Janardhanan [28] proved the success of DQN in cyber threat modeling and dynamic resource allocation with an 80% defense optimization. Unlike these studies, our model does not use the RL structure only as an auxiliary optimization tool. It positions it as the main decision-making authority that directly transforms contextual representations from the Transformer into actions (IGNORE, LOG, ESCALATE, BLOCK).

2.4. Explainability (XAI) and Transfer Learning

The widespread use of deep learning models has brought with it the requirement for transparency in the decision processes of these systems. Charmet et al. [17] describe the use of XAI in the cybersecurity field as a necessity. Rodríguez et al. [15] proposed methods to mitigate the ‘black box’ character inherent in the nature of Transformer models. Bilal et al. [16] integrated Federated Learning and Explainable Artificial Intelligence (XAI) techniques for malicious traffic detection in distributed IoT environments, emphasizing privacy-preserving learning and model transparency. On the other hand, the subject of the portability (robustness) of models to different network topologies has been discussed by Wang et al. [29]. Ullah et al. [9] proved the transfer learning success in IoT environments through Swin Transformer. The approach presented in this study, diverging from the current literature, performs the transfer of universal experience acquired from the CICIDS2018 and CICIoT2023 datasets to a different protocol, such as AWID3, and demonstrates the cross-protocol robustness of the autonomous agent with quantitative experimental results.

2.5. Comparative Analysis of Dynamic Decision-Making and Architectural Capabilities

Since the proposed framework formulates intrusion detection as a reinforcement learning-based dynamic decision-making problem, comparison with existing studies should not be limited to conventional classification metrics such as accuracy and F1-score. Although these metrics indicate detection performance, they do not reveal whether a model can generate adaptive security actions, optimize a reward-driven policy, or account for operational response costs. Therefore,

Table 1. Comparative analysis of recent intrusion detection studies and the proposed Transformer–DDQN framework in terms of scope, decision type, action strategy, reward/cost design, and performance outcomes.

Study and Method	Dataset	Scope	Decision Type	Action Strategy	Reward/Cost	Key Results
Tarafdar et al. [2], DQN-DRL	IoT-23; CICIDS2017	IDS + mitigation	Adaptive detection/mitigation	Block/isolate/alert	FP/FN penalty; OC n.r.	Acc: 98.4%; F1: 98.3%
Dehghan and Khosravian [4], ProAPT	CICAPT-IIoT	APT prediction/mitigation	Next-step APT prediction	MITRE-aligned APT steps	Prediction reward; OC n.r.	Acc: 93.8%; F1: 94.15%
Huang [3], Transformer + Double DQN	Real traffic; NSL-KDD	Monitoring + response	RL-supported response	Double DQN response	Response reward; OC n.r.	Detection F1: 95.6%; Response Acc: 93.1%
Al-Doori and Alheeti [30], DQN + PER + MAFS + GCN	CSE-CIC; NSL-KDD	IDS + feature selection	Detection/feature selection	Normal/attack + MAFS/PER	Accuracy–feature reward; OC n.r.	CSE-CIC: Acc 97.8%, F1 97.0%; NSL-KDD: Acc 99.5%, F1 99.3%
Wu et al. [5], RTIDS	CICIDS2017; CIC-DDoS2019	Passive IDS	Static detection	Transformer classification	N/A	CICIDS2017 F1: 98.02%; CIC-DDoS2019 F1: 98.45%
Ma et al. [21], FedT-DQN	4 NF datasets	Federated IDS	Federated RL detection under heterogeneous clients	Normal/anomalous	Correct detection; FN penalty; OC n.r.	Best Acc: 99.22–99.87%
Proposed Transformer–DDQN	CICIDS2018; CICIoT2023; AWID3	Active IDS/IPS + TL + XAI	Dynamic active defense	IGNORE/LOG/ESCALATE/BLOCK	Risk-aware reward + OC	Source F1: 99.48%/99.13%; Transfer F1: 96.76%/96.61%

Note. OC = operational cost; n.r. = not reported; TL = transfer learning; XAI = explainable artificial intelligence.

compares existing IDS studies not only in terms of detection performance, but also in terms of decision type, action strategy, and reward/cost awareness. Several recent studies have incorporated reinforcement learning into intrusion detection. However, their decision-making scope differs considerably. Some studies employ DQN-based mechanisms mainly for classification tasks or feature selection, while others utilize reinforcement learning to predict the next attack stage or optimize binary decisions. Although these approaches improve adaptability, most of them do not explicitly model hierarchical operational response actions or quantify operational cost. In contrast, the proposed Transformer–DDQN framework learns autonomous defense actions over a hierarchical action space consisting of IGNORE, LOG, ESCALATE, and BLOCK. Moreover, the proposed risk-aware reward design and operational cost analysis allow the learned policy to be evaluated not only in terms of classification accuracy but also in terms of operational decision quality. Thus, the contribution of the proposed model is not limited to achieving competitive detection performance. Instead, it lies in jointly supporting hierarchical action-level defense decisions, risk/cost-aware policy learning, transfer-based cross-domain adaptation, and explainable decision interpretation within a unified framework.

The architectural novelty of the proposed framework lies in formulating the intrusion detection problem as an active decision-making problem based on a Markov Decision Process rather than as a static binary classification problem. Within this structure, the Transformer Encoder generates a contextual state vector from traffic windows consisting of 128 packets. The generated state vector is provided as input to the Dueling DDQN agent, which learns a security policy consisting of four actions: IGNORE, LOG, ESCALATE, and BLOCK. In existing Transformer–RL-based IDS studies, the Transformer is generally employed as a feature extractor, while reinforcement learning is used as a limited decision or optimization component. In the proposed framework, the state representation generated by the Transformer Encoder is directly transferred to the Dueling DDQN decision core, and the agent selects the operational security action based on Q-values. Therefore, the model not only performs attack/benign classification but also generates a cost-sensitive security action for each traffic window. Another technical distinction of the proposed framework is that the reward function is not defined solely according to detection performance but is formulated using false negative, false positive, and intervention cost components. Therefore, the agent simultaneously considers the cost of missed attacks and the cost of unnecessary interventions during the learning process. In addition, Transformer representations learned from CICIDS2018 and CICIoT2023 are transferred to the AWID3 dataset, enabling the evaluation of the model’s domain transfer capability across wired, IoT, and wireless traffic environments. Integrated Gradients analysis further identifies which traffic features and temporal steps contribute to the selected actions.

In this study, the novelty does not arise from the independent use of Transformer, DDQN, transfer learning, or explainability components individually, since these components have previously been explored in the literature. The novelty originates from defining these components within a unified active-defense decision framework and optimizing them jointly for autonomous security policy generation. Therefore, the proposed framework defines Transformer-based state representation, Dueling DDQN-based action selection, risk-aware reward design, transfer learning, and explainability components within a unified active-defense decision architecture.

Therefore, the proposed framework integrates these distinct modules into a unified active-defense decision architecture. As summarized in

Table 2. Architectural and functional comparison of existing IDS studies and the proposed framework across key design capabilities.

Study	Transformer	Autonomous RL-Based Decision Policy	Active Response	Risk-Aware Reward	Transfer Learning	XAI	Cross-Domain
DRL Intelligent Cybersecurity [2]	✗	✓	✓	△	✗	✗	✗
RTIDS [5]	✓	✗	✗	✗	✗	✗	✗
TransRL [8]	✓	✓	✓	✗	✗	✗	✗
FedT-DQN [21]	✓	✓	✗	✗	✗	✗	△
IDS-INT [9]	✓	✗	✗	✗	✓	✓	✓
Reinforced Transformer Learning [24]	✓	✓	✓	△	✗	✗	✗
ATQ-IDS [31]	✓	✓	✓	✗	✗	✗	✗
Swin-LSTM-TL IDS [32]	✓	✗	✗	✗	✓	✗	✓
XAI-Driven IDS [33]	✗	✗	✗	✗	✗	✓	✗
Transformer + XRL IDRS [34]	✓	✓	✓	△	✗	✓	✗
Our Study	✓	✓	✓	✓	✓	✓	✓

Note. ✓ = fully supported; △ = partially supported; ✗ = not supported.

, most existing studies cover only some of these capabilities, whereas our approach consolidates them to simultaneously leverage representation learning, autonomous decision-making, and explainability.

To further clarify the technical originality of this unified design, the proposed architecture is additionally supported by a component-wise ablation analysis. We distinguish the role of each major component, including the Transformer encoder, the DDQN decision module, transfer learning, and reward adaptation. In this way, the contribution of the framework is not presented only as a combination of existing modules but is explicitly examined in terms of contextual representation learning, autonomous response capability, policy optimization, and computational efficiency.

3. Methodology

In this section, the technical details, data pre-processing steps, and training protocols of the proposed Transformer-DDQN hybrid architecture are presented. The aim of the study is to detect complex attack patterns in network traffic. The proposed approach combines a feature extractor that captures temporal dependencies in network flows with an RL agent that considers the operational cost balance in a single framework. In the following subsections, the methodology is explained through the dataset design, the transfer learning process, and the mathematical basis of the decision mechanism.

3.1. Datasets and Preprocessing

A hierarchical dataset strategy was followed for the proposed Transformer-DDQN architecture to learn large-scale wired network attack patterns and to adapt to heterogeneous wireless network protocols. In the primary training and validation stages, the CICIDS2018 dataset provided by the Canadian Institute for Cybersecurity reflects the modern network topologies was used. This set, containing current attack vectors such as DoS, DDoS, botnet, and infiltration, was preferred for the model to learn a strong initial representation [35]. To evaluate the domain adaptation capability, the AWID3 dataset was used in the transfer learning stage [9,32,36]. The fact that the AWID3 dataset contains wireless layer-specific threats, such as WPA2 password cracking and deauthentication, provided a suitable evaluation ground for measuring cross-protocol robustness [36,37].

AWID3 was selected as a cross-domain benchmark because it was specifically designed for modern IEEE 802.1X/WPA2-Enterprise wireless networks with Protected Management Frames (PMF) enabled. Thereby representing realistic enterprise Wi-Fi environments. In addition to classical wireless-layer threats, AWID3 also incorporates selected higher-layer IEEE 802.3 attack traffic, enabling cross-layer intrusion analysis. In the implementation used in this study, the processed AWID3 CSV files contain 254 columns, consisting of 253 traffic-related attributes and one label field. Transfer-learning experiments were conducted after column normalization, schema alignment, and binary label transformation (benign = 0, attack = 1), using representative wireless intrusion subsets including Malware and Botnet scenarios. This design enabled the pretrained Transformer encoder learned on CICIDS2018 to be adapted to heterogeneous wireless traffic conditions [38].

To further strengthen the practical relevance of the proposed framework with a recent benchmark, additional experiments were also conducted on the CICIoT2023 dataset. CICIoT2023 was generated in a realistic IoT cyber-range containing 105 heterogeneous IoT devices communicating through multiple protocols such as Wi-Fi, Zigbee, and Z-Wave, thereby reflecting contemporary smart-home and industrial IoT environments. The dataset includes 33 attack types grouped into 7 major categories: DDoS, DoS, Reconnaissance, Web-based, Brute Force, Spoofing, and Mirai-family attacks. The inclusion of CICIoT2023 complements the CICIDS2018 wired-network evaluation and the AWID3 wireless transfer-learning stage, thereby providing a broader assessment of generalization capability across contemporary cybersecurity environments [39].

A multi-step pre-processing protocol was implemented to harmonize the heterogeneous nature of the CICIDS2018, CICIoT2023, and AWID3 datasets. First, column names were converted to a common format. Then, feature alignment was performed across datasets based on common attributes through column standardization, intersection, and reindexing procedures. To limit the dimensionality problem (curse of dimensionality), a Random Forest-based feature selection approach with 300 trees was adopted. As a result of the analysis, features with a cumulative importance score exceeding the threshold of 0.95 were retained, and the input space was narrowed [10,20]. Random Forest-based feature selection was preferred instead of transformation-based dimensionality reduction approaches such as Principal Component Analysis (PCA). Preserving the original semantic meaning of network traffic attributes was considered important for subsequent explainability analyses. While PCA generates latent variables through linear combinations of features, Random Forest directly estimates feature importance within the original feature space and therefore maintains interpretability. In addition, Random Forest can capture nonlinear relationships and interaction effects among network traffic attributes, which frequently occur in intrusion detection datasets. A forest size of 300 trees was selected to provide stable feature-importance estimation while maintaining computational efficiency. Furthermore, a cumulative importance threshold of 95% was adopted to preserve most discriminative information while removing low-contributing and potentially noisy variables. This strategy provides a balance between dimensionality reduction, computational efficiency, and feature interpretability.

The StandardScaler method was used for normalization, and scaling parameters were calculated only over the training sections of the source datasets. The preprocessing pipeline was consistently applied across the test data and transfer-learning stages, while scaling parameters were fitted only on the corresponding training partitions to reduce the risk of data leakage. Thus, the obtained performance indicates that the model relies on transferable representations rather than data memorization [9,10].

In the input layer of the proposed architecture, raw network traffic data from the datasets were passed through a multi-stage pre-processing pipeline to make them suitable for complex deep learning processes. In the first stage, infinite (+Inf, −Inf) and missing (NaN) values, which are frequently encountered in network flow records and disrupt the gradient convergence of the model, were cleaned by filling them with a constant of 0.0 to ensure numerical stability. To optimize dimensionality reduction and computational efficiency, a ‘Gini Importance Score’ analysis was performed via the Random Forest algorithm, and the dataset was purified from noise by selecting the most distinctive features representing 95% of the total information gain. The selected features were subjected to Z-score normalization by pulling them to the unit variance and zero mean plane using the StandardScaler method. To transform static tabular data into a temporal storyline, a ‘Sliding Window’ mechanism with a length of L = 128 and a step shift parameter of ST = 1 was constructed. The sequence length of L = 128 was selected to balance contextual coverage and computational efficiency. Shorter sequence windows may provide insufficient temporal context and limit the representation of long-range behavioral dependencies, whereas substantially larger windows may introduce redundant information and increase computational overhead without necessarily improving learned representations. Furthermore, transfer-learning experiments across CICIDS2018, CICIoT2023, and AWID3 demonstrate that this sequence length effectively preserves meaningful representations under heterogeneous traffic environments. However, completely unseen attack patterns that are not represented in existing datasets remain outside the direct empirical scope of the present benchmark-based evaluation. Selecting a step shift value of 1 allows the windows to overlap with one-packet differences, enabling continuous scanning of network traffic. This high-resolution sampling strategy reduces the risk of missing critical attack patterns that could be located between two adjacent windows and provides the Transformer Encoder with a rich training set for analyzing micro-behavioral changes in wired, wireless, and IoT network traffic.

3.2. System Architecture

The Transformer–DDQN hybrid architecture treats network traffic not as a static classification problem, but as a dynamic decision-making process. The proposed approach systematically combines contextual representations generated by a context encoder, which resolves complex temporal dependencies in network flows, with an RL agent that dynamically generates adaptive security decisions according to contextual information and environmental feedback [8]. The architecture aligns directly with the self-attention mechanisms emphasized by Wei [27] and the autonomous decision framework proposed by Janardhanan [28], which processes the cybersecurity state as a feature vector (S). Operationally, the architecture is based on modeling network traffic on an MDP plane. In this modeling, each traffic sequence represents the ‘state’ space, the operational responses the system can deploy represent the ‘action’ space, and the accuracy of the decisions represents the ‘reward’ signal that feeds the system [10,14,40]. As stated in the study by Janardhanan [28], this type of MDP model allows for the dynamic allocation of cyber resources and enables the system to provide continuous adaptation against evolving threat vectors.

Figure 1 illustrates the two-pronged transfer learning architecture that combines Transformer-based representation learning with the Dueling Double DQN decision core. The transfer learning process is implemented through a selective encoder-transfer strategy rather than transferring the entire Transformer–DDQN model. In the source stage, the Transformer encoder and the Dueling DDQN decision core are trained jointly on CICIDS2018 or CICIoT2023. After pretraining, only the learned Transformer encoder weights are exported as a source encoder checkpoint. In the AWID3 stage, the target Transformer–DDQN model is initialized by loading only compatible source encoder parameters. To preserve the general temporal representations learned from the source domain, most encoder layers are frozen. The input projection layer, when required by feature-dimensional differences, the final Transformer block, and the Dueling DDQN Core remain trainable so that the model can adapt to the feature distribution and decision policy of AWID3. During this fine-tuning process, only trainable parameters are updated, while the XAI module provides heatmap-based rationale analysis for the final security decisions.

In the first stage, source datasets (CICIDS2018 and contemporary CICIoT2023) are cleaned, selected features are scaled, and the RL loop is executed within CyberSequenceEnv by generating sequential observations with an L = 128 sliding window. Subsequently, the pretrained Transformer encoder weights are exported as a source encoder checkpoint. In the second stage, AWID3 is prepared with the same preprocessing and sliding-window flow, and the target Transformer–DDQN model is initialized using only compatible source encoder parameters. During AWID3 fine-tuning, most encoder layers are frozen, while the input projection layer, the final Transformer block, and the Dueling DDQN Core remain trainable for target-domain adaptation. The architecture reports which time steps and features the decisions are based on via heatmaps using the IG and self-attention-based explainability component.

The data input layer of the architecture, remaining faithful to the chronological nature of network traffic, processes over time-ordered sequences rather than independent records. The raw data transferred from the pre-processing phase is divided into blocks of L = 128 time steps in length using the sliding-window technique. As a result of this process, the data is transformed into inputs in matrix form that the Transformer Encoder unit can analyze with high resolution [5,6,9].

The ${Online}_q$ model learns temporal patterns by reading network traffic through a fixed-length window. In this window, L represents the number of packets provided to the model simultaneously. St denotes how many packets the sliding window is advanced during each update. The CyberSequenceEnv environment produces a raw observation sequence with a length of L = 128 at each step. The next observation is generated by shifting the window by St packets. As St decreases, consecutive observations overlap more, and the model captures behavioral changes with finer time resolution. As St increases, computational cost decreases, but short-term anomalies may be missed more easily.

The decision mechanism ${Online}_q$ is a hybrid deep reinforcement learning structure that combines Transformer–Encoder with Dueling DDQN. The Transformer Encoder block takes the L-packet observation matrix from CyberSequenceEnv as input and begins processing.

The positional encoding layer carries chronological order information to the model by representing the position of each packet within the sequence with sinusoidal components. In this way, the model can distinguish the order in the packet flow and the inter-arrival time of packets. Multi-head self-attention extracts contextual dependencies by scoring the relationship of each packet in the sequence with other packets through Query, Key, and Value representations. The output is refined with feed-forward layers and residual connections and transformed into a high-level representation vector. This representation is decomposed into value and advantage components in the Dueling DDQN structure, supporting more stable learning of action values.

The contextual embedding vector obtained from the Transformer unit is transferred to the Dueling DDQN Decision Core, which is the strategic decision layer. Unlike classical deep Q-learning, this core splits the decision calculation into two parallel streams. The Value stream (V) calculates the overall value of the current network state independently of the selected action and summarizes the risk level of the system at that moment. The Advantage stream (A) separately scores the relative impact of options in the hierarchical action space (IGNORE, LOG, ESCALATE, BLOCK) within the same state. These two streams are combined to obtain Q-values, and the policy selects the action with the highest expected reward. Since the cost of intervention is also taken into account when traffic is secure, unnecessary actions are reduced and operational efficiency is preserved.

The online network, which generates decisions in the DDQN structure, continuously updates its weights during training. The target network, copied from the online network at specific intervals, limits oscillations and overestimation errors observed in learning by keeping target Q-values more stable. The decision signal generated by the {Online}_q model is directly transmitted to the Security Actions hierarchy, and the system executes an active defense policy that establishes a dynamic balance between cyber risk and the cost of false positives.

3.2.1. Weight Update and Reward Loop

Each security decision selected by the model from the hierarchical action space is scored instantaneously by the Risk-Aware Reward mechanism. This mechanism balances the gain from preventing an attack with the false positive cost generated by the erroneous blocking of normal traffic within the same reward function. The produced reward or penalty signal is the direct input of the weight update process. With this feedback, the agent learns a Q-policy over time that reduces risk and maintains business continuity. During training, ${Online}_q$ takes past experience sampled from the Replay Buffer as mini-batches and uses the quality signal produced by Risk-Aware Reward for each experience. These data are jointly involved in calculating target Q-values and deriving the loss function. Subsequently, network weights are updated via gradient descent. This arrangement prevents the model from reacting only to the latest observations. Learning proceeds in a way that simultaneously considers both the diversity of past samples and the risk and cost profile of each action.

3.2.2. Save Encoder Weights

At the end of Stage 1 training, when the model reaches the targeted performance on the source datasets (CICIDS2018 and CICIoT2023), the weights of the Transformer-Encoder block are saved. The Save Encoder Weights step refers to the persistence of the learned parameters representing the temporal dependencies of traffic and inter-packet relationships. This process is necessary to preserve the representational power obtained during training and is used as the starting point for the transition to Stage 2. When the model is transferred to AWID3 in Stage 2, these encoder weights are not loaded as a complete model without compatibility checking. Instead, only source encoder parameters that are compatible with the target encoder structure are loaded, while incompatible parameters are skipped. This prevents architectural or feature-dimensional mismatches from corrupting the target-domain adaptation process. Thus, training is not started from scratch but fine-tuned on previously learned patterns. This transfer aims to reduce the domain gap between wired, IoT, and wireless (AWID3) network environments and to achieve faster convergence with fewer samples.

3.2.3. Encoder Freezing and Fine-Tuning

During the transfer-learning stage, a selective fine-tuning strategy was adopted rather than retraining the entire Transformer–DDQN model from scratch. After source-domain pretraining on CICIDS2018 or CICIoT2023, the pretrained Transformer encoder weights were exported as a source encoder checkpoint. In the AWID3 transfer stage, only compatible source encoder parameters were loaded into the target encoder, while incompatible parameters were skipped to avoid architectural or feature-dimensional mismatches.

Most pretrained encoder parameters were frozen to preserve the general temporal representations acquired during source-domain pretraining and to reduce catastrophic forgetting. Selected components were kept trainable for adaptation to the AWID3 target domain. Specifically, the input projection layer, which handles feature-dimensional adaptation, the final Transformer encoder block, and the Dueling DDQN Decision Core were optimized on AWID3 samples, while the remaining encoder layers continued operating as fixed feature extractors. This strategy allows low-level temporal representations to remain stable while enabling high-level contextual features and the target-domain security policy to adapt to the new wireless traffic distribution. During fine-tuning, only trainable parameters were updated by the optimizer.

3.3. Proposed Model

The proposed framework was developed to transform network traffic analysis from a static classification task into a dynamic decision-making process. The novelty of the proposed framework lies in the integration of a Transformer Encoder, which deeply analyzes temporal dependencies in 128-packet windows, with a Dueling DDQN agent that transforms the representations obtained from these analyses into autonomous security decisions. By modeling network flows within a Markov Decision Process (MDP) framework, the architecture evolves traditional passive detection mechanisms into an active and proactive defense strategy [10,40]. This designed ecosystem is reinforced with a Transfer Learning strategy in which universal traffic characteristics obtained from joint source datasets (CICIDS2018 and CICIoT2023) are successfully transferred to the wireless AWID3 environment, and it is made transparent with an XAI layer that makes the mathematical foundations of the decisions visible [17,32,36].

3.3.1. Transformer-Based Contextual State Representation

The feature extractor component of the proposed architecture is based on a specialized Transformer Encoder structure designed to resolve temporal dependencies and multi-scale contextual patterns in network traffic. Unlike classical sequential models (RNN, LSTM), this structure leverages the self-attention mechanism to calculate the correlation of packets within a sequence independently of their distance [5,6,9]. Since the network traffic windows used in our study consist of a medium-scale sequence of 128 time steps (L = 128), the quadratic complexity of the full self-attention mechanism does not pose a disadvantage. On the contrary, it enables the model to gain high expressive power against both local anomaly signals and global attack patterns across diverse wired and IoT-based protocols.

In the input layer of the system, feature vectors from the pre-processing and Random Forest-based selection stages are first transformed into the model’s internal dimension (d_model = 32) via a linear projection layer (input_proj). To maintain the chronological order of the packets within the sequence, positional embedding data is incorporated into each step at this stage. The two-layer Transformer Encoder architecture produces a refined contextual representation by dynamically weighting the interaction of each packet with all other steps. The outputs obtained at the end of the process are subjected to a mean pooling operation to be transformed into a dense contextual state vector (state embedding) that summarizes the entire sequence. The 32-dimensional state summary produced after these operations is transferred as input to the Dueling DDQN agent, which serves as the decision mechanism of the autonomous system [7,21,35].

3.3.2. Dueling Double Deep Q-Network Decision Mechanism

The decision-making unit, which manages the security of the network as an autonomous authority, is built upon the Dueling DDQN architecture [41]. To eliminate the tendency of overestimating action values (overestimation), which has become a chronic problem in traditional DQN structures, the processes of action selection and value evaluation are functionally distributed between ‘Online’ and ‘Target’ networks [11,12,13]. To reinforce learning stability, the Dueling structure we preferred calculates the action-value function $(Q$) through two separate channels as the state-value function $V\left(s\right)$ and the action advantage function $A\left(s,a\right)$ This decoupling strategy enables the agent to more quickly grasp which states are critical and which action makes a difference in those states. Thus, it is made possible for the system to develop a much more consistent policy, especially in cases where the choice of action does not affect the final outcome. The state-action value $\left(Q\right)$ is combined on a mathematical plane with the following formulation:

$$Q\left(s,a\right)=V\left(s\right)+\left(A\left(s,a\right)-{\displaystyle \frac{1}{\left|\mathcal{A}\right|}}\sum _{a^{\prime }}A\left(s,a^{\prime }\right)\right)$$

(1)

Referencing the 32-dimensional state vector received from the Transformer unit, the agent makes decisions within a discrete action space consisting of four main categories. This hierarchical space covers the actions of IGNORE (0) for non-risk-posing flows, LOG (1) for tracking suspicious activities, ESCALATE (2) for reporting critical anomalies, and BLOCK (3) for the direct termination of detected active attacks [14,21,23].

3.3.3. Risk-Aware Reward Function

The reward function R, which governs the learning dynamics of the proposed model, is designed to balance security risk and operational cost in cybersecurity environments. The primary objective is to strongly penalize false negatives (i.e., missed attacks) while explicitly modeling the cost associated with false positives, such as unnecessary interventions on benign traffic.

As summarized in

Table 3. Risk-aware reward matrix showing action-specific rewards and penalties for benign and attack traffic during pretraining and fine-tuning.

Traffic State	Action	Pretraining	Fine-Tuning
Benign	IGNORE	+30.0	+20.0
	LOG	−10.0	0.0
	ESCALATE	−1.0	+0.5
	BLOCK	−100.0	−150.0
Attack	IGNORE	−150.0	−200.0
	LOG	+1.0	+0.1
	ESCALATE	+4.8	+4.95
	BLOCK	+5.0	+5.0

, the reward structure is defined separately for benign (y = 0) and attack (y = 1) traffic conditions. During the pretraining stage, the agent is encouraged to correctly ignore benign traffic with a high positive reward (+30.0), while incorrect blocking is heavily penalized (−100.0), reflecting the critical impact of service disruption. Intermediate actions such as LOG (−10.0) and ESCALATE (−1.0) introduce graded penalties to discourage unnecessary inspection while still allowing flexible decision-making. For attack scenarios, the reward design prioritizes effective threat mitigation. The BLOCK action yields the highest reward (+5.0), reinforcing direct neutralization, whereas choosing IGNORE incurs a severe penalty (−150.0), representing the high risk of missed attacks. Intermediate actions, LOG (+1.0) and ESCALATE (+4.8), support a hierarchical response strategy based on threat severity. The proposed framework follows a two-stage learning procedure. In the first stage, the agent is trained on heterogeneous source-domain datasets (CICIDS2018 and CICIoT2023) using the base reward configuration. In the second stage, the pretrained model is adapted to the AWID3 target domain through transfer learning. To account for domain-specific characteristics and differing traffic distributions, the reward structure is refined during fine-tuning. Specifically, for benign traffic, the reward assigned to the IGNORE action is reduced from +30.0 to +20.0 to maintain stability while avoiding overconfidence. The penalty for ESCALATE is relaxed from −1.0 to +0.5, enabling more flexible intermediate responses, whereas the penalty for BLOCK is increased in magnitude from −100.0 to −150.0 to more strongly discourage false blocking in the wireless environment. For attack traffic, the penalty assigned to IGNORE is intensified from −150.0 to −200.0, emphasizing the critical importance of avoiding missed detections in the target domain. The rewards for LOG and ESCALATE are adjusted to +0.1 and +4.95, respectively, enabling finer-grained adaptation of response strategies. The reward for BLOCK remains unchanged at +5.0, preserving the learned preference for direct mitigation of malicious activity. This staged reward adaptation allows the model to retain the core defense policy learned during source-domain training while adjusting its decision boundaries to better align with the operational characteristics of the target environment.

As justified in

Table 4. Risk hierarchy and reward design rationale based on operational severity and reward assignment logic.

Event Type	Operational Impact	Relative Severity	Relative Cost Weight	Reward Design Logic	Example Reward Trend
Missed attack (False Negative)	Malicious traffic remains undetected and may compromise system security	Critical	5	Missing attacks create the highest security risk and therefore receive the strongest penalty	Highest negative value
Unnecessary blocking (False Positive)	Legitimate users or services may be interrupted	High	3	False alarms cause operational disruption but generally create a lower impact than missed attacks	Moderate negative value
Escalation action	Requires additional verification and resource utilization	Moderate	1	Escalation introduces additional processing cost while preserving decision flexibility	Slight negative/neutral
Logging action	Generates monitoring overhead only	Low	0.5	Logging provides passive monitoring with minimal operational burden	Minimal penalty
Correct attack mitigation	Malicious activity is successfully prevented	Positive	–	Effective mitigation strengthens security posture and should be positively reinforced	Positive reward

, the reward coefficients were designed according to the relative operational severity associated with security outcomes. In practical cybersecurity environments, missed attacks generally create more severe consequences than unnecessary interventions caused by false positive events. Therefore, the reward structure was organized as a hierarchical risk-aware strategy prioritizing critical security failures over lower-impact operational actions. During the pre-training phase, reward coefficients were initialized according to proportional operational priorities, where missed attacks received the strongest penalties while intermediate actions such as ESCALATE and LOG were assigned moderate values. During the fine-tuning stage, these coefficients were further adjusted based on reward sensitivity analysis to better reflect high-risk cybersecurity conditions. Specifically, the penalty for missed attacks was increased from −150.0 to −200.0, while the penalty for unnecessary blocking was increased from −100.0 to −150.0. Correct escalation actions under attack conditions were assigned positive rewards (4.8–4.95) to encourage controlled intervention under uncertainty.

Also, across the CICIDS2018, CICIoT2023, and AWID3 experiments, an over-budget penalty of −0.3 was applied when the predefined intervention limits were exceeded. The maximum numbers of ESCALATE and BLOCK actions were limited to 20 and 10 per episode, respectively. These constraints were introduced to discourage excessive intervention, reduce unnecessary operational cost, and encourage the agent to learn a balanced response policy rather than overusing high-impact mitigation actions.

3.3.4. Hyperparameter Configuration and Optimization

The core hyperparameters determining the training performance of the Transformer–DDQN model were configured in line with the temporal dynamics inherent in network traffic and the conducted experimental observations. A Transformer encoder architecture was utilized to effectively analyze packet sequences consisting of 128 time steps. The Adam optimization algorithm was employed for the stable and precise updating of model weights. The initial learning rate was set to $1\times {10}^{-4}$, while the discount factor (γ), which controls the importance the agent assigns to long-term rewards, was held constant at 0.99 to maintain system stability [12,14]. An $ϵ$-greedy strategy was adopted to manage the exploration–exploitation balance in the decision-making process. In this context, the ε parameter was configured to exhibit a linear decay from 1.0 toward 0.1 in the baseline setting and 0.05 in the transfer learning stage throughout the training process. The length of the epsilon decay period was adjusted to 100,000 steps for baseline training and 50,000 steps for transfer learning, ensuring adequate exploration across different environments, from traditional networks to IoT ecosystems. This approach enabled the agent to explore the network environment from a broad perspective in the early stages of training and to focus on learned optimal policies in later stages. This gradual decay strategy contributed to a more stable and balanced learning process by reducing the risk of the model getting trapped in local optima during complex and variable cyberattack scenarios.

All hyperparameter configurations shaping the model’s operational capacity and learning dynamics are presented collectively in

Table 5. Hyperparameter settings used for Transformer-based sequence modeling, Dueling DDQN policy learning, replay-buffer training, and exploration scheduling.

Category	Hyperparameter	Value	Description
Transformer	Window Size $\left(L\right)$	128	Sequence length of input packets
	Hidden Size $\left(d_{model}\right)$	32	Transformer embedding dimension
	Encoder Layers/Heads	2/4	Depth and parallel attention count
Dueling DDQN	Hidden Size	128	Neurons in decision core layers
	Optimizer/Learning Rate	Adam/0.0001	Optimization algorithm and step size
	Discount Factor $\left(\gamma \right)$	0.99	Importance of future rewards
RL Training	Batch Size	64	Samples per training step
	Replay Buffer Size	100,000	Experience storage size
	Minimum Buffer Size	15,000	Minimum number of stored experiences before training
	Target Network Update	1000	Interval for target sync (steps)
Exploration	Epsilon Strategy	Linear Decay	${\epsilon }_{start}$ = 1.0 → ${\epsilon }_{min}$; The number of decay steps is provided in the following row.
	Minimum epsilon (${\in }_{min})$	0.10/0.05	Exploration boundaries
	Epsilon Decay Steps	100,000/50,000	Total steps for epsilon reduction

. The epsilon decay duration and fine-tuning scope were standardized separately for each dataset, considering traffic density and distributional complexity.

3.3.5. Reproducibility and Experimental Settings

To ensure full reproducibility, all preprocessing pipelines, data partitions, random seeds, checkpoints, and stopping criteria were standardized across all experiments. Deterministic behavior was enforced by fixing random seeds across Python, NumPy, and PyTorch/CUDA environments, ensuring consistent data shuffling, parameter initialization, and sampling processes. In the CICIDS2018 experiments, all publicly available CSV files were merged and preprocessed to construct a unified dataset comprising over 1.7 million records. Only numerical features were retained, and infinite values were converted to missing values and subsequently replaced with zeros to ensure numerical stability. Feature selection was performed using a RandomForestClassifier with 300 trees, and features contributing to 95% cumulative importance were preserved, resulting in 120 selected attributes. The dataset was partitioned using a stratified 70/15/15 split for training, validation, and testing, respectively (random_state = 42), generating 1,212,131 training samples, 259,743 validation samples, and 259,743 test samples while preserving class distributions. The fitted scaler, selected-feature list, and trained feature-selection model were saved as reusable artifacts.

For the CICIoT2023 dataset, after preprocessing and balancing, the data were partitioned using a stratified 70/15/15 split, resulting in 249,853 training samples, 53,540 validation samples, and 53,540 test samples. Feature names were standardized to ensure consistency across partitions. The same preprocessing pipeline used in CICIDS2018 was applied, including missing-value handling and normalization. Feature selection using the same Random Forest protocol resulted in a feature set of 46 numerical attributes, reflecting the compact and domain-specific nature of IoT traffic patterns. The Transformer-DDQN framework was trained using the training partition, while the validation partition was used for checkpoint monitoring and model selection. The independent test partition was strictly excluded from training and used only for final evaluation to prevent data leakage.

In the AWID3 transfer-learning stage, the dataset contained 1,429,971 samples after preprocessing and was partitioned using a stratified 70/15/15 split, resulting in 1,000,979 training samples, 214,496 validation samples, and 214,496 test samples. Due to heterogeneous schemas across files, column names were normalized and aligned through column union and reindexing before merging into a unified dataframe. Target labels were converted into binary classes (benign = 0, attack = 1), and feature selection yielded 80 relevant attributes. The same preprocessing pipeline used for CICIDS2018 was consistently applied, including normalization and temporal sequence construction. Transfer learning was performed by initializing the Transformer encoder with pretrained weights obtained from CICIDS2018 and fine-tuning on AWID3 data.

Across all datasets, features were normalized using a StandardScaler fitted exclusively on the training data to prevent data leakage. Input data were transformed into temporal sequences using a sliding window mechanism with length L = 128 and step shift ST = 1, enabling fine-grained temporal modeling of network behavior. The reinforcement learning component was trained using the Adam optimizer (learning rate = 1 × 10⁻⁴), a replay buffer size of 100,000, and a target network update interval of 1000 steps.

Exploration was controlled via a linear epsilon decay strategy, where ε decreased from 1.0 to 0.1 during baseline training and from 1.0 to 0.05 during transfer learning. The decay duration was set to 100,000 steps for baseline experiments and 50,000 steps for transfer learning. To reduce stochastic variability and ensure statistical robustness, all experiments were repeated five times using different random seeds (42, 53, 62, 72, and 82). Early stopping was employed to prevent overfitting and unnecessary training. Training was terminated when validation performance did not improve for 50 consecutive episodes, and the best-performing checkpoint was automatically restored for final evaluation.

3.3.6. Transfer Learning and Domain Adaptation Strategy

The transfer-learning process followed a two-stage domain adaptation protocol. In the first stage, the Transformer–DDQN architecture was trained end-to-end under CICIDS2018 or CICIoT2023 source-domain pretraining scenarios. During this phase, all encoder and decision-network parameters were jointly optimized to capture robust representations across wired and IoT environments. In the second stage, the learned knowledge was transferred to the AWID3 target domain representing wireless-network traffic [9,32,36]. Domain adaptation was performed using the selective fine-tuning strategy described in Section 3.2.3. This approach preserved low-level temporal representations while allowing higher-level contextual features and decision policies to adapt to the target-domain distribution. Consequently, representation learning and policy learning remained functionally decoupled while maintaining a balance between knowledge retention and domain-specific adaptation, thereby reducing catastrophic forgetting and improving cross-domain robustness across heterogeneous network protocols [29,37].

3.3.7. Decision Explainability and Transparency (XAI)

To make the decision mechanism more transparent, the Integrated Gradients (IG) method has been integrated into the model. Thus, the quantitative impact of each attribute on action selection within every 128-time-step traffic window is calculated [15,17,42]. The “ladder-XAI” approach proposed in this study does more than just provide a justification for the selected action; it also decomposes the Q-value differences between competing actions at the attribute level.

Heatmaps and temporal importance curves demonstrate that behavioral variables such as flow duration, packet variance, and Inter-Arrival Time (IAT) are the primary signals driving the decisions. This confirms that the model identifies threats not through superficial indicators like IP addresses, but through patterns spread over time [6,10]. The resulting transparency improves the interpretability and auditability of autonomous defense decisions [16,43].

3.3.8. Operational Mechanism and Training Procedure

The implementation of the proposed model is structured across two main phases. In the first stage, defined as “Phase I,” the system was trained end-to-end on a joint source domain consisting of CICIDS2018 and CICIoT2023 datasets. During this phase, the Transformer Encoder and the DDQN agent were optimized simultaneously. This joint optimization ensured that the representations carrying the cyber defense decisions were learned more consistently and deeply across both wired and IoT network environments. In the “Phase II” stage, a transfer learning strategy was applied to adapt to wireless network Dynamics using the AWID3 dataset [36]. In this phase, representation learning and policy development steps were deliberately decoupled; the pre-trained Transformer encoder weights were frozen, thereby reducing computational costs while increasing cross-domain adaptation performance. The resulting methodology aims to directly address the lack of autonomous defense frequently highlighted in the literature by combining the temporal contextual power of the Transformer with the decision-making capability of the DDQN agent [8,44,45].

Algorithm 1 pre-processes the joint source datasets (CICIDS2018 and CICIoT2023), denoted as $D_C$, where numerical feature matrices $X_C$ and corresponding binary label vectors $y_C$ are extracted. The dataset is partitioned into training, validation, and test subsets using a stratified 70/15/15% splitting strategy to preserve class distributions. A subset of informative features, denoted as $F_C$, is selected based on Random Forest importance scores, and a scaling transformation ${\mathrm{Scaler}}_C$ (StandardScaler) is fitted only on the training data to ensure normalized feature distributions and cross-domain compatibility while avoiding data leakage. Subsequently, the processed data are transformed into sequential samples $S_C$ using a fixed-length sliding window mechanism with sequence length $L=128$. These sequences define the reinforcement learning environment. The Transformer encoder $f_{\theta }$ and the dueling DDQN agent $Q_{\psi }$, where $\psi$ represents the parameters of the value network, are jointly initialized and optimized. During training, the model is iteratively updated over $N_C$ episodes. At regular intervals defined by $V$, a greedy policy is evaluated on the validation set to compute the F1-score. If an improvement is observed, the encoder parameters are updated and stored as the best-performing representation, denoted by ${\theta }^{*}$. The training process is conducted using a replay buffer to stabilize learning and improve sample efficiency. The optimized encoder parameters ${\theta }^{*}$, which capture both wired and IoT traffic representations, are exported and used as a fixed feature extraction component for the second-stage transfer learning process.

Algorithm 1: CICIDS2018 or CICIoT2023: RL Training of Transformer–DDQN and Encoder Freezing

Input: Dataset $D_C$, sequence length $L$, action set $A$ Output: Frozen encoder parameters ${\theta }^{*}$ 1. Load and preprocess $D_C$; extract numeric features $X_C$ and binary labels $y_C$; 2. Stratified split $X_c,y_c$ into training, validation, and test subsets; 3. Select feature subset $F_C$ via Random Forest importance; fit scaler $Scale{r}_C$; 4. Construct sequences $S_C=MAKESEQ\left(Scale{r}_C\right(X_C\left[F_C\right]),L)$; 5. Initialize environment $E_C$ with $S_C$ and $y_C$; 6. Initialize Transformer encoder $f_{\theta }$ and dueling DDQN agent $Q_{\psi }$; 7. for episode $=1$ to $N_C$ do 8.  Train Transformer–DDQN on $E_C$ using replay buffer; 9.    if episode $\left( mod V\right)=0$ then 10.    Evaluate greedy policy on validation set and compute F1-score; 11.    if F1 improves then 12.     Update ${\theta }^{*}$; 13. Final step: 14. Freeze and export encoder parameters ${\theta }^{*}$; 15. return ${\theta }^{*}$;

Algorithm 2 describes the AWID3 transfer-learning procedure, where the target-domain dataset is denoted by $D_A$. Numerical features $X_A$, binary labels $y_A$, and subtype labels $a_A$ are extracted after preprocessing, followed by a stratified train-validation-test split (70/15/15%). The independent test subset was strictly reserved for final evaluation to ensure unbiased performance assessment. The selected AWID3 features are scaled using $Scale{r}_A$ and converted into fixed-length sequences $S_A$ with sequence length $L=128$.

Algorithm 2: AWID3: Transfer Learning with Pretrained Encoder and DDQN Fine-Tuning

Input: Dataset $D_A$, pretrained encoder ${\theta }^{*}$, sequence length $L$, action set $A$ Output: Transfer-trained agent $(f_{{\theta }_A},Q_{{\psi }_A})$ 1. Load and preprocess $D_A$; extract $X_A$, binary labels $y_A$, and subtype labels $a_A$; 2. Stratified split $X_A,y_A,a_A$ into training, validation, and test subsets; fit scaler $Scale{r}_A$; 3. Construct sequences $S_A=MAKESEQ\left(Scale{r}_C\right(X_A),L)$; 4. Initialize environments ${\epsilon }_A^{tr}$ and ${\epsilon }_A^{va}$; 5. Initialize encoder $f_{{\theta }_A}$ and load ${\theta }^{*}$ (shape-compatible parameters only); 6. Freeze encoder layers; unfreeze input projection if feature dimensions mismatch; 7. Optionally unfreeze last Transformer block for domain adaptation; 8. Initialize DDQN head $Q_{{\psi }_A}$ and train on ${\epsilon }_A^{tr}$ using replay buffer; 9. Periodically evaluate greedy policy on ${\epsilon }_A^{va}$ and track F1-score; early stopping if needed; 10. return $(f_{{\theta }_A},Q_{{\psi }_A})$;

The pretrained encoder parameters ${\theta }^{*}$, obtained from source-domain training, are loaded into the target encoder $f_{{\theta }_A}$, while incompatible input projection parameters are excluded when feature dimensions differ. Most encoder layers are frozen, whereas the input projection layer, the last Transformer block, and the dueling DDQN head $Q_{{\psi }_A}$ remain trainable for domain adaptation. The transfer agent $(f_{{\theta }_A},Q_{{\psi }_A})$ is then fine-tuned using replay-buffer samples, and the validation F1-score is monitored periodically to apply early stopping when necessary.

3.3.9. Reward Function and Operational Budget Constraint

In this reinforcement learning environment, the agent does not merely focus on increasing classification accuracy. It is designed to account for operational constraints in real-world network security scenarios. Therefore, the reward function has a multi-component structure that reflects both decision accuracy and the cost imposed on the system by the selected actions. At each time step, the chosen action is compared against the ground truth label to generate a base reward. While correct attack detections are reinforced with positive rewards, false alarms (false positives) and missed attacks (false negatives) are penalized with negative rewards. Additionally, since certain actions incur operational costs (such as blocking traffic or escalating the incident to higher-level security mechanisms), these actions are deducted from a budget defined within the system. When the agent exceeds the budget, an additional environmental penalty (over-budget penalty) is applied to guide learning under realistic constraints. This structure limits the uncontrolled use of expensive and aggressive actions, pushing the agent to learn a more balanced and sustainable policy. Consequently, the reward function encourages the agent not only to target high accuracy but also to develop a resource-aware attack detection strategy that is applicable in the field.

4. Experimental Study and Analysis

In this section, the performance of the Transformer–DDQN-based hybrid IDS architecture is examined through core metrics such as detection success, decision stability, and cross-domain generalization. The experiments aimed to measure the model’s level of learning complex attack patterns across a joint source domain (CICIDS2018 and CICIoT2023) while testing how well the knowledge acquired during this process was transferred to the AWID3 dataset. The results demonstrate that the system offers a detection accuracy competitive with state-of-the-art approaches in the literature. Beyond mere accuracy, the results indicate that the framework effectively balances operational risks while producing a consistent autonomous defense policy [3,45].

4.1. Experimental Setup and Hardware

The training and testing processes of the proposed architecture were conducted on the Google Colab Pro+ cloud infrastructure to meet the high computational demands of deep learning and reinforcement learning algorithms. The software environment was deployed in an Ubuntu 22.04.5 LTS container using Python v3.10. During the experiments, an NVIDIA A100-SXM4 GPU with 40 GB of memory was utilized to accelerate the self-attention matrix operations in the Transformer layers and the experience replay processes of the DDQN agent through NVIDIA CUDA v12.4. The deep reinforcement learning models and Transformer architectures were implemented using the PyTorch v2.5.1 framework, which enables high-performance tensor operations. For the Integrated Gradients analyses used to improve the interpretability of the decision-making mechanisms, the Captum v0.7.0 library was employed. In addition, data preprocessing and feature scaling were performed using Pandas v2.2.2 and Scikit-learn v1.5.2.

4.2. Training Protocol and Early Stopping

During training, both total reward and F1 score were monitored on the validation set post-episode. This approach ensures that the model learns by simultaneously considering both operational objectives (reward) and detection performance (F1). The final model selection was not based on the F1 score alone; instead, the checkpoint with the highest validation total reward was selected. The validation set was used for checkpoint monitoring and early stopping, while the independent test set was used only for final performance evaluation. The F1 score was reported as a complementary indicator for interpreting overall performance. The training protocol was conducted in two stages. In the first stage, the model was trained on a joint source domain (CICIDS2018 and CICIoT2023) to strengthen temporal representation learning across wired and IoT environments. In the second stage, it was adapted to the AWID3 dataset through transfer learning. During this transfer phase, the Dueling DDQN decision core was reinitialized to ensure fast and stable policy learning, while selective fine-tuning was applied to the Transformer encoder. The early stopping rule was defined to automatically terminate training if no significant improvement was observed in validation performance for 50 consecutive episodes. The criteria for improvement were evaluated based on both validation total reward and the F1 score.

Before presenting the dataset-specific performance results,

Table 6. Benchmark-oriented comparison and validation coverage of IDS/DRL-based studies.

Study	Benchmark Dataset (s)	Comparison Type	Comparison Focus	Notes
Shaker et al. [46]	NF-UNSW-NB15, NF-BoT-IoT, NF-ToN-IoT	Literature-based	Multi-benchmark IoT IDS comparison	DNN/CNN/RNN comparison; ~98% binary accuracy
Afraji et al. [47]	ToN-IoT, CICIDS2017	Literature-based	ToN-IoT and cross-dataset generalization	CNN-LSTM-GRU IDS; 100% binary and 97% multiclass accuracy
Ayad et al. [48]	BoT-IoT, ToN-IoT, CIC-DDoS2019	Literature-based	IoT benchmark and detection-time analysis	Feature-selection IDS; 99.82–100% accuracy, 0.02–0.15 s detection time
Proposed framework	CICIDS2018, CICIoT2023, AWID3	Direct experimental	Main validation of the proposed framework	Transfer learning, adaptive response, ablation, statistical validation, operational cost, deployment analysis

summarizes the benchmark coverage and validation scope of the proposed framework. The table distinguishes datasets used for direct experimental validation from those included for literature-based benchmark positioning. Studies involving UNSW-NB15, ToN-IoT, and BoT-IoT are discussed to contextualize the proposed model within widely used IDS benchmarks; however, they are not treated as direct numerical equivalents because preprocessing strategies, feature sets, label mappings, class distributions, and evaluation protocols differ across studies. The proposed framework is directly validated on CICIDS2018, CICIoT2023, and AWID3, which provide complementary wired, IoT, and wireless network security scenarios. This design enables heterogeneous validation while avoiding non-equivalent benchmark comparisons.

4.3. CICIDS2018 and CICIoT2023 Performance and Comparative Analysis

In the initial stage of the model development process, the end-to-end learning behavior of the Transformer Encoder and Dueling DDQN agent was analyzed using the CICIDS2018 and CICIoT2023 datasets. Figure 2 illustrates the episode-based total reward evolution for training and evaluation on both datasets. Figure 2a shows that the training reward for CICIDS2018 gradually increases from negative values and converges to a stable positive reward region after approximately 100 episodes. The evaluation reward also improves rapidly after the early episodes and remains consistently high, indicating that the learned policy achieves stable decision behavior in the validation environment. Figure 2b presents the learning behavior on CICIoT2023. Compared with CICIDS2018, the training reward starts from a lower negative region and improves more gradually, reflecting the higher complexity and variability of IoT traffic patterns. The evaluation reward exhibits more frequent stochastic fluctuations, but it generally remains around the positive reward region after the initial learning phase. Since training is performed under an exploration-inclusive ε-greedy policy, training rewards remain lower and more variable than those obtained by the greedy evaluation policy. Overall, the convergence patterns in both datasets indicate that the agent successfully learns stable response policies for both traditional network traffic and modern IoT-based attack scenarios.

Figure 3 presents the variation in Precision, Recall, and F1-score calculated on the validation set on an episode basis for both CICIDS2018 and CICIoT2023 datasets. As shown in Figure 3a, the CICIDS2018 dataset exhibits a rapid improvement in all metrics after the initial episodes, where Precision, Recall, and F1-score increase sharply and stabilize close to 1.0. This behavior indicates that the model quickly learns a highly discriminative representation and achieves consistent classification performance.

In contrast, Figure 3b illustrates the results for the CICIoT2023 dataset, where the metrics start at relatively high values and remain within a narrow high-performance range throughout training, with minor stochastic fluctuations. Compared to CICIDS2018, the performance progression is less abrupt but remains consistently stable, reflecting the model’s ability to generalize effectively in more complex and dynamic IoT traffic environments.

Figure 4 presents the episode-based evolution of the confusion matrix components (TP, FP, FN, and TN). The results show that TP and TN remain consistently high after the initial stabilization period, indicating that the model correctly identifies both attack and benign traffic throughout training. FP values decrease sharply after the early episodes and remain close to zero, while FN values also stay at very low levels. This pattern demonstrates that the model progressively reduces misclassifications and maintains stable classification performance across episodes. A comparable trend was also observed for the CICIoT2023 dataset, with minor fluctuations due to the higher variability of IoT traffic.

4.4. AWID3 Transfer Learning and Decision Stability

In this stage, where temporal representations learned from wired network traffic are transferred to wireless protocols, the domain adaptation capacity of the model was evaluated. Experiments conducted on the AWID3 dataset demonstrate that preserving the encoder knowledge from pre-training and applying partial fine-tuning to selected layers facilitates the agent’s faster convergence to wireless traffic-specific patterns and its attainment of a more stable performance regime. These results indicate that the reuse of representations in cross-protocol transfer, combined with a limited level of adaptation, offers an effective balance.

To visualize the autonomous defense strategy learned by the model against cyber threats, Figure 5 reports the action distributions (IGNORE, LOG, ESCALATE, BLOCK) for each attack class. The distributions demonstrate that the policy is not locked into a single action and gradually adjusts the level of intervention according to the type of attack. The established framework makes it possible to directly convert the classification output into an operational decision and strikes a practical balance between false positive costs and risk mitigation. Since differences in the number of samples between classes could affect action ratios, the results were interpreted through attack-specific decision profiles rather than absolute frequencies. This interpretation helps differentiate which attack types warrant more aggressive blocking and which types favor logging or escalation.

In the Botnet, Evil_Twin, and Website_spoofing scenarios, it is observed that the agent predominantly selects the BLOCKaction with a high ratio. This distribution indicates that the model has learned a more stringent intervention policy for attacks directly targeting network integrity. The concentration of decisions around the BLOCKaction in these classes also suggests that the risk component of the reward function has become practically dominant.

In the Malware, RogueAP, and SQL_Injection classes, performance is lower, and the agent makes more cautious selections. In these cases, instead of directly selecting BLOCK, the agent tends to choose ESCALATE or LOG for further investigation and monitoring. In particular, the diversity of actions in the SQL_Injection class supports the risk-aware reward design in producing a more controlled policy under uncertainty. Experimental results demonstrate that the agent does not make its decisions solely based on signature similarity; rather, it considers operational costs and risk objectives together.

The operational decision structure of the agent was analyzed to investigate how different source-domain pretraining strategies influence the learned defense policy after transfer learning to the AWID3 target domain. Figure 6a,b presents the intra-class normalized action ratios (IGNORE, LOG, ESCALATE, BLOCK) obtained on AWID3 after transferring models pretrained on CICIDS2018 and CICIoT2023, respectively. To ensure a fair comparison across classes, action ratios are normalized within each class rather than using raw counts. For BENIGN, both transferred policies consistently favor the IGNORE action, indicating that the agent preserves normal traffic without unnecessary intervention.

The results indicate that the transferred policy exhibits noticeable variations depending on the source domain used during pretraining. CICIDS2018-based transfer produces a direct, mitigation-oriented behavior, where the BLOCK action becomes dominant in several attack categories, including management frame attacks such as (RE) ASSOC (Reassociation) and DEAUTH (Deauthentication), as well as architectural and network-scale threats including EVIL_TWIN, ROGUE, ROGUEAP (Rogue Access Point), SSDP (Simple Service Discovery Protocol), and SSDP_2. However, for application and transport layer classes such as BOTNET, MALWARE, SSH (Secure Shell), and WEBSITE_SPOOFING, the agent relies more heavily on LOG actions, reflecting a less aggressive, monitoring-oriented response pattern.

In comparison, CICIoT2023-based transfer results in a more cautious, gradual policy. Here, the ESCALATE action becomes dominant or highly frequent in several classes, including cryptographic and protocol-specific exploits such as KRACK (Key Reinstallation Attack), NAN (Neighbor Awareness Networking), SSDP, SSH, and WEBSITE_SPOOFING, while BLOCK remains reserved for high-risk categories such as DEAUTH, DISAS (Disassociation), and ROGUEAP.

Although both configurations utilize the same reward function, their action distributions differ. This divergence arises from variations in the learned state representations and state visitation patterns induced by different source-domain pretraining. In reinforcement learning, the policy is not solely determined by the reward function, but also by how the environment is represented in the latent space. Since the Transformer encoder is pretrained on different datasets (CICIDS2018 vs. CICIoT2023), it captures distinct feature dependencies and traffic dynamics. When transferred to the AWID3 domain, these differences result in varying state representations, which in turn lead to different Q-value estimations and action-selection behaviors. Therefore, even under an identical reward structure, the final policy can diverge significantly depending on the source-domain representation learned during pretraining.

Figure 7 illustrates the normalized action distribution during transfer learning from CICIoT2023 to AWID3. After the initial adaptation phase, the policy becomes relatively stable. IGNORE remains the dominant action, while BLOCK continues to be selected for high-risk cases. ESCALATE is more frequent in the early episodes but decreases as the policy stabilizes. These results indicate that the agent refines its action-selection behavior during fine-tuning and adapts to the AWID3 target domain without extensive retraining.

4.5. Decision Explainability (XAI) and Operational Action Analysis

Integrated Gradients (IG) analysis reduces the black-box nature of the model by providing feature-level explanations of the decision process. The results show that features associated with attack behavior contribute significantly to the model’s decisions. The attention patterns indicate that the Transformer encoder captures temporal dependencies within packet sequences and utilizes time-localized information for action selection.

The heatmap in Figure 8 reveals that feature contributions are sparse and highly localized, concentrating on distinct structural points within the sequence. Crucially, the model heavily relies on information from the middle of the sequence (around step 56) across specific radio and routing features, which then culminates in a highly dominant attribution to frame.len at the very final position (step 127). This suggests that the model relies on context-rich temporal patterns and definitive packet-size characteristics at the sequence boundary when making decisions, especially for BLOCK actions in SSDP attack scenarios.

Figure 9 explains which temporal feature regions contribute to the agent’s action-value decisions in the AWID target domain. In Figure 9a, the sparse attribution pattern indicates that the IGNORE–LOG decision margin is driven by a limited number of localized feature–time interactions rather than by the entire sequence, with a precise concentration on the wlan.fc.protected feature at the final position (step 127). In Figure 9b, the significantly stronger localized attributions at the sequence boundary show that the ESCALATE decision relies critically on terminal states of key wireless/protocol-level features, such as radiotap.datarate and wlan.radio.data_rate. Therefore, the IG maps reveal not only when the model focuses within the 128-packet sequence but also which feature groups support the selected operational action.

Figure 10 shows how important information is distributed along the time axis by averaging the absolute attribution values (|IG|) obtained through Integrated Gradients analysis across all features. The peaks appearing at specific packet indices indicate that the agent’s decision margin is predominantly influenced by packet-level signals at these particular time steps. The distinct increase observed toward the end of the sequence suggests that later packets provide stronger contextual evidence for the final action decision. This temporal concentration demonstrates that the model focuses on highly informative time regions rather than relying uniformly on the entire 128-packet sequence.

4.6. Reward Sensitivity Analysis

To verify that the selected reward coefficients were not arbitrary, four alternative reward configurations were evaluated and compared with the proposed setting. The analysis focused on how reward shaping changes policy aggressiveness, operational cost, and detection performance. In the security-dominant scenario, where missed attacks were penalized under an alternative security-oriented reward setting (IGNORE on attack = −100) and successful blocking was explicitly rewarded (BLOCK on attack = +5), the agent became highly aggressive. The BLOCK count increased to 633 actions during evaluation, while Precision decreased to 97.4%, and F1-score decreased to 98.7%. Although Recall remained at 100.0%, the large increase in blocking actions indicates excessive intervention and reduced operational efficiency. In the cost-minimization scenario, where penalties for false alarms were strongly reduced (e.g., benign BLOCK = −0.5, benign LOG = −0.1), the agent shifted toward persistent monitoring behavior. The LOG count rose to 994 actions, while BLOCK actions dropped to 0 during evaluation. Precision reached 98.0% and the F1-score was 99.0%, showing that minimizing intervention cost can preserve classification metrics but leads to an operationally passive policy. In the ambiguous-margin scenario, where rewards of LOG, ESCALATE, and BLOCK actions were made too similar (attack rewards: +3.8, +3.9, +4.0), the agent produced unstable action preferences. Evaluation results showed 541 LOG actions and 459 BLOCK actions, while ESCALATE was almost never selected. Precision was 98.0% and the F1-score was 99.0%. This indicates that insufficient reward separation weakens the semantic role of intermediate actions. In the under-penalized attack-miss scenario, where ignoring attacks had little or no penalty, detection quality deteriorated substantially. Recall dropped to 78.4% and F1-score fell to 87.9%, despite Precision reaching 100.0%. This demonstrates that weak false-negative penalties encourage risk-insensitive behavior and cause critical attacks to be missed. Compared with these alternatives, the proposed reward configuration achieved the most balanced policy with Precision = 97.8%, Recall = 100.0%, and F1-score = 98.9%, while maintaining moderate action usage (LOG = 760, ESCALATE = 8, BLOCK = 232). Based on these findings, the final reward coefficients were selected to preserve high recall, avoid excessive blocking, maintain operational efficiency, and sustain meaningful differentiation between hierarchical actions. These results confirm that the final reward coefficients are not arbitrarily selected, but empirically grounded to achieve a balanced trade-off between detection performance, action stability, and operational cost.

To provide a more rigorous quantification of the operational burden beyond intervention counts alone, a normalized Operational Cost (OC) metric was introduced. Unlike raw intervention counts, the proposed metric jointly captures routine logging operations, analyst escalation workload, unnecessary operational disruption caused by false positives, and residual security risk caused by missed attacks. The OC metric is defined as follows:

$$OC ={\displaystyle \frac{0.5N_{LOG}+ 1N_{ESC}+ 3FP + 5FN}{N}}$$

(2)

• $N_{LOG}$: number of logging actions
• $N_{ESC}$: number of escalation actions
• $FP$: number of false positive interventions
• $FN$: number of missed attacks
• $N$: total number of evaluation decisions

The weighting scheme was designed according to the operational severity hierarchy adopted in the reward design. Logging actions were assigned the lowest weight because they introduce minimal operational burden. Escalation actions were assigned a moderate cost since they require additional analyst involvement. False positive interventions were assigned a higher cost due to their potential to cause unnecessary operational disruption, while missed attacks were assigned the highest cost because they represent critical security failures. These weights are not intended to model exact monetary costs; rather, they encode relative severity differences among operational outcomes and provide a normalized comparative indicator of operational burden.

For consistency with the reporting protocol used for the other operational indicators, the OC value was calculated for each evaluation episode and then averaged over the last ten converged episodes. The resulting operational cost values are reported in

Table 7. Operational cost analysis based on the average of the last ten converged episodes.

Dataset/Scenario	F1-Score	Avg F1 Last 10	Avg LOG	Avg ESC	Avg FP	Avg FN	Avg OC
CICIDS2018	99.48%	96.32%	247.7	54.2	33.4	2.9	0.293
CICIDS2018 → AWID3	96.76%	94.99%	103.9	414.8	48.0	3.5	0.628
CICIoT2023	99.13%	98.53%	167.0	2.3	2.9	11.9	0.154
CICIoT2023 → AWID3	96.61%	95.15%	110.4	297.1	44.0	4.5	0.507

. The CICIoT2023 pretraining scenario achieved the lowest operational cost value of 0.154, indicating that the learned policy maintained high detection capability while minimizing unnecessary operational burden. In contrast, the transfer-learning scenarios exhibited relatively higher operational costs compared with the direct pretraining experiments. This increase was mainly associated with elevated escalation behavior during cross-domain adaptation, particularly in the CICIDS2018 → AWID3 scenario, where the average escalation frequency reached 414.8 actions. This behavior suggests a more conservative decision strategy, in which the agent preferred additional verification under uncertainty rather than performing potentially incorrect interventions. Nevertheless, despite the increased operational burden, both transfer-learning models preserved F1-scores above 96%, indicating that the proposed framework maintained an effective balance between detection performance and operational cost.

4.7. Comparative Analysis of Architectures and Operational Validation

Figure 11 visualizes the episode-based frequencies of the actions selected by the agent throughout training. As training progresses, the IGNORE action gradually becomes dominant, while LOG and ESCALATE exhibit an overall decreasing trend despite periodic fluctuations. The BLOCK action remains at a moderate and variable level throughout the training process. This pattern suggests that the learned policy evolves from a monitoring-oriented behavior toward a more stable and selective defense strategy that reduces unnecessary interventions while preserving risk-aware decision making.

The proposed DDQN-based decision mechanism was designed to promote stable learning under the non-stationary and sequential characteristics of network traffic. First, the Double Q-learning formulation separates action selection from action evaluation, thereby reducing the overestimation bias frequently observed in conventional DQN methods. Second, the dueling architecture decomposes each Q-value into state-value and action-advantage components, allowing the agent to distinguish the overall security risk of a traffic state from the relative utility of alternative actions. This decomposition improves policy consistency, particularly in states where multiple actions yield similar outcomes. Additional stabilization is provided through experience replay with a buffer capacity of 100,000 transitions and a target-network synchronization period of 1000 steps. The replay mechanism reduces temporal correlation among consecutive sliding-window observations, while the delayed target-network updates prevent rapidly oscillating temporal-difference targets. Moreover, a discount factor of $\gamma =0.99$ encourages long-term reward optimization rather than myopic decisions, which is important in cyber defense scenarios where delayed consequences may occur.

The exploration–exploitation balance is controlled through a linear epsilon-decay schedule from 1.0 to 0.05. During baseline training, epsilon decays over 100,000 steps toward a minimum value of 0.10, whereas in the AWID3 transfer stage, it decays over 50,000 steps toward a minimum value of 0.05 to allow smoother adaptation to the target domain. This strategy enables broad exploration in the early episodes and gradually shifts the agent toward a more stable decision policy as training progresses. Empirically, the episode-based reward curves exhibit an initial growth phase followed by a relatively stable region with controlled fluctuations, indicating convergence toward a stable operating regime. The evolution of selected actions shows that the agent gradually stabilizes its decision policy, with IGNORE becoming dominant for benign or low-risk traffic, while BLOCK remains consistently utilized for high-risk conditions. ESCALATE actions are more prominent during early episodes and decrease as the policy stabilizes, reflecting a transition from exploratory to more confident decision-making. This behavior demonstrates that the learned policy is driven by a consistent risk-aware optimization of security benefit and operational cost rather than static class memorization. These observations are consistent with the theoretical stabilization mechanisms of Double Q-learning, dueling value decomposition, and target-network updates, which collectively reduce estimation bias and prevent unstable policy oscillations.

Figure 12 demonstrates that the proposed architecture offers a more consistent overall performance compared to alternative RL-based representation learning approaches. The Transformer + RL model achieved a high and balanced success rate with 0.992 precision, 0.998 recall, and a 0.995 F1-score. While the CNN + RL approach reached a 0.997 F1-score, the GRU + RL model produced similarly competitive results with a 0.994 F1-score. In contrast, the MLP + RL structure remained at a 0.886 F1-value, and the LSTM + RL model exhibited a significant performance drop with a 0.640 F1-level. The comparison indicates that the representation capacities of MLP and especially LSTM-based structures remain limited, resulting in more imbalanced decision boundaries across classes.

Figure 13 compares the reward–loss behavior of the Transformer-based reinforcement learning variants. Although Transformer + DDQN, Transformer + DQN, and Transformer + Vanilla RL achieved comparable classification-level performance, the proposed Transformer + DDQN obtained the highest evaluation reward (2254.50) while maintaining a substantially lower average loss (11.47) than Transformer + DQN (23.54) and Transformer + Vanilla RL (26.35). As a result, DDQN provided more efficient reward optimization and more stable value learning under the same sequential intrusion-response setting.

Figure 14 shows the evaluation-stage action distributions of the compared models. The proposed Transformer + Dueling DDQN model achieved 99.8% Precision, 98.46% Recall, and 99.13% F1-score, while also producing the highest evaluation reward (2254.5) and the lowest average training loss (11.47) among the value-based baselines. Transformer + DQN reached similar classification scores, but with a lower reward (2236.9) and substantially higher loss (23.54), indicating less stable value optimization. Transformer + Vanilla RL also attained 99.7% F1-score, yet required a markedly different policy behavior characterized by 462 ESCALATE actions and only 4 BLOCK actions during evaluation, suggesting over-reliance on intermediate responses rather than decisive mitigation. In contrast, the proposed Transformer + Dueling DDQN policy selected 448 BLOCK actions, 75 LOG actions, and only 17 ESCALATE actions during evaluation, demonstrating a more direct and risk-consistent intervention behavior while avoiding excessive escalation. The Transformer + PPO baseline showed substantially weaker results, with 49.3% Precision and 66.0% F1-score despite 100.0% Recall. Its evaluation policy collapsed into 1000 LOG actions with zero BLOCK decisions, revealing passive monitoring behavior and poor active-defense capability. These results indicate that several methods achieved similar classification metrics. However, the proposed Transformer + Dueling DDQN framework provided superior reward efficiency, lower optimization loss, and more operationally effective action policies. This preference is technically justified by the dueling architecture, which separately estimates state criticality and action-specific advantages, enabling more accurate and risk-aware mitigation decisions, particularly for decisive actions such as BLOCK. Beyond serving as a high-performing classifier, the proposed approach also represents a more stable and practically deployable autonomous intrusion response mechanism.

To address the concern regarding single-run reporting and statistical reliability, we conducted five additional independent experiments using different random seeds under the same experimental configuration. To ensure reproducibility, all experiments were executed using fixed preprocessing scripts and controlled random seeds. Five independent repetitions were performed using random seeds 42, 53, 62, 72, and 82. Statistical robustness was evaluated using mean performance, standard deviation, and 95% confidence intervals across all runs. As shown in

Table 8. Statistical robustness results over five independent runs, including mean (%), standard deviation (dec), and 95% confidence intervals (dec.) for Precision, Recall, and F1-score.

Metric	Mean	Std	95% CI
Precision	99.15%	0.0036	±0.0045
Recall	97.62%	0.0039	±0.0048
F1-score	98.38%	0.0028	±0.0034

, the proposed Transformer–DDQN model achieved an average F1-score of 0.9838 ± 0.0028 with a 95% confidence interval of ±0.0034. Precision and Recall were 0.9915 ± 0.0036 and 0.9762 ± 0.0039, respectively.

4.8. Computational Efficiency, Resource Usage, and Scalability Analysis

Although the proposed framework integrates Transformer-based feature learning with reinforcement learning, the overall architecture remains computationally efficient and practically deployable. The model contains only 21,381 trainable parameters, which is significantly smaller than typical deep learning architectures, resulting in low memory overhead and fast execution.

As presented in

Table 9. Training efficiency and convergence statistics for baseline and transfer-learning scenarios.

Scenario	Total Time	Training Time	Avg. Time/Episode	Early Stopping
CICIoT2023 Baseline	1940.04 s	1705.75 s	14.70 s	Episode 116
CICIDS2018 Baseline	2421.72 s	2169.17 s	19.03 s	Episode 114
CICIoT2023 $\to$ AWID3 Transfer	1959.77 s	1729.00 s	15.58 s	Episode 111
CICIDS2018 $\to$ AWID3 Transfer	1930.79 s	1702.38 s	15.34 s	Episode 111

, the framework achieves stable convergence with practical training times ranging from 1930.79 to 2421.72 s (approximately 32–40 min), depending on the dataset and transfer learning configuration. All scenarios remain within a reasonable offline training budget, demonstrating that the proposed architecture can be trained efficiently even under transfer learning settings. Early stopping was consistently triggered between Episode 111 and Episode 116, indicating efficient convergence without unnecessary computation.

The inference performance demonstrates strong computational efficiency. As shown in

Table 10. Inference throughput, packet-level latency, and GPU resource usage for baseline and transfer-learning scenarios.

Scenario	Throughput (pkt/s)	Latency (ms/pkt)	Avg. GPU Utilization (%)	Peak GPU Memory (MB)
CICIoT2023 Baseline	207,629	0.004816	31%	1721.66
CICIDS2018 Baseline	220,862	0.004527	<1%	1863.48
CICIoT2023 $\to$ AWID3 Transfer	198,337	0.005041	18%	2213.31
CICIDS2018 $\to$ AWID3 Transfer	202,188	0.004946	<1%	2213.31

, the model sustains high throughput between 198,337 and 220,862 packets per second, with ultra-low latency below 0.0051 ms per packet. GPU memory consumption remains modest, ranging from 1721.66 MB to 2213.31 MB. These results indicate that the proposed model imposes low computational overhead and does not saturate hardware resources, enabling scalable and parallel deployment.

These results confirm that, despite combining Transformer and reinforcement learning components, the proposed architecture does not introduce prohibitive computational overhead. Instead, it operates as a lightweight and efficient decision engine capable of real-time processing in high-throughput environments. Therefore, the proposed Transformer + Dueling DDQN framework achieves a favorable balance between model complexity, detection capability, and computational efficiency, making it suitable for real-world, high-concurrency network environments.

Across all datasets, the model sustained a stable throughput near ${2\times 10}^5$ packets/s with packet-level latency below 0.0051 ms, while GPU memory demand remained below 2.3 GB. These findings indicate that the proposed architecture is suitable for high-volume traffic inspection and real-time autonomous response pipelines. It should also be emphasized that reinforcement learning remains an active component of the decision-making process during operation. Nevertheless, the measured latency and throughput results demonstrate that the RL-based decision mechanism does not introduce prohibitive runtime overhead. From a scalability perspective, the architecture can be deployed through GPU batch parallelism, stream partitioning, distributed inference workers, SDN-based traffic steering, or cloud/edge cybersecurity nodes. Therefore, the proposed Transformer + Dueling DDQN framework offers not only strong detection accuracy, but also realistic operational efficiency for large-scale and high-concurrency network environments.

4.9. Component-Wise Ablation Study Design

To better highlight the technical originality and contribution of the proposed Transformer–DDQN framework, a component-wise ablation study was designed within the CICIoT2023 → AWID3 transfer-learning setting. This setting was selected because it represents a challenging cross-domain adaptation scenario in which traffic representations learned from IoT network data are transferred to heterogeneous wireless traffic. The aim of the ablation analysis was not limited to measuring classification performance, but also included evaluating autonomous response capability, convergence behavior, and computational efficiency.

The ablation design evaluates the individual contribution of four major components of the proposed framework: the Transformer encoder, the DDQN decision module, the transfer-learning mechanism, and the reward-adaptation strategy. First, the Transformer encoder was removed to examine whether the DDQN-based decision mechanism can operate reliably without contextual temporal representations. Second, the DDQN module was removed to evaluate the behavior of the model when the framework is reduced to a Transformer-based static detection. Third, the transfer-learning mechanism was disabled to assess the contribution of pretrained source-domain representations to target-domain adaptation. Finally, reward adaptation was removed to analyze whether domain-specific reward refinement improves policy optimization and computational efficiency during fine-tuning.

This ablation design is particularly important because the proposed architecture is not intended to function merely as a high-performing binary classifier. Instead, it is designed as an active intrusion detection and response framework that transforms contextual traffic representations into operational security actions. Therefore, each ablated configuration was evaluated not only using Precision, Recall, and F1-score, but also in terms of action-selection behavior (LOG/ESCALATE usage), preservation of the autonomous response policy, and execution time. This enables a more complete assessment of whether each component contributes to detection accuracy, autonomous decision-making, or operational efficiency.

Table 11. Component-wise ablation study considering detection performance, autonomous response capability, and computational efficiency.

Variant	Precision	Recall	F1-Score	LOG/ESCALATE Capability	Autonomous Response Policy	Train Time (s)	Eval Time (s)	Total Time (s)
Without Transformer	0.6631	0.9711	0.7881	Yes	Partially preserved, but representation quality is weak	1871.27	155.09	2026.85
Without DDQN	0.9818	0.9941	0.9879	No	Not available; reduced to static detection	12,130.37	970.85	13,101.45
Without Transfer Learning	0.9443	0.9828	0.9632	Yes	Preserved, but convergence is slower	1857.94	229.58	2087.72
Without Reward Adaptation	0.9441	0.9913	0.9671	Yes	Preserved, but policy optimization is less efficient	3728.27	481.69	4210.24
Full Model	0.9500	0.9828	0.9661	Yes	Fully preserved: IGNORE/LOG/ESCALATE/BLOCK	1729.00	230.56	1959.77

presents the component-wise ablation results obtained under the same preprocessing, transfer-learning, and evaluation protocol. Execution times are reported as measured wall-clock values for each configuration. Training and evaluation times are shown separately, while total time represents the end-to-end execution time including training, evaluation, checkpoint handling, and minor measurement overheads; therefore, it may slightly differ from the direct arithmetic sum of training and evaluation times.

The ablation results show that removing the Transformer encoder causes the most severe degradation in detection capability. The F1-score decreases to 0.7881, indicating that contextual and temporal representation learning is essential for the proposed architecture. Although the DDQN-based action mechanism remains partially available in this setting, the weak representation quality limits the reliability of the learned policy. This confirms that the Transformer encoder is not only an auxiliary feature extractor, but the main component responsible for producing informative state representations for the decision module. The configuration without the DDQN module achieves a high F1-score of 0.9879; however, this result must be interpreted in relation to the framework’s core objective. When DDQN is removed, the model no longer preserves the autonomous response capability of the proposed architecture. In this case, the framework is reduced to a static Transformer-based detector that distinguishes benign and attack traffic, but cannot generate hierarchical operational actions such as IGNORE, LOG, ESCALATE, and BLOCK. Therefore, although its classification metrics appear strong, this variant does not represent an active intrusion detection and response system. The removal of transfer learning results in a slight decrease in F1-score, from 0.9661 in the full model to 0.9632. However, the main contribution of transfer learning is more clearly observed in terms of convergence and computational efficiency. The full model reaches stable performance with a total execution time of 1959.77 s, whereas the configuration without transfer learning requires 2087.72 s. This indicates that transferred source-domain representations provide a more informative initialization for the AWID3 target domain and reduce the adaptation cost. The configuration without reward adaptation produces an F1-score of 0.9671, which is slightly higher than the full model. However, this isolated increase in classification performance is obtained at a substantially higher computational cost. The total execution time increases from 1959.77 s to 4210.24 s when reward adaptation is removed. Therefore, reward adaptation is retained because it improves policy optimization efficiency and provides a more favorable balance between detection performance and operational cost.

5. Discussion

Beyond classification accuracy, our study shows that policy behavior is strongly influenced by learned representations, not only by the reward function. The observed differences between CICIDS2018-based and CICIoT2023-based transfer policies indicate that latent feature distributions shape action-selection dynamics, leading to different operational strategies under identical reward settings. Furthermore, the proposed risk-aware reward design ensures a balanced trade-off between high recall and operational cost, preventing both excessive blocking and passive monitoring behaviors, as confirmed by the reward sensitivity analysis. From a practical deployment perspective, future implementations may further benefit from lightweight attention mechanisms, model compression techniques, and edge-cloud offloading strategies to reduce computational overhead in resource-constrained environments.

The integration of IG-based explainability reveals that the model’s decisions are grounded in temporal and behavioral features such as flow duration, packet variance, and IAT, rather than superficial indicators like IP addresses. This demonstrates that the architecture performs semantically meaningful reasoning over packet sequences, while the RL component translates these representations into consistent and risk-aware actions. The model maintains computational efficiency, achieving high throughput and low latency despite combining Transformer and RL components, which supports its feasibility for real-time deployment.

Although the proposed framework incorporates Transformer-based representation learning and reinforcement learning components, the primary computational cost is associated with the offline training phase rather than deployment. The use of Google Cloud infrastructure and an NVIDIA GPU was mainly required for large-scale training and experimental evaluation. During practical deployment, only the inference stage is executed, which maintains low latency and moderate resource requirements. In addition, transfer learning reduces retraining cost by reusing pretrained representations, thereby improving practical applicability in real-world environments.

6. Conclusions

This study presented a Transformer–Dueling DDQN-based hybrid IDS/IPS framework that formulates network security as a sequential decision-making problem rather than a static classification task. By integrating Transformer-based temporal representation learning with a risk-aware reinforcement learning policy, the proposed architecture enables context-aware and autonomous security actions over a hierarchical action space. The results demonstrate that the model successfully captures long-range dependencies in network traffic while producing stable and operationally meaningful defense strategies through the DDQN decision mechanism.

The experimental results confirm that the proposed approach achieves high detection performance across heterogeneous environments. The model attains F1-scores of 99.48% and 99.13% on CICIDS2018 and CICIoT2023, respectively, indicating strong representation learning capability in both traditional and current IoT network domains. More importantly, under transfer learning-based cross-domain adaptation to AWID3, the model preserves a high level of performance, with F1-scores of 96.76% and 96.61%, demonstrating that the learned representations generalize effectively across wired, wireless, and IoT network protocols. The controlled degradation further indicates that the system captures behavioral attack patterns rather than relying on dataset-specific artifacts, which represents a critical distinction for real-world deployment.

The proposed Transformer–Dueling DDQN framework provides a balanced, stable, and generalizable solution for autonomous intrusion detection and response. It advances the field by jointly addressing representation learning, decision-making, explainability, and operational constraints within a unified architecture.

While the proposed framework demonstrated strong performance and cross-domain adaptability, several targeted research directions may further improve its applicability and robustness. First, future studies may investigate continual and online learning mechanisms to enable adaptation against evolving attack patterns and concept drift in dynamic cybersecurity environments. Second, the current single-agent architecture may be extended to multi-agent reinforcement learning frameworks where multiple security entities collaboratively generate coordinated defense policies across distributed network infrastructures. Third, the proposed framework should be evaluated under more realistic deployment conditions, including encrypted traffic analysis and zero-day attack scenarios. Testing the architecture within large-scale enterprise network environments will provide further insight into operational performance and scalability.