Q-Learning for Resource-Aware and Adaptive Routing in Trusted-Relay QKD Network

Abstract

Efficient and scalable quantum key scheduling remains a critical challenge in trusted-relay Quantum Key Distribution (QKD) networks due to imbalanced key resource utilization, dynamic key consumption, and topology-induced congestion. This paper presents a Q-learning-based adaptive routing framework designed to optimize quantum key delivery in dynamic QKD networks. The model formulates routing as a Markov Decision Process, with a compact state representation that combines the current node, destination node, and discretized key occupancy levels. The reward function is designed to jointly penalize resource imbalance and rapid key depletion while promoting traversal through links with sustainable key generation, guiding the agent toward balanced and congestion-aware decisions. Simulation results demonstrate that the Q-learning scheduler outperforms non-adaptive baseline algorithms, achieving an average distribution time of approximately 100 s compared with 170–590 s for the baseline algorithms, a throughput of 61 keys/s compared with 32–55 keys/s, and a failure ratio limited to 0–0.1, demonstrating superior scalability, congestion resilience, and resource-efficient decision-making in dynamic QKD networks.

1. Introduction

With the widespread deployment of 5G network and the ongoing development of 6G technologies, communication systems are advancing toward higher data rates, lower latency, and stronger security. In light of the growing severity of information security challenges in future mobile communications, quantum communication—particularly Quantum Key Distribution (QKD)—has emerged as a pivotal technology for enabling next-generation secure networks [1,2]. QKD, grounded in the no-cloning theorem and Heisenberg’s uncertainty principle, allows for theoretically unconditional secure key exchange, ensuring absolute confidentiality of shared keys between communication parties [3,4]. QKD demonstrates strong potential for applications in high-security domains, including military communications [5], financial data protection [6], and confidential government communications [7].

As QKD network technology matures, its architecture is evolving from simple point-to-point links to more complex topologies supporting multi-hop routing, multi-user access, and concurrent services. Currently, QKD network architectures can be broadly categorized into three types: optical node-based, quantum node-based, and trusted-relay node-based [8]. Optical and quantum node networks generate point-to-point quantum keys by establishing temporary quantum channels between users on demand. While these approaches support multi-user and long-distance key distribution in principle, they are constrained by signal attenuation and the immaturity of critical technologies such as quantum memory. As a result, optical node QKD networks are mainly suited for local-area applications, and quantum node QKD network remain in the experimental stage. In contrast, trusted-relay QKD networks perform key relaying at the upper layer of quantum key generation, enabling multi-user, long-distance key distribution via hop-by-hop forwarding. Due to its relative technological maturity and scalability, this architecture has become the most practical approach for current large-scale QKD network deployments [9].

At the current stage of development, a critical challenge facing trusted-relay QKD networks is efficient scheduling and stable transmission of key resources through appropriate routing strategies under complex physical conditions and limited resources. Unlike traditional communication networks that primarily optimize metrics such as latency and bandwidth, the performance of QKD links is fundamentally governed by physical-layer factors. These factors result in inherently low key generation rates, pronounced spatial heterogeneity, and significant temporal variability [10]. Consequently, resource management in such network faces challenges that can be broadly categorized into the following three aspects:

• Quantum key generation is fundamentally constrained by physical factors such as quantum bit error rate (QBER) and channel attenuation, leading to substantial heterogeneity in link performance. Links with insufficient availability are unable to sustain frequent transmissions, which can cause local key pool exhaustion and disrupt services.
• Key demand is tightly coupled with dynamic traffic patterns and often exhibits unpredictable and bursty behavior. During traffic surges or directional load concentrations, key resources on specific paths can be rapidly exhausted. In addition, imbalanced path selection and uneven request distribution may cause localized link overuse, leading to resource bottlenecks and reduced transmission efficiency.
• Existing routing strategies, such as shortest-path-first and maximum residual key, rely on static or instantaneous network states without modeling key pool dynamics. This limits their adaptability to resource fluctuations, leading to greater key transmission failures and degraded network performance. Routing in QKD networks must intelligently adapt to dynamic conditions, integrating key generation, consumption, and control feedback to ensure efficient and reliable key delivery.

The primary contributions of this paper are summarized as follows:

1.

We propose a resource-aware key scheduling framework for trusted-relay QKD networks that integrates real-time link state monitoring, online Q-Learning–based adaptive routing, and multidimensional path feasibility verification to ensure dynamic congestion avoidance and stable key distribution under time-varying traffic and network conditions.

2.

We constructed a discrete-time model to characterize key dynamics, where the normalized occupancy ratio was uniformly discretized into states, and the action space was defined by adjacent neighbor sets. A composite reward function, integrating occupancy deviation, consumption penalty, and generation incentive, enabled adaptive balancing between network load and key resource replenishment.

3.

The simulation results demonstrate that the proposed method substantially enhances trusted-relay QKD network performance by improving transmission efficiency, optimizing resource utilization, and effectively mitigating congestion to ensure robust stability under high-load conditions.

The remainder of this paper is organized as follows. Section 2 reviews related work, followed by the presentation of the system model and problem formulation in Section 3. Section 4 reports the simulation results and provides a comprehensive performance discussion, before the paper concludes with key insights and future directions in Section 5.

2. Related Work

With the continuous expansion of trusted-relay QKD networks, the demand for efficient quantum key resource allocation and stable service delivery have increased significantly. Integrating efficient and adaptive routing algorithms has become essential for achieving balanced load distribution and improving overall network service capacity. To support ongoing research and practical deployment of trusted-relay QKD network, a variety of routing algorithms have been proposed.

The Open Shortest Path First (OSPF) protocol, a classical link state routing mechanism, was first applied in the experimental DARPA QKD network [11]. Building on this foundation, the SECOQC project modified OSPF to address the specific demand of relay path selection in QKD network [12]. The Tokyo QKD network adopted a hybrid routing strategy combining static and dynamic elements. Despite its limited scale and relatively simple dynamic algorithms, it effectively coordinated data and key transmission by incorporating link quality and key generation rate into node configurations [13]. In a larger deployment, the Beijing–Shanghai QKD backbone implemented an enhanced OSPF protocol, integrating real-time metrics such as key generation rate, link reliability, and relay node security to improve path selection adaptability [14]. As network architectures and service demand become more complex, recent efforts have focused on advanced dynamic routing strategies. For example, Bi et al. [15] proposed an environment-sensitive algorithm that balances key generation and traffic load across diverse topologies. Yu et al. [16] further introduced a hierarchical routing framework that integrates OSPF with quantum key management, significantly enhancing scalability and robustness.

To enhance key-awareness in routing, reference [17] incorporated the effective key volume of each link into OSPF-based path computation. Expanding on this, ref. [18] jointly considered both the current key pool size and the key generation rate, enabling more precise and dynamic resource scheduling. To mitigate key exhaustion at bottleneck links, ref. [19] introduced a hierarchical routing-based path optimization method to enhance transmission flexibility and overall performance. Reference [20] selected optimal routes using a combined metric of physical distance and key availability, while reference [21] incorporated controlled randomness to increase path diversity and robustness. Reference [22] developed a multifactor link cost model in conjunction with a key-aware routing strategy, significantly improving the key exchange success ratio. Within the realm of multipath routing, Han et al. [23] designed a multi-user routing algorithm aimed at optimizing quality of service (QoS) in optical QKD networks. Reference [24] presented a dynamic routing scheme comprising a Hello protocol, routing algorithm, and link state update mechanism to adapt to evolving network conditions. Furthermore, reference [25] implemented a node labeling strategy for multipath selection, effectively preventing routing loops and node conflicts, thus ensuring routing stability and efficiency.

Given the structural similarities between trusted-relay QKD networks and Wireless Ad Hoc Networks (WANETs)—notably their dynamic topologies—several studies have adapted classical WANET routing protocols to address frequent link state changes in QKD scenarios [26,27]. In quantum key distribution over Optical Networks (QKD-ON), joint routing, wavelength, and time-slot assignment (Routing and Wavelength Assignment/Routing and Resource Assignment, RWA/RRA) poses significant complexity beyond that of traditional optical networks [28,29]. To cope with this, key-on-demand strategies based on software-defined optical networks and key pool mechanisms were introduced for adaptive resource management [30]. Time scheduling models have also been developed to coordinate provisioning across domains [31], while heuristic RWA/RRA algorithms using enhanced node structures and auxiliary graphs help minimize key waste and avoid untrusted relays [32]. For secure multicast, distributed subkey relay trees have been designed to support efficient delivery of one to many keys [33]. Research on routing protocols for QKD networks has also been conducted from various perspectives, including adaptation to different types of QKD networks [34,35], enhancement of security [36], and suppression of crosstalk [37]. In this context, ref. [38] demonstrates an experimental B92 protocol for quantum key distribution path verification, while [39] proposes a hybrid QKD framework that dynamically selects protocols to optimize end-to-end key distribution routing security and efficiency.

Despite progress, key challenges remain in QKD routing. Most protocols rely on instantaneous link states and fixed cost functions, limiting foresight and adaptability, while resource allocation often prioritizes short-term gains over long-term efficiency. Machine Learning/Reinforcement Learning (ML/RL) techniques have been applied at various levels: ref. [40] optimized optical path routing and resource allocation in QKD networks with fixed topologies using deep reinforcement learning (DRL); ref. [41] combined autoencoders with the Cascade protocol to predict the QBER and key rates; ref. [42] proposed OptiNode for trusted node placement. These approaches primarily focus on performance or topology, without explicitly modeling the consumable and dynamically generated nature of QKD keys.

In contrast, our method uses online Q-learning with real-time key pool states to guide routing decisions that account for consumable resources, time-varying availability, and QKD-specific security requirements. Unlike classical ML/RL routing methods that assume reusable resources [43,44], our approach integrates both resource- and security-aware constraints. DRL has been shown to efficiently manage high-dimensional states, adapt to real-time dynamics, and optimize resource allocation and traffic scheduling in communication networks [45,46,47]. Future work could further integrate DRL with key pool dynamics to enable intelligent, scalable, and secure QKD routing.

3. System Model and Problem Formulation

To address the challenges of uneven resource distribution and dynamically changing key consumption in trusted relay QKD network, we propose a Q-learning-based key scheduling framework for adaptive and resource-aware path selection. As shown in Figure 1, the framework comprises three key modules. The Neighbor State Awareness Module continuously monitors adjacent links, extracting critical features such as key occupancy rate, deviation from ideal balance, consumption rate, and generation capability, constructing a comprehensive state representation. The episodic online Q-learning framework enables each routing request to evaluate the current policy and update the Q-table based on immediate rewards. Serving as the core decision engine, the Online Q-learning scheduler uses a reward function to balance resource uniformity and consumption efficiency, penalizing over-concentration and rapid depletion while promoting stable replenishment. The Routing and Validation module then generates trusted relay paths from the learned policies, ensuring topology compliance and efficient and stable key distribution. Although keys can be generated in parallel across links, our framework applies sequential, hop-by-hop forwarding decisions at the strategy level based on local and adjacent link states, ensuring balanced resource usage, bottleneck avoidance, and stable key distribution. The appendix, which presents the QKD Network Architecture (Figure A1) and the Relay Routing Mechanism in QKD Networks(Figure A2), provides the foundational knowledge framework underpinning this study.

3.1. QKDN Model and Constraints

3.1.1. Network Topology Model

In this study, the trusted-relay QKD network is modeled as an undirected graph $G=(V,E)$, where the vertex set $V=\{v_1,v_2,\dots ,v_n\}$ represents the trusted-relay nodes within the network. Each node $v_i\in V$ is equipped with QKD devices as well as key management (KM) modules, enabling it to generate keys locally via physical quantum channels, store these keys, and securely forward keys to neighboring nodes through the key management layer.

The edge set E consists of key management links $e_{ij}\in E$, which represent logical secure key relay channels between nodes $v_i$ and $v_j$. These links serve as the backbone for multi-hop key forwarding and distribution within the trusted relay architecture. It should be noted that each key management link $e_{ij}$ is supported by underlying physical quantum links that enable raw key generation between adjacent nodes. Edges in the graph correspond to the logical connections used for key relay rather than the physical quantum channels themselves.

To accurately characterize the operational attributes of each key management link, two key parameters are defined for every ${\mathrm{e}}_{ij}$: the key transmission rate limit $B_{ij}$ and the key pool size $H_{ij}$. Specifically, $B_{ij}^{max}$ denotes the maximum rate at which quantum keys can be distributed through link $e_{ij}$, while $H_{ij}^{max}$ represents the maximum size of the key pool between adjacent nodes.

3.1.2. Quantum Key Pool Model

Each link in the trusted-relay QKD network is associated with a key pool ${\mathrm{H}}_{ij}$, which stores the quantum keys shared between adjacent nodes $v_i$ and $v_j$. This key pool supports key generation, storage and consumption and dynamically reflects the availability of key resources on the corresponding link. The key pool is bidirectionally symmetric, i.e., for any node pair $v_i$ and $v_j$, the shared key pool satisfies ${\mathrm{H}}_{ij}={\mathrm{H}}_{ji}$. The state of the key pool on link ${\mathrm{e}}_{ij}$ at time t is defined as a triplet:

$${\mathrm{S}}_{H_{ij}}\left(t\right)=\left(h_{ij}\left(t\right),g_{ij}\left(t\right),c_{ij}\left(t\right)\right)$$

(1)

where $h_{ij}\left(t\right)\in \left[0,H_{ij}^{max}\right]$ denotes the amount of remaining usable keys; $g_{ij}\left(t\right)\ge 0$ represents the key generation rate; and $c_{ij}\left(t\right)\ge 0$ represents the key consumption rate.

To ensure system stability and prevent the key pool from exceeding its maximum size, the key pool evolves over discrete time steps of duration $\Delta t$ according to the following update rule:

$$h_{ij}(t+\Delta t)=min\left\{H_{ij}^{max},max\left[0,h_{ij}\left(t\right)+g_{ij}\left(t\right)·\Delta t-c_{ij}\left(t\right)·\Delta t\right]\right\}$$

(2)

where the inner max ensures that the remaining usable keys cannot be negative. This formulation guarantees that the number of available keys remains within the allowed limits, thereby enabling dynamic control over key resources and supporting the sustainable operation of the QKD network.

3.1.3. Routing Constraints

The quantum key distribution task is defined as a triplet $\left(s,d,R\right)$, where the source node $s\in V$ intends to transmit quantum keys to the destination node $d\in V$ with a transmission demand of R. To ensure the feasibility of key transmission and efficient allocation of network resources, the selected routing path must satisfy the following constraints.

1.

Connectivity constraint: For any pair of adjacent nodes $(v_i,v_j)$ along the routing path, a key management link $e_{ij}\in E$ must exist to guarantee topological continuity.

2.

Key transmission rate constraint: At any time t, for each link $e_{ij}\in E$, the cumulative key transmission rate of all active tasks traversing this link must not exceed $B_{ij}^{max}$:

$$\sum _{(s,d,R):e_{ij}\in P_{s,d}}{\delta }_{s,d,R}\left(t\right)\le B_{ij}^{max}$$

(3)

where ${\delta }_{s,d,R}\left(t\right)$ denotes the instantaneous key transmission rate of task $(s,d,R)$ on link $e_{ij}$ at time t.

3.

Key pool size constraint: To ensure path viability, all links must maintain a positive key pool size at time t, satisfying

$$h_{ij}\left(t\right)>0,\forall (i,j)\in P_{s,d}$$

(4)

where $P_{s,d}=(v_s,v_1,v_2,\dots ,v_d)$ represents the path from source node s to destination node d.

3.1.4. Evaluation Metrics and Definitions

1.

Average Key Distribution Time: $T_{\mathrm{avg}}$ quantifies the mean time required to deliver successfully distributed quantum keys during a single simulation cycle. It reflects the overall efficiency of routing and scheduling strategies in dynamic trusted-relay QKD networks. The total delivery time for the k-th distribution is defined as

$$T_k=t_{\mathrm{queue},k}+t_{\mathrm{trans}}+t_{\mathrm{rand},k},$$

(5)

The queuing delay, driven by total key volume and effective path key transmission rate, reflects network load and dominates variations in $T_{\mathrm{avg}}$, while fixed transmission and random perturbation delays capture link characteristics and environmental fluctuations:

$$t_{\mathrm{queue},k}={\displaystyle \frac{{\mathrm{KeyAmount}}_k}{B_k}}.$$

(6)

where ${\mathrm{KeyAmount}}_k$ is the total keys delivered in the k-th trial, $B_k$ the path’s effective key transmission rate, $t_{\mathrm{trans}}$ the fixed propagation delay, and $t_{\mathrm{rand},k}$ a small random perturbation capturing minor link fluctuations or transient rerouting; the average delivery time over K trials is then

$$T_{\mathrm{avg}}={\displaystyle \frac{1}{K}}\sum _{k=1}^K\left({\displaystyle \frac{{\mathrm{KeyAmount}}_k}{B_k}}+t_{\mathrm{trans}}+t_{\mathrm{rand},k}\right)$$

(7)

2.

Maximum Key Utilization: The highest utilization among active links at the moment of path selection:

$$\max \_\mathrm{util}=\underset{(i,j)\in E_{\mathrm{active}}}{max}{\rho }_{ij}=\underset{(i,j)\in E_{\mathrm{active}}}{max}\left(1-{\displaystyle \frac{h_{ij}}{H_{ij}^{max}}}\right)$$

(8)

This metric captures the link that is most critically depleted in terms of key pool size. A value of $\max \_\mathrm{util}\to 1$ indicates severe congestion and highlights the need for dynamic rerouting or pool scaling. E is the set of all links in the network, with $\left|E\right|$ denoting the total number of links. The active link set is defined as

$$E_{\mathrm{active}}=\{(i,j)\in E:h_{ij}<H_{ij}^{max}\}$$

(9)

which includes links that are not fully occupied and thus can contribute to further key distribution. For any active link $(i,j)\in E_{\mathrm{active}}$, the instantaneous utilization is defined as

$${\rho }_{ij}=1-{\displaystyle \frac{h_{ij}}{H_{ij}^{max}}},0\le {\rho }_{ij}<1$$

(10)

where $h_{ij}$ denotes the current key pool size on link $(i,j)$ at the time of path selection. $H_{ij}^{max}$ is the maximum key pool size of link $(i,j)$.

3.

Proportion of Over-Threshold Key Resource Links: Given a utilization threshold $\theta$, the over-threshold link set includes all links whose utilization exceeds $\theta$:

$$E_{\mathrm{over}}=\{(i,j)\in E:{\rho }_{ij}>\theta \},r_{\mathrm{over}}={\displaystyle \frac{|E_{\mathrm{over}}|}{\left|E\right|}}.$$

(11)

where ${\rho }_{ij}=1-h_{ij}/H_{ij}^{max}$ denotes the utilization of link $(i,j)$ at the time of path selection. The ratio $r_{\mathrm{over}}$ quantifies the fraction of links that are potentially congested, providing an indicator of network-wide resource stress.

4.

Network Throughput: The network throughput quantifies the efficiency of key distribution by measuring the total number of quantum keys successfully delivered per unit time across all links. It reflects the effectiveness of routing and key management strategies and provides insight into the trade-off between performance and congestion. Let $N_{\mathrm{succ}}$ denote the total number of keys successfully transmitted during a single distribution task, and $\Delta t$ the corresponding duration. The network throughput is then given by

$$\mathrm{Throughput}={\displaystyle \frac{N_{\mathrm{succ}}}{\Delta t}}$$

(12)

5.

Key Distribution Failure Ratio: This metric quantifies the fraction of failed key distribution attempts:

$$R_{\mathrm{fail}}=1-{\displaystyle \frac{N_{\mathrm{succ}}}{N_{\mathrm{total}}}},$$

(13)

where $N_{\mathrm{succ}}$ is the number of successful key transmissions and $N_{\mathrm{total}}$ is the total number of key distribution attempts. $R_{\mathrm{fail}}$ serves as a direct indicator of the reliability and effectiveness of the routing strategy. Failures may result from key pool depletion, disconnected topologies, insufficient path length, or suboptimal routing decisions.

3.2. Online Q-Learning Model Design

3.2.1. State Space Definition

The agent’s state representation captures both routing information and the dynamic status of key resource utilization. Based on this, the state space is formally defined as

$${\mathrm{S}}_t=\left({\mathrm{curPos}}_t,{\mathrm{endPos}}_t,s_{key}^{ij}\left(t\right)\right)$$

(14)

where ${\mathrm{curPos}}_t\in V$ denotes the current node of the quantum key at time t, and ${\mathrm{endPos}}_t\in V$ represents the destination node for the key distribution task. The set V includes all trusted-relay nodes in the network. $s_{key}^{ij}\left(t\right)$ represents the agent’s perception of the status of key resources on the link $(i,j)$, which is a crucial input to decision-making as it reflects the degree of resource contention after normalization and discretization.

To accommodate the discrete nature of the Q-learning model, the continuous key pool size $h_{ij}\left(t\right)$ is discretized as follows.

1.

Normalization: Given the known maximum size of the key pool on each link $H_{ij}^{max}$, the remaining key amount is first normalized into a key occupancy ratio:

$${\rho }_{ij}\left(t\right)=1-{\displaystyle \frac{h_{ij}\left(t\right)}{H_{ij}^{max}}},{\rho }_{ij}\left(t\right)\in [0,1)$$

(15)

2.

Interval Partitioning: The range of occupancy ratios $[0,1)$ is uniformly divided into M intervals of equal width ${\displaystyle \frac{1}{M}}$, forming a discrete set of states:

$$B=\left\{\left[0,{\displaystyle \frac{1}{M}}\right),\left[{\displaystyle \frac{1}{M}},{\displaystyle \frac{2}{M}}\right),\dots ,\left[{\displaystyle \frac{M-1}{M}},1\right)\right\}$$

(16)

3.

State Mapping: The continuous occupancy ratio is mapped to a discrete state label using

$$s_{key}^{ij}\left(t\right)=\varphi \left(h_{ij}\left(t\right)\right)=min\left(M-1,⌊M·{\rho }_{ij}\left(t\right)⌋\right),\varphi \left(h_{ij}\left(t\right)\right)\in \{0,1,\dots ,M-1\}$$

(17)

where $s_{key}^{ij}\left(t\right)$ denotes the discretized state of the key pool on link $(i,j)$. A higher discrete level indicates a higher key occupancy ratio, implying greater resource contention on the link. This feature enables the agent to make routing decisions that are more responsive to the availability of real-time resources.

3.2.2. Action Space Definition

For an agent currently located at node $v_i\in V$, its set of possible actions is constrained by the set of neighboring nodes. The neighbor set is formally defined as

$$\mathrm{A}\left(i\right)=\left\{j\in V\mid (i,j)\in E\right\}$$

(18)

where A(i) represents all directly connected adjacent nodes to which the quantum key can be forwarded from node $v_i$. Therefore, the agent’s action space at time t is defined as

$${\mathrm{A}}_t=\left\{a_t\mid a_t\in \mathrm{Neighbors}\left({\mathrm{curPos}}_t\right)\right\}$$

(19)

where the action $a_t$ denotes the agent’s decision to transmit the quantum key from its current node $v_i$ to one of its neighboring nodes. The action space characterizes all feasible and permissible movements available to the agent at a given decision point.

3.2.3. Reward Function Design

In Q-learning-based routing, the reward function should comprehensively account for link resource utilization, key consumption, and replenishment dynamics to ensure balanced allocation, alleviate congestion, and mitigate the risk of key exhaustion. To this end, a reward function is constructed based on the effectiveness of quantum key resource usage, defined as

$$R_t=-\left[\alpha ·\left|{\rho }_{ij}\left(t\right)-{\rho }_{\mathrm{eq}}\right|+\beta ·{\tilde{c}}_{ij}\left(t\right)-\gamma ·{\tilde{g}}_{ij}\left(t\right)\right]$$

(20)

$${\tilde{c}}_{ij}\left(t\right)={\displaystyle \frac{c_{ij}\left(t\right)}{B_{ij}^{max}}},{\tilde{g}}_{ij}\left(t\right)={\displaystyle \frac{g_{ij}\left(t\right)}{B_{ij}^{max}}}$$

(21)

where ${\rho }_{ij}\left(t\right)=1-{\displaystyle \frac{h_{ij}\left(t\right)}{H_{ij}^{max}}}$ denotes the key occupancy ratio on link $(i,j)$ at time t, while ${\rho }_{\mathrm{eq}}$ represents the predefined ideal occupancy ratio. The absolute deviation $\left|{\rho }_{ij}\left(t\right)-{\rho }_{\mathrm{eq}}\right|$ serves as a penalty for imbalance in resource distribution.

The term $c_{ij}\left(t\right)$ represents the instantaneous key consumption rate, penalizing links with rapid key depletion, while $g_{ij}\left(t\right)$ denotes the key generation rate, incentivizing routing through links with sustainable key supply. Together, they capture the dynamic consumption and replenishment of the key pool, essential for efficient resource utilization. The hyperparameters $\alpha$, $\beta$, and $\gamma$ tune the relative influence of each component in the reward. By combining positive and negative incentives, this reward structure discourages resource imbalance while promoting link selection that ensures adequate key availability and moderate usage.

3.2.4. Policy and Update Mechanism

In the Q-learning framework tailored for quantum key distribution routing, the core quantity is the action-value function $Q(s,a)$, which represents the expected cumulative reward obtained by taking action a in state s and subsequently following the optimal policy ${\pi }^{*}$. Formally, the optimal Q-function $Q^{*}(s,a)$ satisfies

$$Q^{*}(s,a)=\mathbb{E}\left[\sum _{k=0}^{\infty }{\lambda }^k{R}_{t+k}|s_t=s,a_t=a,{\pi }^{*}\right],$$

(22)

where $R_t$ denotes the immediate reward, $\lambda \in [0,1)$ is the discount factor, and ${\pi }^{*}$ is the optimal policy. The Q-function guides the agent’s decision-making by prioritizing actions that maximize long-term rewards.

Within this framework, the environment is designed to simulate the dynamic evolution of key pools on all links after each action. This real-time feedback mechanism ensures that the agent consistently perceives key resource availability, enabling context-aware routing decisions. To balance exploration and exploitation, a temporal-difference (TD) learning approach with an $\epsilon$-greedy policy is employed. The behavior policy ${\pi }_b$ is defined as

$${\pi }_b\left(a\right|s)=\left\{\begin{array}{cc}1-\epsilon ,\hfill & \mathrm{if}a=arg\underset{a^{\prime }}{max}Q(s,a^{\prime })\hfill \\ {\displaystyle \frac{\epsilon }{|A_t|-1}},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(23)

where $\epsilon \in [0,1]$ denotes the exploration rate, and $|A_t|$ is the number of feasible actions in state s. This strategy ensures sufficient exploration during early training while enabling convergence to the optimal policy ${\pi }^{*}$ as learning progresses. The Q-values are updated online according to the TD-maximization rule:

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\eta \left[R_t+\lambda ·\underset{a^{\prime }}{max}{Q}_t(s_{t+1},a^{\prime })-Q_t(s_t,a_t)\right],$$

(24)

where $\eta \in (0,1]$ is the learning rate. The reward $R_t$ jointly considers key consumption, generation, and occupancy balance. Guided by a task-specific reward model aligned with resource efficiency, the agent progressively acquires optimal routing strategies through interaction with the dynamic QKD environment. The detailed training procedure is summarized in Algorithm 1, in which, at each hop, the agent selects an action, observes the reward, updates the Q-table, and proceeds to the next state.

Algorithm 1 Episodic online Q-Learning for key scheduling.

Input: QKD network $G=(V,E)$, max key pool size $H_{ij}^{max}$, learning rate $\eta$, discount factor $\lambda$, exploration rate $\epsilon$, interval count M, reward weights $\alpha$, $\beta$, $\gamma$, target occupancy ${\rho }_{\mathrm{eq}}$ Output: Optimal routing policy ${\pi }^{*}\left(s\right)=arg{max}_{a}Q(s,a)$ 1: for each episode do 2:     Randomly initialize ${\mathrm{curPos}}_t$ and ${\mathrm{endPos}}_t$ 3:     Observe $h_{ij}\left(t\right)$ on outgoing links from ${\mathrm{curPos}}_t$ 4:     Discretize: $s_{key}^{ij}\left(t\right)=min(M-1,\lfloor M·(1-h_{ij}\left(t\right)/H_{ij}^{max})\rfloor )$ 5:     Construct state $s_t=({\mathrm{curPos}}_t,{\mathrm{endPos}}_t,s_{key}^{ij}\left(t\right))$ 6:     while ${\mathrm{curPos}}_t\ne {\mathrm{endPos}}_t$ do 7:         Select $a_t=(i\to j)$ using $\epsilon$-greedy strategy 8:         Execute $a_t$, observe updated key pool $h_{ij}\left(t\right)$ 9:         Observe new state $s_{t+1}=({\mathrm{curPos}}_{t+1},{\mathrm{endPos}}_t,s_{key}^{ij}(t+1))$ 10:       Compute reward: $$R_t=-\left[\alpha ·|{\rho }_{ij}\left(t\right)-{\rho }_{\mathrm{eq}}|+\beta ·c_{ij}\left(t\right)-\gamma ·g_{ij}\left(t\right)\right]$$ 11:        Update Q-value using TD rule: $$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\eta \left[R_t+\lambda ·\underset{a^{\prime }}{max}Q(s_{t+1},a^{\prime })-Q(s_t,a_t)\right]$$ 12:        $s_t\leftarrow s_{t+1}$ 13: return${\pi }^{*}\left(s\right)=arg{max}_{a}Q(s,a)$

Q-learning incurs a training cost of $O(N_{\mathrm{ep}}·L·D)$, where $N_{\mathrm{ep}}$ is the number of training episodes, L the path length, and $D=2\left|E\right|/\left|V\right|$ the average node degree. Once trained, its per-path routing complexity reduces to $O\left(LD\right)$, which is generally lower than the $O\left(\right|E|log|V\left|\right)$ cost of Dijkstra-based algorithms, reflecting significant training overhead but low per-decision computation suitable for real-time routing in large network. In this context (Figure 2), the end-to-end quantum key routing process is structured into four tightly integrated phases—Initialization, Local State Perception, Path Decision & Value Update, and Feasibility Verification & Feedback—which together form a closed-loop optimization cycle, where the virtual key channels represent abstracted logical links modeling resource availability and key generation/consumption for adaptive routing evaluation. During live operation, each routing decision triggers an immediate Q-table update, and this cycle continues iteratively until the policy converges.

4. Results and Performance Discussion

4.1. Simulation Setting

4.1.1. Simulation Parameters

We develop a modular simulation framework based on QKDNetSim (NS-3.41) to model quantum key management and routing in dynamic trusted-relay QKD networks. Simulations are conducted on Ubuntu 22.04 with an Intel i7 CPU and 128 GB RAM, and algorithms are implemented in Python 3.9 via Anaconda 3.2.0.

Training and evaluation are performed on Barabási–Albert (BA) networks of 50–200 nodes (Figure 3). This scale-free topology captures the heterogeneity and hub-centric structure typical of real communication networks, where highly connected nodes form potential congestion points, providing a challenging environment for adaptive routing. Each edge represents a virtual key channel, characterized by static attributes (initial key pool and consumption/generation rates) and dynamic properties reflecting network variability.

Table 1. Network and experimental parameters with descriptions.

Parameter	Value	Description/Purpose
Number of nodes $\left|V\right|$	50–200	Network scale; affects path lengths, congestion, and failure probability
Topology model	Barabási–Albert (BA) model	Simulates a hub-centric scale-free topology to analyze the impact of node degree distribution on routing
Average node degree	4	Network connectivity and path redundancy; influences routing feasibility and robustness
Link maximum key transmission rate $B_{ij}^{max}$	100 keys/s	Maximum key transmission rate per link; fundamental resource constraint for routing
Single-node key transmission rate	10–100 keys/s	Rate at which an individual node transmits keys
Link key pool size $H_{ij}^{max}$	1000 keys	Maximum key storage per link; ensures sustained service under transient high load
Initial key consumption rate $c_{ij}\left(0\right)$	50 keys/s	Initial link key consumption; influences early-stage routing decisions and reward computation
Initial key generation rate $g_{ij}\left(0\right)$	50 keys/s	Initial link key generation; guides early exploration in training
Minimum usable key count $h_{min}$	1 key	Minimum key threshold below which a link is considered unavailable
Ideal occupancy ratio ${\rho }_{\mathrm{eq}}$	0.50	Target key utilization per link; guides load-balancing strategy
Overload detection threshold $\theta$	0.65	Key occupancy level above which a link is considered overloaded; used in scheduling and penalty calculation
Time step duration $\Delta t$	1 s	Simulation time step; defines the interval for network updates and agent decisions
Network load	500–3500	Total network key request volume; used to evaluate algorithm performance under varying load conditions
Granularity level of discretization M	10	State discretization granularity for Q-learning; balances representation accuracy and computational complexity
Traffic modulation pattern	Sinusoidal ±20%	Periodic traffic fluctuations; simulates environmental or user-induced load variations
Link drift standard deviation $\sigma$	0.10	Random link performance drift; enhances training robustness under stochastic conditions
Link failure probability	0.01–0.5 per step	Probability of link failure at each time step; tests routing resilience
Link recovery probability	0.01 per step	Probability of link restoration; models dynamic network recovery
Transmission delay per hop $t_{\mathrm{trans}}$	0.002 s	Per-hop propagation delay; contributes to total distribution time and reward computation
Random transmission jitter $t_{\mathrm{rand}}$	±0.005 s	Random variation in transmission delay; simulates network or physical-layer noise
Random seed	2025	Ensures reproducibility of simulation results
Max steps per episode	150	Number of time steps per training episode; represents a complete key distribution cycle
Initial Q-table $Q(s,a)$	[0, 0.01]	Initial Q-values for the learning agent; balances early exploration and reward scaling
Max training episodes	50	Total training episodes; determines accumulated experience and policy convergence

summarizes the core network and experimental parameters. Network parameters—including node degree, key generation rates, key pool sizes, and initial key consumption rates—are configured to reflect practical QKD constraints. Overload thresholds and ideal key pool utilization ratios guide routing and load balancing, while traffic fluctuations, link drift, per-hop delays, transmission jitter, and probabilistic link failures evaluate algorithmic resilience. Simulations proceed in discrete time steps, with each episode representing a full key distribution cycle.

Table 2. Q-learning hyperparameter settings across training phases.

Parameter	Ep.1–5	Ep.6–15	Ep.19–23	Ep.24–30
$\epsilon$ (exploration rate)	1.0 → 0.5 (linear)	0.5 → 0.1 (exp)	0.1	0.01
$\eta$ (learning rate)	0.01	0.01 → 0.005	0.005	0.002
$\alpha$ (load balance weight)	0.5	0.6	0.4	0.5
$\beta$ (penalty weight)	0.5	0.4	0.6	0.5
$\gamma$ (quality reward weight)	0.2	0.3	0.3	0.3
$\lambda$ (discount factor)	0.8	0.9	0.95	0.95

presents Q-learning hyperparameters across training phases. The exploration rate $\epsilon$ decreases from high to low to shift from broad exploration to policy exploitation, while the learning rate $\eta$ diminishes to stabilize convergence. Weights $\alpha$, $\beta$, and $\gamma$ are adjusted to balance load, penalties, and quality rewards, guiding the agent from general learning to fine-tuned optimization. The discount factor $\lambda$ increases to emphasize long-term rewards, enabling progressive learning of effective routing strategies in dynamic networks.

Each episode spans 150 time steps, with 50 episodes totaling 7500 s of virtual time. Each network configuration is simulated 30 times to reduce stochastic variability, ensuring statistically reliable metrics for reward convergence, key distribution time, key utilization, and failure ratio, while reflecting the algorithm’s adaptability in dynamic QKD network.

4.1.2. Benchmark Methods

To evaluate the proposed QKD routing scheme, three representative baseline algorithms are considered, OSPF (Open Shortest Path First), CAD (Congestion-Aware Dijkstra), and RAKP (Residual-Adaptive Key Provisioning), covering routing strategies from topology-driven to local and global resource-aware methods. In this study, CAD is introduced as a congestion-aware extension of Dijkstra, while RAKP is adopted from prior work [48].

OSPF is a classical shortest-path protocol that assigns uniform cost (typically 1) to each link and selects paths minimizing total hop count. It relies solely on network topology and ignores key resources.

CAD incorporates local congestion awareness into Dijkstra’s algorithm. For each link $(i,j)$, the weight is defined as

$$w_{ij}={\displaystyle \frac{1}{h_{ij}+\xi }}$$

(25)

where $h_{ij}$ denotes the residual key pool size and $\xi ={10}^{-6}$ is a small constant to avoid singularities. Paths are chosen to minimize cumulative weight, favoring links with higher residual key pool size. CAD performs local optimization, considering only the current path’s link states.

RAKP is a global key provisioning. For each link $(i,j)$, the residual ratio is

$$r_{ij}={\displaystyle \frac{H_{ij}-h_{ij}^{\prime }}{H_{ij}}}$$

(26)

where $H_{ij}$ is the maximum key pool size and $h_{ij}^{\prime }$ is the currently allocated key volume. During each allocation round, links are proportionally allocated keys based on $r_{ij}$, and RAKP continuously adapts to global network dynamics.

4.2. Experimental Results and Analysis

4.2.1. Reward Convergence Analysis

The training rewards exhibit a clear exploration–convergence pattern (Figure 4). In the initial phase (Ep.1–5), high exploration leads to large fluctuations and predominantly negative rewards ($-1.5$ to $-0.6$), as the agent explores randomly without an effective policy. During the intermediate phase (Ep.6–15), decreasing $\epsilon$ and adaptive learning rate drive rewards toward 0, reflecting the agent’s emerging ability to balance load and key resource utilization. In the later phase (Ep.16–30), rewards stabilize around $-0.04$ with minor fluctuations ($\pm 0.01$), indicating that the Q-learning policy has largely converged and maintains stable performance despite link perturbations.

4.2.2. Total Quantum Key Volume: Performance Variation

This section systematically compares QKD routing algorithms under varying key volumes, considering average distribution time, maximum key utilization, failure ratio, and network throughput. The results reveal how network constraints interact with algorithmic design. Q-learning consistently achieves high throughput and low failure ratios by anticipating congestion and dynamically balancing load. In contrast, RAKP and CAD incur higher distribution time and failures due to reactive or localized strategies, while OSPF, insensitive to load, exhibits the highest utilization and failure accumulation.

As shown in Figure 5, the evolution of key distribution performance reflects congestion dynamics and algorithmic adaptation. Q-learning maintains low distribution times and moderate key utilization at low key volumes, exploiting underutilized links and avoiding nascent bottlenecks. As key volume increases, distribution time rises to 28–31 s and maximum utilization reaches 0.92, showing that even adaptive policies are limited by finite link key pool size. RAKP reacts slower to changing conditions, leading to higher delays (37–43 s) and utilization (0.93–0.96), while CAD, constrained by local information, suffers accelerated congestion at critical nodes (37–49 s, 0.945–0.985). OSPF consistently overloads frequently used links (33–54 s, 0.995).

Failure ratio distributions further illustrate algorithmic foresight. Q-learning confines nearly all failures to 0–$0.1$, reflecting proactive path selection. RAKP exhibits 57% of failures in $0.1$–$0.3$ due to delayed global adjustments, while CAD shows 14% in $0.3$–$0.4$, highlighting the impact of local decisions at bottlenecks. OSPF reaches 14% in $0.4$–$0.5$, consistent with unbalanced traffic accumulation. Throughput trends (Q-learning 34.6–61 keys/s, RAKP/CAD 32–55 keys/s, and OSPF 29–41 keys/s) reflect the combined effects of adaptive load balancing and link saturation. Despite rising failure ratios, heterogeneous link utilization ensures continued key delivery, while marginal key pool size constraints gradually slow throughput growth and induce longer queues and localized failures.

4.2.3. Single-Node Key Transmission Rate: Performance Variation

This section presents a detailed comparison of QKD routing algorithms across varying single-node key transmission rates, highlighting the coupled behavior of average distribution time ($T_{\mathrm{avg}}$), over-threshold link ratio ($r_{\mathrm{over}}$), and failure ratio ($R_{\mathrm{fail}}$). These interdependent trends reflect the combined influence of algorithmic adaptation, cumulative multi-hop effects, and the hub-centric structure of BA topologies.

As shown in Figure 6, Q-learning consistently outperforms other algorithms under varying key rates by dynamically learning optimal paths and balancing network load. At low rates, initial exploration leads to moderate delays ($T_{\mathrm{avg}}\approx 124\mathrm{s}$) and minimal overload ($r_{\mathrm{over}}\approx 0.2\%$), as the agent gradually discovers efficient paths. RAKP, CAD, and OSPF exhibit higher delays and slightly increased overload, reflecting limited adaptivity or load-insensitive routing that cannot fully exploit underutilized links. As key rates increase, Q-learning maintains moderate delay growth and low failure ratio by proactively avoiding emerging bottlenecks and distributing traffic across less congested links. In contrast, RAKP and CAD experience higher delays and failures due to partial or local adaptivity, while OSPF suffers sharp escalations (delays up to $T_{\mathrm{avg}}\approx 590\mathrm{s}$ and a substantial fraction of failures in the 0.6–0.8 range), illustrating the vulnerability of non-adaptive strategies to multi-hop congestion and saturation of core links.

Network performance exhibits nonlinear trends with increasing single-node key transmission rates, reflecting cumulative multi-hop effects as minor link overloads propagate. At low key rates, Q-learning exhibits a U-shaped delay due to queue initialization and exploration. As the key rate increases, CAD and OSPF experience sharp rises in average delay and failure ratio, whereas Q-learning consistently maintains lower latency. Extreme OSPF degradation highlights the limitations of topology-driven routing in hub-centric BA networks. These patterns arise from the interplay of algorithmic adaptability—Q-learning dynamically mitigates bottlenecks, RAKP partially optimizes paths, CAD struggles under congestion, and OSPF remains non-adaptive—and topology-induced hub congestion.

4.2.4. Network Scale: Performance Variation

The network size strongly affects QKD performance, impacting average distribution time and failure ratio. As the network expands from 50 to 200 nodes, longer paths, increased topology complexity, and intensified link contention naturally lead to higher distribution delays and elevated failure probabilities. These metrics jointly reflect the network’s capacity to balance load, mitigate congestion, and sustain reliable key delivery.

As shown in Figure 7, analysis under increasing node counts reveals distinct algorithmic patterns. Q-learning shows the slowest growth in distribution time (44 → 60 s) and failure ratio (0.056 → 0.086), benefiting from global load awareness and online learning that disperse traffic and prevent bottlenecks. RAKP exhibits moderate increases (50.5 → 85 s, 0.082 → 0.170), with mid-node acceleration followed by high-node stabilization, reflecting its residual-ratio allocation that mitigates local congestion while preserving balance. CAD incurs higher delays (57.5 → 105 s) and sharper failure growth (0.165 → 0.310) as its local congestion-aware routing accumulates bottlenecks with scale. OSPF performs worst, with delay increasing (65.5 → 123 s) and failure ratio rising (0.246 → 0.350), as its load-unaware shortest-path routing leads to escalating congestion and limited capacity for load balancing compared with adaptive strategies.

Cross-comparison shows that node expansion amplifies path length, contention, and blocking probability, thereby magnifying algorithmic differences. Adaptive strategies such as Q-learning, with global load awareness, and RAKP, with residual-ratio allocation, effectively redistribute traffic to contain delay and failure growth. In contrast, CAD accumulates congestion due to reliance on local remaining-key information, while OSPF suffers persistent overload under shortest-path routing. Thus, scaling not only increases delay and failure but also highlights each algorithm’s efficiency in load balancing and congestion resilience.

Q-learning for QKD routing has low online computational complexity, as decisions depend only on the current node and path length, while offline training is more demanding due to the size of the state–action space and number of training episodes. The algorithm scales well in moderately sized or highly connected networks through adaptive load balancing, but in extremely large-scale or ultra-dense networks, performance may be limited by state-space growth, memory requirements, and restricted generalization, potentially requiring hierarchical or function-approximated reinforcement learning.

4.2.5. Link Failure Probability: Performance Variation

This section examines how increasing link failure probability affects QKD network performance, highlighting its substantial impact on average key distribution time and failure ratio, with algorithmic performance gaps widening at higher failure probabilities.

As shown in Figure 8, with increasing link failure probability, both distribution time and failure ratio exhibit coherent yet differentiated trends across algorithms. Q-learning exhibits the slowest growth in average delay, rising from 43.4 s to 85.0 s. Its failure ratio remains low and stable at low link failure probabilities due to proactive avoidance of sparse failures and local congestion but increases to 0.244 at higher probabilities as network constraints become unavoidable. RAKP maintains moderate performance (49.97 → 69.12 s) at low to medium link failure probability, but under higher probabilities, delay surges to 103.94 s and failures (0.082 → 0.372) accelerate as local bottlenecks emerge. CAD initially behaves like RAKP but proves more congestion-sensitive, with delay expanding from 57.46 to 134.34 s and failure ratio from 0.165 to 0.517, reflecting its lack of global load balancing. OSPF, constrained by shortest-path routing, consistently performs worst: distribution time escalates to 184.21 s and failure ratio from 0.246 to 0.670, as it cannot reroute around congested or failed links.

At low link failure probabilities, abundant path availability allows global optimization algorithms (Q-learning and RAKP) to fully exploit network resources, resulting in distribution time and failure curves that increase slowly and remain relatively low. As link failure probabilities increase, path scarcity and congestion bottlenecks disproportionately affect local and non-adaptive strategies, amplifying algorithmic differences. The observed nonlinear growth and widening algorithmic differences reflect the interplay of path availability, congestion dynamics, and algorithmic design. At high link failure probabilities, performance gaps stabilize as network resources become saturated, with Q-learning mitigating degradation through dynamic path selection and load balancing.

5. Conclusions

This paper presents a novel Q-learning-based key scheduling framework designed for trusted-relay QKD networks, addressing the challenges posed by dynamic key consumption and heterogeneous resource allocation. The proposed framework comprises three integral components: a Neighbor State Awareness Module that provides comprehensive monitoring of adjacent link parameters; an Online Q-learning Scheduler that adaptively optimizes resource utilization through a rigorously defined reward function; and a Routing and Validation Module that guarantees feasible and resource-aware path selection under network constraints.

The episodic online learning paradigm enables real-time, fine-grained policy updates, ensuring balanced key pool utilization, preventing resource over-concentration, and maintaining stable replenishment. Simulation results demonstrate that Q-learning consistently outperforms RAKP, CAD, and OSPF across dynamic QKD network. Under total key volume variation, it limits failure ratios to 0–0.1 and achieves 61 keys/s throughput; when varying the single-node key transmission rate, average distribution time remains around 100 s with only 3.8% over-threshold links, indicating effective congestion mitigation; with increasing network scale, it sustains the lowest failure ratios; and under higher link failure probabilities, distribution time stays below 85 s, reflecting robust load balancing, adaptability, and efficiency. Future work could integrate reinforcement learning, graph neural networks, and hybrid optimization strategies to enable scalable, self-adaptive QKD network that optimize reliability, efficiency, and quantum resource utilization in ultra-large, heterogeneous, and dynamic environments.