Quantum Deep Q-Network for Intelligent Packet Routing in 6G Heterogeneous Wireless Networks

Abstract

Intelligent packet routing in sixth-generation (6G) heterogeneous wireless networks must contend with stochastic link failures, heterogeneous delay profiles, and the severe memory constraints of edge nodes. We propose a quantum deep Q-network (Q-DQN) that replaces the multi-layer perceptron in a standard DQN agent with a six-qubit variational quantum circuit (VQC) employing ring-topology entanglement and angle embedding. The total trainable parameter count follows the closed-form expression $\left|\mathsf{\varphi }\right|=12L+7n$, growing at only seven parameters per additional network node. On a 10-node heterogeneous topology with stochastic link failures, Q-DQN achieves an average end-to-end delay of $54.29\pm 1.72$ ms with only 106 parameters, a 49.6× reduction relative to the MLP-based DQN baseline (5258 parameters, $52.89\pm 2.67$ ms). A three-seed scalability evaluation across $n\in \{6,8,10,12\}$ nodes shows that under a limited 200-episode training budget DQN converges more consistently, while Q-DQN matches DQN performance under full 500-episode training at a fraction of the parameter cost. Ablation experiments confirm that local-topology entanglement substantially outperforms full-connection alternatives. These results indicate that VQC-based routing agents can match classical counterparts at a fraction of the parameter cost, providing a path toward ultra-lightweight intelligent routing in 6G edge deployments.

1. Introduction

The design of routing protocols for sixth-generation (6G) wireless systems must contend with challenges that traditional approaches handle only imperfectly. Heterogeneous access technologies (including millimetre-wave backhaul, sub-6 GHz mesh links, and free-space optical segments) produce highly variable and non-stationary delay distributions [1,2]. Link failures due to mobility, interference bursts, and hardware faults occur at rates that make pre-computed routing tables stale within seconds [3,4]. At the same time, the edge nodes responsible for forwarding decisions are often microcontroller-class devices with kilobytes, not gigabytes, of working memory [5]. This combination of dynamic environments and hard resource constraints leaves a gap that neither classical shortest-path algorithms nor large neural network-based policies can fully address.

Dijkstra’s algorithm [6] and its successors deliver near-optimal delay when the complete link-state information is available. In practice, however, gathering and distributing global topology data in a large heterogeneous network introduces communication overhead that itself degrades performance; these centralised algorithms also cannot react to transient failures faster than the link-state update period. Reinforcement learning (RL)-based routing, pioneered by work such as [7,8], addresses both shortcomings by learning decentralised routing policies from local observations, without requiring global topology knowledge. Deep Q-networks (DQN) [9] extend Q-learning [10] with a non-linear function approximator, enabling generalisation to continuous or large state spaces. A growing body of work has demonstrated the effectiveness of DQN-based routing in 5G and 6G-style networks [11,12,13,14], and the survey [5] provides a comprehensive treatment of model-driven versus data-driven routing paradigms.

Despite these advances, the parameter footprint of deep neural networks remains a barrier to deployment on edge nodes. A two-hidden-layer MLP acting as a DQN policy for a 10-node network already requires on the order of 5000 parameters; scaling to 100-node topologies pushes this to hundreds of thousands, far exceeding the flash memory available on typical embedded forwarding chips. Pruning and quantisation can reduce this overhead, but they typically degrade policy quality and add engineering complexity.

Variational quantum circuits (VQC) offer a structurally different path to compact function approximation. The quantum state space of $n_q$ qubits scales as $2^{n_q}$ dimensions, providing rich representational capacity with a number of trainable parameters that is linear in $n_q$ and the circuit depth L [15,16,17]. Recent theoretical work has established that VQCs can represent function classes inaccessible to polynomial-size classical circuits under standard complexity assumptions [18,19,20], and early empirical studies have shown that VQC-based agents can match classical DQN performance on simple control tasks with one to two orders of magnitude fewer parameters [21,22,23]. However, the application of VQC-based reinforcement learning to network routing, a domain with discrete, high-dimensional action spaces and non-stationary environments, has received limited attention.

This paper introduces a Quantum Deep Q-Network (Q-DQN) designed specifically for intelligent routing in 6G heterogeneous networks. The key idea is to replace the MLP inside a DQN agent with an $n_q$ = 6 qubit VQC that uses angle embedding for state encoding, ring-topology CNOT gates for entanglement, and a classical linear layer for action-space readout. The resulting architecture contains only $12L+7n$ parameters for an n-node network with L circuit layers, compared with thousands for a classical MLP. We train and evaluate the agent on a 10-node heterogeneous topology subject to random link failures and recoveries, comparing against a DQN baseline and Dijkstra’s algorithm, which serves as a lower bound on achievable delay under perfect global link-state knowledge. Ablation studies examine entanglement topology, circuit depth, network scale, and generalisation to unseen topologies.

The main contributions of this work are summarised as follows.

• We propose Q-DQN, a VQC-based routing agent whose total parameter count follows the closed-form expression $\left|\mathsf{\varphi }\right|=12L+7n$, growing at only seven parameters per additional network node. Existing DRL routing methods [11,12,13] rely on MLP-based policies with thousands of parameters (5258 for the DQN baseline in our setting) and provide no analytical parameter bound; Q-DQN reduces the parameter count by 49.6× while achieving a routing delay within 1.40 ms of the MLP baseline. The quantum routing agents of Wu et al. [24] and Li et al. [25] adopt heuristic circuit designs without systematic entanglement topology analysis; by contrast, we conduct a controlled topology ablation showing that local-topology entanglement (ring and linear) substantially outperforms full-connection alternatives (56.59–57.10 ms vs. 84.44 ms), and we provide a theoretical account of why ring entanglement mitigates barren-plateau gradient suppression [26] via closed-loop connectivity in routing circuits.
• Through three-seed experiments across four network sizes ($n\in \{6,8,10,12\}$), we provide multi-seed scalability evidence that Q-DQN maintains competitive routing performance under the full 500-episode training budget at 44.9×–64.1× fewer parameters across all tested scales. Under a shorter 200-episode budget, Q-DQN shows higher variance than DQN (e.g., 83.44 ± 38.18 ms vs. 46.17 ± 5.52 ms at n = 10), reflecting the longer warm-up required by the variational quantum circuit; under the full budget, Q-DQN achieves 54.29 ± 1.72 ms (within 1.40 ms of DQN) at 49.6× fewer parameters. Prior quantum RL studies of Lockwood and Si [22], Chen et al. [21], and Skolik et al. [23] evaluate only small fixed-size toy environments (CartPole, MountainCar) and do not investigate scale dependence; to the best of our knowledge, this work is one of the few attempts to examine how VQC-based routing performance varies with network size. The parameter-efficiency advantage (49.6× reduction) is the primary demonstrated contribution of Q-DQN over classical baselines at comparable routing delay.
• We conduct a systematic empirical study of the proposed routing agent covering link failure rate sweeps ($p_f\in [0.005,0.08]$), recovery rate sensitivity ($p_r\in [0.05,0.30]$), quantum gate noise (${\sigma }_{\mathrm{noise}}\in [0,0.20]$), circuit depth ablation ($L\in \{1,2,3,4\}$), entanglement topology ablation, and topology generalisation across four unseen graph seeds. The robustness evaluations in the concurrent works [24,27] report results on at most two failure-probability settings and a single fixed topology; our evaluation provides statistically grounded conclusions (three independent evaluation runs, mean ± $\sigma$) across the full operating envelope relevant to 6G deployments.

The remainder of this paper is organised as follows. Section 2 surveys related work. Section 3 introduces the network model and MDP formulation. Section 4 details the Q-DQN architecture and training procedure. Section 5 presents and analyses the experimental results. Section 7 concludes and outlines future directions.

2. Related Work

2.1. Deep Reinforcement Learning for Network Routing

The use of DRL for routing dates to early demonstrations of Q-learning applied to packet networks. Valadarsky et al. [7] showed that a neural Q-function could adapt routes in response to changing traffic matrices without requiring explicit topology knowledge. Xu et al. [8] subsequently demonstrated that experience-driven DRL surpasses traditional OSPF-based routing in terms of throughput and delay under dynamic traffic conditions. More recent work has focused on heterogeneous 5G/6G settings: Nguyen et al. [11] provide a comprehensive survey of DRL routing methods for 5G-and-beyond networks, while Lin et al. [12] propose a multi-objective DRL framework specifically tailored to 6G heterogeneous topologies. Wang et al. [13] extend this line of work by addressing link-failure dynamics explicitly through an adaptive DRL policy trained with simulated failure scenarios, and Fu et al. [14] study resilient routing in wireless mesh networks under failure-injection stress tests. Despite the demonstrated effectiveness of these approaches, all of them rely on classical MLPs with thousands of parameters, leaving the on-device deployment challenge unaddressed.

2.2. Variational Quantum Circuits and Quantum ML

Variational quantum circuits [15] are parameterised quantum operations trainable via classical optimisers through gradient estimation, either analytically via the parameter-shift rule or via finite differences. Mitarai et al. [28] established the theoretical foundations of quantum circuit learning and showed that VQCs can approximate arbitrary functions given sufficient depth and width. The data re-uploading technique of Pérez-Salinas et al. [29] provides a practical mechanism for encoding high-dimensional classical data into low-qubit circuits by repeated angle embedding across layers. Schuld et al. [30] analysed the effect of different encoding strategies on expressibility, providing guidelines that inform our angle embedding design. The elementary gates employed in each variational block ($R_y$, $R_z$, and CNOT) form a universal gate set for one- and two-qubit quantum operations [31,32]: any unitary transformation on $n_q$ qubits can be approximated to arbitrary precision by a finite composition of these operations, which underpins the theoretical expressibility of the proposed VQC ansatz. A known challenge in VQC training is the barren plateau phenomenon [26], where gradients vanish exponentially with qubit count under random initialisation; ring entanglement with localised circuit structure is known to mitigate this problem [15]. Theoretical limits on VQC expressibility are studied in [18,19,20].

2.3. Quantum Reinforcement Learning

The application of quantum circuits to reinforcement learning was introduced by Lockwood and Si [22] and by Chen et al. [21], who independently demonstrated that VQC-based DQN agents can match classical agents on CartPole and similar benchmarks with dramatically fewer parameters. Skolik et al. [23] extended this to the Gym suite and provided a theoretical analysis of the expressibility of quantum policies. Jerbi et al. [33] further showed that VQC policies can surpass classical ones on certain structured tasks, suggesting a genuine computational advantage rather than a parameter-count coincidence. From the network domain, Wu et al. [24] applied Q-DQN to 6G adaptive control and reported parameter savings consistent with ours. Kwak et al. [34] provide an accessible comparison of quantum and classical RL, and Kim et al. [35] propose a hybrid quantum-classical DRL for heterogeneous resource allocation. Surveys of VQC applications to wireless networking [36] and quantum routing specifically [25,27] confirm that this area is attracting growing attention. Our work advances this literature by providing a systematic ablation and robustness evaluation of a VQC-based routing agent in a dynamic 6G-inspired routing simulation, and by offering preliminary evidence of scale-dependent parameter efficiency under limited training budgets.

3. System Model and Problem Formulation

3.1. Network Topology and Link Model

We consider a heterogeneous wireless network modelled as a directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of $n=\left|\mathcal{V}\right|$ nodes and $\mathcal{E}$ is the set of directed links. The primary evaluation topology has $n=10$ nodes and 20 directed links (10 bidirectional physical connections, each represented as a pair of directed edges in $\mathcal{E}$). Each link $(i,j)\in \mathcal{E}$ carries a propagation delay ${\delta }_{ij}$ drawn from a uniform distribution over $[1,10]$ ms at topology initialisation, reflecting the heterogeneous delay characteristics of mixed millimetre-wave, sub-6 GHz, and wired backhaul segments in a 6G deployment.

Link availability follows a two-state Markov model. At each discrete time step, the binary state of link $(i,j)$ evolves as

$${\mathcal{l}}_{ij}^{(t+1)}=\left\{\begin{array}{cc}0\hfill & \mathrm{with}\mathrm{prob}.p_f,\mathrm{if}{\mathcal{l}}_{ij}^{\left(t\right)}=1,\hfill \\ 1\hfill & \mathrm{with}\mathrm{prob}.p_r,\mathrm{if}{\mathcal{l}}_{ij}^{\left(t\right)}=0,\hfill \end{array}\right.$$

(1)

where failures and recoveries are independent across links and time steps. The stationary probability that a link is active is

$$\overline{\mathcal{l}}={\displaystyle \frac{p_r}{p_f+p_r}},$$

(2)

which evaluates to $\overline{\mathcal{l}}\approx 0.833$ under the base configuration $p_f=0.02$, $p_r=0.10$. The expected number of concurrently active links is therefore $\overline{\mathcal{l}}·\left|\mathcal{E}\right|\approx 16.7$ out of 20, which is representative of the transient blockage dynamics of millimetre-wave channels [2].

3.2. MDP Formulation

The routing problem is cast as a Markov Decision Process (MDP) $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma )$. The agent’s observation at step t is a six-dimensional feature vector

$${\mathbf{s}}_t={[f_1,f_2,f_3,f_4,f_5,f_6]}^{\top }\in {[-1,1]}^6,$$

(3)

computed from locally observable quantities at the current node. The six features are: $f_1$, the minimum propagation delay among active outgoing links, normalised by the maximum link delay; $f_2$, the minimum packet-loss rate among active outgoing links, normalised by the maximum loss rate; $f_3$, the maximum available bandwidth among active outgoing links, normalised by the maximum bandwidth; $f_4$, the absolute difference between the current node index and the destination node index, normalised by n, serving as a proximity proxy; $f_5$, the fraction of the current node’s links that are presently active (link-availability ratio); and $f_6$, the current step index normalised by the episode length, capturing time pressure. All features lie in $[0,1]$ and are linearly scaled to $[-1,1]$ before use, matching the Bloch-sphere convention of the angle embedding. The six-dimensional state is directly supplied to both the Q-DQN and DQN agents, requiring no additional dimensionality reduction.

The action $a_t\in \mathcal{A}=\{0,1,\dots ,n-1\}$ selects the next-hop node from the current position. The valid action set ${\mathcal{A}}_{\mathrm{valid}}\left({\mathbf{s}}_t\right)\subseteq \mathcal{A}$ contains only those next-hop nodes reachable through currently active links; invalid actions are suppressed via the Q-value masking in Equation (6), which applies identically during both training and evaluation.

The transition $P({\mathbf{s}}_{t+1}\mid {\mathbf{s}}_t,a_t)$ is governed by the packet movement along the selected hop and the stochastic link-state updates. Both are assumed Markov: the link-state update follows Equation (1), and the next state ${\mathbf{s}}_{t+1}$ is recomputed from the updated link attributes at the new current node.

The per-step reward combines a delay penalty, a packet-loss penalty, and a completion bonus:

$$r_t=\left\{\begin{array}{cc}B-{\displaystyle \frac{{\delta }_{a_t}}{{\delta }_{max}}}-{\displaystyle \frac{{\rho }_{a_t}}{{\rho }_{max}}}\hfill & \mathrm{if}{a}_t\mathrm{reaches}\mathrm{destination},\hfill \\ -{\displaystyle \frac{{\delta }_{a_t}}{{\delta }_{max}}}-{\displaystyle \frac{\lambda {\rho }_{a_t}}{{\rho }_{max}}}\hfill & \mathrm{if}\mathrm{valid}\mathrm{hop},\mathrm{not}\mathrm{yet}\mathrm{at}\mathrm{destination},\hfill \end{array}\right.$$

(4)

where ${\delta }_{a_t}\in [1,10]$ ms is the link delay, ${\rho }_{a_t}\in [0.01,0.10]$ is the link packet-loss rate, ${\delta }_{max}=10$ ms, ${\rho }_{max}=0.10$, $B=10$ is the completion bonus, and $\lambda =0.5$ down-weights the loss penalty on intermediate hops. Invalid actions are suppressed by the Q-value masking (Equation (6)). The primary evaluation metric, average end-to-end delay, is the sum of per-hop delays ${\sum }_t{\delta }_{a_t}$ over each episode, computed separately from the training reward. The agent maximises the expected discounted return $G_t={\sum }_{k=0}^{\infty }{\gamma }^k{r}_{t+k}$ with discount factor $\gamma =0.95$.

The optimal Q-function $Q^{*}:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ satisfies the Bellman optimality equation:

$$Q^{*}(\mathbf{s},a)=\mathbb{E}\left[r(\mathbf{s},a)+\gamma \underset{a^{\prime }}{max}{Q}^{*}({\mathbf{s}}^{\prime },a^{\prime })|\mathbf{s},a\right],$$

(5)

where the expectation is taken over the stochastic transition $P({\mathbf{s}}^{\prime }\mid \mathbf{s},a)$ induced by Equation (1). The optimal policy is recovered by ${\pi }^{*}\left(\mathbf{s}\right)=arg{max}_a{Q}^{*}(\mathbf{s},a)$. Because the Q-function is defined over a continuous state space, we approximate $Q^{*}$ by the VQC-based function approximator $\widehat{Q}(\mathbf{s},a;\mathsf{\varphi })$ described in Section 4.

To prevent the agent from selecting links that are currently failed, we apply an availability mask ${\mathbf{m}}_t\in {\{0,1\}}^n$, where $m_t^{\left(a\right)}={\mathcal{l}}_{sa}^{\left(t\right)}$ for the link from the current node s to candidate next-hop a. The masked Q-values used for action selection are

$$\tilde{Q}({\mathbf{s}}_t,a;\mathsf{\varphi })=\widehat{Q}({\mathbf{s}}_t,a;\mathsf{\varphi })-\mathcal{M}\left(1-m_t^{\left(a\right)}\right),$$

(6)

where $\mathcal{M}={10}^9$ is a large constant that effectively sets $\tilde{Q}\to -\infty$ for unavailable actions, ensuring they are never selected greedily.

Dijkstra’s algorithm [6] is computed at each episode using the full, instantaneous link-state to provide a lower bound on achievable end-to-end delay, assuming perfect global link-state information is available.

3.3. Performance Metrics

We evaluate all agents on three primary metrics: (i) average end-to-end delay(ms), measured over 80 evaluation episodes after training; (ii) TD loss, measuring the mean Bellman residual over 80 test episodes after training; and (iii) trainable parameter count, which proxies for on-device memory cost. Secondary metrics include delay variance and per-episode reward convergence curves.

4. Proposed Quantum Deep Q-Network

4.1. Architecture

The Q-DQN agent maintains a quantum function approximator $\widehat{Q}(\mathbf{s},a;\mathsf{\varphi })$ that estimates the expected discounted return for taking action a in state $\mathbf{s}$, given trainable parameters $\mathsf{\varphi }$ (see

Table 1. Parameter count for Q-DQN vs. DQN as a function of network size n (fixed $L=3$) and circuit depth L (fixed $n=10$).

	Varying n ($\mathit{L}=3$)		Varying L ($\mathit{n}=10$)
	Q-DQN	DQN	Q-DQN	DQN
$n=6$, $L=3$	78	4998	—	—
$n=8$, $L=3$	92	5128	—	—
$n=10$, $L=3$	106	5258	—	—
$n=12$, $L=3$	120	5388	—	—
$n=10$, $L=1$	—	—	82	5258
$n=10$, $L=2$	—	—	94	5258
$n=10$, $L=3$	—	—	106	5258
$n=10$, $L=4$	—	—	118	5258
Growth rate	$+7$/node vs. $+65$/node		$+12$/layer vs. 0/layer

for a complete breakdown of parameter counts across configurations). The agent selects actions $\epsilon$-greedily:

$$a_t=\left\{\begin{array}{cc}\mathrm{random}\in {\mathcal{A}}_{\mathrm{valid}}\left({\mathbf{s}}_t\right)\hfill & \mathrm{with}\mathrm{prob}.{\epsilon }_t,\hfill \\ arg\underset{a}{max}\tilde{Q}({\mathbf{s}}_t,a;\mathsf{\varphi })\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(7)

The overall Q-DQN framework is illustrated in Figure 1, which decomposes the proposed system into four sequential stages.

As shown in Figure 1, the pipeline proceeds as follows. Stage 1 (dynamic network and MDP) models the heterogeneous 10-node topology as a directed graph subject to stochastic two-state Markov link failures; at each step the agent observes a six-dimensional local state ${\mathbf{s}}_t={[f_1,\dots ,f_6]}^{\top }$ and selects a next-hop from the failure-aware valid action set ${\mathcal{A}}_{\mathrm{valid}}$, with Dijkstra’s algorithm retained solely as a global-state lower bound. Stage 2 (variational quantum Q-function approximator) encodes the state via six-qubit angle embedding $U\left({\mathbf{s}}_t\right)={⨂}_{i=1}^6{R}_y\left(\pi s_i^{\mathrm{norm}}\right)H^{\otimes 6}$, processes it through L variational layers that each apply trainable $R_y$/$R_z$ rotations interleaved with a parameter-free ring-CNOT entangler, and maps the resulting Pauli-Z measurement vector to per-action Q-values via a lightweight classical readout $\widehat{Q}({\mathbf{s}}_t;\mathsf{\varphi })=W_{\mathrm{out}}\mathbf{m}+b_{\mathrm{out}}$, yielding a total of $\left|\mathsf{\varphi }\right|=12L+7n=106$ trainable parameters for n = 10, L = 3. Stage 3 (masked TD learning) suppresses failed-link actions through a large-constant Q-value mask $\tilde{Q}({\mathbf{s}}_t,a)=\widehat{Q}({\mathbf{s}}_t,a)-\mathcal{M}(1-m_t^{\left(a\right)})$, computes TD targets with a periodically frozen target network, and minimises the Huber loss $\mathcal{L}\left(\mathsf{\varphi }\right)=\mathbb{E}\left[{\mathcal{l}}_H(y_t-{\widehat{Q}}_t)\right]$ via Adam, with VQC rotation gradients obtained exactly by the parameter-shift rule ${\partial }_{{\mathsf{\varphi }}_k}\langle O\rangle =\frac{1}{2}({\langle O\rangle }_{+}-{\langle O\rangle }_{-})$. Stage 4 (routing policy) executes the greedy policy ${\pi }_t=arg{max}_a{\tilde{Q}}_t\left(a\right)$ under live failure conditions, achieving $54.29\pm 1.72$ ms end-to-end delay with 49.6× fewer parameters than the DQN baseline. The bottom row of the figure summarises the four key theoretical anchors: the closed-form parameter-count expression $\left|\mathsf{\varphi }\right|=12L+7n\ll |{\theta }_{\mathrm{MLP}}|$, the failure-aware action space ${\mathcal{A}}_{\mathrm{valid}}\left({\mathbf{s}}_t\right)$, the quantum feature map $\mathcal{Q}:{[-1,1]}^6\to {\mathbb{R}}^n$, and the Pauli-Z measurement output $\langle Z_i\rangle \in [-1,1]$ feeding the classical readout. The architecture of the quantum function approximator is described in detail below.

4.2. VQC Architecture

4.2.1. State Encoding via Angle Embedding

The state ${\mathbf{s}}_t\in {\mathbb{R}}^6$ is a fixed-length vector of six engineered features derived directly from the network observation at time t: (1) minimum neighbour link delay normalised by ${\delta }_{max}$; (2) minimum link loss rate normalised by ${\rho }_{max}$; (3) maximum link bandwidth normalised by $b_{max}$; (4) absolute node-index difference to destination, normalised by n; (5) fraction of available outgoing links; and (6) time-step progress $t/T_{max}$. All six features are linearly mapped to $[-1,1]$ before embedding. The 6-dimensional vector is encoded into an $n_q=6$ qubit register without introducing any trainable parameters. Each qubit i is first prepared in the $|+\rangle$ state by a Hadamard gate H and then rotated by the i-th normalised feature:

$$U\left({\mathbf{s}}_t\right)=\left[{\displaystyle \underset{i=1}{\overset{n_q}{⨂}}}{R}_y\left({\tilde{s}}_i\right)\right]·H^{\otimes n_q},{\tilde{s}}_i=\pi ·s_i^{\left(\mathrm{norm}\right)}\in [0,\pi ],$$

(8)

where $s_i^{\left(\mathrm{norm}\right)}=(s_i+1)/2\in [0,1]$ (since $s_i\in [-1,1]$) and $R_y\left(\theta \right)=e^{-i\theta Y/2}$ is the single-qubit rotation about the y-axis with Y the Pauli-Y operator. Explicitly,

$$R_y\left(\theta \right)=\left(\begin{array}{cc}cos(\theta /2)& -sin(\theta /2)\\ sin(\theta /2)& cos(\theta /2)\end{array}\right).$$

(9)

The resulting encoded state is $|{\psi }_{\mathrm{enc}}\rangle =U\left({\mathbf{s}}_t\right){|0\rangle }^{\otimes n_q}$, a product state whose Bloch angles carry the network observation. Classical MLP inputs pass raw features linearly to the first hidden layer; angle embedding instead maps each feature to a point on the Bloch sphere, enabling the subsequent quantum gates to exploit interference between features encoded in different qubits, a mechanism with no direct classical analogue.

4.2.2. Variational Layers with Ring Entanglement

The circuit contains L identical variational blocks. Each block l applies, in order: (i) single-qubit $R_y$ rotations with trainable angles ${\left\{{\mathsf{\varphi }}_{l,i}\right\}}_{i=1}^{n_q}$; (ii) single-qubit $R_z$ rotations with trainable angles ${\left\{{\phi }_{l,i}\right\}}_{i=1}^{n_q}$; and (iii) a ring of parameter-free CNOT gates, where qubit i is the control and qubit $(imod{n}_q)+1$ is the target. The ring entangler is

$${\mathrm{CNOT}}_{\mathrm{ring}}={\displaystyle \prod _{i=1}^{n_q}}{\mathrm{CNOT}}_{i,(imod{n}_q)+1},$$

(10)

and the full l-th block is

$$W_l({\mathsf{\varphi }}_l,{\mathsf{\phi }}_l)={\mathrm{CNOT}}_{\mathrm{ring}}·\left[{\displaystyle \underset{i=1}{\overset{n_q}{⨂}}}{R}_z\left({\phi }_{l,i}\right)\right]·\left[{\displaystyle \underset{i=1}{\overset{n_q}{⨂}}}{R}_y\left({\varphi }_{l,i}\right)\right],$$

(11)

contributing exactly $2n_q=12$ trainable rotation parameters per layer. The choice of ring entanglement is motivated by its empirically favourable gradient landscape. Full-connection entanglement (all-to-all CNOT) introduces $n_q(n_q-1)/2=15$ CNOT gates per layer and causes gradient suppression akin to the barren-plateau phenomenon [26]; the ring topology instead generates local quantum correlations that preserve gradient magnitude during training [15].

4.2.3. Complete Circuit State and Q-Value Estimation

The output quantum state after all L blocks is

$$|\mathsf{\Psi }({\mathbf{s}}_t;\mathsf{\varphi })\rangle =W_L({\mathsf{\varphi }}_L,{\phi }_L)\cdots W_1({\mathsf{\varphi }}_1,{\phi }_1)U\left({\mathbf{s}}_t\right){|0\rangle }^{\otimes n_q}.$$

(12)

The expectation value of the Pauli-Z observable on qubit i is obtained as

$$m_i({\mathbf{s}}_t;\mathsf{\varphi })=\langle \mathsf{\Psi }|{\mathbb{I}}^{\otimes (i-1)}\otimes Z\otimes {\mathbb{I}}^{\otimes (n_q-i)}|\mathsf{\Psi }\rangle \in [-1,1],$$

(13)

yielding the measurement vector $\mathbf{m}={[m_1,\dots ,m_{n_q}]}^{\top }$. A classical linear readout maps $\mathbf{m}$ to Q-values:

$$\widehat{Q}({\mathbf{s}}_t;\mathsf{\varphi })={\mathbf{W}}_{\mathrm{out}}\mathbf{m}({\mathbf{s}}_t;\mathsf{\varphi })+{\mathbf{b}}_{\mathrm{out}},$$

(14)

where ${\mathbf{W}}_{\mathrm{out}}\in {\mathbb{R}}^{n\times n_q}$ and ${\mathbf{b}}_{\mathrm{out}}\in {\mathbb{R}}^n$ together contribute $n(n_q+1)=7n$ trainable parameters. Value-network architectures in prior quantum RL work [21,23] use a separate measurement per action and thus scale the circuit width with the action-space size. Our readout design avoids this by keeping the quantum circuit fixed at $n_q=6$ qubits; the action dimensionality n is absorbed entirely by the cheap classical linear layer, so the circuit depth, not the qubit count, is the primary complexity knob.

4.2.4. Parameter Count and Comparison with Classical MLP

Combining the VQC rotations and the readout layer:

$$\left|\mathsf{\varphi }\right|=2n_{q}L+n(n_q+1)=12L+7n.$$

(15)

For the primary setting ($L=3$, $n=10$): $\left|\mathsf{\varphi }\right|=36+70=106$. The DQN baseline uses the same 6-dimensional engineered state input and 10-dimensional action output; a two-hidden-layer MLP with 64 units per layer on this input requires 5258 parameters, yielding a parameter ratio of $5258/106\approx 49.6\times$. Table 1 contrasts the parameter growth rates of the two architectures as n and L vary.

4.3. Entanglement Topology Variants

We consider three entanglement topologies for the CNOT layer:

• Ring: qubit i controls qubit $(imod{n}_q)+1$, forming a closed loop.
• Linear: qubit i controls qubit $i+1$ for $i=1,\dots ,n_q-1$, an open chain.
• Full: every qubit controls every other qubit, introducing $n_q(n_q-1)/2$ CNOT gates per layer.

Ring entanglement is adopted as the default configuration for its closed-loop connectivity, which theoretically supports better gradient flow than the open-chain linear topology; the ablation in Section 5.6 confirms that both local topologies substantially outperform full-connection entanglement.

4.4. Training Algorithm

Q-DQN is trained with experience replay and a periodically- updated target network following the DQN framework [9]. The training loss is the Huber (SmoothL1) Bellman error, which combines the stability of mean-absolute error for large TD errors with the smooth convergence of mean-squared error near zero:

$$\mathcal{L}\left(\mathsf{\varphi }\right)={\mathbb{E}}_{(\mathbf{s},a,r,{\mathbf{s}}^{\prime })\sim \mathcal{B}}\left[{\mathcal{l}}_H\left(y-\widehat{Q}(\mathbf{s},a;\mathsf{\varphi })\right)\right],$$

(16)

where ${\mathcal{l}}_H\left(\delta \right)=\frac{1}{2}{\delta }^2$ if $\left|\delta \right|\le 1$ and ${\mathcal{l}}_H\left(\delta \right)=\left|\delta \right|-\frac{1}{2}$ otherwise, $\mathcal{B}$ is the experience replay buffer, and the TD target is

$$y=r+\gamma (1-d)\underset{a^{\prime }\in {\mathcal{A}}_{\mathrm{valid}}\left({\mathbf{s}}^{\prime }\right)}{max}\tilde{Q}({\mathbf{s}}^{\prime },a^{\prime };{\mathsf{\varphi }}^{-}),$$

(17)

where $d\in \{0,1\}$ is the terminal flag ($d=1$ when the packet reaches the destination, reducing the target to $y=r$), ${\mathcal{A}}_{\mathrm{valid}}\left({\mathbf{s}}^{\prime }\right)$ excludes failed links via Equation (6), and ${\mathsf{\varphi }}^{-}$ denotes the frozen target-network parameters updated every C training steps.

VQC gradients cannot be obtained by auto-differentiation of a simulation graph when the circuit is executed on quantum hardware, unlike classical neural networks where back-propagation operates through piecewise-differentiable activations. We therefore use the parameter-shift rule [28], which states that for any parameter ${\varphi }_k$ in a rotation gate $R\left({\varphi }_k\right)=e^{-i{\varphi }_{k}G/2}$ with generator G satisfying $G^2=\mathbb{I}$:

$${\displaystyle \frac{\partial \langle O\rangle }{\partial {\varphi }_k}}={\displaystyle \frac{1}{2}}\left[\langle O\rangle {|}_{{\varphi }_k+\pi /2}-\langle O\rangle {|}_{{\varphi }_k-\pi /2}\right],$$

(18)

where $\langle O\rangle$ denotes the expectation value of any observable O. This identity holds exactly, not as an approximation, and requires only two circuit evaluations per parameter. Applying the chain rule through the readout layer (14), the full gradient of the TD loss with respect to ${\varphi }_k$ is

$${\displaystyle \frac{\partial \mathcal{L}}{\partial {\varphi }_k}}=-\mathbb{E}\left[{\mathcal{l}}_H^{\prime }\left(y-\widehat{Q}\right){\mathbf{W}}_{\mathrm{out}}^{(a,:)}{\displaystyle \frac{\partial \mathbf{m}}{\partial {\varphi }_k}}\right],$$

(19)

where ${\mathcal{l}}_H^{\prime }\left(\delta \right)=\delta \mathbf{1}\left[\right|\delta |\le 1]+sign\left(\delta \right)\mathbf{1}[\left|\delta \right|>1]$ is the Huber sub-gradient, ${\mathbf{W}}_{\mathrm{out}}^{(a,:)}$ is the a-th row of the readout weight matrix, and $\partial \mathbf{m}/\partial {\varphi }_k$ is computed via Equation (18) applied component-wise. The readout parameters ${\mathbf{W}}_{\mathrm{out}}$ and ${\mathbf{b}}_{\mathrm{out}}$ are updated by standard back-propagation. Both VQC and readout parameters are jointly optimised with the Adam optimiser [37] at a single learning rate $\alpha =0.003$ for Q-DQN and $\alpha =0.001$ for DQN. Gradients are clipped to a maximum ${\mathcal{l}}_2$ norm of 1.0 before each parameter update to prevent gradient explosion.

Algorithm 1 summarises the full procedure.   

Algorithm 1: Q-DQN Training for Network Routing

5. Experimental Evaluation

Table 2. Simulation and training hyperparameters.

Parameter	Value	Description
Network topology
n	10	Number of nodes
$\left|\mathcal{E}\right|$	20	Directed links, corresponding to 10 bidirectional physical connections
${\delta }_{ij}$	$\mathcal{U}(1,10)$ ms	Link propagation delay
$p_f$	0.02	Link failure probability
$p_r$	0.10	Link recovery probability
VQC-based Q-DQN
$n_q$	6	Number of qubits
L	3	Default number of variational layers
Entanglement	Ring	CNOT ring topology
Embedding	Angle $R_y$	${\tilde{s}}_i=\pi s_i^{\mathrm{norm}}$
$\left|\mathsf{\varphi }\right|$	106	Trainable parameters
DQN baseline
Architecture	2 × 64 MLP	Two hidden layers with ReLU
$\left|\mathsf{\theta }\right|$	5258	Trainable parameters
Training
Episodes E	500	Total training episodes
Steps/episode	30	Environment steps per episode
Buffer $\left|\mathcal{B}\right|$	10,000	Replay buffer capacity
Batch size	16/32	Q-DQN/DQN mini-batch size
$\gamma$	0.95	Discount factor
$\epsilon$ decay	$1.0\to 0.05$	Exploration schedule
Optimiser	Adam	$\alpha =0.003$ for Q-DQN, $0.001$ for DQN
Loss	Huber	SmoothL1 TD-error loss
Gradient clipping	1.0	Maximum gradient norm
Target update C	20/100	Q-DQN/DQN target-network update period

lists all simulation and training hyperparameters. The main evaluation uses a 10-node topology with $p_f=0.02$ and $p_r=0.10$. Each configuration is evaluated over three independent runs of 80 episodes; mean and standard deviation are reported. The scalability experiment (Section 5.5) uses three independent random seeds (mean ± std reported). Entanglement topology and circuit depth ablation (Section 5.6) use a reduced budget of 200 training episodes with a single random seed per configuration; these results should be read as indicative trends rather than definitive findings, and multi-seed replication is a priority for follow-up work. Different ablation tables employ independent seeds, so results for nominally identical settings (e.g., $n=10$, $L=3$, ring) may differ slightly due to seed-induced variance across independent runs. To ensure a fair comparison, the DQN baseline receives the same6-dimensional engineered state ${\mathbf{s}}_t$ as the Q-DQN agent. A two-hidden-layer MLP with 64 units per layer and 10 action outputs on this 6-dimensional input yields exactly 5258 trainable parameters. Hyperparameters of both agents (learning rate, batch size, target-update period) were chosen separately through a small validation grid to achieve stable convergence for each model; the reported values correspond to the best validation performance per agent rather than a setting biased toward either. Note that the state vector encodes aggregated per-node statistics rather than per-link attributes for each candidate next-hop; this compact local representation may limit the agent’s ability to distinguish fine-grained next-hop quality, and action-conditioned state representations are left for future work. All simulations are run on a machine with an Intel Core i7-12700K CPU and 32 GB RAM, using PennyLane 0.35 for VQC simulation and PyTorch 2.1 for gradient computation and replay buffer management.

5.1. Training Convergence

Figure 2 shows the per-episode average reward curves for Q-DQN and DQN during static training (no link failures). Both agents converge within the 500-episode budget. The Q-DQN curve exhibits more pronounced oscillation, which is attributable to the limited approximation capacity of the compact 106-parameter VQC: the circuit cannot represent the true Q-function as smoothly as the 5258-parameter MLP, so small distributional shifts in the replay buffer cause larger swings in the estimated Q-values. Additionally, on real quantum hardware the parameter-shift rule introduces finite-shot measurement noise absent from classical back-propagation; although our simulator computes parameter-shift gradients exactly, the structural difference in gradient estimation may still contribute to training variability through the non-stationarity of Bellman targets. The DQN curve is comparatively smoother due to the higher approximation capacity of the ReLU-based MLP. Despite this oscillation, the Q-DQN final average reward ($-4.80$ at episode 500) is comparable with DQN ($-4.50$), indicating that both policies converge to solutions of similar quality.

Figure 3 extends this analysis to the dynamic scenario with link failures, and additionally shows checkpoint delays evaluated every 100 episodes. The Q-DQN achieves its best checkpoint at episode 400 (41.59 ms), while DQN reaches its best at episode 500 (34.75 ms). The checkpoint trajectories illustrate that both agents improve progressively but neither monotonically, which is expected in non-stationary environments where each checkpoint corresponds to a slightly different failure configuration.

5.2. Baseline Performance Comparison

Table 3. Baseline performance comparison on the 10-node topology ($p_f=0.02$, $p_r=0.10$; averaged over 3 independent evaluation runs, 80 episodes each). DDQN results are reported as mean ± std across 3 independent seeds.

Method	Delay (ms)	TD Loss	Parameters	Global State?
Dijkstra [6]	$7.04\pm 0.03$	—	0	✓
DQN [9]	$52.89\pm 2.67$	$0.4872\pm 0.0292$	5258	×
DDQN [38]	$45.26\pm 2.84$	$0.4709\pm 0.1105$	5258	×
Q-DQN (ours)	$54.29\pm 1.72$	$0.6192\pm 0.0255$	106	×
Q-DQN vs. DQN gap	$+1.40$	$+0.1320$	$49.6\times$ fewer	—

Note: ✓/× under “Global State?” indicates whether the method requires full global link-state knowledge. Bold parameter count denotes the minimum among learned methods. The italic row summarises the Q-DQN/DQN performance gap.

and Figure 4 report the end-of-training evaluation results over 80 episodes. Q-DQN achieves $54.29\pm 1.72$ ms and DQN achieves $52.89\pm 2.67$ ms, a difference of only 1.40 ms, well within one standard deviation of either estimate. Double DQN (DDQN) [38], which reduces Q-value overestimation by selecting next-hop actions with the online network and evaluating them with the target network, achieves $45.26\pm 2.84$ ms across three independent seeds, approximately 7 ms lower than standard DQN and 9 ms lower than Q-DQN. This confirms that Q-DQN achieves competitive routing performance with 49.6× fewer parameters than both classical baselines. Both RL agents show substantially higher delay than Dijkstra’s $7.04\pm 0.03$ ms. This gap reflects the fundamental information asymmetry: Dijkstra operates as an oracle with full global link-state at every step, whereas Q-DQN and DQN use only local per-node observations. A simple local greedy heuristic that always selects the currently available neighbour with the minimum link delay was also tested and produced substantially higher delays than both learned policies, confirming that the Dijkstra gap is primarily attributable to partial observability rather than a failure of policy learning.

The comparison is intentionally scoped to DQN and DDQN as the primary learned baselines because they share an identical algorithmic framework with Q-DQN (experience replay, target network, $\epsilon$-greedy exploration) and differ only in the Q-function approximator, making the parameter-efficiency trade-off directly measurable without confounding algorithmic differences. Extending the comparison to proximal policy optimisation or graph-neural-network routing would each require adapted state representations and is left for future work.

Figure 5 presents a two-panel Pareto analysis aggregating results from all ablation configurations (entanglement topology and circuit depth, detailed in Section 5.6). The left panel plots average delay against parameter count on a log scale; the right panel shows a horizontal bar chart of absolute delays sorted by parameter count. Both panels show that the local-entanglement Q-DQN variants ($L=3$, ring or linear) occupy the Pareto frontier: they achieve substantially lower delay than any configuration with more parameters, and both outperform the full-entanglement variant at the same parameter count.

Figure 6 reinforces this finding by plotting delay versus the number of training samples consumed. Q-DQN reaches its best delay approximately 150 episodes earlier than DQN in this single-seed experiment at $n=10$, suggesting a potential sample efficiency advantage under these conditions. This is consistent with fewer parameters reducing the effective hypothesis space and limiting overfitting to the replay buffer. However, multi-seed scalability experiments (Section 5.5) show that under a 200-episode budget across multiple seeds and network sizes, DQN converges more consistently; the single-seed observation here should therefore not be interpreted as a general sample efficiency advantage.

Figure 7 shows the TD loss curves for both agents in static (left) and dynamic (right) training scenarios. In the static case, Q-DQN settles at a slightly lower training-time TD loss (≈0.21) than DQN (≈0.29) after 500 episodes. We attribute this to the compact VQC’s constrained hypothesis class, which prevents it from chasing small Bellman residuals that the MLP’s larger capacity would otherwise exploit in a stationary environment. Under dynamic conditions (right panel), the gap widens further: DQN’s training-time TD loss (≈0.56) is approximately twice that of Q-DQN (≈0.26); we tentatively attribute this gap to the larger MLP’s greater sensitivity to distributional shift in the non-stationary replay buffer, though the explanation should be tested with multi-seed runs. The evaluation-time TD losses (Table 3) show the reverse ordering (Q-DQN $0.6192>$ DQN $0.4872$). These two quantities are not directly comparable: training-time loss is computed over replay-buffer samples during an ongoing optimisation with changing target weights, whereas evaluation-time loss is measured on fresh rollouts with a fixed trained policy. The compact VQC approximates the Q-function less precisely under a clean evaluation distribution, consistent with its smaller parameter budget.

5.3. Robustness to Link Failures

Figure 8, Figure 9, Figure 10 and Figure 11 characterise system behaviour under varying link failure, recovery, and gate-noise conditions.

Figure 8 sweeps $p_f$ from 0.005 to 0.080 while holding $p_r=0.10$ fixed. Both Q-DQN and DQN exhibit moderate and roughly symmetric delay increases as $p_f$ grows, with Q-DQN delays ranging from 52.69 to 62.87 ms and DQN from 54.48 to 61.43 ms. Neither method degrades catastrophically up to $p_f=0.08$, indicating that both learned policies are robust to the expected operating range of 6G link failure rates. Dijkstra’s delays remain at ≈7 ms throughout, as it always routes around failures given its perfect link-state knowledge.

Figure 9 sweeps the link recovery probability $p_r$ from 0.05 to 0.30 with fixed $p_f=0.02$. As $p_r$ increases, failed links recover faster, reducing average delay for all methods. Across this sweep, Q-DQN delays vary between 55.12 and 57.67 ms ($\sigma \approx 0.8$ ms) while DQN varies between 49.20 and 62.33 ms ($\sigma \approx 5.5$ ms). The lower standard deviation of Q-DQN across recovery rates indicates a more stable routing policy, which we attribute to the VQC’s smooth parameter landscape, a known property of sinusoidal quantum activation functions.

Figure 10 presents a two-dimensional heatmap of Q-DQN delay over the joint $(p_f,p_r)$ space, confirming that the agent maintains acceptable performance across the entire tested parameter grid. The worst-case cell (high $p_f$, low $p_r$) still yields delays below 70 ms, demonstrating the agent’s suitability for challenging 6G failure scenarios.

In practice, quantum hardware introduces gate noise that perturbs the expectation values in Equation (13). We simulate this by adding Gaussian noise $\mathcal{N}(0,{\sigma }_{\mathrm{noise}}^2)$ to each Pauli-Z expectation value during evaluation, sweeping ${\sigma }_{\mathrm{noise}}\in \{0.00,0.05,\dots ,0.20\}$. Figure 11 shows that Q-DQN delays remain within 4 ms of the noiseless baseline at all noise levels tested, and that the degradation is symmetric with respect to DQN’s own sensitivity to network-state estimation noise. Both methods degrade by $+3.7$ ms (Q-DQN) and $+3.9$ ms (DQN) over the noiseless-to-maximum-noise range. This confirms that Q-DQN achieves equal noise resilience at 49.6× fewer parameters, a notable result given that VQC outputs are expected to be more sensitive to Pauli measurement noise than the outputs of a classical sigmoid or ReLU network.

5.4. Topology Generalisation

To assess generalisation, we evaluate policies trained on the primary topology (seed 42) on three additional topologies generated with different random seeds (123, 456, 789). Figure 12 shows the results. Q-DQN and DQN achieve comparable delays on seeds 42 and 123; on seeds 456 and 789, Q-DQN exhibits higher delays (73.69 vs. 65.17 ms and 98.50 vs. 87.82 ms, respectively). The larger gaps at topologically dissimilar seeds suggest that the compact VQC, while parameter-efficient, may be more sensitive to structural deviations in the graph than the over-parameterised MLP. This is consistent with the bias–variance trade-off: fewer parameters reduce variance within the training distribution but increase bias when applied to out-of- distribution inputs. We consider on-policy fine-tuning on new topologies as a direction for future work.

5.5. Network Scalability

Table 4. Scalability comparison across network sizes ($L=3$, 200 training episodes, 3 independent seeds; mean ± std reported for delay).

Nodes n	Delay (ms)		Parameters		Ratio
	Q-DQN	DQN	Q-DQN	DQN
6	$33.07\pm 8.85$	$\mathbf{14}.\mathbf{31}$ ± 7.02	78	4998	64.1×
8	$78.40\pm 10.17$	$\mathbf{71}.\mathbf{46}$ ± 14.75	92	5128	55.7×
10	$83.44\pm 38.18$	$\mathbf{46}.\mathbf{17}$ ± 5.52	106	5258	49.6×
12	$69.54\pm 16.22$	$\mathbf{52}.\mathbf{93}$ ± 8.89	120	5388	44.9×

Note: Bold denotes the lower (better) delay between Q-DQN and DQN for each network size.

and Figure 13 report delay and parameter counts for both agents across $n\in \{6,8,10,12\}$ nodes. Several observations merit discussion.

Across all tested sizes, DQN achieves lower mean delays under the 200-episode training budget. This indicates that the classical MLP converges more reliably than the compact VQC under limited training. The large standard deviation for Q-DQN at $n=10$ ($\pm 38.18$ ms) reflects high seed-to-seed variance: one of the three seeds failed to converge within 200 episodes (delay ≈135 ms), while the other two achieved 45–70 ms. Similarly, at $n=8$ and $n=12$, Q-DQN shows substantially higher std than DQN, confirming that the VQC requires more training samples to reach stable performance under this budget. Under the full 500-episode budget used in the main comparison (Table 3), Q-DQN reaches $54.29\pm 1.72$ ms, within 1.40 ms of DQN, confirming that the VQC ultimately matches classical performance. These multi-seed results clarify that Q-DQN’s primary advantage is parameter efficiency ($44.9$–$64.1\times$ fewer parameters across all tested sizes), not sample efficiency.

The parameter growth rate of Q-DQN is only 14 parameters per two additional nodes (Equation (15)), compared with 130 for DQN. Even at $n=12$, Q-DQN uses 44.9× fewer parameters (120 vs. 5388), confirming substantially lower memory requirements across all tested scales.

5.6. Ablation Studies

5.6.1. Entanglement Topology

Table 5. Entanglement topology ablation ($L=3$, $n=10$, 200 training episodes, single seed).

Entanglement	Delay (ms)	Parameters
Ring (default)	57.10	106
Linear	56.59	106
Full	84.44	106
VQC (no ent.)	57.11	106

Note: Bold denotes the lowest delay among all entanglement configurations.

compares ring, linear, and full entanglement at fixed $L=3$, $n=10$. All three topologies share the same 106-parameter budget (CNOT gates are parameter-free). Linear achieves the marginally lowest numerical delay (56.59 ms), with ring (57.10 ms) within 0.51 ms, a difference within expected single-seed run-to-run variance. Full entanglement performs considerably worse (84.44 ms), which we attribute to over-entanglement: dense all-to-all CNOT scrambles feature locality encoded per qubit, creating training difficulties reminiscent of the barren plateau problem [26]. Ring entanglement is adopted as the default configuration for its closed-loop topology, which theoretically supports better gradient flow than the open-chain linear variant [15]. Importantly, full entanglement is the only configuration that significantly degrades performance; ring, linear, and no-entanglement variants all achieve comparable delays (56.59–57.11 ms), indicating that the primary design principle is to avoid dense all-to-all entanglement rather than to maximise entanglement degree. Entanglement appears to offer convergence-stability benefits in more complex settings rather than raw delay gains under the current routing task and qubit count.

The no-entanglement variant (independent single-qubit circuits) achieves 57.11 ms, almost identical to ring, suggesting that for this particular routing task and qubit count, entanglement primarily aids convergence stability rather than raw delay performance.

5.6.2. Circuit Depth

Table 6. Circuit depth ablation (ring entanglement, $n=10$, 200 training episodes, single seed).

Depth L	Delay (ms)	TD Loss	Params	Formula ($12\mathit{L}+70$)
1	56.08	0.4725	82	$12+70=82$
2	60.34	0.7707	94	$24+70=94$
3	63.30	0.5829	106	$36+70=106$
4	52.71	0.6665	118	$48+70=118$

Note: Bold denotes the lowest delay among all circuit depths tested.

and Figure 14 examine $L\in \{1,2,3,4\}$ with ring entanglement and $n=10$. The delay does not decrease monotonically with depth: $L=4$ achieves the lowest delay (52.71 ms) but its TD loss (0.6665) is higher than $L=3$ (0.5829) and $L=1$ (0.4725), indicating that the deeper circuit has not yet converged under the 200-episode budget and its Q-values remain poorly calibrated. $L=2$ exhibits the highest TD loss overall (0.7707) despite a worse delay than $L=4$; we attribute this single-seed result to variance under the short training budget. $L=3$ is adopted as a balanced default among deeper configurations ($L\ge 3$): it has lower TD loss than $L=4$ (0.5829 vs. 0.6665), and the full 500-episode evaluation (Table 3, 54.29 ms) confirms it continues to improve with additional training. Note that $L=4$ achieves the lowest 200-episode delay (52.71 ms) and $L=1$ achieves the lowest TD loss (0.4725); a definitive depth ranking under 500-episode training is reserved for future work. $L=1$ is the recommended option for memory-critical deployments (82 parameters) given its strongest TD-loss stability and competitive 200-episode delay (56.08 ms).

6. Discussion

The 106-parameter Q-DQN policy occupies approximately 424 bytes in single-precision floating point, a footprint compatible with the SRAM available on low-power ARM Cortex-M4 microcontrollers commonly used in network edge hardware. Deploying such a policy eliminates the need for a centralised path-computation server, enabling fully distributed routing decisions at sub-millisecond inference latency. While our simulation uses classical VQC emulation, the architecture is directly compatible with emerging quantum co-processors integrated with classical baseband chips, as anticipated in proposals for quantum-native 6G hardware [36,39].

Several aspects of the present study warrant further investigation. First, the current state representation encodes compact aggregated per-node statistics (minimum delay, minimum loss, maximum bandwidth, node-index proximity, link-availability ratio, time progress) rather than per-link attributes for each candidate next-hop. As a result, the agent cannot directly compare individual next-hop options in its input; it relies on learned node-index-dependent Q-values to differentiate candidates, which may limit fine-grained routing precision. Future work will investigate action-conditioned representations that explicitly encode per-link delay, loss, and bandwidth for each candidate next-hop. Second, the topology generalisation gap observed at seeds 456 and 789 (Section 5.4) suggests that either a larger qubit count or a fine-tuning step on new topologies may be necessary for production deployment. Third, at $n=8$ the multi-seed scalability experiment (Table 4) shows Q-DQN achieving $78.40\pm 10.17$ ms versus DQN’s $71.46\pm 14.75$ ms under a 200-episode budget; the high seed-to-seed variance of both agents at this size suggests that a longer training budget or warm-start initialisation may be needed to close the remaining gap. Fourth, our evaluation relies on classical VQC simulation; experiments on actual quantum hardware would introduce real noise profiles that our Gaussian approximation (Section 5.3) may not fully capture. Fifth, the current implementation uses a single-agent formulation; extending to multi-agent or hierarchical settings relevant to large 6G core networks is a natural next step.

Regarding NISQ hardware feasibility, running Q-DQN on near-term quantum hardware involves practical constraints not fully captured by classical simulation alone. Superconducting-qubit processors currently achieve two-qubit CNOT gate fidelities of approximately 99.0–99.5%, with individual gate times of roughly 200–500 ns. For the proposed $L=3$, six-qubit circuit, each inference pass requires $L\times n_q=18$ two-qubit CNOT gates together with single-qubit rotation gates, yielding an estimated circuit execution time of 10–30 μs per Q-value evaluation, well within the coherence times ($T_2\approx 50$–300μ s) of state-of-the-art superconducting devices. Trapped-ion platforms provide higher two-qubit gate fidelities (∼99.9%) but slower gate times (∼1 ms), making them better suited to offline policy updates than real-time packet routing. Among existing NISQ platforms, superconducting processors represent the most feasible near-term target for hardware deployment of Q-DQN [40]. Actual hardware deployment would introduce additional error sources (state-preparation errors, readout noise, and crosstalk) that our Gaussian perturbation model does not fully capture; hardware-in-the-loop experiments are the natural next step.

The random directed graph used in this study, with uniform link delays and stationary Markov failure dynamics, is a deliberate simplification that captures the features of 6G links most relevant here (variable delay and intermittent availability) while keeping the evaluation tractable. Real-world 6G access and backhaul topologies typically exhibit additional structure: geographic clustering, backbone–access hierarchy, and degree distributions that deviate substantially from the Erdos–Rényi model used here. The generalisation experiment (Section 5.4) provides a first indication of how structural deviations affect performance: Q-DQN suffers a larger penalty on topologically dissimilar seeds than the over-parameterised DQN, suggesting that the compact VQC is more sensitive to distributional shift in the graph structure. Adding topology embeddings, for instance via graph neural network feature extraction, to the agent’s state representation could help the agent generalise to real-world deployment settings.

Wu et al. [24] report a $52\times$ parameter reduction for a VQC-based routing agent in a 6G simulation, consistent with our $49.6\times$ finding. Their work uses a five-qubit circuit on a smaller seven-node topology, whereas we evaluate across four network sizes with experiments spanning failure robustness, gate-noise sensitivity, topology generalisation, scalability, and circuit ablation studies. Li et al. [25] propose a hybrid quantum-classical routing protocol and report gains in heterogeneous ultra-dense networks, though they do not evaluate circuit depth or entanglement topology systematically. Our work provides the ablation depth and robustness breadth that these studies leave as future work.

7. Conclusions

This paper presented Q-DQN, a quantum deep Q-network for intelligent packet routing in 6G heterogeneous wireless networks. By replacing the conventional MLP with a six-qubit variational quantum circuit equipped with ring-topology entanglement and angle embedding, the proposed agent achieves an average end-to-end delay of $54.29\pm 1.72$ ms, within 1.40 ms of a classical DQN baseline, while using only 106 trainable parameters, a 49.6× reduction. The parameter count scales as $12L+7n$, growing at only seven parameters per additional network node versus 65 for DQN, and a three-seed scalability evaluation across $n\in \{6,8,10,12\}$ confirms that under limited training DQN converges more consistently, while under full training Q-DQN matches DQN at 49.6× lower parameter cost, establishing that the VQC’s advantage is parameter efficiency rather than sample efficiency. Robustness sweeps confirm that Q-DQN maintains comparable stability to DQN across link failure and recovery rates, and gate-noise experiments show equal resilience to Gaussian measurement perturbations. Ablation studies identify full-connection entanglement as significantly inferior to local alternatives; ring, linear, and no-entanglement configurations achieve comparable delays, with ring adopted as the default for theoretical gradient-flow reasons. Circuit depth $L=3$ is selected as a balanced default among deeper configurations, with $L=1$ recommended for memory-critical deployments.

Future work will address three open questions raised by this study: (i) topology-aware meta-learning to close the generalisation gap; (ii) hardware-in-the-loop experiments on NISQ-era quantum co-processors to characterise real gate-noise effects; and (iii) extension to multi-agent or hierarchical routing settings appropriate for large-scale 6G core and access networks.