Coordinated Multi-Intersection Traffic Signal Control Using a Policy-Regulated Deep Q-Network

Abstract

Coordinated control across multiple signalized intersections is essential for mitigating congestion propagation in urban road networks. However, existing DQN-based approaches often suffer from unstable action switching, limited interpretability, and insufficient capability to model spatial spillback between adjacent intersections. To address these limitations, this study proposes a Policy-Regulated and Aligned Deep Q-Network (PRA-DQN) for cooperative multi-intersection signal control. A differentiable policy function is introduced and explicitly trained to align with the optimal Q-value-derived target distribution, yielding more stable and interpretable policy behavior. In addition, a cooperative reward structure integrating local delay, movement pressure, and upstream–downstream interactions enables agents to simultaneously optimize local efficiency and regional coordination. A parameter-sharing multi-agent framework further enhances scalability and learning consistency across intersections. Simulation experiments conducted on a 2 × 2 SUMO grid show that PRA-DQN consistently outperforms fixed-time, classical DQN, distributed DQN, and pressure/wave-based baselines. Compared with fixed-time control, PRA-DQN reduces maximum queue length by 21.17%, average queue length by 18.75%, and average waiting time by 17.71%. Moreover, relative to classical DQN coordination, PRA-DQN achieves an additional 7.53% reduction in average waiting time. These results confirm the effectiveness and superiority of the proposed method in suppressing congestion propagation and improving network-level traffic performance. The proposed PRA-DQN provides a practical and scalable basis for real-time deployment of coordinated signal control and can be readily extended to larger networks and time-varying demand conditions.

1. Introduction

Urban traffic congestion imposes substantial economic and societal costs worldwide. According to the INRIX 2025 Global Traffic Scorecard, the typical U.S. driver lost 49 h annually to traffic delays, corresponding to approximately $894 per driver and $85.8 billion nationwide in lost productivity [1,2]. Signalized intersections are critical control points for congestion mitigation, and field evidence indicates that adaptive signal control can reduce control delay and the number of stops by up to 20% and 30%, respectively [3]. Recent studies have also reported notable improvements under heterogeneous traffic and peak-demand conditions, including delay reductions exceeding 70% and reductions of around 50% in queue dissipation time using a hybrid Max Pressure and reinforcement learning approach [4], as well as reductions of up to 24% in average travel time and 45% in time loss using Proximal Policy Optimization (PPO)-based deep reinforcement learning [5]. Importantly, traffic demand varies markedly across hour-of-day, day-of-week, and public-holiday versus workday conditions. Prior empirical work shows that public holidays can significantly reshape both traffic volume levels and intraday profiles, highlighting the need to account for such time-varying demand when designing and deploying signal control programs [6].

Traffic-flow theory has described congestion evolution using traffic phase transitions, including congestion boundary approaches that characterize regime changes between undersaturated and oversaturated states [7,8]. In addition, simplified traffic models have interpreted congestion dynamics as a spatiotemporal clustering phenomenon [9]. These theoretical viewpoints imply that congestion propagation is driven by upstream–downstream coupling and spillback, which directly links the operational states of adjacent intersections. These insights motivate coordination mechanisms that explicitly address spillback and inter-intersection interactions in multi-intersection.

Traffic signal control plays a pivotal role in urban traffic management and is widely recognized as a complex sequential decision-making problem under dynamic and uncertain traffic conditions. Traditional rule-based or model-based signal control schemes, such as pre-timed signal plans or actuated signal control, often rely on predefined parameters and simplified assumptions, which limit their adaptability to heterogeneous and time-varying traffic conditions [10]. With the rapid development of sensing, communication, and computing technologies, deep reinforcement learning (DRL) has gained prominence as a promising approach for adaptive traffic signal control, enabling agents to learn control policies directly through interaction with the traffic environment [11].

Early DRL-based studies mainly focused on isolated intersections and showed that deep neural networks enable reinforcement learning to handle high-dimensional traffic states, thereby improving scalability. For example, Shabestary and Abdulhai [12] proposed an adaptive controller that directly processes high-resolution sensory inputs from connected vehicles, thereby avoiding handcrafted feature extraction. Chu et al. [13] developed an end-to-end off-policy DRL framework that takes real-time images of an intersection as input and learns a near-optimal timing policy to minimize average waiting time. Ma et al. [14] further integrated temporal traffic pattern mining with an actor–critic architecture, transforming traffic conditions into sequential image representations to improve the robustness of signal timing. Chen et al. [15] constructed a 3DQN-based model that combines dueling networks, double DQN, and prioritized experience replay to optimize single-intersection timing using floating car data. These studies collectively demonstrate the effectiveness of DRL for adaptive control at individual intersections. As research attention shifted toward network-level performance, multi-agent DRL began to play a central role in multi-intersection signal coordination. Hu and Li [16] proposed a multi-agent Double DQN framework in which local agents are trained independently and then integrated into a global agent to coordinate corridor signals. Zhang et al. [17] designed a fully scalable multi-agent proximal policy optimization (PPO) algorithm with parameter sharing to coordinate arterial corridors under dynamic traffic conditions. Park et al. [18] developed DQN-based models for both isolated and two-intersection coordinated control and showed that learned policies can outperform optimized fixed-time plans. To alleviate scalability and non-stationarity issues in large-scale networks, Jiang et al. [19] introduced a distributed multi-agent reinforcement learning scheme with graph decomposition, while Gu et al. [20] proposed RegionLight, which partitions the network into star-shaped regions and applies an adaptive branching dueling Q-network for regional control. Related works have also explored knowledge-sharing schemes [21], cooperative group-based Multi-Agent Reinforcement Learning (MARL) [22], and topology-embedding propagation strategies [23] to enhance coordination across large networks.

Beyond basic efficiency objectives, DRL-based traffic signal control has been extended to a wide range of application scenarios, including mixed traffic environments, multi-modal flows, and cooperative vehicle–infrastructure systems. Related predictive control studies in coupled transportation–energy systems further highlight the importance of accurate demand forecasting and multi-stage decision-making [24]. Yang [25] proposed a PPO-based controller with a multi-discrete action space and combined state representation to enhance robustness in mixed connected and non-connected traffic. Wang et al. [26] developed a human-centric multimodal deep signal control scheme to coordinate vehicles, buses, and pedestrians, explicitly targeting per-capita waiting time and social equity. Yazdani et al. [27] designed an intelligent vehicle–pedestrian light controller that minimizes total user delay by jointly considering vehicle–vehicle, vehicle–pedestrian, and pedestrian–pedestrian interactions. Multi-modal and transit-priority settings have also been studied through decentralized MARL and DRL-based Transit Signal Priority (TSP) models, demonstrating the potential of RL to balance bus reliability and car performance in complex networks [28,29,30]. In the context of connected and automated vehicles (CAVs), several works have investigated cooperative signal and vehicle control to jointly optimize travel time, emissions, and safety, or to integrate signal control with vehicle navigation tasks [31,32,33,34]. Other studies have considered privacy-preserving signal control [35], software-defined or 6G-enabled smart transportation architectures [36,37,38], and democratic multi-goal control schemes based on voting among DRL-trained controllers [39].

At the methodological level, value-based and policy-based DRL algorithms have both been extensively applied to traffic signal control. Beyond standard DQN, various improvements—such as double DQN, dueling architectures, prioritized replay, and hybrid actor–critic schemes—have been introduced to address overestimation, instability, and sample inefficiency [40]. Policy-gradient and PPO-based methods have been proposed to optimize phase sequences and durations in continuous or hybrid action spaces, enhancing flexibility under diverse demand patterns [41,42,43]. However, several fundamental challenges remain. First, many DQN-based controllers still rely on ε-greedy exploration directly on noisy Q-value estimates, which can lead to unstable phase switching and limited interpretability of the learned policies, particularly in multi-agent settings. Second, although reward functions commonly include delay, queue length, or pressure-based terms [44], they often underrepresent upstream–downstream interactions and spillback effects, which are critical for robust multi-intersection coordination. Third, robustness and generalization are not fully resolved: DRL agents are sensitive to abnormal input data, sensor failures, or partially blinded intersections, and improvements via specialized reward designs or robustness-enhancing mechanisms are still an active research topic [45,46,47].

Recent studies have explored policy–value separation and stabilization techniques to improve learning robustness, as well as graph-based multi-agent reinforcement learning to encode network topology [25,41]. Centralized-critic architectures under the Centralized Training with Decentralized Execution (CTDE) paradigm and hierarchical reinforcement learning frameworks further address coordination by introducing centralized value estimation or multi-level decision structures [48,49]. In contrast, PRA-DQN focuses on stabilizing value-based learning through explicit policy–Q alignment and parameter sharing, without relying on centralized critics or explicit graph encoders, which distinguishes it from these network-level control paradigms.

Against this background, this study aims to enhance the stability, interpretability, and coordination capability of value-based multi-intersection controllers by proposing a Policy-Regulated and Aligned Deep Q-Network (PRA-DQN). The proposed approach builds upon the classical DQN framework and targets the aforementioned gaps from three perspectives.

(1)

A structured multi-intersection state representation is constructed, integrating waiting time, queue length, and movement-level pressure, so that agents can better capture both local congestion and upstream–downstream interactions.

(2)

A hybrid reward function is designed to combine delay reduction, pressure balancing, and adjacent-intersection influence, thereby encouraging each agent to consider not only its own performance but also the impact on neighboring intersections and network-wide coordination.

(3)

A policy-regulation and Q-alignment mechanism is introduced, in which an explicit, differentiable policy function is trained to align with the Q-value–induced target distribution, smoothing action-selection noise and improving interpretability. Together with a parameter-sharing multi-agent structure, this mechanism improves learning stability and ensures consistent behavior across intersections.

The proposed PRA-DQN is evaluated on a SUMO-based 2 × 2 grid network and compared against six representative baselines, including classical DQN, distributed DQN, stochastic action selection, Max Pressure, MaxWave, and fixed-time signal control. The experimental results demonstrate that PRA-DQN can effectively reduce maximum queue length, average queue length, and average waiting time, confirming its potential as a practical solution for coordinated control of multi-intersections.

This study focuses on coordinated signal control at the intersection level under coupled traffic dynamics (e.g., queue spillback and upstream–downstream interactions). To ensure clarity and reproducibility, all state variables used for learning are explicitly defined with their measurement units and aggregation rules. The proposed neural-network-based controller learns a mapping from the measured traffic states to discrete signal phase actions, enabling adaptive phase selection under time-varying congestion in the studied multi-intersection setting.

The remainder of this paper is organized as follows. Section 2 describes the problem setting and the proposed multi-intersection state/action formulation. Section 3 presents the PRA-DQN methodology, including the reward design and policy–Q alignment mechanism. Section 4 reports the simulation environment, experimental setup, and comparative results. Section 5 concludes the paper and outlines directions for future work.

2. Multi-Intersection Signal Coordination Model

2.1. State Space

The state space represents the traffic conditions perceived by each agent at every decision step. In the proposed PRA-DQN framework, the state of intersection i is constructed from four key traffic indicators that jointly capture congestion levels, delay accumulation, vehicle demand, and local movement dynamics.

Let ${\mathrm{L}}_{\mathrm{i}}$ denote the set of incoming lanes at intersection $\mathrm{i}$. For each lane $\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}$, let ${\mathrm{q}}_{\mathrm{l}}$ be the queue length, ${\mathrm{w}}_{\mathrm{l}}$ the total waiting time, ${\mathrm{n}}_{\mathrm{l}}$ the number of approaching vehicles, and ${\mathrm{v}}_{\mathrm{l}}$ the total speed of vehicles on that lane.

The intersection-level aggregated features are defined as:

$${\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{q}}_{\mathrm{l}}}\left(\mathrm{t}\right)$$

(1)

$${\mathrm{w}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{w}}_{\mathrm{l}}}\left(\mathrm{t}\right)$$

(2)

$${\mathrm{n}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{n}}_{\mathrm{l}}}\left(\mathrm{t}\right)$$

(3)

$${\mathrm{v}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{v}}_{\mathrm{l}}}\left(\mathrm{t}\right)$$

(4)

Accordingly, the complete state vector for intersection $\mathrm{i}$ is expressed as:

$${\mathrm{s}}_{\mathrm{i}}=\left\{{\mathrm{q}}_{\mathrm{i}},{\mathrm{w}}_{\mathrm{i}},{\mathrm{n}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}}\right\}$$

(5)

This four-dimensional representation provides the necessary information for the PRA-DQN agent to evaluate current intersection conditions and predict short-term traffic evolution. It also forms the basis for constructing structured feature tensors used in the convolutional neural network described in Section 3.2.

2.2. Action Space

The action space specifies the set of signal control operations that can be executed at each intersection. In general, signal controllers may operate under either fixed or variable phase sequences. To ensure safety, avoid phase conflicts, and simplify coordinated decision-making across intersections, this study adopts a fixed four-phase sequence for all intersections in the network. Each cycle consists of four movements: east–west through (EW-T), east–west left-turn (EW-L), north–south through (NS-T), and north–south left-turn (NS-L), as illustrated in Figure 1. The fixed four-phase structure is adopted to ensure safety and comparability in this controlled study. For intersections with heterogeneous geometries or phase requirements, the framework can be extended by defining intersection-specific phase sets and applying action masking to restrict infeasible actions, without altering the core PRA-DQN learning mechanism.

For a control region containing N signalized intersections, the action taken at each decision step corresponds to selecting one of the four predefined phases. Let the action set of intersection i be denoted by

$${\mathrm{A}}_{\mathrm{i}}=\{{\mathrm{a}}_{\mathrm{i},1},{\mathrm{a}}_{\mathrm{i},2},{\mathrm{a}}_{\mathrm{i},3},{\mathrm{a}}_{\mathrm{i},4}\}$$

(6)

The joint action space of the multi-intersection system is then expressed as the Cartesian product:

$$\mathrm{A}={\mathrm{A}}_1\times {\mathrm{A}}_2\times \mathrm{L}\times {\mathrm{A}}_{\mathrm{N}}$$

(7)

with a total size of $\left|\mathrm{A}\right|=4^{\mathrm{N}}$. Each joint action corresponds to a coordinated release pattern across all intersections in the network.

At each time step t, the agent determines whether to maintain the current phase or switch to the next phase in the fixed sequence. A green interval of 10 s is allocated to each activated phase, followed by a 3 s yellow interval to ensure safe clearance. Figure 2 illustrates how different actions correspond to changes in the displayed signal at each intersection.

By constructing the global joint action space in this manner, the control framework supports coordinated phase decisions across the entire multi-intersection network. This design facilitates information sharing and cooperative optimization among agents, ultimately contributing to improved traffic flow efficiency and reduced congestion propagation.

2.3. Reward Function

The reward function provides feedback for learning based on the current traffic state and the selected signal action [50]. In this study, the reward is formulated using lane-level measures of waiting time and queue-related congestion, and all reward terms are computed using the same lane-level measurements and detection segment definitions as the state variables to ensure consistency between the input representation and the optimization objective.

All model parameters are defined in

Table 1. Description of Model Parameters.

Symbol	Definition	Unit	Aggregation/Measurement Rule
$\mathrm{i},\mathrm{j}$	Index of intersections, $\mathrm{i},\mathrm{j}\in 1,\dots ,\mathrm{N}$	—	—
$\mathrm{N}$	Number of intersections in the network	—	—
${\mathrm{L}}_{\mathrm{i}}$	Set of incoming lanes at intersection $\mathrm{i}$	—	—
$\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}$	Index of an incoming lane of intersection i	—	—
$\Delta \mathrm{t}$	Decision interval for signal control	s	Fixed to 10 s per decision step (followed by a 3 s yellow clearance)
${\mathrm{q}}_{\mathrm{l}}\left(\mathrm{t}\right)$	Queue length on lane l at time t	veh	Instantaneous count per decision step; measured on lane l (same detection segment as other lane-level variables)
${\mathrm{w}}_{\mathrm{l}}\left(\mathrm{t}\right)$	Total waiting time on lane l at time t	s	Accumulated waiting time of vehicles on lane l at each decision step
${\mathrm{n}}_{\mathrm{l}}\left(\mathrm{t}\right)$	Number of approaching vehicles on lane l at time t	veh	Instantaneous vehicle count per decision step; not time-aggregated flow
${\mathrm{v}}_{\mathrm{l}}^{\mathrm{sum}}\left(\mathrm{t}\right)$	Total (summed) speed of vehicles on lane l at time t	m/s	Sum of instantaneous vehicle speeds on lane l at each decision step
${\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}\right)$	Total queue length at intersection $\mathrm{i}$	veh	${\mathrm{q}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{q}}_{\mathrm{l}}}\left(\mathrm{t}\right)$
${\mathrm{w}}_{\mathrm{i}}\left(\mathrm{t}\right)$	Total waiting time at intersection $\mathrm{i}$	s	${\mathrm{w}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{w}}_{\mathrm{l}}}\left(\mathrm{t}\right)$
${\mathrm{n}}_{\mathrm{i}}\left(\mathrm{t}\right)$	Total number of approaching vehicles at intersection $i$	veh	${\mathrm{n}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{n}}_{\mathrm{l}}}\left(\mathrm{t}\right)$
${\mathrm{v}}_{\mathrm{i}}^{\mathrm{sum}}\left(\mathrm{t}\right)$	Total (summed) speed at intersection $\mathrm{i}$	m/s	${\mathrm{v}}_{\mathrm{i}}^{\mathrm{sum}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{v}}_{\mathrm{l}}^{\mathrm{sum}}}\left(\mathrm{t}\right)$
${\mathrm{C}}_{\mathrm{l}}$	Maximum capacity of lane $\mathrm{l}$	veh	Used for normalization in ${\rho }_l$; represents the maximum number of vehicles that can be accommodated on lane $l$ (under the adopted lane capacity definition)
${\mathsf{\rho }}_{\mathrm{l}}\left(\mathrm{t}\right)$	(Normalized) density of lane $l$ at time $\mathrm{t}$	—	${\mathsf{\rho }}_{\mathrm{l}}\left(\mathrm{t}\right)={\displaystyle \frac{{\mathrm{n}}_{\mathrm{l}}\left(\mathrm{t}\right)}{{\mathrm{C}}_{\mathrm{l}}}}$ (dimensionless)
$\mathrm{m}\in {\mathrm{M}}_{\mathrm{i}}$	Traffic movement at intersection $\mathrm{i}$	—	A movement corresponds to a specific incoming–outgoing lane pair governed by a phase
${\mathrm{M}}_{\mathrm{i}}$	Set of all possible movements at intersection $\mathrm{i}$	—	—
$\mathrm{In}\left(\mathrm{m}\right),\mathrm{Out}\left(\mathrm{m}\right)$	Incoming and outgoing lanes of movement $\mathrm{m}$	—	—
${\mathrm{P}}_{\mathrm{m}}\left(\mathrm{t}\right)$	Pressure of movement m at time t	—	Defined using incoming/outgoing lane densities (see Section 2.3); computed per decision step
${\mathrm{P}}_{\mathrm{i}}\left(\mathrm{t}\right)$	Intersection pressure at intersection $\mathrm{i}$		${\mathrm{P}}_{\mathrm{i}}\left(\mathrm{t}\right)={\displaystyle \sum _{\mathrm{m}\in {\mathrm{M}}_{\mathrm{i}}}}{\displaystyle \mid {\mathrm{P}}_{\mathrm{m}}\left(\mathrm{t}\right)\mid }$
${\mathrm{a}}_{\mathrm{i}}\left(\mathrm{t}\right)\in \mathrm{A}$	Action (phase selection) at intersection $\mathrm{i}$	—	One action selected per decision step
$\mathrm{A}$	Action set (four-phase scheme)	—	$\mathrm{A}=\mathrm{EW}-\mathrm{T},\mathrm{EW}-\mathrm{L},\mathrm{NS}-\mathrm{T},\mathrm{NS}-\mathrm{L}$
${\mathrm{s}}_{\mathrm{i}}\left(\mathrm{t}\right)$	State vector of intersection $\mathrm{i}$	—	A four-dimensional state constructed from $\{{\mathrm{q}}_{{\mathrm{i}}^{\prime }},{\mathrm{w}}_{{\mathrm{i}}^{\prime }},{\mathrm{n}}_{{\mathrm{i}}^{\prime }},{\mathrm{v}}_{\mathrm{i}}^{\mathrm{sum}}\}$ (Section 2.1)
${\mathrm{r}}_{\mathrm{i}}\left(\mathrm{t}\right)$	Reward of intersection $\mathrm{i}$	—	Computed from waiting-time-related and pressure-related components (Section 2.3)
$\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)$	Q-value function	—	—
$\mathrm{G}(\mathrm{s},\mathrm{a};\mathsf{\theta })$	Adjustable policy function (PRA-DQN)	—	—
$\mathsf{\theta },{\mathsf{\theta }}^{-}$	Trainable parameters and target network parameters	—	—
$\mathsf{\gamma }$	Discount factor	—	—
$\mathsf{\alpha }$	Learning rate	—	—

.

All traffic state variables are sampled once per decision step (Δt = 10 s) with a 3 s yellow clearance. The number of approaching vehicles is defined as an instantaneous count within a predefined upstream detection segment on each lane, rather than a time-aggregated flow. All variables follow standard traffic engineering units and aggregation rules consistent with fundamental-diagram-based and heterogeneous-data traffic analysis [51].

(1)

Waiting Time

Waiting time is a critical indicator in signal control systems, impacting traffic flow, vehicle delays, and overall efficiency. The total waiting time across all lanes at each intersection is first computed, and its negative value is then used as the reward to minimize the time vehicles remain stopped at red lights. This encourages smoother traffic flow and higher operational efficiency. Let the set of incoming lanes at intersection $\mathrm{i}$ be denoted by ${\mathrm{L}}_{\mathrm{i}}$, and let ${\mathrm{w}}_{\mathrm{l}}$ denote the total waiting time on lane $\mathrm{l}$. The total waiting time at intersection i is given by

$${\mathrm{W}}_{\mathrm{i}}={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}}{\mathrm{w}}_{\mathrm{l}}}$$

(8)

The corresponding waiting-time-related reward is defined as the negative of ${\mathrm{W}}_{\mathrm{i}}$

$${\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{w}\right)}=-{\mathrm{W}}_{\mathrm{i}}$$

(9)

By minimizing the total waiting time $\mathrm{Wi}$ through maximizing ${\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{w}\right)}$, the time vehicles spend stopped at red lights is reduced, thereby enhancing the dynamic flow of traffic and improving the overall efficiency of the intersection.

(2)

Queue Length

Queue length represents the number of vehicles waiting on a signalized approach and is one of the most important indicators of intersection performance. A longer queue typically reflects insufficient traffic discharge during the preceding phases and indicates a higher likelihood of spillback and congestion propagation. To quantify capture this effect, this study adopts the pressure formulation commonly used in multi-intersection signal control.

The density on lane $\mathrm{l}$ is defined as

$${\mathsf{\rho }}_{\mathrm{l}}={\mathrm{n}}_{\mathrm{l}}/{\mathrm{C}}_{\mathrm{l}}$$

(10)

where ${\mathrm{n}}_{\mathrm{l}}$ denotes the number of vehicles on lane $\mathrm{l}$ and ${\mathrm{C}}_{\mathrm{l}}$ is the maximum capacity of lane $\mathrm{l}$. For a traffic movement m at intersection $\mathrm{i}$, let $\mathrm{In}\left(\mathrm{m}\right)$ and $\mathrm{Out}\left(\mathrm{m}\right)$ denote its incoming and outgoing lanes, respectively. The movement pressure ${\mathrm{P}}_{\mathrm{m}}$ is then defined as

$${\mathrm{P}}_{\mathrm{m}}={\mathsf{\rho }}_{\mathrm{In}\left(\mathrm{m}\right)}-{\mathsf{\rho }}_{\mathrm{Out}\left(\mathrm{m}\right)}$$

(11)

The intersection pressure at intersection $i$ is defined as the sum of the absolute values of all movement pressures (Figure 3):

$${\mathrm{P}}_{\mathrm{i}}={\displaystyle \sum _{\mathrm{m}\in {\mathrm{M}}_{\mathrm{i}}}\left|{\mathrm{P}}_{\mathrm{m}}\right|}$$

(12)

Here, ${\mathrm{M}}_{\mathrm{i}}$ denotes the set of all movements at intersection $\mathrm{i}$. A larger value of ${\mathrm{P}}_{\mathrm{i}}$ indicates a more imbalanced distribution of vehicles between incoming and outgoing lanes. Therefore, the pressure-related reward at intersection $\mathrm{i}$ is defined as

$${\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{p}\right)}=-{\mathrm{P}}_{\mathrm{i}}$$

(13)

By minimizing ${\mathrm{P}}_{\mathrm{i}}$ through maximizing ${\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{p}\right)}$, the controller promotes a more balanced traffic distribution across the intersection, helping to reduce congestion and improve overall traffic efficiency.

(3)

Influence of Adjacent Intersections

In addition to local queue length and waiting time, the collaborative influence of adjacent intersections is also incorporated into the reward design. This term penalizes large queues and long waiting times at neighboring intersections so that each agent takes into account upstream–downstream spillback effects and contributes to network-wide coordination.

Let $\mathrm{N}\left(\mathrm{i}\right)$ denote the set of intersections adjacent to intersection $\mathrm{i}$. For each neighboring intersection $\mathrm{j}\in \mathrm{N}\left(\mathrm{i}\right)$, let ${\mathrm{q}}_{\mathrm{j}}$ and ${\mathrm{w}}_{\mathrm{j}}$ denote the total queue length and total waiting time at intersection j, respectively. They can be computed as

$${\mathrm{q}}_{\mathrm{j}}={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{j}}}{\mathrm{q}}_{\mathrm{l}}}$$

(14)

$${\mathrm{w}}_{\mathrm{j}}={\displaystyle \sum _{\mathrm{l}\in {\mathrm{L}}_{\mathrm{j}}}{\mathrm{w}}_{\mathrm{l}}}$$

(15)

The neighborhood influence term ${\mathrm{C}}_{\mathrm{i}}$ is then defined as

$${\mathrm{C}}_{\mathrm{i}}={\displaystyle \sum _{\mathrm{j}\in \mathrm{N}\left(\mathrm{i}\right)}({\mathrm{q}}_{\mathrm{j}}+{\mathrm{w}}_{\mathrm{j}})}$$

(16)

and the corresponding neighborhood-related reward is given by

$${\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{n}\right)}={\mathrm{C}}_{\mathrm{i}}$$

(17)

By minimizing ${\mathrm{C}}_{\mathrm{i}}$ through maximizing ${\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{n}\right)}$, the controller encourages each intersection to reduce the risk of congestion spillback and to support smoother traffic propagation in its surrounding area.

(4)

Overall Reward and Objective

Combining the above three components, the total reward for intersection $\mathrm{i}$ at each decision step is defined as

$${\mathrm{r}}_{\mathrm{i}}={\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{w}\right)}+{\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{l}\right)}+{\mathrm{r}}_{\mathrm{i}}^{\left(\mathrm{n}\right)}$$

(18)

The overall objective of the multi-intersection signal control problem is to maximize the cumulative reward over all intersections in the control area:

$$\max {\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{r}}_{\mathrm{i}}}$$

(19)

3. Signal Coordination Control Algorithm Based on PRA-DQN

3.1. Policy-Regulated and Aligned DQN (PRA-DQN) Algorithm

In reinforcement learning, the policy function determines how an agent selects actions based on the observed state. Traditional DQN implicitly embeds this policy within the ϵ-greedy mechanism: with probability ϵ, a random action is selected for exploration, and with probability 1 − ϵ, the agent exploits the Q-function by choosing the action with the maximal Q-value. While simple, this implicit policy structure often leads to unstable and non-smooth behavior in multi-intersection traffic signal control, where consistency and interpretability are critical.

To overcome these limitations, this study introduces a Policy-Regulated and Aligned DQN (PRA-DQN) framework, which makes the policy explicit and optimizable. PRA-DQN incorporates an adjustable policy function $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$, where s represents the current state (e.g., queue length, waiting time, approaching traffic), a is a candidate signal phase, and θ is the set of trainable policy parameters. The output of G is transformed (e.g., by a Hardmax/Softmax operation) into a probability distribution over actions, enhancing interpretability and reducing noise in action selection.

(1)

Explicit Policy Function

The explicit representation of the policy allows direct optimization instead of relying solely on ϵ-greedy exploitation. During interaction with the environment, PRA-DQN still applies ϵ-greedy exploration—selecting a random action with probability ϵ—while the exploitation step uses the policy distribution generated by $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$. This design preserves exploration capability while enabling a more stable and interpretable exploitation phase.

(2)

Policy–Q Alignment Mechanism

A distinguishing feature of PRA-DQN is the policy–Q alignment mechanism, which ensures that the policy function selects the same optimal action as the Q-function. The policy–Q alignment mechanism can be interpreted as training an explicit policy to match a Q-induced target distribution over actions. Compared with direct ε-greedy action selection on noisy Q estimates, the explicit policy provides a smoother state–action mapping, which reduces high-frequency switching among signal phases and improves learning stability. For each observed state, the Q-network computes the optimal action:

$${\mathrm{a}}^{*}=\arg \underset{{\mathrm{a}}^{\prime }}{\max }\mathrm{Q}(\mathrm{s},{\mathrm{a}}^{\prime };{\mathsf{\theta }}_{\mathrm{Q}})$$

(20)

The alignment target is then defined as:

$$\mathrm{y}(\mathrm{s},\mathrm{a})=\left\{\begin{array}{cc}1,& \mathrm{ifa}={\mathrm{a}}^{*},\\ 0,& \mathrm{otherwise}.\end{array}\right.$$

(21)

This target reflects whether each action matches the Q-value–derived optimal choice. The policy parameters θ are optimized by minimizing a log-loss function:

$${\mathrm{L}}_{\mathrm{G}}=-{\displaystyle \sum _{\mathrm{a}}\mathrm{y}(\mathrm{s},\mathrm{a})\mathrm{logG}(\mathrm{s},\mathrm{a};\mathsf{\theta })}$$

(22)

and updated by stochastic gradient descent (SGD). Through repeated updates, the output $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$ progressively aligns with the optimal action dictated by the Q-function. This mechanism enhances policy interpretability and suppresses the action-selection noise typically seen in classical DQN.

Mini-batch training is applied to stabilize updates: samples are drawn from the shared replay buffer, and gradients are computed for each batch before updating θ.

(3)

Q-Network Optimization

The Q-network retains the standard DQN structure. The target value for each transition $(\mathrm{s},\mathrm{a},\mathrm{r},{\mathrm{s}}^{\prime })$ is computed as:

$${\mathrm{y}}_{\mathrm{Q}}=\mathrm{r}+\mathsf{\gamma }\underset{{\mathrm{a}}^{\prime }}{\max }\mathrm{Q}({\mathrm{s}}^{\prime },{\mathrm{a}}^{\prime };{\mathsf{\theta }}_{\mathrm{Q}}^{-})$$

(23)

where ${\mathsf{\theta }}_{\mathrm{Q}}^{-}$ denotes the target network parameters. The Q-network parameters ${\mathsf{\theta }}_{\mathrm{Q}}$ are trained by minimizing the Huber loss using the Adam optimizer. This process ensures stable value estimation, while the target network mitigates the non-stationarity of the learning process.

(4)

Combined Learning Process

During training, PRA-DQN alternates between two coupled learning steps:

• Value Learning: Updating the Q-network parameters ${\mathsf{\theta }}_{\mathrm{Q}}$ based on sampled transitions.
• Policy adaptation: updating the policy parameters θ so that $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$ aligns with the Q-value–based optimal action.

This two-step process ensures that the policy remains consistent with value estimation, while reducing the instability and noise that may arise from purely ε-greedy action selection.

(5)

Practical Training Procedure

The complete algorithmic implementation is shown in Algorithm 1:

Algorithm 1. Policy-Regulated and Aligned DQN (PRA-DQN)

Initialization:           • Initialize replay buffer D with capacity N.           • Initialize Q-network $\mathrm{Q} (\mathrm{s},\mathrm{a};{\mathsf{\theta }}_{\mathrm{Q}})$ with random weights.           • Initialize target Q-network Q−with ${\mathsf{\theta }}_{\mathrm{Q}}^{-}={\mathsf{\theta }}_{\mathrm{Q}}$           • Initialize the adjustable policy function $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$ with parameters θ = [1,…,1].      Main Loop (for each episode):           • For each episode t = 1 to T:   1. Action Selection using ε-greedy:      ■ With probability ϵ, select a random action at;      ■ Otherwise, select action according to the policy function G(st, a; θ).   2. Execute action $a_t$, observe reward $r_t$ and next state $s_{t+1}$.   3. Store transition $\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}},{\mathrm{r}}_{\mathrm{t}},{\mathrm{s}}_{\mathrm{t}+1}\right)$ in the experience replay buffer D.      Q-function Update:           • Randomly sample a minibatch from buffer D.           • For each transition sample $\left({\mathrm{s}}_{\mathrm{j}},{\mathrm{a}}_{\mathrm{j}},{\mathrm{r}}_{\mathrm{j}},{\mathrm{s}}_{\mathrm{j}+1}\right)$:   1. Compute Temporal Difference (TD) target: ${\mathrm{y}}_{\mathrm{j}}={\mathrm{r}}_{\mathrm{j}}+\mathsf{\gamma }\underset{{\mathrm{a}}^{\prime }}{\max } \mathrm{Q}\left({\mathrm{s}}_{\mathrm{j}+1},{\mathrm{a}}^{\prime };\mathsf{\theta }\right)$   2. Update Q-network: Minimize the Huber loss between $y_j$ and $\mathrm{Q}({\mathrm{s}}_{\mathrm{j}},{\mathrm{a}}_{\mathrm{j}};{\mathsf{\theta }}_{\mathrm{Q}})$.   3. Every C steps, update the target Q-function ${\mathsf{\theta }}_{\mathrm{Q}}^{-}={\mathsf{\theta }}_{\mathrm{Q}}$      Training the Policy Function G:           • For each sample in the minibatch:   1. Compute alignment target:      ■ $\mathrm{y}(\mathrm{s},\mathrm{a})=\left\{\begin{array}{cc}1,& \mathrm{ifa}={\mathrm{a}}^{*},\\ 0,& \mathrm{otherwise}.\end{array}\right.$.   2. Compute log-loss:      ■ ${\mathrm{L}}_{\mathrm{G}}=-{\displaystyle \sum _{\mathrm{a}}\mathrm{y} ({\mathrm{s}}_{\mathrm{j}},\mathrm{a}\left)\mathrm{logG}\right({\mathrm{s}}_{\mathrm{j}},\mathrm{a};\mathsf{\theta })}$.   3. Update policy parameters:      ■ Update θ using stochastic gradient descent.

3.2. Neural Network Architecture Design

The PRA-DQN framework employs a deep neural network to approximate the state–action value function $\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)$. To exploit spatial relationships among lanes, the lane-level features $\left({\mathrm{q}}_{\mathrm{l}},{\mathrm{w}}_{\mathrm{l}},{\mathrm{n}}_{\mathrm{l}},{\mathrm{v}}_{\mathrm{l}}\right)$ for all incoming lanes $\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}$ are arranged into a two-dimensional tensor ${\mathrm{X}}_{\mathrm{i}}\in {\mathbb{R}}^{\mathrm{M}\times 4}$, where M is the number of incoming lanes.

(1)

Input Feature Organization

For each intersection, the lane-level feature $\left({\mathrm{q}}_{\mathrm{l}},{\mathrm{w}}_{\mathrm{l}},{\mathrm{n}}_{\mathrm{l}},{\mathrm{v}}_{\mathrm{l}}\right)$ of all incoming lanes $\mathrm{l}\in {\mathrm{L}}_{\mathrm{i}}$ is organised into a two-dimensional tensor of size: ${\mathrm{X}}_i\in {\mathbb{R}}^{\mathrm{M}\times 4}$, where M is the number of incoming lanes.

Each row corresponds to one lane, and each column corresponds to one feature dimension.

(2)

Convolutional Feature Extraction

Two convolutional layers with 3 × 3 kernels and ReLU activation are applied to the feature map. A max-pooling operation follows each convolutional layer to reduce dimensionality and enhance robustness to local noise.

Let ${\mathrm{F}}^{\left(2\right)}$ denote the output of the second convolutional layer.

(3)

Fully Connected Layers and Output

The final convolutional feature map is flattened and passed through two fully connected layers with ReLU activation:

$${\mathrm{h}}_1=\mathsf{\sigma }({\mathrm{W}}_1\mathrm{z}+{\mathrm{b}}_1)$$

(24)

$${\mathrm{h}}_2=\mathsf{\sigma }({\mathrm{W}}_2{\mathrm{h}}_1+{\mathrm{b}}_2)$$

(25)

where $\mathrm{z}=\mathrm{Flatten}\left({\mathrm{F}}^{\left(2\right)}\right)$.

${\mathrm{W}}_1,{\mathrm{W}}_2$: weight matrices of the first and second fully connected layers;

${\mathrm{b}}_1,{\mathrm{b}}_2$: bias vectors;

$\mathsf{\sigma }(•)$: ReLU activation function;

${\mathrm{h}}_1,{\mathrm{h}}_2$: hidden-layer activation vectors.

The output layer computes the Q-value of each action:

$$\mathrm{Q}(\mathrm{s},\mathrm{a};{\mathsf{\theta }}_{\mathrm{Q}})={\mathrm{W}}_{\mathrm{o}}{\mathrm{h}}_2+{\mathrm{b}}_{\mathrm{o}}$$

(26)

where

${\mathrm{W}}_{\mathrm{o}}$ and ${\mathrm{b}}_{\mathrm{o}}$: weights and bias of the output layer;

${\mathsf{\theta }}_{\mathrm{Q}}=\{{\mathrm{W}}_1,{\mathrm{W}}_2,{\mathrm{W}}_{\mathrm{o}},{\mathrm{b}}_1,{\mathrm{b}}_2,{\mathrm{b}}_{\mathrm{o}}\}$: the set of all learnable parameters of the Q-network.

The output layer contains $\mid \mathrm{A}\mid =4$ neurons, corresponding to the four signal phases available at each intersection.

(4)

Loss Function and Optimization

The Q-network parameters ${\mathsf{\theta }}_{\mathrm{Q}}$ are optimized using the Huber loss:

$${\mathrm{L}}_{\mathrm{Q}}=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{({\mathrm{y}}_{\mathrm{Q}}-\mathrm{Q}(\mathrm{s},\mathrm{a};{\mathsf{\theta }}_{\mathrm{Q}}\left)\right)}^2,& \mathrm{if}\left|{\mathrm{y}}_{\mathrm{Q}}-\mathrm{Q}<1\right|,\\ \left|{\mathrm{y}}_{\mathrm{Q}}-\mathrm{Q}(\mathrm{s},\mathrm{a};{\mathsf{\theta }}_{\mathrm{Q}})\right|-{\displaystyle \frac{1}{2}},& \mathrm{otherwise}.\end{array}\right.$$

(27)

The temporal-difference target is computed following the standard DQN formulation:

$${\mathrm{y}}_{\mathrm{Q}}=\mathrm{r}+\mathsf{\gamma }\underset{{\mathrm{a}}^{\prime }}{\max } \mathrm{Q}\left({\mathrm{s}}^{\prime },{\mathrm{a}}^{\prime };{\mathsf{\theta }}_{\mathrm{Q}}^{-}\right)$$

(28)

where ${\theta }_Q^{-}$ denotes the parameters of the target network.

Adam is used for optimization, and the target network is updated every C steps to stabilize training.

(5)

Experience Replay

Minibatches are uniformly sampled from a shared replay buffer to reduce temporal correlation in samples.

All agents use the same convolutional neural network fully connected (CNN–FC) architecture and shared parameters, enabling consistent and efficient representation learning in the multi-intersection setting.

The network parameters are set as shown in

Table 2. Network Parameter Settings.

Parameter	Description	Value
Learning rate (Q-network)	Step size for updating Q-network weights (Adam)	0.001
Learning rate (policy function G)	SGD learning rate for policy alignment	0.01
Discount factor γ	Weight of future rewards	0.99
Replay buffer size	Capacity of stored transitions	10,000
Batch size	Number of samples per update	32
Target network update frequency	Soft update interval	500 steps
Convolutional layers	CNN feature extraction layers	2
Kernel size	Size of convolution kernels	3 × 3
Pooling	Type of pooling	MaxPooling
Activation function	Nonlinear activation	rectified linear unit (ReLU)
Fully connected layers	Number of FC layers	2

.

Although the state consists of four feature channels, the lane-level features form a structured representation across incoming lanes. Convolutional layers enable shared kernels to capture local correlations among adjacent lanes while maintaining parameter efficiency. This inductive bias is beneficial for generalization across intersections with similar lane configurations.

3.3. Parameter Sharing Mechanism

To enhance learning efficiency and promote consistent behavior across multiple intersections, a parameter-sharing mechanism is adopted in the PRA-DQN framework. In this design, all agents share a unified Q-network and policy alignment module, while interacting with the traffic environment independently. For each intersection, the local state vector ${\mathrm{s}}_{\mathrm{i}}=\left\{{\mathrm{q}}_{\mathrm{i}},{\mathrm{w}}_{\mathrm{i}},{\mathrm{n}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}}\right\}$ is fed into the shared network, which outputs the corresponding Q-values and recommends the signal phase action.

During each interaction cycle, intersection $i$ executes the action suggested by the shared network, observes the resulting next state and reward, and stores the transition $({\mathrm{s}}_{\mathrm{i}},{\mathrm{a}}_{\mathrm{i}},{\mathrm{r}}_{\mathrm{i}},{\mathrm{s}}_{\mathrm{i}}^{\prime })$ into a common replay buffer. This shared replay memory allows agents to benefit from one another’s experience, providing richer and more diverse samples that enhance training stability and accelerate value propagation throughout the network.

Parameter sharing offers two important advantages. First, it significantly reduces the number of trainable parameters compared to maintaining separate networks for each agent, thereby improving computational efficiency and scalability. Second, it encourages partially synchronized updates across intersections, which is beneficial for coordinated signal control in a grid network where local traffic states influence one another. The overall architecture of the parameter-sharing framework is illustrated in Figure 4.

To ensure stable convergence, key hyperparameters are chosen based on widely adopted practices in DQN-based traffic signal control. Specifically, the discount factor is uniformly set to $\gamma =0.99$ for both the Q-network and training environment. The Q-network parameters are updated using the Adam optimizer with a learning rate of 0.001, while the policy alignment module $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$ is optimized using stochastic gradient descent with a learning rate of 0.01. These parameter settings help maintain numerical stability and prevent divergence during training.

3.4. Signal Control Based on Parameter-Sharing PRA-DQN Algorithm

Figure 5 presents the complete workflow of the proposed parameter-sharing PRA-DQN signal control algorithm. At each decision step, the agent at intersection $i$ observes the current state vector ${\mathrm{s}}_{\mathrm{i}}=\left\{{\mathrm{q}}_{\mathrm{i}},{\mathrm{w}}_{\mathrm{i}},{\mathrm{n}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}}\right\}$. The shared PRA-DQN network evaluates the state and outputs estimated Q-values for all possible signal phases. The adjustable policy function $\mathrm{G}\left(\mathrm{s},\mathrm{a};\mathsf{\theta }\right)$ then selects the action based on the policy-aligned probability distribution, ensuring consistency between the learned policy and the Q-value–derived optimal action.

Once an action is selected, the corresponding traffic phase is executed for a fixed time interval, after which the environment returns the next state and reward. This experience is appended to the shared replay buffer for batch sampling and network updates. The Q-network is trained using the Huber loss and Adam optimizer, while the policy alignment module is trained using a log-loss objective and SGD. By alternating between Q-value updates and policy alignment updates, PRA-DQN simultaneously improves value estimation accuracy and action selection stability.

Through repeated interaction with the traffic environment, the PRA-DQN framework gradually converges to an effective coordinated control strategy. The shared structure enables agents to implicitly learn spatial dependencies between intersections, considering factors such as traffic spillback, movement pressure, and local delay. As a result, the algorithm not only optimizes each intersection’s phase transitions but also fosters network-level cooperation, reducing queue accumulation and preventing congestion propagation across the grid.

The parameter-sharing PRA-DQN algorithm provides four main advantages:

Local decision-making: Each agent reacts promptly to its own traffic conditions using a unified model.

Pressure-aware coordination: By incorporating intersection pressure into the reward, the agents balance inflows and outflows to maintain smooth traffic progression.

Efficient learning: Shared parameters reduce computational cost and improve model generalisation across different intersections.

Collaborative adaptation: Shared experience enables agents to learn cooperative behaviors that mitigate spillback and enhance overall traffic flow in the multi-intersection network.

4. Simulation Experiments and Results Analysis

4.1. Simulation Environment

All experiments are conducted using the open-source traffic simulation platform SUMO. The simulation environment is implemented on a Windows 10 (64-bit) system equipped with an Intel Core i5-7200U Central Processing Unit (CPU) (2.50 GHz) and 8 GB of Random-Access Memory (RAM). SUMO version 1.20.0 is used, together with the Traffic Control Interface (TraCI) interface, to retrieve traffic states and apply signal control actions in real time. The PRA-DQN algorithm is implemented in Python 3.7 with TensorFlow 1.19.0. The reinforcement learning agent communicates with SUMO at each decision step through TraCI to perform signal-phase adjustments and collect transition samples. Signal control is executed at a fixed decision interval of Δt = 10 s with a 3 s yellow clearance, and all learning-related traffic measurements are collected at each decision step to align the simulation interface with the defined state and reward formulations.

Unless otherwise specified, vehicles follow SUMO’s default microscopic models, namely the Krauss car-following model and the LC2013 lane-changing model. The test network is a 2 × 2 bidirectional grid with four signalized intersections spaced 500 m apart. Each approach is configured with two incoming lanes (a dedicated through lane and a dedicated left-turn lane) to match the four-phase signal plan. Vehicles are generated from boundary entry links located 250 m upstream of each stop line. The maximum speed on all links is set to 50 km/h, and vehicles are inserted with an initial speed of 35 km/h.

Traffic demand follows a time-varying peak-period profile, starting from a low level, increasing to a peak, and then gradually decreasing, with an average arrival rate of 1.25 veh/s. For each vehicle, the origin is sampled from boundary entry links and the destination is randomly selected from boundary links, yielding diversified turning movements and inter-intersection interactions. The peak-demand period produces congested (oversaturated) conditions and potential spillback. Signal control uses a fixed four-phase definition (east–west through, east–west left-turn, north–south through, and north–south left-turn). The minimum and maximum green times are set to 15 s and 60 s, respectively; a 3 s yellow clearance is applied between phases; and the all-red time is set to 0 s. All intersections start with a zero initial offset to ensure consistent initialization across runs.

4.2. Road Network Configuration

The simulation experiments are conducted on a 2 × 2 grid network, as illustrated in Figure 6. The network consists of four signalized intersections with identical geometric structures and phase plans. Each intersection supports four movement phases: east–west through, east–west left-turn, north–south through, and north–south left-turn.

All links are bidirectional, and the distance between adjacent intersections is set to 500 m. Vehicle entry points are located 250 m upstream of each stop line, enabling the simulation to capture traffic demand propagation and potential queue spillback within and across intersections under dynamic traffic conditions. Accordingly, the 2 × 2 grid network is adopted as a controlled experimental setting to examine congestion propagation across adjacent intersections, including spillback and upstream–downstream coupling. This configuration provides a coupled multi-intersection environment while retaining a well-defined and reproducible topology for systematic evaluation of coordination mechanisms.

4.3. Traffic Flow Simulation Setup

To emulate realistic urban peak-hour traffic, a dynamically varying traffic flow is generated for all incoming approaches. Vehicle arrivals follow a stochastic process with variable intensities to reflect fluctuations commonly observed during congested periods. This configuration captures the inherent randomness of traffic conditions and aims to learn control strategies that remain effective under variations in demand.

The experiments focus on ordinary vehicles without the presence of connected or prioritized vehicle types. This ensures that the learned policy is evaluated under general traffic conditions and that performance improvements can be attributed solely to the proposed signal control method.

4.4. Simulation Parameter Settings

The simulation and algorithm training are performed over 300 episodes, with each episode consisting of 400 simulation steps. Before training begins, 6000 pre-training steps are executed to populate the replay buffer with initial experience samples. Each decision step corresponds to a fixed signal duration of 10 s, followed by a 3-s yellow interval. Minimum and maximum green times are enforced for each phase to ensure safety and operational feasibility.

The discount factor γ is set to 0.99 to emphasize long-term traffic performance, consistent with the settings in Section 3. The Q-network is trained using the Adam optimizer with a learning rate of 0.001, while the policy alignment function G (s, a; θ) is trained using stochastic gradient descent with a learning rate of 0.01. These settings follow standard practices in deep reinforcement learning for traffic signal control and contribute to stable convergence.

A summary of all simulation and training parameters is presented in

Table 3. Simulation Model Parameter Settings.

Parameter	Description	Value
Number of episodes	Total training episodes	300
Steps per episode	Simulation steps per episode	400
Pre-training steps	Steps for warm-up/memory initialization	6000
Discount factor γ	Same as Table 2	0.99
ε-initial	Initial exploration rate	1.0
ε-final	Minimum exploration rate	0.05
ε-decay	Linear decay steps	220
Phase duration	Decision interval	10 s
Yellow time	Yellow phase duration	3 s

.

4.5. Simulation Setup

To evaluate the learning performance and convergence stability of the proposed PRA-DQN algorithm, ten independent simulation runs were conducted under identical settings. During each run, the cumulative reward obtained in every training episode was recorded. After all runs were completed, the episode rewards were averaged to eliminate randomness and highlight the overall training trend.

As shown in Figure 7, the average reward exhibits a clear upward trend as training progresses, indicating continuous policy improvement. In the early stages of training, the reward fluctuated considerably due to high exploration and limited samples in the replay buffer. As more experience is collected and both the Q-network and the policy function become better optimized, the reward curve gradually stabilizes and converges toward a higher value.

These results demonstrate that the explicit policy–Q alignment mechanism and parameter-sharing architecture adopted by PRA-DQN effectively enhance training stability and improve learning efficiency. The stable convergence trend also indicates that the algorithm successfully learns coordinated phase decisions across intersections, reducing congestion and improving network performance.

Benchmark Algorithms

To provide a comprehensive and fair performance evaluation, PRA-DQN is compared with six representative baseline controllers. These baselines include reinforcement learning–based approaches, rule-based algorithms, and classical fixed-time coordination.

(1)

DQN-based Coordinated Signal Control (DQN-CoC): This method approximates the Q-value function using a deep neural network and employs an ε-greedy strategy for action selection. In coordinated mode, agents share information and jointly optimize signal timings to enhance network-wide performance.

(2)

DQN-based Independent Signal Control (DQN-IC): Similarly to DQN-CoC, this method also relies on a deep neural network and ε-greedy exploration. However, each agent makes decisions solely based on its own local state without sharing information, serving as a baseline to assess the benefits of coordination.

(3)

Stochastic Control (STOCHASTIC): In this approach, the signal phase is selected randomly without considering traffic state or historical patterns. It serves as a lower-bound benchmark for assessing learning-based strategies.

(4)

Max-Pressure Control (MAXPRESSURE): This rule-based controller selects the phase with the highest pressure, computed as the density difference between incoming and outgoing lanes. It aims to resolve local flow imbalance and reduce congestion.

(5)

Max-Wave Pressure Control (MAXWAVE): This algorithm estimates shockwave propagation and calculates wave-front pressure. The phase associated with the highest wave pressure is selected to mitigate spillback and delays.

(6)

Fixed-Time Control: This classical method assigns predetermined cycle lengths and green splits at all intersections. By synchronizing signal timings, it aims to create a green-wave effect and provide a reference for evaluating adaptive control methods.

These baselines collectively cover classic, stochastic, pressure-based, and Reinforcement Learning (RL)-based strategies, ensuring a comprehensive comparison with the proposed PRA-DQN controller.

4.6. Analysis of Simulation Results

In addition to lane-level performance indicators derived from the local reward function, network-level metrics are reported to evaluate system-wide effects. Specifically, average travel time is computed as the mean entry-to-exit duration of completed trips, and throughput is measured as the number of vehicles discharged from the network per unit time. These metrics help verify whether improvements obtained under local control objectives are associated with system-level benefits.

To assess the effectiveness of PRA-DQN, all six baseline algorithms described in Section Benchmark Algorithms were implemented and tested under identical traffic conditions. Three key performance indicators were used for quantitative comparison: maximum queue length, average queue length, and average waiting time.

All results in

Table 4. Performance Comparison of Different Control Algorithms.

Model	Maximum Queue Length		Average Queue Length		Average Waiting Time
	Value	Optimization Ratio	Value	Optimization Ratio	Value	Optimization Ratio
PRA-DQN-CoC	17.60	21.17%	16.30	18.75%	8.60	17.71%
DQN-CoC	19.20	13.88%	17.50	12.50%	9.30	11.43%
DQN-IC	20.10	9.87%	18.30	8.50%	9.80	6.67%
STOCHASTIC	21.50	3.59%	19.80	1.00%	10.20	2.86%
MAXWAVE	18.70	16.14%	17.10	14.50%	9.00	14.29%
MAXPRESSURE	20.00	10.31%	18.00	10.00%	9.70	7.62%
Fixed-Time	22.30	—	20.00	—	10.50	—

are averaged over ten independent simulation runs with different random seeds. For each metric, mean values and corresponding standard deviations are reported to reflect performance variability across runs.

Table 4 summarizes the results. The performance improvement from independent DQN to coordinated DQN reflects the effect of inter-intersection coordination, whereas the additional gain from coordinated DQN to PRA-DQN indicates the contribution of the policy–Q alignment mechanism. Parameter sharing further supports consistent representation learning across intersections by exposing the shared network to diverse but structurally similar traffic states. Overall, PRA-DQN demonstrates superior performance across all evaluation metrics. Compared with Fixed-Time control, PRA-DQN reduces:

• maximum queue length by 21.17%,
• average queue length by 18.75%,
• and average waiting time by 17.71%.

These improvements highlight PRA-DQN’s ability to dynamically adjust signal phases based on real-time state observations, thereby mitigating congestion buildup.

Compared with classical DQN (DQN-IC), PRA-DQN achieves an additional 7.53% reduction in average waiting time. This improvement arises from the explicit policy function and the policy–Q alignment mechanism, which enhance action-selection stability and reduce oscillatory behavior typical in standard DQN.

Moreover, PRA-DQN outperforms both MAXPRESSURE and MAXWAVE. Although these heuristic controllers react to immediate congestion or wave-front changes, they lack long-term optimization capability. PRA-DQN, through reinforcement learning, captures both short-term conditions and long-term network effects, resulting in better overall coordination and smoother traffic operations.

These comparative results confirm that PRA-DQN effectively balances local delay, intersection pressure, and spillback mitigation, making it a robust and efficient solution for multi-intersection signal control tasks.

5. Conclusions

This study developed a cooperative multi-intersection signal control framework based on deep reinforcement learning. A comprehensive mathematical formulation was established, including the state space, action space, and reward function, which provides a rigorous foundation for the proposed control strategy. Based on this formulation, we designed a Policy-Regulated and Aligned Deep Q-Network (PRA-DQN) to alleviate learning instability and strengthen coordination in DQN-based multi-intersection signal control. The proposed method integrates explicit policy regulation, policy–Q alignment, and parameter sharing, enabling stable and interpretable phase selection across multiple intersections. In particular, the cooperative reward explicitly integrates local delay, movement pressure, and upstream–downstream interactions to better capture spillback-coupled dynamics in multi-intersection networks. SUMO-based simulations were conducted on a controlled 2 × 2 grid network under congested and coupled traffic dynamics. The results show that PRA-DQN consistently improves maximum queue length, average queue length, and average waiting time compared with multiple baseline controllers, including Fixed-Time, classical DQN, distributed DQN, and pressure/wave-based methods. Compared with fixed-time control, PRA-DQN reduces maximum queue length by 21.17%, average queue length by 18.75%, and average waiting time by 17.71%. Moreover, relative to classical DQN coordination, PRA-DQN achieves an additional 7.53% reduction in average waiting time, indicating improved coordination performance beyond standard value-based learning. Overall, these results indicate that PRA-DQN can mitigate congestion and suppress spillback propagation, thereby improving multi-intersection operational efficiency in the tested setting. Methodologically, the results suggest that aligning an explicitly parameterized policy with the Q-induced target distribution can reduce action-selection noise and mitigate unstable phase switching, while incorporating upstream–downstream coupling into the reward design helps agents explicitly account for spillback effects rather than optimizing each intersection in isolation.

Several directions remain for future research. Future work will validate the proposed approach on larger-scale and non-uniform urban networks, beyond the controlled 2 × 2 testbed used in this study. Improving real-time performance and robustness under highly dynamic and stochastic traffic conditions is also important; robust training, adaptive reward shaping, and multi-objective formulations are promising directions. In addition, integrating cooperative perception or communication mechanisms among intersections and connected vehicles may further strengthen coordination. In practical deployments, incorporating additional context information (e.g., incident reports, temporary restrictions, or priority rules) into the state representation may further improve decision-making. Finally, extending PRA-DQN to irregular intersection geometries, heterogeneous phase structures, and mixed traffic environments is another valuable avenue for investigation.