Adaptive Traffic Control Based on P-DQN+HER+TD3: Toward Real-World Deployments

Abstract

Urban traffic congestion adversely impacts vehicle mobility, fuel consumption, and environmental sustainability. Classical traffic-control schemes—fixed-time and actuated methods—suffer from rigidity and limited responsiveness, failing to adapt to dynamic, stochastic demand or capture network-wide spatiotemporal correlations. Although reinforcement learning has emerged as a promising alternative, prevalent discrete-only approaches constrain controllers to coarse-grained phase selection, yielding suboptimal performance under continuous traffic dynamics. To address these limitations, this study proposes a hybrid deep reinforcement learning framework integrating a Parameterized Deep Q-Network, Twin Delayed Deep Deterministic Policy Gradient, and Hindsight Experience Replay for intelligent traffic-signal optimization. The model jointly handles discrete-phase selection and continuous-phase duration within a unified parameterized action space. An analytical state-dependent queue model grounds the framework in traffic-flow theory, capturing dynamic queue transitions and server capacities. Both synthetic traffic data and preprocessed Next Generation Simulation trajectory data derive state variables including density, speed, queue length, and delay. The framework is evaluated in an intersection simulation against PressLight, Max-Pressure, and Frame-based Reinforcement Learning with All-Phase competition models. The proposed approach achieves lower delay, queue length, and travel time with higher throughput. These findings demonstrate that integrating trajectory-based state modeling, analytical queue formulations, and hybrid discrete–continuous reinforcement learning offers an effective and scalable solution for real-world adaptive traffic-signal control.

1. Introduction

The escalation of urban roads traffic congestion poses a significant challenge to contemporary metropolitan infrastructure, resulting in increased travel latency, elevated anthropogenic emissions, suboptimal fuel efficiency, and compromised vehicular safety [1]. Therefore, the development of cost-effective strategies for optimizing Traffic Light Control (TLC) is of critical importance in mitigating the economic and societal impacts of traffic congestion. TLC constitutes a highly complex and non-trivial optimization problem, as traffic environments are inherently dynamic, non-linear, and uncertain in nature. The stochastic fluctuations in traffic demand, heterogeneous vehicle interactions, and time-varying network conditions further exacerbate the difficulty of achieving efficient and adaptive signal control [2]. Inefficiencies in traffic-signal timing primarily result from suboptimal allocation of green-phase durations, insufficient responsiveness to real-time traffic dynamics, and lack of coordinated control across adjacent intersections [3]. Optimizing signal operations has been shown to significantly reduce congestion, minimize vehicle delays, and increase the effective throughput of urban road networks. The classical Traffic Signal Control (TSC) strategies are commonly categorized as pretimed, actuated, adaptive, and optimization-based control [4,5]. These arterial TSC schemes have traditionally relied on predetermined cycle lengths, phase sequences, and offsets to coordinate traffic progression across multiple intersections [6]. The fixed-time and coordinated signal-timing strategies can provide effective traffic progression when traffic-demand patterns are relatively stable and predictable; however, their limited ability to adapt to stochastic and spatially heterogeneous traffic conditions can result in inefficient signal utilization, increased delay, and queue propagation under varying demand [7]. Conventional implementations of these methods typically depend on predefined traffic models or rule-based heuristics derived from expert knowledge, which restricts their scalability and limits their capacity to respond to rapidly changing traffic conditions [8]. Consequently, these approaches often demonstrate constrained responsiveness in dynamic and stochastic traffic environments, leading to suboptimal network performance under high variability and fluctuating traffic-demand patterns [9]. Moreover, such systems often face challenges related to scalability, particularly as traffic volumes rise and congestion becomes more rampant. This limitation has prompted the development of better traffic-signal controllers, capable of accurately evaluating varying traffic conditions and making more effective decisions regarding TSC operations. Recent reinforcement learning-based (RL-based) traffic-signal control techniques have increasingly addressed two important challenges: coordination among multiple intersections and operation under oversaturated traffic conditions [10]. Multi-agent and network-wide RL-based approaches coordinate neighboring intersections by sharing traffic information or learning cooperative policies, improving traffic progression and reducing network-wide congestion [11]. In parallel, recent studies have incorporated pressure-based and traffic-flow information into RL-based frameworks to improve control under heavy and oversaturated demand, where queue growth and spillback can substantially degrade network performance [12]. These studies highlight the need for adaptive controllers that account for both spatial interactions among intersections and rapidly increasing queue dynamics under high demand. However, challenges remain in jointly optimizing discrete signal phases and continuous timing parameters while maintaining stable queue evolution under varying demand levels. Furthermore, these schemes may be less efficient under variable demand or unequal intersection spacing. While these methods are computationally efficient, they lack adaptability to fluctuating traffic conditions. They often fail to optimize multiple performance objectives simultaneously, such as delay, throughput, and queue length. Empirical studies have shown that trajectory-informed control strategies can effectively reduce spillover severity and improve traffic efficiency in real-world urban networks [13]. These findings have motivated continued research toward intelligent traffic-signal controllers capable of operating efficiently in large-scale urban environments with dynamic and heterogeneous traffic demands. Recent advances in deep reinforcement learning (DRL) have demonstrated strong potential for adaptive and data-driven TSC, enabling controllers to learn policies directly from observed traffic states and dynamically adjust signal phases to changing traffic conditions [14]. Early RL-based TSC approaches commonly formulated signal control as a discrete Markov Decision Process (MDP), in which agents selected signal phases based on traffic states such as vehicle counts, queue lengths, or occupancy rates [15]. More recent developments have incorporated adaptive pressure mechanisms and generalized phase-pressure frameworks to improve policy robustness and coordination across multiple intersections [16]. Nevertheless, many existing approaches primarily focus on discrete-phase selection, providing limited flexibility in continuously adjusting signal-timing parameters such as green duration. Moreover, most existing DRL approaches focus either on discrete-phase selection or continuous green-time optimization separately, limiting their ability to fully exploit the control space of traffic signals. However, existing reinforcement learning-based traffic-signal control methods often have limited capability to jointly handle discrete signal-phase selection and continuous traffic duration control parameters. In addition, effectively exploiting sparse reward signals in congested and highly dynamic traffic environments remains challenging, potentially resulting in slower convergence and reduced policy stability. This limitation motivates the integration of discrete-phase selection with continuous green-time optimization, as considered in the proposed P-DQN+HER+TD3 framework. The main contributions are summarized as follows:

• A hybrid reinforcement learning model is proposed that jointly optimizes discrete traffic phase selection and continuous-phase durations by combining Parameterized Deep Q-Network with Twin Delayed Deep Deterministic Policy Gradient (TD3), enabling more efficient traffic-signal control.
• A data-driven evaluation is conducted using preprocessed NGSIM trajectory data, capturing lane-level traffic dynamics such as speed, density, queue length, and delay to ensure realistic policy learning.
• Enhanced learning is achieved through the integration of Hindsight Experience Replay, enabling the agent to learn effectively from a goal-conditioned experience-reuse mechanism and improve sample efficiency.
• An extensive study is performed to evaluate the proposed hybrid agent against state-of-the-art baseline traffic-control models, including PressLight, Max-Pressure, and Frame-based Reinforcement learning with All-Phase competition, using multiple metrics to provide a comprehensive performance assessment.
• The study compares computation time to evaluate model efficiency and its trade-off with the other performance metrics.

The paper is organized as follows: Section 2 reviews the existing literature related to RL-based traffic-control and hybrid deep learning approaches, highlighting key limitations that motivate this study. Section 3 presents the proposed methodology, including the problem formulation and the design of the integrated P-DQN+HER+TD3 architecture. Section 4 details NGSIM data preprocessing and the experimental setup. Section 5 evaluates the performance of the proposed approach using both synthetic traffic scenarios and real-world NGSIM datasets. Finally, Section 6 concludes the paper by summarizing the key findings, outlining practical implications, and highlighting directions for future research.

2. Related Works

Recent studies have generally classified TSC optimization approaches into three core paradigms: traditional model-based techniques, adaptive reinforcement learning (RL) strategies, and back-pressure methods. Model-based approaches optimize signal timing using representations such as macroscopic fundamental diagrams, but they often face scalability challenges and real-time implementation due to computational complexity [17]. Adaptive RL-based traffic-signal control approaches can be distinguished according to how traffic-domain knowledge is incorporated into the learning architecture. Recent TSC research schemes have increasingly shifted from isolated intersection optimization toward coordinated control of multiple intersections within an urban traffic network. Multi-agent reinforcement learning (MARL) provides a suitable framework for this transition by assigning agents to individual intersections while enabling them to learn cooperative policies from local and neighboring traffic information. The authors in [18] developed a network-wide coordinated RL framework using centralized training and decentralized execution, with an attention mechanism to capture spatial–temporal dependencies among neighboring intersections. Similarly, a two-layer coordinated MARL framework that combines local cooperation with global coordination to optimize traffic congestion and vehicle emissions across multiple intersections is further proposed in [19]. Other cooperative MARL approaches have also introduced communication or neighborhood information sharing to mitigate partial observability and improve coordination between adjacent signalized intersections [20]. These studies demonstrate that effective network-level signal control requires consideration of interactions between neighboring intersections, since decisions at one junction can influence downstream queues, traffic progression, and congestion at other junctions. An important challenge in TSC under saturated and oversaturated demand arises when traffic inflow approaches or exceeds the effective discharge capacity of the intersections. Under these conditions, queues increase rapidly, downstream storage capacity becomes constrained, and queue spillback can propagate congestion to upstream intersections, thereby limiting the effectiveness of conventional signal-control strategies. This issue is addressed in [21] by incorporating a virtual mesoscopic traffic model into an RL-based signal controller to explicitly represent oversaturated traffic conditions and evaluate signal strategies at a congested intersection. Their results demonstrate the importance of representing network traffic conditions beyond short-term intersection states when demand exceeds available capacity. Similarly, the authors in [22] developed a meta-reinforcement learning approach that adapts the reward function according to the saturation level of traffic flow, recognizing that the control objective changes between unsaturated and saturated regimes. Under heavy demand, therefore, effective TSC must simultaneously consider queue accumulation, throughput, phase allocation, and the prevention of excessive downstream occupancy. State-of-the-art TSC models, such as PressLight and Frame-based Reinforcement learning with All-Phase competition (FRAP), have been proposed in the literature to regulate traffic flow, which use pressure-based and phase-competition strategies by employing RL algorithms for adaptive signal timing [23]. Pressure-guided methods build on the Max-Pressure principle by using upstream and downstream traffic conditions to characterize the urgency of competing movements [24]. PressLight represents an important integration of this principle with reinforcement learning, in which pressure information is incorporated into the traffic-state and reward formulation and an RL agent learns the signal-control policy from interactions with the traffic environment. Advanced traffic-state representation that captures pressure and effective running-vehicle information is proposed in [25]. The authors of this work subsequently integrated RL in their model to develop Advanced-MPLight and Advanced-CoLight, achieving strong performance across multiple real-world traffic networks. In [26], the authors further integrated RL with Max-Pressure by employing policy-gradient learning to optimize pressure-control parameters while accounting for phase-switching losses, thereby combining the theoretical advantages of pressure-based control with data-driven policy optimization. More recently, the authors of [27] developed a hybrid Max-Pressure and PPO framework in which Max-Pressure dynamically determines phase order while RL optimizes phase timings, providing another explicit mechanism for combining traffic-pressure knowledge with adaptive learning. In contrast, phase-symmetric approaches, such as FRAP, are designed around signal control through competition among conflicting phases rather than explicitly using pressure-based control. Collectively, these studies demonstrate that RL can be integrated with traffic-control principles through state representation, reward design, policy optimization, or hybrid phase-selection and timing mechanisms, providing adaptive responses to dynamic and network-level traffic conditions. Additionally, FRAP introduces multi-agent phase competition to prioritize conflicting traffic movements to mitigate queue spillback [28]. Max-Pressure algorithms were initially developed for isolated intersections and later extended to large, heterogeneous networks. They enable decentralized signal control based on queue lengths, offering practical advantages over model-based methods [29]. However, real-world implementations of back-pressure-based TSC have remained limited, with only a small number of experimental deployments reported in the literature. For example, an experimental Max-Pressure traffic controller based on travel-time delays, including the use of real-time Bluetooth data for traffic-state estimation and signal control is proposed in [30]. Despite these promising results, such approaches have largely been validated on small-scale testbeds, highlighting the need for further investigation into their scalability and practical deployment in large urban networks. The MoveLight framework adopts a movement-centric deep reinforcement learning approach that performs traffic-signal control at the lane/movement level, enabling more fine-grained representation of traffic demand. Experimental results using real-world datasets from Cologne and Hangzhou demonstrated improvements in queue length, delay, and throughput compared with existing TSC methods. However, the approach relies on detailed real-time traffic information and deep reinforcement learning, which can increase the computational and data requirements associated with training and deployment [31]. Deep Q-Networks (DQN) and its variants, such as Double DQN and dueling DQN, have been successfully applied to this problem, demonstrating improvements in delay reduction and throughput compared to traditional fixed-time and actuated control strategies [32]. However, these methods are constrained to discrete action spaces, restricting the granularity of green-time optimization and limiting performance in scenarios requiring fine-grained control [33]. Recent work has introduced advanced dueling and double-architecture enhancements to improve convergence stability and fairness in discrete traffic environments, highlighting both the continued relevance and limitations of discrete RL approaches [34]. Continuous control approaches, such as Deep Deterministic Policy Gradient (DDPG) and TD3, have been explored to optimize continuous signal parameters, including phase durations and offsets. However, these methods still struggle to handle discrete-phase selection while simultaneously optimizing continuous timing, limiting their applicability in practical TSC tasks where both decisions are essential. Hybrid action-space models developed for TSC reveal a fundamental limitation in conventional RL approaches, as purely discrete or purely continuous formulations alone are inadequate to fully capture the complexity of traffic-control decisions [35]. Instead, these models highlight the necessity of integrating both discrete decisions (e.g., phase selection) and continuous control (e.g., phase-duration adjustment) within a unified learning framework to effectively capture the complexity and dynamics of real-world traffic systems. Although recent advances in RL-based TSC have significantly improved decision-making in complex environments, existing approaches still suffer from structural limitations that constrain their practical effectiveness. Although these approaches effectively combine discrete and continuous actions, they often suffer from learning instability and limited exploration efficiency, particularly in environments with sparse rewards or highly dynamic states. Other studies have explored model-based RL for parameterized action spaces to improve sample efficiency and long-term decision performance [36]. Complementary work applying enhanced TD3 variants and graph-based architectures has demonstrated improvements in continuous-phase timing optimization but still does not solve the discrete decision component alone when applied directly [37]. Moreover, RL methods for TSC often encounter sparse reward environments, leading to slow convergence and sub-optimal policies without careful design of reward structures or experience sampling. However, the sample efficiency and convergence remain a fundametal challenge in large-scale networks. Recent surveys highlight these limitations in both single- and multi-agent RL approaches to urban TSC and emphasize ongoing research needs for scalable hybrid systems. Parameterized DQN (P-DQN) models combine discrete action selection with continuous parameters, allowing joint optimization of traffic-light phase selection and time-duration sequencing [38,39]. HER is further used to improve learning efficiency by allowing agents to learn from unsuccessful or partially achieved goals, mitigating the challenge of sparse rewards. Furthermore, few studies integrate advanced replay strategies such as HER with actor–critic optimization methods such as TD3 within a unified parameterized reinforcement learning framework. To address this gap, this study proposes a data-driven TSC framework that integrates P-DQN, HER, and TD3 within a unified learning architecture, enabling more efficient policy learning and improved traffic management performance in complex urban intersections. The proposed model integrates parameterized action-space learning for simultaneous discrete-phase selection and continuous green-time optimization, HER as a goal-oriented experience relabeling mechanism, and TD3 for stable continuous-action learning optimization. This integration enables the controller to learn goal-directed traffic-signal policies and adapt to varying traffic conditions, thereby addressing the core limitations of existing RL methods in real-world signal-control optimization. Leveraging preprocessed NGSIM trajectory data, our approach jointly optimizes discrete-phase selection and continuous-phase durations while benefiting from enhanced sample efficiency and improved convergence stability. The proposed method is benchmarked against PressLight, Max-Pressure, and FRAP to demonstrate superior performance in terms of delay reduction, throughput, travel time, and queue management under realistic traffic dynamics. To the best of the authors’ knowledge, no previous study has integrated P-DQN, HER, and TD3 within a unified parameterized RL-based framework for data-driven TSC using NGSIM (Lankershim and Peachtree) traffic datasets. The proposed P-DQN+HER+TD3 framework is motivated by the inherent characteristics of ATSC, particularly the presence of hybrid discrete–continuous actions, sparse or delayed learning signals, and the need for stable continuous-parameter optimization. The three components are therefore assigned complementary roles rather than being used as independent or redundant learning algorithms.

• First, P-DQN is adopted to address the hybrid action space encountered in TSC. A traffic-signal controller typically needs to make a discrete decision regarding the active signal phase while simultaneously determining continuous parameters such as phase duration, green extension, or timing adjustments. P-DQN provides a unified mechanism for representing each discrete signal-phase action together with its associated continuous parameters.
• Second, HER is incorporated to improve learning efficiency when the desired traffic objective is difficult to achieve directly. Traffic-control episodes may contain delayed or sparse reward information because the effect of a signal decision on queue dissipation, vehicle delay, and traffic throughput may only become evident after several subsequent control steps. HER increases the diversity and usefulness of the replay experience by reconsidering unsuccessful trajectories with alternative achieved outcomes.
• Third, TD3 is employed to improve the stability of continuous-parameter learning. Continuous signal-timing parameters are sensitive to estimation errors in the critic, and overestimation of action values can lead to unstable policy updates. TD3 addresses this issue through its twin-critic architecture, delayed policy updates, and target-policy smoothing. These mechanisms provide more reliable value estimation for the continuous-timing component and reduce undesirable oscillations in the learned control policy.

3. Methodology

Figure 1 shows a generalized block diagram for a data-driven RL framework for intelligent TSC for an urban road intersection. The traffic environment block models a real-world road intersection where vehicles move following traffic dynamics. The traffic enviroment generates observable traffic states such as vehicle positions, queue lengths, lane occupancies, and arrival rates. These raw traffic measurements are passed to the State Data Extraction and Processing module, where they are structured, normalized, and transformed into a compact state representation suitable for training and RL policy making. The processed state vectors capture the real-time traffic condition of the intersection. Additionally, the formatted states are then fed into the P-DQN Actor Network, which selects a hybrid control action (discrete-phase selection with continuous-timing parameters) to optimize signal performance. The selected action is applied back to the traffic environment, influencing vehicle movements and generating a reward signal based on traffic efficiency metrics such as delay reduction or queue minimization. Each interaction tuple (state, action, reward, next state) is stored in the Replay Buffer, which incorporates HER to improve data sample efficiency by relabeling goals and enhancing learning from sparse rewards. During training, mini-batches sampled from the replay buffer are used to update the network parameters. The TD3 Critic Network evaluates the selected actions by estimating Q-values and minimizes the target loss function to stabilize learning and reduce overestimation bias. The actor network is updated through policy gradients derived from the critic’s evaluation. This iterative loop continues until convergence, resulting in an optimized traffic-control policy capable of adaptively managing intersection traffic flow under varying conditions.

Figure 1. Conceptual framework of P-DQN-based traffic-control training architecture: The arrows illustrate the systematic flow of the proposed process, while the distinct colors are utilized solely to enhance visual clarity and differentiate between the various architectural components.

3.1. P-DQN+HER+TD3 Model for Traffic-Signal Control (Lankershim and Peachtree Datasets)

Following the hierarchical discrete–continuous reinforcement learning formulation commonly used in hybrid control problems (e.g., UAV navigation), a traffic-light control system can be modeled in a symmetric way as a hybrid Markov Decision Process (MDP) with a parameterized action space [40], as follows:

$$\mathcal{M}=⟨\mathcal{S},\mathcal{A},P,r,\gamma ⟩,$$

(1)

where $s_t\in \mathcal{S}$ denotes the traffic states at time t, $a_t\in \mathcal{A}$ is the control action, $r_t=r(s_t,a_t)$ is the reward function, P represents the transition dynamics, and $\gamma \in (0,1)$ is the discount factor. At each decision step t, the RL environment provides the agent with the current traffic state $s_t$, including the observed or estimated queue conditions, signal status, and traffic-flow information. The P-DQN component selects the discrete-signal phase, while the continuous-action component determines the corresponding green-time parameter. The resulting hybrid action is applied to the developed traffic environment, which produces the next state $s_{t+1}$ and reward $r_t$. The transition $(s_t,a_t,r_t,s_{t+1})$ is subsequently stored in the replay buffer for policy learning. Unlike purely discrete- or purely continuous-control strategies, traffic-signal optimization naturally exhibits a hierarchical structure. At each decision step t, the RL-agent performs the following actions:

• Selection of a high-level discrete action: Selection of traffic-signal phases (e.g., north–south green, east–west green, and protected turning phase).
• Determination of a low-level continuous parameter: Adjustment of signal-timing parameters such as phase duration or green-time allocation.

The action consists of a discrete-phase selection and a continuous-duration parameter:

$$a_t=(k_t,{\theta }_{k_t}),$$

(2)

where $k_t\in \{1,\dots ,K\}$ represents the selected signal phase and ${\theta }_{k_t}\in {\mathcal{X}}_{k_t}\subset {\mathbb{R}}^d$ denotes the associated continuous green-time parameter. Thus, the hybrid action space is defined as

$$\mathcal{A}=\left\{(k,{\theta }_k)\mid k\in \{1,\dots ,K\},{\theta }_k\in {\mathcal{X}}_k\right\}$$

(3)

3.1.1. Reward Function

The reward function is designed to reflect the primary control objective of adaptive traffic-signal control, namely, to reduce congestion while maintaining efficient vehicle movement through the controlled intersection. To maintain consistency between the reinforcement learning state representation and the reward formulation, the reward is constructed primarily from the four state variables, namely, vehicle count, average speed, aggregated queue length, and traffic density. Throughput is additionally incorporated as a control-performance signal because it directly measures the number of vehicles successfully served by the intersection. The four-dimensional state at time step t is given by

$$s_t={[n_t,{\overline{v}}_t,q_t,{\rho }_t]}^{\mathrm{T}},$$

(4)

where $n_t$ denotes the number of vehicles, ${\overline{v}}_t$ is the average vehicle speed, $q_t$ is the aggregated queue length, and ${\rho }_t$ represents traffic density. Accordingly, the instantaneous reward is formulated as

$$r_t^{\mathrm{base}}=-\left(\alpha {\widehat{n}}_t+\beta {\widehat{q}}_t+\gamma {\widehat{\rho }}_t\right)+\delta {\widehat{v}}_t+\lambda {\widehat{T}}_t,$$

(5)

where ${\widehat{n}}_t$, ${\widehat{v}}_t$, ${\widehat{q}}_t$, and ${\widehat{\rho }}_t$ are the normalized vehicle count, average speed, aggregated queue length, and traffic density, respectively, while ${\widehat{T}}_t$ denotes the normalized intersection throughput. The coefficients $\alpha$, $\beta$, $\gamma$, $\delta$, and $\lambda$ determine the relative contribution of the individual traffic-performance measures. Because the traffic-performance measures have different numerical scales, each metric is normalized before being incorporated into the reward function. Table 1 summarizes representative configurations considered during the sensitivity analysis. The final coefficients are selected based on their ability to provide a balanced trade-off between congestion reduction and traffic efficiency while maintaining stable learning. In particular, greater emphasis is placed on queue length and delay because these measures directly characterize congestion and the quality of traffic progression.

Table 1. Sensitivity analysis of reward-weighting coefficients and their impact on traffic performance.

$\mathit{\alpha }$	$\mathit{\beta }$	$\mathit{\gamma }$	$\mathit{\delta }$	$\mathit{\lambda }$	Avg. Queue	Avg. Delay (s)	Throughput	Convergence
0.4	0.3	0.1	0.1	0.5	12.5	18.2	420	Stable
0.5	0.3	0.1	0.05	0.5	10.8	15.6	445	Stable
0.6	0.2	0.1	0.05	0.4	9.9	14.8	438	Stable
0.3	0.4	0.1	0.1	0.6	11.7	16.9	460	Slightly Unstable
0.2	0.3	0.2	0.2	0.7	13.9	19.5	470	Unstable

Thus, the reward provides a direct trade-off between congestion reduction and traffic service. Because the traffic variables have different physical units and numerical ranges, each component is normalized before being incorporated into the reward. For a generic traffic variable $x_t$, min–max normalization is expressed as

$$\widehat{x}\ast t={\displaystyle \frac{x_t-x\ast \min }{x_{\max }-x_{\min }+ϵ}},$$

(6)

where $x_{\min }$ and $x_{\max }$ are the lower and upper bounds obtained from the training data, respectively, and $ϵ$ is a small positive constant introduced to avoid division by zero. For the average-speed term, the normalized quantity is defined so that higher speeds produce higher rewards,

$$\widehat{v}\ast t={\displaystyle \frac{\overline{v}\ast t-v\ast \min }{v\ast \max -v_{\min }+ϵ}}.$$

(7)

The normalized throughput is similarly defined as

$$\widehat{T}\ast t={\displaystyle \frac{T_t-T\ast \min }{T_{\max }-T_{\min }+ϵ}},$$

(8)

where $T_t$ represents the number of vehicles successfully served during time step t. The final reward may subsequently be clipped to a bounded interval to improve training stability,

$$r_t^{\mathrm{clip}}=\max \left(r_{\min },\min (r_t,r_{\max })\right),$$

(9)

where $r_{\min }$ and $r_{\max }$ denote the lower and upper reward limits, respectively. In the implemented framework, reward clipping is applied to constrain extreme reward values and improve the stability of the P-DQN+HER+TD3 learning process.

3.1.2. Traffic-Flow Normalization via Passenger Car Units (PCUs)

To accurately reflect heterogeneous real-world traffic compositions within the simulation environment, all raw vehicle counts were converted into equivalent Passenger Car Units (PCUs) at the ingestion phase. This normalization ensures that traffic density, queue lengths, and state representations adhere strictly to standard traffic engineering practices rather than treating structurally diverse vehicles identically. The dynamic PCU conversion is computed continuously as

$${\mathrm{PCU}}_t=\sum _i{N}_{i,t}\times {\alpha }_i$$

(10)

where $N_{i,t}$ represents the active count of vehicle type i at time step t, and ${\alpha }_i$ denotes the corresponding standard PCU conversion factor listed in Table 2. Consequently, all model training, reward calculations, and subsequent comparative analyses were executed on these normalized PCU metrics from the initialization of the simulation.

Table 2. Standard Passenger Car Unit (PCU) conversion factors embedded in the simulation.

Vehicle Type	PCU Factor
Passenger Car	1.0
Motorcycle	0.5
Light Commercial Vehicle	1.5
Bus	2.5
Heavy Truck	3.0

Let ${\mathcal{N}}_i\left(t\right)$ be the set of active vehicles of a specific type i at time t, and let $\mathcal{N}\left(t\right)$ be the total set of all vehicles in the system. The mathematical relationship is defined as follows:

$${\mathcal{N}}_i\left(t\right)\subseteq \mathcal{N}\left(t\right)$$

(11)

By taking the cardinality (count) of these sets, we obtain their respective scalar values where $N_{i,t}=\left|{\mathcal{N}}_i\left(t\right)\right|$ represents the active count of vehicle type i, and $n\left(t\right)=\left|\mathcal{N}\right(t\left)\right|$ represents the total number of vehicles:

$$N_{i,t}\le n\left(t\right)\mathrm{where}n\left(t\right)=\sum _i{N}_{i,t}$$

(12)

3.1.3. Parameterized Q-Network (P-DQN)

The reward objective function formulation enables the RL-agent to learn TSC policies that balance congestion mitigation with efficient traffic flow, leading to improved intersection performance under varying traffic demand conditions [41]. Therefore, the action-value function in parameterized action spaces is defined as

$$Q(s,k,{\theta }_k;\omega ),$$

(13)

where $\omega$ denotes the critic network parameters. The discrete phase is selected by maximizing over discrete actions while using continuous-parameter policies:

$$k^{\ast }=\arg \underset{k}{\max }Q\left(s,k,{\mu }_k(s;{\varphi }_k);\omega \right),$$

(14)

which is a standard formulation in hybrid action reinforcement learning. Where ${\mu }_k(s;{\varphi }_k)$ is the continuous parameter policy for phase k. The critic network is trained using a temporal-difference loss:

$${\mathcal{L}}_Q\left(\omega \right)={\mathbb{E}}_{(s,a,r,s^{\prime })}\left[{\left(y-Q(s,k,{\theta }_k;\omega )\right)}^2\right],$$

(15)

with target:

$$y=r+\gamma \underset{k^{\prime }}{\max }Q\left(s^{\prime },k^{\prime },{\mu }_{k^{\prime }}(s^{\prime };{\varphi }_{k^{\prime }}^{-});{\omega }^{-}\right).$$

(16)

The continuous parameter policy is updated using the deterministic policy gradient:

$${\nabla }_{{\varphi }_k}J={\mathbb{E}}_s\left[{\nabla }_{{\theta }_k}Q(s,k,{\theta }_k;\omega ){\nabla }_{{\varphi }_k}{\mu }_k\left(s\right)\right].$$

(17)

which follows the Bellman optimality principle adapted to parameterized action spaces.

3.1.4. TD3 Stabilization

To mitigate overestimation bias and improve stability, two critics are used as in [42] as follows:

$$Q_1(s,k,{\theta }_k;{\omega }_1),Q_2(s,k,{\theta }_k;{\omega }_2),$$

(18)

The TD3 target is computed using clipped double Q-learning as follows:

$$y=r+\gamma \underset{i=1,2}{\min }{Q}_i\left(s^{\prime },k^{\prime },{\tilde{\theta }}_{k^{\prime }};{\omega }_i^{-}\right),$$

(19)

where target policy smoothing is applied:

$${\tilde{\theta }}_{k^{\prime }}={\mu }_{k^{\prime }}(s^{\prime };{\varphi }_{k^{\prime }}^{-})+ϵ,ϵ\sim \mathrm{clip}(\mathcal{N}(0,\sigma ),-c,c),$$

(20)

which improves robustness by reducing variance in value estimates. Each critic minimizes a temporal-difference loss:

$${\mathcal{L}}_{\mathrm{TD}3}\left({\omega }_i\right)=\mathbb{E}\left[{\left(y-Q_i(s,k,{\theta }_k;{\omega }_i)\right)}^2\right].$$

(21)

The actor is updated using delayed policy updates to enhance stability:

$${\nabla }_{{\varphi }_k}J={\mathbb{E}}_s\left[{\nabla }_{{\theta }_k}{Q}_1(s,k,{\theta }_k;{\omega }_1){\nabla }_{{\varphi }_k}{\mu }_k\left(s\right)\right],$$

(22)

where the policy is updated every d steps as proposed in TD3.

3.1.5. Hindsight Experience Replay for P-DQN+HER+TD3 Traffic-Signal Control Model

Hindsight Experience Replay enhances off-policy reinforcement learning by augmenting the replay buffer with additional transitions in which the originally intended goal is replaced by goals that were actually obtained during ealier episodes. This relabeling strategy significantly improves sample efficiency and accelerates convergence, particularly in environments characterized by sparse or delayed rewards. A trajectory that does not achieve its initially specified traffic objective may nevertheless contain useful information about a traffic state that was successfully achieved. To exploit such experiences, HER is incorporated into the proposed P-DQN + TD3 framework as a goal-conditioned experience-reuse mechanism. HER augments the replay buffer by allowing previously collected trajectories to be reconsidered with respect to alternative traffic-state goals. This provides additional goal-conditioned learning signals and increases the diversity of useful experiences available during policy optimization. A transition in the goal-conditioned replay buffer is represented in [43,44] as follows:

$$(s_t,g,a_t,r_t,s_{t+1}),$$

(23)

where $s_t$ denotes the traffic state, $a_t$ is the hybrid control action, and g represents the desired traffic-state goal. The goal is defined using a selected representation of the desired traffic condition, such as queue, delay, waiting-time, occupancy, or throughput characteristics. Let $f\left(s\right)$ denote the mapping from the traffic state to the corresponding goal representation. For an episode terminating at state $s_T$, HER constructs an achieved goal from the terminal traffic state according to

$$g^{\prime }=\varphi \left(s_T\right),$$

(24)

where $\varphi (·)$ extracts the relevant traffic-performance variables from the terminal state. The relabeled goal $g^{\prime }$ therefore represents a traffic condition that was actually achieved during the episode. The goal g represents the desired traffic condition at the controlled intersection and is defined using the key traffic-performance measures:

$$g=\left[Q^{\ast },D^{\ast },W^{\ast },O^{\ast },T^{\ast }\right],$$

(25)

where $Q^{\ast }$, $D^{\ast }$, $W^{\ast }$, $O^{\ast }$, and $T^{\ast }$ denote the desired queue length, vehicle delay, waiting time, lane occupancy, and throughput, respectively. The corresponding achieved goal at the end of an episode is obtained from the terminal traffic state as

$$g^{\prime }=\varphi \left(s_T\right)=\left[Q_T,D_T,W_T,O_T,T_T\right],$$

(26)

where $s_T$ denotes the terminal state and $\varphi (·)$ is a mapping that extracts the relevant traffic-performance measures from the state. To maintain consistency with the dense traffic-performance objective defined in Equation (5), the reward associated with a relabeled transition is recomputed using a goal-conditioned formulation. Specifically,

$$r_t(s_t,a_t,g)=r_t^{\mathrm{base}}-\eta d\left(f\left(s_{t+1}\right),g\right),$$

(27)

where $r_t^{\mathrm{base}}$ is the original dense traffic-performance reward, $d(·,·)$ measures the deviation between the achieved traffic state and the desired goal, and $\eta$ controls the contribution of the goal-conditioned term. A normalized distance measure can be used so that the goal-conditioned component remains comparable in magnitude to the traffic-performance reward. During hindsight relabeling, the original goal g is replaced by the achieved goal $g^{\prime }$, and the corresponding reward is recomputed as

$$r_t^{\prime }=r_t(s_t,a_t,g^{\prime }).$$

(28)

This process allows a trajectory that failed to achieve its original goal to be treated as successful with respect to a traffic condition that it actually achieved. The resulting hindsight transitions provide additional training information without requiring additional interaction with the traffic environment. In this study, a terminal-state relabeling strategy is adopted, in which the final state of an episode is used to construct the achieved goal. This choice is motivated by the system-level nature of TSC, where the overall quality of a control policy is evaluated in terms of aggregate traffic conditions over an episode. Restricting relabeling to terminal states also limits the number of additional transitions introduced into the replay buffer and reduces the computational overhead associated with more extensive relabeling strategies. The augmented replay buffer is consequently defined as

$$\mathcal{D}=\mathcal{D}\mathrm{original}\cup \mathcal{D}\mathrm{HER},$$

(29)

where $\mathcal{D}\mathrm{original}$ contains transitions generated using the original traffic-control goals and $\mathcal{D}\mathrm{HER}$ contains transitions relabeled using achieved traffic-state goals. Both forms of experience contribute to policy learning, with the original transitions preserving the primary dense traffic-performance objective and the hindsight transitions improving goal-conditioned experience utilization. The resulting formulation establishes a clear distinction between the two mechanisms: the dense reward function provides continuous feedback on traffic-control performance, while HER enhances the reuse of collected trajectories by introducing alternative achieved traffic-state goals. Thus, HER complements rather than replaces the multi-objective reward formulation and is intended to improve the efficiency and robustness of learning under diverse traffic conditions.

3.1.6. Integrated Optimization Objective

The final critic objective under P-DQN + TD3 + HER is expressed as follows:

$$\underset{{\omega }_1,{\omega }_2}{\min }{\mathbb{E}}_{(s,g,a,r,s^{\prime })\sim \mathcal{D}}\left[{\displaystyle \sum _{i=1}^2}{\left(r+\gamma \underset{j}{\min }{Q}_j(s^{\prime },g,k^{\prime },{\tilde{\theta }}_{k^{\prime }})-Q_i(s,g,k,{\theta }_k)\right)}^2\right],$$

(30)

The actor objective is similarly expressed as follows:

$$\underset{{\varphi }_k}{\max }{\mathbb{E}}_{s,g}\left[Q_1(s,g,k,{\mu }_k(s,g))\right],$$

(31)

Under bounded rewards and Lipschitz continuity assumptions, the Bellman operator is expressed as follows:

$$\mathcal{T}Q=r+\gamma {\mathbb{E}}_{s^{\prime }}\underset{i}{\min }{Q}_i\left(s^{\prime }\right).$$

(32)

where a $\gamma$-contraction ensures convergence to a fixed point.

3.2. Analytical Modeling of Urban Traffic via State-Dependent Queues (Peachtree Trajectory-Based Dataset)

This section presents the proposed P-DQN+HER+TD3 framework for adaptive TSC at signalized urban intersections. The proposed P-DQN+TD3+HER controller is trained using a goal-conditioned experience-replay mechanism in which future achieved traffic states were retrospectively used as alternative goals. The methodology is designed to address the hybrid nature of TSC, where the controller must simultaneously select a discrete signal phase and determine the corresponding continuous green-time duration. To ensure a realistic representation of traffic operations, the proposed framework incorporates movement-specific queue dynamics, traffic demand, phase-dependent saturation flow, signal lost time, finite queue-storage capacity, and spillback effects into the traffic environment. The proposed methodology comprises four interconnected stages: First, empirical traffic characteristics are extracted from the NGSIM trajectory Peachtree datasets and used to parameterize the traffic environment. Second, an interactive intersection environment is developed based on movement-level queue conservation, phase-dependent discharge capacity, signal-timing constraints, and finite queue-storage capacity. Third, the P-DQN+HER+TD3 framework is employed to jointly determine the discrete signal phase and the associated continuous green-time parameter. Finally, the trained controller is evaluated under varying traffic-demand conditions using key performance indicators, including average delay, throughput, and average queue length. Let the intersection consist of a set of traffic movements $\mathcal{M}=\{1,2,\dots ,M\}$ and a set of signal phases $\mathcal{P}=\{1,2,\dots ,P\}$. Each movement represents a traffic stream associated with a particular intersection approach and permitted movement. The environment maintains an independent queue for each traffic movement.

3.2.1. Movement-Specific Queue Dynamics

Let $q_m\left(k\right)$ denote the number of vehicles waiting in movement m at decision interval k. The total intersection queue is expressed in [45] as

$$Q\left(k\right)=\sum _{m\in \mathcal{M}}{q}_m\left(k\right).$$

(33)

The evolution of each movement-specific queue is governed by a vehicle conservation equation:

$$q_m(k+1)=\min \left\{K_m,\max \left[0,q_m\left(k\right)+A_m\left(k\right)-D_m\left(k\right)\right]\right\},$$

(34)

where $A_m\left(k\right)$ is the number of vehicles arriving at movement m during decision interval k, $D_m\left(k\right)$ is the number of vehicles discharged from movement m, and $K_m$ represents the maximum storage capacity of the movement. Equation (34) explicitly imposes the physical limits of the intersection. The lower bound prevents negative queue lengths, while the upper bound prevents the number of queued vehicles from exceeding the available storage space. The formulation therefore provides a direct mechanism for representing queue accumulation and potential spillback under congested traffic conditions.

3.2.2. Traffic Demand and State-Dependent Arrival Process

The arrival process is parameterized using traffic characteristics obtained from the processed NGSIM observations. Let ${\lambda }_m\left(k\right)$ denote the baseline arrival rate of movement m during decision interval k. To represent increasing traffic pressure under congested conditions, the effective arrival rate is formulated in [46] as

$${\lambda }_m^{\mathrm{eff}}\left(k\right)={\lambda }_m\left(k\right)\left[1+{\beta }_m{\displaystyle \frac{q_m\left(k\right)}{K_m}}\right],$$

(35)

where ${\beta }_m$ is a movement-specific congestion sensitivity coefficient. The state-dependent formulation allows the arrival process to respond to the current traffic state and provides a mechanism for representing increasing traffic pressure as queues become congested [47]. For controlled traffic-demand experiments, the number of arrivals during a decision interval $\Delta t$ can be generated according to

$$A_m\left(k\right)\sim Poisson\left({\lambda }_m^{\mathrm{eff}}\left(k\right)\Delta t\right).$$

(36)

When empirical demand profiles are used, the observed or estimated movement-level arrival counts obtained from the NGSIM data are supplied directly to the environment. Thus, the state-dependent formulation is primarily used to construct controlled demand conditions and evaluate controller robustness under increasing traffic intensity.

3.2.3. Hybrid Traffic-Signal Control Action

Traffic-signal control is formulated as a hybrid action consisting of a discrete signal-phase decision and a continuous green-time parameter. The action selected at decision interval k is defined as

$$a\left(k\right)=\left(p\left(k\right),g_{p\left(k\right)}\left(k\right)\right),$$

(37)

where $p\left(k\right)\in \mathcal{P}$ denotes the selected signal phase and $g_{p\left(k\right)}\left(k\right)$ represents its corresponding green duration. The continuous green-time parameter is constrained according to

$$g_{\min }^p\le g_p\left(k\right)\le g_{\max }^p,p\in \mathcal{P}.$$

(38)

These constraints prevent the RL controller from generating physically infeasible signal timings. The P-DQN component determines the discrete phase, whereas the TD3 component optimizes the continuous green-time parameter associated with the selected phase.

3.2.4. Phase-Dependent Saturation Flow and Lost Time

The discharge capacity of a signalized intersection depends on the active phase and the saturation flow of each movement [48]. Let $s_m^p$ denote the saturation flow rate of movement m when phase p is active. The effective green time is reduced by the signal lost time $L_p$ and is therefore given by

$$g_p^{\mathrm{eff}}\left(k\right)=\max \left[0,g_p\left(k\right)-L_p\right].$$

(39)

The effective green time is defined as

$$g_p^{\mathrm{eff}}\left(k\right)=\max \left\{0,g_p\left(k\right)-L_p\right\},$$

(40)

where $g_p\left(k\right)$ is the selected green duration for phase p and $L_p$ denotes the associated lost time, including the effects of yellow and all-red intervals. Consequently, only the effective green time contributes to vehicle discharge. The phase-dependent effective discharge rate for movement m is then defined as

$${\mu }_m^{\mathrm{eff}}\left(k\right)=s_m^{p\left(k\right)}{\displaystyle \frac{g_{p\left(k\right)}^{\mathrm{eff}}\left(k\right)}{C\left(k\right)}},$$

(41)

where $C\left(k\right)$ represents the signal-cycle duration. The maximum number of vehicles that can be discharged from movement m during decision interval k is consequently

$$D_m\left(k\right)=\min \left[q_m\left(k\right)+A_m\left(k\right),{\mu }_m^{\mathrm{eff}}\left(k\right)\Delta t\right].$$

(42)

For a movement that is not served by the active phase,

$$D_m\left(k\right)=0.$$

(43)

This formulation establishes a direct physical relationship between the RL control action and traffic evolution. The RL-agent does not directly modify the physical saturation flow. Instead, it selects the signal phase and green duration, which determine the portion of the movement-specific saturation capacity available for vehicle discharge.

3.2.5. Signal-Timing and Cycle Constraints

To ensure physically feasible signal operation, the proposed environment enforces both phase-level and cycle-level constraints. The green duration of each phase is bounded by

$$g_{\min }^p\le g_p\left(k\right)\le g_{\max }^p.$$

(44)

The cycle duration is constrained according to

$$C_{\min }\le \sum _{p\in \mathcal{P}}\left[g_p\left(k\right)+L_p\right]\le C_{\max }.$$

(45)

The constraints account for both effective green time and signal lost time and prevent the learned controller from producing unrealistic signal cycles.

3.2.6. Storage Constraints and Spillback

Finite queue storage is explicitly incorporated into the traffic environment. For every movement m,

$$0\le q_m\left(k\right)\le K_m.$$

(46)

When the number of vehicles that would otherwise occupy a movement exceeds its available storage, the excess is treated as blocked demand:

$$B_m\left(k\right)=\max \left[0,q_m\left(k\right)+A_m\left(k\right)-D_m\left(k\right)-K_m\right].$$

(47)

The total blocked demand at the intersection is

$$B\left(k\right)=\sum _{m\in \mathcal{M}}{B}_m\left(k\right).$$

(48)

The storage constraint prevents unrealistic unbounded queue growth and allows the environment to represent congestion propagation and potential spillback when traffic demand exceeds available discharge capacity.

3.2.7. Heavy-Traffic Characterization

The proposed formulation retains the heavy-traffic concept of the queueing-based approach while replacing the single-server representation with a movement-specific demand-to-capacity formulation. The traffic intensity associated with movement m is defined as

$${\rho }_m\left(k\right)={\displaystyle \frac{{\lambda }_m^{\mathrm{eff}}\left(k\right)}{{\mu }_m^{\mathrm{eff}}\left(k\right)+ϵ}},$$

(49)

where $ϵ$ is a small positive constant introduced for numerical stability. When ${\rho }_m\left(k\right)<1$, the available discharge capacity is sufficient to serve the expected traffic demand under the current operating conditions. As ${\rho }_m\left(k\right)$ approaches or exceeds unity, the arrival demand approaches or exceeds the available discharge capacity, resulting in increasing queue accumulation. This condition is particularly important in evaluating the ability of the proposed controller to maintain stable traffic operations under heavy and oversaturated demand. An intersection-level traffic-pressure measure is defined as

$${\rho }_{\mathrm{int}}\left(k\right)=\underset{m\in \mathcal{M}}{\max }{\rho }_m\left(k\right).$$

(50)

This measure is used to characterize the severity of congestion and to evaluate the robustness of the proposed controller under increasing traffic demand.

3.2.8. Traffic-State Representation

The RL-agent observes a state vector containing the traffic and signal variables required to characterize the current operating condition. The state is represented as

$$s\left(k\right)=\left[\mathbf{q}\left(k\right),{\mathit{\lambda }}^{\mathrm{eff}}\left(k\right),\mathbf{d}\left(k\right),p\left(k\right),\mathbf{g}\left(k\right),\mathbf{w}\left(k\right)\right],$$

(51)

where $\mathbf{q}\left(k\right)$ represents movement-specific queue lengths, ${\mathit{\lambda }}^{\mathrm{eff}}\left(k\right)$ represents effective arrival rates, $\mathbf{d}\left(k\right)$ represents movement discharge rates, $p\left(k\right)$ is the current signal phase, $\mathbf{g}\left(k\right)$ represents the current signal-timing parameters, and $\mathbf{w}\left(k\right)$ represents waiting-time information. The movement-level state representation enables the controller to identify queue imbalance across approaches and adapt the signal operation according to traffic demand, queue occupancy, and current signal status.

3.2.9. P-DQN-Based Discrete-Phase Selection

The P-DQN architecture is employed to address the hybrid discrete–continuous action space. The discrete action network estimates the value of each candidate signal phase, while the continuous parameter network produces the corresponding green-time parameter. For each candidate phase p, the continuous parameter is obtained as

$${\eta }_p\left(k\right)=g_{\min }^p+{\pi }_{{\theta }_p}\left(s\left(k\right)\right)\left(g_{\max }^p-g_{\min }^p\right),$$

(52)

where ${\pi }_{{\theta }_p}$ represents the continuous parameter network. The discrete phase is selected using

$$p\left(k\right)=\arg \underset{p\in \mathcal{P}}{\max }{Q}_{{\theta }_Q}\left(s\left(k\right),p,{\eta }_p\left(k\right)\right),$$

(53)

where $Q_{{\theta }_Q}$ denotes the P-DQN action-value function. The final hybrid action is therefore

$$a\left(k\right)=\left(p\left(k\right),{\eta }_{p\left(k\right)}\left(k\right)\right).$$

(54)

The selected action is subsequently passed to the traffic environment, where it determines the effective green time and movement discharge capacity.

3.2.10. TD3 Optimization of Continuous Green Time

The TD3 component optimizes the continuous green-time parameter; thus, two independent critic networks are employed to reduce overestimation bias. The target value is calculated as

$$y\left(k\right)=r\left(k\right)+\gamma \underset{j\in \{1,2\}}{\min }{Q}_{{\theta }_{C_j}^{-}}\left(s(k+1),\tilde{a}(k+1)\right),$$

(55)

where ${\theta }_{C_j}^{-}$ denotes the parameters of the target critic networks. Target-policy smoothing is introduced using clipped Gaussian noise:

$${ϵ}_k\sim clip\left(\mathcal{N}(0,{\sigma }^2),-c,c\right).$$

(56)

The smoothed target action is then expressed as

$$\tilde{a}(k+1)=clip\left[{\pi }_{{\theta }_{\pi }^{-}}\left(s(k+1)\right)+{ϵ}_k,g_{\min },g_{\max }\right].$$

(57)

The two critic networks are trained by minimizing

$${\mathcal{L}}_{C_j}={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\left(Q_{{\theta }_{C_j}}(s,a)-y\right)}^2\right],j\in \{1,2\}.$$

(58)

The actor is updated less frequently than the critics according to the delayed policy-update mechanism of TD3. The target networks are updated using

$${\theta }^{-}\leftarrow \tau \theta +(1-\tau ){\theta }^{-},$$

(59)

where $\tau$ is the soft-update coefficient.

3.2.11. Hindsight Experience Replay in State-Dependent Queue-Based Traffic Modeling

Traffic control can involve sparse or difficult-to-achieve objectives, particularly when the network is heavily congested. HER is therefore incorporated to improve experience utilization and learning efficiency. Let g denote the desired traffic-control goal and let $g_a\left(k\right)$ denote the achieved goal at decision interval k. The achieved goal is represented using traffic performance variables:

$$g_a\left(k\right)=\left[W_{\mathrm{avg}}\left(k\right),Q\left(k\right),B\left(k\right)\right],$$

(60)

where $W_{\mathrm{avg}}\left(k\right)$ is the average vehicle waiting time. For a trajectory that does not achieve its original goal, HER selects a future achieved state and replaces the original goal with

$$g^{\prime }=g_a(k+\Delta k),\Delta k>0.$$

(61)

The corresponding reward is recalculated as

$$r^{HER}\left(k\right)=R(s\left(k\right),a\left(k\right),g^{\prime }).$$

(62)

The relabeled transition is subsequently sampled from the experience replay buffer for policy learning. This enables experiences from unsuccessful traffic-control episodes to contribute to learning by treating subsequently achieved traffic states as alternative goals.

3.2.12. Multi-Objective Traffic-Control Reward

The reward function is designed to simultaneously discourage queue accumulation, reduce waiting time, maximize vehicle discharge, and penalize blocked demand. The instantaneous reward is defined in [49] as follows:

$$r\left(k\right)=-w_q\sum _{m\in \mathcal{M}}{\displaystyle \frac{q_m\left(k\right)}{K_m}}-w_w{W}_{\mathrm{avg}}\left(k\right)+w_d\sum _{m\in \mathcal{M}}{D}_m\left(k\right)-w_{b}B\left(k\right),$$

(63)

where $w_q$, $w_w$, $w_d$, and $w_b$ are weighting coefficients. The first term penalizes queue occupancy, the second penalizes vehicle waiting time, the third rewards vehicle discharge, and the fourth penalizes storage violations and blocked demand. This reward formulation therefore links the learning objective directly to the physical traffic performance of the intersection.

3.2.13. Integrated Closed-Loop Traffic Environment

The proposed P-DQN+HER+TD3 framework operates within a closed-loop traffic environment. At each decision interval, the agent observes the current traffic state and selects a hybrid action consisting of a discrete phase and a continuous green duration. The environment then translates this action into traffic-flow consequences through the phase-dependent saturation-flow and queue-conservation models. The complete interaction can be summarized as

$$s\left(k\right)\to \left[p\left(k\right),g_{p\left(k\right)}\left(k\right)\right]\to g_{p\left(k\right)}^{\mathrm{eff}}\left(k\right)\to {\mu }_m^{\mathrm{eff}}\left(k\right)\to D_m\left(k\right)\to q_m(k+1)\to s(k+1).$$

(64)

Thus, the RL action does not directly alter the original NGSIM trajectories. Instead, the action changes the signal operating conditions within the constructed environment, which in turn determines vehicle discharge, queue evolution, and the next traffic state. This closed-loop formulation provides the interactive mechanism required for reinforcement learning and establishes a clear distinction between the empirical NGSIM observations and the simulation environment used to evaluate alternative signal-control policies.

3.3. Methodological Flow Diagram of P-DQN+HER+TD3 for Traffic-Signal Control Using Lankershim and Peachtree Datasets

Figure 2 summarizes the analysis methodology used in the study. The process begins with the Lankershim and Peachtree traffic datasets, which are cleaned, filtered, transformed into PCU-based measures, and normalized to obtain the RL-ready traffic states. The processed data are then used to establish the state-dependent traffic environment and network configuration. The proposed P-DQN+HER+TD3 framework subsequently selects signal phases and continuous green-time parameters, interacts with the traffic environment, and updates its policy using replayed and hindsight-relabeled experiences. Finally, the learned control strategy is evaluated using average delay, average queue, throughput, and computation time, and its performance is compared with MaxPressure, PressLight, FRAP, and Fixed-Time using the Welch two-sample t-test. This workflow clearly distinguishes the roles of the datasets and shows how data preparation, model development, learning, and performance evaluation are connected.

Figure 2. Workflow of P-DQN+HER+TD3 for TSC on the Lankershim and Peachtree Datasets.

3.4. Comparative Analysis of Methodological Novelty

Table 3 summarizes the methodological components that distinguish the proposed framework from the individual learning approaches. In particular, the proposed P-DQN+HER+TD3 framework combines discrete–continuous (Disc.-Cont) action selection with TD3-based continuous optimization (TD3.-Opt) and HER-based goal relabeling (Goal Rel). Unlike conventional HER formulations that use generic achieved states, the proposed implementation defines the achieved traffic-control goal using average waiting time, queue length, and blocked vehicles, thereby aligning experience relabeling with traffic-specific performance objectives. This integration enables previously unsuccessful traffic-control experiences to be reused through future achieved traffic states while maintaining stable continuous-control learning.

Table 3. Comparative analysis of methodological features for reinforcement learning-based traffic signal control. Here, ✓ indicates that the feature is fully incorporated or supported, × indicates that the feature is not incorporated, and ∼ indicates partial or indirect support. Bold font highlights the proposed integrated P-DQN+HER+TD3 approach.

Method	Disc.-Cont.	TD3.-Opt	Goal Rel.	Traffic HER	Integrated
P-DQN	✓	×	×	×	×
TD3	∼	✓	×	×	×
HER	×	×	✓	∼	×
P-DQN + HER	✓	×	✓	✓	∼
P-DQN+HER+TD3	✓	✓	✓	✓	✓

4. Experimental Setup up for Synthetic and Real-World NGSIM Datasets

This section presents the experimental configuration established to evaluate the performance of the proposed P-DQN+HER+TD3 framework for adaptive TSC. To assess the reinforcement learning approach under diverse traffic-demand levels and urban conditions, the experimental design combines empirical traffic data, analytical traffic formulations, and generated synthetic data scenarios. The framework integrates two distinct data layers to build a comprehensive, high-fidelity testing environment:

• Generated Synthetic Data: To bridge the gap between historical observations and mathematical queue models, a synthetic data generator is employed. This source creates fully customized traffic scenarios, including edge-case demand spikes, extreme weather-induced slowdowns, and asymmetric lane arrivals. Generating these synthetic data enables the environment to simulate highly volatile or rare traffic patterns that are absent from the historical NGSIM data but critical for stress-testing policy resilience.
• Empirical Data (NGSIM): Raw vehicle trajectory data from the Lankershim Boulevard and Peachtree Street sites under the Next Generation Simulation (NGSIM) program provide realistic baselines for vehicle movements, arrival demand, and intersection behavior. Because these historical datasets are observational, they are used exclusively to extract traffic characteristics and calibrate the environment, rather than serving as interactive control environments (Source accessed on 20 February 2026: https://catalog.data.gov/dataset/next-generation-simulation-ngsim-vehicle-trajectories-and-supporting-data)

The National Institute of Standards and Technology NGSIM trajectory data are used as empirical sources of vehicle-movement information in this study. The preprocessing and simulation workflow translates these data sources into an interactive RL environment, as follows:

• State Calibration: Preprocessed NGSIM trajectory information and generated synthetic data streams are transformed into traffic-state variables—including vehicle accumulation, queue conditions, and flow characteristics—to parameterize the intersection environments.
• Closed-Loop Control: Within the interactive environment, the P-DQN+HER+TD3 agent and baseline controllers select actions consisting of discrete-phase decisions and continuous timing parameters. The environment updates to the next traffic state and returns a reward signal for closed-loop policy learning.
• Evaluation Integrity: Control actions do not alter the historical NGSIM trajectories. Instead, NGSIM data set the realistic boundaries of the environment, while the queueing model and synthetic data test policy resilience under stress.

4.1. NGSIM Lankershim and Peachtree Dataset Preprocessing and Environment Calibration

The NGSIM program preprocessing workflow of Figure 3 prepares raw vehicle trajectory data for reinforcement learning-based TSC. The extracted trajectory information is used to characterize and calibrate the data-driven intersection-control environments developed for reinforcement learning. The interactive environments allow the proposed P-DQN+HER+TD3 controller and the baseline methods to select alternative signal phases and timing parameters and to evaluate their resulting control policies. Consequently, the original NGSIM trajectories are not modified in response to RL actions; rather, the empirical traffic characteristics obtained from the datasets provide the basis for the interactive evaluation environments. The flow diagram in Figure 4 represents intersections as nodes and road links as directed edges. The edge width indicates traffic volume and color reflects congestion (green = low, yellow = moderate, red = high). Each diagram models traffic entering a central intersection from four different directions (North, South, East, and West), where lane blocks represent incoming approaches and directed arrows indicate vehicle movement toward the intersection. The Lankershim-based diagram reflects a dominant North–South arterial flow, characteristic of heavily loaded main corridors, whereas the Peachtree-based diagram exhibits a more balanced distribution of traffic across all approaches. These visualizations provide an intuitive representation of traffic-demand patterns and congestion behavior, supporting analysis and evaluation of intelligent traffic-control strategies. Table 4 summarizes the hyperparameters, neural network architectures, and Python packages used for PressLight, FRAP, and the proposed Hybrid (P-DQN+HER+TD3) models. It includes RL settings (state/action dimensions, replay buffer, learning rates, exploration, TD3 components) and network details (layer sizes, activations, outputs). The listed Python libraries ensure reproducibility. Figure 5 illustrates the proposed framework integrating the NGSIM trajectory dataset with a hybrid P-DQN architecture for intelligent intersection control. Raw vehicle trajectory data, including positions, speeds, accelerations, and lane identifiers, are first preprocessed through temporal aggregation and spatial mapping onto the intersection layout. Specifically, vehicle positions are projected onto predefined lane segments to determine lane occupancy, from which queue lengths are computed as the number of vehicles with speeds below a threshold $v_{\mathrm{th}}$. Traffic density is estimated as the ratio of occupied space to total lane length, while waiting time is accumulated for each vehicle when its speed falls below $v_{\mathrm{th}}$ over consecutive time steps. These features are further aggregated over a fixed time window to reduce noise and capture short-term traffic dynamics. The extracted features are then used to construct the MDP state $s_t$, which includes queue length vectors per approach, current signal phase encoding, cumulative delay, and estimated traffic density. The control module consists of a hybrid reinforcement learning controller combining P-DQN for discrete traffic phase selection and TD3-based actor–critic learning for continuous green-time optimization. An experience replay buffer enhanced with HER improves sample efficiency and goal-conditioned learning. The selected phase and optimized duration are applied through the signal controller interface, and system performance is evaluated using metrics such as average waiting time, travel time, throughput, and delay. The resulting reward $r_t$ is fed back to the learning agent, forming a closed-loop training framework for adaptive and data-driven TSC. The complete training and decision-making process of the proposed P-DQNPC scheme is formalized in Algorithm 1, which summarizes the interactions within the traffic environment. This pseudocode jointly optimizes TSC across multiple intersections using a hybrid RL framework. For each intersection, the current traffic state is observed and a hybrid action is selected, consisting of a discrete-phase choice (via P-DQN) and a continuous-phase duration (via TD3 with exploration noise). The environment is then updated, and the resulting reward and next state are stored in a replay buffer. The P-DQN critic is updated to evaluate discrete phase actions, while TD3 critics and actor are updated to optimize continuous-phase durations. Networks are periodically soft-updated to stabilize learning. This process is repeated across all intersections and episodes, resulting in a learned hybrid policy that coordinates both phase selection and timing across the traffic network.

Figure 3. NGSIM traffic dataset preprocessing flow chart. Accessed on 20 February 2026: (https://catalog.data.gov/dataset/next-generation-simulation-ngsim-vehicle-trajectories-and-supporting-data).

Figure 3. NGSIM traffic dataset preprocessing flow chart. Accessed on 20 February 2026: (https://catalog.data.gov/dataset/next-generation-simulation-ngsim-vehicle-trajectories-and-supporting-data).

Figure 4. Traffic-flow diagrams for Lankershim and Peachtree showing directional movement.

Figure 4. Traffic-flow diagrams for Lankershim and Peachtree showing directional movement.

Table 4. Hyperparameters and Python packages used for PressLight, FRAP, and Hybrid (P-DQN+HER+TD3).

Parameter/Component	Value/Structure	Description
Reinforcement Learning Configuration
State dimension	4	State observation vehicle count, average speed, aggregated queue length, and traffic density.
State vector	${\mathbf{s}}_t={[n_t,{\overline{v}}_t,q_t,{\rho }_t]}^{\mathrm{T}}$	RL observation supplied to the controller at decision time t.
Discrete actions	4	Available traffic-signal phases.
Continuous parameter dimension	1	Continuous adjustment of the selected green-time duration.
Replay buffer	50,000	Experience memory capacity.
Batch size	64	Number of transitions sampled per training update.
Discount factor ($\gamma$)	0.99	Weight assigned to future rewards.
Learning rate	0.001	Adam optimizer learning rate.
Exploration ($ϵ$)	1.0 → 0.05	Initial exploration probability with decay factor 0.995.
TD3 parameters	$\tau =0.005$, noise = 0.2, clip = 0.5, delay = 2	Target update, policy noise, noise clipping, and delayed policy update.
Training	200 episodes × 200 steps	Maximum training steps per episode.
Random seeds	5	Independent experimental runs for reproducibility.
Reward clipping	[−5,5]	Constrains reward magnitude to improve training stability.
HER	Enabled	Hindsight Experience Replay for experience augmentation.
Neural Network Architectures
PressLight Q-network	$4\to 128\to 64\to 4$	Q-values associated with the four signal phases.
FRAP	$4\to 128\to 128\to 4$	Shared representation, phase competition, and output layers.
Hybrid discrete Q-network	$4\to 128\to 64\to 4$	Selects the discrete traffic-signal phase.
Hybrid actor	$4\to 128\to 64\to 1$	Generates the continuous green-time adjustment.
Hybrid critic	$(4+1)\to 128\to 64\to 1$	Evaluates state-action values using twin critics.
Activation	ReLU/Tanh	Hidden-layer and continuous-action output activations.
Loss	MSE	Mean squared error used for value-function learning.
Python Packages and Computing Environment
Python	3.11	Programming language.
PyTorch	2.1.0	Actor, critic, and Q-network implementation.
NumPy	1.26.0	Numerical and matrix computations.
Pandas	2.1.0	Data loading and preprocessing.
Matplotlib	3.8.0	Visualization of rewards and traffic metrics.
SciPy	1.11.0	Scientific computing utilities.
Collections (deque)	Built-in Python	Replay-buffer implementation.
CUDA	Optional GPU	Accelerates neural-network training.

Note: All experiments were run on GPU when available; otherwise CPU is used. Random seeds ensure reproducibility of results.

Table 4. Hyperparameters and Python packages used for PressLight, FRAP, and Hybrid (P-DQN+HER+TD3).

Parameter/Component	Value/Structure	Description
Reinforcement Learning Configuration
State dimension	4	State observation vehicle count, average speed, aggregated queue length, and traffic density.
State vector	${\mathbf{s}}_t={[n_t,{\overline{v}}_t,q_t,{\rho }_t]}^{\mathrm{T}}$	RL observation supplied to the controller at decision time t.
Discrete actions	4	Available traffic-signal phases.
Continuous parameter dimension	1	Continuous adjustment of the selected green-time duration.
Replay buffer	50,000	Experience memory capacity.
Batch size	64	Number of transitions sampled per training update.
Discount factor ($\gamma$)	0.99	Weight assigned to future rewards.
Learning rate	0.001	Adam optimizer learning rate.
Exploration ($ϵ$)	1.0 → 0.05	Initial exploration probability with decay factor 0.995.
TD3 parameters	$\tau =0.005$, noise = 0.2, clip = 0.5, delay = 2	Target update, policy noise, noise clipping, and delayed policy update.
Training	200 episodes × 200 steps	Maximum training steps per episode.
Random seeds	5	Independent experimental runs for reproducibility.
Reward clipping	[−5,5]	Constrains reward magnitude to improve training stability.
HER	Enabled	Hindsight Experience Replay for experience augmentation.
Neural Network Architectures
PressLight Q-network	$4\to 128\to 64\to 4$	Q-values associated with the four signal phases.
FRAP	$4\to 128\to 128\to 4$	Shared representation, phase competition, and output layers.
Hybrid discrete Q-network	$4\to 128\to 64\to 4$	Selects the discrete traffic-signal phase.
Hybrid actor	$4\to 128\to 64\to 1$	Generates the continuous green-time adjustment.
Hybrid critic	$(4+1)\to 128\to 64\to 1$	Evaluates state-action values using twin critics.
Activation	ReLU/Tanh	Hidden-layer and continuous-action output activations.
Loss	MSE	Mean squared error used for value-function learning.
Python Packages and Computing Environment
Python	3.11	Programming language.
PyTorch	2.1.0	Actor, critic, and Q-network implementation.
NumPy	1.26.0	Numerical and matrix computations.
Pandas	2.1.0	Data loading and preprocessing.
Matplotlib	3.8.0	Visualization of rewards and traffic metrics.
SciPy	1.11.0	Scientific computing utilities.
Collections (deque)	Built-in Python	Replay-buffer implementation.
CUDA	Optional GPU	Accelerates neural-network training.

Note: All experiments were run on GPU when available; otherwise CPU is used. Random seeds ensure reproducibility of results.

Figure 5. NGSIM-based hybrid parameterized reinforcement learning framework for intersection control.

Figure 5. NGSIM-based hybrid parameterized reinforcement learning framework for intersection control.

Algorithm 1 Proposed P-DQN+HER+TD3 Learning Algorithm

4.2. Preprocessing of the Peachtree Trajectory Dataset via State-Dependent Urban Traffic Queue Modeling

The operational sequence of the proposed framework, as detailed in Algorithm 2, proceeds through offline dataset calibration as illustrated in Section 4.1 followed by a hybrid reinforcement learning loop. Initially, raw vehicle trajectories from the Peachtree dataset are processed to extract empirical arrival rates and turning behaviors, which are then used to calibrate the mathematical parameters of the state-dependent urban traffic queue model. During each step of the online training loop, the framework handles a complex hybrid action space by utilizing P-DQN to select the discrete-signal phase configuration, while the TD3 continuous actor computes the exact green-time duration allocated to that phase. This combined action is integrated directly into the traffic environment, which evaluates vehicle movements through movement-specific queue conservation, phase-dependent saturation flows, lost times, and storage constraints. Based on these physical network boundaries, the queue environment updates traffic-flow metrics and generates a performance reward penalizing delays and queue growth. To maximize sample efficiency under sparse traffic conditions, HER dynamically re-labels failed trajectories with virtual, achieved goals to compute alternative reward feedback, while the network weights are optimized using TD3’s twin critics and a delayed actor update mechanism to ensure smooth, non-oscillatory convergence toward a stable, adaptive signal-control policy.

Algorithm 2 P-DQN+HER+TD3 ATSC via State-Dependent Queue Modeling

5. Experimental Results and Findings

This section evaluates the proposed P-DQN+HER+TD3 hybrid framework, examining its effectiveness, stability, and adaptability for TSC in both synthetic environments and real-world arterial intersection scenarios based on the NGSIM dataset. The performance of the proposed method is compared with baseline approaches under realistic traffic conditions. NGSIM data are chosen for their high-resolution vehicle trajectories, which allow for realistic modeling of traffic dynamics through detailed speed, acceleration, position and lane occupancy information. Using this dataset allows the reinforcement learning frameworks to interact with traffic patterns that closely resemble real-world driving behavior. The experiments assess the convergence, effectiveness, and adaptability of the proposed P-DQN+HER+TD3 scheme in reducing congestion and delays across both simulated and realistic traffic enviroments.

5.1. Experimental Results for Synthetic (Generated) Data

This section presents a comprehensive comparative performance analysis of the proposed P-DQN+HER+TD3 model, to justify how it surpasses the limitations of leading discrete baseline models. Experimental results are analysed to evaluate performance across multiple key metrics, including cumulative reward, queue length, throughput, and average delay. Figure 6 compares the performance of the proposed model against state-of-the-art FRAP and PressLight RL-schemes over 50 training episodes using cumulative rewards as a perfomance metric. Since rewards are negative, values closer to zero indicate superior performance and learning efficiency. A critical observation is that the figure exhibits similar behavior during the initial 10 episodes, followed by clear divergence. The P-DQN+TD3+HER model achieved the best performance with the highest final reward (≈−60) and fastest convergence. FRAP attains moderate performance (≈−120), while PressLight shows the slowest learning and lowest final reward (≈−170).

Figure 6. Convergence speed and sample efficiency.

PressLight and FRAP struggle with sparse rewards during early training; however, the HER mechanism in the proposed model facilitates reaching the same final performance as PressLight with 30% fewer episodes, demonstrating faster learning. Furthermore, HER enhanced sample efficiency by leveraging virtual goals, resulting in a 22% higher cumulative reward by the end of the 50-episode cycle. The P-DQN+TD3+HER model demonstrated a significant improvement in learning acceleration with consistently higher cumulative rewards. Figure 7 evaluates traffic optimization performance by comparing the average queue length over 50 training episodes, where lower values indicate improved efficiency and reduced congestion. Initially, all models exhibited high congestion levels (17–21 vehicles), but the P-DQN+TD3+HER approach demonstrated superior convergence speed, stability, and overall traffic efficiency. The proposed model achieved the best performance, reducing average queue length by approximately 15% and 25% relative to the FRAP and PressLight, respectively, converging to around six vehicles. This improvement is attributed to its ability to optimize continuous-phase durations, enabling adaptive extension and reduction of green times, thereby minimizing idle periods and vehicle waiting times. FRAP demonstrated competitive performance but with minor late-stage fluctuations, converging to approximately seven vehicles. In contrast, PressLight showed the weakest performance, characterized by a prolonged learning plateau and the highest final queue length of about 11 vehicles. Figure 8 evaluates traffic efficiency in terms of average travel time over 50 training episodes, where lower values indicate improved throughput and reduced intersection delay. All models initially exprienced high travel times (56–64 s), but the proposed P-DQN+TD3+HER approach demonstrated superior stability, convergence speed, and overall traffic-flow efficiency. The proposed model achieved the best performance, reducing average travel time by 19.2% compared to the FRAP baseline and converging within 40 s. This improvement is attributed to its ability to optimize continuous-phase durations, effectively avoiding premature signal switching and ensuring efficient clearance of high-density vehicle platoons. FRAP demonstrated moderate performance but exhibited a significant mid-training instability, including a temporary delay spike, before converging to about 45.5 s. In contrast, PressLight showed the weakest performance, characterized by an early optimization plateau and the highest final travel time of approximately 48 s. Figure 9 evaluates intersection performance using average vehicle delay over 50 training episodes, where lower values indicate improved traffic flow and reduced commuter wait times. Initially, all models exhibited high delay levels (22–29 s); however, the proposed approach showed superior convergence speed, stability, and optimization capability, resulting in significantly improved intersection efficiency and smoother traffic flow. The P-DQN+TD3+HER model achieved the best overall performance, recording a 24% reduction in average delay and converging to approximately 7 s. This improvement is attributed to its ability to learn the non-linear relationship between signal timing and vehicle discharge rates within the first 13 episodes, enabling precise temporal control of signal transitions and more efficient traffic clearance. FRAP demonstrated competitive performance, particularly in phase competition, but lacked fine-grained temporal precision, ultimately converging to around 10 s after a late-stage improvement. In contrast, PressLight, while effective at balancing lane pressure, showed a limited adaptability under dynamic traffic conditions, leading to performance instability and the highest final delay of approximately 14 s following a mid-training regression. The summary Table 5 presents a comparative performance analysis of the proposed P-DQN + TD3 + HER scheme against two state-of-the-art baselines, PressLight and FRAP. Across all six traffic indicators, the proposed model demonstrated substantial improvements. The model obtained 26.0% reduction in queue length and a 21.4% reduction in waiting time compared to PressLight. These gains are primarily driven by the transition from fixed-step phase durations to continuous, real-time duration tuning. Throughput is increased by up to 14.2%, while average delay and travel times are reduced by 24.1% and 19.2%, respectively. This highlights the model’s ability to maintain higher vehicle discharge rates during peak surges. The significant improvement in total reward (+28.4% vs. PressLight) confirms that the integration of TD3’s twin-critic stability and HER’s goal-relabeling effectively addresses the sparse reward and overestimation challenges inherent in complex urban traffic environments.

Figure 7. Average queue length.

Figure 8. Commuter experience (travel time).

Figure 9. Network average delay (flow rate).

Table 5. Performance gains of P-DQN + TD3 + HER compared to baseline models (last 10 episodes average).

Metric	vs. PressLight (%)	vs. FRAP (%)	Key Driver
Total Reward	$+28.4\%$	$+15.2\%$	TD3 Stability
Queue Length	$-26.0\%$	$-18.5\%$	Continuous Duration
Throughput	$+14.2\%$	$+9.1\%$	Dynamic Timing
Avg. Wait Time	$-21.4\%$	$-12.8\%$	Precision Switching
Avg. Delay	$-24.1\%$	$-16.5\%$	HER Sample Efficiency
Travel Time	$-19.2\%$	$-11.0\%$	Adaptive Flow

5.2. Experimental Results for NGSIM (Lankershim Boulevard and Peachtree Street) Trajectory Datasets

This section examines the performance evolution of P-DQN+HER+TD3, PressLight, and FRAP across the Lankershim and Peachtree datasets, using the key traffic metrics of average delay, throughput, queue length, and computation time. They are tracked over training episodes to highlight learning progression and model stability. The analysis enables a clear comparison of effectiveness across both datasets and all models.

5.2.1. Model Performance Across Episodes for Lankershim and Peachtree Datasets

This subsection presents a comprehensive evaluation of the performance evolution of the proposed P-DQN+HER+TD3 traffic-control framework in comparison with state-of-the-art baseline models, namely, PressLight and FRAP. The analysis is conducted across two representative urban traffic datasets, Lankershim and Peachtree, to assess the effectiveness, robustness, and generalization capability of the proposed approach under diverse traffic conditions. The trajectory datasets provide empirical vehicle-movement information used to characterize traffic conditions and calibrate the evaluation environments. Since the original datasets contain historically observed trajectories and do not provide a closed-loop mechanism through which alternative signal phases and green durations modify subsequent vehicle trajectories, the interactive evaluation of the proposed and baseline controllers is performed within the developed RL environments using traffic characteristics derived from the datasets. The graphs track a specific traffic metric, including average vehicle delay, throughput, queue length, and computation time, over evaluation episodes. Line styles and markers differentiate both models and datasets, enabling clear comparison of learning progression and stability. As illustrated in Figure 10, the proposed approach demonstrates stable policy improvement after approximately 100 episodes, establishing a steady convergence plateau near a positive cumulative reward of $+125$. Crucially, P-DQN+HER enters a distinct plateau phase past 350 episodes, establishing clear convergence and demonstrating that the model has fully maximized its potential gain. In contrast, PressLight fails to optimize within this environment, rapidly descending and stabilizing at a highly suboptimal negative reward region near $-145$. Because PressLight does not converge to a functional policy, a direct speed-ratio comparison is less indicative of performance than the absolute gap in policy efficacy. This performance disparity stems from structural action-space limitations: while PressLight is constrained by its discrete-phase-switching architecture, the proposed approach uses a hybrid action design that enables simultaneous, continuous longitudinal and lateral trajectory control. Furthermore, the integration of HER effectively mitigates the sparse reward problem. This prevents the prolonged stagnation in negative reward regions that characterizes both baselines, enabling the ego-vehicle to discover successful trajectories early in the training phase. Conversely, FRAP exhibits continuous oscillations without establishing a clear steady-state plateau within the training window. This indicates that its phase-competition architecture is less suited for microscopic, single-vehicle trajectory tasks compared to the goal-conditioned HER approach. Figure 11 presents the average delay performance of the proposed P-DQN+HER+TD3 framework compared with PressLight and FRAP in the data-driven intersection-control environments developed using the Lankershim and Peachtree trajectory datasets. For the Lankershim-derived environment, the proposed P-DQN+HER+TD3 framework achieved an average delay of 4.79 s, compared with 14.82 s for PressLight and 15.18 s for FRAP. This represents reductions in average delay of 67.68% and 68.45% relative to PressLight and FRAP, respectively. A similar performance advantage is observed in the Peachtree-derived environment, where the proposed framework achieved an average delay of 7.00 s, compared with 17.90 s for PressLight and 18.45 s for FRAP. These results correspond to reductions of 60.89% and 62.06%, respectively. The learning behavior shown in Figure 11 further indicates that the proposed framework progressively reduced average delay during training and converged towards lower-delay control policies. In contrast, PressLight and FRAP exhibited comparatively higher and more variable delay values over the evaluated training period. The consistent performance of the proposed framework across both dataset-derived environments suggests that the combination of parameterized action selection, HER, and TD3 enables more effective learning of hybrid traffic-control policies. In particular, the P-DQN component provides discrete signal-phase selection, while the continuous control component enables the optimization of signal-timing parameters. HER further improves the utilization of unsuccessful experiences by reformulating them with alternative goals, while TD3 improves the stability of continuous-action learning. Overall, the results demonstrate that the proposed P-DQN+HER+TD3 framework achieves substantially lower average delay than the selected baseline methods under the traffic conditions represented by the Lankershim- and Peachtree-derived environments. These findings provide evidence of the effectiveness of the proposed approach for data-driven adaptive TSC. Figure 12 presents the traffic flow rate, expressed in vehicles per hour (veh/h), obtained from the evaluation of the proposed P-DQN+HER+TD3 framework and the baseline methods within the data-driven intersection-control environments derived from the Lankershim and Peachtree trajectory datasets. The proposed framework consistently achieved higher traffic-flow rates than PressLight and FRAP in both dataset-derived environments. In the Lankershim-derived environment, the proposed method achieved a flow rate of 1802.50 veh/h, representing improvements of 12.83% and 13.54% over PressLight and FRAP, respectively. Similarly, in the Peachtree-derived environment, the proposed framework achieved 1429.08 veh/h, corresponding to improvements of 10.46% over PressLight and 10.56% over FRAP. These results indicate that the proposed controller can achieve higher traffic throughput under the traffic conditions represented by the two empirical datasets. The learning trajectories in Figure 12 further show that the proposed framework progressively increased the traffic-flow rate during training, indicating improved policy learning and more effective traffic clearance. In comparison, the baseline methods exhibited relatively limited improvement, with their flow rates remaining within lower performance ranges. PressLight and FRAP stabilized at approximately 1600 veh/h in the Lankershim-derived environment and between 1250 and 1300 veh/h in the Peachtree-derived environment. The comparatively higher flow rates achieved by the proposed framework suggest that its hybrid discrete–continuous action formulation enables more effective coordination of signal-phase selection and signal-timing decisions. Overall, the proposed P-DQN+HER+TD3 framework improved the evaluated traffic-flow rate by approximately 10–14% relative to the considered baseline methods across the two dataset-derived environments. The results demonstrate the potential of combining parameterized action selection, HER, and TD3 for adaptive TSC. Nevertheless, because the Lankershim and Peachtree datasets contain historically observed vehicle trajectories rather than closed-loop responses to alternative signal policies, these findings should be interpreted as evidence from NGSIM-derived interactive evaluation environments rather than direct real-world deployment results. Figure 13 shows the average queue length obtained from the evaluation of the proposed model and the baseline methods within the data-driven intersection-control environments derived from the Lankershim and Peachtree trajectory datasets. For the Lankershim-derived environment, the proposed framework achieved an average queue length of 2.43 vehicles, compared with 6.12 vehicles for PressLight and 5.97 vehicles for FRAP. This corresponds to reductions of approximately 60.29% and 59.30% relative to PressLight and FRAP, respectively. Similarly, in the Peachtree-derived environment, the proposed framework achieved an average queue length of 2.79 vehicles, compared with 7.52 and 7.58 vehicles for PressLight and FRAP, respectively. These results represent reductions of approximately 62.90% and 63.19%, respectively. The learning trajectories in Figure 13 further indicate that the proposed framework progressively reduced the average queue length during training, suggesting improved policy learning and more effective regulation of vehicle accumulation at the intersection. In contrast, the baseline methods exhibited comparatively limited improvement, with queue lengths remaining at higher levels over the evaluated training period. The lower queue lengths achieved by the proposed framework can be attributed to its hybrid action formulation, in which P-DQN enables discrete signal-phase selection while the continuous action component optimizes signal-timing parameters. The incorporation of HER improves the utilization of experience during learning, while TD3 supports more stable optimization of the continuous control parameters. Overall, the proposed P-DQN+HER+TD3 framework achieved substantially lower average queue lengths than PressLight and FRAP across both dataset-derived evaluation environments.

Figure 10. Cumulative reward comparison: P-DQN+HER+TD3 against baseline models.

Figure 11. Average delay time comparison: P-DQN+HER+TD3 against baseline models.

Figure 12. Traffic throughput comparison: P-DQN+HER+TD3 against baseline models.

Figure 13. Traffic queue length comparison: P-DQN+HER+TD3 against baseline models.

The results demonstrate the potential of the proposed hybrid RL approach to improve queue management and traffic-flow regulation under the traffic conditions represented by the Lankershim and Peachtree datasets. However, since the underlying datasets contain historically observed trajectories rather than closed-loop responses to alternative signal-control actions, these results should be interpreted as evidence from the developed data-driven interactive environments.

5.2.2. Impact of Traffic-Flow Variations on Model Performance

Figure 14 illustrates the effect of increasing traffic demand on average delay for the proposed P-DQN+HER+TD3 framework and the baseline methods. The experiment evaluates traffic-demand scaling factors ranging from 0.6 to 1.4 within the data-driven intersection-control environments derived from the Lankershim and Peachtree trajectory datasets. The trajectory datasets provide empirical traffic characteristics for constructing the evaluation environments, while the traffic-demand scaling and subsequent interactive control experiments are performed within the developed RL environments. As shown in Figure 14, average delay generally increases across all evaluated methods as the traffic-demand scaling factor increases from 0.6 to 1.4. This behavior reflects the increasing pressure placed on the intersection as traffic demand approaches and exceeds the conditions represented by the baseline traffic scenarios. Despite the increased demand, the proposed P-DQN+HER+TD3 framework consistently maintained the lowest average delay across both dataset-derived environments. Relative to PressLight, the proposed framework achieved delay reductions of approximately 30–40%, while reductions of approximately 15–25% were observed relative to FRAP. These results indicate that the proposed controller maintains a performance advantage as traffic demand varies across the evaluated range. The proposed framework also exhibited a comparatively positive increase in average delay as traffic demand intensified. This behavior is exhibited across both the Lankershim- and Peachtree-derived environments, although the magnitude of the delay varied according to the characteristics of each traffic scenario. The difference between the proposed framework and the baseline methods is particularly evident under higher demand levels. For example, in the Peachtree-derived environment, PressLight reached a maximum delay of approximately 23.6 s at the highest evaluated traffic scale, whereas the proposed framework maintained a substantially lower delay. FRAP demonstrated intermediate performance but remained consistently above the proposed method across the evaluated demand levels. The observed results suggest that the proposed hybrid control architecture is better able to adapt its signal-phase and continuous timing decisions as traffic demand changes. The parameterized action formulation allows the controller to jointly optimize discrete phase selection and continuous signal-timing parameters, while HER and TD3 support effective learning and optimization of the resulting hybrid control policy.

Figure 14. Average traffic time delay.

In summary, the traffic-demand sensitivity analysis demonstrates that the P-DQN+HER+TD3 framework maintains its performance advantage across a range of demand conditions, rather than being limited to a single traffic scenario. However, these findings represent evaluations within the developed data-driven interactive environments; they do not provide direct evidence that the original trajectory datasets responded dynamically to the applied signal-control policies. Figure 15 illustrates the variation in traffic throughput with increasing traffic demand for the proposed P-DQN+HER+TD3 framework and the baseline methods. As shown in the figure, all evaluated methods generally exhibit an increase in throughput as the traffic-demand scaling factor increases from 0.6 to 1.4. However, the proposed P-DQN+HER+TD3 framework consistently achieves the highest throughput across the evaluated demand levels. The proposed framework provides approximately 20–30% higher throughput than PressLight and 10–18% higher throughput than FRAP, indicating its comparative effectiveness in utilizing available intersection capacity under varying traffic-demand conditions. The proposed framework also exhibits a relatively stable increase in throughput as traffic demand increases, reaching peak values of approximately 2720 veh/h in the Lankershim-derived environment and 2200 veh/h in the Peachtree-derived environment. Although the maximum throughput obtained in the Peachtree-derived environment is lower than that observed in the Lankershim-derived environment, the proposed framework maintains a consistent performance advantage over both baseline methods. This suggests that the learned hybrid control policy can adapt its signal-phase and timing decisions across traffic scenarios with different characteristics. In comparison, the baseline methods exhibit comparatively lower throughput and less pronounced improvements as traffic demand increases. FRAP demonstrates intermediate performance, whereas PressLight exhibits a greater reduction in relative performance at higher demand levels. The increasing separation between the proposed framework and the baseline methods at higher demand levels indicates that the proposed controller can make more effective use of available signal-control actions under increasing traffic pressure. Overall, the results show that the proposed P-DQN+HER+TD3 framework achieves a throughput advantage of approximately 10–30% over the considered baseline methods across the evaluated traffic-demand range. This performance is attributed to the hybrid action formulation, which combines discrete signal-phase selection with continuous signal-timing optimization, together with HER and TD3 for improved learning of the continuous control parameters. The results therefore demonstrate the potential of the proposed framework to improve traffic throughput within the evaluated data-driven interactive environments. Accordingly, the throughput results reported in Figure 15 represent the performance of the learned controllers within the developed dataset-derived interactive environments, rather than direct modifications of the original recorded trajectories. Figure 16 evaluates the response of the proposed P-DQN+HER+TD3 model to increasing traffic demand by examining the average queue length over traffic-demand scaling factors ranging from 0.6 to 1.4. As shown in Figure 16, the average queue length generally increases for all evaluated methods as traffic demand increases. This behavior reflects the increasing traffic pressure placed on the intersection at higher demand levels. The Peachtree-derived environment exhibits comparatively higher queue accumulation than the Lankershim-derived environment, indicating differences in the traffic conditions represented by the two datasets. Despite the increasing demand, the proposed P-DQN+HER+TD3 framework consistently maintains the lowest average queue length across the evaluated traffic-demand levels. Quantitatively, the proposed framework achieves reductions in average queue length of approximately 25–35% relative to PressLight and 10–20% relative to FRAP over the evaluated demand range. At the highest traffic-demand level, the proposed framework reaches peak queue lengths of approximately 4.9 veh in the Lankershim-derived environment and 6.2 veh in the Peachtree-derived environment. In comparison, FRAP exhibits intermediate performance, while PressLight experiences substantially greater queue accumulation, reaching approximately 8.6 veh at the maximum traffic-demand level. The comparatively gradual increase in queue length exhibited by the proposed framework indicates that its learned control policy remains effective as traffic demand intensifies. This behavior can be attributed to the hybrid action formulation of P-DQN+HER+TD3, which enables the controller to jointly adapt discrete signal-phase decisions and continuous signal-timing parameters. The combination of HER and TD3 further supports the learning of effective continuous control policies under varying traffic conditions. Consequently, the proposed framework demonstrates greater resilience to increasing traffic demand within the evaluated environments, maintaining lower queue accumulation and supporting more stable traffic-flow regulation than the considered baseline methods. Notably, while the Peachtree environment (represented by dashed lines) presents a higher traffic burden than Lankershim (represented by solid lines) for all configurations, the proposed model demonstrates superiority and capacity to effectively mitigate intersection congestion under both light and highly congested traffic regimes. These results provide evidence of the potential of the proposed hybrid RL architecture for adaptive TSC under increasing network load. Figure 17 illustrates the average computation time per step across varying traffic-flow scales. The computational overhead remains completely invariant to fluctuations in traffic demand for all tested models. Notably, because the computational complexity of each algorithm is structurally identical across both the Lankershim and Peachtree datasets, their respective performance curves perfectly overlap, rendering as three distinct trajectories in the plot. While the proposed P-DQN+HER+TD3 framework exhibits a slightly higher execution time (∼2.8 ms) compared to FRAP (∼1.2 ms) and PressLight (∼0.7 ms), this sub-millisecond overhead remains well within the hard real-time constraints required for physical deployment. This marginal trade-off is highly justified given the substantial gains in trajectory accuracy and overall rewards demonstrated by the hybrid model.

Figure 15. Intersection vehicle throughput.

Figure 16. Average traffic-flow queue length.

Figure 17. The aggregated commutation time for the three models.

5.2.3. Statistical Analysis

Table 6 presents the mean, standard deviation, and 95% confidence intervals for the evaluated methods over 10 independent runs ($N=10$). Pairwise differences between P-DQN+HER+TD3 and the baseline methods are evaluated using Welch’s two-sample t-test with a significance threshold of $p<0.05$. As shown, the proposed model achieved an Average Delay of only 10.80 s, which represents a substantial reduction of 27.2% compared to PressLight (14.84 s) and 15.8% compared to FRAP (12.83 s). Furthermore, it maximized intersection capacity by achieving a Throughput of 1996 pcu/h (a 45.2% increase over PressLight and 17.4% over FRAP) while maintaining the shortest Average Queue length at 3.74 pcu. P-DQN+HER+TD3 achieves the lowest averaged delay (10.80 s) and queue length (3.74 pcu), together with the highest throughput (1996 pcu/h). Relative to PressLight, the proposed method reduces average delay and queue length by 27.3% and 31.3%, respectively, while increasing throughput by 45.2%. Compared with FRAP, the corresponding improvements are 15.8%, 17.3%, and 17.4%, respectively. These results show that the proposed hybrid controller provides more effective congestion reduction and vehicle processing than both baseline methods, with the largest gains observed relative to PressLight. Because physical traffic-signal controllers operate on a scale of seconds, an execution latency of 2.8 ms is practically negligible and safely avoids any real-time deployment bottlenecks. Sacrificing less than two milliseconds of computation time to achieve drastic reductions in commuter delays and bottleneck queues presents a clear net benefit that fully justifies the architectural complexity of the P-DQN+HER+TD3 framework. Despite this increase, the computational overhead remains within practical real-time limits, suggesting that the performance gains outweigh the additional cost. Table 7 shows the percentage change in the mean performance of P-DQN+HER+TD3 relative to baselines. For metrics where lower values indicate better traffic performance, negative percentages represent reductions, whereas positive percentages represent improvements for metrics where higher values are desirable. The statistics demonstrate that P-DQN+HER+TD3 provides the strongest mean traffic-control performance among the evaluated methods, achieving lower congestion and delay and higher throughput, while maintaining relatively consistent performance across independent training runs at the expense of moderately increased computational complexity.

Table 6. Performance comparison of P-DQN+HER+TD3 against baseline methods. Results are reported as mean ± standard deviation with 95% confidence intervals.

Metric	P-DQN+HER+TD3	PressLight	FRAP
Avg. Delay (s)	10.80 ± 2.90 [9.12–12.48]	14.84 ± 3.96 [12.52–17.16]	12.83 ± 3.44 [10.82–14.84]
Throughput (pcu/h)	1996 ± 540 [1670–2322]	1375 ± 380 [1151–1599]	1700 ± 470 [1424–1976]
Avg. Queue (pcu)	3.74 ± 1.02 [3.14–4.34]	5.44 ± 1.45 [4.59–6.29]	4.52 ± 1.20 [3.81–5.23]
Comp. Time (ms)	2.8 ± 0.0 [2.8–2.8]	0.7 ± 0.0 [0.7–0.7]	1.2 ± 0.0 [1.2–1.2]

Note: Values in brackets represent the 95% confidence intervals. Pairwise statistical comparisons are performed using Welch’s two-sample t-test with $p<0.05$ as the significance threshold.

Table 7. Percentage improvement of P-DQN+HER+TD3 over baseline methods (computed from mean values).

Metric	vs. PressLight (%)	vs. FRAP (%)
Avg. Delay	−27.3	−15.8
Throughput	+45.2	+17.4
Avg. Queue	−31.3	−17.3
Comp. Time	+300.0	+133.3

5.2.4. Discussion of Experimental Results for Lankershim Boulevard and Peachtree Street Trajectory Datasets

This study evaluates the performance of the proposed P-DQN+HER+TD3 traffic-control model against PressLight and FRAP using both synthetic traffic scenarios and real-world NGSIM datasets (Lankershim and Peachtree). The results demonstrated that the proposed model consistently outperforms the baseline approaches in reducing average vehicle delay and queue lengths, while maintaining high traffic throughput across multiple training episodes. The superior performance of the P-DQN+HER+TD3 model can be attributed to three main components. First, the P-DQN mechanism improves sample efficiency by prioritizing more informative experiences during training. Second, the incorporation of HER enhanced learning in sparse and heavy traffic environments by enabling the agent to learn from goal-oriented experiences. Third, the TD3 algorithm stabilized policy updates and reduced overestimation bias, leading to more consistent convergence. Together, these components enabled the model to learn robust and adaptive TSC strategies under varying traffic conditions. Further analysis indicates that the model performs particularly well under moderate-to-high traffic-demand conditions, where congestion is more pronounced. In such scenarios, the agent effectively balances the trade-off between minimizing queue lengths and delay time and maximizing throughput. However, under low traffic-demand conditions, the performance improvement over baseline methods is less significant, suggesting that simpler control strategies may be sufficient in such cases.

5.3. Experimental Results for Urban Traffic via State-Dependent Queues (Peachtree-Based Dataset)

To evaluate the operational efficiency, scalability, and robustness of the proposed P-DQN + TD3 + HER framework, a comprehensive comparative analysis is conducted against four baseline methods—MaxPressure, PressLight, FRAP, and a conventional Fixed-Time controller. The simulation environment is parameterized using empirical traffic characteristics extracted from the microscopic NGSIM Peachtree dataset [50]. Figure 18 compares the average delay achieved by the proposed P-DQN+TD3+HER framework against four baseline methods across varying traffic volumes. As the traffic-demand scale escalates from 0.6 to 1.4, the traditional Fixed-Time baseline consistently demonstrates the worst performance, with average delays climbing near-linearly from 96 s to over 210 s. Under low-to-moderate traffic conditions ($0.6\le \mathrm{scale}\le 1.0$), all learning-based and adaptive algorithms exhibit comparable efficiency, maintaining stable delays below 30 s. However, a significant performance divergence emerges as demand surpasses the critical threshold of 1.0. The FRAP and PressLight models degrade rapidly under heavy saturation, with FRAP experiencing a sharp non-linear spike in latency up to 95 s at a 1.4 traffic scale.

Figure 18. Average delay under increasing traffic demand.

In contrast, the proposed reinforcement learning framework (P-DQN + TD3 + HER) demonstrates superior traffic stabilization, closely matching the performance of the MaxPressure heuristic. Both frameworks effectively suppress congestion at peak demand, restricting maximum average vehicle delays to under 50 s and demonstrating robust adaptability to highly congested intersection environments. An integrated analysis of vehicle delays and cumulative signal lost time highlights the superior efficiency and operational trade-offs of the proposed framework. While the Fixed-Time baseline minimizes signal transition overhead (75 s), its rigid scheduling causes catastrophic vehicle delays exceeding 210 s under heavy traffic. Conversely, traditional adaptive baselines like MaxPressure keep vehicle delays low but incur maximum signal lost time (415–460 s) due to continuous, aggressive phase switching. The proposed P-DQN + TD3 + HER framework uniquely optimizes this trade-off under light traffic, it minimizes unnecessary phase changes to preserve green-light momentum, restricting lost time to just 261 s while maintaining low delays. Under extreme saturation (1.4 scale), the model intelligently scales up phase transitions (408 s) to clear competing queues, successfully capping average vehicle delays under 50 s and demonstrating robust, demand-dependent optimization. Figure 19 shows the intersection queue growth rate, measured in vehicles per simulation step, across increasing traffic-demand scales (0.6–1.4).

Figure 19. Queue growth rate under increasing traffic demand.

The traditional Fixed-Time baseline exhibits the lowest queue growth trajectory, rising linearly from 2.9 to 5.5 veh/step, because its rigid cycle lengths meter traffic uniformly at the cost of high individual vehicle delays. Conversely, the adaptive baselines (MaxPressure, PressLight, and FRAP) cluster tightly together, showing a steeper accumulation that peaks between 6.7 and 7.1 veh/step due to frequent phase switching interrupting traffic momentum. The proposed P-DQN + TD3 + HER framework demonstrates the most efficient operational profile; it minimizes early bottlenecking with the lowest initial queue rate of 2.5 veh/step under light demand and consistently maintains a lower growth trajectory than all adaptive baselines until extreme saturation (1.4 scale), where it converges at 6.7 veh/step to effectively stabilize peak intersection pressure. To evaluate worst-case intersection congestion and storage lane overfilling, the network’s maximum queue length (vehicles) under scaling traffic demands is analyzed. As illustrated in Figure 20, the proposed P-DQN + TD3 + HER demonstrates excellent ability to gradually minimize queue length, particularly under high-demand conditions. While it exhibits slightly longer maximum queues at lower traffic volumes (0.6 to 1.0) compared to specialized adaptive baselines like PressLight and FRAP, it dramatically outperforms the traditional Fixed-Time control, which hits a maximum capacity ceiling of 30 vehicles by the 1.0 demand scale. Crucially, as the network scales to heavy and extreme saturation (1.2 to 1.4), the proposed model minimizes worst-case gridlock accumulation much better than its learning-based counterparts; at the highest demand scale of 1.4, it restricts the maximum queue to just 14 vehicles, successfully tying with the network-theory heuristic MaxPressure while significantly outperforming PressLight (18 vehicles) and FRAP (20 vehicles), which experience substantial queue spikes under severe loads. As illustrated in Figure 21, the proposed P-DQN+TD3+HER framework demonstrates a highly stable and optimized phase-switching strategy compared to adaptive baselines. The model maintains the lowest cumulative lost time over the entire range of traffic demand scale (0.6–1.4). Traditional adaptive models such as MaxPressure, PressLight, and FRAP exhibit excessively high phase-switching frequencies (exceeding 100 switches across all demand levels), indicating a tendency toward aggressive and inefficient phase toggling. Conversely, our proposed model intelligently scales its switching frequency from approximately 65 to 102 switches in direct proportion to the traffic demand. This model exhibits a highly intelligent, logical trend: as the traffic demand scale increases, the number of phase switches grows linearly. This proves that under low traffic, the model efficiently keeps green lights open longer for streaming vehicles (fewer switches), and only increases phase rotations when traffic builds up and multi-lane queues require clearing. This progressive adjustment minimizes intersection lost time (such as yellow and all-red phases) during low demand while maintaining the necessary agility to clear complex queues under heavy traffic conditions. Figure 22 evaluates the intersection storage utilization percentage across increasing traffic demand scales from 0.6 to 1.4. In traffic management, storage utilization represents the percentage of physical lane capacity occupied by queuing or stationary vehicles; higher values signal severe space consumption and impending gridlock. The traditional Fixed-Time baseline (purple line) exhibits a critical operational failure, starting at an already elevated 33.5% utilization under light traffic (0.6 scale) and climbing linearly to 50% under extreme congestion scale.

Figure 20. Maximum queue length analysis.

Figure 21. Cumulative signal lost time under increasing traffic demand.

Figure 22. Intersection storage utilization under increasing traffic demand.

This steep trajectory indicates that rigid scheduling fails to clear approaches in a timely way, allowing vehicles to accumulate and choke available roadway storage. Conversely, the proposed P-DQN + TD3 + HER framework and all learning-based basics (MaxPressure, PressLight, and FRAP) demonstrate exceptional space management, effectively clustering near the baseline between 0% and 6% across the entire demand spectrum. The proposed model (blue line) maintains a highly stable profile, peaking minutely at 2.5% during mid-tier demand (0.8 scale) before dropping back near 0% at maximum saturation, verifying that the intelligent phase adaptations successfully keep the physical intersection clear.

5.3.1. Performance Summary Table

Table 8 showing the quantitative comparison highlights the superior operational efficiency and demand-dependent scalability of the proposed framework over conventional baselines under light (0.6) and extreme (1.4) traffic-demand scales. While the Fixed-Time controller maintains minimal signal phase switching overhead (19 switches, 75 s lost time), it suffers catastrophic failure under saturated conditions, with individual vehicle delays exceeding 210 s and intersection storage utilization soaring to 50%. Conversely, traditional intelligent baselines like MaxPressure, PressLight, and FRAP successfully mitigate delays but exhibit extreme over-switching behaviors (up to 115 switches, 460 s lost time) even under sparse traffic conditions. The proposed P-DQN + TD3 + HER framework uniquely balances these trade-offs by running dynamically across demand tiers; it minimizes phase transitions to 65 switches and 261 s of lost time during low demand to preserve flow momentum, while smoothly escalating operations under peak congestion to cap average vehicle delays below 50 s, limit queue growth to 6.7 veh/step, and maintain a safe physical intersection buffer with an ultra-low storage utilization of 0.6%.

Table 8. Performance summary across varying traffic-demand scales.

Control Method	Average Delay (s)		Lost Time (s)		Queue Growth		Phase Switches		Storage Util. (%)
	0.6	1.4	0.6	1.4	0.6	1.4	0.6	1.4	0.6	1.4
Fixed-Time	96.0	>210.0	75.0	75.0	2.9	5.5	19	19	33.5	50.0
MaxPressure	15.0	≈45.0	415.0	460.0	2.9	7.1	105	115	0.5	5.8
PressLight	15.0	≈80.0	415.0	460.0	2.9	7.0	105	110	0.2	5.8
FRAP	15.0	≈95.0	415.0	460.0	2.8	6.7	105	105	0.0	0.1
Proposed Framework	15.0	<50.0	261.0	408.0	2.5	6.7	65	102	0.3	0.6

5.3.2. Discussion of Experimental Results for Urban Traffic via State-Dependent Queues (Peachtree-Based Dataset)

The experimental results obtained using the Peachtree-based trajectory dataset demonstrate that the proposed P-DQN + TD3 + HER framework provides effective and adaptive control of urban traffic under varying demand conditions. By incorporating state-dependent queue information into the learning process, the framework dynamically adjusts signal phases and green-time allocation according to the evolving traffic state rather than relying on fixed timing or purely instantaneous pressure measures. The results indicate that the controller maintains stable traffic operation under low, moderate, and high demand while preventing excessive queue accumulation and maintaining efficient utilization of the available intersection storage. As traffic demand increases, the proposed framework responds progressively by adapting phase selection and continuous green-time parameters, allowing increased traffic volumes to be accommodated without a proportional deterioration in delay or queue conditions. The integration of P-DQN for discrete-phase decisions, TD3 for continuous signal-timing optimization, and HER for learning from alternative traffic-goal states contributes to improved learning flexibility and operational robustness. Overall, the experimental findings confirm that the proposed state-dependent queue-based framework can translate observed trajectory-level traffic characteristics into adaptive signal-control decisions, achieving stable and scalable intersection performance across different traffic-demand regimes.

6. Conclusions and Future Work

This study presented a hybrid P-DQN+HER+TD3 reinforcement learning framework for adaptive traffic-signal control in urban traffic networks. The proposed framework combines discrete signal-phase selection through P-DQN with continuous signal-timing optimization using TD3, while Hindsight Experience Replay (HER) improves sample efficiency by relabeling achieved traffic states as alternative learning goals. In particular, the TD3 component provides stable continuous-control optimization through twin critics, delayed policy updates, and target-policy smoothing, thereby reducing value overestimation and improving the stability of green-time parameter learning under dynamic traffic conditions. The proposed controller operates across multiple intersections, where each intersection observes its local traffic state and generates a parameterized hybrid control action consisting of a discrete-phase decision and a continuous green-time parameter. The traffic environment incorporates movement-specific queue dynamics, phase-dependent saturation flows, signal lost times, and storage constraints, thereby providing an interactive framework for evaluating alternative signal-control actions. The NGSIM Lankershim Boulevard and Peachtree Street trajectory datasets are used as empirical sources for characterizing and calibrating traffic conditions, while the interactive traffic-control environment evaluates the response of the learned policies to alternative signal actions. The experimental results demonstrate that the proposed P-DQN+HER+TD3 framework achieves improved traffic-control performance relative to the considered baseline methods, particularly in reducing average delay and queue formation while increasing traffic throughput under varying traffic-demand conditions. The results further indicate that the combination of parameterized discrete–continuous control, HER-based experience relabeling, and TD3-based continuous policy optimization provides a robust mechanism for adapting signal timings to changing traffic conditions. Overall, the findings demonstrate the potential of integrating P-DQN, HER, and TD3 within a unified traffic-control framework. P-DQN provides effective discrete-phase selection, HER improves learning efficiency in sparse-reward traffic-control environments, and TD3 enables stable optimization of continuous signal-timing parameters. Their integration therefore provides a flexible foundation for adaptive multi-intersection traffic-signal control and supports further investigation of scalable, real-time deployment using larger traffic networks and field-calibrated traffic data.

The main limitation is that NGSIM provides historical vehicle trajectories rather than a closed-loop environment in which vehicle movements dynamically respond to alternative signal decisions. Therefore, the dataset is used for traffic-state characterization, environment construction, calibration, and validation, while policy evaluation is conducted within the developed interactive environment. Accordingly, the results should not yet be interpreted as direct field deployment. Future work will focus on large-scale coordinated control across multiple intersections, including inter-intersection information exchange and scalable multi-agent learning. Further closed-loop validation using a microscopic traffic simulator calibrated with NGSIM data, followed by hardware-in-the-loop and field-oriented experiments, will be pursued to assess the robustness and practical deployment potential of the proposed framework.