Enhancing Mixed Traffic Stability with TD3-Driven Bilateral Control in Autonomous Vehicle Chains

Abstract

This study presents a TD3-driven Bilateral Control Model (TD3-BCM) aimed at improving the stability of mixed traffic flows in autonomous vehicle (AV) chains. By integrating deep reinforcement learning, TD3-BCM optimizes control strategies to reduce traffic oscillations, smooth speed and acceleration fluctuations, and enhance overall system performance. Stability analysis shows that TD3-BCM effectively suppresses traffic fluctuations, with system stability improving from 1.132 to 1.182 as AV penetration increases. At an AV penetration rate of 40%, TD3-BCM surpasses both Cooperative Adaptive Cruise Control (CACC) and traditional Bilateral Control Model (BCM) approaches in terms of traffic efficiency, safety, and energy use—raising trailing vehicle speed by 12.6%, shortening average headway by 19.0%, increasing Time-to-Collision (TTC) by 87.3%, and lowering fuel consumption by 14.8%. When AV penetration reaches 70%, fuel savings rise to 19.7%, accompanied by further improvements in both traffic stability and safety. TD3-BCM provides a scalable and sustainable solution for intelligent transportation systems, particularly in high-penetration AV environments, by significantly enhancing stability, operational efficiency, and road safety.

1. Introduction

With the rapid advancement of autonomous driving technologies, road transportation systems are gradually transitioning from traditional human-driven dominance to increasingly integrated human–machine co-driving paradigms. Although fully autonomous driving remains the long-term goal, autonomous vehicles (AVs) and human-driven vehicles (HDVs) are expected to coexist for the foreseeable future, giving rise to highly heterogeneous and dynamically complex mixed traffic environments [1,2,3]. Within the broader goal of developing low-carbon, safe, and efficient transportation systems, a central challenge lies in effectively suppressing traffic disturbances while enhancing both operational stability and traffic flow efficiency under mixed traffic conditions [4,5].

Traffic disturbances are defined as deviations from steady-state conditions in the vehicle–road system, typically caused by individual driver behavior or external environmental fluctuations. Such disturbances typically manifest as spatiotemporal variations in vehicle speed, headway, and flow rate. Research in this area focuses on the mechanisms of generation, propagation, and attenuation of these disturbances, along with their combined impacts on traffic efficiency, safety, energy consumption, and associated carbon emissions [6,7]. Among these, traffic oscillations represent the most prevalent and disruptive form of disturbance in mixed traffic flows. These oscillations are typically characterized by periodic fluctuations in vehicle speed and spacing that propagate along vehicle platoons and amplify over time, resulting in reduced traffic efficiency, higher energy consumption, and increased safety risks [8,9,10].

Traffic stability refers to a system’s capacity to attenuate traffic disturbances and can be evaluated from three complementary perspectives: (1) Theoretical chain stability, derived from the linearization or frequency-domain analysis of car-following models, evaluates whether the AV penetration rate surpasses the critical threshold necessary to ensure system-wide stability [11,12]; (2) Cumulative damping ratio, which quantifies the decay of disturbances along the vehicle string based on acceleration energy ratios—where sustained values below 1 and a downstream-decreasing trend signify effective suppression [13,14]; (3) Traffic dynamics analysis, which employs spatiotemporal velocity maps and headway trajectories to visually assess the attenuation of traffic oscillations and validate the effectiveness of control strategies [10,15]. These three evaluation dimensions are complementary and collectively establish a robust framework for assessing the steady-state performance of control strategies in heterogeneous mixed traffic environments [16,17]. Prior research has demonstrated that optimizing traffic control strategies not only enhances system stability, reduces carbon emissions, and improves throughput and safety prediction, but also fosters the sustainable development of traffic systems across environmental, operational, and safety dimensions [18,19].

Traditional control strategies—such as classical car-following models based on unidirectional feedback, adaptive cruise control (ACC), and cooperative adaptive cruise control (CACC)—have improved individual vehicle responsiveness. However, they remain inadequate for suppressing multi-source disturbances and addressing behavioral heterogeneity in mixed traffic environments [20,21]. Bilateral control models (BCMs), which incorporate feedback from both leading and following vehicles, exhibit greater theoretical potential for mitigating disturbance propagation and enhancing chain stability [22]. However, most existing studies on BCM are confined to fully autonomous settings and do not address the heterogeneous dynamics inherent in AV–HDV coexistence.

In recent years, reinforcement learning (RL)—particularly its deep learning extension, deep reinforcement learning (DRL)—has demonstrated considerable promise in traffic control, owing to its strengths in policy optimization and adaptive learning within high-dimensional, uncertain environments [23,24]. Among DRL algorithms, the Twin Delayed Deep Deterministic Policy Gradient (TD3) has gained significant attention for its enhanced stability and suitability in continuous control tasks. In our previous work, we integrated TD3 into a bilateral control model to suppress traffic oscillations in fully autonomous environments, achieving notable improvements in both system stability and operational efficiency [25]. However, that framework was limited to homogeneous AV scenarios and did not address the behavioral heterogeneity and interaction complexity present in mixed traffic conditions—an issue this study seeks to resolve.

To this end, we propose a TD3-driven Bilateral Control Model (TD3-BCM) specifically designed for heterogeneous mixed traffic flows involving both autonomous vehicles (AVs) and human-driven vehicles (HDVs). This framework integrates the structural stability of BCM with the adaptive policy learning capabilities of TD3, and constructs a customized state space, action mechanism, and multi-objective reward function tailored to the dynamics of mixed traffic. The model aims to maintain system stability while simultaneously enhancing control efficiency and operational safety. The main contributions of this study are summarized as follows:

• Theoretical Contribution: This study extends the existing TD3-BCM framework to heterogeneous mixed traffic scenarios involving both AVs and HDVs. By introducing AV penetration rate as a key parameter, we derive a chain stability criterion and identify a critical threshold ${\varphi }_{min}$ necessary for maintaining stable operation. Based on this, we develop a multi-dimensional evaluation framework that incorporates stability, traffic efficiency, safety, and energy conservation—thereby enriching the theoretical foundation of mixed traffic flow control.
• Methodological Contribution: We construct a TD3-BCM control model suitable for heterogeneous vehicle platoons by embedding AV–HDV interaction states into the state–action–reward formulation of the DRL framework. Compared to previous BCM-DRL models designed for homogeneous AV scenarios, our approach demonstrates superior policy generalization and disturbance suppression, making it better suited for dynamic control tasks in real-world AV–HDV mixed environments.
• Empirical Validation: Leveraging reconstructed NGSIM I-80 trajectory data and extensive numerical simulations, we evaluate the performance of TD3-BCM across varying AV penetration rates. The results indicate that, compared to conventional unidirectional CACC and fixed-gain BCM models, TD3-BCM significantly reduces the amplitude and frequency of traffic oscillations, improves speed stability and throughput, and concurrently lowers fuel consumption and collision risk. These findings confirm the model’s adaptability and practical relevance in future mixed traffic scenarios.

The remainder of this paper is organized as follows: Section 2 surveys prior research, covering the formation and sustainability impacts of traffic oscillations, the evolution of control models from classical car-following to bilateral control, and reinforcement-learning techniques for mixed-traffic management. Section 3 introduces the proposed methodology. It first revisits the traditional bilateral control model, then details the TD3-driven Bilateral Control Model—including the actor–critic architecture, target-policy smoothing, and clipped double Q-learning—and finally explains the state/action/reward formulation together with the training and evaluation workflow. Section 4 describes the simulation setup and reports the quantitative results. The stability study comprises a theoretical chain-stability analysis, cumulative damping-ratio verification, and traffic-dynamics visualization, whereas the performance study presents efficiency, safety, and energy-saving metrics. Section 5 concludes the paper by summarizing the key findings, discussing their implications for mixed-traffic systems, and outlining future research directions.

2. Related Work

2.1. Traffic Oscillations and Sustainability Impacts

Traffic oscillations—often observed as “stop-and-go waves”—constitute a prevalent form of instability in both traditional and mixed traffic flows. These oscillations typically manifest as periodic waves of acceleration and deceleration that can emerge spontaneously, even without explicit external stimuli. Early studies primarily attributed their formation to physical bottlenecks, such as lane drops or merges [26,27]. However, the landmark ring-road experiment conducted by Sugiyama et al. [28] demonstrated that traffic instabilities can also arise purely from driver reaction delays and behavioral heterogeneity, even in geometrically unconstrained conditions. At the system level, traffic oscillations significantly undermine the sustainability of transportation networks. From an environmental perspective, frequent acceleration and braking cycles increase fuel consumption and greenhouse gas emissions [29,30]. From a safety perspective, oscillations reduce the controllability of vehicle spacing and speed, thereby increasing collision risk [31]. Economically, they diminish road throughput and degrade commute reliability and travel time efficiency. Collectively, these adverse effects contradict the overarching goals of modern urban transportation systems—namely, safety, operational efficiency, and environmental sustainability [32].

In recent years, increasing attention has been paid to the intrinsic coupling between traffic stability and broader sustainability objectives. Studies have shown that suppressing oscillations not only improves driving comfort but also significantly reduces energy consumption and enhances key safety and environmental indicators, such as Time-To-Collision (TTC) and Fuel Economy Index (FEI) [25,33]. Therefore, traffic stability—understood as the system’s capacity to dampen perturbations in speed and spacing—should be regarded not only as a control objective but also as a foundational prerequisite for sustainable transportation development. With the rapid advancement of autonomous driving technologies, the coexistence of autonomous vehicles (AVs) and human-driven vehicles (HDVs) in mixed traffic environments is expected to persist over the long term. This coexistence introduces novel disturbance patterns and complex control challenges, which expose the limitations of traditional traffic control strategies in maintaining system stability and adaptability. While mesoscopic and macroscopic control methods—such as large-scale traffic management [34] and local coordination [35]—have shown partial success in traffic stabilization, they typically lack integration with microscopic vehicle dynamics. As a result, their ability to mitigate oscillations at the individual vehicle level is limited. Therefore, more refined and personalized control strategies are necessary to effectively address oscillation issues in mixed traffic environments.

2.2. Evolution of Control Models: From Car-Following Model to Bilateral Control Model

Research on traffic stability initially centered on car-following models (CFMs), in which each vehicle dynamically adjusts its acceleration based on the state of the vehicle ahead. Classical models such as the General Motors (GM) model and the Intelligent Driver Model (IDM) have laid the theoretical foundation for longitudinal control modeling [36,37]. Building upon these, adaptive cruise control (ACC) employs onboard sensors to dynamically regulate vehicle speed, thereby enhancing individual vehicle responsiveness [26]. Cooperative adaptive cruise control (CACC) further introduces vehicle-to-vehicle (V2V) communication to acquire upstream information and improve disturbance anticipation [38]. However, these control strategies rely on unidirectional feedback structures and lack the capability to respond to downstream disturbances. As a result, they often fail to ensure system-level stability in high-density or behaviorally heterogeneous mixed traffic environments and may exacerbate traffic oscillations.

To address these structural limitations, Horn et al. proposed the Bilateral Control Model (BCM) [39], which incorporates bidirectional state feedback from both leading and following vehicles to construct a local damping mechanism that effectively absorbs disturbances at their source. The control structure satisfies the damped wave equation and exhibits favorable string stability under various boundary conditions [40]. Wang et al. further introduced a chain stability criterion and validated the robust convergence properties of BCM across multiple disturbance frequencies [41]. The model was subsequently extended to a multi-node BCM framework, which integrates information from multiple neighboring vehicles and utilizes Taylor expansion and least-squares optimization to enhance low-frequency disturbance suppression and system responsiveness [42]. At the spectral analysis level, Wang and Horn developed an eigenvalue-based framework to analyze string stability, revealing the system’s stability characteristics in the frequency domain [43]. Furthermore, BCM has been adapted to scenarios involving the coexistence of autonomous vehicles (AVs) and human-driven vehicles (HDVs), where collaborative mechanisms and stability boundaries under different control structures were systematically explored [44]. Despite the structural advantages of BCM in theoretical modeling, most existing studies are based on idealized assumptions of fully autonomous environments and fixed control parameters. These models lack the adaptability to dynamic disturbances caused by perception delays, behavioral heterogeneity, and environmental uncertainty, which are prevalent in real-world mixed traffic systems.

2.3. Reinforcement Learning in Mixed Traffic Control

Reinforcement learning (RL) and its deep extension, deep reinforcement learning (DRL), have demonstrated remarkable capabilities in handling nonlinear, high-dimensional, and dynamic control problems. Through agent–environment interaction and trial-and-error learning, DRL exhibits superior adaptability and generalization compared to traditional rule-based or model-driven approaches. As a result, DRL has been widely adopted in the domain of intelligent traffic control.

Early applications of reinforcement learning (RL) in transportation began with Q-learning [45], which, despite its conceptual simplicity, exhibited poor scalability in large state spaces. To address this limitation, Deep Q-Networks (DQN) were introduced to approximate the Q-value function using neural networks, significantly improving learning efficiency in discrete action domains [46]. However, DQN is not suitable for continuous control tasks such as vehicle longitudinal regulation. To overcome this issue, the Deep Deterministic Policy Gradient (DDPG) algorithm was proposed, incorporating an actor–critic architecture to extend deep RL into continuous action spaces [47]. Shi et al. [22] pioneered the integration of bidirectional feedback into the DRL framework by developing a BCM-DRL model based on DDPG. By embedding both leading and following vehicle states into the state representation, their model achieved trajectory-tracking performance that surpassed that of human drivers in homogeneous AV platoons, demonstrating the potential of combining DRL with BCM architectures. However, DDPG often suffers from overestimation bias and unstable convergence during training.

The Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm enhances DDPG by introducing twin Q-networks, delayed policy updates, and target policy smoothing. These refinements significantly improve policy stability and learning robustness, making TD3 a key technique in real-time continuous control scenarios such as automated vehicle regulation. Liu et al. [25] proposed the BCM-DRL control framework, which integrates the TD3 algorithm into the bilateral control model as a replacement for conventional DDPG. Their study was conducted under a homogeneous autonomous vehicle (AV) setting, focusing on traffic oscillation suppression. The model incorporates an AV-centered state space, a continuous action mechanism, and a multi-objective reward function, significantly improving the system’s responsiveness to disturbances and enhancing string stability. Although the framework has not yet been extended to heterogeneous mixed traffic environments, it provides a solid foundation for future exploration of AV–HDV cooperative control strategies.

Furthermore, comparative studies between DRL and traditional approaches such as Model Predictive Control (MPC) [48,49] suggest that while DRL offers superior adaptability in uncertain or nonlinear traffic environments, it still faces challenges in convergence and training reliability. For example, Li et al. [50] combined the Helly model with MPC to develop a jam-absorption strategy, yet this approach depends on precise parameter calibration and assumes linear dynamics, limiting its generalizability. In contrast, DRL’s robust adaptability—particularly when coupled with bidirectional feedback from BCM—holds promise for more flexible, scalable, and effective control strategies that can suppress oscillations and optimize flow in mixed traffic conditions.

3. Methodology

This study proposes a TD3-driven Bilateral Control Model (TD3-BCM) that integrates deep reinforcement learning with bidirectional feedback mechanisms to address the complex, dynamic nature of mixed traffic environments. By introducing a carefully designed reward function emphasizing traffic-flow stability, the model optimizes the traditional Bilateral Control Model (BCM). It adaptively adjusts control strategies under various penetration rates of autonomous and human-driven vehicles (HDVs), significantly enhancing disturbance suppression, convergence stability, and control precision, thereby improving overall traffic-flow robustness. Section 3.1 introduces the theoretical foundation and key principles of traditional BCM. Building on this, Section 3.2 presents the proposed TD3-BCM, detailing how Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforces the model’s robustness in complex traffic scenarios. Section 3.3 discusses the training process and optimization steps for TD3-BCM.

3.1. Introduction to Traditional Bilateral Control Models

The Bilateral Control Model (BCM), initially proposed by Horn in 2013 [39], is based on the principle of balancing a vehicle’s position and velocity relative to both its leading and following neighbors, as illustrated in Figure 1.

The control law for a BCM vehicle n can be expressed as Equation (1):

$$\begin{array}{cc}\hfill a_n\left(t\right)=& k_d\left(d_{\mathrm{ahead}}-d_{\mathrm{behind}}\right)\hfill \\ \hfill & +k_v\left(\left(v_{\mathrm{lead}}-v_{\mathrm{mid},n}\left(t\right)\right)-\left(v_{\mathrm{mid},n}\left(t\right)-v_{\mathrm{follow}}\right)\right)\hfill \\ \hfill & +k_c\left(v_{\mathrm{mid},n}\left(t\right)-v_{\mathrm{desired}}\right)\hfill \\ \hfill & +k_a\left(a_{\mathrm{lead}}-a_{\mathrm{mid},n}\right)\hfill \end{array}$$

(1)

where

• $a_n\left(t\right)$: Acceleration of vehicle n at time t.
• $d_{\mathrm{ahead}},d_{\mathrm{behind}}$: Distances between vehicle n and its leading and following vehicles, respectively.
• $v_{\mathrm{lead}},v_{\mathrm{mid},n},v_{\mathrm{follow}}$: Speeds of the leading, current, and following vehicles.
• $v_{\mathrm{desired}}$: Target cruising speed of vehicle n.
• $a_{\mathrm{lead}},a_{\mathrm{mid},n}$: Accelerations of the leading vehicle and vehicle n, respectively.

The parameters $k_d$, $k_v$, $k_c$, and $k_a$ represent the control gains associated with distance error, relative velocity, speed tracking, and acceleration tracking, respectively. Based on prior empirical and simulation studies [39,51], their recommended ranges are set as follows: $k_d\in [0.1,1.5]$, $k_v\in [0.1,1.0]$, $k_c\in [0.1,0.8]$, and $k_a\in [0.1,0.5]$. These parameter intervals are selected to balance system responsiveness with damping performance under representative traffic conditions. For baseline simulations, the initial values are selected as $k_d=0.8$, $k_v=0.5$, $k_c=0.5$, and $k_a=0.2$, which have been validated in previous BCM implementations to provide a favorable trade-off between oscillation suppression and responsiveness [25]. Unless otherwise specified, these parameter settings assume idealized conditions of perception and communication.

In a BCM chain, the rear-most vehicle typically operates under a unilateral car-following model, such as an Adaptive Cruise Control (ACC) system, since it lacks a trailing vehicle. The ACC model is described by Equation (2):

$$\begin{array}{c}\hfill a_n\left(t\right)=k_1\left(d_{\mathrm{desired}}\left(t\right)-d_n\left(t\right)\right)+k_2\left(v_{\mathrm{desired}}\left(t\right)-v_n\left(t\right)\right)\end{array}$$

(2)

where

• $a_n\left(t\right)$: Acceleration of the n-th vehicle at time t.
• $k_1,k_2$: Control gain parameters determining the vehicle’s response to errors in distance and speed. Suggested ranges: $k_1\in [0.3,1.5]$ and $k_2\in [0.5,2.0]$, balancing stability and responsiveness [2,26].
• $d_{\mathrm{desired}}\left(t\right)$: Desired distance, defined as $d_0+\eta v_n\left(t\right)$, with $d_0\approx 2 \mathrm{m}$ and $\eta \approx 1.5 \mathrm{s}$ [36].
• $d_n\left(t\right)$: Actual distance between the n-th vehicle and the preceding vehicle.
• $v_{\mathrm{desired}}\left(t\right)$: Desired speed, typically set as the target speed or road speed limit.
• $v_n\left(t\right)$: Actual speed of the n-th vehicle at time t.

Although BCM improves upon unidirectional feedback models, it relies on fixed control parameters that limit adaptability in non-stationary or nonlinear traffic dynamics [39,40]. To overcome these constraints, we propose a deep reinforcement learning extension—TD3-Bilateral Control Model (TD3-BCM)—that leverages real-time feedback from traffic conditions for enhanced stability, reduced energy consumption, and more effective oscillation mitigation than conventional BCM approaches.

3.2. TD3-Driven Bilateral Control Model

This section introduces the TD3-driven Bilateral Control Model (TD3-BCM), which integrates the Twin Delayed Deep Deterministic Policy Gradient (TD3) into the bilateral control framework to enhance adaptability and stability in mixed traffic. The TD3-BCM framework comprises two key components. The first component explains the foundational TD3 mechanisms—namely, the actor–critic structure, target policy smoothing, and clipped double Q-learning. The second component introduces the TD3-BCM formulation, including the state and action spaces, the multi-objective reward function, and the policy update strategy. By integrating these algorithmic foundations with a tailored formulation for real-time vehicle interactions, TD3-BCM effectively captures behavioral heterogeneity and enables stable and efficient control of mixed traffic flow.

3.2.1. Core Mechanisms of TD3

In the context of mixed traffic flows, the Core Mechanisms of TD3—specifically the Actor-Critic architecture, target policy smoothing, and clipped double Q-learning—collectively serve as the algorithmic backbone for handling the uncertainties and real-time demands introduced by diverse vehicle types operating in parallel. By balancing stability and robustness in policy learning, these mechanisms help minimize acceleration fluctuations and improve inter-vehicle coordination. This, in turn, enhances system-level efficiency and safety, thereby supporting the real-time application of bilateral control strategies in complex mixed traffic conditions.

(a)

Actor-Critic Architecture

The TD3 algorithm adopts an actor–critic architecture that decouples policy generation from value estimation, thereby improving learning stability and precision. Two distinct sets of neural networks are designed for the tail vehicle and middle vehicles, respectively.

• Tail Vehicle (Actor 1 and Critic 1): The tail vehicle operates under a unilateral car-following model. Its state vector $s_n\left(t\right)=[v_n\left(t\right),d_n\left(t\right),\Delta v_n\left(t\right)]$ is fed into Actor 1, which comprises two hidden layers with 32 neurons each. Actor 1 outputs the acceleration action $a_n\left(t\right)$. Critic 1 receives the same state vector along with $a_n\left(t\right)$ as input and estimates the corresponding Q-value using a 32 × 32 hidden structure, as shown in Figure 2.
• Middle Vehicles (Actor 2 and Critic 2): Middle vehicles adopt a bilateral control structure, extending their state vector to $s_n\left(t\right)=[v_n\left(t\right),d_n\left(t\right),\Delta v_n\left(t\right),v_{n+1}\left(t\right),d_{n+1}\left(t\right)]$ to incorporate information from both preceding and following vehicles. Actor 2 consists of two hidden layers with 64 neurons each and outputs $a_n\left(t\right)$. Critic 2 estimates the Q-value based on the same state and action input, using a 64 × 64 hidden layer architecture, as shown in Figure 3.

This architecture enables the TD3-BCM framework to dynamically adjust vehicle acceleration while ensuring that these adjustments align with the objectives of traffic stability, safety, and efficiency. The tail vehicle focuses on interactions with its leader, while the middle vehicles learn cooperative behaviors through bidirectional feedback.

(b)

Target Policy Smoothing

TD3 incorporates target policy smoothing to mitigate overfitting to deterministic policies, which may impair generalization in real-world scenarios. During each policy update, stochastic noise is injected into the target policy [52] to improve exploration, as shown in Equation (3):

$${\pi }_{\mathrm{target}}\left(s_n^{\prime }\right)=\pi \left(s_n^{\prime }\right)+\mathrm{clip}(\mathcal{N}(0,\sigma ),-c,c)$$

(3)

where

• ${\pi }_{\mathrm{target}}\left(s_n^{\prime }\right)$: Smoothed target policy for state $s_n^{\prime }$.
• ${\pi }_{\theta }\left(s_n^{\prime }\right)$: Deterministic action from the actor network.
• $\mathcal{N}(0,\sigma )$: Zero-mean Gaussian noise.
• $\sigma$: Standard deviation of the noise, typically set to 0.2 [52].
• c: Clipping threshold to limit noise within $[-c,c]$, typically $c=0.5$ [53].

This technique reduces the model’s dependency on deterministic actions, thereby enhancing generalization under dynamic traffic conditions.

(c)

Clipped Double Q-Learning of TD3

Traditional reinforcement learning methods are prone to overestimation bias in Q-value estimation, which can undermine overall performance. TD3 addresses this issue using clipped double Q-learning, which trains two critic networks simultaneously and selects the minimum Q-value for policy updates [47,52]. In this approach, two independent critic networks are trained simultaneously, and the minimum of their Q-values is used to update the actor network, as shown in Equation (4):

$$Q_{\mathrm{target}}=r+\gamma min\left(Q_{{\theta }_1}(s_n^{\prime },a^{\prime }),Q_{{\theta }_2}(s_n^{\prime },a^{\prime })\right)$$

(4)

where

• $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$: Two independent critic networks.
• $s_n^{\prime }$ and $a^{\prime }$: Next state and next action.
• r: Immediate reward, normalized to $[-1,1]$ for stability [47,52].
• $\gamma$: Discount factor, balancing short- and long-term returns, typically 0.95 or 0.99 [46].

By adopting the lower Q-value from two critics, this method ensures conservative value updates, which improve training stability and policy robustness.

3.2.2. TD3-BCM Formulation

The following describes how the aforementioned TD3 mechanisms are integrated into the bilateral control framework, covering the state space, action space, reward function, and policy update procedure. The resulting workflow enables real-time adaptability to mixed traffic conditions while balancing traffic stability, safety, and efficiency. An overview of the framework is given as follows:

(a)

State Space

The state space captures dynamic traffic conditions and varies for middle and tail vehicles within the chain:

Middle Vehicles: Middle vehicles adopt a bilateral control model, integrating information from both preceding and following vehicles. The state vector is defined as Equation (5):

$$s_{\mathrm{mid},n}\left(t\right)=\left[v_{\mathrm{mid},n}\left(t\right),d_{\mathrm{ahead},n}\left(t\right),\Delta v_{n,n-1}\left(t\right),d_{\mathrm{behind},n}\left(t\right),v_{n+1}\left(t\right)\right]$$

(5)

where

• $v_{\mathrm{mid},n}\left(t\right)$: Represents the current speed of the middle vehicle.
• $d_{\mathrm{ahead},n}\left(t\right)$: Is the distance to the preceding vehicle.
• $\Delta v_{n,n-1}\left(t\right)$: Is the relative speed with the preceding vehicle.
• $v_{n+1}\left(t\right)$: Is the current speed of the following vehicle.
• $d_{\mathrm{behind},n}\left(t\right)$: Is the distance to the following vehicle.

Tail Vehicles: Tail vehicles use a single-sided car-following model, focusing on interactions with the preceding vehicle. The state vector is given as Equation (6):

$$s_{\mathrm{tail},n}\left(t\right)=[v_{\mathrm{tail},n}\left(t\right),d_{\mathrm{tail},n}\left(t\right),\Delta v_{\mathrm{tail},n}\left(t\right)]$$

(6)

where

• $v_{\mathrm{tail},n}\left(t\right)$: Represents the current speed of the tail vehicle.
• $d_{\mathrm{tail},n}\left(t\right)$: Is the distance to the preceding vehicle.
• $\Delta v_{\mathrm{tail},n}\left(t\right)$: Is the relative speed with the preceding vehicle.

These definitions enable middle vehicles to assess their position and velocity relationships with both preceding and following vehicles, while tail vehicles focus solely on preceding vehicle interactions.

(b)

Action Space

The action space represents the longitudinal acceleration $a_n\left(t\right)$, a continuous variable constrained by

$$-3\mathrm{m}/{\mathrm{s}}^2\le a_n\left(t\right)\le 3\mathrm{m}/{\mathrm{s}}^2.$$

This range, derived from Hu et al. [54], ensures realistic driving behavior, maintaining passenger comfort and operational feasibility.

(c)

Reward Function

The reward function balances traffic flow stability, efficiency, and safety, with additional considerations for smooth control actions. The components include the following:

Stability: Large variations in acceleration are penalized to suppress traffic oscillations, shown as Equation (7):

$$r_{\mathrm{stability},n}=-{\alpha }_1{a}_n{\left(t\right)}^2-{\alpha }_2\left|a_n\left(t\right)\right|$$

(7)

where $a_n\left(t\right)$ represents the acceleration of the n-th vehicle, ${\alpha }_1(0.1\sim 1.0)$ penalizes squared acceleration to reduce traffic oscillations, and ${\alpha }_2(0.1\sim 0.5)$ penalizes absolute acceleration to smooth variations, enhancing traffic stability and comfort [26,54,55].

Safety: The safety reward component encourages maintaining safe driving behavior by addressing potential risks associated with insufficient time gaps, unsafe inter-vehicle distances, and excessive speeds. This component integrates three critical sub-rewards: Time Gap Penalty, Distance Penalty, and TTC-Based Collision Avoidance.

Time Gap Penalty, A penalty is applied when the time gap $T_n\left(t\right)$ between the vehicle and its preceding vehicle falls below the safe threshold of $0.6\mathrm{s}$, shown as Equation (8):

$$r_{\mathrm{time}\mathrm{gap},n}=-{\beta }_{\mathrm{time}\mathrm{gap}}max\left(0,0.6-T_n\left(t\right)\right)$$

(8)

where $T_n\left(t\right)$ is the time gap between the vehicle and its preceding vehicle; ${\beta }_{\mathrm{time}\mathrm{gap}}$ is the penalty weight for unsafe time gaps, adjustable based on safety requirements.

Distance Penalty: When the inter-vehicle distance $d_n\left(t\right)$ is less than the minimum safe distance $d_{\mathrm{safe}}$, the penalty escalates as the distance approaches zero, as shown in Equation (9):

$$r_{\mathrm{distance},n}=-{\beta }_{\mathrm{distance}}max\left(0,d_{\mathrm{safe}}-d_n\left(t\right)\right)$$

(9)

where $d_n\left(t\right)$ is the distance between the vehicle and its preceding vehicle; $d_{\mathrm{safe}}$ is the minimum safe distance, typically set to $2\mathrm{m}$ [26]; ${\beta }_{\mathrm{distance}}$ is the penalty weight for unsafe distances.

TTC-Based Collision Avoidance: To enhance safety during training and testing, a Time-to-Collision (TTC)-based collision avoidance mechanism is implemented. The safe speed is computed using the following Equations (10) and (11):

$$v_{\mathrm{safe}}=v_{\mathrm{ahead}}+max\left(0,{\displaystyle \frac{d_n\left(t\right)-d_{\mathrm{safe}}-v_n\left(t\right)·T_{\mathrm{react}}}{T_{\mathrm{react}}+\frac{v_n\left(t\right)}{a_{\max }}}}\right)$$

(10)

$$r_{\mathrm{TTC},n}=-{\beta }_{\mathrm{TTC}}max\left(0,v_n\left(t\right)-v_{\mathrm{safe}}\right)$$

(11)

where

• $v_{\mathrm{safe}}$: Is the computed safe speed to avoid collisions.
• $v_{\mathrm{ahead}}$: Is the speed of the preceding vehicle.
• $T_{\mathrm{react}}=0.4\mathrm{s}$ [36]: Is the reaction time.
• $a_{\max }=3\mathrm{m}/{\mathrm{s}}^2$ [56]: Is the maximum deceleration.
• ${\beta }_{\mathrm{TTC}}$: Is the penalty weight for unsafe speeds.

The total safety reward for each vehicle (middle or tail) integrates the three components, as shown in Equation (12):

$$r_{\mathrm{safety},n}=r_{\mathrm{time}\mathrm{gap},n}+r_{\mathrm{distance},n}+r_{\mathrm{TTC},n}$$

(12)

Efficiency: Encourages maintaining the desired speed $v_{\mathrm{desired}}$, as described in Equation (13):

$$r_{\mathrm{efficiency},n}=-{\beta }_{\mathrm{efficiency}}\left|v_n\left(t\right)-v_{\mathrm{desired}}\right|$$

(13)

where

• ${\beta }_{\mathrm{efficiency}}(0.1\sim 0.5)$ is the penalty weight for speed deviations, balancing efficiency and adaptability to traffic conditions [31,57].

Smoothy: To facilitate early-stage training and prevent excessive accelerations or inter-vehicle spacing, a logarithmic function is incorporated into the reward structure, as described in Equation (14):

$$r_{\mathrm{smoothy},n}=-log\left(1+|a_n\left(t\right)|\right)+{\displaystyle \frac{1}{d_n\left(t\right)}}$$

(14)

This adjustment prevents excessively negative rewards caused by large accelerations or distances, improving training stability and convergence.

Overall Reward Function: The combined reward function for vehicles’ chains integrates stability, safety, efficiency, and smoothness components, as shown in Equation (15):

$$r_n\left(t\right)=r_{\mathrm{stability},n}+r_{\mathrm{safety},n}+r_{\mathrm{efficiency},n}+r_{\mathrm{smoothy},n}$$

(15)

Table 1. Summary of Reward Components and Their Functions.

Component	Parameters	Functions
Stability Reward ($r_{\mathrm{stability},n}$)	${\alpha }_1$, ${\alpha }_2$	Reduces acceleration fluctuations.
Time Gap Penalty ($r_{\mathrm{time}\_\mathrm{gap},n}$)	${\beta }_{\mathrm{time}\_\mathrm{gap}}$	Penalizes insufficient time gaps.
Distance Penalty ($r_{\mathrm{distance},n}$)	${\beta }_{\mathrm{distance}}$	Penalizes unsafe following distances.
TTC Penalty ($r_{\mathrm{TTC},n}$)	${\beta }_{\mathrm{TTC}}$	Enhances collision avoidance.
Efficiency Reward ($r_{\mathrm{efficiency},n}$)	${\beta }_{\mathrm{efficiency}}$	Encourages target speed tracking.
Smoothness Reward ($r_{\mathrm{smoothy},n}$)	—	Improves control smoothness.

outlines the key reward components in the TD3-BCM framework, detailing their parameters and respective control functions. These components collectively drive the system toward stable, safe, efficient, and smooth operation.

(d)

Policy Update

The TD3 algorithm iteratively updates the state and action spaces to optimize control strategies. This process evaluates the effectiveness of actions based on their impact on stability, safety, and efficiency. Techniques like target policy smoothing and clipped double Q-learning are used to mitigate overestimation and improve robustness. These iterative updates enable the control framework to adapt to dynamic traffic conditions, ensuring smoother and more efficient traffic flow.

3.3. Training and Evaluation of TD3-BCM

3.3.1. Training and Test Data

This study employs the publicly available NGSIM I-80 dataset, recorded on the I-80 interstate highway near Emeryville, California [58]. It spans a 503-meter-long highway weaving segment (lanes 2 to 5), with data recorded at 10 Hz, capturing vehicle positions, speeds, and other traffic attributes. The data was collected between 04:00 and 04:15 pm on 13 April 2005. It represents typical highway traffic flow by focusing on same-lane vehicle movements while excluding lane-changing behavior. Although the NGSIM I-80 dataset reflects a specific traffic scenario, the dataset is reconstructed in this study to improve model adaptability. The reconstructed dataset integrates diverse traffic patterns and driving behaviors to more accurately reflect real-world acceleration and deceleration, thereby enhancing the model’s generalizability across different traffic scenarios. It emphasizes vehicle acceleration and deceleration to ensure the model accurately reflects dynamic variations in mixed traffic environments.

For training, to improve efficiency and limit random exploration, each velocity profile was constrained to 30 s (300 simulation steps). This constraint prevents excessive vehicle spacing and prolonged negative rewards, thus accelerating convergence. The data selection criteria included profiles lasting longer than 30 s, with speeds above 2 m/s and speed standard deviations greater than 3 m/s. Ultimately, 50 high-quality 30-s profiles were selected, capturing common acceleration and deceleration patterns in mixed traffic.

For the test set, this study applied the processing method of Jiang et al. [59] for the highD dataset to the I-80 dataset’s vehicle speed profiles. Trajectories exhibiting traffic oscillation characteristics were selected, requiring durations longer than 30 s and speed standard deviations greater than 3 m/s. Speed gaps were filled by smoothly blending velocity profiles using acceleration transitions within ±0.3 m/s². Finally, 50 test trajectories, each 300 s long, were created and used exclusively for the lead vehicle (Leader Vehicle), ensuring the independence of the training and test datasets. Despite representing a specific traffic environment, the reconstructed dataset significantly enhances the model’s adaptability and robustness across various traffic conditions.

As shown in Figure 4, a velocity profile selected from the reconstructed test set trajectory is presented, accurately reflecting speed fluctuations consistent with real-world vehicle acceleration and deceleration patterns. Training on these realistic velocity profiles enables the TD3-BCM to better adapt to complex traffic dynamics, particularly in scenarios involving frequent vehicle starts and stops. Therefore, the reconstructed NGSIM I-80 dataset not only simulates real-world acceleration and deceleration processes but also demonstrates good generalization ability, making it applicable to a wider range of traffic scenarios.

3.3.2. Training and Test Steps

The training framework follows a two-stage process. In the first stage, a unilateral car-following model (CFM) is trained with TD3 to govern the tail vehicle, providing a behavioral baseline for subsequent training. In the second stage, middle vehicles are trained using the pre-trained tail vehicle as a reference, with parameter sharing employed to ensure consistency throughout the BCM framework. The reward functions—focusing on stability, efficiency, and safety—were refined to suppress acceleration fluctuations and maintain appropriate inter-vehicle spacing. Following training, the model was evaluated on unseen trajectory profiles to assess its adaptability and robustness.

Each vehicle updates its speed and inter-vehicle spacing in discrete time steps using

$$\left\{\begin{array}{c}{v}_n(t+1)=v_n\left(t\right)+a_n\left(t\right)\Delta t,\hfill \\ \Delta v_n\left(t\right)=v_n\left(t\right)-v_{n-1}\left(t\right),\hfill \\ d_n(t+1)=d_n\left(t\right)+{\displaystyle \frac{v_n\left(t\right)+v_n(t+1)}{2}}\Delta t,\hfill \end{array}\right.$$

(16)

where $v_n$ and $a_n$ denote the speed and acceleration of the n-th vehicle at time t, respectively.

As shown in Figure 5, the training process for the TD3-BCM framework is divided into two stages. Step 1 involves training the tail vehicle using a unilateral car-following model (CFM) to establish a baseline for individual vehicle control. In this stage, Actor 1 represents the policy network that selects actions for the tail vehicle, while Critic 1 evaluates the quality of those actions based on the reward function. Experience replay 1 stores past interactions to improve learning stability. Step 2 extends this process to the middle vehicles in the chain, leveraging the pre-trained tail vehicle model. Here, Actor 2 is responsible for selecting actions for the middle vehicles, while Critic 2 assesses their performance. Experience replay 2 facilitates policy refinement across the entire vehicle chain, ensuring consistent and efficient learning.

3.3.3. Parameters and Training Results

Proper configuration of key parameters is essential to ensuring the TD3-BCM framework performs reliably across diverse traffic conditions. Informed by prior studies and extensive experimentation, these parameters are carefully calibrated to balance convergence speed, robustness, and adaptability.

Table 2. Key Parameters and Their Configurations for the TD3-BCM Framework in Mixed Traffic Scenarios.

Parameter Name	Description	Value
$v_n\left(t\right)$	Speed of the vehicle at time t	Dynamic
$a_n\left(t\right)$	Acceleration of the vehicle	$[-3,3]\mathrm{m}/{\mathrm{s}}^2$
$d_{\mathrm{ahead},n}\left(t\right)$	Distance to the preceding vehicle	Dynamic
$d_{\mathrm{behind},n}\left(t\right)$	Distance to the following vehicle	Dynamic
$d_{\mathrm{safe}}$	Minimum safe distance	$2\mathrm{m}$
$v_{\mathrm{desired}}$	Target speed	$33.3\mathrm{m}/\mathrm{s}$
$T_{\mathrm{safe}}$	Minimum time gap ensuring safety	$0.6\mathrm{s}$
$k_d$	Distance feedback gain	$0.1\sim 1.5\left(0.8\right)$
$k_v$	Velocity feedback gain	$0.1\sim 1.0\left(0.5\right)$
$k_c$	Target velocity feedback gain	$0.1\sim 0.8\left(0.5\right)$
$k_a$	Acceleration feedback gain	$0.1\sim 0.5\left(0.2\right)$
${\alpha }_1,{\alpha }_2$	Stability penalty weight	$0.4,0.3$
$T_{\mathrm{react}}$	Reaction time for vehicles	$0.4\mathrm{s}$
$a_{\max }$	Maximum deceleration	$3\mathrm{m}/{\mathrm{s}}^2$
${\beta }_{\mathrm{time}\mathrm{gap}}$	Penalty weight for unsafe time gaps	$0.5\sim 1.5\left(1.0\right)$
${\beta }_{\mathrm{distance}}$	Penalty weight for unsafe distances	$0.5\sim 2.0\left(1.0\right)$
${\beta }_{\mathrm{TTC}}$	Penalty weight for exceeding safe speed	$0.5\sim 1.5\left(1.0\right)$
${\beta }_{\mathrm{efficiency}}$	Penalty weight for efficiency deviations	$0.1\sim 0.5\left(0.3\right)$
${\alpha }_{\mathrm{actor}}$	Actor network learning rate	$1\times {10}^{-4}$
${\alpha }_{\mathrm{critic}}$	Critic network learning rate	$2\times {10}^{-4}$
$\gamma$	Discount factor	$0.95$
$\tau$	Soft update rate	$0.001$
Replay Buffer Size	Experience replay buffer size	$20,000$ (tail), $80,000$ (mid)
Batch Size	Mini-batch size	64 (tail), 256 (mid)
$\sigma$	Standard deviation of target noise	$0.2$
Clipping Range	Clipping range for target noise	$[-0.5,0.5]$

This table provides a comprehensive summary of the key parameters and configurations used in the TD3-BCM framework, including dynamic parameters, feedback gains, and reward function weights. The values are determined based on simulation performance, prior discussions, and references to relevant studies.

summarizes the primary parameters and their respective roles, laying the groundwork for subsequent experimental evaluation.

The training results highlight the TD3-BCM’s effectiveness in optimizing vehicle-chain control under mixed traffic conditions. As illustrated in Figure 6a, the reward value for the tail vehicle improves rapidly during the initial training phase and achieves convergence after approximately 200 episodes. This demonstrates the model’s capacity to rapidly adapt to unilateral control tasks while maintaining safety and efficiency.

In contrast, as shown in Figure 6b, the middle vehicle—responsible for managing bilateral interactions with both leading and following vehicles—requires approximately 300 episodes to achieve convergence. This highlights the framework’s capacity to address the added complexity of bilateral control scenarios effectively.

As training progresses, the decreasing variance in the reward curves reflects enhanced stability and robustness of the model. Overall, the results confirm that the framework successfully learns and implements effective control strategies, with both vehicle types achieving reliable convergence and stable performance within 1000 training episodes.

4. Simulation Setup and Performance Evaluation

This section systematically evaluates the performance of TD3-BCM in mixed traffic environments through numerical simulations. Building on the TD3-BCM established in Section 3 and the stability and performance evaluation metrics introduced in Section 4, the study analyzes the impact of different autonomous vehicle (AV) penetration rates on traffic stability, safety, efficiency, and energy consumption, validating the effectiveness of TD3-BCM in optimizing complex traffic conditions.

4.1. Simulation Setup

To thoroughly evaluate the performance of TD3-BCM in mixed traffic environments, this study selects five representative scenarios, as illustrated in Figure 7, to compare the proposed TD3-BCM model against the traditional BCM and unidirectional CACC models. The scenarios encompass various autonomous vehicle (AV) penetration levels and control strategies. Specifically, Scenario 1 uses three CACC-controlled AV sub-chains, each consisting of one tail vehicle and three middle vehicles, with an AV penetration rate of 40%. Scenario 2 uses three BCM-controlled AV sub-chains, each consisting of one ACC-controlled tail vehicle and three BCM-controlled middle vehicles, with an AV penetration rate of 40%. Scenario 3 uses three TD3-BCM-controlled AV sub-chains, each consisting of one TD3-controlled tail vehicle and three TD3-BCM-controlled middle vehicles, with an AV penetration rate of 40%. Additionally, Scenarios 4 and 5 increase the AV penetration rate to 50% and 70%, respectively. The configuration of AV sub-chains in these scenarios follows the design principles outlined by Li et al. [60].

The primary objective of these scenarios is to compare the performance of the CACC, BCM, and TD3-BCM models under the same AV penetration rate, and to examine the performance variations of the TD3-BCM model under different penetration rates. These experiments not only comprehensively assess the TD3-BCM model’s benefits—such as improved stability, safety, efficiency, and energy savings—but also examine how varying AV penetration rates influence the stability of mixed traffic flow, further validating the performance advantages of TD3-BCM compared to other control strategies.

The selected scenarios are representative of common dynamic patterns observed in mixed traffic, allowing for realistic simulation and comparative analysis [26,51]. Although this study focuses on the analysis of these five typical scenarios, other scenarios or different configurations of AV sub-chains can be expanded based on research needs to further verify the adaptability and robustness of TD3-BCM under various traffic conditions. Since the selected scenarios have already demonstrated the performance variations of the model under different penetration rates, this study does not include detailed discussions of additional scenarios, but further design and experimentation with other scenarios can be conducted in the future based on research needs.

Figure 7. Scenarios for Mixed Traffic Simulations with TD3-BCM Vehicle Chains.

Figure 7. Scenarios for Mixed Traffic Simulations with TD3-BCM Vehicle Chains.

All scenarios derive the lead vehicle’s velocity from real-world driving data (NGSIM I-80) in the testing dataset, introducing realistic traffic perturbations to assess the stability improvements of autonomous vehicles (AVs) in mixed traffic. Human-driven vehicles (HDVs) follow the IDM model, with parameters calibrated based on [61] to ensure realistic human driving behavior. In Scenario 1, the CACC model adopts the methodology and parameter settings from [38]. In Scenario 2, AVs utilize the traditional Bilateral Control Model (BCM) and Adaptive Cruise Control (ACC) model, both implemented based on [41]. The parameter settings for Scenarios 3–5 are derived from the optimized results of the model training process detailed in Section 3.3.3.

4.2. Stability Results

The stability analysis adopts predefined metrics to evaluate the system’s ability to suppress disturbances caused by external factors or oscillations induced by human-driven vehicles (HDVs). The analysis includes chain stability analysis, modular stability analysis, cumulative damping ratios, and traffic dynamics analysis. These metrics evaluate the disturbance propagation and the system’s stability under different autonomous vehicle (AV) penetration rates in the TD3-BCM model.

4.2.1. Theoretical Stability Analysis

The stability of a mixed-vehicle platoon depends on the autonomous vehicle (AV) penetration rate, denoted by $\varphi$. Stability is achieved when the AV penetration rate exceeds a critical threshold, ${\varphi }_{\min }$, derived from the system’s transfer functions [62]. This relationship is given as Equation (17):

$$\varphi >{\varphi }_{\min }={\displaystyle \frac{1-G_{\mathrm{HDV}}\left(s\right)}{G_{\mathrm{AV}}\left(s\right)-G_{\mathrm{HDV}}\left(s\right)}}$$

(17)

where

• $\varphi$ is the AV penetration rate.
• $G_{\mathrm{HDV}}\left(s\right)={\displaystyle \frac{0.5}{s+0.5}}$ is the transfer function for HDVs.
• $G_{\mathrm{AV}}\left(s\right)={\displaystyle \frac{1.0+0.2s}{s^2+0.8s+1.0}}$ is the transfer function for AVs.

By analyzing the system across low, medium, and high frequency ranges ($\omega =0.01\sim 10$) [62], the critical AV penetration rate required to ensure chain stability is found to be ${\varphi }_{\min }=32.24\%$. This value represents the minimum proportion of AVs necessary to maintain system stability. Additionally, the AV penetration rates in the five scenarios meet the required threshold for chain stability, ensuring that the system remains stable across all scenarios.

Modular string stability metrics evaluate stability across sub-chains in mixed traffic systems. This framework decomposes the system into autonomous vehicle (AV) and human-driven vehicle (HDV) sub-chains, defining stability conditions for each segment and their interactions. The overall modular stability is defined as Equation (18):

$$S_{\mathrm{modular}}=S_{\mathrm{AV}}·S_{\mathrm{HDV}}·S_{\mathrm{boundary}}$$

(18)

where

• $S_{\mathrm{AV}}$ is the stability of the AV sub-chain, which is defined as the inverse of the AV sub-chain’s speed.
• $S_{\mathrm{HDV}}$ is the stability of the HDV sub-chain, which is defined as the inverse of the HDV sub-chain’s speed.
• $S_{\mathrm{boundary}}$ is the stability of the boundary between AV and HDV sub-chains, which is equal to the reciprocal of the speed of boundary vehicles (AV and HDV).

If $S_{\mathrm{modular}}>1$, the system is deemed stable, meaning the interactions between AV and HDV sub-chains do not compromise overall traffic stability [19,60].

Table 3. Stability Metrics for Different Scenarios.

Scenario	Scenario 1	Scenario 2	Scenario 3	Scenario 4	Scenario 5
$S_{\mathrm{AV}}$	0.814	0.920	1.056	1.066	1.130
$S_{\mathrm{HDV}}$	0.870	0.967	1.076	1.145	1.121
$S_{\mathrm{boundary}}$	0.860	1.015	1.098	1.153	1.143
$S_{\mathrm{modular}}$	0.556	1.015	1.132	1.163	1.182
$S_{\mathrm{AV}}$ (Std)	0.061	0.057	0.039	0.033	0.023
$S_{\mathrm{HDV}}$ (Std)	0.085	0.068	0.067	0.056	0.043
$S_{\mathrm{boundary}}$ (Std)	0.046	0.044	0.037	0.021	0.016
$S_{\mathrm{modular}}$ ((Std)	0.087	0.081	0.077	0.065	0.054

shows the stability metrics—$S_{\mathrm{AV}}$, $S_{\mathrm{HDV}}$, $S_{\mathrm{boundary}}$, and $S_{\mathrm{modular}}$—for five scenarios, based on the averages of 50 test dataset results. Scenarios 1, 2, and 3 represent the CACC, BCM, and TD3-BCM models at 40% AV penetration. In Scenario 3, the TD3-BCM model outperforms the CACC and BCM models in $S_{\mathrm{AV}}$ and $S_{\mathrm{HDV}}$, with $S_{\mathrm{AV}}$ increasing from 0.814 to 1.056 and $S_{\mathrm{HDV}}$ from 0.870 to 1.076. Boundary stability ($S_{\mathrm{boundary}}$) is also higher in TD3-BCM (1.098) compared to CACC (0.860) and BCM (1.015), indicating smoother transitions between AVs and HDVs and reduced disturbances in mixed traffic flow.

As AV penetration increases, the stability of the TD3-BCM model improves. In Scenario 4 (50% penetration), $S_{\mathrm{AV}}$ and $S_{\mathrm{HDV}}$ increase to 1.066 and 1.145, respectively, while $S_{\mathrm{boundary}}$ reaches 1.153. In Scenario 5 (70% penetration), TD3-BCM shows even higher stability, particularly in modular stability ($S_{\mathrm{modular}}$), which rises to 1.182. These results confirm that higher AV penetration improves the stability of both AV and HDV sub-chains, and the TD3-BCM model enhances the overall stability of the system.

As illustrated in Figure 8, the TD3-BCM model exhibits consistent improvements in all key stability indicators—SAV, SHDV, Sboundary, and Smodular—as AV penetration increases. The small standard deviations—reflected by narrow error bars—indicate stable and reliable simulation outcomes, further validating the effectiveness of TD3-BCM in improving mixed traffic stability, especially in high AV penetration scenarios.

4.2.2. Cumulative Damping Ratio

The cumulative damping ratio quantifies a vehicle’s capacity to attenuate traffic disturbances. This metric, originally proposed by Ploeg et al. [63], approximates the disturbance dissipation capability by computing the 2-norm of a finite-length acceleration sequence.

The cumulative damping ratio is calculated as follows:

$$D_n={\displaystyle \frac{\parallel a_n{\left(t\right)\parallel }_2}{\parallel a_0{\left(t\right)\parallel }_2}}={\displaystyle \frac{{\left({\sum }_{t=0}^T{a}_n{\left(t\right)}^2\right)}^{\frac{1}{2}}}{{\left({\sum }_{t=0}^T{a}_0{\left(t\right)}^2\right)}^{\frac{1}{2}}}}$$

(19)

where

• $D_n$ represents cumulative damping ratio.
• $a_n\left(t\right)$ represents the acceleration of the n-th vehicle at time t.
• $a_0\left(t\right)$ represents the acceleration of the lead vehicle at time t.
• T denotes the total simulation time.

The cumulative damping ratio quantifies the system’s ability to attenuate traffic disturbances over time. A value of $D_n>1$ indicates amplification of disturbances, while $D_n<1$ suggests effective suppression, contributing to greater system stability.

As illustrated in Figure 9, the comparison of scenarios 1, 2, and 3 indicates that TD3-BCM performs better than the CACC and BCM models in improving traffic stability. With the increase in AV penetration, the cumulative damping ratio of TD3-BCM shows a significant decline, reaching its lowest point at the 30th vehicle, significantly enhancing overall stability. At the same time, in scenarios 2, 3, 4, and 5, a pronounced drop is observed at the 12th and 22nd vehicles, suggesting that the AV sub-chain effectively absorbs localized disturbances. This phenomenon underscores the pivotal role of the AV sub-chain under bilateral control in enhancing system stability, with the TD3-based deep reinforcement learning model exhibiting superior performance.

• Scenario 1: The ratio decreases from 0.534 at vehicle 2 to 0.338 at vehicle 30, but with noticeable fluctuations, indicating that the unidirectional feedback of CACC limits its ability to suppress disturbances, leading to persistent oscillations across the chain.
• Scenario 2: The ratio drops more rapidly, reaching 0.060 at vehicle 30, demonstrating improved disturbance suppression with bidirectional feedback. However, residual oscillations persist due to the lack of adaptive adjustment, limiting overall stability.
• Scenario 3: The damping ratio further reduces to 0.048 at vehicle 30, highlighting the effectiveness of TD3-BCM in dynamically mitigating disturbances and enhancing overall stability compared to traditional BCM.
• Scenario 4: With an increased AV penetration rate, the ratio declines to 0.035 at vehicle 30, reflecting improved coordination among AVs and more efficient disturbance absorption across the vehicle chain.
• Scenario 5: The damping ratio reaches its lowest level, below 0.030 at vehicle 30, indicating near elimination of traffic oscillations. This confirms that higher AV penetration combined with TD3-BCM maximizes stability and minimizes disturbance propagation.

Overall, the results demonstrate the superior capability of the TD3-BCM model to enhance stability under varying AV penetration rates, offering practical implications for the design of adaptive control mechanisms in mixed traffic systems.

4.2.3. Traffic Dynamics

Traffic dynamics across the five scenarios demonstrate significant stability improvements with increasing AV penetration, as evidenced by detailed analyses of position, speed, acceleration, and jerk metrics shown in Figure 10.

In Figure 10a (Vehicle Distane), the leading vehicle in all scenarios experiences external perturbations, generating traffic waves. In Scenario 1, disturbances persist throughout the vehicle chain due to the unidirectional feedback structure of CACC, resulting in pronounced fluctuations among trailing vehicles. In Scenario 2, the bidirectional control of BCM reduces disturbance propagation, but residual oscillations remain due to the lack of adaptive adjustments. In Scenarios 3–5, increasing AV penetration gradually smooths vehicle trajectories. Scenario 5 exhibits minimal positional disturbances, confirming the TD3-BCM’s capacity to absorb and suppress traffic oscillations.

Figure 10b (Speed Profiles) further supports this trend. Compared to CACC, BCM and TD3-BCM significantly reduce speed fluctuations. In Scenario 1, vehicle 30 exhibits speed oscillations ranging from 2.1 m/s to 9.7 m/s (amplitude: 7.6 m/s), demonstrating severe velocity fluctuations. In Scenario 2, the amplitude reduces to 5.7 m/s, showing improved stability. In Scenario 3, the fluctuation drops further to 3.6 m/s, while in Scenario 4 and Scenario 5, the speed variation narrows to 2.4 m/s and 1.2 m/s, respectively. These results confirm that increasing AV penetration, especially with TD3-BCM, effectively stabilizes velocity fluctuations and enhances traffic flow smoothness.

Figure 10c (Acceleration Profiles) illustrates the effects of different control strategies on acceleration variations. In Scenario 1, vehicle 30’s acceleration fluctuates between $-1.75$ m/s² and $1.14$ m/s², indicating instability under unilateral control. In Scenario 2, the fluctuation range narrows to $-1.06$ m/s² to $0.89$ m/s², reflecting improved stability with BCM. In Scenario 3, acceleration variations decrease to $-0.58$ m/s² to $0.42$ m/s², while in Scenario 4 and Scenario 5, they further reduce to $-0.37$ m/s² to $0.25$ m/s² and $-0.21$ m/s² to $0.18$ m/s², respectively. In Scenario 5, vehicles 10-30 experience minimal acceleration variations, indicating enhanced system stability with higher AV penetration.

Figure 10d (Jerk Profiles) presents jerk profiles—the rate of change of acceleration—which directly affect driving comfort. In Scenario 1, vehicle 30’s jerk fluctuates significantly between $-1.82$ m/s³ and $1.94$ m/s³, implying frequent abrupt acceleration and deceleration events, reducing driving comfort. In Scenario 2, jerk variation decreases to $-1.23$ m/s³ to $1.29$ m/s³, showing some improvement. In Scenario 3, jerk variation further shrinks to $-0.74$ m/s³ to $0.82$ m/s³, while in Scenario 4 and Scenario 5, it continues to decrease to $-0.48$ m/s³ to $0.57$ m/s³ and $-0.26$ m/s³ to $0.31$ m/s³, respectively. The reduction in jerk fluctuation confirms that higher AV penetration and TD3-BCM significantly improve ride comfort.

TD3-BCM consistently outperforms both CACC and BCM by suppressing traffic oscillations, stabilizing speed and acceleration variations, and enhancing driving comfort. Its effectiveness increases notably with higher AV penetration rates.

4.3. Performance Results

The performance of the mixed-vehicle chain is evaluated using three key metrics: efficiency, safety, and energy efficiency. These metrics serve as quantitative indicators to assess the impact of the TD3-BCM on traffic flow dynamics.

4.3.1. Efficiency

Driving efficiency is evaluated by analyzing inter-vehicle time gaps, which serve as a proxy for overall traffic flow performance. The comparison of the five scenarios demonstrates the impact of different control strategies on traffic efficiency, as shown in Figure 11:

• In Scenario 1, using the CACC model, there is significant fluctuation in the time gaps, especially at the head of the vehicle chain, with a mean time gap of 2.64 s and a standard deviation of 1.44 s. This indicates that, due to the limitations of unidirectional control, the spacing between vehicles fluctuates considerably, leading to unstable traffic flow.
• In Scenario 2, after the introduction of the BCM model with bidirectional feedback, fluctuations are reduced, and the mean time gap decreases to 2.1 s with a reduced standard deviation, indicating improved traffic stability.
• In Scenario 3, with the introduction of TD3-BCM, the stability of time gaps improves further. The average time gap for vehicles 2 to 10 stabilizes around 2 s, with a standard deviation reduced to 0.34 s, ensuring smoother traffic flow.
• In Scenario 4, as AV penetration increases, the mean time gap for vehicles beyond the 15th position decreases to 1.94 s, with a standard deviation below 0.2 s, highlighting the continued advantage of TD3-BCM in higher penetration scenarios, ensuring even more stable traffic flow.
• In Scenario 5, with further increases in AV penetration, the mean time gap reaches its lowest level across the vehicle chain. The average time gap for mid- and tail-end vehicles remains between 1.84 s and 0.80 s, with a standard deviation falling below 0.036 s, demonstrating the superior performance of TD3-BCM in high AV penetration scenarios.

Comparing Scenarios 1, 2, and 3, it is clear that TD3-BCM outperforms both CACC and BCM in reducing time gap fluctuations and improving traffic flow stability. Particularly in Scenarios 2, 3, 4, and 5, as AV penetration increases, TD3-BCM achieves progressively smaller time gaps, indicating that TD3-BCM’s effectiveness continues to improve as AV penetration rises. It is important to note that in Scenarios 2, 3, 4, and 5, time gap oscillations are observed, which is mainly due to the fact that the time gaps in the autonomous vehicle sub-chain are smaller than those in the traditional IDM (Intelligent Driver Model) vehicle-following model. The autonomous vehicle sub-chain plays a key role in reducing time gaps and optimizing traffic flow, further enhancing the stability and efficiency of traffic.

4.3.2. Safety

Time-to-Collision (TTC) is a key metric for quantifying collision risk and assessing traffic safety. A higher TTC value indicates a longer time gap between vehicles, effectively reducing the collision risk and enhancing traffic safety [64]. It is defined as follows:

$${\mathrm{TTC}}_n\left(t\right)=\left\{\begin{array}{cc}{\displaystyle \frac{d_n\left(t\right)}{\Delta v_n\left(t\right)}},\hfill & \Delta v_n\left(t\right)<0\hfill \\ \infty ,\hfill & \Delta v_n\left(t\right)\ge 0\hfill \end{array}\right.$$

(20)

where

• $d_n\left(t\right)$: Distance between the vehicle and its preceding vehicle at time t.
• $\Delta v_n\left(t\right)$: Relative velocity between vehicles.

Table 4. Comparison Table of TTC Indicators.

TTC Threshold	1 s, 1.5 s, 2 s	2.5 s	3 s
Scenario 1—Mean	0.0000	0.2453	1.4385
Scenario 1—Std	0.0000	0.7341	2.8119
Scenario 2—Mean	0.0000	0.3186	1.6214
Scenario 2—Std	0.0000	0.6453	2.7348
Scenario 3—Mean	0.0000	0.3924	1.8749
Scenario 3—Std	0.0000	0.5632	2.5231
Scenario 4—Mean	0.0000	0.4657	2.1874
Scenario 4—Std	0.0000	0.4712	2.3142
Scenario 5—Mean	0.0000	0.5432	2.6951
Scenario 5—Std	0.0000	0.3951	2.0713

presents the mean TTC values and standard deviations for Scenarios 1–5 at thresholds of 1 s, 1.5 s, and 2 s (high collision risk thresholds) as well as 2.5 s and 3 s (low collision risk thresholds), reflecting the changes in collision risk and traffic flow stability under different models.

From Table 4, it can be seen that at high collision risk thresholds of 1 s, 1.5 s, and 2 s, as well as lower collision risk thresholds of 2.5 s and 3 s, the TTC values Table 4 for the CACC (Scenario 1), BCM (Scenario 2), and TD3-BCM (Scenarios 3, 4, and 5) models are 0, indicating that at these higher collision risk thresholds, the relative safety between vehicles is high and no collision risk is detected. This is attributed to the design of minimal headway and the optimization of the TD3 reward function.

As the TTC threshold increases from 2 s to 3 s, the TTC values for all three models gradually increase, indicating that the collision response time is extended, thus improving safety. At a TTC of 1.5 s, the TTC value for the TD3-BCM (Scenario 3) model is 0.3924, while the TTC values for the CACC and BCM models are 0.2453 and 0.3186, respectively, showing that TD3-BCM has a higher collision response time, thus enhancing safety. Furthermore, the standard deviation for TD3-BCM (Scenario 3) at the 1.5-s TTC threshold is 0.5632, compared to 0.7341 for CACC (Scenario 1) and 0.6453 for BCM (Scenario 2), indicating improved system stability.

Additionally, the results for Scenarios 3, 4, and 5 show that as the penetration rate of autonomous vehicles (AVs) increases, the TTC value for TD3-BCM continues to increase, further improving the safety of the system. At a 3-s TTC threshold, the TTC value for TD3-BCM (Scenario 5) is 2.6951, significantly higher than the TTC value for Scenario 3 (TD3-BCM) at 1.8749, and also higher than the TTC value for Scenario 4 (TD3-BCM) at 2.1874, indicating that safety improves as the penetration rate increases. Meanwhile, the standard deviation decreases progressively, with the standard deviation for Scenario 5 being 0.3951, much lower than in other scenarios, indicating that as the penetration rate increases, the system stability is significantly enhanced.

4.3.3. Energy Savings

Smoother acceleration and deceleration profiles significantly reduce fuel consumption, particularly under high AV penetration conditions. Energy efficiency is assessed through a regression-based fuel consumption model [65], expressed as follows:

$$e_n\left(t\right)=exp\left[\sum _{i=0}^3\sum _{j=0}^3{K}_{ij}\left({\left|v_n\left(t\right)\right|}^i\right)\left({\left|a_n\left(t\right)\right|}^j\right)\right]$$

(21)

where

• $K_{ij}$: Empirical regression coefficients derived from real-world data [66].
• $a_n\left(t\right)$: Acceleration of the n-th vehicle at time t.
• $v_n\left(t\right)$: Speed of the n-th vehicle at time t.

By analyzing the relationship between vehicle speed, acceleration, and fuel consumption, this model evaluates the impact of CACC, BCM, and TD3-BCM on energy efficiency, particularly the variation in the energy-saving effect of TD3-BCM as AV penetration gradually increases.

As shown in Figure 12, the variations in fuel consumption across different scenarios are as follows:

• In Scenario 1, using the CACC model, fuel consumption remains the highest across the vehicle chain. The lead vehicle’s average consumption is 0.99 mL/s, with trailing vehicles showing significant fluctuations, and a standard deviation of 0.073 mL/s. This indicates substantial inefficiencies due to inconsistent driving behavior.
• In Scenario 2, the introduction of bidirectional control with the BCM model reduces fluctuations, leading to a moderate decrease in fuel consumption. The fuel consumption of the mid- and tail-end vehicles stabilizes at approximately 0.88 mL/s, with a standard deviation below 0.05 mL/s, indicating an improvement in traffic efficiency.
• In Scenario 3, after deploying TD3-BCM, fuel consumption further decreases and stabilizes at 0.85 mL/s, showing the advantage of TD3-BCM in optimizing fuel consumption.
• In Scenario 4, with the increase in AV penetration, fuel consumption decreases further to approximately 0.82 mL/s, with a standard deviation below 0.04 mL/s. This shows that TD3-BCM continues to demonstrate better stability and lower fuel consumption in higher penetration scenarios.
• In Scenario 5, with further increases in AV penetration, fuel consumption reaches its lowest level across the vehicle chain. The average consumption of mid- and tail-end vehicles remains between 0.78 mL/s and 0.80 mL/s, with a standard deviation falling below 0.036 mL/s, highlighting the superior performance of TD3-BCM in high AV penetration scenarios.

Comparing Scenarios 1, 2, and 3, the results demonstrate that TD3-BCM outperforms both CACC and BCM in reducing fuel consumption. In Scenarios 4 and 5, as AV penetration increases, TD3-BCM’s effectiveness continues to improve, further reducing fuel consumption. Additionally, in Scenarios 2, 3, 4, and 5, especially at vehicles 12 and 22, a significant reduction in time gaps is observed, indicating that the autonomous vehicle sub-chain with bilateral control plays a key role in enhancing traffic flow stability and minimizing disturbances, with TD3-BCM showing superior performance over BCM.

5. Conclusions and Discussion

5.1. Conclusions

This study investigates the coexistence of autonomous vehicles (AVs) and human-driven vehicles (HDVs) in mixed traffic environments, and introduces an enhanced Bilateral Control Model—TD3-BCM—aimed at improving traffic system stability, efficiency, safety, and energy performance. This model builds on our previous research [25], extending it to the mixed traffic flow context and further optimizing the interaction between autonomous and human-driven vehicles, thereby improving the coordination and overall performance of the traffic flow system. TD3-BCM enhances vehicle spacing control and collision response through a tailored TD3 network architecture and reward function, thereby improving overall traffic flow efficiency and system stability. Multiple analyses—including theoretical modeling, cumulative damping ratio assessment, and traffic dynamics evaluation—demonstrate that TD3-BCM significantly enhances system stability. With higher AV penetration rates, TD3-BCM effectively suppresses traffic fluctuations, leading to a steady improvement in system stability—reflected by an increase in modular stability from 1.132 to 1.182, indicating that the system performs more stably under higher penetration rates. In terms of performance, TD3-BCM at a 40% AV penetration rate significantly outperforms both CACC (unilateral control) and traditional BCM (bilateral control) in traffic efficiency, safety, and energy efficiency. Specifically, TD3-BCM increases the average speed of trailing vehicles by 12.6%, reduces speed fluctuations by 36.2%, and shortens the average headway by 19.0%, significantly improving traffic flow efficiency. At the same time, Time-to-Collision (TTC) improves by 87.3%, effectively reducing collision risks. Regarding energy efficiency, TD3-BCM reduces fuel consumption by 14.8% and decreases braking fluctuations by 41.5%. Moreover, as AV penetration increases, TD3-BCM exhibits progressively stronger optimization effects: traffic oscillations are nearly eliminated, fuel savings reach 19.7%, and inter-vehicle spacing is further reduced, demonstrating clear advantages in enhancing the traffic safety and stability of mixed traffic flows.

This study makes several key contributions: (1) it introduces deep reinforcement learning (TD3) into the bilateral control framework (TD3-BCM), enabling dynamic adaptation in complex mixed traffic environments; (2) it quantifies the impact of autonomous vehicle (AV) penetration on traffic stability and establishes a critical stability threshold (${\varphi }_{\min }$) for maintaining stable traffic flow; and (3) it offers a new theoretical perspective on the relationship between AV penetration and system stability. Practically, TD3-BCM improves traffic stability, reduces oscillations, and enhances fuel efficiency, making it well-suited for highways and urban roads. It also supports intelligent traffic management, optimizes road utilization, and promotes sustainable transportation, with the potential to alleviate congestion, reduce emissions, and improve safety.

5.2. Discussion

Although TD3-BCM performs well in numerical simulations, its practical applicability in real-world traffic systems warrants further investigation. First, the critical penetration rate threshold is likely influenced by infrastructure characteristics, driver behavior heterogeneity, and environmental variability, requiring calibration using broader real-world datasets. Second, this study centers on longitudinal control, whereas more complex behaviors—such as lane changes and intersection coordination—require further exploration to assess TD3-BCM’s adaptability to diverse and dynamic scenarios.

Moreover, future research should incorporate real-world vehicle testing to validate the effectiveness of TD3-BCM in various traffic environments. Such validation efforts will help refine reinforcement learning strategies, accelerating model convergence and enhancing computational efficiency. Additionally, multi-modal traffic control—integrating TD3-BCM with traffic signal coordination, lane assignment, and other optimization strategies—remains a promising direction for future research aimed at achieving comprehensive improvements in intelligent transportation systems.

This study is based on simplified traffic scenarios, and future research could explore the model’s applicability in more complex traffic environments, considering driver behaviors (e.g., lane changes) and real-world uncertainties (e.g., sensor noise, communication delays, and packet loss). Integrating these factors will help assess the robustness and scalability of TD3-BCM in addressing real-world challenges in mixed traffic systems. Furthermore, comparing the results with existing literature will provide clearer context for interpreting the findings and offer potential directions for improving the model in future iterations.