A Multi-Agent Cooperative Decision-Making Method with Risk-Aware Dual Masking for Highway On-Ramp Merging

Abstract

Highway on-ramp merging in mixed-traffic environments is a high-dimensional combinatorial optimization problem that requires joint consideration of driving safety, traffic efficiency, and multi-agent coordination. To address this issue, this paper proposes BQDOM, namely Beam-searched Q-learning with Dual-mask for On-ramp Merging, a joint-action scoring-based multi-agent Q-learning method. The proposed method first constructs historical observation sequences from environmental perception information and uses them to generate trajectory embeddings and short-term future trajectory predictions. The trajectory embeddings enhance individual action-value estimation, while the predicted trajectories support action mask generation. By integrating trajectory embeddings with real-time observations, a shared individual Q-network evaluates the action values of each agent. Meanwhile, future trajectories are used to construct a risk mask, which is combined with a legality mask to form a dual action mask that filters high-risk and invalid actions. A fixed-width Beam Search strategy is then adopted to build candidate joint-action sequences. Key-action weights are assigned according to vehicle spatial positions, and candidate joint actions are iteratively scored. The joint action with the highest score is selected for execution, and the executed joint action is used to construct a system-level temporal-difference target for updating the action-value networks. Experiments in a highway-based simulation environment compare BQDOM with MAA2C, MAPPO, and MAACKTR across the Easy, Normal, and Hard scenarios. BQDOM achieves the best overall performance among the compared methods, reducing the mean collision rate relative to the best baseline for this metric by 75.00%, 22.22%, and 18.18%, respectively. These results suggest a favorable balance between safety and traffic efficiency within the evaluated simulation scenarios.

1. Introduction

Highway on-ramp merging is one of the typical bottleneck scenarios in traffic systems. Vehicles are required to complete speed adjustment and lane-changing maneuvers within a limited distance; inappropriate decisions may easily lead to a decline in overall efficiency, an increase in conflict risks, and even the propagation of congestion [1,2]. With the development of vehicle–infrastructure cooperation and autonomous driving technologies, connected and automated vehicles (CAVs), which are equipped with vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication and perception capabilities, have received increasing attention [3,4]. By sharing the states of surrounding vehicles and road information, CAVs are expected to achieve safer and more efficient cooperative driving in complex interactions, thereby improving overall traffic efficiency and reducing accident risks. However, in mixed-traffic environments, CAVs coexist with human-driven vehicles (HDVs), whose stochastic and heterogeneous behaviors significantly amplify interaction uncertainty [5,6]. Meanwhile, multi-lane on-ramp merging involves coupled longitudinal speed control and lateral lane-changing decisions [7]. For N agents sharing a discrete action set A, the full joint action space contains ${\left|A\right|}^N$ combinations. With the five discrete actions adopted in this study, this yields $5^N$ combinations before action masking, illustrating the exponential growth of exhaustive joint-action enumeration and motivating a computationally manageable search strategy for online decision-making [8].

Existing studies on on-ramp merging and multi-vehicle cooperative control have mainly adopted traditional rule-based or optimization-based methods, both of which exhibit notable limitations [9,10]. Rule-based methods are feasible in simple scenarios, but their practicality rapidly deteriorates when facing complex environments [11,12]. Such methods can hardly characterize the randomness and heterogeneity of vehicle interactions accurately, nor do they possess sufficient macroscopic traffic-flow regulation capability. As a result, they tend to fall into local optima and struggle to improve global efficiency [13]. Optimization-based methods rely on accurate vehicle dynamics and human driving behavior models, and they usually require online receding-horizon optimization at each time step, resulting in considerable computational overhead [14]. In multi-vehicle and multi-constraint scenarios, this computational burden is more likely to become a bottleneck. Moreover, such methods have limitations in long-term planning capability, and their adaptability and real-time performance are insufficient when inputs and outputs fluctuate in dynamic traffic environments [15]. Therefore, in recent years, deep reinforcement learning (DRL), owing to its strong self-learning capability, has been widely applied to autonomous driving decision-making tasks [16,17] and has gradually been extended from single-vehicle decision-making to multi-vehicle cooperation under the framework of multi-agent reinforcement learning (MARL) [18,19].

Although MARL provides a unified learning-based modeling approach for multi-vehicle cooperative merging, directly applying it to multi-lane on-ramp merging still faces several key bottlenecks. First, cooperative benefits and individual benefits often constrain each other. If the reward allocation or coordination mechanism is improperly designed, unstable phenomena may occur, such as “individual self-interest leading to overall inefficiency” or “excessive altruism impairing individual feasibility” [20,21]. Second, the uncontrollability and uncertainty of HDVs in mixed traffic make decisions based solely on instantaneous observations more prone to misjudgment. Many studies have pointed out that introducing the prediction of surrounding vehicles’ intentions or trajectories can improve the quality and safety of decision information in highly interactive tasks [22,23]. Third, the excessively large joint action space leads to high costs in joint-action search. In safety-critical highway on-ramp merging scenarios, if high-risk actions are not explicitly filtered, training exploration will be slowed down by a large number of invalid samples and may even result in a higher collision rate [24,25]. Fourth, although centralized training with decentralized execution (CTDE) can use global information during training to facilitate coordination, action selection during execution is typically performed locally [26,27]. In strongly coupled on-ramp merging situations, explicit comparison of candidate joint actions can provide an additional means of coordinating gap creation, yielding, and merging maneuvers.

To address the above issues, this paper proposes BQDOM, a joint-action scoring-based multi-agent cooperative decision-making method for mixed-traffic flow. Based on the idea of value decomposition, BQDOM realizes scalable cooperative control through prediction-enhanced observation, dual action masking, and finite-width joint-action search. Specifically, the proposed method first encodes historical observation sequences to generate trajectory embeddings and short-term future trajectories, thereby incorporating forward-looking information into the decision-making process. Then, a dual action mask mechanism is designed to filter invalid and high-risk actions, constraining model exploration within a safe and feasible subspace. Finally, a system-level scoring function with key-action weighting is constructed, and fixed-width Beam Search is employed to approximate the optimal joint action, effectively mitigating the online decision-making difficulty caused by the exponential explosion of the multi-agent joint action space.

Beam Search is adopted because it fits the masked, joint-action scoring structure of BQDOM. The dual action mask defines the candidate actions for each vehicle, while the scoring function evaluates candidate action combinations as the joint action is constructed vehicle by vehicle. Exhaustive enumeration becomes costly as the number of vehicles increases, whereas retaining only a single partial candidate may prematurely discard useful cooperative alternatives. By retaining up to B candidates at each expansion stage, Beam Search preserves multiple alternatives while bounding the number of retained candidates. Therefore, Beam Search offers a practical balance between search breadth and computational cost, making it well suited to the joint-action scoring architecture of BQDOM.

In summary, the main contributions of this paper are as follows.

• A dual action masking mechanism is developed for mixed-traffic on-ramp merging. By combining rule-based legality constraints with trajectory-prediction-based risk filtering, it excludes invalid and predicted high-risk actions before joint-action search, reducing the candidate action space considered for cooperative decision-making.
• A merging-oriented joint decision-making mechanism is developed by integrating key-action weighting, joint-action scoring, and fixed-width Beam Search. The weighting scheme emphasizes merging actions of key ramp vehicles and assistive actions of mainline vehicles, while the search compares alternative action combinations to support coordinated merging with a bounded number of retained candidates.
• The framework is evaluated through comparative and ablation experiments in a Highway-based mixed-traffic on-ramp merging environment. Experiments across different traffic densities and CAV penetration rates assess its safety and efficiency, while ablation experiments examine the contributions of dual action masking, key-action weighting, and joint-action scoring under the tested simulation conditions.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 introduces the proposed joint-action scoring framework. Section 4 presents the experimental design and result analysis. Section 5 concludes the paper and discusses future research directions.

2. Related Work

Existing studies on safety constraints, forward-looking perception, and multi-vehicle cooperative decision-making in highway on-ramp merging under mixed-traffic flow mainly focus on the following categories: rule-based and optimization-based methods, supervised learning and trajectory prediction methods, deep reinforcement learning methods, cooperative decision-making and value decomposition methods, and action masking and online supervision methods for safety constraints. Accordingly, this section reviews the related literature from these perspectives.

2.1. Rule-Based and Optimization-Based Methods

Traditional studies on on-ramp merging and bottleneck management have largely relied on rule-based control and optimization-based control frameworks. Typical technical routes include ramp metering, variable speed limits (VSLs), and their coordinated optimization within model predictive control (MPC) frameworks. For example, local feedback-based ramp control and its adaptive extensions can suppress mainline congestion and improve traffic efficiency to a certain extent [11]. VSL methods mitigate congestion and bottleneck queues by regulating the speed distribution on the mainline and can be integrated with ramp control under an MPC framework for coordinated optimization [12]. In studies more closely related to the merging mechanism, some works actively shape acceptable merging gaps to guide vehicles into the mainline. Scarinci et al. proposed periodically compressing mainline traffic to create merging gaps and induce vehicles to enter them [28]. Chen et al. introduced the concept of batch merging into the merging process to reduce the additional gap loss caused by lane changes [29].

Although these methods have significant advantages in terms of interpretability, engineering deployability, and stability, their effectiveness highly depends on assumptions or approximate models of traffic participants, especially human-driven vehicles (HDVs). In strongly interactive mixed-traffic scenarios, the behavioral heterogeneity and stochasticity of HDVs can easily lead to model mismatch and performance degradation. In addition, online optimization under multi-lane and multi-agent coupling often faces dimensional expansion and real-time computational challenges, making it difficult to cover combinatorial conflicts and cooperative games in high-density merging scenarios [30].

2.2. Supervised Learning and Trajectory Prediction

In lane-changing and merging decision-making tasks, supervised learning is commonly applied to key modules such as intention recognition, trajectory prediction, and behavior classification, thereby compensating for the inability of instantaneous observations alone to capture interaction trends. For example, Wang et al. combined long short-term memory (LSTM) networks with conditional random fields (CRFs) to construct a human-like maneuver decision model that characterizes sequential maneuver selection during driving [31]. Liu et al. employed support vector machines to build a lane-changing decision model for driving behavior recognition [32]. Other studies have explicitly modeled driving style differences, enabling lane-changing decisions to exhibit individualized style-aware characteristics [33]. Meanwhile, trajectory prediction and intention inference techniques have also been widely used to provide forward-looking environmental information for downstream decision-making, thereby improving the ability to identify potential conflicts, such as assisting in the evaluation of gap availability and interaction intensity through short-horizon prediction [34].

However, supervised learning methods have obvious limitations. On the one hand, their performance is highly dependent on high-quality annotated data or rule-generated labels, which are costly to obtain and limited in scenario coverage. On the other hand, these models are sensitive to distribution shift. Once traffic density, driving style, or interaction structure changes, their generalization ability may deteriorate or even lead to systematic bias [35].

2.3. Deep Reinforcement Learning

Deep reinforcement learning (DRL) reduces the dependence on explicit traffic behavior models to a certain extent through interactive learning with simulation environments. Representative methods include the deep deterministic policy gradient (DDPG) for continuous control [17], the value-based deep Q-network (DQN) family [19], and proximal policy optimization (PPO), which is known for stable policy updates [36]. These methods have been widely applied to the modeling and optimization of autonomous driving decision-making and motion planning tasks. For the strongly interactive scenario of on-ramp merging, existing studies have modeled the merging problem as a multi-agent reinforcement learning (MARL) problem, enabling mainline and ramp vehicles to learn cooperative policies under shared objectives. In mixed-traffic scenarios where CAVs and HDVs coexist, learning policies that adapt to the uncertainty of HDV behaviors has become a key research direction [37]. In addition, collaborative decision-making in mixed traffic has also seen the exploration of hybrid paradigms, such as combining different reinforcement learning architectures and policy components to improve decision quality and training stability. One example is the use of hybrid DRL frameworks for CAV cooperative decision-making [38].

Nevertheless, MARL still faces typical challenges in traffic tasks. There is often a mismatch between individual returns and system-level returns, and improper reward design can easily lead to training instability and policy oscillation. In multi-lane environments, the joint action space expands rapidly with the number of participating vehicles, resulting in high exploration costs and excessive invalid samples. Moreover, the uncontrollability of HDV behavior in mixed traffic makes decision-making based solely on direct policy-network outputs more likely to generate risky actions and conflicting action combinations. Therefore, researchers have begun to integrate hierarchical control, centralized training with decentralized execution (CTDE), and safety-enhancement mechanisms to achieve a balance between scalability and safety [39,40].

2.4. Cooperative Decision-Making and Value Decomposition

To address the core problems of joint action evaluation and credit assignment in multi-agent reinforcement learning, value decomposition methods structurally decompose the value function, thereby improving the feasibility of cooperative multi-agent learning. Among them, value decomposition networks (VDNs) decompose the team Q-value into the sum of individual Q-values of all agents, greatly simplifying cooperative learning and decentralized execution [41]. The QMIX algorithm further introduces monotonicity constraints and a mixing network, enabling the model to learn a more expressive global value function during centralized training while still supporting decentralized execution by individual agents [42]. For cooperative lane changing, MQLC combines individual and global Q-networks through QCOMBO-based consistency regularization [43]. During execution, high-priority vehicles retain their individually highest-valued action, while low-priority vehicles retain their top-n actions. The global Q-network evaluates the resulting joint-action combinations and selects the one with the highest estimated global return, coordinating individual preferences with system-level objectives.

The candidate-generation procedure in MQLC nevertheless introduces a structural restriction: the global Q-network can only evaluate and select combinations of individually preselected actions. An action excluded at this stage cannot be recovered by the global Q-network, even if it could contribute to a beneficial cooperative maneuver. Moreover, retaining only one action for a high-priority vehicle fixes that vehicle’s choice during subsequent joint-action evaluation and selection. These observations motivate a different treatment of candidate actions and merging priorities in BQDOM. Rather than restricting each vehicle to its individually top-ranked actions, BQDOM constructs candidate sets through legality and predicted-risk filtering, incorporates merging urgency and assistive actions into joint-action scoring, and applies fixed-width Beam Search to select a coordinated action combination.

2.5. Safety Enhancement

In safety-critical traffic decision-making tasks, frequent collisions during reinforcement learning exploration can contaminate the sample distribution and significantly reduce training efficiency. Therefore, safety-enhancement mechanisms have become an important support for applying MARL in this field [44]. Among them, action masking is a direct and effective method. Its core idea is to filter out clearly invalid or unsafe actions in advance at each decision step, thereby reducing meaningless model updates and lowering collision risks. Building on this idea, Chen et al. proposed a priority-based safety supervisor in a mixed-traffic on-ramp merging scenario. By combining forward-looking prediction techniques, the supervisor detects potential collision conflicts in real time and replaces policy outputs with safer alternative actions when necessary, effectively accelerating model training and reducing collision rates [37]. In the field of safe reinforcement learning, the concept of shielding has also been widely applied. Its core idea is to insert a constraint layer between policy output and environmental execution to ensure that system operation always satisfies safety requirements and avoids dangerous regions [45].

In summary, existing studies provide an important foundation for cooperative decision-making in mixed-traffic on-ramp merging, but several limitations remain. Rule-based and optimization-based methods rely on accurate modeling and are difficult to adapt to the uncertainty of HDV behavior. Supervised learning and trajectory prediction methods can enhance forward-looking perception, but they are limited by data distribution and annotation quality. Deep reinforcement learning and value-based coordination methods provide mechanisms for aligning individual and system-level objectives, while candidate-action construction and joint-action selection remain important design considerations for strongly coupled merging tasks. For mixed-traffic on-ramp merging, integrating predictive risk filtering with explicit merging priorities and joint-action search provides a task-oriented approach to coordinating ramp vehicles and mainline assistance. Therefore, it is necessary to construct a multi-agent cooperative decision-making method that integrates forward-looking perception, safety filtering, and joint action search, so as to improve on-ramp merging efficiency and system-level cooperative decision quality while ensuring safety and feasibility.

3. Methodology

3.1. Problem Formulation

This study considers a mixed-traffic environment for multi-lane on-ramp merging, where the system consists of controllable CAVs and uncontrollable HDVs. CAVs are required to complete speed adjustment and lane-changing maneuvers within a limited spatial range while cooperatively interacting with mainline vehicles. This process involves significant uncertainty: HDV behaviors are difficult to model accurately, and the interactions among vehicles are strongly coupled. Therefore, the problem is more suitably described as a multi-agent partially observable Markov decision process (POMDP), formulated as

$$I,S,{\left\{{\Omega }_i\right\}}_{i\in I},A,P,R,O,\gamma .$$

(1)

here, I denotes the set of agents; S denotes the global state space, which contains the motion states, positional information, and road topology information of all vehicles within the merging scenario; ${\Omega }_i$ denotes the observation space of agent i; A denotes the discrete action set; P denotes the state transition function; R denotes the reward function; O denotes the observation function; and $\gamma$ is the discount factor used to balance immediate rewards and long-term returns.

At time step t, each agent $i\in I_t$ can only obtain its local observation $o_i^t\in {\Omega }_i$ and select a discrete action $a_i^t\in A$. The joint action is defined as

$${\mathbf{a}}^t=\left(a_1^t,\dots ,a_{N_t}^t\right)\in A^{N_t},$$

(2)

where $N_t=\left|I_t\right|$ denotes the number of agents currently present in the scenario. The objective is to maximize the long-term system return while satisfying safety and executability constraints:

$$\underset{\pi }{\max }{\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }}{\gamma }^{t}R\left(s^t,{\mathbf{a}}^t\right)\right].$$

(3)

It should be emphasized that, in multi-lane merging tasks, the optimal policy depends not only on the immediate return of each individual vehicle, but also on the coupled combination of actions among multiple vehicles. Therefore, joint action selection and system-level coordination must be explicitly considered.

3.2. Overview of BQDOM

This paper proposes BQDOM, namely Beam-searched Q-learning with Dual-mask for On-ramp Merging, to address the multi-agent cooperative decision-making problem in mixed-traffic flow under multi-lane on-ramp merging scenarios. Unlike conventional reinforcement learning methods that only impose collision risk implicitly through reward penalties, BQDOM explicitly formulates cooperative decision-making as a safety-constrained joint action filtering and scoring search problem. The overall decision-making process consists of prediction-enhanced observation, dual action masking, joint action scoring, and approximate solution via Beam Search.

Specifically, the proposed framework first extracts temporal features of traffic interactions from historical observation sequences of vehicles, generating trajectory embeddings and short-term future trajectory predictions. These outputs provide forward-looking state representations for subsequent individual action-value estimation and risky action identification. Then, the individual action-value estimation module combines the current observation with trajectory embeddings to output the discrete action Q-values of each agent. Meanwhile, the action masking module constructs a dual action mask through a legality mask and a risk mask. The legality mask removes physically unreachable and traffic-rule-violating invalid actions, whereas the risk mask eliminates dangerous actions that may lead to high collision risks within a short horizon, thereby restricting the candidate action space to a safe and feasible range.

On this basis, the joint action scoring module identifies ramp key vehicles and non-key mainline vehicles according to vehicle spatial positions and merging task requirements. Higher weights are assigned to key actions that promote ramp vehicle merging and to assistive actions by mainline vehicles that release merging gaps. Candidate joint actions are then ranked according to their system-level scores. Finally, BQDOM adopts a fixed-width Beam Search strategy to expand joint action sequences vehicle by vehicle within the filtered safe candidate action space, approximately searching for high-quality joint actions and outputting the selected action to the environment for execution. Through the above design, BQDOM explicitly enhances multi-vehicle cooperation in merging scenarios while ensuring action safety and feasibility. At the same time, it avoids exhaustive enumeration of the full joint action space, thereby improving multi-agent cooperative decision-making quality under controllable computational complexity. The overall framework is shown in Figure 1.

Figure 1. Framework of the multi-agent cooperative decision-making method for on-ramp merging in mixed-traffic flow.

3.3. Risk Field Construction and Reward Function

In the simulation environment, the one-step reward of each CAV consists of five components: collision penalty, speed reward, merging-lane penalty, longitudinal car-following risk penalty, and lane-changing penalty. Let the state of the i-th controlled CAV at time step t be $x_i\left(t\right)$, $y_i\left(t\right)$, and $v_i\left(t\right)$, and let its corresponding action be $a_i\left(t\right)$. The reward can be expressed as

$$r_i\left(t\right)=r_{\mathrm{col},i}\left(t\right)+r_{\mathrm{spd},i}\left(t\right)+r_{\mathrm{merge},i}\left(t\right)+r_{\mathrm{long},i}\left(t\right)+r_{\mathrm{lc},i}\left(t\right).$$

(4)

The components are defined as follows.

3.3.1. Collision Penalty

When vehicle i collides at the current time step, the environment provides a fixed negative reward; otherwise, this term is zero:

$$r_{\mathrm{col},i}\left(t\right)=-{\lambda }_{\mathrm{col}}\mathbb{I}\left(\mathrm{vehicle}i\mathrm{collides}\right),$$

(5)

where ${\lambda }_{\mathrm{col}}>0$ is the collision penalty coefficient, and $\mathbb{I}(·)$ is the indicator function.

3.3.2. Speed Reward

The vehicle speed is first linearly mapped to the interval $[0,1]$. Given the reward speed range $[v_{\min },v_{\max }]$, the normalized speed is defined as

$${\overline{v}}_i\left(t\right)={\displaystyle \frac{v_i\left(t\right)-v_{\min }}{v_{\max }-v_{\min }}}.$$

(6)

the speed reward is then given by

$$r_{\mathrm{spd},i}\left(t\right)={\lambda }_{\mathrm{spd}}{\overline{v}}_i\left(t\right),$$

(7)

where ${\lambda }_{\mathrm{spd}}>0$ is the speed reward coefficient. This term encourages CAVs to maintain a relatively high speed within the specified speed range, thereby improving overall traffic efficiency.

3.3.3. Merging-Lane Penalty

When a vehicle is located in the merging lane, namely the dedicated lane where the mainline and the ramp converge, an exponential cost term related to the longitudinal position is introduced to discourage vehicles from staying in the middle of the merging lane for a long time. Let $x_m$ denote the reference longitudinal position of the merging lane and $L_m$ denote the length scale of the merging section. The merging-lane cost is defined as

$$C_{\mathrm{merge},i}\left(t\right)=\left\{\begin{array}{cc}-\exp \left(-{\displaystyle \frac{{\left(x_i\left(t\right)-x_m\right)}^2}{10L_m}}\right),\hfill & \begin{array}{c}\mathrm{if}\mathrm{vehicle}i\mathrm{is}\mathrm{in}\hfill \\ \mathrm{the}\mathrm{merging}\mathrm{lane},\hfill \end{array}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(8)

the corresponding reward component is

$$r_{\mathrm{merge},i}\left(t\right)={\lambda }_{\mathrm{merge}}{C}_{\mathrm{merge},i}\left(t\right),$$

(9)

where ${\lambda }_{\mathrm{merge}}>0$ is the merging-lane cost coefficient. This term assigns a relatively large negative value near the merging area, guiding CAVs to complete merging as soon as possible and return to the mainline lane.

3.3.4. Longitudinal Car-Following Risk Penalty

In the longitudinal interactions of on-ramp merging, rear-end collision risk is mainly reflected in two cases: insufficient time headway and excessively short time-to-collision (TTC). This paper constructs the longitudinal car-following risk penalty based on the time-headway safety constraint and the TTC-based risk field.

Let $d_i^t$ denote the longitudinal distance between vehicle i and its preceding vehicle, $v_i^t$ denote the speed of vehicle i, and $v_{\mathrm{pre},i}^t$ denote the speed of the preceding vehicle. Given the desired time-headway threshold $T_h$, the time headway is defined as

$$h_i^t={\displaystyle \frac{d_i^t}{v_i^t+\epsilon }},$$

(10)

where $\epsilon$ is a small constant used to prevent division by zero. When $h_i^t<T_h$, insufficient headway risk is considered to exist, and the headway risk intensity is defined as

$${\rho }_h^t=\max \left(0,\log {\displaystyle \frac{T_h}{h_i^t}}\right).$$

(11)

To further characterize high-risk states in which a vehicle rapidly approaches its preceding vehicle, the approaching speed of vehicle i relative to its preceding vehicle is defined as

$$\Delta v_i^t=\max \left(v_i^t-v_{\mathrm{pre},i}^t,0\right).$$

(12)

When $\Delta v_i^t>0$, vehicle i is approaching its preceding vehicle; when $\Delta v_i^t=0$, the longitudinal distance between the two vehicles does not exhibit a decreasing trend. Accordingly, TTC is defined as

$$TT{C}_i^t=\left\{\begin{array}{cc}{\displaystyle \frac{d_i^t}{\Delta v_i^t+\epsilon }},\hfill & \Delta v_i^t>0,\hfill \\ +\infty ,\hfill & \Delta v_i^t=0.\hfill \end{array}\right.$$

(13)

Let $TT{C}_{\mathrm{safe}}$ denote the safe TTC threshold and $TT{C}_{\mathrm{crit}}$ denote the critical high-risk threshold, where $TT{C}_{\mathrm{crit}}<TT{C}_{\mathrm{safe}}$. TTC is mapped to a risk scale within $[0,1]$ as follows:

$${\rho }_{\mathrm{ttc}}^t=\left\{\begin{array}{cc}0,\hfill & {\mathrm{TTC}}_i^t\ge {\mathrm{TTC}}_{\mathrm{safe}},\hfill \\ 1,\hfill & {\mathrm{TTC}}_i^t\le {\mathrm{TTC}}_{\mathrm{crit}},\hfill \\ {\displaystyle \frac{{\mathrm{TTC}}_{\mathrm{safe}}-{\mathrm{TTC}}_i^t}{{\mathrm{TTC}}_{\mathrm{safe}}-{\mathrm{TTC}}_{\mathrm{crit}}}},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(14)

Finally, the two longitudinal risks are integrated into a unified risk measure and converted into a reward penalty:

$$r_{\mathrm{long},i}\left(t\right)=-{\lambda }_h{\rho }_h^t-{\lambda }_{\mathrm{ttc}}{\rho }_{\mathrm{ttc}}^t.$$

(15)

Here, ${\lambda }_h,{\lambda }_{\mathrm{ttc}}>0$ are weighting coefficients used to balance the relative importance of the insufficient-headway penalty, which emphasizes steady-state safety constraints, and the TTC urgency penalty, which suppresses dynamic approaching risks. This unified penalty can simultaneously restrain aggressive close-following behavior and high-risk states caused by rapid approach during Q-learning, thereby imposing consistent constraints on longitudinal car-following safety.

3.3.5. Lane-Changing Penalty

To suppress frequent and unnecessary lane-changing behaviors, a fixed penalty is assigned when a vehicle takes a lateral action such as lane changing to the left or right. This term does not affect lane keeping or purely longitudinal acceleration/deceleration actions:

$$r_{\mathrm{lc},i}\left(t\right)=\left\{\begin{array}{cc}-{\lambda }_{\mathrm{lc}},\hfill & a_i\left(t\right)\in \{\mathtt{Lane}\_\mathtt{Left},\mathtt{Lane}\_\mathtt{Right}\},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(16)

where ${\lambda }_{\mathrm{lc}}>0$ is the lane-changing cost coefficient. This term is designed to avoid traffic disturbances and safety risks caused by frequent lane changes within the merging area.

Overall, the coefficients in the reward function are specified by the environment configuration parameters. Structurally, the reward design reflects a comprehensive trade-off among the above five factors and provides a unified optimization objective for subsequent multi-agent cooperative decision-making based on the risk field and dual action masking.

3.4. Trajectory Prediction and Individual Action-Value Estimation

The key risks in on-ramp merging are often caused by short-horizon vehicle interaction dynamics, such as the following vehicle rapidly approaching due to sudden deceleration of the leading vehicle, or lateral conflicts caused by simultaneous lane changes of adjacent vehicles. Relying only on instantaneous single-step observations makes it difficult to capture the evolution trend of interactive behaviors. Therefore, a historical observation window is defined for agent i as

$$O_i^t=\left\{o_i^{t-L+1},\dots ,o_i^t\right\}\in {\mathbb{R}}^{L\times D},$$

(17)

where L denotes the history length, and D denotes the dimension of the single-step observation vector, namely the feature dimension of each $o_i^t$. The predictor uses a long short-term memory network (LSTM) to encode $O_i^t$ and outputs two types of information:

$$\left(e_i^t,{\widehat{Z}}_i^{t:t+H}\right)=f_{\mathrm{pred}}\left(O_i^t\right).$$

(18)

Here, $e_i^t$ denotes the temporal embedding used for individual action-value estimation, and ${\widehat{Z}}_i^{t:t+H}$ denotes the sequence of H future CAV states at steps $t+1,\dots ,t+H$. Each predicted state contains position $(x,y)$, heading $\psi$, and speed v. The shared LSTM encoder feeds separate embedding and prediction heads; the latter predicts a residual around constant-velocity extrapolation. The default configuration uses $L=8$ historical observations and $H=8$ future steps with a sampling interval of $\Delta t=0.2$ s. HDV trajectories are obtained separately using the model-based rollout described in Section 3.5.2.

The shared encoder and prediction head are pretrained on simulated CAV trajectories using future-state labels. Both are frozen during decision-policy training, while the embedding head and value networks are optimized. Thus, the prediction parameters are not jointly optimized with the reinforcement-learning TD objective.

The pretraining objective is

$${\mathcal{L}}_{\mathrm{pre}}={\displaystyle \frac{1}{N_{\mathrm{valid}}}}{\displaystyle \sum _{n=1}^{N_b}}{\displaystyle \sum _{h=1}^H}{m}_{nh}\left[{\displaystyle \frac{1}{2}}\left({\displaystyle \frac{\Delta x_{nh}^2}{s_x^2}}+{\displaystyle \frac{\Delta y_{nh}^2}{s_y^2}}\right)+1-\cos (\Delta {\psi }_{nh})+{\displaystyle \frac{\Delta v_{nh}^2}{s_v^2}}\right].$$

(19)

Here, $N_b$ is the prediction-training batch size, n indexes trajectory samples, and h indexes future steps. For $q\in \{x,y,\psi ,v\}$, $\Delta q_{nh}={\widehat{q}}_{nh}-q_{nh}$ is the prediction error relative to the simulated label. Position, heading, and speed are expressed in meters, radians, and meters per second, respectively. The label-validity mask $m_{nh}\in \{0,1\}$ excludes missing future labels, and $N_{\mathrm{valid}}={\sum }_{n=1}^{N_b}{\sum }_{h=1}^H{m}_{nh}>0$. The fixed normalization scales are $s_x=10$ m, $s_y=2$ m, and $s_v=5$ m/s. The cosine term accounts for the periodicity of heading. All prediction targets are simulated CAV future states; no real human-driving recordings are used.

At the individual level, a parameter-shared discrete action-value network is adopted to estimate the preference of each vehicle for each action:

$$Q_i^t\left(a\right)=Q_i\left([o_i^t;e_i^t],a\right),a\in A.$$

(20)

The observation feature $o_i^t$ and the temporal embedding feature $e_i^t$ are concatenated. The former describes the current state of the vehicle, whereas the latter contains the temporal evolution information of surrounding vehicle interactions. In on-ramp merging scenarios, the spatiotemporal state of a vehicle directly affects the reasonableness and feasibility of its candidate actions. Introducing $e_i^t$ can significantly improve the sensitivity of value estimation to short-horizon dynamic interactions.

The resulting action-value vector is

$${\mathbf{q}}_i^t={\left(Q_i^t\left(a\right)\right)}_{a\in A}\in {\mathbb{R}}^{\left|A\right|}.$$

(21)

The parameter-sharing mechanism reduces the number of model parameters and sample complexity. It also ensures policy consistency and generalization capability among multiple CAVs in complex mixed-traffic environments with variable numbers of vehicles and lanes. This is a commonly adopted basic design in homogeneous CAV systems.

3.5. Dual Action Masking

Training and execution in multi-lane on-ramp merging scenarios face a significant redundancy problem in the action space. Many candidate actions are either physically infeasible under the current state or executable but highly risky within a short horizon. If such actions are directly included in agent exploration and joint action search, invalid samples are likely to be generated, slowing down training convergence and greatly increasing the collision rate. Therefore, this paper introduces a dual action masking mechanism to filter low-quality candidates in advance at the action-space level and impose hard constraints on the candidate set.

3.5.1. Legality Mask

For each agent i and candidate action $a\in A_i$, a rule-based mask is constructed according to lane boundaries, road speed limits, and vehicle dynamics constraints:

$$M_{\mathrm{legal}}^t(i,a)=\left\{\begin{array}{cc}1,\hfill & a\mathrm{satisfies}\mathrm{lane}-\mathrm{boundary}\hfill \\ & \mathrm{and}\mathrm{dynamic}\mathrm{constraints},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(22)

The legality mask is used to filter actions that are physically unreasonable or prohibited by traffic rules. It takes the value 1 when action a satisfies basic constraints such as the current lane boundary, permitted lane-changing direction, and speed or acceleration limits; otherwise, it takes the value 0. This mask mainly addresses the problem of obviously invalid actions and provides the most fundamental executability guarantee.

3.5.2. Risk Mask

The risk mask further filters actions that have significant conflict risks within a short horizon. Let

$$M_{\mathrm{risk}}^t(i,a)\in \{0,1\}.$$

(23)

Its computation relies on two types of forward-looking trajectories. The first is the action-conditioned ego trajectory under candidate action a:

$${\tilde{Z}}_{i,a}^{t:t+H}=g\left(o_i^t,a\right).$$

(24)

The second is the future trajectory ${\widehat{Z}}_j^{t:t+H}$ of surrounding vehicle j. For surrounding CAVs, it is supplied by the LSTM predictor in Section 3.4. HDVs are simulated human-driven vehicles whose longitudinal behavior follows the Intelligent Driver Model (IDM) and whose lane-changing behavior follows the MOBIL rules. HDV acceleration and steering are initialized using the IDM/MOBIL-based control logic and held over the finite prediction horizon, subject to controller limits, while the motion equations advance position, heading, and speed. The rollout is reinitialized at each decision step. This implementation therefore uses a model-based rollout with held controls rather than constant-velocity extrapolation. The HDVs supply the surrounding interaction context for simulated CAV training trajectories; no neural HDV predictor is trained on human-driving recordings.

Based on these two types of forward-looking trajectories, the minimum distance $d_{\min }(i,a)$ and minimum time-to-collision $TT{C}_{\min }(i,a)$ between agent i and surrounding vehicles under candidate action a are defined as

$$d_{\min }(i,a)=\underset{h\in [1,H]}{\min }\underset{j\ne i}{\min }∥{\tilde{\mathbf{p}}}_{i,a}^{t+h}-{\widehat{\mathbf{p}}}_j^{t+h}∥,$$

(25)

$$TT{C}_{\min }(i,a)=\underset{h\in [1,H]}{\min }\underset{j\ne i}{\min }{\displaystyle \frac{\Delta d_{ij}^{t+h}}{\max \left(\Delta v_{ij}^{t+h},\epsilon \right)}}.$$

(26)

here, h denotes the current step within the prediction horizon, $h\in \{1,\dots ,H\}$; ${\tilde{\mathbf{p}}}_{i,a}^{t+h}$ denotes the position of the ego vehicle at time $t+h$ under action a; ${\widehat{\mathbf{p}}}_j^{t+h}$ denotes the predicted position of surrounding vehicle j at time $t+h$; $\Delta d_{ij}^{t+h}$ denotes the relative distance between the two vehicles within the prediction horizon; and $\Delta v_{ij}^{t+h}$ denotes the approaching speed between the two vehicles within the prediction horizon.

Given the safe distance threshold $d_{\mathrm{safe}}$ and the safe TTC threshold ${\tau }_{\mathrm{safe}}$, the risk mask is defined as

$$M_{\mathrm{risk}}^t(i,a)=\mathbb{I}\left(d_{\min }(i,a)\ge d_{\mathrm{safe}}\wedge TT{C}_{\min }(i,a)\ge {\tau }_{\mathrm{safe}}\right).$$

(27)

This definition follows a conservative but not overly conservative principle: it should effectively block high-probability conflicts within a short horizon while avoiding excessive shrinkage of the action space, which may lead to policy rigidity.

Accordingly, the final safety mask is

$$M_{\mathrm{safe}}^t(i,a)=M_{\mathrm{legal}}^t(i,a)\wedge M_{\mathrm{risk}}^t(i,a).$$

(28)

The safe candidate set is defined as

$$A_{i,\mathrm{safe}}^t=\left\{a\in A\mid M_{\mathrm{safe}}^t(i,a)=1\right\}.$$

(29)

If the safe candidate action set of an agent is empty under the current state, the lane-keeping action or the legal action with the minimum risk is preferentially retained as a fallback action to ensure the executability of the joint action search process. The masked action value is then defined as

$${\tilde{Q}}_i^t\left(a\right)=\left\{\begin{array}{cc}{Q}_i^t\left(a\right),\hfill & M_{\mathrm{safe}}^t(i,a)=1,\hfill \\ -\infty ,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(30)

3.6. Weighted Scoring-Based Beam Search for Joint Decision-Making

Under the dual action masking constraint, the safe action set of agent i at time step t is denoted by $A_{i,\mathrm{safe}}^t$. Although the legality mask and risk mask effectively reduce the action space, the multi-agent joint action space ${\prod }_{i\in I_t}{A}_{i,\mathrm{safe}}^t$ still grows rapidly with the number of vehicles present in the scenario. If only single-agent greedy decision-making is used, system-level cooperative optimality cannot be guaranteed. This is particularly true when multiple lanes compete for limited merging gaps, where locally optimal action combinations may lead to reduced system efficiency. Therefore, this paper transforms joint action selection into a safety-constrained sequential scoring and search problem and employs Beam Search for an approximate optimal solution.

Considering that ramp vehicles must complete merging within a limited distance, vehicles located in key ramp regions have a more direct influence on the success of the current merging process. Moreover, the closer a vehicle is to the merging terminal point, the more significant its decision becomes for conflict resolution and traffic efficiency. Therefore, this paper identifies ramp key vehicles according to whether a vehicle is located in the key ramp region and its relative distance to the merging terminal point. For key vehicles, actions that facilitate safe and smooth merging are emphasized. For non-key mainline vehicles, assistive actions that help release merging space for ramp key vehicles, alleviate local conflicts, and avoid occupying the merging area are emphasized. In this way, the influence of key CAV actions on the merging task is incorporated into joint action evaluation.

First, let $d_i^{\mathrm{end}}\left(t\right)$ denote the longitudinal distance from vehicle i to the merging terminal point, and let $L_m$ denote the length of the merging area. The merging urgency of a ramp vehicle is defined as

$${\kappa }_i\left(t\right)=clip\left(1-{\displaystyle \frac{d_i^{\mathrm{end}}\left(t\right)}{L_m}},0,1\right).$$

(31)

A larger ${\kappa }_i\left(t\right)$ indicates that the vehicle is closer to the merging terminal point and that its action has a more significant influence on the current merging process. Let the candidate joint action at time step t be

$${\mathbf{a}}^t={\left(a_i^t\right)}_{i\in I_t},a_i^t\in A_{i,\mathrm{safe}}^t,$$

(32)

where $I_t$ denotes the set of agents currently present. The key-action weight coefficient ${\beta }_i(a_i^t,s_g^t)$ is defined as

$${\beta }_i\left(a_i^t,s_g^t\right)=\left\{\begin{array}{cc}{\varphi }_i^{\mathrm{key}}\left(a_i^t,s_g^t\right),\hfill & i\in I_t^{\mathrm{key}},\hfill \\ {\varphi }_i^{\mathrm{assist}}\left(a_i^t,s_g^t\right),\hfill & i\in I_t^{\mathrm{main}},\hfill \\ 1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(33)

$${\varphi }_i^{\mathrm{key}}\left(a_i^t,s_g^t\right)=\left\{\begin{array}{cc}2+{\kappa }_i\left(t\right),\hfill & a_i^t\in A_i^{\mathrm{key}}\left(t\right),\hfill \\ 1,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(34)

$${\varphi }_i^{\mathrm{assist}}\left(a_i^t,s_g^t\right)=\left\{\begin{array}{cc}1+{\lambda }_a,\hfill & a_i^t\in A_i^{\mathrm{assist}}\left(t\right),\hfill \\ 1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(35)

here, $s_g^t$ denotes the global state at time t, $I_t^{\mathrm{key}}$ denotes the set of ramp key vehicles at the current time step, and $I_t^{\mathrm{main}}$ denotes the set of mainline assistive vehicles that directly interact with ramp key vehicles. ${\varphi }_i^{\mathrm{key}}(a_i^t,s_g^t)$ evaluates the contribution of action $a_i^t$ to the merging task of ramp key vehicles. $A_i^{\mathrm{key}}\left(t\right)$ denotes the key-action set of ramp key vehicles, mainly including safe actions that help the vehicle capture the target merging gap, improve speed matching with the target gap, or advance the merging process into the mainline. The closer the vehicle is to the merging terminal point, the larger ${\kappa }_i\left(t\right)$ becomes, and the stronger the influence of its key actions on joint action scoring.

Similarly, ${\varphi }_i^{\mathrm{assist}}(a_i^t,s_g^t)$ evaluates the assisting effect of action $a_i^t$ taken by a non-key mainline vehicle on the merging task of the key vehicle. $A_i^{\mathrm{assist}}\left(t\right)$ denotes the assistive-action set of mainline assistive vehicles. When a mainline vehicle is located within the front or rear neighborhood of the potential merging gap of a ramp key vehicle, and its candidate action can enlarge the target merging gap, reduce relative conflict risk with the ramp key vehicle, or avoid continuing to occupy the merging area, this action is identified as an assistive action. ${\lambda }_a\in (0,1)$ is the gain coefficient for mainline assistive actions. This setting grants a moderate weight increase to mainline assistive actions without exceeding the influence weight of ramp key-vehicle actions, ensuring that the joint action scoring process still prioritizes the ramp merging task.

On this basis, the joint action scoring function is defined as

$$\begin{array}{cc}\hfill F\left(s_g^t,{\mathbf{o}}^t,{\mathbf{a}}^t\right)& ={\displaystyle \frac{{\displaystyle \sum _{i\in I_t}{\beta }_i\left(a_i^t,s_g^t\right)Q_i\left([o_i^t;e_i^t],a_i^t\right)}}{{\displaystyle \sum _{i\in I_t}{\beta }_i\left(a_i^t,s_g^t\right)}}}.\hfill \end{array}$$

(36)

In Equation (36), ${\beta }_i(a_i^t,s_g^t)$ directly weights individual Q-values in the normalized joint-action score according to the traffic state and candidate action, explicitly representing ramp-merging urgency and mainline assistance. Changing a candidate action can therefore alter both its individual Q-value and its weight. This mechanism expresses merging priorities through joint-action scoring rather than fixing a high-priority vehicle to a single individually preferred action. Combined with dual action masking and Beam Search, it supports the evaluation and selection of candidate combinations according to their weighted contributions to the merging task.

The core objective of joint decision-making is to solve for the optimal action sequence:

$$\begin{array}{cc}\hfill {\mathbf{a}}^{t,\ast }& =\arg \underset{{\mathbf{a}}^t}{\max }F\left(s_g^t,{\mathbf{o}}^t,{\mathbf{a}}^t\right),\hfill \\ & \mathrm{s}.\mathrm{t}.a_i^t\in A_{i,\mathrm{safe}}^t,\forall i\in I_t.\hfill \end{array}$$

(37)

Since directly enumerating all feasible sequences incurs a complexity that grows exponentially with the number of agents, this paper adopts Beam Search with a fixed width B for approximate solution. During the search process, safe joint action candidates at time step t are expanded layer by layer, and the scoring function $F(·)$ is used to approximately evaluate candidate partial joint actions. At each layer, only the top B candidates with the highest scores are retained until all agents have been expanded. The joint action with the highest score is then selected and executed in the environment. At the next time step, key-action weighting coefficients are recalculated based on new observations, the safe action set is updated, and the search process is executed again. This forms a closed-loop cooperative control mechanism of prediction, scoring, search, and execution.

Algorithm 1 summarizes the decision-making procedure of BQDOM. The LSTM supplies temporal embeddings to the shared Q-network and predicted CAV trajectories for risk assessment. During Beam Search, unassigned actions are temporarily completed using their highest individual Q-values within the candidate sets to evaluate partial prefixes; these actions remain subject to expansion at subsequent stages. Ramp vehicles are expanded first, with those closer to the merging end given higher priority; ties are resolved by vehicle index. The learning procedure is described separately in Section 3.7.

From the perspective of search complexity, let $N=|I_t|$ denote the number of agents at time step t. Exhaustive enumeration over the masked action sets considers ${\prod }_{i\in I_t}\left|A_{i,\mathrm{safe}}^t\right|$ joint action combinations, which is bounded above by ${\left|A\right|}^N$. In contrast, Beam Search retains at most B candidate prefixes at each layer. The total number of candidate expansions is therefore bounded above by $B{\sum }_{i\in I_t}\left|A_{i,\mathrm{safe}}^t\right|$, giving an expansion-count complexity of

$$\mathcal{O}\left(B\sum _{i\in I_t}\left|A_{i,\mathrm{safe}}^t\right|\right)\subseteq \mathcal{O}\left(BN\left|A\right|\right).$$

(38)

For fixed B and $\left|A\right|$, this bound grows linearly with N, rather than exponentially.

Since the dual action mask significantly compresses the safe action set and the fixed beam width controls the search overhead within an acceptable range, the proposed method can satisfy the online decision-making requirements of multi-lane on-ramp merging scenarios. In summary, through key-action weighted scoring, joint action sequence evaluation, and approximate solution via Beam Search, this paper achieves a unified consideration of safety, cooperation, and forward-looking decision-making in multi-lane on-ramp merging.

Algorithm 1 Decision-Making Procedure of BQDOM

Require: Trained LSTM encoder, embedding and prediction heads, and shared Q-network; history length L, prediction horizon H, and beam width B.

Ensure: Executed joint actions.

1: Initialize the environment and vehicle history buffers.    2: while the episode is not terminated do    3:  Obtain $s_g^t$, observations $o_i^t$, and history windows $O_i^t$.    4:  Encode $O_i^t$ with the LSTM; generate $e_i^t$ and ${\widehat{Z}}_i^{t:t+H}$ through the embedding and prediction heads.    5:  Compute $Q_i^t\left(a\right)=Q([o_i^t;e_i^t],a)$ using the shared Q-network.    6:  Generate HDV trajectories using the model-based rollout.    7:  Construct action-conditioned ego trajectories and compute legality and risk masks.    8:  Form candidate sets ${\mathcal{A}}_i^t$; apply the prescribed legal fallback rule if a set is empty.    9:  Identify key and assistive actions and compute ${\beta }_i(a,s_g^t)$.  10:  Determine expansion order $\pi$; initialize $\mathcal{S}\leftarrow \{\varnothing \}$.  11:  for  $k=1,\dots ,|I_t|$ do  12:   $i\leftarrow \pi \left(k\right)$; $\mathcal{U}\leftarrow \varnothing$.  13:   for each $p\in \mathcal{S}$ and $a\in {\mathcal{A}}_i^t$ do  14:    Append vehicle i’s action a to p, obtaining $p^{\prime }$.  15:    Temporarily complete unassigned actions by ${\tilde{a}}_j\leftarrow \arg {\max }_{a\in {\mathcal{A}}_j^t}{Q}_j^t\left(a\right)$, obtaining $\tilde{\mathbf{a}}$.  16:    Evaluate $F(s_g^t,{\mathbf{o}}^t,\tilde{\mathbf{a}})$ using Equation (36).  17:    Add $p^{\prime }$ and its score to $\mathcal{U}$.  18:   end for  19:   Retain up to B distinct highest-scoring prefixes from $\mathcal{U}$ in $\mathcal{S}$.  20:  end for  21:  Select ${\mathbf{a}}^{t,\ast }\leftarrow \arg {\max }_{\mathbf{a}\in \mathcal{S}}F(s_g^t,{\mathbf{o}}^t,\mathbf{a})$.  22:  Execute ${\mathbf{a}}^{t,\ast }$ and update the history buffers.  23: end while

3.7. Learning Objective and Optimization

This section presents the learning objective and optimization strategy of BQDOM. The objective below optimizes the decision networks and embedding head; the shared LSTM encoder and prediction head remain frozen during this stage. Their supervised training is specified in Section 3.4. Off-policy Q-learning combined with the Double Q mechanism is adopted to construct the temporal-difference (TD) target, thereby stabilizing the training process and suppressing value overestimation bias. A transition tuple $(s_g^t,{\mathbf{o}}^t,{\mathbf{a}}^t,r^t,s_g^{t+1},{\mathbf{o}}^{t+1},d^t)$ is sampled from the experience replay buffer, where $d^t$ denotes the terminal flag of the scenario. The TD target is defined as

$$y^t=r^t+(1-d^t)\gamma Q_{\mathrm{tot}}^{-}\left(s_g^{t+1},{\mathbf{o}}^{t+1},{\mathbf{a}}^{t+1,\ast }\right).$$

(39)

here, ${\mathbf{a}}^{t+1,\ast }$ denotes the optimal joint action selected by the online network under the dual action masking constraint, and $r^t$ denotes the system-level reward aggregated from the one-step rewards of all controlled CAVs at the current time step. The optimal joint action is defined as

$$\begin{array}{c}\hfill {\mathbf{a}}^{t+1,\ast }=\arg \underset{{\mathbf{a}}^{t+1}\in {\prod }_{i\in I_{t+1}}{A}_{i,\mathrm{safe}}^{t+1}}{\max }F\left(s_g^{t+1},{\mathbf{o}}^{t+1},{\mathbf{a}}^{t+1}\right).\end{array}$$

(40)

and the i-th component of the optimal joint action is

$$a_i^{t+1,\ast }={\left[{\mathbf{a}}^{t+1,\ast }\right]}_i.$$

(41)

The system-level value estimate of the current joint action is

$$Q_{\mathrm{tot}}\left(s_g^t,{\mathbf{o}}^t,{\mathbf{a}}^t\right)=F\left(s_g^t,{\mathbf{o}}^t,{\mathbf{a}}^t\right).$$

(42)

To improve training stability, the Huber loss is adopted to construct the TD error loss, and the system-level value estimate constructed from the actually executed joint action is updated as

$${\mathcal{L}}_{\mathrm{TD}}=\mathbb{E}\left[Huber\left(Q_{\mathrm{tot}}\left(s_g^t,{\mathbf{o}}^t,{\mathbf{a}}^t\right)-y^t\right)\right].$$

(43)

In addition, to suppress the interference caused by abnormally high values assigned to actions filtered out by the risk mask, a lightweight safety regularization term ${\mathcal{L}}_{\mathrm{safe}}$ is introduced. The dangerous action set is defined as

$${\mathcal{B}}_i^t=\left\{a\in A|M_{\mathrm{legal}}^t(i,a)=1,M_{\mathrm{safe}}^t(i,a)=0\right\}.$$

(44)

The safety regularization term is then written as

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{safe}}=\mathbb{E}\left[{\displaystyle \frac{1}{\max \left(\left|{\mathcal{B}}^t\right|,1\right)}}\sum _i\sum _{a\in {\mathcal{B}}_i^t}ReLU\left(Q_i\left([o_i^t;e_i^t],a\right)\right)\right].\end{array}$$

(45)

here, $|{\mathcal{B}}^t|$ denotes the total number of dangerous actions and is used for normalization. The final loss function is

$$\mathcal{L}={\mathcal{L}}_{\mathrm{TD}}+{\lambda }_{\mathrm{safe}}{\mathcal{L}}_{\mathrm{safe}}.$$

(46)

where ${\lambda }_{\mathrm{safe}}$ is the safety regularization weight, which balances value-fitting accuracy and safety-constraint strength. The target network is synchronized through soft update:

$${\theta }^{-}\leftarrow (1-\tau ){\theta }^{-}+\tau \theta .$$

(47)

where ${\theta }^{-}$ denotes the parameters of the target value network, $\theta$ denotes the parameters of the corresponding online network, and $\tau \in (0,1)$ is the soft update coefficient. This learning objective is highly consistent with the execution mechanism of filtering before searching. During training, next-step action selection is also conducted under the dual action masking constraint, thereby effectively reducing policy deviation caused by training-execution distribution mismatch.

4. Results and Discussions

4.1. Experimental Environment and Parameter Settings

To verify the effectiveness of the proposed BQDOM method in highway on-ramp merging scenarios under mixed-traffic flow, a multi-agent mixed-traffic simulation platform is constructed based on the open-source autonomous driving simulation environment highway. The platform consists of mainline highway lanes and an on-ramp merging area, where CAVs and HDVs are simultaneously distributed on both the mainline and the ramp. Vehicles are required to complete longitudinal speed adjustment and lateral lane-changing cooperation within a limited merging distance, ultimately achieving a safe and efficient merging process.

This paper adopts a five-dimensional discrete high-level action set, including lane change to the left, lane keeping, lane change to the right, acceleration, and deceleration. The observation space is composed of vehicle kinematic information, which is used to describe the relative positions, velocities, and other state features of the ego vehicle and surrounding vehicles.

To ensure the fairness of comparative experiments, the basic experimental configuration is kept consistent with the common settings of mainstream baseline methods. The physical update frequency of the simulation environment is set to 15 Hz, and the policy decision frequency of agents is set to 5 Hz, which means that one decision action is executed every 0.2 s. The maximum duration of a single episode is set to 20 s, corresponding to a maximum of 100 decision steps. During training, the reward discount factor $\gamma$ is set to 0.99, the capacity of the experience replay buffer is set to 50,000, the batch size is set to 128, the rollout length is set to 10, and the reward scaling factor is set to 20. The total number of training episodes is set to 20,000. These settings generally follow the core framework of training frequency, reward design, and evaluation procedure adopted in baseline studies.

To facilitate reproducibility, Table 1 summarizes the reward, risk-filtering, and joint-action scoring parameters, while Table 2 reports the optimization and network settings.

Table 1. Reward, risk-filtering, and joint-action scoring parameters of BQDOM.

Parameter	Setting
Collision penalty coefficient ${\lambda }_{\mathrm{col}}$	400
Speed reward coefficient ${\lambda }_{\mathrm{spd}}$	1
Merging-lane penalty coefficient ${\lambda }_{\mathrm{merge}}$	4
Headway penalty coefficient ${\lambda }_h$	4
TTC penalty coefficient ${\lambda }_{\mathrm{ttc}}$	0.2
Lane-changing penalty coefficient ${\lambda }_{\mathrm{lc}}$	0.5
Risk-mask distance threshold $d_{\mathrm{safe}}$	2 m
Risk-mask TTC threshold ${\tau }_{\mathrm{safe}}$	1 s
History length	8 observations
Beam Search width B	4
Mainline assistive-action gain ${\lambda }_a$	0.5
Ramp key-action weight	$2+{\kappa }_i\left(t\right)\in [2,3]$
Safety regularization coefficient ${\lambda }_{\mathrm{safe}}$	0.2

Table 2. Optimization and network settings of BQDOM.

Parameter	Setting
Target-network soft-update coefficient $\tau$	0.01
Optimizer	Adam
Learning rate	$5\times {10}^{-4}$
Initial exploration rate	0.30
Minimum exploration rate	0.05
Exploration decay factor	0.999 per episode
LSTM layers	3
LSTM hidden dimension	128
Trajectory embedding dimension	64
Embedding-head dimensions	$128\to 128\to 64$
Prediction-head dimensions	$128\to 128\to 32$
Individual Q-network dimensions	$89\to 128\to 128\to 5$
Scoring hypernetwork dimensions	$25\to 128\to 1$ per branch

The exploration rate follows ${ϵ}_e=\max (0.05,0.30\times {0.999}^e)$, where e denotes the number of completed training episodes. The prediction head produces $H\times 4=32$ outputs for the default prediction horizon $H=8$. ReLU activations are applied between the fully connected layers of the embedding head, prediction head, individual Q-network, and scoring hypernetworks.

To enhance the comparability among different experimental groups and clearly quantify the performance of the proposed method under different traffic scales, the traffic density configuration is fixed, and three typical traffic scenarios are constructed: Easy Mode with 3 CAVs and 3 HDVs, Normal Mode with 6 CAVs and 6 HDVs, and Hard Mode with 9 CAVs and 9 HDVs. These scenarios are used to investigate the performance variation of different methods as the traffic scale gradually increases.

This paper evaluates the proposed method and baseline methods from three perspectives: safety, traffic efficiency, and learning stability. Safety is quantified by the collision rate. Traffic efficiency is characterized by the average driving speed and the average episode reward. Learning stability is analyzed through the cumulative return curves and average speed variation curves during training. These metrics provide a comprehensive evaluation of the overall performance of different methods in mixed-traffic on-ramp merging scenarios.

4.2. Comparative Experiments

To systematically verify the effectiveness and superiority of the proposed BQDOM method, three mainstream multi-agent reinforcement learning algorithms are selected as comparative baselines, namely MAA2C, MAPPO, and MAACKTR [46]. Specifically, MAA2C is a parameter-sharing multi-agent advantage actor–critic algorithm, which can learn cooperative policies for homogeneous multi-agent systems based on local rewards [47]. MAPPO is a classical paradigm of multi-agent proximal policy optimization, which improves policy optimization stability through generalized advantage estimation and clipped policy updates, and is commonly used as a benchmark method for multi-agent cooperative tasks [36]. MAACKTR introduces a K-FAC-based approximate second-order optimization method into the actor–critic framework to improve policy update efficiency and convergence performance [48].

To ensure the fairness and reliability of the comparative experiments, all baseline algorithms are trained under exactly the same environmental configuration, action space, and number of training episodes. The training random seeds, reward function scale, and evaluation procedure are also kept consistent. Only the native policy update mechanisms and action selection logic of each algorithm are retained. In contrast, the proposed BQDOM method fully adopts the decision-making framework described above, namely “trajectory prediction–risk filtering–joint search–joint scoring.” Experiments are conducted under easy, normal, and hard traffic scenarios, and the quantitative results are shown in Figure 2, Figure 3 and Figure 4.

Figure 2. Reward curvecomparison under the easy mode.

Figure 3. Reward curve comparison under the normal mode.

Figure 4. Reward curve comparison under the hard mode.

The episode reward curves under the three traffic density modes show that the proposed BQDOM method achieves the best training performance in Easy, Normal, and Hard scenarios. Specifically, in Easy Mode, BQDOM exhibits the fastest reward increase, enabling rapid policy improvement within fewer training episodes. It then converges stably to a level significantly higher than those of the other comparative methods. The final reward of BQDOM is approximately 40% higher than that of MAA2C and approximately 55% higher than that of MAPPO, whereas MAACKTR only stabilizes around a marginally positive reward level.

In Normal Mode, as the complexity of traffic interactions increases, BQDOM still maintains a continuous upward trend with relatively small fluctuations. Its final converged reward is substantially higher than those of MAA2C, MAPPO, and MAACKTR. Compared with MAA2C and MAPPO, the final reward is improved by approximately 140% and 120%, respectively. In Hard Mode, although high-density mixed traffic reduces the overall reward levels of all algorithms and significantly increases training difficulty, BQDOM still maintains the leading convergence result and favorable training stability, as shown in Figure 4. This indicates its stronger adaptability to complex scenarios.

By comparison, MAA2C shows a certain learning capability under all three modes, but its reward improvement speed and final convergence level are clearly lower than those of BQDOM. Moreover, its fluctuation becomes more pronounced in the high-density scenario. MAPPO achieves a relatively favorable reward level in the easy mode; however, as traffic density increases, its performance degrades significantly. In the middle and later stages of Normal and Hard modes, MAPPO remains within a relatively low reward range, indicating insufficient adaptability to complex vehicle interactions. MAACKTR performs relatively weakly in all three modes, with rewards remaining at a low level for a long time, making it difficult to learn high-quality cooperative policies. Overall, under the idealized simulation conditions considered in this study, BQDOM exhibits faster reward improvement and higher final rewards than the selected baseline methods across the three traffic-density settings. These results suggest that the proposed framework supports effective learning for multi-vehicle cooperative decision-making within the evaluated simulation scenarios.

To further compare the performance variation of different methods under different traffic densities during the evaluation phase, Figure 5 presents the collision rate, average driving speed, and evaluation reward as the scenario difficulty increases. It can be observed that, as the traffic density increases from Easy Mode to Hard Mode, the collision rates of all methods generally increase, while the evaluation rewards generally decrease. This indicates that high-density scenarios significantly increase the decision-making difficulty of multi-vehicle cooperative merging. In contrast, BQDOM maintains a relatively low collision rate and the highest evaluation reward across all three density scenarios, while its average driving speed remains at a relatively high level. This suggests that BQDOM can more stably balance safety and traffic efficiency in complex traffic interactions. To further quantify the performance differences among the methods, Table 3, Table 4 and Table 5 report the detailed results of each evaluation metric.

Figure 5. Performance comparison of the four models under different traffic density scenarios.

Table 3. Performance comparison of the four models under the easy scenario.

Method	Collision Rate	Average Speed	Evaluation Reward
BQDOM	$\mathbf{0.02}\pm \mathbf{0.02}$	$\mathbf{28.03}\pm \mathbf{0.28}$	$\mathbf{81.38}\pm \mathbf{3.72}$
MAA2C	$0.09\pm 0.03$	$26.47\pm 0.47$	$50.68\pm 5.78$
MAPPO	$0.08\pm 0.02$	$24.26\pm 0.53$	$40.74\pm 6.77$
MAACKTR	$0.23\pm 0.07$	$24.96\pm 0.25$	$4.19\pm 9.85$

Note: Bold values indicate the best result for each metric among the compared methods or configurations.

Table 4. Performance comparison of the four models under the normal scenario.

Method	Collision Rate	Average Speed	Evaluation Reward
BQDOM	$\mathbf{0.07}\pm \mathbf{0.03}$	$\mathbf{26.58}\pm \mathbf{0.13}$	$\mathbf{58.48}\pm \mathbf{2.79}$
MAA2C	$0.47\pm 0.11$	$24.09\pm 0.16$	$4.86\pm 10.52$
MAPPO	$0.09\pm 0.02$	$12.04\pm 0.09$	$1.04\pm 0.87$
MAACKTR	$0.61\pm 0.04$	$23.51\pm 0.40$	$-26.75\pm 4.47$

Note: Bold values indicate the best result for each metric among the compared methods or configurations.

Table 5. Performance comparison of the four models under the hard scenario.

Method	Collision Rate	Average Speed	Evaluation Reward
BQDOM	$\mathbf{0.18}\pm \mathbf{0.06}$	$23.21\pm 0.16$	$\mathbf{47.14}\pm \mathbf{3.47}$
MAA2C	$0.54\pm 0.09$	$21.25\pm 0.06$	$8.69\pm 5.28$
MAPPO	$0.22\pm 0.08$	$10.97\pm 0.01$	$-3.14\pm 1.79$
MAACKTR	$0.84\pm 0.03$	$\mathbf{26.18}\pm \mathbf{0.18}$	$-36.22\pm 3.78$

Note: Bold values indicate the best result for each metric among the compared methods or configurations.

The quantitative results in Table 3, Table 4 and Table 5 further demonstrate that BQDOM achieves the best overall performance under all three traffic modes. In Easy Mode, the collision rate, average driving speed, and evaluation reward of BQDOM are 0.02, 28.03, and 81.38, respectively, all outperforming the other comparative methods. Compared with MAA2C, MAPPO, and MAACKTR, the collision rate of BQDOM is reduced by 0.07, 0.06, and 0.21, respectively. Its average driving speed is improved by 5.89%, 15.54%, and 12.30%, respectively, while its evaluation reward is increased by 60.58%, 99.75%, and 1842.24%, respectively.

In Normal Mode, where traffic interaction complexity further increases, BQDOM still achieves the highest evaluation reward of 58.48 and the highest average driving speed of 26.58. Compared with MAA2C, BQDOM reduces the collision rate by 0.40, improves the average driving speed by 10.34%, and increases the evaluation reward by 53.62. Compared with MAACKTR, BQDOM reduces the collision rate by 0.54, improves the average driving speed by 13.06%, and substantially increases the evaluation reward. Although MAPPO achieves a collision rate close to that of the proposed method in this scenario, its average driving speed is only 12.04 and its evaluation reward is only 1.04, indicating a clearly conservative policy. In contrast, BQDOM achieves higher traffic efficiency and overall returns while maintaining safety.

In Hard Mode, faced with higher traffic density and stronger interaction conflicts, BQDOM still achieves the highest evaluation reward of 47.14, demonstrating better robustness. Compared with MAA2C, BQDOM reduces the collision rate by 0.36, improves the average driving speed by 9.22%, and increases the evaluation reward by 442.46%. Compared with MAACKTR, BQDOM reduces the collision rate by 0.66 and improves the evaluation reward by 83.36. Compared with MAPPO, although their collision rates are relatively close, BQDOM improves the average driving speed by 111.58% and increases the evaluation reward by 50.28. This indicates that the proposed method does not obtain local safety through excessive deceleration; instead, it achieves a better balance between safety and efficiency under risk constraints.

In summary, under the idealized highway simulation conditions used in this study, BQDOM achieves the lowest mean collision rate and the highest mean evaluation reward among the compared methods across the Easy, Normal, and Hard scenarios, while maintaining relatively high average driving speeds. These results suggest that the proposed dual action mask and Beam Search-based cooperative decision-making framework provides a favorable balance between safety and traffic efficiency in the simulated on-ramp merging scenarios.

To complement the performance comparisons above, an additional evaluation is conducted to examine the statistical significance of the differences between BQDOM and the baseline methods. The evaluation covers four methods under three traffic densities, yielding 12 model–density combinations. Each combination is evaluated over 1000 episodes, resulting in 12,000 evaluation episodes in total. Within each density, all methods use the same 1000 test scenarios, and their results are compared within each scenario. This additional evaluation quantifies the performance differences between the trained policies, while Figure 2, Figure 3 and Figure 4 describe their learning processes.

Two-sided permutation tests with 100,000 random permutations are used for average speed and evaluation reward, with the methods’ results exchanged only within the same test scenario. Collision events are compared using a two-sided exact McNemar test. Pointwise 95% confidence intervals for the performance differences are obtained using 10,000 bootstrap resamples, with both methods’ results from each selected scenario resampled together. Holm adjustment is applied jointly to all 27 comparisons, covering three metrics, three baselines, and three traffic densities. The significance level is set to 0.05, and the resulting values are reported as p-values in Table 6, Table 7 and Table 8. For collision rate, the difference is defined as the baseline value minus the BQDOM value and is expressed in percentage points. For average speed and evaluation reward, the difference is defined as BQDOM minus the baseline. Thus, a positive difference favors BQDOM.

Table 6. Statistical comparisons between BQDOM and the baseline methods in the additional Easy evaluation.

Baseline	Metric	Difference	95% CI	p-Value	Result
MAA2C	Collision rate	$+7.200$	$[5.062,9.338]$	<0.001	Better
MAA2C	Average speed	$+1.519$	$[1.440,1.596]$	<0.001	Better
MAA2C	Evaluation reward	$+27.106$	$[23.656,30.694]$	<0.001	Better
MAPPO	Collision rate	$+4.900$	$[2.939,6.861]$	<0.001	Better
MAPPO	Average speed	$+3.643$	$[3.512,3.776]$	<0.001	Better
MAPPO	Evaluation reward	$+32.123$	$[28.799,35.370]$	<0.001	Better
MAACKTR	Collision rate	$+22.400$	$[21.200,27.600]$	<0.000001	Better
MAACKTR	Average speed	$+3.474$	$[3.354,3.598]$	<0.001	Better
MAACKTR	Evaluation reward	$+82.340$	$[78.213,86.448]$	<0.001	Better

CI denotes confidence interval. Collision-rate differences are in percentage points; speed differences are in m/s. Better and Worse indicate statistically significant differences in favor of and against BQDOM, respectively, at the 0.05 level.

Table 7. Statistical comparisons between BQDOM and the baseline methods in the additional Normal evaluation.

Baseline	Metric	Difference	95% CI	p-Value	Result
MAA2C	Collision rate	$+39.900$	$[36.484,43.316]$	<0.000001	Better
MAA2C	Average speed	$+2.468$	$[2.356,2.575]$	<0.001	Better
MAA2C	Evaluation reward	$+66.093$	$[63.492,68.686]$	<0.001	Better
MAPPO	Collision rate	$+3.200$	$[0.896,5.504]$	$0.013189$	Better
MAPPO	Average speed	$+14.541$	$[14.435,14.643]$	<0.001	Better
MAPPO	Evaluation reward	$+59.317$	$[57.602,60.966]$	<0.001	Better
MAACKTR	Collision rate	$+55.500$	$[52.148,58.852]$	<0.000001	Better
MAACKTR	Average speed	$+3.031$	$[2.896,3.164]$	<0.001	Better
MAACKTR	Evaluation reward	$+88.141$	$[86.994,89.288]$	<0.001	Better

Differences, confidence intervals, and result labels follow the definitions in Table 6.

Table 8. Statistical comparisons between BQDOM and the baseline methods in the additional Hard evaluation.

Baseline	Metric	Difference	95% CI	p-Value	Result
MAA2C	Collision rate	$+33.800$	$[29.862,37.738]$	<0.000001	Better
MAA2C	Average speed	$+1.870$	$[1.769,1.970]$	<0.001	Better
MAA2C	Evaluation reward	$+33.182$	$[31.126,35.162]$	<0.001	Better
MAPPO	Collision rate	$+3.900$	$[0.331,7.469]$	$0.032421$	Better
MAPPO	Average speed	$+12.300$	$[12.221,12.380]$	<0.001	Better
MAPPO	Evaluation reward	$+48.216$	$[46.669,49.767]$	<0.001	Better
MAACKTR	Collision rate	$+64.800$	$[61.465,68.135]$	<0.000001	Better
MAACKTR	Average speed	$-2.942$	$[-3.038,-2.844]$	<0.001	Worse
MAACKTR	Evaluation reward	$+82.084$	$[80.480,83.749]$	<0.001	Better

Differences, confidence intervals, and result labels follow the definitions in Table 6.

Table 6, Table 7 and Table 8 report higher evaluation rewards for BQDOM than for all three baselines at each traffic density, with all nine comparisons yielding $p<0.001$ and confidence intervals entirely above zero. The reward gains over MAA2C, MAPPO, and MAACKTR are 27.106, 32.123, and 82.340 in Easy; 66.093, 59.317, and 88.141 in Normal; and 33.182, 48.216, and 82.084 in Hard, respectively. These results provide quantitative evidence for the evaluation-reward advantage across the three densities, complementing the training-curve comparisons.

The collision-rate comparisons also favor BQDOM across all three densities, with all nine reported p-values below 0.05. Relative to MAPPO, collision rates decrease by 4.9, 3.2, and 3.9 percentage points in Easy, Normal, and Hard, with $p<0.001$, $p=0.013189$, and $p=0.032421$, respectively. Average speed is higher in eight of the nine comparisons, while it is 2.942 m/s lower than MAACKTR in Hard; all speed comparisons yield $p<0.001$. In the latter comparison, BQDOM also achieves a collision-rate reduction of 64.8 percentage points and a reward increase of 82.084. Thus, the reported reward advantage is accompanied by lower collision rates and higher average speeds in most comparisons.

4.3. Parameter Sensitivity Experiment

This section examines the sensitivity of BQDOM to the trajectory prediction horizon H, the risk-mask distance threshold $d_{\mathrm{safe}}$, the risk-mask TTC threshold ${\tau }_{\mathrm{safe}}$, the mainline assistive-action gain ${\lambda }_a$, the safety regularization coefficient ${\lambda }_{\mathrm{safe}}$, the collision penalty coefficient ${\lambda }_{\mathrm{col}}$, and the Beam Search width B. The analysis considers collision avoidance, traffic efficiency, merging completion, and search cost to explain the parameter choices.

4.3.1. Trajectory Prediction Horizon

The trajectory prediction horizon H is a key hyperparameter of BQDOM. It directly determines the forward-looking time range of the risk assessment module, thereby affecting the filtering accuracy of the dual action mask and the reliability of joint action scoring. When H is too small, the model can only utilize interaction information within a very short horizon, making it difficult to anticipate potential merging conflicts. This may lead to delayed risk perception and short-sighted policies. Conversely, when H is too large, trajectory prediction errors may accumulate over time, and the uncertainty of human-driven vehicle (HDV) behavior may be amplified along the prediction horizon, resulting in distorted risk assessment and overly conservative decision-making.

To quantify the influence of H on model performance, a single-variable comparative experiment is conducted under the normal mode while keeping all other training parameters unchanged. Four experimental groups are configured with H = 4, H = 6, H = 8, and H = 10, respectively. The average episode reward during training is used as the evaluation criterion, and the experimental results are shown in Figure 6. Meanwhile, the collision rate, average driving speed, and evaluation reward under different prediction horizons are reported in Table 9.

Figure 6. Reward curve comparison under different prediction horizons.

Table 9. Evaluation metrics under different prediction horizons in the normal scenario.

Prediction Step	Collision Rate	Average Speed	Evaluation Reward
$H=4$	$0.21$	$24.79$	$52.34$
$H=6$	$0.10$	$25.96$	$60.69$
$H=8$	$\mathbf{0.07}$	$\mathbf{26.50}$	$\mathbf{60.97}$
$H=10$	$0.12$	$24.99$	$56.37$

Note: Bold values indicate the best result for each metric among the compared methods or configurations.

As shown in Figure 6, all four groups exhibit a rapid increase in reward during early training, followed by differences in reward level and fluctuation. The curve for $H=8$ maintains the highest reward level during the middle and later stages, with smaller overall fluctuations, indicating favorable learning performance at this prediction horizon.

The shorter horizons, $H=4$ and $H=6$, yield lower final rewards, while extending the horizon to $H=10$ also reduces the final reward and increases fluctuations relative to $H=8$. This non-monotonic trend supports a moderate prediction horizon that provides useful look-ahead information without excessively extending the prediction range.

Table 9 further shows that $H=8$ achieves the lowest collision rate of 0.07, the highest average driving speed of 26.50 m/s, and the highest evaluation reward of 60.97. Although $H=6$ achieves a similar evaluation reward, $H=8$ provides a lower collision rate and a higher average speed. Together with the training curves, these results support the favorable overall performance of $H=8$ in safety, traffic efficiency, and reward.

Therefore, $H=8$ is adopted as the default trajectory prediction horizon to balance forward-looking interaction information and overall decision-making performance.

4.3.2. Risk-Mask Thresholds

Table 10 and Table 11 report the influence of the risk-mask distance and TTC thresholds, respectively. A distance threshold of $d_{\mathrm{safe}}=2$ m achieves the lowest collision rate and the highest merging success rate in both the Normal and Hard scenarios. A smaller threshold favors higher driving speeds, whereas increasing the threshold further does not improve safety or merging completion. These results indicate that a moderate distance threshold balances available merging opportunities against conflict avoidance.

Table 10. Performance under different risk-mask distance thresholds.

Scenario	${\mathit{d}}_{\mathbf{Safe}}$ (m)	Collision Rate (%)	Average Speed (m/s)	Evaluation Reward	Merging Success Rate (%)
Easy	1	0.00	28.54	88.31	100.00
	2	0.00	28.26	87.30	100.00
	3	3.33	28.18	83.14	100.00
Normal	1	6.67	27.30	59.27	96.67
	2	3.33	26.63	63.89	98.33
	3	10.00	26.92	54.63	96.67
Hard	1	33.33	26.05	8.67	83.33
	2	20.00	24.44	18.31	94.44
	3	40.00	25.42	3.91	82.22

Note: Bold values indicate the best result for each metric within each scenario, including ties.

Table 11. Performance under different risk-mask TTC thresholds.

Scenario	${\mathit{\tau }}_{\mathbf{Safe}}$ (s)	Collision Rate (%)	Average Speed (m/s)	Evaluation Reward	Merging Success Rate (%)
Easy	0.5	0.00	28.51	87.71	100.00
	1.0	0.00	28.26	87.30	100.00
	1.5	0.00	28.18	86.46	100.00
Normal	0.5	10.00	27.62	58.00	95.00
	1.0	3.33	26.63	63.89	98.33
	1.5	3.33	26.90	66.04	100.00
Hard	0.5	43.33	26.01	1.47	80.00
	1.0	20.00	24.44	18.31	94.44
	1.5	33.33	24.85	8.59	90.00

Note: Bold values indicate the best result for each metric within each scenario, including ties.

For the TTC threshold, ${\tau }_{\mathrm{safe}}=1$ s achieves the lowest collision rate and the highest evaluation reward and merging success rate in the Hard scenario. The larger threshold performs favorably in Normal but provides no further improvement in Hard, showing that the benefit of a longer warning time depends on traffic interactions. Accordingly, $d_{\mathrm{safe}}=2$ m and ${\tau }_{\mathrm{safe}}=1$ s are adopted as the default risk-mask thresholds.

4.3.3. Mainline Assistive-Action Gain

Table 12 examines the mainline assistive-action gain ${\lambda }_a$. The zero-gain comparison removes the additional weighting of mainline assistive actions. The default ${\lambda }_a=0.5$ achieves the lowest collision rate and the highest evaluation reward and merging success rate in Hard. It also achieves the highest evaluation reward in Easy and Normal. Increasing the gain further does not yield additional improvement, indicating that moderate assistance weighting promotes cooperative merging without overemphasizing mainline assistive actions in joint-action scoring.

Table 12. Performance under different mainline assistive-action gains.

Scenario	${\mathit{\lambda }}_{\mathit{a}}$	Collision Rate (%)	Average Speed (m/s)	Evaluation Reward	Merging Success Rate (%)
Easy	0	0.00	28.06	85.95	100.00
	0.5	0.00	28.26	87.30	100.00
	0.9	0.00	28.13	87.27	100.00
Normal	0	3.33	26.88	61.92	100.00
	0.5	3.33	26.63	63.89	98.33
	0.9	10.00	26.65	56.98	96.67
Hard	0	30.00	25.37	6.72	82.22
	0.5	20.00	24.44	18.31	94.44
	0.9	26.67	24.79	10.69	83.33

Note: Bold values indicate the best result for each metric within each scenario, including ties.

4.3.4. Safety Regularization Coefficient

As shown in Table 13, ${\lambda }_{\mathrm{safe}}=0.2$ achieves the lowest collision rate in Normal and Hard and the highest evaluation reward in all three scenarios. Its merging success rate is also tied for the highest in each scenario. The lower average speeds in Normal and Hard accompany improved collision avoidance and merging completion, reflecting a preference for safer interactions rather than speed alone. These results support ${\lambda }_{\mathrm{safe}}=0.2$ as the default safety regularization coefficient.

Table 13. Performance under different safety regularization coefficients.

Scenario	${\mathit{\lambda }}_{\mathbf{Safe}}$	Collision Rate (%)	Average Speed (m/s)	Evaluation Reward	Merging Success Rate (%)
Easy	0	0.00	28.38	87.12	100.00
	0.1	0.00	27.63	84.51	100.00
	0.2	0.00	28.26	87.30	100.00
Normal	0	10.00	27.16	57.52	98.33
	0.1	6.67	26.81	59.27	96.67
	0.2	3.33	26.63	63.89	98.33
Hard	0	63.33	26.05	−3.99	72.22
	0.1	30.00	25.27	10.63	94.44
	0.2	20.00	24.44	18.31	94.44

Note: Bold values indicate the best result for each metric within each scenario, including ties.

4.3.5. Collision Penalty Coefficient

Table 14 presents the effect of the collision penalty coefficient. The default ${\lambda }_{\mathrm{col}}=400$ achieves a tied-lowest collision rate in each scenario, the highest merging success rate in Normal and Hard, and the highest evaluation reward in all three scenarios. In Hard, it maintains the collision rate obtained with the intermediate coefficient while improving merging completion and evaluation reward. Its average speed also remains above that of the intermediate setting. The results show that the stronger collision penalty supports safe merging while preserving traffic efficiency.

Table 14. Performance under different collision penalty coefficients.

Scenario	${\mathit{\lambda }}_{\mathbf{col}}$	Collision Rate (%)	Average Speed (m/s)	Evaluation Reward	Merging Success Rate (%)
Easy	100	3.33	28.50	86.35	100.00
	200	0.00	28.12	86.95	100.00
	400	0.00	28.26	87.30	100.00
Normal	100	3.33	26.61	60.71	96.67
	200	6.67	26.42	55.27	96.67
	400	3.33	26.63	63.89	98.33
Hard	100	26.67	24.53	10.62	88.89
	200	20.00	24.35	13.18	91.11
	400	20.00	24.44	18.31	94.44

Note: Bold values indicate the best result for each metric within each scenario, including ties.

4.3.6. Beam Search Width

Table 15 compares closed-loop performance under different Beam Search widths. The results in Easy are identical across the tested widths. In Normal, increasing B from 1 to 2 improves merging success from 97.00% to 98.50%, whereas larger widths provide no clear additional benefit. In Hard, increasing B from 1 to 4 reduces the collision rate from 25.00% to 22.00%, increases evaluation reward from 22.364 to 23.384, and raises merging success from 85.67% to 88.00%. Compared with $B=2$, $B=4$ further improves all three metrics. The results for $B=4$, $B=8$, and $B=16$ are identical, indicating that the observed closed-loop benefit levels off at $B=4$. Thus, $B=4$ is the smallest tested width that achieves the best observed collision rate, evaluation reward, and merging success in Hard.

Table 15. Closed-loop performance under different Beam Search widths.

Scenario	B	Collision Rate (%)	Average Speed (m/s)	Evaluation Reward	Merging Success Rate (%)
Easy	1	1.00	28.168	85.937	99.00
	2	1.00	28.168	85.937	99.00
	4	1.00	28.168	85.937	99.00
	8	1.00	28.168	85.937	99.00
	16	1.00	28.168	85.937	99.00
Normal	1	8.00	26.422	58.232	97.00
	2	8.00	26.426	58.238	98.50
	4	8.00	26.426	58.233	98.50
	8	8.00	26.426	58.193	98.50
	16	8.00	26.426	58.193	98.50
Hard	1	25.00	24.549	22.364	85.67
	2	23.00	24.533	22.571	86.33
	4	22.00	24.537	23.384	88.00
	8	22.00	24.537	23.384	88.00
	16	22.00	24.537	23.384	88.00

Note: Bold values indicate the best result for each metric within each scenario, including ties.

Table 16 further compares search quality and computational cost. $B=4$ is the smallest evaluated width that achieves a 100% optimal-score hit rate. Wider beams retain the same hit rate but expand more candidates and require longer search times. In Hard, $B=4$ reduces the mean candidate count by 45.50% and 69.80% relative to $B=8$ and $B=16$, respectively, and reduces search time by 3.53% and 10.38%. These observations agree with the candidate-expansion bound in Equation (38): increasing the beam width enlarges the search workload. Therefore, $B=4$ is adopted as the default Beam Search width to balance decision quality and search cost.

Table 16. Search cost and optimal-score hit rate under different Beam Search widths.

B	Search Time Easy (ms)	Search Time Normal (ms)	Search Time Hard (ms)	Mean Candidates Hard	Optimal-Score Hit Rate (%)
1	0.060	0.106	0.151	24.38	95
2	0.063	0.112	0.161	45.93	98
4	0.064	0.113	0.164	85.41	100
8	0.064	0.116	0.170	156.72	100
16	0.064	0.121	0.183	282.87	100

Note: Bold values indicate the lowest search time and candidate count, or the highest optimal-score hit rate, including ties.

4.4. Ablation Experiment

To verify the specific contributions of the key modules in the BQDOM framework, this section conducts ablation experiments focusing on the effects of the dual action mask, key-action weighting, and sequence scoring modules. Following the experimental logic of “complete model–key module removal–performance comparison,” four model configurations are designed:

• The complete BQDOM model;
• BQDOM without the dual action mask, where only the basic legality constraints of the environment are retained and the secondary filtering process of the risk mask is removed;
• BQDOM without key-action weighting, where differentiated weights are no longer assigned to key actions of ramp key vehicles and non-key mainline vehicles;
• BQDOM without the key-action-weighted scoring module, where joint action sequences are no longer scored and filtered, and the actions with the maximum Q-values estimated by the individual Q-network are directly output.

All experimental groups are conducted under the same simulation environment, training hyperparameters, and evaluation procedure. The training curves and quantitative results are shown in Figure 7 and Table 17, respectively.

Figure 7. Reward curve comparison under different configurations.

Table 17. Evaluation metrics under different configurations.

Configuration	Collision Rate	Average Speed	Evaluation Reward
BQDOM	$\mathbf{0.02}\pm \mathbf{0.02}$	$\mathbf{28.03}\pm \mathbf{0.28}$	$\mathbf{81.38}\pm \mathbf{3.72}$
BQDOM without dual mask	$0.14\pm 0.04$	$24.80\pm 0.15$	$54.31\pm 3.39$
BQDOM without key-action weighting	$0.08\pm 0.05$	$26.71\pm 0.08$	$60.72\pm 4.32$
BQDOM without sequence scoring	$0.14\pm 0.04$	$25.52\pm 0.06$	$55.83\pm 2.93$

Note: Bold values indicate the best result for each metric among the compared methods or configurations.

From the training process, the complete BQDOM clearly outperforms the other ablated models in terms of convergence speed, final reward level, and training stability. The reward curve of the complete model rises rapidly in the early training stage and remains at the highest level in the middle and later stages. By contrast, after removing the dual action mask, the convergence speed of the model becomes significantly slower, the training fluctuation is the most severe, and the final reward level is the lowest. This indicates that the dual action mask plays a crucial role in constraining high-risk exploration and improving training stability.

After removing the sequence scoring module or the key-action weighting setting, the model still retains a certain learning capability. However, the overall reward curves remain consistently lower than that of the complete model, and the fluctuation amplitude increases in the later training stage. This suggests that relying solely on local action values is insufficient to fully characterize the joint decision-making relationship in multi-vehicle cooperative merging. Moreover, the absence of key-action weighting weakens the model’s ability to focus on the core conflict agents in ramp merging scenarios, thereby affecting the overall decision quality.

The quantitative results in Table 17 show that the complete BQDOM achieves the best performance among the four configurations, with a collision rate of 0.02, an average driving speed of 28.03, and an evaluation reward of 81.38. After removing the dual action mask, the collision rate increases to 0.14, the average driving speed decreases to 24.80, and the evaluation reward decreases to 54.31. Compared with the complete model, the collision rate increases by six times, the average driving speed decreases by 11.52%, and the evaluation reward decreases by 33.26%. This demonstrates that the dual action mask can significantly compress the high-risk action space and serves as the core mechanism for ensuring safety and convergence efficiency.

After removing key-action weighting, the collision rate increases to 0.08, the average driving speed decreases to 26.71, and the evaluation reward decreases to 60.72. After removing the sequence scoring module, the collision rate further increases to 0.14, the average driving speed decreases to 25.52, and the evaluation reward decreases to 55.83. These two groups of results indicate that both key-action weighting and the scoring module contribute significantly to the overall performance of the model. Among them, removing the scoring module causes a slightly greater performance degradation than removing key-action weighting, suggesting that global scoring at the joint action level provides more direct support for multi-agent cooperative optimization. Key-action weighting further enhances the model’s ability to identify ramp key conflict agents and assistive actions.

In summary, the dual action mask, key-action weighting, and sequence scoring module jointly constitute the core mechanisms of BQDOM. The dual action mask suppresses invalid and high-risk exploration from the perspective of action-space constraints and provides the foundation for improving safety and training stability. Key-action weighting highlights the influence of key actions in joint decision-making and strengthens the model’s response to core conflict scenarios in ramp merging. The sequence scoring module improves the quality of joint action selection from the perspective of global cooperative optimization. These three components work together to enable BQDOM to jointly optimize safety, traffic efficiency, and cooperative returns in mixed-traffic on-ramp merging scenarios.

4.5. Generalization Evaluation Under Different CAV Penetration Rates

To evaluate the generalization of a trained policy across different mixed-traffic compositions, the BQDOM model trained in Hard Mode is directly evaluated at CAV penetration rates of 25%, 50%, 75%, and 100%. Hard Mode is selected because its larger traffic scale and more complex vehicle interactions provide a challenging training setting from which to assess policy generalization under changes in traffic composition. The same trained BQDOM model is used across all four penetration conditions, without penetration-specific retraining or fine-tuning. Thus, this experiment evaluates the direct application of a trained policy across penetration rates, rather than policies separately optimized for each penetration condition.

For a consistent comparison protocol, each baseline method (MAA2C, MAPPO, and MAACKTR) likewise uses its own model trained in the same Hard Mode scenario, without retraining or fine-tuning at the evaluated penetration rates. The total traffic volume, the remaining environment settings, and the evaluation procedure are kept consistent while the relative proportions of CAVs and HDVs are varied. The collision rate, average driving speed, and evaluation reward are compared in Figure 8, Figure 9 and Figure 10 and Table 18.

Figure 8. Collision rates of different models under different penetration rates.

Figure 9. Average speeds of different models under different penetration rates.

Figure 10. Evaluation rewards of different models under different penetration rates.

Table 18. Performance comparison of the four models under different penetration rates.

Method	Penetration Rate	Collision Rate	Average Speed	Evaluation Reward
BQDOM	$25\%$	$0.04$	$24.35$	$52.32$
	$50\%$	$0.09$	$24.39$	$50.01$
	$75\%$	$0.17$	$27.68$	$55.43$
	$100\%$	$0.19$	$26.58$	$39.24$
MAA2C	$25\%$	$0.10$	$22.23$	$36.01$
	$50\%$	$0.35$	$22.04$	$18.71$
	$75\%$	$0.42$	$22.19$	$15.37$
	$100\%$	$0.39$	$23.14$	$17.15$
MAPPO	$25\%$	$0.04$	$11.00$	$-0.88$
	$50\%$	$0.11$	$10.98$	$-1.47$
	$75\%$	$0.21$	$10.97$	$-3.42$
	$100\%$	$0.24$	$10.96$	$-3.94$
MAACKTR	$25\%$	$0.43$	$25.41$	$-22.34$
	$50\%$	$0.65$	$26.01$	$-42.31$
	$75\%$	$0.75$	$26.35$	$-58.66$
	$100\%$	$0.70$	$26.29$	$-40.65$

As shown in Figure 8, Figure 9 and Figure 10, different algorithms exhibit noticeable differences in safety, traffic efficiency, and overall returns under different CAV penetration rates. Overall, BQDOM maintains a relatively low collision rate and stable positive evaluation rewards across all four penetration rate scenarios, while its average driving speed remains at a relatively high level. These results support the ability of the Hard Mode-trained BQDOM policy to maintain favorable performance across the tested traffic compositions without penetration-specific retraining. In contrast, MAPPO exhibits a relatively conservative behavior. Although MAACKTR achieves a high average speed, it suffers from a high collision rate and negative rewards. To further quantify the performance differences among different methods, Table 18 reports the detailed evaluation metrics.

The quantitative results in Table 18 further show that BQDOM exhibits significant overall performance advantages across the full range of penetration rates. In terms of evaluation reward, BQDOM achieves reward values of 52.32, 50.01, 55.43, and 39.24 under 25%, 50%, 75%, and 100% penetration rates, respectively, which are consistently higher than those of MAA2C and MAPPO. In terms of traffic efficiency, the average driving speed of BQDOM remains above 24.35 m/s in all scenarios, outperforming MAA2C overall and reaching more than twice that of MAPPO. Although MAACKTR shows relatively high average speeds in repeated evaluations, its collision rate remains consistently high, and its evaluation rewards are significantly negative under all penetration rates. This suggests that MAACKTR tends to adopt an aggressive speed-oriented strategy and lacks an effective balance between safety and cooperation.

Compared with the baseline methods, MAA2C achieves substantially lower rewards and average speeds than BQDOM under all penetration rates, indicating clear limitations in cooperative decision-making. Although MAPPO maintains a relatively low collision rate under some penetration rates, its average driving speed remains around 11 m/s, and its evaluation rewards are negative throughout all scenarios. This indicates that MAPPO adopts a severely conservative policy and cannot effectively balance traffic efficiency. By contrast, BQDOM achieves higher traffic efficiency while ensuring safety, thereby obtaining a better trade-off between safety and efficiency.

Further analysis of the performance variation of BQDOM with respect to penetration rate shows that, as the CAV penetration rate increases from 25% to 75%, the average driving speed and evaluation reward of BQDOM generally exhibit an upward trend, with the best overall performance obtained at the 75% penetration rate. When the penetration rate increases to 100%, the evaluation reward decreases, indicating that increasing CAV penetration does not necessarily lead to monotonic performance improvement. In the fully CAV scenario, although the potential for system-level cooperation is higher, the coupling among multi-agent joint actions and the complexity of decision-making also increase, which affects model performance to some extent. Overall, BQDOM maintains stable positive returns and high traffic efficiency under different CAV penetration rates.These findings demonstrate that the Hard Mode-trained BQDOM policy maintains favorable performance across the evaluated CAV penetration rates without penetration-specific retraining, supporting its generalization across different traffic compositions within the tested simulation conditions.

4.6. Visualization Experiment

To further intuitively analyze the cooperative decision-making capability of the proposed BQDOM method in typical on-ramp merging scenarios, representative segments from the testing phase under Easy Mode, Normal Mode, and Hard Mode are selected for visualization, as shown in Figure 11. The figure presents the vehicle operation process in each scenario at three stages: Early Stage, Middle Stage, and Late Stage. The dashed arrows indicate the main movement direction or lane-changing tendency of CAVs.

Figure 11. Visualization of cooperative on-ramp merging under easy, normal, and hard traffic scenarios.

Figure 11 illustrates typical interactions between mainline and ramp vehicles under different traffic densities. In the selected sequences, mainline CAVs adjust their movements through lane keeping, lane changing, or yielding, while ramp vehicles progressively enter the mainline. These examples illustrate the temporal development of cooperative merging maneuvers.

To complement these qualitative observations, Table 19 summarizes batch evaluation results from 100 test episodes for each of the Easy, Normal, and Hard scenarios. Different evaluation experiments use different test seeds; therefore, numerical results may vary across evaluation batches. Three metrics are considered: collision rate, merging success rate, and mean merging time. Collision rate is the percentage of episodes in which at least one controlled CAV experiences a collision. Merging success rate is calculated as the fraction of initial ramp CAVs that complete merging in each episode, averaged equally across episodes. A merging completion is recorded when a ramp CAV first enters a mainline lane without being in a crashed state at that time.

Table 19. BQDOM performance across the three traffic scenarios used in the visualization analysis.

Scenario	Test Episodes	Collision Rate (%)	Merging Success Rate (%)	Mean Merging Time (s)
Easy	100	1.00	99.00	8.73
Normal	100	8.00	98.50	9.90
Hard	100	20.00	86.67	10.17

Merging time is measured from the beginning of the episode to the first recorded merging completion. The times of successfully merged vehicles are first averaged within each episode and then averaged equally across episodes containing at least one successful merge. Mean merging time is therefore interpreted together with merging success rate, as it describes completion time conditional on successful merging.

As shown in Table 19, BQDOM achieves merging success rates above 98% in both Easy and Normal, indicating that most ramp CAVs complete merging in these scenarios. The collision rate increases from 1.00% in Easy to 8.00% in Normal and 20.00% in Hard, while mean merging time increases from 8.73 to 9.90 and 10.17 s, respectively. Compared with Easy, Hard exhibits a 12.33-percentage-point decrease in merging success rate and a 1.44 s increase in mean merging time. These results indicate that greater scenario complexity is associated with higher collision risk, a lower merging completion rate, and longer successful-merge times. The quantitative results complement the illustrated cooperative maneuvers by characterizing safety and merging efficiency across the three traffic scenarios.

4.7. Quantitative Evaluation of CAV Trajectory Prediction

To independently evaluate the trajectory forecasts used by the risk mask, the LSTM predictor is compared with a constant-velocity (CV) baseline on identical CAV trajectories. This supplementary experiment uses a fixed decision model trained for 12,000 episodes in Easy Mode, separately from the main comparative training protocol in Section 4.1. It includes 30 test episodes in each of the Easy, Normal, and Hard scenarios, with test seeds disjoint from training and validation. The evaluated predictor follows the supervised training procedure in Section 3.4. Evaluation covers horizons from 0.2 to 1.6 s in 0.2 s increments, and only windows with complete future position labels for the corresponding horizon are included.

For a valid prediction window w and a horizon of K steps, the average displacement error (ADE) and final displacement error (FDE) are defined as

$${\mathrm{ADE}}_w\left(K\right)={\displaystyle \frac{1}{K}}{\displaystyle \sum _{h=1}^K}\parallel {\widehat{\mathbf{p}}}_{w,h}-{\mathbf{p}}_{w,h}{\parallel }_2,{\mathrm{FDE}}_w\left(K\right)={\parallel {\widehat{\mathbf{p}}}_{w,K}-{\mathbf{p}}_{w,K}\parallel }_2,$$

(48)

where ${\widehat{\mathbf{p}}}_{w,h}$ and ${\mathbf{p}}_{w,h}$ are the predicted and simulated ground-truth positions. The physical horizon is $K\Delta t$, with $K\in \{1,\dots ,8\}$ and $\Delta t=0.2$ s. Both metrics are measured in meters and lower values indicate higher accuracy. Window errors are first averaged within each episode, and episode means are then averaged with equal weight. The sets of valid windows may differ across horizons, but are identical for LSTM and CV at each horizon.

Figure 12 shows that prediction errors generally increase with the forecast horizon and that LSTM yields lower errors than CV at longer horizons. Figure 13 and Table 20 report the 1.6 s comparison. Relative to CV, LSTM reduces ADE by 26.51%, 20.35%, and 25.36% in Easy, Normal, and Hard Mode, respectively; the corresponding FDE reductions are 25.61%, 19.43%, and 23.64%. These improvements concern both the complete forecast interval and its endpoint.

Figure 12. CAV prediction errors over forecast horizons of 0.2–1.6 s. Dashed lines indicate CV and solid lines indicate LSTM. Results are averaged equally across 30 test episodes per scenario.

Figure 13. CAV prediction accuracy at 1.6 s. Error bars represent episode-bootstrap 95% confidence intervals for each method’s mean error.

Table 20. CAV prediction accuracy and relative reductions versus CV at 1.6 s.

Scenario	CV ADE (m)	LSTM ADE (m)	ADE Reduction (%)	CV FDE (m)	LSTM FDE (m)	FDE Reduction (%)
Easy	0.487	0.358	26.51	1.122	0.834	25.61
Normal	1.067	0.850	20.35	2.441	1.966	19.43
Hard	1.150	0.859	25.36	2.677	2.044	23.64

Note: Bold values indicate the lower ADE and FDE between the two prediction methods within each scenario.

To assess whether the reduction in prediction error relative to CV is statistically supported, we first calculate the error difference between LSTM and CV within each test episode. We then perform 1000 bootstrap resamples with replacement at the episode level, keeping the LSTM and CV results paired within each sampled episode. At the 1.6 s forecast horizon, the 95% confidence intervals for the mean differences (LSTM minus CV) in both ADE and FDE lie entirely below zero in all three scenarios. These results support lower mean prediction errors for LSTM than for CV.

4.8. Risk-Mask Robustness Under HDV Position Noise

To examine the robustness of the risk mask to HDV position measurement errors, a supplementary experiment introduces noise into the HDV positions observed by CAVs. The experiment covers the Easy, Normal, and Hard scenarios, with four position-noise standard deviations, $\sigma \in \{0,0.1,0.3,0.5\}$ m. Each condition is evaluated over 1000 episodes, yielding 12,000 episodes in total. The corresponding trained BQDOM model is used in each scenario, with all network parameters and decision settings fixed across noise levels and without retraining or fine-tuning.

At each decision step, independent zero-mean Gaussian noise with standard deviation $\sigma$ is added to the longitudinal and lateral position observations of HDVs. The perturbations affect perception only and are propagated through observation construction, HDV trajectory rollout, and risk-mask computation; the physical vehicle states remain unchanged. HDV trajectories are generated using the model-based rollout described in Section 3.5.2. This supplementary evaluation uses the model-internal risk mask based on geometric collision checks of predicted trajectories, with additional environment-level safety intervention disabled in all conditions. The same initial test scenes are used across noise levels, and all four performance metrics are calculated from the true simulation states.

As shown in Table 21, the largest absolute changes from the corresponding noise-free results are 2.50 percentage points in collision rate, 2.04 percentage points in merging success rate, 0.028 m/s in average speed, and 5.986 in evaluation reward. Average speed remains nearly unchanged, while safety and reward exhibit scenario-dependent responses to position noise.

Table 21. BQDOM performance under HDV position noise, with 1000 test episodes per condition.

Scenario	$\mathit{\sigma }$	Collision	Merging	Average Speed	Evaluation
	(m)	(%)	Success (%)	(m/s)	Reward
Easy	0.0	2.6	97.40	28.065	76.629
	0.1	3.0	97.20	28.061	76.000
	0.3	3.1	97.10	28.047	75.560
	0.5	3.2	97.30	28.043	75.312
Normal	0.0	8.8	92.55	26.777	53.724
	0.1	9.3	90.65	26.802	50.291
	0.3	9.5	91.70	26.801	50.853
	0.5	10.6	90.70	26.790	48.749
Hard	0.0	18.4	84.67	23.510	38.514
	0.1	19.2	83.60	23.506	34.824
	0.3	18.3	84.33	23.514	36.482
	0.5	20.9	82.63	23.482	32.528

Figure 14 further illustrates these trends. In Easy and Normal, collision rates increase with the noise standard deviation. Reward decreases slightly in Easy and declines overall in Normal, with a partial recovery at $\sigma =0.3$ m. In Hard, collision rate, merging success, and reward all improve at $\sigma =0.3$ m relative to $\sigma =0.1$ m, whereas $\sigma =0.5$ m produces the highest collision rate and the lowest merging success and reward in this scenario. The average-speed curves remain nearly flat across all three scenarios.

Figure 14. Effects of HDV position noise on BQDOM performance: (a) collision rate; (b) merging success rate; (c) average speed; and (d) evaluation reward.

These results show that position noise primarily affects safety and reward rather than traffic speed under the evaluated conditions. The risk mask evaluates action risks based on predicted trajectories, and position observation errors may affect risk assessment and subsequent action selection. This provides a possible explanation for the performance changes observed in the Normal and Hard scenarios. Incorporating position-measurement uncertainty into trajectory-conflict assessment offers a direct way to improve the reliability of risk filtering under noisy perception.

5. Conclusions and Future Work

This study focuses on highway on-ramp merging scenarios under mixed-traffic flow and addresses several key challenges, including the expansion of the joint-action space, the difficulty of jointly optimizing safety and efficiency, and the uncertainty of HDV behavior. To this end, a joint-action scoring-based multi-agent Q-learning method, termed BQDOM, is proposed. The proposed method enhances decision-making foresight through historical observation representation and short-term trajectory prediction, filters invalid and high-risk actions using legality and risk masks, and realizes efficient approximate solution of joint actions by combining key-action weighting with fixed-width Beam Search. Under the idealized highway simulation conditions evaluated in this study, BQDOM achieves mean collision rates of 0.02, 0.07, and 0.18 in the Easy, Normal, and Hard scenarios, respectively, representing relative reductions of 75.00%, 22.22%, and 18.18% compared with the best-performing baseline for this metric in each scenario. Together with the results for the other evaluation metrics, these findings suggest a favorable balance between safety and traffic efficiency within the evaluated simulation scenarios. The ablation experiments further support the contributions of the core modules to the observed performance under the same simulation conditions.

The conclusions are limited by the simulation environment and modeling assumptions used in this study. In particular, HDV behavior is represented by IDM/MOBIL rules, and HDV trajectory prediction uses a held-control approximation over a finite horizon; these assumptions may not capture the full variability of human driving. The trajectory-prediction evaluation is also based on simulated data. Performance under sensing errors beyond the tested Gaussian position noise, extreme weather, and real-world human-driving behavior remains to be evaluated. Future work will incorporate real-world traffic data and measurement uncertainty into trajectory-conflict assessment and examine the effects of behavior-model mismatch on cooperative merging performance.