Adaptive Constraint Regulation for Human Preference-Aware Safe Reinforcement Learning of On-Ramp Merging

Abstract

Reinforcement learning (RL) has been widely utilized for decision-making in highway on-ramp merging scenarios. However, most existing methods incorporate safety through reward functions, which may allow autonomous vehicles to trade safety for higher cumulative rewards. Moreover, personalized human risk preferences are rarely considered, making the learned policies difficult to adapt to heterogeneous user-specific risk requirements and potentially resulting in overly conservative or insufficiently cautious behaviors. To address these issues, this paper proposes a Risk-Aware Personal Preference-Based Safe Reinforcement Learning framework (RAPRL), for autonomous decision-making in on-ramp merging scenarios. Specifically, the high-level decision-making problem is formulated as a constrained Markov decision process (CMDP), in which safety requirements are explicitly represented as constraints rather than reward terms. To enable personalized safety regulation, a fuzzy logic mechanism is developed to adaptively determine the constraint cost limit according to the driver’s risk preference and the surrounding traffic density. The resulting safe RL problem is solved using a Lagrangian-based soft actor-critic algorithm (SAC). Furthermore, an Action Shielding Mechanism is designed to assess the potential risk of candidate actions before execution and replace unsafe or infeasible actions, thereby improving safety during both policy learning and execution. Theoretical analysis shows that the proposed shielding mechanism can reduce unsafe exploration and improve sample efficiency. Extensive simulations in on-ramp merging scenarios demonstrate that RAPRL effectively reduces safety violations while maintaining driving efficiency. Compared with the SAC Discrete method, the proposed method improves the success rate by 4.76% and reduces the collision ratio by 70%, indicating a better safety–efficiency trade-off.

1. Introduction

Merging into congested highway traffic remains a challenging task for autonomous vehicles (AVs), as the decision-making process must balance two conflicting objectives: safety and efficiency [1]. This trade-off becomes more complex in interactive traffic, where the ego CAV must make merging decisions under uncertain HDV responses and vehicle-to-vehicle interactions [2]. Meanwhile, the desired behavior of the ego CAV is not uniform across users. A conservative user or safety-conscious driver may prefer larger safety margins and more cautious merging maneuvers, whereas an aggressive user may accept a higher level of risk to reduce merge delay and improve traffic efficiency. Therefore, diverse driving preferences should be explicitly considered in autonomous decision-making, especially when the vehicle must regulate its safety–efficiency tradeoff in interactive traffic. To satisfy users’ risk expectations, decision-making systems of AVs should incorporate user preferences and provide multiple behavioral options.

Reinforcement learning (RL) has been widely used for AV decision-making tasks [3,4]. However, most methods often neglect user-specific risk preferences, resulting in learned policies that may not align with an individual’s risk perception, thereby reducing user satisfaction and trust in the system [5]. Moreover, ensuring safety within RL still remains challenging. Most existing approaches incorporate safety requirements into the reward function instead of enforcing them as explicit constraints, which may cause the agent to prioritize long-term rewards while compromising safety [6]. Therefore, it is essential to develop a risk-aware reinforcement learning framework that explicitly incorporates personal risk preferences into the decision-making process of autonomous driving.

The key problem addressed in this paper is to learn an on-ramp merging policy that can explicitly regulate safety constraints, adapt the allowable risk level to user-specific preferences and traffic conditions, and prevent unsafe high-level actions from being executed. This problem involves three coupled aspects: constraint-based safety regulation during policy learning, preference-aware adjustment of the safety cost limit, and execution-level filtering of risky actions before they are applied to the vehicle. To address this problem, we propose a Risk-Aware Personal Preference-Based Safe Reinforcement Learning framework (RAPRL) for autonomous on-ramp merging, as shown in Figure 1. The high-level decision-making problem is first formulated as a constrained Markov decision process (CMDP), where safety requirements are represented as explicit cost constraints and the cost limit is adaptively determined according to human risk preference and traffic density, as shown in Figure 2. Based on this formulation, the resulting safe RL problem is solved using a Lagrangian-based soft actor-critic (SAC) algorithm. To further reduce unsafe exploration and prevent risky decisions from being executed, an MPC-based Action Shielding Mechanism (ASM) is introduced to pre-execute candidate RL actions, evaluate their predicted collision risk and feasibility, and replace unsafe or invalid actions before execution. After this shielding process, the selected action is sent to the low-level MPC module for trajectory planning and vehicle control. Finally, theoretical analysis and numerical simulations are conducted to evaluate the safety performance and learning efficiency of the proposed method. The main contributions are summarized as follows:

• A risk-aware safe RL method based on CMDP is proposed for the highway on-ramp merging task, in which the individual’s risk preference is incorporated into the safety constraints to accommodate the safety expectations of users. The safety level of the RL policy can be adjusted by computing the cost limits of CMDP constraints using fuzzy logic based on user preferences and traffic density.
• An Action Shielding Mechanism is built to mask out unsafe RL actions. We pre-execute the RL action with MPC and conduct collision checks with surrounding agents to determine whether the action is safe. Theoretical proof has shown the effectiveness of the shielding mechanism in terms of safety and sampling efficiency.
• Numerical simulations in different levels of traffic densities have shown that our method outperforms the baselines, which can improve safety without sacrificing traffic efficiency. Due to the use of user preference-aware safety constraints and action shielding, risk behaviors can be significantly reduced during the exploration stage of RL, enabling safer policy learning in interactive ramp-merging scenarios.

2. Related Works

2.1. RL-Based Approach

Reinforcement Learning (RL) has demonstrated significant potential in the complex decision-making tasks of autonomous driving, particularly in handling interaction and mixed traffic scenarios [7,8]. While these approaches effectively optimize driving policies for high-level objectives, standard RL algorithms primarily focus on maximizing cumulative rewards and often lack explicit mechanisms to guarantee safety constraints during the learning process. To address this limitation, the constrained Markov decision process (CMDP) framework was introduced to explicitly incorporate safety constraints. For instance, ref. [9] formulated unsignalized intersection navigation as a CMDP, solving it via a Safe-PPO algorithm, while [10] combined the Augmented Lagrangian method with PPO to enforce risk boundaries. However, CMDP-based methods typically treat safety as a statistical constraint or a soft penalty. As highlighted by [11], it remains difficult for these methods to strictly enforce hard safety constraints, leaving the agent prone to falling into unsafe regions during exploration. To mitigate the risk of unsafe actions, the Action Shielding Mechanism (ASM) has been developed to monitor and correct policy outputs [12]. Recent works have incorporated motion prediction into shielding modules to anticipate and mask actions that lead to collisions [13,14,15]. However, most existing shielding approaches rely on simplified kinematic predictions or immediate action masking, often overlooking the discrepancy between high-level decisions and the actual trajectories executed by low-level controllers. Furthermore, a theoretical justification is often absent.

Unlike existing studies, we pre-execute the RL action with MPC to acquire the predictions of the ego vehicle and then conduct collision checks with surrounding agents based on their motion predictions to determine whether the RL action is safe or not. The unsafe actions are replaced with safe ones, which can help the RL agent quickly learn to act safely. Moreover, we provide a theoretical justification that the Action Shielding Mechanism can enhance the safety performance and learning efficiency of RL.

2.2. Human Risk Perception in Decision-Making

Risk perception in driving has gained wide scientific interest over the years, with differences among individuals’ risk perception found to have a significant impact on the safety design of intelligent vehicles [16]. Many studies have used statistical methods to analyze driving risk, among which the most common method is to assess risk by analyzing driving behaviors. To understand the differences in risk perception between drivers, driving style has been studied. Ref. [17] proposes a universal driving risk model, which provides a new way of modeling the differences in driver’s risk perception at a physical level. Risk perception is also the main consideration in decision-making for driving safety. Ref. [18] uses the risk of each sample trajectory as the cost item and obtains a trajectory with minimal cost to guarantee the safety requirement. Ref. [19] proposes a model to describe the acceptable risk and shows how an accepted risk contributes to the decision-making of AVs at the maneuver level. Ref. [20] proposes a risk-aware decision-making framework to handle the epistemic uncertainty arising from training the prediction model on insufficient data.

Unlike previous works, we explicitly incorporate human risk perception into the CMDP framework to align high-level decisions with user expectations. Specifically, we employ a fuzzy logic controller that dynamically adjusts the CMDP cost limits based on individual risk preferences and real-time traffic density, thereby enabling adaptable safety levels.

2.3. Combination of RL and MPC

RL and Model Predictive Control (MPC) represent two distinct paradigms in autonomous decision-making. While RL excels in handling complex system dynamics and learning from interaction, it often struggles with generalization and safety guarantees in unseen environments. Conversely, MPC provides robust constraint handling but is limited by model accuracy and computational burden. Consequently, recent research has increasingly focused on hybrid architectures that leverage the strengths of both. For instance, ref. [21] formulated a dual MPC where an RL agent acts as an adaptive solver, and [22] utilized RL to estimate the value function within a parametric MPC framework. Others employ MPC to guide RL training; specifically, MPC has been used to reduce sample complexity [23] or approximate value functions for high-level objectives [24,25]. Furthermore, ref. [26] demonstrated the feasibility of embedding RL techniques directly into MPC-based policies. These studies collectively highlight the potential of RL–MPC integration in complex control tasks. However, most existing hybrid approaches focus primarily on performance optimization or value approximation, often overlooking the critical aspect of explicit safety constraints during the exploration phase.

Unlike the previous studies, we formulate the on-ramp merging problem within a CMDP framework, where safety violations are explicitly constrained via cost functions rather than implicitly handled through reward shaping. Moreover, we introduce an Action Shielding Mechanism that leverages MPC as a predictive safety filter to prevent unsafe actions during the exploration phase of RL. This design enables safety-aware learning under explicit constraints while maintaining compatibility with downstream MPC-based trajectory planning.

3. Problem Statement

This section formulates the decision-making process of the on-ramp merging problem as a hierarchical framework that consists of a CMDP for high-level maneuver decisions and MPC for low-level motion control.

3.1. Constrained Markov Decision Process

The high-level maneuver decision-making problem is formulated as a CMDP model in this study, which specifies safety requirements as constraint terms. CMDP is described as a tuple $<S,A,P,r,c,\eta ,\gamma >$, where S is the state space, A is the action space, P is the transition probability $p\left(s_{t+1}\right|s_t,a_t)$ from state $s_t$ to the next state $s_{t+1}$ under the action $a_t\in A$, $r:S\times A\to \mathbb{R}$ is the reward function, $c:S\times A\to \mathbb{R}$ is the cost function, and $\gamma \in [0,1)$ is a discount factor.

3.1.1. State Space

Let $s^i=(o^i,x^i,y^i,v_x^i,v_y^i)$ be the state of vehicle i, where $o^i$ is a binary variable indicating whether the vehicle i is observable, $x^i$ and $y^i$ are the longitudinal and lateral distance between the vehicle i and the ego vehicle, and $v_x^i$ and $v_y^i$ are relative speed between the vehicle i and the ego vehicle in longitudinal and lateral directions. The complete state of the environment consists of all vehicles that are observed by the ego vehicle, and the state space S is written as, $S=\left(s^1,s^2,\dots ,s^n\right)$, where n is the maximum number of observed vehicles on the main lane.

3.1.2. Action Space

The high-level decision module outputs a discrete action, including turning left/right, idling, and acceleration/deceleration. The action space A is defined as

$$A=\left[A^{◃},A^{▹},A^{\uparrow },A^{\oslash },A^{\downarrow }\right],$$

(1)

where $A^{◃},A^{▹},A^{\uparrow },A^{\oslash },$ and $A^{\downarrow }$ are turning left, turning right, acceleration, idling, and deceleration, respectively.

3.1.3. Reward

The reward functions are defined considering safety, success ratio, and traffic efficiency, and we summarize the rewards as

$$r=r_{\mathrm{s}}+r_{\mathrm{g}}+r_{\mathrm{v}},$$

(2)

where r is the sum of rewards, consisting of the reward of safety $r_{\mathrm{s}}$, the reward of successfully merging and reaching the goal position $r_{\mathrm{g}}$, and the reward of traffic efficiency $r_{\mathrm{v}}$. We set

$$r_{\mathrm{s}}=\left\{\begin{array}{cc}0.05\hfill & \mathrm{if}\mathrm{TTC}\ge 2.5\mathrm{s},\hfill \\ -1\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(3)

where TTC denotes the time to collision with surrounding vehicles. The term $r_{\mathrm{g}}$ is defined as $r_g=r_{\mathrm{g}1}+r_{\mathrm{g}2}$, with $r_{\mathrm{g}1}=5$ awarded when the ego vehicle successfully merges into the target lane, and $r_{\mathrm{g}2}=10$ awarded when the ego vehicle reaches the designated goal position. The reward for traffic efficiency $r_{\mathrm{v}}$ is defined based on ego vehicle speed and traffic speed, which encourages the ego vehicle to drive at the average speed of the traffic flow,

$$r_{\mathrm{v}}=\left\{\begin{array}{cc}0.1\hfill & \mathrm{if}\left|v_{\mathrm{ego}}-v_{\mathrm{ave}}\right|\le \kappa v_{\mathrm{ave}},\hfill \\ -0.5\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(4)

where $v_{\mathrm{ego}}$ is the speed of ego vehicle, $v_{\mathrm{ave}}$ is the average speed of traffic, and $\kappa$ is the coefficient of the average speed.

3.1.4. Cost

The cost c includes three situations, as illustrated in Figure 3, which are computed based on the predicted states of the ego vehicle and surrounding objects, including (a) fail to merge before reaching the end of the road, (b) collide with other vehicles, and (c) hard to merge when the target lane is occupied. The cost functions can be written as,

$$c=c_1+c_2+c_3,$$

(5)

where c is the sum of costs, including the cost $c_1$ for failing to merge before reaching the end of the road, the cost $c_2$ for colliding with other vehicles, and the cost $c_3$ for hard to merge when the target lane is occupied.

Similar event-based cost designs have been used in reinforcement learning-based autonomous driving, where hazardous driving events are represented by penalty terms and assigned different weights according to their safety severity [27]. Inspired by this design principle, we define the CMDP cost in this study based on the following three safety-related events in the on-ramp merging task.

$$\left\{\begin{array}{cc}{c}_1=0.2\hfill & \mathrm{if}\mathrm{failed}\mathrm{to}\mathrm{merge},\hfill \\ c_2=2\hfill & \mathrm{if}\mathrm{collision}\mathrm{occurs},\hfill \\ c_3=0.3\hfill & \mathrm{if}{l}_{\mathrm{occupied}}=1.\hfill \end{array}\right.$$

(6)

The cost values in Equation (6) are dimensionless penalty weights rather than physical risk quantities. They are selected according to the relative severity of the corresponding events. Collision is regarded as the most severe safety violation and is therefore assigned the largest cost. Failing to merge before the end of the ramp is treated as a task-level failure, while target-lane occupancy represents a local merging difficulty that may lead to unsafe interaction if the ego vehicle continues to merge. Accordingly, the cost values are designed to satisfy the severity ordering $c_2>c_3>c_1$, where $c_2$ strongly penalizes collision events, and $c_1$ and $c_3$ provide additional constraint signals for non-collision risks. The target-lane occupancy cost $c_3$ is set slightly larger than $c_1$ because it acts as an early warning signal for potential merging conflicts and may be triggered repeatedly during the merging process. These values are empirically selected in simulation to maintain a clear distinction between collision and non-collision events while keeping the cumulative cost within a suitable range for Lagrangian-based policy optimization.

In particular, we use the longitudinal position and relative speed to determine whether the target lane is occupied,

$$l_{oc}^i=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\left|\Delta v_x^i\right|\le 1.5\mathrm{and}\left|\Delta x^i\right|\le 5,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(7)

$$l_{\mathrm{occupied}}=\underset{i}{\max }\left[l_{oc}^i\right],$$

(8)

where $l_{\mathrm{occupied}}$ denotes the target lane occupancy status. $\Delta v_x^i$ and $\Delta x^i$ represent the longitudinal relative speed and distance of vehicle i, respectively. Specifically, $l_{\mathrm{occupied}}$ is set to 1 only if a vehicle falls within the defined spatial and speed thresholds, identifying it as an immediate obstacle for merging.

3.1.5. Problem Formulation

We formulate the decision-making problem as a CMDP to maximize cumulative rewards subject to safety constraints:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \underset{\pi }{max}& J_{\pi }^R={\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }}{\gamma }^{t}r(s_t,a_t)\right],\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& J_{\pi }^C={\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }}{\gamma }^{t}c(s_t,a_t)\right]\le \eta ,\hfill \end{array}\end{array}$$

(9)

where $J_{\pi }^R$ and $J_{\pi }^C$ denote the expected return and cost, respectively. $\eta$ is the cost limit reflecting risk preference. By introducing a Lagrange multiplier $\lambda \ge 0$, we recast this as an unconstrained min–max problem:

$$\underset{\pi }{max}\underset{\lambda \ge 0}{min}\left(J_{\pi }^R-\lambda (J_{\pi }^C-\eta )\right).$$

(10)

This Lagrangian formulation penalizes policies that violate the cost constraint ($J_{\pi }^C>\eta$), while maximizing the reward when the constraint is satisfied.

3.2. Human-Aligned Safety Cost Limits

As discussed previously, the cumulative costs $J_{\pi }^C$ are constrained by the cost limit $\eta$ in Equation (9), which determines the risk preference of the learned policy. Setting an appropriate value for $\eta$ is essential to match the users’ preferences for safety. In this study, we utilize the fuzzy logical method [28] to determine the value of the cost limit $\eta$ based on the specific risk preference and traffic density. We employ Mamdani inference [29] to accommodate human-semantic descriptors and encode them as interpretable rules, thereby deriving the safety cost limit from risk preference and traffic density. Unlike quantitative risk assessments that typically assume a single acceptable risk level, fuzzy logic allows the cost limit to adapt jointly to both preference and density.

In this study, we design the membership functions based on expert knowledge to map the fuzzy inputs (risk preference and traffic density) to the fuzzy output (cost limit), as illustrated in Figure 4. To make the membership functions explicit, let $p\in [0,100]$ denote the normalized risk-preference variable, $\rho \in [0.5,1]$ denote the normalized traffic-density variable, and $\eta \in [0,0.1]$ denote the cost limit. The membership functions of risk preference are defined as

$$\begin{array}{c}\begin{array}{ccc}\hfill {\mu }_{\mathrm{con}}\left(p\right)& =\left\{\begin{array}{cc}1,\hfill & 0\le p\le 30,\hfill \\ {\displaystyle \frac{50-p}{20}},\hfill & 30<p<50,\hfill \\ 0,\hfill & 50\le p\le 100,\hfill \end{array}\right.{\mu }_{\mathrm{neu}}\left(p\right)\hfill & \hfill =\left\{\begin{array}{cc}0,\hfill & 0\le p\le 30,\hfill \\ {\displaystyle \frac{p-30}{20}},\hfill & 30<p<50,\hfill \\ {\displaystyle \frac{70-p}{20}},\hfill & 50\le p<70,\hfill \\ 0,\hfill & 70\le p\le 100,\hfill \end{array}\right.\end{array}\\ {\mu }_{\mathrm{agg}}\left(p\right)=\left\{\begin{array}{cc}0,\hfill & 0\le p\le 50,\hfill \\ {\displaystyle \frac{p-50}{20}},\hfill & 50<p<70,\hfill \\ 1,\hfill & 70\le p\le 100.\hfill \end{array}\right.\end{array}$$

(11)

Similarly, the membership functions of traffic density are defined as

$$\begin{array}{c}\begin{array}{ccc}\hfill {\mu }_{\mathrm{low}}\left(\rho \right)& =\left\{\begin{array}{cc}1,\hfill & \rho =0.5,\hfill \\ {\displaystyle \frac{0.7-\rho }{0.2}},\hfill & 0.5<\rho <0.7,\hfill \\ 0,\hfill & 0.7\le \rho \le 1,\hfill \end{array}\right.{\mu }_{\mathrm{med}}\left(\rho \right)\hfill & \hfill =\left\{\begin{array}{cc}0,\hfill & \rho =0.5,\hfill \\ {\displaystyle \frac{\rho -0.5}{0.2}},\hfill & 0.5<\rho <0.7,\hfill \\ 1,\hfill & 0.7\le \rho \le 0.8,\hfill \\ {\displaystyle \frac{1-\rho }{0.2}},\hfill & 0.8<\rho <1,\hfill \\ 0,\hfill & \rho =1,\hfill \end{array}\right.\end{array}\\ {\mu }_{\mathrm{high}}\left(\rho \right)=\left\{\begin{array}{cc}0,\hfill & 0.5\le \rho \le 0.8,\hfill \\ {\displaystyle \frac{\rho -0.8}{0.2}},\hfill & 0.8<\rho <1,\hfill \\ 1,\hfill & \rho =1.\hfill \end{array}\right.\end{array}$$

(12)

The membership functions of the cost limit are defined as

$$\begin{array}{c}\begin{array}{ccc}\hfill {\mu }_{\mathrm{small}}\left(\eta \right)& =\left\{\begin{array}{cc}1,\hfill & 0\le \eta \le 0.01,\hfill \\ {\displaystyle \frac{0.05-\eta }{0.04}},\hfill & 0.01<\eta <0.05,\hfill \\ 0,\hfill & 0.05\le \eta \le 0.1,\hfill \end{array}\right.{\mu }_{\mathrm{medium}}\left(\eta \right)\hfill & \hfill =\left\{\begin{array}{cc}0,\hfill & 0\le \eta \le 0.01,\hfill \\ {\displaystyle \frac{\eta -0.01}{0.04}},\hfill & 0.01<\eta <0.05,\hfill \\ {\displaystyle \frac{0.09-\eta }{0.04}},\hfill & 0.05\le \eta <0.09,\hfill \\ 0,\hfill & 0.09\le \eta \le 0.1,\hfill \end{array}\right.\end{array}\\ {\mu }_{\mathrm{large}}\left(\eta \right)=\left\{\begin{array}{cc}0,\hfill & 0\le \eta \le 0.05,\hfill \\ {\displaystyle \frac{\eta -0.05}{0.03}},\hfill & 0.05<\eta <0.08,\hfill \\ 1,\hfill & 0.08\le \eta \le 0.1.\hfill \end{array}\right.\end{array}$$

(13)

These equations correspond to the membership curves shown in Figure 4a–c, and provide explicit mathematical definitions for the fuzzy inputs and output used in the Mamdani inference process. The above ranges and membership shapes are selected to support smooth safety–efficiency regulation in on-ramp merging. The normalized risk-preference range allows different user-specified risk tolerances to be represented on a common scale, while the normalized traffic-density range reflects the transition from sparse to dense merging conditions considered in the simulation. The cost-limit range is consistent with the sensitivity analysis of the CMDP constraint limit. For the membership shapes, shoulder functions are used for boundary categories, such as conservative/aggressive preference, low/high density, and small/large cost limit, whereas triangular or trapezoidal functions are used for intermediate categories. This overlapping design avoids abrupt changes in the cost limit when the input variables vary slightly, which is important for maintaining stable safety–efficiency tradeoffs during merging decisions.

Consistent with established driving-style taxonomies in driving behavior research [30], the user’s risk preference is represented by three commonly used linguistic levels, i.e., conservative, neutral, and aggressive. As shown in Figure 4a, these levels are modeled as predefined fuzzy membership functions over the normalized risk-preference variable ranging from 0 to 100%. They are not inferred online from driver data in the current study; instead, the membership functions are specified based on prior studies and expert assumptions to systematically evaluate how different risk-tolerance levels affect the adaptive safety cost limit and the learned merging policy. In practical applications, the normalized risk-preference input can be obtained through several interfaces, such as a pre-driving questionnaire, a user-selected driving-style setting, or calibration from historical driving behavior. The corresponding membership degrees with respect to the conservative, neutral, and aggressive sets are then used as inputs to the subsequent fuzzy inference process. Therefore, the focus of this study is not to infer the user’s risk preference, but to translate a given preference into an adaptive CMDP cost limit for safe policy learning. Traffic density includes {‘low’, ‘medium’, ‘high’}, as shown in Figure 4b, and varies between 0.5 and 1.0, where larger values indicate denser traffic conditions with smaller inter-vehicle headways and more surrounding vehicles within the ego vehicle’s interaction range. The medium-density traffic is modeled with a trapezoidal membership function, reaching its maximum membership value at moderate density levels. The membership function of cost is shown in Figure 4c, including {‘small’, ‘medium’, ‘large’}, and the cost limits vary between 0 and 0.1.

After fuzzification, we can define fuzzy rules that map the fuzzy inputs (risk preference and traffic density) to the fuzzy output (cost limit). Fuzzy rules are a crucial aspect of designing a fuzzy logic system, and the rules are usually derived from expert knowledge or empirical data. In this study, the fuzzy rules are defined based on human knowledge, and

Table 1. Fuzzy relations between the cost limit, traffic density, and risk preference.

		Traffic Density	High	Medium	Low
	Cost Limit
Risk Preference
Conservative			Small	Small	Medium
Neutral			Small	Medium	Large
Aggressive			Medium	Large	Large

shows the fuzzy relations between cost limit, risk preference, and traffic density.

The rule table is designed to reflect the safety–efficiency tradeoff in on-ramp merging. For a fixed traffic density, a higher risk preference is assigned a larger cost limit, because the ego vehicle is allowed to accept a slightly higher level of risk to reduce merge delay and improve traffic efficiency. Conversely, for a fixed risk preference, a higher traffic density is assigned a smaller cost limit, because dense traffic provides fewer acceptable merging gaps and requires stricter safety regulation. Therefore, the rule table follows a monotonic and interpretable design: the cost limit increases with user risk tolerance and decreases with traffic density. For example, an aggressive preference under low-density traffic leads to a large cost limit to encourage efficient merging, whereas a conservative preference under high-density traffic leads to a small cost limit to enforce cautious behavior. From Table 1, we can infer the fuzzy output of cost limit given the fuzzy inputs of risk preference and traffic density. For instance, if traffic density is ‘low’ and the risk preference is ‘aggressive’, then the cost limit is ‘large’. We use a matrix $\tilde{R}$ to represent the fuzzy rules, and the process of rule evaluations can be written as

$$\tilde{C}={\displaystyle \bigcup _{k=1}^9}({\tilde{A}}_i\times {\tilde{B}}_j)\circ {\tilde{R}}_k,$$

(14)

where $\tilde{C}$ is the fuzzy output, ${\tilde{A}}_i$ is the traffic density, ${\tilde{B}}_j$ is the risk preference, $i,j\in \{1,2,3\}$ are the indices of different fuzzy sets w.r.t. risk preference and traffic density, and the operator ∘ denotes the composition of fuzzy relations.

Based on the fuzzy rules, the fuzzy output of each rule can be computed. These fuzzy outputs are aggregated into a single fuzzy set and converted to the crisp output value using the centroid method. Figure 5 illustrates an example of aggregation and defuzzification of the Mamdani process. Given that the traffic density and risk preference are 0.57 and 45%, and the membership value of the input is $\tilde{A}=\{\mathrm{Low},\mathrm{Medium}\}$ and $\tilde{B}=\{\mathrm{Conservative},\mathrm{Neutral}\}$, we can compute the fuzzy output for the cost limit of small, medium, and large, as shown in Figure 5, which are 0.25, 0.35, and 0.65, respectively. With the fuzzy output, we can compute the grey area for each fuzzy set and aggregate them into a single fuzzy set, which is the union area of three grey areas. Finally, we take the centroid of the unions of grey areas to obtain a crisp value for the cost limit, which is 0.0595. Therefore, the cost limit is 0.0595 when traffic density is 0.57 and the risk preference is 45%.

4. Model Predictive Control

This section introduces the MPC formulation used for vehicle control, including the linearized discrete-time kinematic bicycle model, the prediction of future states over a finite horizon, and the quadratic optimization of control inputs under constraints.

4.1. Discrete Linear Model

In this study, we use the kinematic bicycle model in MPC [31]. As illustrated in Figure 6, the vehicle state is written as $\mathbf{X}={[x,y,v,\phi ]}^{\top }$, where x and y denote the longitudinal and lateral position, and v and $\phi$ are vehicle speed and the yaw angle. The side-slip angle, wheelbase, and length from the front and rear axles to the center of gravity are denoted as $\beta$, l, $l_f$, and $l_r$, respectively. The control variable is denoted as $\mathbf{U}={[u_1,u_2]}^{\top }$, where $u_1$ and $u_2$ are acceleration a and the steering angle $\delta$, respectively. The referenced position and control variable are defined as ${[x_r,y_r,v_r,{\phi }_r]}^{\top }$ and ${[v_r,{\delta }_r]}^{\top }$. We derive a linear model

$$\begin{array}{c}\hfill \dot{\mathbf{X}}=f(\mathbf{X},\mathbf{U})=\left[\begin{array}{c}vcos(\phi +\beta )\\ vsin(\phi +\beta ))\\ a\\ {\displaystyle \frac{v}{l}}sin\beta \end{array}\right],\end{array}$$

(15)

$$\begin{array}{c}\hfill \beta =arctan({\displaystyle \frac{l_r}{l_f+l_r}}tan\delta ),\end{array}$$

(16)

$$\begin{array}{c}\hfill {\dot{\mathbf{e}}}_{\mathbf{X}}=A_F{\mathbf{e}}_{\mathbf{X}}+B_G{\mathbf{e}}_{\mathbf{U}},\end{array}$$

(17)

$$\begin{array}{c}\hfill A_F=\left[\begin{array}{cccc}0& 0& cos({\phi }_r+{\beta }_r)& -v_{r}sin({\phi }_r+{\beta }_r)\\ 0& 0& sin({\phi }_r+{\beta }_r)& v_{r}cos({\phi }_r+{\beta }_r)\\ 0& 0& 0& 0\\ 0& 0& {\displaystyle \frac{sin{\beta }_r}{l}}& 0\end{array}\right],\end{array}$$

(18)

$$\begin{array}{c}\hfill B_G=\left[\begin{array}{cc}0& {\displaystyle \frac{-v_{r}sin({\phi }_r+{\beta }_r)l_r(l_f+l_r)}{{(l_f+l_r)}^2{cos}^2{\delta }_r+l_r{tan}^2{\delta }_r}}\\ 0& {\displaystyle \frac{-v_{r}cos({\phi }_r+{\beta }_r)l_r(l_f+l_r)}{{(l_f+l_r)}^2{cos}^2{\delta }_r+l_r{tan}^2{\delta }_r}}\\ 1& 0\\ 0& {\displaystyle \frac{\frac{v_r}{l}cos{\beta }_r{l}_r(l_f+l_r)}{{(l_f+l_r)}^2{cos}^2{\delta }_r+l_r{tan}^2{\delta }_r}}\end{array}\right],\end{array}$$

(19)

where $A_F\in {\mathbb{R}}^{4\times 4}$ and $B_G\in {\mathbb{R}}^{4\times 2}$ are the Jacobi matrix of the function f with respect to state X and control U, respectively. The state error between the predicted state and the reference state is denoted as ${\mathbf{e}}_{\mathbf{X}}\in {\mathbb{R}}^4$. The control error between the predicted control and the reference control is denoted as ${\mathbf{e}}_{\mathbf{U}}\in {\mathbb{R}}^2$. We discretize the linear model as

$$\begin{array}{c}\hfill {\mathbf{e}}_{\mathbf{X}}\left(k\right)=A_k{\mathbf{e}}_{\mathbf{X}}(k-1)+B_k{\mathbf{e}}_{\mathbf{U}}(k-1),\end{array}$$

(20)

where ${\mathbf{e}}_{\mathbf{X}}\left(k\right)$ is the state error at time k, ${\mathbf{e}}_{\mathbf{X}}(k-1)$ is the state error at time $k-1$, and ${\mathbf{e}}_{\mathbf{U}}(k-1)$ is the control error at time $k-1$. The system matrix $A_k$ and input matrix $B_k$ are defined as

$$\begin{array}{c}\hfill A_k=A_F·\Delta t+\mathrm{I},\end{array}$$

(21)

$$\begin{array}{c}\hfill B_k=B_G·\Delta t,\end{array}$$

(22)

where $\Delta t$ is the time interval. We define the predicted state error vector $E_{\mathbf{X}}$ and the control error vector $E_{\mathbf{U}}$ as

$$\begin{array}{cc}\hfill E_{\mathbf{X}}& ={[{\mathbf{e}}_{\mathbf{X}}^{\top }(k+1),\dots ,{\mathbf{e}}_{\mathbf{X}}^{\top }(k+N)]}^{\top },\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill E_{\mathbf{U}}& ={[{\mathbf{e}}_{\mathbf{U}}^{\top }\left(k\right),\dots ,{\mathbf{e}}_{\mathbf{U}}^{\top }(k+N-1)]}^{\top },\hfill \end{array}$$

(24)

where N is the predicted time horizon. Then, the optimization problem can be formulated as

$$\begin{array}{c}\hfill \begin{array}{cc}\min & E_{\mathbf{X}}^{\top }{W}_X{E}_{\mathbf{X}}+E_{\mathbf{U}}^{\top }{W}_U{E}_{\mathbf{U}},\end{array}\end{array}$$

(25)

where $W_X$ and $W_U$ are the diagonal matrices representing the weight of the state and control errors respectively. The constraints on the control variables are described as

$$\begin{array}{c}\hfill \begin{array}{cc}& -{\tilde{a}}_{\max }\le u_1(k+i)\le {\tilde{a}}_{\max },i=0,1,\dots ,N-1,\\ & -{\delta }_{\max }\le u_2(k+i)\le {\delta }_{\max },i=0,1,\dots ,N-1,\end{array}\end{array}$$

(26)

where the max acceleration ${\tilde{a}}_{\max }$ and the max steering angle ${\delta }_{\max }$ are the upper bound of the control variables, which are usually designed to ensure driving comfort and tracking performance. This optimization problem is a quadratic programming (QP) problem that can be quickly solved by a nonlinear programming solver, e.g., CVXOPT. The quadratic program solver is denoted as ‘QP.solver’ at line 8 (see Algorithm 1) to solve the optimization problem defined in Equation (25). The solution to this optimization problem is ${[{\mathbf{e}}_{\mathbf{U}}^{\top }\left(k\right),...,{\mathbf{e}}_{\mathbf{U}}^{\top }(k+N-1)]}^{\top }$, and we take the first one ${\mathbf{e}}_{\mathbf{U}}^{\top }\left(k\right)$ as the optimal control.

Algorithm 1 MPC and States Prediction

Input:  $s_t$, $a_t$, $a_t^{\odot }$, QP.solver. MPC is used for state prediction (MPC.predict) and control execution (MPC.execute).   1: if MPC.predict then   2:     $v_r=g\left(a_t\right)$ by (28)                                                     ▹ execute the raw action   3: else if MPC.execute then   4:     $v_r=g\left(a_t^{\odot }\right)$ by (28)                                                    ▹ execute the replaced action   5: end if   6: Get $s_r$ by (27)                                                         ▹ reference positions   7: $(x_r,y_r)\leftarrow (s_r,l_r)$ by coordinate transform   8: ${\mathit{U}}^{*}\leftarrow \mathrm{QP}.\mathrm{solver}(x_r,y_r,v_r)$                                                ▹ solve the optimal control   9: Get ${\mathit{e}}_{\mathit{X}}$ by (20)                                                              ▹state errors 10: Get $x_e^{\sigma }$ by (29)                                                            ▹ predicted states 11: if MPC.predict then 12:     return $x_e^{\sigma }$ 13: else if MPC.execute then 14:     return ${\mathit{U}}^{*}$ 15: end if Output:  $x_e^{\sigma }$ or ${\mathit{U}}^{*}$                                                  ▹ predicted states or optimal control

4.2. States Computation

In this study, we compute the state error ${\mathbf{e}}_{\mathbf{X}}\left(k\right)$ at each time k based on the predicted positions for the entire time horizon, as shown in Algorithm 1. We first need to compute reference positions in the Frenet frame to acquire the predicted positions and then convert them back to the Cartesian coordinates. The reference positions are the waypoints along the reference line in the Frenet frame [32]. In the first step, the reference position $s_{\mathrm{r}}$ in the Frenet frame is derived by $x_{\mathrm{r}}$, which is the orthogonal projection point of x relative to the reference path. In the next step, the relationship between $x_r$ and $s_r$ is already known. For example, if the reference path is a long straight road, then the relationship can be expressed as $x_r=s_r$. It is assumed that the reference line is the center line of the target lane. The target lane is located on the left lane if the decision is to turn left, and vice versa. Therefore, the reference positions in the Frenet frame can be computed as

$$s_r\left(k\right)=s_r(k-1)+v_r\Delta t,$$

(27)

where $s_{\mathrm{r}}\left(k\right)$ is the reference position at time k, and $v_r$ is the reference speed that is determined by the decision $a_t$, which is updated as

$$v_r=\left\{\begin{array}{cc}{v}_t+\Delta v\hfill & \mathrm{if}{a}_t=A^{\uparrow }\hfill \\ v_t-\Delta v\hfill & \mathrm{if}{a}_t=A^{\downarrow }\hfill \\ v_t\hfill & \mathrm{else}\hfill \end{array}\right.,$$

(28)

where $v_t$ is the current speed of the ego vehicle, $a_t$ is the high-level decision determined by the RL agent, and the action space A is defined as $A=[A^{◃},A^{▹},A^{\uparrow },A^{\oslash },A^{\downarrow }]$, which are turning left, turning right, acceleration, idling and deceleration, respectively. In particular, we execute the RL decision in MPC for motion prediction of the ego vehicle, and the predicted position of the ego vehicle $x_e^{\sigma }$ is written as

$$x_e^{\sigma }=x_r\left(\sigma \right)+e_X^x\left(\sigma \right),$$

(29)

where $\sigma \in [1,N]$ is the prediction coefficient, $x_r\left(\sigma \right)$ is the reference longitudinal position in the Cartesian coordinates, and $e_X^x\left(\sigma \right)$ is the longitudinal position error computed by Equation (20).

5. Safe Reinforcement Learning

This section introduces the safe RL algorithm, for which we use a discrete version of the SAC algorithm to solve the CMDP problem. Also, the ASM is built to check the safety of the RL action and replace it when necessary based on the predicted states provided by the MPC module.

5.1. Lagrangian-Based Discrete SAC

Safe RL is employed for the high-level behavioral decision in this study, i.e., to determine the optimal driving decision from a discrete action space A. A discrete action version of the SAC algorithm, i.e., SAC-Discrete (SACD) [33], is used to solve the CMDP problem.

5.1.1. Critic Network and Policy Network

Let ${\mathring{a}}^1,{\mathring{a}}^2,\dots ,{\mathring{a}}^{\left|A\right|}$ be the discrete actions in the action space, i.e., $A={\left\{{\mathring{a}}^h\right\}}_{h=1}^{\left|A\right|}$, $\left|A\right|$ is the size of the action space. A critic network and policy network are used in this study, as shown in Figure 7. The soft Q-function of the discrete SAC outputs a vector of size $\left|A\right|$ that consists of the Q-value of each action, i.e., $q_{\omega }:S\to {\mathbb{R}}^{\left|A\right|}$:

$$q_{\omega }\left(s_t\right)={[Q_{\omega }(s_t,{\mathring{a}}^1),Q_{\omega }(s_t,{\mathring{a}}^2),\dots ,Q_{\omega }(s_t,{\mathring{a}}^{\left|A\right|})]}^T,$$

(30)

where $q_{\omega }\left(s_t\right)$ is the soft Q-function parameterized by $\omega$, and $Q_{\omega }(s_t,{\mathring{a}}^h)$ is the state-action value at state $s_t$ with the discrete action ${\mathring{a}}^h(h=1,2,\dots ,|A\left|\right)$. Likewise, the policy can directly output the action distribution, which outputs a vector that consists of the probability of each action at state $s_t$, i.e., ${\pi }_{\theta }:S\to {[0,1]}^{\left|A\right|}$:

$${\pi }_{\theta }\left(s_t\right)={[{\pi }_{\theta }\left({\mathring{a}}^1\right|s_t),{\pi }_{\theta }\left({\mathring{a}}^2\right|s_t),\dots ,{\pi }_{\theta }\left({\mathring{a}}^{\left|A\right|}\right|s_t)]}^T,$$

(31)

where $\theta$ denotes the policy network parameters (see Figure 7), ${\pi }_{\theta }\left({\mathring{a}}^h\right|s_t)$ is the probability of the action ${\mathring{a}}^h(h=1,2,\dots ,|A\left|\right)$ conditioned on the state $s_t$. In the discrete action settings, the soft state-value function is computed as

$$V\left(s_t\right)={\pi }_{\theta }{\left(s_t\right)}^T\left[q_{\omega }\left(s_t\right)-\xi \log {\pi }_{\theta }\left(s_t\right)\right].$$

(32)

where $\xi$ is the temperature parameter that determines the relative importance of the entropy term versus the reward term. According to [33], the policy network parameters $\theta$ can be learned by minimizing the following loss function:

$$J_{\pi }\left(\theta \right)=\underset{s_t\sim \mathcal{D}}{\mathbb{E}}\left[{\pi }_{\theta }{\left(s_t\right)}^{\top }\left(\xi \log {\pi }_{\theta }\left(s_t\right)-q_{\omega }\left(s_t\right)\right)\right].$$

(33)

The loss function of the temperature parameter is defined as

$$G\left(\xi \right)={\pi }_{\theta }{\left(s_t\right)}^{\top }\left[-\xi \left(\log {\pi }_{\theta }\left(s_t\right)+\overline{\mathcal{H}}\right)\right],$$

(34)

where $\overline{\mathcal{H}}$ is a hyperparameter that represents the target entropy. As shown in Figure 7, the critic network parameter $\omega$ of the soft Q-function can be learned by minimizing the soft Bellman residual:

$$\begin{array}{c}\hfill J_q\left(\omega \right)=\underset{s_t,a_t\sim \mathcal{D}}{\mathbb{E}}\left[{\displaystyle \frac{1}{2}}{\left(Q_{\omega }(s_t,a_t)-\overline{Q}(s_t,a_t)\right)}^2\right],\end{array}$$

(35)

$$\begin{array}{c}\hfill \overline{Q}(s_t,a_t)=r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}}\left[V_{\overline{\omega }}\left(s_{t+1}\right)\right],\end{array}$$

(36)

where $Q_{\omega }(s_t,a_t)$ is the estimated Q function, and $\overline{Q}(s_t,a_t)$ is the target soft Q function. In particular, $V_{\overline{\omega }}\left(s_{t+1}\right)$ is estimated using a target Q network:

$$V_{\overline{\omega }}\left(s_{t+1}\right)={\pi }_{\theta }{\left(s_{t+1}\right)}^{\top }\left[q_{\overline{\omega }}\left(s_{t+1}\right)-\xi \log {\pi }_{\theta }\left(s_{t+1}\right)\right],$$

(37)

where $\overline{\omega }$ is the parameter of the target Q network, which is updated in Line 18 in Algorithm 2.

Algorithm 2 Human-aligned safe RL

Input:  MPC, ASM, $\sigma$, N Input:  initialize $\theta ,{\omega }_{1,2},{\omega }_c,\overline{\omega },{\overline{\omega }}_c,\lambda ,\xi ,\Gamma$ and $\mathcal{D}\leftarrow \varnothing$   1: for each iteration do   2:       for each environment step do   3:             $a_t\sim {\pi }_{\theta }(·\mid s_t)$   4:             $x_e^{\sigma }\leftarrow \mathrm{MPC}.\mathrm{predict}(s_t,a_t)$ by Algorithm 1   5:             $a_t^{\odot }\leftarrow \mathrm{ASM}(a_t,x_e^{\sigma },\sigma ,N)$ by Algorithm 3   6:             $s_{t+1}\leftarrow \mathrm{MPC}.\mathrm{execute}(s_t,a_t^{\odot })$ by Algorithm 1   7:             $r_t\leftarrow r(s_t,a_t^{\odot })$   8:             $c_t\leftarrow c(s_t,a_t^{\odot })$   9:             $\mathcal{D}\leftarrow \mathcal{D}\cup \left\{(s_t,a_t^{\odot },r_t,c_t,s_{t+1})\right\}$ 10:       end for 11:       for each gradient step do 12:             $\lambda \leftarrow \lambda -{\alpha }_{\lambda }{\nabla }_{\lambda }J\left(\lambda \right)$ 13:             ${\omega }_i\leftarrow {\omega }_i-{\alpha }_{\omega }{\nabla }_{{\omega }_i}{J}_q\left({\omega }_i\right)$ for $i=1,2$ 14:             ${\omega }_c\leftarrow {\omega }_c-{\alpha }_{{\omega }_c}{\nabla }_{{\omega }_c}{J}_{q_c}\left({\omega }_c\right)$ 15:             $\theta \leftarrow \theta -{\alpha }_{\theta }{\nabla }_{\theta }{J}_{\pi }^{\lambda }\left(\theta \right)$ 16:             $\xi \leftarrow \xi -{\alpha }_{\xi }{\nabla }_{\xi }G\left(\xi \right)$ 17:             if soft update then 18:                  $\overline{\omega }\leftarrow \Gamma \omega +(1-\Gamma )\overline{\omega }$ 19:                  $\overline{{\omega }_c}\leftarrow \Gamma {\omega }_c+(1-\Gamma )\overline{{\omega }_c}$ 20:             end if 21:       end for 22: end for Output:  Optimal policy ${\pi }^{*}$

Algorithm 3 Action Shielding Mechanism

Input:  $a_t$, $x_e^{\sigma }$, $x_{\mathrm{obj}}$, $\sigma$, N   1: if $a_t$ is $A^{◃}$ then   2:      for each vehicle i do   3:            $x_i\left(N\right)=x_i\left(0\right)+N·v_i\Delta t$   4:            if $x_i\left(0\right)\le x_e^{\sigma }\le x_i\left(N\right)$ and $|y_e^{\sigma }(A^{◃})-y_i|\le d_y$ then   5:                $a_t^{\odot }=A^{\downarrow }$                                   ▹ Situation 1   6:            end if   7:      end for   8: end if   9: if $a_t$ is $A^{▹}$ then 10:     if ego vehicle has merged then 11:            $a_t^{\odot }=A^{\oslash }$                                    ▹ Situation 2 12:      end if 13: end if 14: if $a_t$ is $A^{\uparrow }$ or $A^{\oslash }$ then 15:      if $|x_e^{\sigma }\left(a\right)-x_{\mathrm{obj}}|\le d_x$ then 16:            $a_t^{\odot }=A^{\downarrow }$                                     ▹ Situation 3 17:      end if 18: end if Output:  Safe decision $a_t^{\odot }$

5.1.2. Cost Network

A cost network parameterized by ${\omega }_c$ is built to approximate the cost value function $q_{{\omega }_c}\left(s_t\right)$, as shown in Figure 7. The cost value function $q_{{\omega }_c}\left(s_t\right)$ is defined as

$$q_{{\omega }_c}\left(s_t\right)={[Q_{{\omega }_c}(s_t,{\mathring{a}}^1),Q_{{\omega }_c}(s_t,{\mathring{a}}^2),\dots ,Q_{{\omega }_c}(s_t,{\mathring{a}}^{\left|A\right|})]}^T,$$

(38)

where $Q_{{\omega }_c}(s_t,{\mathring{a}}^h)$ is the soft cost value at state $s_t$ with the discrete action ${\mathring{a}}^h(h=1,2,\dots ,|A\left|\right)$. Similarly, the loss function of the cost network is defined as

$$\begin{array}{c}\hfill J_{q_c}\left({\omega }_c\right)=\underset{s_t,a_t\sim \mathcal{D}}{\mathbb{E}}\left[{\displaystyle \frac{1}{2}}{\left(Q_{{\omega }_c}(s_t,a_t)-{\overline{Q}}_c(s_t,a_t)\right)}^2\right],\end{array}$$

(39)

where $Q_{{\omega }_c}(s_t,a_t)$ is the cost value at state $s_t$ and action $a_t$ and ${\overline{Q}}_c(s_t,a_t)$ is the target cost value:

$$\begin{array}{cc}\hfill {\overline{Q}}_c(s_t,a_t)& =c(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}}\left[V_{{\overline{\omega }}_c}\left(s_{t+1}\right)\right],\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill V_{{\overline{\omega }}_c}\left(s_{t+1}\right)& ={\pi }_{\theta }{\left(s_{t+1}\right)}^{\top }{q}_{{\overline{\omega }}_c}\left(s_{t+1}\right),\hfill \end{array}$$

(41)

where the target cost function $q_{{\overline{\omega }}_c}$ is approximated by the target cost network with parameter ${\overline{\omega }}_c$, which is updated in Line 19 in Algorithm 2.

5.1.3. n-Step TD Learning

In traditional temporal difference (TD) learning, the agent updates the value estimates based on the current estimate and a target value. n-step TD learning generalizes this idea by considering not just the immediate next state but by looking ahead n steps into the future. This means that the update is based on the sum of rewards over the next n steps. In this study, we use n-step TD learning, which is expected to strike a balance between the short-term and the longer-term perspective of Monte Carlo methods. By adjusting the parameter n, the trade-off between bias and variance can be adjusted in the learning process. We also use n-step transitions to approximate the target Q value and cost value functions. According to [34], the target Q-value in Equation (36) and target cost value in Equation (40) are rewritten as

$$\begin{array}{c}\hfill \overline{Q}(s_t,a_t)={\displaystyle \sum _{j=0}^{\mathrm{n}-1}}{\gamma }^{j}r(s_{t+j},a_{t+j})+{\gamma }^n{\mathbb{E}}_{s_{t+1}}\left[V_{\overline{\omega }}\left(s_{t+n}\right)\right],\end{array}$$

(42)

$$\begin{array}{c}\hfill {\overline{Q}}_c(s_t,a_t)={\displaystyle \sum _{j=0}^{\mathrm{n}-1}}{\gamma }^{j}c(s_{t+j},a_{t+j})+{\gamma }^n{\mathbb{E}}_{s_{t+1}}\left[V_{\overline{{\omega }_c}}\left(s_{t+n}\right)\right],\end{array}$$

(43)

where n is the parameter that determines the number of steps that we want to look ahead before updating the Q-function.

5.1.4. Lagrange Multiplier

We have a cost network, the loss function of the policy network is modified by adding the cost item via the Lagrange multiplier:

$$J_{\pi }^{\lambda }\left(\theta \right)=J_{\pi }\left(\theta \right)+\underset{s_t\sim \mathcal{D}}{\mathbb{E}}\left[\lambda {\pi }_{\theta }{\left(s_t\right)}^T{q}_{{\omega }_c}\left(s_t\right)\right],$$

(44)

where $\lambda$ is the Lagrange multiplier:

$${\nabla }_{\theta }{J}_{\pi }^{\lambda }\left(\theta \right)={\displaystyle \frac{\partial J_{\pi }\left(\theta \right)}{\partial \theta }}+\lambda {{\displaystyle \frac{\partial {\pi }_{\theta }\left(s_t\right)}{\partial \theta }}}^T{q}_{{\omega }_c}\left(s_t\right).$$

(45)

We use the state-action value to approximate the $J_{\pi }^C$,

$$J_{\pi }^C=Q_{{\omega }_c}(s_t,a_t)$$

(46)

The loss function of the Lagrange multiplier is denoted as

$$J\left(\lambda \right)=\underset{s_t\sim \mathcal{D}}{\mathbb{E}}\left[-\lambda (Q_{{\omega }_c}(s_t,a_t)-\eta )\right],$$

(47)

where $J\left(\lambda \right)$ is the loss for Lagrange multiplier $\lambda$.

Overall, the proposed human-aligned safe RL is described in Algorithm 2. The raw action $a_t$ sampled from the policy ${\pi }_{\theta }(·|s_t)$ (Line 3) would first be sent to MPC for motion prediction (MPC.predict) (Line 4) and the ASM for safety check (Line 5). The replaced action $a_t^{\odot }$ provided by the ASM is then executed in the low-level MPC controller (MPC.execute, Line 6) and then gets the corresponding cost and reward training (Line 7–8). We use two critic networks parameterized by ${\omega }_i(i=1,2)$ to avoid the overestimation of the soft Q-value [35]. In addition, the target soft Q value and cost value are updated based on the parameters $\overline{\omega }$ and $\overline{{\omega }_c}$.

5.2. Action Shielding Mechanism

The frequent constraint violation behavior usually makes the RL agent hard to learn. In this study, to enhance the sample efficiency for safe RL, we design a rule-based ASM to replace unexpected or unsafe RL actions with safe ones (see Algorithm 3), which can help guide the RL agent to take the right action during the exploration process. In the ASM, the unsafe RL action is detected by checking whether there exist collisions between the ego vehicle and surrounding vehicles based on the predicted states.

We define the unsafe decision set as ${\Omega }_{\mathrm{unsafe}}={\Omega }_1\cup {\Omega }_2\cup {\Omega }_3$, including three typical situations, as shown in Figure 8. If the original RL decisions $a_t\in {\Omega }_{\mathrm{unsafe}}$, the decisions $a_t$ will be substituted with a safe one $a_t^{\odot }$.

5.2.1. Situation 1

A collision is likely to occur when the action turning left $A^{◃}$ is executed (see top Figure 8). The unsafe set ${\Omega }_1$ of this situation is defined as

$${\Omega }_1=\{a\in A|,x_i\left(0\right)\le x_e^{\sigma }(A^{◃})\le x_i\left(N\right),|y_e^{\sigma }(A^{◃})-y_i| \le d_y\},$$

(48)

where $x_e^{\sigma }$ and $y_e^{\sigma }$ are the predicted position of ego vehicle based on the action $A^{◃}$, $\sigma$ is the prediction coefficient, $x_i$ and $y_i$ are the predicted positions of the $i^{\mathrm{th}}$ vehicle, and $d_y$ is the safe distance between the ego vehicle and the $i^{\mathrm{th}}$ vehicle in the lateral direction. In this situation, the action $A^{◃}$ is replaced with the deceleration action $A^{\downarrow }$ to delay the merging maneuver and avoid the predicted lateral conflict. The replaced action $a_t^{\odot }$ is written as

$$a_t^{\odot }=A^{\downarrow }.$$

(49)

5.2.2. Situation 2

The action is labeled as an unexpected decision when the ego vehicle has already merged into the target lane, but the RL agent outputs an action turning right $A^{▹}$, which is considered to be unnecessary (middle in Figure 8). The unsafe set ${\Omega }_2$ of this situation is defined as

$${\Omega }_2=\{a\in A|a=A^{▹},I_{\mathrm{merge}}=1\},$$

(50)

where $I_{\mathrm{merge}}=1$ indicates that the ego vehicle has successfully merged. Therefore, an action idling $A^{\oslash }$ is used to replace the unexpected action $A^{▹}$. The replaced action $a_t^{\odot }$ is written as

$$a_t^{\odot }=A^{\oslash }.$$

(51)

5.2.3. Situation 3

The ego vehicle and its neighbor’s motion have few differences in the longitudinal direction. The ego vehicle would fail to merge before reaching the end of the merge zone if acceleration $A^{\uparrow }$ or idling $A^{\oslash }$ are executed (see bottom of Figure 8). The unsafe set ${\Omega }_3$ of this situation is defined as

$${\Omega }_3=\{a\in A|a=A^{\uparrow } or A^{\oslash },|x_e^{\sigma }\left(a\right)-x_{\mathrm{obj}}| \le d_x\},$$

(52)

where $x_e^{\sigma }\left(a\right)$ is the predicted longitudinal position of the ego vehicle when executing the action a, i.e., $a=A^{\uparrow } or A^{\oslash }$. $x_{\mathrm{obj}}$ is the longitudinal position of the adjacent vehicle that occupies the target lane. $d_x$ is the distance threshold. The condition $|x_e^{\sigma }\left(a\right)-x_{\mathrm{obj}}| \le d_x$ indicates that the ego vehicle and the adjacent vehicle have few differences in the longitudinal direction, making it hard for the ego vehicle to merge. Similarly, the action $A^{\uparrow }$ or $A^{\oslash }$ would be replaced by the deceleration $A^{\downarrow }$, which slows down the ego vehicle to find another opportunity to merge, i.e., $a_t^{\odot }=A^{\downarrow }$.

Overall, the proposed ASM is designed to handle the dominant edge cases in the considered ramp-merging scenario, including predicted lateral collision during lane changing, unexpected right-turn decisions after successful merging, and failure-to-merge risk caused by target lane occupancy. These cases correspond to the main unsafe or infeasible high-level decisions observed in the current simulation environment. For the first and third cases, the deceleration action is used as a shielding fallback to delay the merging maneuver or enlarge the longitudinal gap, allowing the ego vehicle to seek another merging opportunity. It should be noted that this fallback action is implemented through the MPC execution layer and is subject to the acceleration bounds in the vehicle-control optimization. Therefore, it is not equivalent to an emergency braking maneuver in the present implementation.

Nevertheless, deceleration is not universally safe under all traffic configurations. For example, when the following vehicle is very close to the ego vehicle, has delayed response, or does not follow IDM-like longitudinal behavior, a deceleration fallback may increase rear-end collision risk. In the current simulation, this risk is partially mitigated by the bounded MPC execution and the IDM-based response of surrounding vehicles, but rear-side safety is not explicitly modeled as an independent shielding constraint. Therefore, the ASM should be interpreted as a scenario-specific safety augmentation module for reducing risky exploration in the considered ramp-merging setting, rather than a complete safety supervisor for all possible traffic configurations. A more general shielding design can further incorporate rear-side safety margins and select fallback actions from multiple candidate maneuvers.

6. Theoretical Analysis

This section presents a theoretical analysis of the ASM regarding the safety performance of the learned RL policy and the convergence performance of the proposed safe RL algorithm.

6.1. Safety Performance

Theorem 1. 

If the current action $a_t$ is unsafe, i.e., $a_t\in {\Omega }_{\mathrm{unsafe}}$, and when it is replaced with a safe one $a_t^{\odot }$ using ASM, then the state action cost function $Q_c(s_t,a_t^{\odot })$ is lower than the original state action cost function $Q_c(s_t,a_t)$, which indicates improved safety effectiveness contributed by the ASM.

Proof. 

We consider the Bellman equation [36], which is denoted as

$$Q_c(s_t,a_t)=c(s_t,a_t)+\gamma V_c\left(s_{t+1}\right).$$

(53)

For simplicity, let $c_t=c(s_t,a_t)$ and $c_t^{\odot }=c(s_t,a_t^{\odot })$. Then we have

$$Q_c(s_t,a_t)-Q_c(s_t,a_t^{\odot })=c_t-c_t^{\odot }+V_c\left(s_{t+1}\right)-V_c\left(s_{t+1}^{\odot }\right).$$

(54)

Considering the first and second items in the Bellman equation, the original decision $a_t$ will lead to a collision in the next steps as the result of $c_t>c_t^{\odot }$. As for the difference in state cost value, we assume that the following actions will also be replaced by a safe decision if the original is unsafe. Then, we have

$$V_c\left(s_{t+1}\right)-V_c\left(s_{t+1}^{\odot }\right)=E_{\pi }\left[{\displaystyle \sum _{t=0}^T}{\gamma }^t(c(s_{t+1},a_{t+1})-c(s_{t+1}^{\odot },a_{t+1}^{\odot }))\right].$$

(55)

During the rest steps in the episode, the original decision will be replaced by the safe one, and will result in a safe state. Therefore, we have

$$c(s_{t+1},a_{t+1})>c(s_{t+1}^{\odot },a_{t+1}^{\odot }),t\in [0,T].$$

(56)

As a consequence, we have

$$V_c\left(s_{t+1}\right)>V_c\left(s_{t+1}^{\odot }\right),t\in [0,T],$$

(57)

$$Q_c(s_t,a_t)>Q_c(s_t,a_t^{\odot }),t\in [0,T].$$

(58)

□

The performance of the policy with the ASM could be promoted because the safe action can reduce the state-action cost value at each step and guide the agent to choose the safe decision. Furthermore, another benefit of the proposed method is that the training process is not affected by the ASM.

6.2. Convergence Analysis

The optimization objective is to maximize the sum of the discounted reward while assuring the cost value satisfies the constraints. From Equation (10), the Lagrangian dual function [37] is formulated as

$$d\left(\lambda \right)=\underset{\pi }{\min }-J_R^{\pi }+\lambda (J_C^{\pi }-\eta ).$$

(59)

The dual ascent is to find ${\max }_{\lambda \ge 0}d\left(\lambda \right)$. During the learning process, the policy update is imperfect due to two factors. One factor is that the number of iterations is limited for computation efficiency and another factor is that the replaced decision can guarantee safety but not optimality. Hence, we assume the suboptimality of the solution ${\pi }^{*}$ is upper bounded as

$$-J_R^{{\pi }^{*}}+\lambda (J_C^{{\pi }^{*}}-\eta )-d\left(\lambda \right)<ϵ.$$

(60)

We denoted the residual of $\lambda$ before and after the imperfect update in a single step of Algorithm 2 as

$$\widehat{g}(\lambda ,{\alpha }_{\lambda })={\displaystyle \frac{1}{{\alpha }_{\lambda }}}(\max (0,\lambda +{\alpha }_{\lambda }{\nabla }_{\lambda }d\left(\lambda \right))-\lambda ).$$

(61)

Lemma 1. 

Following the imperfect dual ascent with step size ${\alpha }_{\lambda }\le \mu$, we have

$$d\left({\lambda }_{k+1}\right)\ge d\left({\lambda }_k\right)+{\displaystyle \frac{{\alpha }_{\lambda }}{2}}\Vert \widehat{g}({\lambda }_k,{\alpha }_{\lambda }){\Vert }^2-\sqrt{{\displaystyle \frac{2ϵ}{\mu }}}\Vert \widehat{g}({\lambda }_k,{\alpha }_{\lambda })\Vert .$$

(62)

Proof. 

The result follows from standard projected dual ascent for $\mu$-smooth concave dual functions (cf. Theorem 2.2.7 in [38]). In our setting, the policy update at each iteration is only $ϵ$-suboptimal, which introduces a deviation term of order $\sqrt{ϵ/\mu }$; that is, the deviation can be bounded by a constant times $\sqrt{ϵ/\mu }$. Intuitively, one step of the projected ascent guarantees an increase in $d\left(\lambda \right)$ proportional to $\parallel \widehat{g}{\parallel }^2$, minus a correction term of size at most on the order of $\sqrt{ϵ/\mu }$, which accounts for the inexact policy improvement. □

Taking ${\lambda }^{\prime }={\lambda }_k$ and utilizing the fact that ${\lambda }_{k+1}-{\lambda }_k={\alpha }_{\lambda }\widehat{g}$ gives the result.

Theorem 2. 

There exist constant $\chi >0$ such that the imperfect update converges to a dual solution $\widehat{\lambda }$ that satisfies

$$\underset{{\lambda }^{*}\in {\mathcal{P}}^{*}}{\mathit{min}}\Vert {\lambda }^{*}-\widehat{\lambda }\Vert \le \chi \sqrt{{\displaystyle \frac{ϵ}{\mu }}}.$$

(63)

Proof. 

Let $\varphi \left(\lambda \right)={\min }_{{\lambda }^{*}\in {\mathcal{P}}^{*}}\Vert \lambda -{\lambda }^{*}\Vert$. Based on Theorem 4.1 of [39], there exists a constant $\psi$ such that

$$\varphi \left({\lambda }_k\right)+\Vert {\pi }^{*}-\pi \Vert \le \psi \Vert {\lambda }_{k+1}-{\lambda }_k\Vert .$$

(64)

From Lemma 1, when $\Vert \widehat{g}\Vert >{\displaystyle \frac{2}{{\alpha }_{\lambda }}}\sqrt{{\displaystyle \frac{2ϵ}{\mu }}}$, the Lagrangian dual function $d\left(\lambda \right)$ monotonically increases and the imperfect dual ascent would reach a $\widehat{\lambda }$ satisfying $\Vert \widehat{g}(\widehat{\lambda },{\alpha }_{\lambda })\Vert <{\displaystyle \frac{2}{{\alpha }_{\lambda }}}\sqrt{{\displaystyle \frac{2ϵ}{\mu }}}$. Then, it follows that $\psi \Vert {\lambda }_{k+1}-{\lambda }_k\Vert <\psi (2+{\alpha }_{\lambda })\sqrt{{\displaystyle \frac{2ϵ}{\mu }}}$. Taking $\psi ={\displaystyle \frac{\chi }{\sqrt{2}(2+{\alpha }_{\lambda })}}$, we have $\varphi \left(\widehat{\lambda }\right)\le \chi \sqrt{{\displaystyle \frac{ϵ}{\mu }}}$. □

Theorem 2 shows that the proposed method will converge to a near-optimal solution, even under imperfect policy updates.

7. Experimental Setup

This section introduces the implementation details of the experiments, including the simulation environment, scenario setting, model formulation, and implementation details, etc.

7.1. Scenario Settings

We built an on-ramp merging scenario based on the simulation platform highway-env [40]. As shown in Figure 1, the merge zone is comprised of a lane width of 5 m and a length of 70 m. The ego vehicle starts from the entrance of the ramp, 80 m from the merge zone. The longitudinal acceleration of the surrounding vehicles is predicted by the intelligent driver model (IDM) [41]. The longitudinal distance between the front vehicle i and the rear vehicle $i+1$ is denoted as $d_{i,i+1}$, which is defined as $d_{i,i+1}=d_s+v_{i+1}/\rho$, where $d_s$ is the safety distance. $v_{i+1}$ is the speed of the rear vehicle. The initial speed of all vehicles ranges from 17 m/s to 27 m/s. $\rho \in [0.5,1]$ represents the space density of traffic flow. The traffic density is in low level when $\rho \in [0.5,0.7)$, medium level when $\rho \in [0.5,1]$, and high level when $\rho \in (0.8,1]$. A larger value of $\rho$ yields smaller headways, which increases the number of surrounding vehicles within the ego vehicle’s interaction range during the on-ramp merge and results in tighter available gaps. The surrounding vehicles are controlled by IDM-based longitudinal behavior models, which provide a controlled and reproducible traffic environment for evaluating the proposed method. This setting is used as a benchmark scenario and does not aim to fully reproduce all heterogeneous driving behaviors observed in real traffic.

7.2. Implementation Details

The parameter settings of the proposed approach, including MPC and safe RL, are summarized as follows, and the main hyperparameters with explicit symbols are further listed in

Table 2. Hyperparameters of the safe RL algorithm.

Symbol	Description	Value
$\gamma$	Discount factor	0.99
${\alpha }_{\theta }$	Policy network learning rate	$1\times {10}^{-4}$
${\alpha }_{\omega }$	Critic network learning rate	$1\times {10}^{-4}$
${\alpha }_{{\omega }_c}$	Cost network learning rate	$1\times {10}^{-4}$
${\alpha }_{\xi }$	Temperature parameter learning rate	$1\times {10}^{-4}$
${\lambda }_0$	Initial Lagrangian multiplier	1.0
${\alpha }_{\lambda }$	Lagrangian multiplier learning rate	$1\times {10}^{-4}$
$\mathcal{D}$	Replay buffer size	$1\times {10}^5$
$\mathcal{B}$	Batch size	256

. The MPC prediction horizon is set to 10 time steps. To ensure smooth merging, the reference steering angle is set to 0 rad with a maximum magnitude of $\pi /8$ rad, and the acceleration is constrained within [−0.5 g, 0.5 g]. The prediction coefficient $\sigma$ is set to 5, meaning that collision checking in the ASM is performed five steps ahead. For safe RL, the environment simulation frequency is 10 Hz, while the decision-making frequency is 2 Hz. Each episode terminates when the ego vehicle reaches the goal or a collision occurs, and the agent is trained for 0.5 million interaction steps. During training, the agent is coupled with MPC online: the agent outputs actions for motion prediction, which are overridden by the ASM if potential collisions are detected, and the resulting shielded actions are executed by the low-level MPC controller to obtain reward and cost signals. All neural networks in safe RL consist of two hidden layers with 256 units each and use ReLU activations. The optimizer is Adam, the target entropy ratio is set to 0.98, the policy update frequency is 1, and the target network update frequency is 10. Other symbolic hyperparameters, including the learning rates, replay buffer size, batch size, discount factor, and Lagrangian multiplier settings, are reported in Table 2. Moreover, we train the RL agent with five random seeds. All the experiments are conducted on an Intel Core i5-11300H CPU with 16 GB RAM that runs at 3.1 GHz.

8. Results and Discussion

8.1. Convergence Performance

To demonstrate the advantages of RAPRL (ours), we compare the training curves of RAPRL with three baselines: Dueling DQN [42], SACD [33], and PPO [43]. Figure 9 presents the comparative results across three key metrics: Crash Ratio, Average Cost, and Average Reward, plotted against the total environmental interactions. The shaded regions represent the standard deviation across different random seeds.

As illustrated in the Average Reward curve (Figure 9 (right)), RAPRL outperforms all baselines in terms of the final converged value. While Dueling DQN exhibits the fastest initial learning speed, it converges to a sub-optimal policy with a lower final reward compared to RAPRL. In contrast, RAPRL maintains a steady growth rate and ultimately achieves the highest asymptotic performance. PPO shows the slowest convergence rate and the lowest final reward. In Figure 9 (left), both Dueling DQN and RAPRL demonstrate rapid reductions in collision ratios, quickly converging to a lower crash ratio. This indicates that both methods can effectively learn to avoid fatal states at an early stage. However, PPO requires significantly more interactions to reduce the crash risk to an acceptable level. Crucially, the Average Cost curve (Figure 9 (middle)) reveals a significant advantage of our method. Although Dueling DQN achieves a low crash ratio, its average cost exhibits a sharp increase after the initial drop and stabilizes at a high level. Conversely, RAPRL consistently reduces the average cost and converges to the lowest level among all methods.

These results demonstrate that RAPRL not only ensures collision avoidance but also effectively minimizes comprehensive penalties, achieving a superior balance between safety constraints and driving efficiency.

8.2. Performance Evaluation

8.2.1. Traffic Success

Traffic success is primarily evaluated by the success rate, which quantifies the probability of the ego vehicle successfully completing the merging task without terminal failures. As presented in

Table 3. Comparison between the constraint-free RL methods and ours.

Method	Success Rate (%) ↑			Collision Ratio ↓			Average Cost ↓			Average Time (s) ↓
	High	Medium	Low	High	Medium	Low	High	Medium	Low	High	Medium	Low
Dueling DQN [42]	87.0	94.3	99.0	0.013	0.005	0.005	0.50	0.28	0.08	11.78	11.47	10.85
SACD [33]	94.5	97.5	99.2	0.010	0.008	0.005	0.44	0.25	0.10	11.59	11.33	10.82
PPO [43]	99.5	97.7	99.2	0.003	0.018	0.008	0.01	0.03	0.01	12.36	11.62	10.95
RAPRL (ours)	99.0	99.5	99.3	0.003	0.005	0.005	0.02	0.02	0.02	11.87	11.46	10.96

, the proposed RAPRL demonstrates exceptional robustness and superior performance across varying traffic densities. Specifically, RAPRL achieves success rates of 99.0%, 99.5%, and 99.3% in high, medium, and low traffic densities, respectively. In the challenging high-density scenario, where the baseline Dueling DQN experiences a significant performance drop to 87.0%, RAPRL maintains a high success rate, representing a substantial improvement of 12.0%. Similarly, compared to SACD, RAPRL improves the success rate by 4.5%, 2.0%, and 0.1% across the three density levels. Although PPO achieves a marginally higher success rate in the high-density scenario (99.5%), it suffers from performance degradation in the medium density scenario (97.7%). RAPRL consistently maintains a success rate above 99% in all tested simulation scenarios, indicating its stability and adaptability within the considered IDM-based ramp-merging environments.

8.2.2. Traffic Safety

Traffic safety is assessed using collision ratio and average cost, which reflect the frequency of accidents and the agent’s adherence to safety constraints. As shown in Table 3, the unconstrained methods, Dueling DQN and SACD, incur significantly higher average costs, particularly in high-density scenarios (0.50 and 0.44, respectively), indicating aggressive behaviors that frequently violate safety boundaries. In contrast, RAPRL dramatically reduces the average cost to 0.02 across all densities. Regarding collision ratio, while PPO demonstrates strong performance in high and low densities, it exhibits a spike in collision ratio to 0.018 under medium density, suggesting instability in complex interaction scenarios. RAPRL, however, consistently suppresses the collision ratio to a negligible level (0.003∼0.005) across all test cases. This validates that by explicitly optimizing the constrained objective, RAPRL effectively minimizes safety risks.

8.2.3. Traffic Efficiency

Traffic efficiency is evaluated by the average time required to complete the merging maneuver. As illustrated in Table 3, Dueling DQN and SACD achieve the shortest average times (e.g., 11.78 s and 11.59 s in high density) but at the cost of high accident risks and safety violations, as discussed in the safety analysis. Conversely, PPO exhibits the longest average time (12.36 s in high density), indicating that it adopts an overly conservative policy to ensure safety, thereby sacrificing efficiency. RAPRL achieves a well-balanced trade-off between these conflicting objectives. It records an average time of 11.87 s in high density, which is faster than PPO, yet it maintains the lowest safety cost. These results demonstrate that RAPRL enables the agent to complete efficient merging maneuvers without compromising safety standards or adopting overly cautious behaviors.

8.3. Ablation Study

This section conducts an ablation study to evaluate the impact of RAPRL’s essential components: the Action Shielding Mechanism (ASM), safety constraints (SC), and personal preference (PP). We benchmark the full RAPRL method against three ablated variants: (a) RAPRL w/o SC, (b) RAPRL w/o ASM and SC, and (c) RAPRL w/o ASM, SC, and PP. The quantitative performance metrics are summarized in

Table 4. Ablation study results.

PP	ASM	SC	Success Rate (%) ↑			Collision Ratio ↓			Average Cost ↓			Average Time (s) ↓
			High	Medium	Low	High	Medium	Low	High	Medium	Low	High	Medium	Low
-	-	-	94.5	97.5	99.2	0.010	0.008	0.005	0.44	0.25	0.10	11.59	11.33	10.82
-	-	🗸	97.2	97.3	99.3	0.003	0.005	0.005	0.23	0.13	0.04	12.08	11.54	10.99
-	🗸	🗸	98.3	98.8	98.9	0.010	0.008	0.003	0.02	0.02	0.02	12.33	11.58	11.02
🗸	🗸	🗸	99.0	99.5	99.3	0.003	0.005	0.005	0.02	0.02	0.02	11.87	11.64	10.96

, while the training curves for average reward and average cost are illustrated in Figure 10.

It is worth clarifying that SC and ASM play different roles in RAPRL. The CMDP-based safety constraint is a learning-level mechanism that penalizes the expected cumulative cost through the Lagrangian formulation and guides the policy toward lower-risk behavior during optimization, whereas ASM is an execution-level safety filter that evaluates the raw high-level action using MPC-based prediction and replaces it when the action is identified as unsafe or infeasible. During training, the shielded action $a_t^{\odot }$ is executed by the low-level MPC controller, and the resulting transition, reward, and cost are stored for policy updates. Thus, ASM also influences policy learning by reducing unsafe exploration and exposing the agent to corrected lower-risk behaviors. The two mechanisms are therefore complementary: SC encourages the policy itself to satisfy the expected cost constraint, while the ASM handles residual unsafe actions at the action execution level.

8.3.1. Safety Constraints

To verify the fundamental role of safety constraints in risk mitigation, we compare the baseline method (w/o PP, ASM, and SC) with the variant incorporating only SC (RAPRL w/o PP and ASM). As shown in Table 4, the introduction of SC significantly reduces the collision ratio across all traffic densities. Notably, in the high-density scenario, the collision ratio drops from 0.010 to 0.003, and the success rate improves from 94.5% to 97.2%. This indicates that SC encourages the policy of avoiding high-cost behaviors by penalizing safety violations during policy optimization, rather than directly filtering actions at execution time. However, the limitation of using SC alone is also evident: the average time increases from 11.59 s to 12.08 s, suggesting that the agent adopts a conservative policy to satisfy hard constraints, thereby sacrificing driving efficiency. Furthermore, as observed in the training curves (Figure 10 (middle)), although the method with SC (orange line) reduces crash occurrences, its average cost exhibits significant oscillations and remains at a relatively high level compared to the full method. This implies that while hard constraints prevent collisions, they struggle to guide the agent to smoothly navigate away from high-risk regions, leading to frequent boundary-hovering behaviors. Therefore, the improvement in the SC-only variant shows that the CMDP policy itself can learn safer behavior under the Lagrangian cost constraint.

8.3.2. ASM

The Action Shielding Mechanism (ASM) is designed to address the limitations of static constraints by actively correcting the agent’s behavior near safety boundaries. By comparing the variant “RAPRL w/o PP” (which includes SC and ASM) with the version containing only SC, we observe a decisive improvement in safety compliance. The most striking impact is reflected in the average cost. As detailed in Table 4, adding ASM drastically reduces the average cost in high-density scenarios from 0.23 to 0.02, an order-of-magnitude improvement. This result is visually corroborated by the average cost curve in Figure 10, where the green curve, corresponding to the SC and ASM variant without PP, converges rapidly to a near-zero cost level, significantly outperforming the oscillating orange curve (w/o ASM). Additionally, the success rate further increases to 98.3%. These results demonstrate that ASM further suppresses residual unsafe actions that may still be produced by the learned policy, thereby improving constraint satisfaction and stabilizing the average cost during training.

8.3.3. Personal Preference

Finally, we evaluate the contribution of the Personal Preference (PP) module in balancing safety with efficiency and accelerating learning. Comparing the complete RAPRL method with the “RAPRL w/o PP” variant, Table 4 reveals that PP plays a crucial role in recovering the efficiency lost due to safety restrictions. In high-density traffic, the average time decreases from 12.33s to 11.87s, and the success rate peaks at 99.0%. This indicates that preference-aware cost-limit adaptation helps the agent recover part of the efficiency lost under strict safety regulation and guides the policy toward a better safety–efficiency trade-off. Moreover, the average reward curve in Figure 10 (right) shows that the full method (red line) exhibits the fastest convergence speed and achieves the highest asymptotic reward. This confirms that incorporating human-like driving preferences not only optimizes the trade-off between safety and efficiency but also allows the agent to learn optimal policies more rapidly and stably.

8.3.4. Visual Result

We visualize the behaviors of agents trained under three settings for the on-ramp merging task: RAPRL without PP and ASM, RAPRL without PP, and the full RAPRL, as shown in Figure 11. The ego vehicle trained without PP and ASM can select unsafe or suboptimal actions, e.g., turning right despite having merged into the main lane (see Figure 11a). Incorporating ASM prevents such unsafe actions in both RAPRL w/o PP and full RAPRL (see Figure 11b,c). Furthermore, the agent trained by full RAPRL reaches the farthest position from the end of the merge zone (green car in Figure 11) at episode termination, demonstrating that integrating personalized preferences enhances traffic efficiency.

Overall, the ablation results indicate that the CMDP-based safety constraint and ASM contribute to safety in different but complementary ways. The SC-only variant improves the collision-related metrics compared with the unconstrained baseline, showing that the policy can lead to safer behavior under the Lagrangian cost constraint. However, SC alone still exhibits higher average cost and larger oscillations during training, suggesting that the learned policy may still approach unsafe boundaries. After adding ASM, the average cost is significantly reduced and the learning curve becomes more stable, indicating that the shielding module suppresses residual unsafe actions before execution. Therefore, the reported safety performance of RAPRL should be interpreted as the combined result of learned constraint-aware policy optimization and execution-level action shielding, rather than as the effect of either mechanism alone. Accordingly, the learned policy should be understood as a constraint-aware policy trained with execution-level shielding support, rather than a policy whose safety is solely guaranteed by post-processing.

8.4. Sensitivity Analysis

To evaluate the robustness of our framework, we conduct a sensitivity analysis on two critic design parameters that govern the system’s safety characteristics: the prediction coefficient $\sigma$ in the ASM and the cost limit $\eta$ (reflecting risk tolerance) in the CMDP.

8.4.1. Prediction Coefficient

We conduct motion prediction for constraint checking in the ASM of RAPRL and change the prediction coefficient $\sigma$ to test its effect on the average cost and average reward. The prediction coefficient $\sigma$ denotes the look-ahead step used by the ASM to select the predicted ego vehicle state from the MPC prediction horizon. Given the MPC horizon N, the predicted state sequence can be written as $\{x_e^1,x_e^2,\dots ,x_e^N\}$, where $x_e^{\sigma }$ denotes the predicted ego vehicle state at the $\sigma$-th future step. Therefore, the corresponding look-ahead time is

$$t_{\sigma }=\sigma \Delta t,\sigma \in \{1,\dots ,N\},$$

(65)

where $\Delta t$ is the simulation time interval. In this study, the MPC prediction horizon is $N=10$, and the simulation frequency is 10 Hz, resulting in $\Delta t=0.1$ s. Thus, $\sigma =1$, $\sigma =5$, and $\sigma =10$ correspond to approximately 0.1 s, 0.5 s, and 1.0 s look-ahead collision checking, respectively.

The value of $\sigma$ affects the trade-off between responsiveness and prediction reliability. A small $\sigma$ performs near-term collision checking and is more responsive to current states, but it may be too short-sighted to identify future merging conflicts. A large $\sigma$ provides earlier warning of potential conflicts, but the predicted state may contain larger uncertainty and may lead to overly conservative or unstable shielding decisions. Therefore, $\sigma$ is treated as a tunable hyperparameter of the ASM and is selected through sensitivity analysis.

As shown in Figure 12, we conducted five runs for each $\sigma$, initialized with different random seeds. We observe that a larger $\sigma$ leads to faster learning but results in higher variability across the random seeds, indicating increased instability. On the other hand, a smaller $\sigma$ can slow down the training process. However, with an appropriate prediction coefficient of $\sigma =5$, the model achieves the best performance regarding the average cost (the lowest value) and the average reward (the highest value). This result indicates that a mid-horizon prediction step provides a better balance between early conflict detection and reliable state prediction. Accordingly, $\sigma =5$ is used as the default prediction coefficient in the experiments.

8.4.2. Risk Tolerance

In our CMDP framework, the cost limit $\eta$ serves as a quantifiable proxy for the agent’s risk tolerance. Lower values of $\eta$ enforce conservative driving strategies with strict adherence to safety constraints, whereas higher values permit more aggressive policies. To investigate the impact of this preference indicator, we evaluated $\eta$ at 0.1, 0.05, 0.01, and 0.001, as illustrated in Figure 13. These results demonstrate that the policy is highly sensitive to this threshold. Larger $\eta$ values relax the constraints, resulting in instability and increased safety violations, while overly small $\eta$ values restrict exploration and slow down convergence. We empirically identify an optimal balance at $\eta =0.01$, where the agent maximizes safety compliance without compromising driving performance. However, relying on a fixed cost limit reveals a key limitation: a static $\eta$ cannot adapt to varying traffic densities or heterogeneous user preferences (e.g., aggressive driving in sparse traffic versus conservative driving in dense traffic). This limitation motivates our proposed approach, which dynamically adjusts $\eta$ using fuzzy logic to align the agent’s risk sensitivity with evolving environmental and user-specific requirements.

This sensitivity analysis also helps interpret the influence of fuzzy-rule settings on the learned policy. In the proposed framework, the fuzzy membership functions and rule table affect policy learning mainly through the generated CMDP cost limit $\eta$. Therefore, different fuzzy-rule settings would lead to different safety–efficiency behaviors by changing the resulting value of $\eta$. As shown in Figure 13, an overly large $\eta$ relaxes the safety constraint and may increase average cost, whereas an overly small $\eta$ imposes a stricter constraint and may slow down policy learning. These results indicate that the learned policy is sensitive to the cost-limit level produced by the fuzzy module. Accordingly, the adopted fuzzy rules are designed to generate moderate and adaptive cost limits under different risk-preference and traffic-density conditions, avoiding both excessively permissive and excessively conservative behaviors.

9. Discussion

Although the proposed method achieves promising results in the evaluated ramp-merging scenarios, its current validation remains subject to several practical limitations. First, the experiments were conducted in a controlled single-ramp environment with IDM-based surrounding vehicles, which provides a reproducible benchmark but cannot fully represent multi-lane interactions, heterogeneous driver styles, perception uncertainty, and non-IDM behaviors in real traffic. Second, the user risk preference is specified before policy learning and remains fixed during each experiment. This setting enables a controlled analysis of how different preference levels affect the CMDP cost limit and the learned merging behavior, but it does not capture preference variations over time or across traffic contexts. Third, the MPC prediction and execution modules are evaluated by simulation, where the finite-horizon QP problem is solved at each decision step. Therefore, the current results demonstrate algorithmic effectiveness in simulations rather than embedded real-time deployment capability.

Future work will address these limitations by evaluating the framework in more realistic traffic simulators, naturalistic driving datasets, and heterogeneous human driving behavior models. We will also investigate online preference inference and adaptation based on user feedback, historical driving behavior, and interaction histories. For deployment feasibility, hardware-in-the-loop validation and embedded-oriented MPC implementations, such as warm-started QP solvers, sparse optimization, code generation, and reduced-horizon MPC, will be further explored.

10. Conclusions

This study proposes a risk-aware safe RL approach with personal preferences for decision-making in the on-ramp merging task for autonomous driving, in which RL is used for high-level decisions, followed by a low-level MPC. We formulate the high-level decision problem as a CMDP that represents safety in cost terms. Furthermore, we use the fuzzy logical method to compute the threshold of the cost limits based on human risk preferences and traffic density and a Lagrangian-based SAC is used to solve the CMDP. An Action Shielding Mechanism is designed to remove unsafe or invalid RL actions, and we theoretically prove its effectiveness in enhancing safety and sample efficiency. Numerical simulations and theoretical analysis demonstrate the superiority of our method regarding success rate, collision ratio, and average cost. Future work will further improve the practicality and robustness of the proposed framework under more realistic traffic conditions and deployment settings.