Dynamic Platoon Re-Sequencing for Electric Vehicles Based on Bootstrapped DQN

Abstract

The energy consumption imbalance among electric vehicles (EVs) within a fixed platoon primarily originates from the distinct aerodynamic drag forces at different positions. This imbalance further causes practical challenges, such as inconsistent battery degradation rates and divergent charging durations. To tackle these challenges, dynamically adjusting the platoon formations during the journey is essential, which requires identifying the optimal vehicle sequences at designated re-sequencing points. In this research, we formulate the Optimal Re-Sequencing (ORS) problem as a multi-objective optimization problem that minimizes the imbalance of energy consumption, ensures a minimum remaining state of charge (SOC) for energy security, and penalizes excessive formation changes to maintain stability. To solve this optimization problem, we propose a deep reinforcement learning (DRL) framework based on Bootstrapped Deep Q-Networks. This framework integrates multi-head Q-value estimation and prioritized experience replay mechanisms to improve exploration efficiency and learning stability. Through simulation experiments based on a modeled Suzhou–Nanjing route, our proposed approach achieves a final SOC standard deviation of 0.00524, representing a 47% reduction compared with existing works, demonstrating superior efficiency and effectiveness in achieving fair energy consumption across EV platoons.

1. Introduction

Platooning refers to a group of vehicles traveling cooperatively in close formation [1], which is supported by Vehicle-to-Vehicle (V2V) communication and autonomous driving technologies. In recent research, platooning has shown a significant potential to improve traffic throughput [2,3], road safety [4,5], and energy efficiency [6] by reducing aerodynamic drag. Beyond single-platoon coordination, recent developments in large-scale cooperative traffic management—such as the HAC3 framework [7] and multi-agent reinforcement learning (MARL) approaches for dynamic routing—reflect a growing trend toward hierarchical and adaptive decision-making across multiple traffic layers and control scales. Most existing studies on platoon control are based on fixed platoon strategies, where vehicle positions in the platoon remain unchanged during operation [8,9]. In this framework, researchers have explored a variety of control mechanisms, including intervehicle spacing [10], predictive energy management [11], trajectory optimization [12], and speed planning [13,14,15]. Experimental studies have reported energy savings of 6–10% compared to vehicles without platooning [16]. In addition, advanced ecological controllers increase these savings to approximately 21% [17]. Although fixed platooning can effectively reduce overall energy consumption, it does not consider position-dependent disparities in energy use [18]. This imbalance arises from position-dependent aerodynamic exposure: leading vehicles face higher energy consumption due to greater drag, while trailing vehicles benefit from drafting effects, which substantially reduces their aerodynamic load [19,20]. As a result, such position-dependent disparities in energy consumption gradually lead to heterogeneous levels of SOC between EVs, and the EV with the lowest remaining SOC often constrains the range of the EV platoon.

To reduce the impact of position-dependent disparities in energy consumption, recent studies have explored non-fixed platoon methods, allowing vehicles to change their positions dynamically [21,22,23,24]. Ref. [21] proposed a multi-vehicle self-organized cooperative control strategy that uses a Distributed Model Predictive Control (DMPC) algorithm to merge local platoons into the target platoon as a whole, improving merging efficiency and stability. Ref. [22] developed two sequence change algorithms, named Fuel Amount Heuristic and Traveling Time Heuristic, that can dynamically reorder vehicles based on remaining fuel or elapsed time, extending the platoon driving range by approximately 3–9%. Ref. [18] introduced five re-sequencing optimization methods that significantly cut runtime and enhance energy balance compared to fixed-platoon strategies. Despite these advances, several limitations remain. First, most approaches emphasize energy fairness but overlook the dynamic and uncertain nature of real traffic. In congested conditions, limited road space and increased collision risk often render re-sequencing infeasible, and strategies relying solely on instantaneous decisions cannot ensure platoon stability or safety. Recent studies on adaptive energy management have shown improved robustness under complex driving environments by employing environment-aware multi-objective optimization [25], suggesting that adaptive decision-making could enhance stability under such uncertain conditions. However, such adaptive mechanisms have not yet been integrated into cooperative platooning. Second, the minimum remaining SOC between vehicles is rarely explicitly monitored, although it is critical to ensuring fleet-level energy security and to prevent a single low-energy vehicle from destabilizing the formation. Finally, most existing experiments focus only on small platoons, typically consisting of four vehicles, which restricts understanding of scalability and operational complexity in large fleets.

In response to these challenges, we formulate the dynamic platooning task as a multi-objective ORS problem, in which EVs adjust their positions at designated checkpoints to improve energy balance and formation stability. The optimization jointly considers SOC fairness, minimum remaining energy, and platoon stability to achieve balanced energy consumption across all vehicles. To address this problem, we develop a DRL-based control framework that learns adaptive re-sequencing strategies through interaction with the environment. This framework improves both energy efficiency and formation stability at the fleet level.

The main contributions of this paper are summarized as follows:

(1)

We extend the ORS problem in ref. [18] to a multi-objective framework that jointly minimizes SOC deviation, maximizes the minimum remaining SOC, and improves formation stability. This extension turns the original single-objective model into a fleet-level coordination scheme that explicitly balances fairness, energy assurance, and stability.

(2)

We adopt a Bootstrapped DQN with a redesigned reward function that adds an SOC-ranking action at the final re-sequencing stage. Compared with the standard DQN in ref. [23], this design encourages structured exploration and clearer reward guidance, improving convergence and reducing training complexity.

(3)

We benchmark the framework against multiple DRL and heuristic baselines under realistic road and energy profiles. Unlike earlier DRL-based platoon studies limited to small-scale or fixed action spaces, our approach scales to large, combinatorial coordination problems. It achieves low SOC deviation and high minimum SOC, improving both fairness and fleet-level resilience.

The remainder of this paper is organized as follows. Section 2 defines the formulation of the multi-objective ORS problem. Section 3 introduces the proposed DRL framework. Section 4 presents the experimental setup and performance analysis.

2. Problem Formulations

We consider a homogeneous platoon consisting of N EVs ($N\ge 2$), indexed by the set $V=\{1,2,\dots ,N\}$. The platoon travels a total distance d, through M re-sequencing checkpoints along the route. Let $d_0,d_1,\dots ,d_{M-1}$ denote the cumulative distances from the starting point to each of the M checkpoints, where $d_0=0$ represents the point of departure. The corresponding arrival times of the lead vehicle at these checkpoints are indicated by $t_0,t_1,\dots ,t_{M-1}$, and $t_f$ represents the arrival time of the entire platoon at its final destination. The vehicle sequence can be adjusted at checkpoints to balance energy consumption. The sequence of vehicle positions during each segment is represented by a position matrix $P\in {\mathbb{R}}^{N\times M}$, where the entry $P_{ij}\in \{1,2,\dots ,N\}$ indicates the position of vehicle i during the j-th platoon segment. Here, $i=1,\dots ,N$ and $j=1,\dots ,M$. The SOC of vehicle i at time $t_j$ is indicated by $s_i\left(t_j\right)$ and evolves according to the dynamics described in ref. [18]:

$$s_i\left(t_j\right)=s_i\left(t_{j-1}\right)-{\displaystyle \frac{\delta }{\Lambda }}\left(1-e_{P_{ij}}^j\right)\left(d_j-d_{j-1}\right).$$

(1)

This SOC model follows the formulation in ref. [18]. $\delta$ is the baseline energy consumption rate per unit distance and $\Lambda$ denotes the battery capacity. The aerodynamic reduction parameters $e_{P_{ij}}^j\in [0,1)$ represents the reduction rate in electricity usage associated with the $P_{ij}$-th position in the platoon, which are taken from drag measurements in ref. [26] and used directly in our simulations. The overall SOC state vector at time $t_j$ is defined as $s\left(t_j\right)={[s_1\left(t_j\right),\dots ,s_N\left(t_j\right)]}^T$. To express the cumulative energy consumption, we define an energy consumption matrix $C\triangleq \left[C_{ij}^j\right]$ for all vehicles and phases, where each entry is given by:

$$C_{ij}^j=s_i\left(t_{j-1}\right)-s_i\left(t_j\right).$$

(2)

In this work, we assume that

$$e_{P_{i_{2}j}}^j\ge e_{P_{i_{1}j}}^j,\mathrm{if}{P}_{i_{1}j}<P_{i_{2}j}.$$

(3)

Equivalently, vehicles positioned further back benefit from a stronger drag-reduction effect. This assumption is adopted for analytical convenience, while the main results remain valid even without it, requiring only a change in notation.

Based on the above formulation, to enhance both energy efficiency and formation robustness in platoon coordination, we define the optimization problem from the perspectives of fairness, energy security, and formation stability:

(1)

Fairness: Minimize the standard deviation of the final SOCs $\sigma \left(s\left(t_f\right)\right)$ to ensure balanced energy consumption in all vehicles.

(2)

Energy security: Maximize the remaining SOC $min\left(s\left(t_f\right)\right)$, because the vehicle with the lowest remaining energy determines both the charging time and the endurance of the entire platoon.

(3)

Formation stability: Minimize instability during the re-sequencing process, measured by the L1-norm of differences between consecutive formations, defined as

$$\mathcal{I}=\sum _{j=1}^{M-1}{\parallel P(:,j)-P(:,j+1)\parallel }_1.$$

Based on the above optimization goals, we extend the ORS problem proposed in ref. [18] to the following multi-objective optimization problem, defined as follows.

$$\begin{array}{cc}\hfill \underset{P}{max}& F\left(s\left(t_f\right)\right)\triangleq \left[-\sigma \left(s\left(t_f\right)\right),min\left(s\left(t_f\right)\right),-\mathcal{I}\right],\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left(1\right),\hfill \\ \hfill & s_i\left(t_f\right)\ge 0,\forall i\in \{1,\dots ,N\},\hfill \\ \hfill & P_{ij}\in \{1,\dots ,N\},\hfill \\ \hfill & P_{i_{1}j}\ne P_{i_{2}j},\forall i_1\ne i_2,j=1,\dots ,M.\hfill \end{array}$$

(4)

3. Algorithm Design for Dynamic Platoon Re-Sequencing

To address the multiobjective ORS problem introduced in Section 2, we develop a DRL framework that models vehicle re-sequencing as a sequential decision-making task. Specifically, the framework is organized into two parts: an environment simulation module and a DRL agent module. The environment maintains the state of the platoon, updates the SOC according to the position-dependent consumption matrix, and provides rewards based on fairness, energy security, and stability. The DRL agent observes the state, selects actions from the defined action space, and is trained using the Bootstrapped DQN [27] combined with prioritized experience replay to enhance exploration efficiency and training stability. Unlike the conventional $\epsilon$-greedy exploration strategy, which often suffers from low sample efficiency and weak generalization, Bootstrapped DQN maintains an ensemble of independently initialized value heads. At the beginning of each episode, one head is randomly sampled for policy evaluation and update, approximating Thompson sampling and enabling diverse exploration. This ensemble mechanism provides uncertainty-aware decision making and improves convergence stability in the ORS environment. This interaction process is formalized as a Markov Decision Process (MDP) and implemented in a customized OpenAI Gym environment. The details of the state space, action space, and reward function are described in the following subsections.

3.1. Agent Modeling and Environment Design

3.1.1. State Space

A customized Gym-compatible environment is constructed to emulate the platooning dynamics of N homogeneous EVs over M re-sequencing stages. At each stage j, vehicles can be re-sequenced according to their position-dependent energy efficiency. The hybrid system state combines both energy status and temporal information, defined as

$$S\left(t_j\right)=[s_1\left(t_j\right),s_2\left(t_j\right),\dots ,s_N\left(t_j\right),l],$$

(5)

where $s_i\left(t_j\right)$ denotes the SOC of vehicle i at stage j, and $l=M-j$ is the number of remaining re-sequencing opportunities. Each $s_i\left(t_j\right)$ is normalized to the range $[0,1]$, where $s_i\left(t_j\right)=1$ indicates a fully charged battery and $s_i\left(t_j\right)=0$ represents complete depletion. Hence, the state vector has dimension $N+1$.

3.1.2. Action Space

At each re-sequencing stage, the agent selects a new platoon formation from the action space $\mathcal{A}$ based on the current environmental state to perform re-sequencing. The action space $\mathcal{A}$ consists of two parts: (i) all permutations of the vehicle set $\{1,2,\dots ,N\}$, denoted by ${\mathcal{A}}_{\mathrm{perm}}=Perm(1,2,\dots ,N)$ with $N!$ actions, where each action corresponds to a specific platoon formation; and (ii) one additional SOC-ranking action ${\pi }_{\mathrm{SOC}}=argsort(-s\left(t_j\right))$, which assigns vehicles in descending order of their SOC so that high-energy vehicles take the leading positions. The complete action set is therefore

$$\mathcal{A}={\mathcal{A}}_{\mathrm{perm}}\cup \left\{{\pi }_{\mathrm{SOC}}\right\},\left|\mathcal{A}\right|=N!+1,$$

(6)

where ${\pi }_{\mathrm{SOC}}$ is admissible only in the final stage, and any premature execution is penalized as an illegal action. Each action specifies a new formation that directly determines the subsequent position-dependent energy consumption according to (2). It is noted that the factorial growth of the action space with respect to N may lead to high computational complexity for large-scale platoons; however, in practical deployments this issue can be mitigated by restricting the action candidates or adopting hierarchical coordination to maintain tractable decision making.

3.1.3. Reward Function

Building on the standard deviation minimization framework proposed in ref. [23] and the SOC ranking optimality proven in Theorem 2 of ref. [18], we design a multi-objective stage-wise reward function that primarily minimizes the standard deviation of final SOCs $\sigma \left(s\left(t_f\right)\right)$, while also preserving $min\left(s\left(t_f\right)\right)$ and maintaining formation stability. A special SOC ranking action is enabled exclusively at the final re-sequencing stage and is positively rewarded. The reward function is formulated as follows:

$$\begin{array}{c}\hfill r=\left\{\begin{array}{c}-\alpha \sigma \left(s\left(t_f\right)\right)+\beta min\left(s\left(t_f\right)\right)-\gamma \mathcal{I}+\zeta ,j=M,\hfill \\ -\lambda ,\mathrm{if}\mathrm{the}\mathrm{DRL}\mathrm{agent}\mathrm{takes}\mathrm{an}\mathrm{illegal}\mathrm{action}.\hfill \end{array}\right.\end{array}$$

(7)

where $\alpha$, $\beta$, $\gamma$, $\zeta$, and $\lambda$ are used to balance fairness, energy assurance, formation stability, final-stage bonus, and penalty for illegal actions, respectively.

3.2. Policy Architectures

Exploration Improvement Strategy

Efficient exploration is essential in high-dimensional discrete action spaces. To move beyond simple $ϵ$-greedy or NoisyNet-based perturbations, we employ Bootstrapped DQN [27]. This choice allows our policy design to capture uncertainty and encourage structured exploration. The architecture consists of a shared feature extractor followed by multiple independent output heads. At the start of each episode, one head k is sampled and used consistently for both action selection and Q-value evaluation:

$$a=arg\underset{a}{max}{Q}^{\left(k\right)}(s,a;{\theta }_k),$$

(8)

where ${\theta }_k$ are the parameters of the k-th head. This mechanism approximates Thompson sampling by introducing head-specific stochasticity across episodes. The ensemble of heads thus provides uncertainty-aware decision making and improves temporal consistency during exploration. Compared with conventional $ϵ$-greedy, this strategy enables more structured exploration, which is particularly beneficial in ORS tasks characterized by latent state dependencies and delayed feedback.

3.3. Experience Replay Strategy

To improve sample efficiency and accelerate policy learning, we adopt prioritized experience replay (PER) as part of the training framework. Instead of uniformly sampling transitions from the replay buffer, PER selects experiences with higher temporal-difference (TD) errors more frequently, guiding the agent to focus on samples that are more likely to improve the Q-function approximation. For a given transition k, its sampling probability is computed as:

$$p\left(k\right)={\displaystyle \frac{{\mathrm{pri}}_k}{{\sum }_m{\mathrm{pri}}_m}},$$

(9)

where the priority score is defined as ${\mathrm{pri}}_k=\left|{\delta }_k\right|+\varphi$. Here, ${\delta }_k$ denotes the TD error for transition k, and $\varphi$ is a small positive constant added to avoid zero probability and ensure that all samples retain a non-zero chance of being selected.

To implement efficient priority-based sampling and updates, we employ the SumTree structure, which supports $\mathcal{O}(logN)$ time complexity for both sampling and priority adjustment, where N is the buffer size. This prioritization focuses training updates on high-error regions of the state–action space, thereby accelerating convergence and enhancing stability in the ORS environment.

3.4. Computational Complexity

The overall training complexity of the proposed Bootstrapped DQN framework can be analyzed in terms of three major components: (1) the forward and backward propagation of K bootstrap heads, (2) prioritized experience replay sampling and updates implemented using a SumTree structure, and (3) environment interactions and state updates. For each training step, the computational cost is approximately $\mathcal{O}(KP+Blog{N}_{\mathrm{buf}})$, where K is the number of bootstrap heads, P is the number of parameters per head, B is the batch size, and $N_{\mathrm{buf}}$ is the size of the replay buffer. Over T training steps, the total complexity becomes $\mathcal{O}(T·(KP+Blog{N}_{\mathrm{buf}}))$. Although the ensemble design introduces a linear factor K, the complexity remains in the same order of magnitude as a standard DQN implementation while providing improved exploration efficiency and learning stability.

The detailed training procedure of the proposed Bootstrapped DQN-based dynamic platoon re-sequencing framework is provided in Appendix A.

4. Experiments

4.1. Experiment Setting

A simulated platoon of five fully charged Tesla EVs travels along a modeled Suzhou–Nanjing route, as shown in Figure 1. The route parameters are derived from realistic geographic and energy consumption data. The average energy-consumption rate is set to $\delta =340\mathrm{Wh}/\mathrm{mile}$, and each vehicle has a battery capacity of $\Lambda =85\mathrm{kWh}$. Most previous studies use four-vehicle platoons [23]. Here, we set the platoon size to $N=5$, which expands the action space by about five times while keeping the computational cost manageable. This setting allows richer positional interactions—especially among mid-platoon vehicles—and creates a more representative dynamic re-sequencing scenario. It also produces a smoother, more statistically meaningful SOC distribution across vehicle positions, making the evaluation of energy balance clearer. In addition, the larger action space encourages more diverse policy exploration during DRL training, improving the generalization of the learned policy. When the platoon size exceeds six, however, the factorial growth of permutation-based actions makes the search space excessively large and the training time impractical. Thus, $N=5$ provides a practical trade-off between behavioral richness and computational feasibility. All vehicles are assumed to maintain ideal V2V communication for state sharing, so that the analysis can focus on control policy learning and energy coordination.

The platoon is allowed to reconfigure its formation at five designated re-sequencing stages between six checkpoints along the route: Suzhou ($d_0=0$), Wuxi ($d_1=26.2$), Changzhou ($d_2=71.7$), Zhenjiang ($d_3=123.8$), Yizheng ($d_4=149.2$), and Nanjing ($d_5=196.6$). The total route length is approximately 196.6 miles.

Based on the aerodynamic position parameters reported in ref. [26], the electricity usage reduction rates are set as

$$e_1^j=0.043,e_2^j=0.10,e_3^j=0.14.$$

Since aerodynamic measurements beyond the third vehicle are not available in the existing literature and the incremental aerodynamic benefit beyond the third position is relatively small, we assume

$$e_4^j=e_5^j=0.14.$$

Thus, the consumption matrix $\mathbf{C}$ is constructed according to Equation (2) as follows:

$$\mathbf{C}=\left[\begin{array}{ccccc}0.1048& 0.1747& 0.1996& 0.0972& 0.1818\\ 0.0991& 0.1652& 0.1887& 0.0919& 0.1719\\ 0.0951& 0.1586& 0.1810& 0.0882& 0.1650\\ 0.0951& 0.1586& 0.1810& 0.0882& 0.1650\\ 0.0951& 0.1586& 0.1810& 0.0882& 0.1650\end{array}\right].$$

The simulation is executed on an NVIDIA GeForce RTX 5090 (32 GB GDDR7) using the parameters listed in

Table 1. System and Reward Parameters.

Parameter	Value
Hidden dimension	128
Learning episodes	80,000
Replay buffer size	50,000
Discount factor	0.98
Learning rate	0.002
Exploration rate	0.01
Batch size	128
Minimum sample size	1000
Soft update factor	0.01 → 0.005 → 0.001
Smoothing factor	0.005
Priority replay factor	0.6
Prioritized replay (init/incr)	0.4/$1\times {10}^{-5}$
Mask probability (PER)	0.8
Number of heads (Bootstrapped DQN)	5
Fairness weight $\alpha$	500
Remaining SOC reward $\beta$	30
Instability penalty $\gamma$	30
SOC ranking bonus $\zeta$	3
Illegal action penalty $\lambda$	5

. Before fixing the final reward coefficients, we ran preliminary experiments to explore the typical ranges of the three objectives: SOC standard deviation, minimum remaining SOC, and formation stability. Because both $\sigma \left(s\left(t_f\right)\right)$ and $min\left(s\left(t_f\right)\right)$ naturally fall within [0, 1], the stability metric $\mathcal{I}$ was normalized to the same scale. A balanced reward configuration was then chosen to maintain stable optimization among the three objectives. To avoid the agent’s over-reliance on the SOC-ranking action, the terminal bonus $\zeta$ and penalty $\lambda$ were kept as small constants.

4.1.1. Parameter Sensitivity Analysis

To validate the chosen parameter configuration, sensitivity analyses were performed to assess how key factors influence learning performance. Specifically, we examined the impact of the number of bootstrap heads in the Bootstrapped DQN and the reward coefficients $\alpha$, $\beta$, and $\gamma$. The sensitivity tests for the number of heads were conducted over 20,000 training episodes, while those for the reward coefficients were each performed over 5000 episodes. These analyses confirmed that the selected configuration achieves a stable trade-off among fairness, energy assurance, and formation stability.

This sensitivity analysis was conducted to study how the number of bootstrap heads ($N_{\mathrm{head}}$) affects the learning performance of the Bootstrapped DQN. As shown in

Table 2. Sensitivity analysis of the number of bootstrap heads ($N_{\mathrm{head}}$).

${\mathit{N}}_{\mathbf{head}}$	$\mathit{\sigma }\left(\mathbf{SOC}\right)$	$min\left(\mathbf{SOC}\right)$	$\mathcal{I}$
1	0.01655	0.349	0.08
3	0.01397	0.351	0.16
5	0.01053	0.356	0.09
7	0.01885	0.341	0.34
15	0.01932	0.336	0.25

and Figure 2, the results indicate that increasing $N_{\mathrm{head}}$ improves exploration at first, but too many heads may lead to unstable value estimation. The configuration with $N_{\mathrm{head}}=5$ achieves balanced performance in terms of SOC deviation, minimum SOC, stability, and average reward. This value is therefore adopted in the main experiments.

This sensitivity analysis evaluates the effect of the fairness weight $\alpha$ on learning performance. As shown in

Table 3. Sensitivity analysis of the fairness weight $\alpha$.

$\mathit{\alpha }$	$\mathit{\sigma }$(SOC)
100	0.02145
200	0.02052
350	0.01840
650	0.01707
800	0.01617

and Figure 3, the SOC standard deviation $\sigma \left(\mathrm{SOC}\right)$ decreases monotonically as $\alpha$ increases, indicating that a larger weight improves energy fairness among vehicles. Considering convergence stability and computational cost, a moderate value of $\alpha$ was used in the main experiments to maintain a reasonable balance between fairness and overall performance.

This sensitivity analysis evaluates the effect of the energy assurance weight $\beta$ on overall performance. As shown in

Table 4. Sensitivity analysis of the energy assurance weight $\beta$.

$\mathit{\beta }$	Min SOC
15	0.342
25	0.346
45	0.346
60	0.348
65	0.350

and Figure 4, the minimum remaining SOC increases gradually as $\beta$ grows, indicating that a stronger emphasis on energy assurance improves the fleet’s energy security. Beyond moderate values, the improvement becomes marginal, suggesting that a moderate $\beta$ provides sufficient energy protection without noticeably affecting other objectives. The adopted configuration already enables the model to handle the minimum SOC objective effectively, ensuring adequate energy assurance without additional tuning.

This sensitivity analysis investigates how the stability weight $\gamma$ influences formation control. As shown in

Table 5. Sensitivity analysis of the stability weight $\gamma$ (log scale).

$\mathit{\gamma }$	$\mathcal{I}$
0	0.49
15	0.37
25	0.35
40	0.18
55	0.16

and Figure 5, increasing $\gamma$ consistently reduces the instability metric $\mathcal{I}$, indicating that stronger penalization of formation changes improves stability. The reduction is steep when $\gamma$ increases from 0 to 25 and becomes more gradual beyond 40, suggesting diminishing returns at higher values. The chosen moderate $\gamma$ thus provides sufficient stability improvement while keeping formation adjustments responsive and efficient.

4.1.2. Baseline Algorithms

For comparative evaluation, four representative DQN variants are selected as baselines: the standard DQN [28,29], Double DQN [30], Dueling DQN [31], and a static SOC-based ranking heuristic [18]. All baseline algorithms are trained under identical simulation conditions to ensure fair comparison.

4.2. Evaluation Metrics

We evaluate the scheduling performance of the proposed algorithm in terms of fairness, policy stability, formation stability, computational efficiency, and energy guarantee. Fairness is quantified by the SOC standard deviation $\sigma \left(s\left(t_f\right)\right)$; policy stability by the return standard deviation $\sigma \left(R\right)$; formation stability by the stability metric $\mathcal{I}$, where a smaller $\mathcal{I}$ indicates higher stability; computational efficiency by the average inference time; and energy guarantee by the minimum remaining SOC $min\left(s\left(t_f\right)\right)$. These metrics were continuously monitored during training to verify the convergence and stability of each algorithm. As shown in Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10, the trajectories of Q-values, average returns, SOC deviation, minimum SOC, and formation stability together depict the overall training and validation process. The Bootstrapped DQN converged rapidly—within about 10,000 episodes—while maintaining the highest average return and the lowest SOC deviation, highlighting its superior learning efficiency and fairness performance.

4.3. Training Process

To evaluate the policy stability of our algorithm, we take Vanilla DQN, Double DQN, Dueling DQN and SOC ranking algorithms as the baseline algorithms. The training results are shown below.

Figure 6 shows the Q-value trajectories during training. The Bootstrapped DQN quickly rises to a plateau around 12 and then gradually decreases toward 6, remaining consistently higher than those of the other methods despite slight oscillations. Vanilla DQN exhibits a smooth transition from negative to positive values with moderate fluctuations. Double DQN fluctuates widely and often crosses zero, suggesting unstable learning in the later stages. Dueling DQN stays at a low level without sustained improvement, indicating weak value estimation capability. These observations confirm that Bootstrapped DQN achieves faster and more stable value learning than the other baseline algorithms.

Figure 7 shows the evolution of the average return during training. The Bootstrapped DQN rises rapidly after about 10,000 episodes and maintains a high return thereafter, with minor oscillations that reflect active yet stable exploration. Vanilla DQN improves gradually and stabilizes at positive values in the later stage. Double DQN exhibits large fluctuations and frequently drops below zero, suggesting unstable policy learning. Dueling DQN remains close to zero throughout most of the training, showing limited progress. Compared with the other DQN variants, Bootstrapped DQN achieves both faster convergence and smoother learning dynamics.

Figure 8 shows the evolution of the SOC standard deviation during training. The Bootstrapped DQN converges rapidly and consistently maintains the lowest deviation, stabilizing near 0.005 after about 10,000 episodes. This behavior demonstrates that its policy achieves the most balanced energy consumption across the fleet. Vanilla DQN also reduces SOC variance but converges more slowly and less steadily. In contrast, Double DQN and Dueling DQN stay at much higher levels—around 0.015 and 0.017—indicating weaker capability in maintaining fairness among vehicles. Overall, Bootstrapped DQN exhibits clear superiority in ensuring equitable energy distribution within the platoon.

Figure 9 shows the evolution of the minimum remaining SOC during training. The Bootstrapped DQN rapidly raises the minimum SOC to about 0.36 within 20,000 episodes and keeps it at the highest level thereafter, reflecting a strong capability for maintaining energy reserves. Vanilla DQN improves more gradually and eventually reaches a similar level with mild fluctuations. Double DQN oscillates sharply and often drops below 0.345, whereas Dueling DQN stays nearly flat around 0.348. Taken together, these observations show that Bootstrapped DQN provides the most reliable energy assurance across the training process.

Figure 10 shows the evolution of the formation stability metric $\mathcal{I}$ during training. Double DQN maintains the highest instability throughout, while Dueling DQN achieves the lowest $\mathcal{I}$ after convergence. Bootstrapped DQN and Vanilla DQN stay between the two extremes, keeping moderate stability levels. Among all methods, Bootstrapped DQN reaches a steady and balanced performance, indicating stable coordination without excessive formation changes.

To evaluate the impact of reward design, we construct a simplified environment by removing penalties for illegal actions and bonuses for SOC-based ranking, disabling explicit reward guidance during training.

Figure 11 shows the Q-value trajectories during training without reward shaping. The Bootstrapped DQN increases rapidly between 20,000 and 60,000 frames and then stabilizes near 6, staying above the other algorithms throughout the process. Vanilla DQN remains mostly negative with only minor improvement toward the end. Double DQN fluctuates sharply around zero and does not reach convergence, whereas Dueling DQN achieves brief early gains followed by a gradual decline. Overall, Bootstrapped DQN preserves stable value estimation even in the absence of explicit reward guidance.

Figure 12 shows the evolution of the average return during training without reward shaping. The Bootstrapped DQN rises steadily after about 40,000 episodes and stabilizes around 4–5, showing clear convergence and consistently higher returns than the other methods. Vanilla DQN gradually improves in the later stage and levels off near 2–3. Dueling DQN stays relatively flat around 1.5–2.5 with limited progress, whereas Double DQN oscillates strongly and often drops below zero. Even without reward shaping, Bootstrapped DQN maintains smooth learning dynamics and stable policy improvement throughout training.

Figure 13 shows the convergence of the SOC standard deviation during training without reward shaping. The Bootstrapped DQN decreases rapidly and stabilizes around 0.008, reaching the lowest deviation among all algorithms. Vanilla DQN remains near 0.010 with mild fluctuations. Double DQN stays at the highest level between 0.021 and 0.025, showing strong volatility, whereas Dueling DQN stays relatively steady around 0.016 without further decline. Bootstrapped DQN still achieves the most balanced energy distribution among vehicles under this simplified training condition.

Figure 14 depicts the evolution of the minimum remaining SOC during training without reward shaping. The Bootstrapped DQN rises rapidly between 10,000 and 40,000 episodes and stabilizes around 0.365, maintaining the top trajectory throughout. Dueling DQN remains steady near 0.350, Vanilla DQN declines to about 0.330 before recovering, and Double DQN fluctuates heavily below 0.33. Throughout the training, Bootstrapped DQN maintains the highest and most stable minimum SOC among all approaches.

Figure 15 illustrates the evolution of the formation stability metric $\mathcal{I}$ during simulation training without reward shaping. Dueling DQN exhibits the most stable yet overly conservative behavior. Double DQN shows the highest instability, reflecting erratic decision making in the absence of explicit penalties. Bootstrapped DQN and Vanilla DQN maintain moderate stability levels, balancing steadiness and adaptability. These results demonstrate that Bootstrapped DQN preserves robust coordination performance even without reward guidance.

Based on the above analysis, the removal of reward shaping leads to clear degradation of fairness and energy guarantee across all methods, consistent with Theorem 2 in [18]. Nevertheless, Bootstrapped DQN retains comparatively stronger robustness, achieving higher remaining SOC and lower standard deviation than other algorithms, even though its performance is weaker than in the shaped setting. This robustness highlights its adaptability to sparse or poorly designed rewards and its potential for practical use in platoon control.

4.4. Convergence and Stability Analysis

The convergence behavior of the proposed Bootstrapped DQN framework is illustrated in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15. As shown in Figure 6 and Figure 7, Bootstrapped DQN rapidly converges to a high Q-value plateau of 12 and consistently maintains higher returns than the baseline algorithms, indicating fast and stable learning. Figure 8 and Figure 9 further show that the SOC standard deviation and the minimum remaining SOC converge to steady values of 0.005 and 0.36 after 10,000 and 20,000 episodes, respectively, revealing balanced energy consumption and reliable energy assurance across the fleet. Regarding formation stability, Figure 10 indicates that Bootstrapped DQN maintains a moderate instability level ($\mathcal{I}=0.13$), achieving a desirable trade-off between robustness and flexibility.

To evaluate the influence of reward design, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 present the results obtained without explicit reward shaping. Although convergence becomes slower in this ablation setting, Bootstrapped DQN still achieves the highest Q-values and returns, the lowest SOC deviation (0.008), and a moderate stability index ($\mathcal{I}=0.14$), demonstrating that the proposed framework remains robust even under weaker reward guidance.

4.5. Numerical Results

We compare four DRL algorithms with a conventional SOC-ranking heuristic [18]. The results are reported in

Table 6. Comparison of DRL algorithms and SOC ranking heuristic.

Algo	$\mathit{\sigma }\left(\mathit{R}\right)$	$\mathit{\sigma }\left(\mathit{s}\right)$	$\mathcal{I}$	Min SOC	Time (s)
Vanilla DQN	4.8288	0.00986	0.1215	0.3563	0.00056
Double DQN	4.7125	0.01463	0.1526	0.3474	0.00039
Dueling DQN	3.0740	0.01790	0.05658	0.3465	0.00076
Bootstrapped DQN	5.0594	0.00524	0.1272	0.3632	0.00112
SOC Ranking	N/A	0.00466	1.8000	0.2836	0.00026

for the main experiment and in

Table 7. Performance in the ablation experiment.

Algo	$\mathit{\sigma }\left(\mathit{R}\right)$	$\mathit{\sigma }\left(\mathit{s}\right)$	$\mathcal{I}$	Min SOC	Time (s)
Vanilla DQN	3.9072	0.01985	0.005787	0.3382	0.00077
Double DQN	4.8907	0.01736	0.01570	0.3422	0.00073
Dueling DQN	3.2419	0.01635	0.005579	0.3475	0.00099
Bootstrapped DQN	5.2101	0.00846	0.1404	0.00145

for the ablation study where penalty and reward terms are removed.

As shown in Table 6, Vanilla DQN and Double DQN show a limited improvement in fairness, while stability $\mathcal{I}$ values remain relatively high, indicating weaker scheduling robustness. Dueling DQN exhibits a relatively stable return trajectory, but this stability is achieved at the expense of fairness and minimum SOC, suggesting that energy guarantees are compromised. Bootstrapped DQN, in contrast, provides the most balanced trade-off. It achieves the lowest SOC deviation, thereby ensuring fairness across the fleet, while simultaneously maintaining the highest minimum SOC and moderate formation stability. Although its inference time is slightly longer than the other DRL algorithms, the difference remains negligible for practical applications. The SOC-ranking heuristic, serving as a theoretical benchmark, unsurprisingly achieves the lowest standard deviation by design. However, its extremely poor performance in terms of minimum SOC and stability highlights its impracticality for real-world platoon control. Overall, Bootstrapped DQN provides near-optimal fairness together with robustness and strong energy guarantees, demonstrating superior applicability in real-world EV platoon management.

As shown in Table 7, removing penalties and SOC-based ranking rewards leads to a noticeable degradation across all methods. Fairness deteriorates notably compared with the shaped setting, while the minimum SOC values also decrease, reflecting weaker energy guarantees. However, stability remains relatively unaffected. Among the four algorithms, the Bootstrapped DQN demonstrates the highest resilience, achieving the lowest SOC deviation, the highest minimum SOC, and moderate stability despite the absence of reward shaping. These results highlight that reward shaping terms effectively guide training. The Bootstrapped DQN demonstrates strong robustness and adaptability under sparse or poorly designed reward signals, making it a reliable option for real-world platoon scheduling where reward functions may be incomplete or difficult to design precisely.

5. Conclusions

This paper proposed a Bootstrapped DQN framework for energy-balanced coordination in electric vehicle platoons, formulating the re-sequencing task as a multi-objective optimization problem that simultaneously considers fairness, energy guarantee, and formation stability. The framework integrates multi-head exploration, value bootstrapping, and prioritized experience replay to enhance convergence efficiency and robustness. Simulation experiments conducted on a modeled Suzhou–Nanjing route demonstrated that the proposed method consistently outperforms Vanilla, Double, and Dueling DQN, achieving the lowest SOC deviation and the highest minimum SOC while maintaining moderate stability. Compared with the SOC-ranking heuristic, which ensures theoretical fairness but suffers from poor stability and energy assurance, Bootstrapped DQN achieves a more practical trade-off between fairness and applicability. Furthermore, ablation studies confirmed the effectiveness of reward shaping in improving fairness and stability, while highlighting the robustness of Bootstrapped DQN under sparse reward signals. The current framework has been evaluated under simplified conditions, considering a homogeneous five-vehicle platoon and assuming ideal communication without latency or packet loss. The present simulations do not incorporate complex real-world traffic dynamics such as congestion, cut-ins, or diverse driving behaviors, which could influence platoon coordination. Future work will extend the framework to heterogeneous and larger-scale platoons, integrate realistic traffic and communication conditions, and include comparative evaluations with other advanced reinforcement learning and heuristic algorithms such as A2C to further assess scalability and robustness.