Co-Optimization of Cooperative Adaptive Cruise Control and Energy Management for Plug-in Hybrid Electric Truck Platoons

Abstract

To optimize fuel economy for platooning plug-in hybrid electric trucks, this paper proposes a co-optimization framework that integrates cooperative adaptive cruise control and energy management to enhance driving safety and fuel efficiency in complex traffic environments. The control strategy is divided into two layers: in the upper layer, a cooperative adaptive cruise control model based on distributed model predictive control (DMPC) is used to achieve stable platoon following and vehicle spacing, thus improving the overall platoon efficiency. In the lower layer, a distributed soft actor-critic (DSAC) algorithm is used for the fine-grained power distribution of plug-in hybrid electric trucks, enabling efficient energy utilization. The results demonstrate that this strategy significantly enhances the fuel economy and vehicle-following performance of plug-in hybrid truck platoons. Compared with the classical deep deterministic policy gradient (DDPG) algorithm, the energy management strategy based on the distributed soft actor-critic offers higher computational efficiency.

1. Introduction

With continued global economic growth, concerns over environmental pollution and energy use have intensified [1]. In response, hybrid electric vehicles (HEVs) have been widely recognized as an effective approach to curb vehicle fuel consumption and emissions [2]. By combining an internal combustion engine with electric propulsion, HEVs can lower greenhouse gas emissions and regulated pollutants relative to conventional vehicles [3]. Additionally, in contrast to pure electric vehicles, HEVs offer longer driving ranges, effectively addressing the issue of limited battery capacity [4]. However, since HEVs are equipped with multiple power sources, such as engines and electric motors, designing an efficient energy management strategy is crucial for maximizing their performance [5].

Current studies on energy management for plug-in hybrid electric trucks generally focus on two classes of approaches, namely rule-based strategies and optimization-based strategies [6]. Rule-based energy management strategies rely on predefined rules and logic derived from practical experience or heuristics to dynamically control the vehicle’s power sources [7]. Examples include state machine logic controllers and fuzzy logic controllers [8]. Although rule-based strategies offer good real-time performance and stability, they often lack flexibility and have limitations in optimizing performance [9]. On the other hand, optimization-based energy management strategies involve setting control objectives and constraints to search for the optimal control strategy [10]. Examples of such strategies include dynamic programming (DP) [11], model predictive control (MPC) [12], equivalent consumption minimization strategy (ECMS) [13], genetic algorithms (GA) [14], and game theory (GT) [15]. DP provides a globally optimal solution when the entire driving cycle is known a priori, and it is commonly adopted as a reference benchmark for evaluating energy management performance [16]. MPC and ECMS are typical real-time optimization strategies that are widely applied in online energy management [17]. ECMS defines the conversion relationship between electrical energy and fuel consumption through the use of equivalence factors, making the selection of these factors crucial. MPC exhibits robust performance, facilitating enhanced control capabilities for hybrid electric vehicles (HEVs) in complex environments.

Zeng et al. [18] developed a stochastic model predictive control (SMPC)-based energy management method that integrates road grade information into the optimization formulation. By constructing stochastic models for fuel consumption and battery state of charge (SOC), their simulations showed improved fuel economy. Zhang et al. [19] proposed an MPC-based scheme for intelligent plug-in HEVs, where a micro-traffic flow analysis model was used to enhance adaptability across diverse driving conditions. More recently, driven by advances in artificial intelligence, HEV energy management has increasingly leveraged data-driven techniques, including deep learning [20], reinforcement learning [21], and deep reinforcement learning [22]. Reinforcement learning is a learning mechanism that maximizes cumulative rewards by interacting with the environment, without the need for pre-labeled data, and optimizes itself through exploration and feedback from the environment [23]. Deep reinforcement learning (DRL) combines deep neural networks with reinforcement learning to learn policies/value functions in high-dimensional environments [24]. He et al. [25] proposed a hierarchical DRL-based energy management method: DDPG in the upper layer plans the SOC reference, while a DNN-based MPC in the lower layer performs power allocation, achieving 98.61% of the global-optimal fuel economy. Qi et al. [26] proposed an uncertainty-aware reinforcement learning–based energy management strategy, where actions are selected using a distributed uncertainty function. This design improves adaptability under complex driving conditions and leads to better fuel economy. Owing to their strong learning capability and adaptability, deep reinforcement learning methods have therefore attracted increasing attention in energy management research.

Energy management for hybrid electric vehicles is transitioning from traditional rule-based designs to more intelligent control frameworks [27]. However, existing strategies generally do not account for the driving environment [28]. Energy management performance is highly sensitive to the complexity and uncertainty of real-world driving conditions [29]. To better align the powertrain operation with the surrounding traffic environment, integrating eco-driving with energy management is crucial for unlocking additional energy-saving potential under complex driving conditions [30]. In particular, jointly optimizing adaptive cruise control and energy management enables coordinated decision-making beyond independently designed controllers [31]. Zhang et al. [32] proposed a framework for adaptive cruise control and energy management systems, utilizing deep neural networks to address upper-level following distance planning and lower-level power distribution separately, achieving significant reductions in vehicle energy consumption. Xie et al. [33] proposed an integrated model predictive control (IMPC) framework that unifies power management and adaptive speed control, jointly optimizing SOC and vehicle speed to improve both fuel economy and driving safety. By contrast, in conventional designs, adaptive cruise control primarily targets longitudinal following performance, whereas energy management is executed separately to allocate power-source torque according to the commanded acceleration [34]. This separated optimization approach fails to fully utilize the potential of joint control between adaptive cruise control and energy management strategies [35]. Khayyam et al. [36] proposed an adaptive cruise control model based on an adaptive neuro-fuzzy inference system, which takes into account factors such as aerodynamic drag, road gradient, and rolling resistance, achieving a 3% reduction in fuel consumption. Zhang et al. [37] developed a multi-objective integrated MPC-based adaptive cruise control scheme that enhances both tracking performance and collision avoidance. Peng et al. [38] proposed a heterogeneous multi-agent deep reinforcement learning method to enable coordinated optimization between adaptive cruise control and energy management. Furthermore, their approach improved computational performance by 10% through the use of prioritized experience replay techniques. By taking into account the coupling relationship between ACC and energy management, new integrated control strategies can be developed that more precisely match the operating states and efficiencies of the power sources, thus avoiding unnecessary stop-start cycles and sudden accelerations, which are highly energy-consuming driving behaviors [39]. The significance of co-optimizing adaptive cruise control and energy management strategies lies not only in improving the performance of individual systems but also in achieving overall vehicle energy efficiency and driving experience optimization through their coordinated functioning. This approach drives vehicles towards a more intelligent and environmentally friendly future.

Recent years have also witnessed increasing interest in integrating model predictive control (MPC) with deep reinforcement learning (DRL) to exploit their complementary strengths. In such hybrid designs, MPC is often used to explicitly handle constraints and provide receding-horizon optimization, while DRL is leveraged to enhance adaptability, learn complex mappings, or improve decision-making under uncertainty. This integration logic has been explored in other energy-related control problems [40], and related MPC–DRL paradigms have also been reported in vehicle and energy management contexts [41,42,43]. While these studies demonstrate the potential of MPC–DRL integration, the coupling architectures and problem formulations vary considerably, and hierarchical or co-optimization ACC–EM frameworks have already been reported in the literature as well. However, for plug-in hybrid electric truck (PHET) platoons, directly transferring existing designs can be nontrivial because platooning introduces stringent requirements on safety distance keeping, ride comfort, and stability under inter-vehicle coupling, and may also operate under limited information exchange and heterogeneous vehicle/powertrain characteristics. In addition, the traction demand of a truck platoon is strongly time-varying, which increases the difficulty of simultaneously achieving high-quality following performance and robust fuel-economy benefits.

Motivated by these platoon-specific challenges, the proposed method adopts a distributed hierarchical decomposition tailored to PHET platoons: the upper-layer DMPC focuses on safety- and comfort-oriented cooperative speed planning to shape the traction power demand, and the lower-layer DSAC performs continuous power-split optimization under the planned demand and powertrain constraints. This design is particularly suitable for the formulated problem because (i) platoon motion planning naturally requires constraint handling and coordination, which aligns with MPC, and (ii) energy management faces strongly time-varying demand and constraint-induced penalties, where distributional RL can improve learning robustness and convergence behavior.

To fully exploit the energy-saving potential of coordinated longitudinal control and power management in plug-in hybrid electric truck platoons, this paper proposes a co-optimization framework that jointly designs cooperative adaptive cruise control and energy management to improve both driving safety and fuel economy. The main contributions of this paper are as follows:

(1)

A distributed hierarchical co-optimization architecture for PHET platoons. We develop a DMPC–DSAC cooperative framework that decouples safety/comfort-constrained platoon motion planning from power-split optimization, enabling coordinated cruise control and energy management without requiring centralized data aggregation.

(2)

A distributed MPC layer tailored to platoon motion quality and coordination. The upper-layer DMPC explicitly targets stability and ride comfort while planning the platoon speed trajectory, providing a coordination-aware motion demand profile that supports smooth following and efficient operation under varying driving conditions.

(3)

A distributional RL-based energy management layer with improved learning robustness. The lower-layer DSAC performs continuous power allocation under the planned motion demand and powertrain constraints. By adopting distributional value learning within an entropy-regularized framework, the proposed strategy emphasizes convergence efficiency and training stability, and its effectiveness is validated through comparisons with representative benchmarks across different driving cycles.

The structure of this paper is organized as follows: Section 2 introduces the platoon model and powertrain model of plug-in hybrid electric trucks. Section 3 presents the co-optimization strategy for cooperative adaptive cruise control and energy management, including the cooperative adaptive cruise control model based on DMPC and the energy management strategy based on DSAC. Section 4 provides the simulation results and related discussions. Finally, Section 5 concludes the paper.

2. Modeling of Plug-in Hybrid Electric Trucks Systems

As shown in Figure 1, the plug-in hybrid electric trucks form a platoon and exchange information through vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. The entire platoon consists of n + 1 trucks, with the first truck serving as the lead vehicle and the subsequent n trucks as following vehicles. To ensure safe and efficient operation of the platoon, this paper presents a cooperative adaptive cruise control model based on a distributed model predictive control (DMPC) algorithm. The following vehicles track the lead vehicle while maintaining an appropriate distance, thereby enhancing the overall throughput of the platoon.

2.1. Truck Platoon Model

To better capture platoon dynamics, we formulate a nonlinear vehicle model for each truck. Assuming there are N trucks in the queue, the nonlinear longitudinal dynamics of the $i$-th truck at time t + 1 are as follows:

$$\left\{\begin{array}{l}{s}_i(t+1)=s_i(t)+v_i(t)\mathrm{\Delta }t,\\ v_i(t+1)=v_i(t)+\left({\displaystyle \frac{{\eta }_i}{R_i}}{T}_i(t)-{\displaystyle \frac{C_D\rho A_i}{2}}{v}_i^2(t)-m_{i}g{f}_i\right){\displaystyle \frac{\mathrm{\Delta }t}{m_i}},i\in N\\ T_i(t+1)=T_i(t)-{\displaystyle \frac{1}{{\mu }_i}}{T}_i(t)\mathrm{\Delta }t+{\displaystyle \frac{1}{{\mu }_i}}{\widehat{T}}_i(t)\mathrm{\Delta }t,\end{array}\right.$$

(1)

where $s_i(t)$ represents the displacement of the $i$-th truck at time t, $v_i(t)$ is the speed of the $i$-th truck at time t, $T_i(t)$ is the actual torque of the $i$-th truck at time t, $\mathrm{\Delta }t$ is the sampling time, ${\eta }_i$ is the mechanical transmission efficiency of the vehicle, $R_i$ is the wheel radius of the vehicle, $C_D$ is the air drag coefficient, $\rho$ is the air density, $A_i$ is the frontal area of the vehicle, $m_i$ is the mass of the vehicle, $g$ is the gravitational acceleration, $f_i$ is the rolling resistance coefficient, ${\mu }_i$ is the longitudinal dynamic time delay constant, and ${\widehat{T}}_i(t)$ is the desired torque.

This model uses the vehicle’s position, speed, and torque as state variables, represented by $x_i(t)={\left[s_i(t),v_i(t),T_i(t)\right]}^T$. The desired torque is used as the control variable, represented by $u_i(t)={\widehat{T}}_i(t)$, with the constraint conditions for the control variable being $T_{\min }\le u_i(t)\le T_{\max }$. Therefore, Equation (1) can be simplified to:

$$\left\{\begin{array}{l}{x}_i(t+1)=A_i(x_i(t))+B_i\cdot u_i(t)\\ y_i(t)=C{x}_i(t)\end{array}\right.$$

(2)

where $A_i=\left[\begin{array}{c}{s}_i(t)+v_i(t)\mathrm{\Delta }t\\ v_i(t)+{\displaystyle \frac{\mathrm{\Delta }t}{m_i}}\left({\displaystyle \frac{{\eta }_i}{R_i}}{T}_i(t)-{\displaystyle \frac{C_D\rho A_i}{2}}{v}_i^2(t)-m_{i}g{f}_i\right)\\ T_i(t)-{\displaystyle \frac{1}{{\mu }_i}}{T}_i(t)\mathrm{\Delta }t\end{array}\right]$, $B_i=\left[\begin{array}{c}0\\ 0\\ {\displaystyle \frac{1}{{\mu }_i}}\mathrm{\Delta }t\end{array}\right]$, $C=\left[\begin{array}{ccc}\begin{array}{l}1\\ 0\end{array}& \begin{array}{l}0\\ 1\end{array}& \begin{array}{l}0\\ 0\end{array}\end{array}\right]$.

Let the state vector of the vehicle be denoted as $X(t)$, the output vector as $Y(t)$, and the input vector as $U(t)$:

$$\left\{\begin{array}{l}X(t)={\left[x_1^T,x_2^T,\dots ,x_N^T\right]}^T\\ Y(t)={\left[y_1^T,y_2^T,\dots ,y_N^T\right]}^T\\ U(t)={\left[u_1^T,u_2^T,\dots ,u_N^T\right]}^T\end{array}\right.$$

(3)

The discrete nonlinear dynamic model of the entire vehicle platoon is as follows:

$$\left\{\begin{array}{l}X(t+1)=A(X(t))+B\cdot U(t)\\ Y(t)=CX(t)\end{array}\right.$$

(4)

To enforce the desired inter-vehicle spacing, the desired position of the vehicle is defined as:

$$s_{i,des}(t)=s_0(t)-i{d}_0$$

(5)

where $d_0$ represents the desired distance between adjacent vehicles.

The speed tracking error of the vehicles within the platoon is:

$$e_{v,i}(t)=v_i(t)-v_0(t)$$

(6)

The inter-vehicle distance tracking error is:

$$e_{s,i}(t)=s_i(t)-s_{i,des}(t)$$

(7)

For safe and stable platooning, the vehicle speed and inter-vehicle spacing are required to satisfy:

$$\left\{\begin{array}{l}\underset{t\to \infty }{\lim }{e}_{v,i}(t)=0\\ \underset{t\to \infty }{\lim }{e}_{s,i}(t)=0\end{array}\right.$$

(8)

2.2. Plug-in Hybrid Electric Truck Powertrain Model

This study considers a single-axle parallel plug-in hybrid electric truck, whose powertrain architecture is illustrated in Figure 2. The system comprises an engine, traction motor, battery pack, transmission, and supervisory controller. Key vehicle parameters and powertrain specifications are summarized in

Table 1. Powertrain parameters of the plug-in hybrid electric truck [45].

Category	Parameters	Symbol	Values	Unit
Vehicle body	Gross vehicle mass	$m$	18,000	kg
Vehicle body	Frontal area	$A$	5.1	m2
Vehicle body	Aerodynamic drag coefficient	$C_D$	0.527	-
Traction motor	Peak power	$P_{m,max}$	158.3	kW
Traction motor	Peak torque	$T_{m,max}$	293	N·m
Traction motor	Maximum speed	$n_{m,max}$	12,000	rpm
Engine	Peak power	$P_{e,max}$	169.1	kW
Engine	Peak torque	$T_{e,max}$	734	N·m
Engine	Nominal speed	$n_{e,nom}$	2200	rpm
Battery pack	Nominal voltage	$U_{bat}$	560.28	V
Battery pack	Capacity	$Q_{bat}$	5	Ah
Battery pack	Nominal power	$P_{bat,nom}$	78.4	kW

.

Based on the longitudinal dynamics of the vehicle, the driving resistance of the vehicle is given by:

$$\left\{\begin{array}{l}{F}_r=F_f+F_w+F_g+F_a\\ F_f=mgf\cos \alpha \\ F_w={\displaystyle \frac{C_{D}A{u}^2}{21.15}}\\ F_g=mg\sin \alpha \\ F_a=\delta m{\displaystyle \frac{du}{dt}}\end{array}\right.$$

(9)

where $F_f$ represents the rolling resistance, $F_w$ is the air resistance, $F_g$ is the gradient resistance, $F_a$ is the inertial resistance, $f$ is the rolling resistance coefficient, $\alpha$ is the road gradient, $C_D$ is the air resistance coefficient, $A$ is the frontal area, $\delta$ is the rotational mass conversion factor, $m$ is the vehicle mass, $g$ is the gravitational acceleration, and $u$ is the vehicle speed.

The required torque of the vehicle is given by:

$$T_{req}=\left\{\begin{array}{l}{\displaystyle \frac{F_{r}R}{i\eta }},F_r>0\\ {\displaystyle \frac{F_{r}R\eta }{i}},F_r\le 0\end{array}\right.$$

(10)

where R is the wheel radius, $i$ is the vehicle’s gear ratio, and $\eta$ is the mechanical transmission efficiency.

The required rotational speed of the vehicle is given by:

$$n_{req}={\displaystyle \frac{ui}{R}}$$

(11)

The required power of the vehicle is given by:

$$P_{req}=T_{req}{n}_{req}=P_e+P_m$$

(12)

where $P_e$ is the engine power and $P_m$ is the motor power.

The fuel consumption map of the engine is shown in Figure 3. The map (fuel consumption rate, g/kWh) is derived from steady-state engine dynamometer bench test data and implemented as a 2-D lookup table with interpolation in simulation. Based on the engine’s characteristic curve, the fuel consumption rate of the engine can be expressed as:

$$b_e=f(n_e,T_e)$$

(13)

where $n_e$ is the engine speed and $T_e$ is the engine torque.

The instantaneous fuel consumption of the engine can be expressed as:

$$\left\{\begin{array}{l}{\dot{m}}_{fuel}={\displaystyle \frac{P_e{b}_e}{3600}}\\ P_e=T_e{n}_e/9.55\end{array}\right.$$

(14)

The efficiency map of the motor is shown in Figure 4. The efficiency of the motor can be expressed as:

$${\eta }_m=f(n_m,T_m)$$

(15)

where $n_m$ is the motor speed and $T_m$ is the motor torque.

The relationship between the motor power and the battery power can be expressed as:

$$P_m={\displaystyle \frac{T_m{n}_m}{9.55}}=\left\{\begin{array}{l}{P}_{bat}\cdot {\eta }_m,\mathrm{Motor}\mathrm{mode}\\ P_{bat}/{\eta }_m,\mathrm{Generator}\mathrm{mode}\end{array}\right.$$

(16)

where $P_{bat}$ represents the battery power.

An equivalent internal-resistance model is adopted for the battery, which can be written as:

$$I_{bat}={\displaystyle \frac{V_{oc}-\sqrt{V_{oc}^2-4R_{bat}{P}_{bat}}}{2R_{bat}}}$$

(17)

$$SOC(k)=SO{C}_0-{\displaystyle \frac{{\displaystyle {\int }_0^k{I}_{bat}(t)dt}}{Q_{bat}}}$$

(18)

where $I_{bat}$ represents the battery current, $V_{oc}$ is the open-circuit voltage of the battery, $R_{bat}$ is the internal resistance of the battery, $SOC(k)$ is the state of charge at the current time, $SO{C}_0$ is the state of charge at the initial time, and $Q_{bat}$ is the battery capacity.

3. Cooperative Optimization Strategy for Adaptive Cruise Control and Energy Management

3.1. Energy Management Framework

Figure 5 presents the proposed cooperative co-optimization framework for the PHET platoon. The upper layer employs a DMPC-based cooperative adaptive cruise control scheme to generate platoon speed profiles, with objectives emphasizing stability and ride comfort to support safe and efficient operation. The lower layer adopts a DSAC–based energy management policy, which performs power-split optimization using the upper-layer planned vehicle states, thereby improving the platoon’s fuel economy. In this hierarchical scheme, the interaction between the two layers follows a one-way coupling. The DMPC layer outputs a planned speed trajectory over the prediction horizon, which determines the corresponding traction power demand profile. The DSAC layer then performs power-split decisions under this demand profile and the powertrain constraints to optimize fuel economy. In the current implementation, the DSAC outputs are not fed back to the DMPC layer for replanning; thus, the coupling effect is mainly reflected through the DMPC-shaped driving demand that DSAC responds to. Moreover, to verify the effectiveness of the proposed DSAC-based energy management module, DP and DDPG are adopted as benchmark methods for comparison. The proposed coordination is distributed rather than centralized. The DMPC layer relies on local onboard measurements and limited information exchange with neighboring vehicles, avoiding global data aggregation by a central coordinator. The DSAC-based energy management is executed locally on each vehicle based on the planned motion information and local powertrain states.

3.2. Cooperative Adaptive Cruise Control Model Based on Distributed Model Predictive Control

In this paper, DMPC [46] is used for platoon speed planning by decomposing the global optimization problem into a set of coupled local subproblems, one for each vehicle. A predecessor-following communication topology is used to facilitate information exchange between adjacent vehicles, enabling the optimization problem for each vehicle in the platoon to be solved.

The communication topology of the platoon is represented by a graph, $G=\left\{V,E\right\}$, where $V=\left\{0,1,2,\dots ,N\right\}$, $E\subseteq V\times V$. $G$ can be simplified as follows: $A$ is the adjacency matrix, $L$ is the Laplacian matrix, and $P$ is the fixed matrix.

$A$ is used to represent the communication direction between vehicles within the platoon, such that:

$$A=\left[a_{i,j}\right]\in R^{N\times N}$$

(19)

where $a_{i,j}=\left\{\begin{array}{l}1,\mathrm{if}\left\{j,i\right\}\in E\\ 0,\mathrm{if}\left\{j,i\right\}\notin E\end{array}\right.$.

The Laplacian matrix $L$ can be expressed as:

$$L=D-A$$

(20)

where the in-degree matrix $D$ is given by $D=\mathrm{diag}\left\{{\mathrm{deg}}_1,{\mathrm{deg}}_2,\dots ,{\mathrm{deg}}_N\right\}$.

The pinning matrix $P$ can be represented as:

$$P=\mathrm{diag}\left\{p_1,p_2,\dots ,p_N\right\}$$

(21)

where $p_i=\left\{\begin{array}{l}1,\mathrm{if}\left\{0,i\right\}\in E\\ 0,\mathrm{if}\left\{0,i\right\}\notin E\end{array}\right.$.

The set of information received by vehicle $i$ from the leading vehicle is represented as:

$$P_i=\left\{\begin{array}{l}0,\mathrm{if}{p}_i=1\\ \varnothing ,\mathrm{if}{p}_i=0\end{array}\right.$$

(22)

The set of information received by vehicle $i$ from the neighboring vehicle $j$ is represented as:

$$N_i=\left\{j\left|a_{i,j}=1,j\in N\right.\right\}$$

(23)

The set of information received by vehicle $i$ from all vehicles in the platoon is represented as:

$$I_i=N_i\cup P_i$$

(24)

The distributed model predictive control (DMPC) method aims to achieve a globally optimal objective through local optimization of each subsystem. Each subsystem minimizes the local tracking error within the prediction horizon $\left[t,t+N_p\right]$, while also considering the optimization of driving comfort and local performance. Therefore, a multi-objective optimization function is designed in this paper as follows:

$$\begin{array}{l}{J}_{1,i}(k\left|t\right.)={‖Q_i(y_i^p(k\left|t\right.)-y_{i,des}(k\left|t\right.))‖}_2\\ J_{2,i}(k\left|t\right.)={\displaystyle \sum _{j\in F_i}{‖G_i(y_i^p(k\left|t\right.)-y_j^a(k\left|t\right.)-d_{j,i})‖}_2}\\ J_{3,i}(k\left|t\right.)={‖R_i(u_i^p(k\left|t\right.)-h_i(v_i^p(k\left|t\right.)))‖}_2\\ J_{4,i}(k\left|t\right.)={‖F_i(y_i^p(k\left|t\right.)-y_i^a(k\left|t\right.))‖}_2\end{array}$$

(25)

where $J_{1,i}$ and $J_{2,i}$ are the tracking error cost functions of the vehicles, $J_{3,i}$ is the driving comfort cost function, $J_{4,i}$ is the communication stability cost function between vehicles, $Q_i$ is the error coefficient matrix between the following vehicle and the leading vehicle, $G_i$ is the error coefficient matrix between the following vehicle and its neighboring vehicles, $R_i$ is the driving comfort coefficient matrix, $F_i$ is the communication stability coefficient matrix between vehicles, $d_{j,i}$ is the deviation of the following distance between adjacent vehicles, $y_i^p(k\left|t\right.)$ is the predicted output sequence of the vehicle, $y_{i,des}(k\left|t\right.)$ is the desired state sequence of the leading vehicle, and $y_i^a(k\left|t\right.)$ is the hypothesized output sequence of the vehicle.

The optimization problem of the $i$-th vehicle in the platoon at time $t$ can be expressed as:

$$\begin{array}{l}\underset{U_i}{\min }{J}_i(y_i^p(:\left|t\right.),u_i^p(:\left|t\right.),y_i^a(:\left|t\right.),y_{-i}^a(:\left|t\right.))\\ ={\displaystyle \sum _{k=0}^{N_{p-1}}(J_{1,i}(k\left|t\right.)+J_{2,i}(k\left|t\right.)+J_{3,i}(k\left|t\right.)+J_{4,i}(k\left|t\right.))}\\ st:{x^{\prime }}_i^p(k+1\left|t\right.)=A_i(x_i^p(k\left|t\right.))+B_i{u}_i^p(k\left|t\right.)\\ y_i^p(k\left|t\right.)=C_i{x}_i^p(k\left|t\right.)\\ x_i^p(0\left|t\right.)=x_i(t)\\ T_{\min }\le u_i^p(k\left|t\right.)\le T_{\max }\\ y_i^p(N_p\left|t\right.)={\displaystyle \frac{1}{\left|{\mathrm{II}}_i\right|}}{\displaystyle \sum _{j\in {\mathrm{II}}_i}(y_j^a(N_p\left|t\right.)-d_{j,i})}\\ T_i^p(N_p\left|t\right.)=h_i(v_i^p(N_p\left|t\right.))\end{array}$$

(26)

where $U_i={\left[u_i^p(0\left|t\right.),u_i^p(1\left|t\right.),\dots ,u_i^p(N_p-1\left|t\right.)\right]}^T$ represents the control sequence to be optimized, and $d_{j,i}={\left[-(j-i)d_0,0\right]}^T$ is the distance deviation between adjacent vehicles.

3.3. Energy Management Strategy Based on Distributional Soft Actor-Critic

3.3.1. DSAC Algorithm

DSAC [47] combines distributional reinforcement learning with the soft actor–critic framework in an off-policy manner, leading to more stable learning and enhanced exploration. DSAC uses distributional Bellman update to estimate the distribution of the Q-function, rather than solving it through a single expected value. This allows DSAC to model the distribution of Q-values, thus better handling noise and uncertainty in the environment. By combining entropy regularization with distributed Q-function estimation, DSAC improves its exploration ability, enabling superior performance in complex environments.

In this work, DSAC is selected mainly considering the characteristics of the PHET power-split problem, which involves a continuous action space, strongly time-varying power demand, and potentially noisy/uncertain dynamics induced by different driving cycles and operating constraints. Compared with widely used alternatives, DSAC offers several practical advantages in this context. First, PPO is an on-policy method, which typically requires substantially more interaction samples to reach comparable performance, making it less suitable for training energy management policies that rely on extensive environment rollouts. Second, TD3 and conventional SAC are expectation-based value methods, whereas DSAC explicitly learns the return (Q-value) distribution, which can provide more robust value estimation under fluctuating rewards and constraint-induced penalties that are common in energy management tasks. Consequently, DSAC tends to exhibit improved training stability and faster convergence. Finally, although the fuel economy improvement over a strong off-policy baseline such as DDPG may appear modest in some cases, DSAC remains attractive due to its higher convergence efficiency and learning stability, which are important for reliable training and evaluation across different driving conditions.

Figure 6 shows the DSAC architecture. DSAC consists of a distributional critic network and an actor (policy) network. The critic predicts the return distribution for each state–action pair rather than a single expected value, while the actor outputs actions conditioned on the current state. During training, DSAC samples transitions from the replay buffer and updates the critic to better approximate the target return distribution, after which the actor is optimized based on the critic’s estimates. These updated value estimates are subsequently used to update the policy network, improving action selection. DSAC is suitable for reinforcement learning tasks with continuous action spaces, where traditional Q-learning methods struggle to provide effective solutions. DSAC, on the other hand, offers a more stable and efficient solution.

(1)

Distributed reinforcement learning frameworks

Within the maximum-entropy framework, DSAC maximizes the expected cumulative reward with an entropy regularizer to promote exploration. The resulting objective is:

$$J_{\pi }=E_{\begin{array}{l}(s_i\ge t,a_i\ge t)~{\rho }_{\pi }\\ r_i\ge t~R(\cdot \left|s_i,\right.a_i)\end{array}}\left[{\displaystyle \sum _{i=t}^{\infty }{\gamma }^{i-t}\left[r_i+\alpha H(\pi (\cdot \left|s_i\right.))\right]}\right]$$

(27)

where ${\rho }_{\pi }$ represents the distribution of the state-action pair $(s,a)$ under the policy $\pi$, $\gamma$ is the discount factor, $\alpha$ is the coefficient of entropy, $H$ is the entropy of the policy $\pi$, and $r_i(s_i,a_i)$ is the reward at each step.

The soft Q-value function under policy $\pi$ is defined as:

$$Q^{\pi }(s_t,a_t)=E_{r~R(\cdot \left|s_t\right.,a_t)}\left[r\right]+\gamma E_{\pi }\left[G_{t+1}\right]$$

(28)

where $G_t={\displaystyle {\sum }_{i=t}^{\infty }{\gamma }^{i-t}}\left[r_i-\alpha \log \pi (a_i\left|s_i\right.)\right]$.

The optimal policy is obtained via soft policy iteration, which alternates between soft policy evaluation and soft policy improvement. In this procedure, the value function is updated using the soft Bellman operator $T^{\pi }$:

$$T^{\pi }{Q}^{\pi }(s,a)=E_{r~R(\cdot \left|s,a\right.)}\left[r\right]+\gamma E_{s^{\prime }~p,a^{\prime }~\pi }\left[Q^{\pi }(s^{\prime },a^{\prime })-\alpha \log \pi (a^{\prime }\left|s^{\prime }\right.)\right]$$

(29)

To improve the performance of the new policy ${\pi }_{new}$, we update the policy by maximizing the entropy-augmented objective:

$${\pi }_{new}=\arg \underset{\pi }{\max }{J}_{\pi }=\arg \underset{\pi }{\max }{E}_{s~{\rho }_{\pi },a~\pi }\left[Q^{{\pi }_{old}}(s,a)-\alpha \log \pi (a\left|s\right.)\right]$$

(30)

Distributed soft policy models the distribution of the soft return, which is defined as:

$$Z^{\pi }(s_t,a_t)=r_t+\gamma G_{t+1}{|}_{(s_i>t,a_i>t)~{\rho }_{\pi }{r}_{i\ge t}~R(\cdot |s_i,a_i)}$$

(31)

The expected state-action return function is given by:

$$Q^{\pi }(s,a)=E\left[Z^{\pi }(s,a)\right]$$

(32)

The soft state-action return distribution is defined as: ${Z^{\prime }}^{\pi }(Z^{\pi }(s,a)|s,a):S\times A\to P(Z^{\pi }(s,a))$. A distributional soft policy evaluation scheme is adopted to derive the optimal policy. Under the maximum-entropy formulation, the corresponding distributional Bellman operator is:

$$T_D^{\pi }{Z}^{\pi }(s,a)\stackrel{D}{=}r+\gamma (Z^{\pi }(s^{\prime },a^{\prime })-\alpha \log \pi (a^{\prime }|s^{\prime }))$$

(33)

In the above equation, both sides represent distributions, and $A\stackrel{D}{=}B$ indicates that the two distributions have the same probabilistic form. The strategy iteration that combines maximum entropy with distributional return is referred to as distributional soft policy iteration (DSPI), and DSPI can converge the policy to the optimal policy.

(2)

Principle of reducing overestimation

A quantitative analysis of the overestimation error in the return distribution learning is performed. For convenience, assume the entropy weight coefficient $\alpha =0$. The greedy objective is defined as $y=E\left[r\right]+\gamma E_{s^{\prime }}\left[{\max }_{a^{\prime }}{Q}_{\theta }(s^{\prime },a^{\prime })\right]$, and the Q-estimate $Q_{\theta }(s,a)$ can be updated by minimizing the loss function ${(y-Q_{\theta }(s,a))}^2/2$, with the parameters $\theta$ defined as:

$${\theta }_{new}=\theta +\beta (y-Q_{\theta }(s,a)){\nabla }_{\theta }{Q}_{\theta }(s,a)$$

(34)

where $\beta$ is the learning rate.

Since there is some error between the updated Q estimate $Q_{\theta }$ and the true Q value $\tilde{Q}(s,a)$, we assume:

$$Q_{\theta }(s,a)=\tilde{Q}(s,a)+{\epsilon }_Q$$

(35)

where ${\epsilon }_Q$ is the random error.

The parameters updated according to the true objective $\tilde{y}$ are:

$${\theta }_{true}=\theta +\beta (\tilde{y}-Q_{\theta }(s,a)){\nabla }_{\theta }{Q}_{\theta }(s,a)$$

(36)

where $\tilde{y}=E\left[r\right]+\gamma E_{s^{\prime }}\left[{\max }_{a^{\prime }}\tilde{Q}(s^{\prime },a^{\prime })\right]$.

The error in the updated Q estimate $Q_{{\theta }_{new}}(s,a)$ is given by:

$$\mathrm{\Delta }(s,a)=E_{\epsilon Q}\left[Q_{{\theta }_{new}}(s,a)-Q_{{\theta }_{true}}(s,a)\right]=\beta \gamma \delta {‖{\nabla }_{\theta }{Q}_{\theta }(s,a)‖}_2^2$$

(37)

where $\delta =E_{{\epsilon }^{\prime }Q}\left[E_{s^{\prime }}\left[\underset{a^{\prime }}{\max Q(s^{\prime },a^{\prime })}\right]\right]-E_{s^{\prime }}\left[\underset{a^{\prime }}{\max \tilde{Q}(s^{\prime },a^{\prime })}\right]\ge 0$, which means $\mathrm{\Delta }(s,a)\ge 0$. This indicates that $\mathrm{\Delta }(s,a)$ represents an upward bias. Although the single-step upward bias is small, it can accumulate and grow over time through temporal difference (TD) learning, leading to overestimation error.

To analyze the overestimation, we introduce the distributional return. Assume that the return follows a Gaussian distribution $Z^{\prime }(\cdot |s,a)$, where the mean and standard deviation are represented by two independent functions $Q_{\theta }(s,a)$ and ${\sigma }_{\psi }(s,a)$, with $\theta$ and $\psi$ being the parameters of these functions, i.e., ${Z^{\prime }}_{\theta ,\psi }(\cdot |s,a)=N(Q_{\theta }(s,a),{\sigma }_{\psi }{(s,a)}^2)$. Similarly, assume that the target distribution is ${Z^{\prime }}^{\mathrm{target}}(\cdot |s,a)=N(y,{\sigma }^{{\mathrm{target}}^2})$. Therefore, the overestimation error of $Q_{{\theta }_{new}}(s,a)$ is given by:

$${\mathrm{\Delta }}_D(s,a)\approx {\displaystyle \frac{\mathrm{\Delta }(s,a)}{{\sigma }_{\psi }{(s,a)}^2}}$$

(38)

From the above equation, it can be seen that the overestimation error ${\mathrm{\Delta }}_D(s,a)$ is inversely proportional to the square of ${\sigma }_{\psi }(s,a)$, while ${\sigma }_{\psi }(s,a)$ is directly proportional to the system’s uncertainty. As the randomness of the return function and the return distribution increases, ${\sigma }_{\psi }(s,a)$ becomes larger. Therefore, by learning distributional returns, the randomness of the task can be reduced, thereby lowering the overestimation.

(3)

Design of the DSAC algorithm

In the DSAC algorithm, both the distributional value function ${Z^{\prime }}_{\theta }(\cdot |s,a)$ and the policy function ${\pi }_{\theta }(\cdot |s)$ follow Gaussian distributions, with parameters $\theta$ and $\varphi$, respectively. Their means and variances are computed by neural networks. During the policy evaluation phase, we choose the Kullback–Leibler (KL) divergence as the metric for the distributional distance, and optimize the return distribution corresponding to the current policy:

$$J_{Z^{\prime }}(\theta )=-\underset{(s,a,r,s^{\prime })~B}{E}\left[\log P(T_D^{{\pi }_{{\varphi }^{\prime }}}Z(s,a)|{Z^{\prime }}_{\theta }(\cdot |s,a))\right]$$

(39)

The gradient of parameter $\theta$ is given by:

$${\nabla }_{\theta }{J}_{Z^{\prime }}(\theta )=-\underset{(s,a,r,s^{\prime })~B}{E}\left[{\nabla }_{\theta }\log P(T_D^{{\pi }_{{\varphi }^{\prime }}}Z(s,a)|{Z^{\prime }}_{\theta }(\cdot |s,a))\right]$$

(40)

where ${Z^{\prime }}_{\theta }$ is a Gaussian model, ${Z^{\prime }}_{\theta }(\cdot |s,a)=N(Q_{\theta }(s,a),{\sigma }_{\theta }{(s,a)}^2)$.

From the above gradient formula, it can be observed that when the standard deviation of the distribution ${\sigma }_{\theta }(s,a)\to 0$, the gradient ${\nabla }_{\theta }{J}_{Z^{\prime }}(\theta )$ will explode, and when ${\sigma }_{\theta }(s,a)\to \infty$, gradient vanishing occurs. Therefore, clipping techniques are employed to address this issue by limiting the standard deviation within a reasonable range to prevent large changes in the target distribution that could cause instability in learning. We constrain the target distribution to stay near the expectation of the current return distribution:

$${\sigma }_{\theta }(s,a)=\mathrm{clip}({\sigma }_{\theta }(s,a),{\sigma }_{\min },{\sigma }_{\max })$$

(41)

$$\stackrel{\_\_\_\_\_\_\_\_\_\_\_\_\_\_}{T_D^{{\pi }_{{\varphi }^{\prime }}}Z(s,a)}=\mathrm{clip}(T_D^{{\pi }_{{\varphi }^{\prime }}}Z(s,a),Q_{\theta }(s,a)-b,Q_{\theta }(s,a)+b)$$

(42)

where b is the clipping boundary.

The target network is updated using slow-moving updates:

$${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime },{\varphi }^{\prime }\leftarrow \tau \varphi +(1-\tau ){\varphi }^{\prime }$$

(43)

In the policy improvement phase, we can learn the policy by maximizing the soft Q-value:

$$J_{\pi }(\varphi )=E_{s~B,a~{\pi }_{\varphi }}\left[Q_{\theta }(s,a)-\alpha \log ({\pi }_{\varphi }(a|s))\right]$$

(44)

To reduce the gradient estimation variance, we use the reparameterization trick to compute the gradient of the policy. Since $Q_{\theta }(s,a)$ can be explicitly parameterized by $\theta$, we convert the action a into a deterministic variable to simplify the process:

$$a=f_{\varphi }({\xi }_a;s)=a_{\mathrm{mean}}+{\xi }_a\odot a_{\mathrm{std}}$$

(45)

where ${\xi }_a$ is an auxiliary variable, $a_{\mathrm{mean}}$ is the mean of ${\pi }_{\varphi }(\cdot |s)$, $a_{\mathrm{std}}$ is the standard deviation of ${\pi }_{\varphi }(\cdot |s)$, and $\odot$ denotes the Hadamard product.

The gradient of the policy update is given by:

$${\nabla }_{\varphi }{J}_{\pi }(\varphi )=E_{s~B,{\xi }_a}\left[-\alpha {\nabla }_{\varphi }\log ({\pi }_{\varphi }(a|s))+({\nabla }_a{Q}_{\theta }(s,a)-\alpha {\nabla }_a\log ({\pi }_{\varphi }(a|s))){\nabla }_{\varphi }{f}_{\varphi }({\xi }_a;s)\right]$$

(46)

The entropy coefficient $\alpha$ is updated as follows:

$$J(\alpha )=E_{(s,a)~B}\left[\alpha (-\log {\pi }_{\varphi }(a|s)-\overline{H})\right]$$

(47)

The pseudocode of the DSAC algorithm is shown in Algorithm 1.

Algorithm 1. DSAC algorithm

Initialize parameters $\theta$, $\varphi$, $\alpha$ Initialize target parameters ${\theta }^{\prime }\leftarrow \theta$, ${\varphi }^{\prime }\leftarrow \varphi$ Initialize learning rate ${\beta }_{Z^{\prime }}$, ${\beta }_{\pi }$, ${\beta }_{\alpha }$, $\tau$ Initialize iteration index $k=0$ For $k=0\to k_{\max }$ do  Select action $a~{\pi }_{\varphi }(a|s)$  Observe reward $r$ and new state $s^{\prime }$  Store transition tuple $(s,a,r,s^{\prime })$ in buffer B  Sample N transitions $(s,a,r,s^{\prime })$ from B  Update soft return distribution $\theta \leftarrow \theta -{\beta }_{Z^{\prime }}{\nabla }_{\theta }{J}_{Z^{\prime }}(\theta )$   if k mod m then   Update policy $\varphi \leftarrow \varphi +{\beta }_{\pi }{\nabla }_{\varphi }{J}_{\pi }(\varphi )$   Adjust temperature $\alpha \leftarrow \alpha -{\beta }_{\alpha }{\nabla }_{\alpha }J(\alpha )$   Update target networks ${\theta }^{\prime }\leftarrow \tau \theta +(1+\tau ){\theta }^{\prime }$, ${\varphi }^{\prime }\leftarrow \tau \varphi +(1-\tau ){\varphi }^{\prime }$   end if end for

3.3.2. Energy Management Strategy Based on DSAC

This section presents a DSAC-based energy management strategy for platooned hybrid electric trucks to optimize power allocation. At each control step, the agent observes the current vehicle state, and the DSAC policy outputs a control action that is applied to the powertrain. In this study, the state vector comprises the battery SOC, vehicle speed, and longitudinal acceleration:

$$s=\left\{v,a_c,SOC\right\}$$

(48)

In this study, torque compensation control is implemented for the motor torque using the vehicle’s required torque and the engine’s output torque [48]. Accordingly, the engine is taken as the control input, and the DSAC action is defined as the engine power:

$$a=\left\{P_e\right\}$$

(49)

To enhance platoon energy efficiency, the DSAC agent is trained using a reward that penalizes both fuel and electricity consumption. The reward is defined as:

$$r=\left\{\alpha \cdot m_{fuel}+\beta \cdot P_{bat}/3600\right\}$$

(50)

where $\alpha$ represents the diesel fuel price [49], $\alpha =3.8 \mathrm{CNY}/\mathrm{L}$, and $\beta$ represents the electricity price, $\beta =0.8 \mathrm{CNY}/\mathrm{kWh}$.

4. Results and Discussion

4.1. Parameter Settings

The proposed hierarchical framework is implemented and simulated in Python. For the upper layer, a DMPC-based CACC platoon model is built with six PHETs, including one leader and five followers. The parameters of the truck platoon are shown in

Table 2. Simulation parameters for the plug-in hybrid electric truck platoon.

Parameters	Value
Number of follower vehicles	5
Desired inter-vehicle distance	20 m
Prediction horizon	7
Cost function weights ${\mathrm{Q}}_i$/${\mathrm{G}}_i$/${\mathrm{R}}_i$/${\mathrm{F}}_i$	10/5/10/8

. For the lower layer, the DSAC-based energy management strategy is employed, and the hyperparameters of the DSAC algorithm are shown in

Table 3. Hyperparameters of the DSAC algorithm.

Hyperparameters	Value
Optimizer	Adam
Number of hidden layers	5
Number of hidden units per layer	256
Batch size	256
Value learning rate	0.0001
Policy learning rate	0.0001

. All simulations were performed in Python on a laptop equipped with an Intel Core i5-1240P CPU, 16 GB RAM, and Intel Iris(R) Xe Graphics, using Python 3.14.

In all simulations, regenerative braking is enabled. When the traction power demand becomes negative during deceleration, the motor operates in generator mode to recover braking energy and charge the battery within the motor/generator capability and battery charging power constraints. A plug-in charging period is not considered in the constructed drive profile, which represents the driving stage only. The initial battery SOC is set to 0.8, the SOH is assumed to be 100% (kept constant during the simulation), and the ambient temperature is fixed at 25 °C (room temperature).

4.2. Driving Condition Data Collection

In order to make the vehicle’s driving conditions more aligned with local actual conditions and thus improve the accuracy of the control strategy, this study uses an onboard vehicle CAN-bus data recorder to collect driving condition data on a road section in Liuzhou. A total of 18,812 data points were collected. The data collection equipment is shown in Figure 7. The data recorder recorded 18 signals, including time, vehicle speed, motor voltage, motor current, etc. The collected CAN signals were replayed and exported using TSMaster (v2023.8.30.958) software. Figure 8 shows part of the driving conditions collected in Liuzhou during the experiment.

4.3. Analysis of Vehicle Following Performance

The vehicle-following capability of the DMPC platoon model is evaluated using two representative driving cycles, namely CHTC and the Liuzhou city cycle. The characteristics of the driving conditions are shown in

Table 4. Characteristics of driving conditions.

Driving Cycle	Time (s)	Distance (km)	Maximum Speed (m/s)	Maximum Acceleration (m/s2)	Average Speed (m/s)
CHTC	1800	23.22	24.44	0.81	12.90
Liuzhou	1923	20.03	15.54	0.83	10.41

. The CHTC is a national standard cycle used to evaluate the energy consumption of commercial vehicles, with an average speed of 12.90 m/s and a maximum speed of 24.44 m/s. The Liuzhou cycle represents the actual driving conditions in Liuzhou city, with an average speed of 10.41 m/s and a maximum speed of 15.54 m/s.

Figure 9 presents the platoon simulation results under the CHTC driving cycle. Figure 9a compares the speed profiles of the leader and follower trucks. As shown, the following vehicles are able to closely follow the speed curve of the lead truck. To more accurately describe the following performance of the vehicles, we define the average speed deviation of the following vehicles as ${\mathsf{\delta }}_{mean}={{\displaystyle \sum }}_{i=1}^N\left|v_l-v_i\right|/N$, where $v_l$ is the lead vehicle’s speed and $v_i$ is the speed of the following vehicle.

Table 5. The average speed deviation of the following vehicle.

Driving Cycle	Following Vehicle 1	Following Vehicle 2	Following Vehicle 3	Following Vehicle 4	Following Vehicle 5
CHTC	0.167 m/s	0.186 m/s	0.207 m/s	0.212 m/s	0.213 m/s
Liuzhou	0.114 m/s	0.127 m/s	0.141 m/s	0.145 m/s	0.147 m/s

presents the average speed deviations of the following vehicles. From Table 5, the first follower exhibits the smallest mean speed deviation (0.167 m/s), whereas the fifth follower shows the largest (0.213 m/s). Overall, the mean speed deviation increases progressively with the follower index. This is due to the cumulative effect of speed deviations, which becomes larger as more vehicles follow, resulting in a greater average speed deviation for the following vehicles.

Figure 9b shows the position curves of all vehicles. From the figure, it can be seen that each curve is arranged in sequence without any intersection, indicating that the following vehicles did not collide during the driving process, demonstrating good safety. Figure 9c illustrates the position error curves of the following vehicles. As shown, the position error of the following vehicles remains within a range of $\pm$2 m, maintaining a reasonable inter-vehicle distance.

Table 6. The average position error of the following vehicle.

Driving Cycle	Following Vehicle 1	Following Vehicle 2	Following Vehicle 3	Following Vehicle 4	Following Vehicle 5
CHTC	0.032 m	0.089 m	0.214 m	0.257 m	0.339 m
Liuzhou	0.033 m	0.093 m	0.218 m	0.256 m	0.337 m

presents the average position errors of the following vehicles. The first following vehicle has the smallest average position error, at 0.032 m, while the fifth following vehicle has the largest average position error, at 0.339 m. The average position error of the following vehicles increases with the number of following vehicles, which is due to the accumulation of position errors from preceding vehicles during the driving process, leading to larger average position errors. Figure 9d shows the acceleration curves of all vehicles. As can be seen, the vehicle accelerations are within a $\pm 1\mathrm{m}/{\mathrm{s}}^2$ range, and the maximum acceleration of the following vehicles is smaller than that of the lead vehicle, further improving the comfort of the vehicles.

To assess the robustness of the DMPC-based CACC model, simulations were performed using the Liuzhou driving cycle, with results shown in Figure 10. Figure 10a plots the vehicle speed profiles, indicating that all followers track the leader effectively. Table 5 summarizes the mean speed deviations. All followers exhibit deviations below 0.15 m/s, indicating good tracking performance. Figure 10b plots the vehicle position trajectories, showing that the followers remain uniformly spaced behind the leader while maintaining safe inter-vehicle distances.

Figure 10c shows the position error trajectories. The followers’ position errors remain within ±2 m, comparable to those under the CHTC, indicating the high stability of the DMPC-based CACC model. Figure 10d plots the vehicle acceleration profiles. The accelerations remain within ±1 m/s², consistent with the results obtained under the CHTC. Through the analysis of the control effects under both the CHTC and Liuzhou driving cycles, we found that the cooperative adaptive cruise control model based on DMPC has good robustness, can solve for different driving conditions, and yields good simulation results with regard to vehicle following, maintaining inter-vehicle distance, and driving comfort.

4.4. Fuel Economy Analysis

For benchmarking, we additionally implemented DP and DDPG energy management strategies to compare against the proposed DSAC-based method. We analyze the simulation results for the first following vehicle. Figure 11 shows the SOC curve for the first following vehicle. Figure 11a,b displays the SOC curves for the CHTC and Liuzhou driving cycles, respectively. The red, green, and blue curves represent the SOC values for DP, DDPG, and DSAC, respectively. The variation in the SOC value reflects the vehicle’s electricity consumption during operation. In Figure 11, the initial SOC value is 0.8, and the final value is around 0.35 for all methods. The SOC trajectories for DDPG and DSAC are quite similar, while the trajectory for DP shows a significant difference. This is because DP is a global optimal control method, whereas DDPG and DSAC are local optimal control methods, resulting in a noticeable deviation in the final SOC compared to DP.

Table 7. Fuel consumption of different control strategies under the CHTC.

Method	SOC Final Value	Fuel Consumption (L/100 km)	Fuel Economy (%)
DP	0.350	12.356	100
DDPG	0.359	13.686	90.28
DSAC	0.346	13.427	92.02

shows the fuel consumption simulation results under the CHTC for different energy management strategies. Among them, DP has the lowest fuel consumption of 12.356 L/100 km, followed by DDPG with 13.686 L/100 km, and DSAC with 13.427 L/100 km. Compared to DP’s fuel consumption, DDPG achieves 90.28% of DP’s fuel economy, while DSAC achieves 92.02%. This indicates that DSAC has a higher fuel economy than DDPG. However, the fuel economy of DSAC is only 1.74% better than DDPG, as both energy management strategies are deep reinforcement learning methods developed based on actor-critic network architectures, and DSAC is an optimization of the DDPG algorithm.

Table 8. Fuel consumption of different control strategies under the Liuzhou driving cycle.

Method	SOC Final Value	Fuel Consumption (L/100 km)	Fuel Economy (%)
DP	0.350	8.647	100
DDPG	0.348	9.516	90.87
DSAC	0.353	9.295	93.03

shows the fuel consumption under the Liuzhou driving cycle, where DDPG achieves 90.87% fuel economy, and DSAC achieves 93.03%. The results demonstrate that the energy management strategy based on DSAC offers better fuel economy than DDPG and exhibits stronger robustness, adapting well to different driving conditions.

To provide a clearer comparison of fuel use, we examine the engine operating points. Figure 12 illustrates their distributions under the CHTC and Liuzhou driving cycles, shown in Figure 12a and Figure 12b, respectively.

The DP solution places most operating points in the high-efficiency band. By comparison, the DDPG- and DSAC-based controllers exhibit a wider spread that includes low-efficiency regions. Compared with DDPG, DSAC places a larger share of engine operating points in the high-efficiency region, which translates into better fuel economy.

In deep reinforcement learning algorithms, the average reward is an important metric for evaluating the completion of training. Figure 13 plots the average reward curves for different algorithms; Figure 13a corresponds to the CHTC. As shown, DDPG converges around episode 72, whereas DSAC converges earlier at approximately episode 55, indicating faster convergence for DSAC. Figure 13b shows the results under the Liuzhou driving cycle. DDPG converges at about episode 74, whereas DSAC converges earlier at approximately episode 53. Combined with the CHTC results, these simulations confirm that DSAC achieves faster convergence than DDPG. This is because the DSAC algorithm incorporates the return distribution function into maximum entropy reinforcement learning, and by directly learning the continuous return distribution, it addresses the issues of gradient explosion and vanishing gradients, thus improving the convergence speed.

In this study, the training data for the reinforcement learning agents are generated through online interaction with the simulation environment; that is, transition samples are collected while the agent operates the powertrain under the given driving cycle. The average reward curves in Figure 13 are used to assess training progress and convergence. For transparency and reproducibility, it should be noted that the training process is conducted for a fixed number of 500 episodes for both algorithms and both driving cycles. Convergence is identified when the average reward curve becomes stable and shows no further noticeable improvement, indicating that the learned policy has reached a steady performance level. This setup makes the learning process and stopping criterion explicit and repeatable.

Given the platoon-level focus of this study, the fuel economy of each vehicle is further examined.

Table 9. Fuel consumption of vehicles in the platoon.

Driving Cycle	Performance Index	FV1	FV2	FV3	FV4	FV5
CHTC	Fuel consumption (L/100 km)	13.427	13.503	13.419	13.437	13.398
Liuzhou	Fuel consumption (L/100 km)	9.295	9.224	9.343	9.251	9.304

reports the fuel consumption of all platoon members under the DSAC-based EM strategy. Under the CHTC, fuel consumption varies only slightly, from 13.398 to 13.503 L/100 km, and the five followers exhibit nearly identical values. This consistency is mainly attributed to their similar speed profiles, which lead to comparable energy demand and thus similar fuel use. Figure 14 presents the SOC trajectories of the five followers. The curves almost overlap, indicating similar battery energy usage across the platoon. Overall, these results demonstrate that the proposed DSAC-based EM strategy is well-suited for platooning and can effectively enhance the fuel economy of platoon driving.

4.5. Limitations of Simulation-Only Verification and Practical Considerations

Although the proposed energy management strategy demonstrates consistent improvements across the presented simulation cases, it is important to clarify that the validation in this study is conducted in a model-in-the-loop (MIL) simulation environment. The driving cycles are constructed from real-world CAN-bus recordings, which helps ensure that the longitudinal speed demand and operating profiles are representative of practical truck usage. However, using CAN-bus data to build driving cycles does not constitute an experimental validation of the proposed control strategy itself. Therefore, the practical credibility of the reported quantitative gains should be interpreted within the scope and assumptions of the adopted models and simulation settings.

(1)

Model fidelity and unmodeled dynamics.

The powertrain and battery models employed in this paper, while widely adopted for energy management research, inevitably simplify certain high-fidelity behaviors. For example, thermal dynamics, transient actuator behaviors, drivetrain backlash, and nonlinearities associated with real hardware may influence fuel consumption, battery current trajectories, and aging-related indicators in real operation. Moreover, battery life estimation models typically rely on a subset of aging mechanisms and may not capture all degradation pathways under varying temperatures, current ripple, and long-term calendar aging effects. These simplifications may lead to discrepancies between simulated and real-world performance.

(2)

Real-time implementation constraints.

The proposed strategy involves optimization and/or learning-based decision-making, whose practical deployment requires consideration of real-time computational limits (e.g., ECU/industrial PC constraints), solver convergence behavior under strict timing, and robustness to numerical issues. In real vehicles, additional constraints such as torque rate limits, actuator saturation, and controller scheduling may affect the achievable performance compared with offline simulation.

(3)

Communication, sensing, and uncertainty factors.

In practical settings, sensor noise, estimation errors (e.g., SOC estimation), and disturbances such as vehicle mass variations, road grade uncertainty, and aerodynamic changes can impact control outcomes. If the strategy relies on information exchange (e.g., for platooning/cooperative control), communication latency, packet loss, and time synchronization errors may further degrade performance. These factors are not fully represented in the current MIL simulations.

(4)

Implications and future validation.

Given the above limitations, the results presented in this paper should be viewed as demonstrating the potential effectiveness of the proposed framework under the assumed modeling conditions. As future work, we will (i) conduct robustness tests incorporating uncertainty, noise, and delay effects, (ii) perform cross-validation using higher-fidelity co-simulation platforms/models, and (iii) implement hardware-in-the-loop (HIL) and/or vehicle-level experiments to validate real-time feasibility and practical performance under realistic constraints.

4.6. Practical Implementation Issues: Communication Imperfections, Real-Time Computation, and Scalability

While the proposed DMPC–DSAC framework shows promising performance in the presented model-in-the-loop simulations, several practical implementation factors are not explicitly modeled and may affect real-world deployment. This subsection discusses key issues, including communication delays and packet loss in V2V/V2I networks, real-time computational burden, and scalability to larger platoons, and outlines corresponding research directions.

4.6.1. Communication Delays and Packet Loss in V2V/V2I Networks

In real platooning scenarios, the information exchanged among vehicles (e.g., states, predicted trajectories, or control intentions) may suffer from time-varying communication delays and intermittent packet loss. Such imperfections can lead to outdated or missing information, which may degrade tracking performance and potentially compromise platoon stability if not properly addressed. The current study does not explicitly incorporate these network effects; thus, the reported performance should be interpreted under an idealized communication assumption.

Future work will investigate delay-/loss-aware cooperative control by: (i) incorporating time-stamped information and delay compensation (prediction-based alignment) within DMPC; (ii) formulating robust or stochastic DMPC to account for bounded delays and probabilistic packet drops; and (iii) adopting event-triggered or asynchronous coordination mechanisms to reduce reliance on high-rate, lossless communications while maintaining safety constraints.

4.6.2. Real-Time Computational Burden of DMPC and DSAC

Practical deployment requires the controller to meet strict timing constraints on embedded hardware. DMPC may involve iterative optimization and coordination among multiple vehicles, and its computational cost can increase with prediction horizon length, constraint complexity, and coupling strength among agents. In addition, DSAC introduces learning-based decision components whose training is typically performed offline; however, online execution still requires real-time inference and integration with DMPC. The current manuscript focuses on algorithmic effectiveness in simulation and does not provide a detailed real-time feasibility study (e.g., worst-case execution time under a fixed sampling period).

As future research, we will report runtime statistics under representative sampling rates and hardware configurations, and explore computationally efficient implementations, such as warm-starting, reduced-order prediction models, limited-iteration DMPC, parallel/distributed solvers, and policy distillation, where a trained policy approximates the optimization-based controller for fast online execution while preserving safety constraints.

4.6.3. Scalability to Larger Platoons

As platoon size increases, the communication load, coordination complexity, and computational requirements of multi-agent optimization can grow significantly. Moreover, ensuring robust platoon behavior (e.g., string stability and constraint satisfaction) becomes more challenging under heterogeneous vehicle dynamics and uncertain road conditions. The current study evaluates the proposed framework in a limited-size platoon setting; therefore, scalability to larger platoons is not fully demonstrated.

To enhance scalability, future work will investigate hierarchical or clustered coordination architectures, where local sub-platoons are controlled with neighborhood interactions and higher-level coordination ensures global objectives. Additionally, we will conduct systematic scalability studies by increasing platoon size and reporting performance–complexity trade-offs (e.g., energy savings, battery-life metrics, tracking errors, and computation/communication overhead) to characterize deployment limits and guide practical parameter selection.

5. Conclusions

This paper presents a collaborative optimization framework for cooperative adaptive cruise control and energy management of plug-in hybrid electric truck platoons by integrating distributed model predictive control (DMPC) with a distributed soft actor-critic (DSAC) approach. The main findings are summarized as follows:

(1)

A hierarchical cooperative control structure is developed, where the upper-layer DMPC generates the platoon speed trajectory to improve car-following smoothness and ride comfort, and the lower-layer DSAC allocates power based on the planned motion information to enhance fuel economy.

(2)

Under the China heavy-duty commercial vehicle test cycle (CHTC) and the Liuzhou city driving cycle, the DMPC-based cruise control achieves reliable following and spacing performance, maintaining the position error within ±2 m and acceleration within ±1 m/s², indicating good robustness and driving comfort.

(3)

The DSAC-based energy management strategy demonstrates favorable fuel economy performance, achieving 92.02% fuel efficiency on the CHTC and 93.03% on the Liuzhou cycle, and outperforming the DDPG-based method in the reported comparisons.

(4)

The training curves show that DSAC converges faster than DDPG under both driving cycles, indicating improved learning efficiency for the studied energy management task.

The limitations of this work are summarized as follows:

(1)

The framework is evaluated through simulation-based studies, and further verification under more realistic implementation conditions is still needed.

(2)

Some deployment-related factors (e.g., communication imperfections, real-time computation constraints, and larger platoon scalability) are not fully covered in the current evaluation.

(3)

The current hierarchical interaction follows a feed-forward structure, and the motion planning layer is not adjusted online based on the energy management outcomes.

Future work will focus on:

(1)

Conducting more realistic validation and implementation-oriented testing to further assess real-time feasibility and practical performance.

(2)

Enhancing robustness under practical uncertainties and extending evaluations to larger platoons and more complex traffic scenarios.

(3)

Exploring tighter coordination mechanisms between motion planning and energy management to further improve overall performance.