Cooperative Control and Energy Management for Autonomous Hybrid Electric Vehicles Using Machine Learning

Abstract

The growing deployment of connected and autonomous vehicles (CAVs) requires coordinated control strategies that jointly address safety, mobility, and energy efficiency. This paper presents a novel two-stage cooperative control framework for autonomous hybrid electric vehicle (HEV) platoons based on machine learning. In the first stage, a metric learning-based distributed model predictive control (ML-DMPC) strategy is proposed to enable cooperative longitudinal control among heterogeneous vehicles, explicitly incorporating inter-vehicle interactions to improve speed tracking, ride comfort, and platoon-level energy efficiency. In the second stage, a multi-agent twin-delayed deep deterministic policy gradient (MATD3) algorithm is developed for real-time energy management, achieving an optimal power split between the engine and battery while reducing Q-value overestimation and accelerating learning convergence. Simulation results across multiple standard driving cycles demonstrate that the proposed framework outperforms conventional distributed model predictive control (DMPC) and multi-agent deep deterministic policy gradient (MADDPG)-based methods in fuel economy, stability, and convergence speed, while maintaining battery state of charge (SOC) within safe limits. To facilitate future experimental validation, a dSPACE-based hardware-in-the-loop (HIL) architecture is designed to enable real-time deployment and testing of the proposed control framework.

1. Introduction

Advances in vehicle onboard sensing and intelligent control technologies offer significant potential to improve the efficiency of connected and autonomous vehicles [1,2]. This has led to extensive research on decentralized cooperative control for multiple vehicles and multi-agent systems (MASs), with a focus on leveraging traffic information and inter-vehicle interactions [3]. At the same time, there is increasing attention to enhancing energy management systems (EMSs) within hybrid electric vehicle (HEV) platoons to design an EMS for each vehicle that maximizes overall system efficiency [4,5,6]. In multi-vehicle energy management, determining the optimal speed for each vehicle requires accounting for the velocity of the vehicle ahead. This information is then shared with the following vehicles via vehicle-to-vehicle (V2V) communication, and the power split is optimized to improve each vehicle’s fuel economy [7].

Modern advanced driving assistance systems (ADAS) incorporate adaptive cruise control (ACC) to enhance control and energy efficiency in CAVs. A proportional-integral (PI) controller-based ACC was proposed to control heavy trucks [8]. The longitudinal control exhibited favorable characteristics, and the simulation results demonstrated improved fuel economy and pollution control performance. Dynamic programming (DP), based on a proportional-integral-derivative (PID) controller framework, generated an ACC reference trajectory, reducing fuel consumption by allowing larger tracking errors [9]. To improve driving comfort and coordination between the engine and motor, ref. [10] introduced a group-preserving scheme-based control strategy. Similarly, ref. [11] proposed a collaborative optimization approach for plug-in HEVs that combines a Bayesian traffic recognition model with an equivalent consumption minimization strategy (ECMS) to reduce overall energy consumption. This method demonstrates notable improvements in reducing frictional losses and conserving energy. Nevertheless, key driving objectives, such as tracking performance, ride comfort, and fuel efficiency, often conflict. Therefore, careful consideration of these objectives is essential, which naturally aligns with a model predictive control (MPC) framework for vehicle energy management.

An MPC framework that combines predictive and receding-horizon optimization is widely employed to optimize vehicle energy management at each sampling interval. A novel energy management scheme is formulated by incorporating a machine-learning (ML)-based driving-style recognition model into the MPC [12]. Simulation results show that the proposed model reduces fuel cost compared with the charge-depleting and charge-sustaining (CD-CS) method. The power management strategies in [13,14] employ an MPC-based cooperative adaptive cruise control (CACC) with two predictive layers, demonstrating fuel savings through platoon formation and cooperative energy management relative to single-vehicle-based EMS. The potential of MPC is further explored in [15], which addresses the mixed-vehicle platoon control problem to improve traffic efficiency. The applied distributed MPC (DMPC) strategy, using a neural network (NN)-based data-driven approach, exhibits superior performance, with lower computational cost and faster convergence than a centralized algorithm. In [16], a multi-objective DMPC scheme based on a new $\gamma$-string stability criterion with bidirectional topology is realized. Cooperative control achieves better energy efficiency than real-time optimization (RTO) under variable-speed references. The DMPC problem for CAVs is formulated with coupled safety and inter-vehicle distance constraints and solved cooperatively to achieve superior platoon control objectives [17,18]. The effectiveness of DMPC is influenced by predictive model accuracy, prediction-horizon length, and the algorithm’s optimization capability. Hence, current EMS methods for HEVs primarily focus on “single-stage & two-level” vehicle control and power management. However, advanced vehicle control and energy management systems should optimally distribute power under transient and stochastic conditions, with support from intelligent transportation systems (ITS).

Machine learning (ML), a branch of artificial intelligence (AI), is rapidly advancing and has significant applications in autonomous vehicles [19,20]. Integrating intelligent transportation systems (ITS) with reinforcement learning (RL) and deep reinforcement learning (DRL) enables the development of adaptive and optimized solutions. RL has been applied to HEV energy management [21,22], demonstrating effective control of battery charging/discharging and improved fuel efficiency compared with conventional methods. Deep Q-learning (DQN) has been used for autonomous power-split control in PHEVs, achieving greater fuel savings than rule-based approaches [23]. In [24], a deep deterministic policy gradient (DDPG)-based EMS was developed by embedding prior powertrain knowledge, demonstrating superior fuel economy compared with DQN. To further enhance RL/DRL performance, ref. [25] proposed hybrid experience replay (HER) with twin-delayed DDPG (TD3), achieving improved fuel optimality and faster convergence than traditional methods.

The RL and DRL methods discussed above are single-agent approaches. While single-agent DRL methods may struggle to learn efficient policies, multi-agent DRL (MADRL) methods [26] have attracted greater attention. In a multi-agent setting, multiple agents interact with one another and the environment at each time step, and each agent learns and makes decisions concurrently. Each agent’s decisions affect the environment and, consequently, influence the learning process of other agents. The MADRL approach was developed to enhance collaborative control among multiple autonomous vehicles at unsignalized intersections [27]. This strategy offers strong adaptability for transferring knowledge and reusing information from agents’ policies to handle unfamiliar traffic scenarios. An integrated optimization framework based on multi-agent settings for vehicle control and powertrain management demonstrated reduced fuel consumption compared with a separate optimization method [28].

Several studies have explored hierarchical approaches to improve platoon control and energy management, including a data-driven RL-based EMS [29], multi-step predictive models [30,31], and a two-term EMS strategy [32]. Integrating transportation data with big data analysis has also shown potential to enhance HEV fuel economy. However, V2I connectivity requires a more detailed vehicle description, thereby increasing the state space. Since EMS algorithms must process large-scale data, their complexity often imposes heavy computational demands in real applications. To address this, the present work introduces a metric learning-based platoon control combined with a fast multi-agent DRL framework to achieve low computational cost, improved fuel efficiency, and better energy efficiency. Specifically, a double-centralized, critics-based MADRL method within a “two-stage & two-level” EMS is proposed, where the upper-level controller determines optimal acceleration/deceleration to avoid collisions, and the lower-level controller optimizes the power split between the engine and the battery.

In summary, prior studies reveal several limitations. This paper addresses them through three main objectives:

(i)

Optimizing the vehicle speed profile with respect to safety, comfort, and energy consumption, and comparing two distinct DMPC strategies;

(ii)

Optimizing vehicle power management based on the derived optimal speed profile, using the centralized training and decentralized execution (CTDE) framework within a MADRL approach;

(iii)

Reducing overestimation bias in Q-value estimation for multi-agent DRL by applying the MATD3 algorithm to HEV energy management.

The paper is structured as follows: Section 2 presents the nonlinear vehicle platoon and dynamic system models; Section 3 describes the control framework for the ML-DMPC controller and the MATD3-based EMS; Section 4 reports the optimization results with comparative analysis and Section 5 proposes an HIL experiment; and Section 6 concludes the work.

2. Modeling of Vehicle Platoon and HEV

This paper considers a heterogeneous vehicle platoon comprising $N+1$ vehicles (nodes): a leading vehicle (LV, denoted as 0) and N following vehicles (FVs, labeled from 1 to N) traveling on a flat road, as depicted in Figure 1. A predecessor-leader following (PLF) topology is adopted for V2V communication to ensure efficient information exchange [33]. The problem is formulated as a decentralized leader-following task, in which multiple vehicles track the leader while maintaining the desired inter-vehicle spacing. The target vehicle type is a hybrid electric vehicle with a uniform powertrain architecture.

Figure 1. Vehicle platoon communication framework using PLF topology. Solid arrows denote vehicle-following interactions, while dashed arrows indicate V2V communication links.

2.1. HEV Powertrain Configuration

In this research, we adopted an HEV with a powertrain comprising a battery, an auxiliary power unit (APU), and a drive motor, as depicted in Figure 2. The APU comprises an internal combustion engine (ICE) along with an integrated starter and generator (ISG). It serves as an assistive energy source under dynamic, high-power-demand conditions to improve the efficiency of the hybrid energy storage system.

Figure 2. The structure and configuration of HEV. Black arrows indicate mechanical power flow, while blue arrows indicate electrical power flow.

2.2. Nonlinear Modeling of Vehicle Dynamics

For powertrain analysis, this work adopts an inverse (backward) simulation framework within the nonlinear vehicle dynamics model. In designing the controller algorithm, both modeling complexity and accuracy are considered, leading to the formulation of a nonlinear discrete-time model for each following vehicle ${\left(\mathrm{FV}\right)}_j\in \mathcal{N}$ in the platoon, as expressed in Equation (1) [34]:

$$\begin{array}{cc}\hfill x_{cj}(k+1)& =x_{cj}\left(k\right)+u_{cj}\left(k\right)\delta k,\hfill \\ \hfill u_{cj}(k+1)& =u_{cj}\left(k\right)+{\displaystyle \frac{1}{m_j}}\left({\displaystyle \frac{{\varphi }_j}{R}}{M}_j\left(k\right)-C_{ad,j}{u}_{cj}^2\left(k\right)-m_j{g}_0{C}_{rr,j}\right)\delta k,\hfill \\ \hfill M_j(k+1)& =M_j\left(k\right)+{\displaystyle \frac{1}{{\tau }_j}}\left(u_{cj}\left(k\right)-M_j\left(k\right)\right)\delta k,\hfill \end{array}$$

(1)

where

• $x_{cj}\left(k\right)$ and $u_{cj}\left(k\right)$ are the position and velocity states of vehicle j at time step k, and $\delta k$ is the discretization interval;
• ${\varphi }_j$ is the transmission efficiency;
• R is the wheel radius;
• $C_{ad,j}$ is the aerodynamic drag coefficient;
• $g_0$ is the gravitational constant;
• $m_j$ is the total vehicle mass;
• $C_{rr,j}$ is the rolling resistance coefficient;
• $M_j\left(k\right)$ is the applied driving or braking torque;
• ${\tau }_j$ is the inertial lag; and
• $u_j\left(k\right)$ is the requested torque demand.

For platoon control, the state vector is defined as

$$s_j\left(k\right)={\left[x_{cj}\left(k\right),u_{cj}\left(k\right),M_j\left(k\right)\right]}^{\top },$$

(2)

with the control input given by $u_j\left(k\right)=M_j\left(k\right)$, and the output vector defined as

$$y_j\left(k\right)={\left[x_{cj}\left(k\right),u_{cj}\left(k\right)\right]}^{\top }.$$

(3)

The state-space representation of Equation (1) is given by

$$\begin{array}{cc}\hfill s_j(k+1)& ={\mathrm{\Lambda }}_j^{\prime }\left(s_j\left(k\right)\right)+{\mathrm{\Psi }}_j^{\prime }{u}_j\left(k\right),\hfill \\ \hfill y_j\left(k\right)& ={\mathrm{\Omega }}_j^{\prime }{s}_j\left(k\right),\hfill \end{array}$$

(4)

where ${\mathrm{\Lambda }}_j^{\prime }$, ${\mathrm{\Psi }}_j^{\prime }$, and ${\mathrm{\Omega }}_j^{\prime }$ are the system matrices:

$${\mathrm{\Lambda }}_j^{\prime }={\left[x_{cj}(k+1),u_{cj}(k+1),M_j(k+1)\right]}^{\top },{\mathrm{\Psi }}_j^{\prime }={\left[0,0,\delta k/{\tau }_j\right]}^{\top },{\mathrm{\Omega }}_j^{\prime }=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\end{array}\right].$$

(5)

The state-space representation of the platoon system is given by

$$\begin{array}{cc}\hfill S(k+1)& =\mathrm{\Lambda }S\left(k\right)+\mathrm{\Psi }U\left(k\right),\hfill \\ \hfill Y\left(k\right)& =\mathrm{\Omega }S\left(k\right),\hfill \end{array}$$

(6)

where $S\left(k\right)$, $U\left(k\right)$, and $Y\left(k\right)$ represent the state, input, and output vectors defined for all nodes in the platoon. The corresponding matrices are

$$\mathrm{\Lambda }={\left[{\mathrm{\Lambda }}_1^{\top }\left(x_1\right),{\mathrm{\Lambda }}_2^{\top }\left(x_2\right),\dots ,{\mathrm{\Lambda }}_N^{\top }\left(x_N\right)\right]}^{\top },$$

(7)

$$\mathrm{\Psi }=diag\left[{\mathrm{\Psi }}_1,{\mathrm{\Psi }}_2,\dots ,{\mathrm{\Psi }}_N\right],$$

(8)

$$\mathrm{\Omega }=diag\left[{\mathrm{\Omega }}_1,{\mathrm{\Omega }}_2,\dots ,{\mathrm{\Omega }}_N\right].$$

(9)

2.3. Powertrain System Model

The internal combustion engine (ICE) characteristic map and the battery model are detailed below.

2.3.1. ICE Model

The ICE model utilizes the brake-specific fuel consumption (BSFC) map. The focus is on adhering to the optimal BSFC curve rather than conducting a global search across the map, which is beneficial for the energy management system to achieve improved fuel economy. The BSFC metric quantifies the engine fuel consumption $F\left(t\right)$ via the BSFC surface, typically obtained from experimental measurements, available at https://github.com/lryz0612/DRL-Energy-Management/blob/master/Eng_bsfc_map.mat (accessed on 20 March 2025). The fuel consumption $F\left(t\right)$ is determined by interpolation from tabulated BSFC data as a function of engine torque $T_e$ and speed $n_e$ as follows [35]:

$$F\left(t\right)=\left\{\begin{array}{cc}\mathrm{LuT}\left(T_e,n_e\right),\hfill & n_e>n_{\mathrm{idle}},\hfill \\ 0,\hfill & n_e\le n_{\mathrm{idle}},\hfill \end{array}\right.$$

(10)

where $\mathrm{LuT}$ is the lookup table. It is a 2D dataset derived from engine test measurements, and $n_{\mathrm{idle}}$ corresponds to the engine speed at idle.

2.3.2. Battery Model

In this work, the battery cell is modeled using a simplified open-circuit voltage ($V_{\mathrm{oc}}\left(t\right)$) in series with an internal resistance $R_{\mathrm{in}}$, as illustrated in Figure 3. This model provides an effective trade-off between accuracy and computational efficiency, making it suitable for energy management and real-time control applications.

Figure 3. Equivalent circuit of battery cell.

Applying Kirchhoff’s voltage law (KVL) to the equivalent circuit, the battery terminal voltage is expressed as

$$V_{\mathrm{b}}\left(t\right)=V_{\mathrm{oc}}\left(t\right)-I_{\mathrm{b}}\left(t\right)R_{\mathrm{in}}.$$

(11)

The instantaneous battery output power is defined as

$$P_{\mathrm{b}}\left(t\right)=V_{\mathrm{b}}\left(t\right)I_{\mathrm{b}}\left(t\right).$$

(12)

Substituting the terminal voltage Equation (11) into the power expression yields

$$P_{\mathrm{b}}\left(t\right)=\left(V_{\mathrm{oc}}\left(t\right)-I_{\mathrm{b}}\left(t\right)R_{\mathrm{in}}\right)I_{\mathrm{b}}\left(t\right),$$

(13)

which expands to

$$P_{\mathrm{b}}\left(t\right)=V_{\mathrm{oc}}\left(t\right)I_{\mathrm{b}}\left(t\right)-R_{\mathrm{in}}{I}_{\mathrm{b}}^2\left(t\right).$$

(14)

This power balance can be rewritten as

$$P_{\mathrm{b}}\left(t\right)=P_{\mathrm{useful}}\left(t\right)-P_{\mathrm{loss}}\left(t\right),$$

(15)

where

$$P_{\mathrm{useful}}\left(t\right)=V_{\mathrm{oc}}\left(t\right)I_{\mathrm{b}}\left(t\right),P_{\mathrm{loss}}\left(t\right)=R_{\mathrm{in}}{I}_{\mathrm{b}}^2\left(t\right).$$

(16)

Rearranging the power expression results in a quadratic equation in $I_{\mathrm{b}}\left(t\right)$:

$$R_{\mathrm{in}}{I}_{\mathrm{b}}^2\left(t\right)-V_{\mathrm{oc}}\left(t\right)I_{\mathrm{b}}\left(t\right)+P_{\mathrm{b}}\left(t\right)=0.$$

(17)

Solving for the battery current gives

$$I_{\mathrm{b}}\left(t\right)={\displaystyle \frac{V_{\mathrm{oc}}\left(t\right)-\sqrt{V_{\mathrm{oc}}^2\left(t\right)-4R_{\mathrm{in}}{P}_{\mathrm{b}}\left(t\right)}}{2R_{\mathrm{in}}}}.$$

(18)

The State-of-Charge (SOC) dynamics follow the Coulomb counting method:

$${\displaystyle \frac{d\mathrm{SOC}\left(t\right)}{dt}}=-{\displaystyle \frac{I_{\mathrm{b}}\left(t\right)}{Q_{\mathrm{nom}}}},$$

(19)

and integrating over time yields

$$\mathrm{SOC}\left(t\right)=\mathrm{SOC}\left(0\right)-{\displaystyle \frac{1}{Q_{\mathrm{nom}}}}{\int }_0^t{I}_{\mathrm{b}}\left(t\right)dt.$$

(20)

Here, $P_{\mathrm{b}}$ is the battery output power, and $V_{\mathrm{oc}}\left(t\right)$ and $I_{\mathrm{b}}\left(t\right)$ denote the open-circuit voltage and battery current, respectively. The internal resistance switches based on current direction:

$$R_{\mathrm{in}}=\left\{\begin{array}{cc}{R}_{\mathrm{dis}},\hfill & I_{\mathrm{b}}\left(t\right)>0\hfill \\ R_{\mathrm{chg}},\hfill & I_{\mathrm{b}}\left(t\right)<0.\hfill \end{array}\right.$$

(21)

$Q_{\mathrm{nom}}$ is the nominal battery capacity in ampere-hours (Ah). $\mathrm{SOC}\left(0\right)$ is the initial SOC. Detailed battery characteristics are available in the referenced GitHub source at https://github.com/lryz0612/DRL-Energy-Management/blob/master/Mot_eta_quarter.mat (accessed on 20 March 2025). For analysis, external effects such as temperature variations and battery aging are not considered in this model.

The SOC operating range of $[0.4,0.85]$ is selected based on literature findings and experimental evidence from [24,35]. Operating the battery outside this window increases internal resistance, induces voltage instability, and accelerates degradation. Test-bench experiments reported in these studies confirm that maintaining SOC within this range ensures stable terminal voltage, reduces resistance variation, and improves energy efficiency under representative HEV driving conditions.

The motor power consumption of vehicle node j at time step k over the total time steps t is calculated as follows [35]:

$$P_{\mathrm{mot},j}(k\mid t)=f\left(T_{\mathrm{mot},j},u_{c,j}\right)=\left\{\begin{array}{cc}{\displaystyle \frac{T_{\mathrm{mot},j}(k\mid t)u_{c,j}(k\mid t)}{R{\eta }_j{\eta }_{\mathrm{mot}}}},\hfill & T_{\mathrm{mot},j}(k\mid t)\ge 0,\hfill \\ {\displaystyle \frac{T_{\mathrm{mot},j}(k\mid t)u_{c,j}(k\mid t)}{R{\eta }_j{\eta }_{\mathrm{brk}}}},\hfill & T_{\mathrm{mot},j}(k\mid t)<0,\hfill \end{array}\right.$$

(22)

where

• $T_{\mathrm{mot},j}$, $u_{cj}$, R, and ${\eta }_j$ are the torque, velocity, wheel radius, and conversion efficiency of vehicle node j, respectively;
• ${\eta }_{\mathrm{mot}}$ and ${\eta }_{\mathrm{brk}}$ represent the efficiency of the motor drive and the braking efficiency, respectively.

3. Methodology

The details of the proposed ML-DMPC control and MATD3 methodology are outlined in this section.

3.1. Design of ML-DMPC Controller

Metric learning is an essential machine learning (ML) framework for deriving a distance or similarity function that effectively captures relationships among data points in high-dimensional spaces [36]. In this work, we introduce a distributed metric learning approach [37] that leverages the alternating direction method of multipliers (ADMM) [38]. This technique addresses the global metric learning task by enabling each node to solve it using only local data with minimal information sharing, thereby alleviating computational overhead. Using the ADMM scheme, the distributed convex optimization problem is solved iteratively, and convergence is ensured under mild assumptions [39]. To the best of our knowledge, this study is the first to extend metric learning to distributed model predictive control (DMPC) for connected and autonomous vehicle platooning and energy management, using ADMM as the optimization backbone. This section outlines the development of a CACC based on an ML-DMPC-driven DMPC controller for heterogeneous platoons. In the ML-DMPC controller setup, multiple CAVs are linked through an information structure, and the global control objective is achieved through coordinated control of each vehicle. Figure 4 illustrates the flow of control-variable transfer in CACC based on ML-DMPC.

Figure 4. Control variable transfer flow. Solid arrows represent local interactions, while dashed arrows represent V2V communication.

The multi-objective cost function for each vehicle node j is defined as [40]

$$\underset{U_j^p\left(k\right)}{min}{\mathrm{CF}}_j\left(y_j^{\mathrm{pre}},u_j^{\mathrm{pre}},y_j^{\mathrm{ass}},y_{-j}^{\mathrm{ass}}\right),$$

(23)

where $y_j^{\mathrm{pre}}$, $u_j^{\mathrm{pre}}$, $y_j^{\mathrm{ass}}$, and $y_{-j}^{\mathrm{ass}}$ denote the predicted output, predicted control input, assumed output, and the assumed output from neighboring vehicles for node j, respectively.

The objective function in Equation (24) is computed by summing the open-loop local cost functions as follows [41]:

$$\begin{array}{cc}\hfill {\mathrm{CF}}_j=\sum _{k=0}^{N_p-1}(& ∥\left(y_j^{\mathrm{pre}}(k\mid t)-y_{\mathrm{des},j}(k\mid t)\right){\mathrm{\Theta }}_j∥+∥\left(y_j^{\mathrm{pre}}(k\mid t)-y_j^{\mathrm{ass}}(k\mid t)-d_{j,i}\right){\mathrm{\Gamma }}_j∥\hfill \\ \hfill & +∥\left(u_j^{\mathrm{pre}}(k\mid t)-h_j\left(u_{c,j}^{\mathrm{pre}}\right)\right){\mathrm{\Xi }}_j∥+∥\left(y_j^{\mathrm{pre}}(k\mid t)-y_j^{\mathrm{ass}}(k\mid t)\right){\mathrm{\Sigma }}_j∥)\hfill \\ \hfill & +∥{{\rm Y}}_j{P}_j(k\mid t)∥,\hfill \end{array}$$

(24)

where $d_{j,i}$ is the inter-vehicle spacing between nodes j and i.

The matrices ${\mathrm{\Theta }}_j$, ${\mathrm{\Gamma }}_j$, ${\mathrm{\Xi }}_j$, and ${\mathrm{\Sigma }}_j$ are symmetric and positive definite and are chosen to penalize deviations from the desired performance:

• ${\mathrm{\Theta }}_j$ penalizes LV–FV interaction errors;
• ${\mathrm{\Gamma }}_j$ penalizes deviations from neighboring-vehicle trajectories;
• ${\mathrm{\Xi }}_j$ regulates acceleration and deceleration magnitudes;
• ${\mathrm{\Sigma }}_j$ maintains FV communication stability.

The matrix ${\mathrm{{\rm Y}}}_j$ serves as the weighting factor for energy consumption. The procedure of the ML-DMPC algorithm is summarized in Algorithm 1.

3.2. Formulation of MATD3-Based EMS for HEV Platoon

This section models the HEV platoon as a multi-agent system (MAS), in which multiple agents cooperate to achieve a shared objective, thereby improving optimization efficiency [42]. To train the MADRL framework, the actor-critic policy gradient method is employed within the CTDE paradigm. In this setup, the critic is provided with supplementary knowledge about other agents’ policies, whereas the actor relies solely on its local observations. After the training phase, only the decentralized actors are used for execution, with each operating independently. The cumulative expected return for an agent $j\in \mathcal{N}$ is formulated as

$$J\left(w_j\right)={\mathbb{E}}_{a_j\sim {\pi }_j}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_{t,j}\left(s_{t,j},a_{t,j},s_{t+1,j}\right)\right].$$

(25)

The gradient of this expected return is expressed as

$${\nabla }_{w_j}J\left(w_j\right)={\mathbb{E}}_{s\sim {\rho }^{\pi },a_j\sim {\pi }_j}\left[{\nabla }_{w_j}log{\pi }_j(a_j\mid o_j)Q_j^{\pi }(x,a_1,a_2,\dots ,a_N)\right],$$

(26)

where $Q_j^{\pi }(x,a_1,a_2,\dots ,a_N)$ represents a centralized action-value function of agent j. It is determined by taking into account the actions of all agents $a_1,a_2,\dots ,a_N$ and the collection of state observations $x=(o_1,o_2,\dots ,o_N)$ from all agents.

For the centralized critic network, the centralized action-value function is defined as $Q_j^{\mu }$, and its gradient is calculated as follows:

$${\nabla }_{w_j}J\left({\mu }_j\right)={\mathbb{E}}_{x,a\sim D}{\left[{\nabla }_{w_j}{\mu }_j(a_j\mid o_j){\nabla }_{a_j}{Q}_j^{\mu }(x,a_1,a_2,\dots ,a_N)\right]}_{a_j={\mu }_j\left(o_j\right)},$$

(27)

where D represents the experience replay tuple, which contains the experiences of all agents, $(x,x^{\prime },a_1,a_2,\dots ,a_N,r_1,r_2,\dots ,r_N)$.

The centralized $Q_j^{\mu }$-function is updated as

$$\begin{array}{cc}\hfill L\left(w_j\right)& ={\mathbb{E}}_{x,a,r,x^{\prime }}\left[{\left(Q_j^{\mu }(x,a_1,a_2,\dots ,a_N)-y\right)}^2\right],\hfill \\ \hfill y& =r_j+\gamma Q_j^{{\mu }^{\prime }}(x^{\prime },a_1^{\prime },a_2^{\prime },\dots ,a_N^{\prime }){|}_{a_j^{\prime }={\mu }_j^{\prime }\left(o_j\right)},\hfill \end{array}$$

(28)

where $L\left(w_j\right)$ is the MSE loss function; $x^{\prime }$, $a_j^{\prime }$, and $Q_j^{{\mu }^{\prime }}$ are the state, action, and action-value function of the target networks, respectively; y represents the network output; and ${\mu }^{\prime }=\{{\mu }_{w_1^{\prime }},{\mu }_{w_2^{\prime }},\dots ,{\mu }_{w_N^{\prime }}\}$ represents the set of target policies with delayed weight parameters $w_1^{\prime },w_2^{\prime },\dots ,w_N^{\prime }$.

Algorithm 1 ML-DMPC

1: For each vehicle node j, define the initial state values $(x_{cj}\left(0\right),u_{cj}\left(0\right))$ and the dual variable. The assumed control input and output across the prediction horizon are initialized as $$\begin{array}{cc}\hfill u_j^{ass}(k\mid 0)& =h_j\left(u_{cj}\left(0\right)\right),\hfill \\ y_j^{ass}(k\mid 0)\hfill & \hfill =y_j^{pre}(k\mid 0),k=0,1,2,\dots ,N_p-1.\end{array}$$ $\hspace{1em}{y}_j^{pre}$ is calculated as $$\begin{array}{cc}\hfill s_j^{pre}(k+1\mid 0)& ={\mathrm{\Lambda }}_j^{\prime }\left(s_j^{pre}(k\mid 0)\right)+{\mathrm{\Psi }}_j^{\prime }{u}_j^{ass}(k\mid 0),\hfill \\ \hfill y_j^{pre}(k\mid 0)& ={\mathrm{\Omega }}_j^{\prime }{s}_j^{pre}(k\mid 0),\hfill \\ \hfill s_j^{pre}(0\mid 0)& =s_j\left(0\right).\hfill \end{array}$$ 2: For each time $t>0$, obtain the current states of all platoon vehicles. 3: For ADMM iteration $m>n$ (repeat until convergence), for each node $j=1$ to N, do the following steps: (i) Solve the local optimization using the vehicle’s current state, its assumed output, and the assumed outputs of neighbors to get the optimal control sequence: $$u_j^{\ast }(k\mid t),k=0,1,\dots ,N_p-1.$$ (ii) Compute the optimal predicted states over the horizon using the optimal inputs: $$\begin{array}{cc}\hfill s_j^{\ast }(k+1\mid t)& ={\mathrm{\Lambda }}_j^{\prime }\left(s_j^{\ast }(k\mid t)\right)+{\mathrm{\Psi }}_j^{\prime }{u}_j^{\ast }(k\mid t),\hfill \\ s_j^{\ast }(0\mid t)& =s_j\left(t\right).\hfill \end{array}$$ (iii) Update the assumed control sequence for the next time step by shifting and appending a new terminal input: $$u_j^{ass}(k\mid t+1)=\left\{\begin{array}{cc}{u}_j^{*}(k+1\mid t),\hfill & k=0,1,\dots ,N_p-2,\hfill \\ h_j\left(u_{cj}^{\ast }(N_p\mid t)\right),& k=N_p-1.\hfill \end{array}\right.$$ Using the updated assumed control sequence, propagate the plant model to form the assumed state and output over the horizon. (iv) Update the dual variable. if updated constraints $\le \epsilon$ then (v) Transmit $y_j^{ass}(k\mid t+1)$ to the other vehicles, receive $y_{(-j)}^{ass}(k\mid t+1)$ from the neighboring vehicles, and compute the desired output using the leading vehicle. (vi) Apply the first element of the optimal control sequence: $$u_j\left(t\right)=u_j^{*}(0\mid t).$$ else (vii) Increment m and repeat steps (i)–(iv). 4: Increment t and go to Step (1).   end

It is worth emphasizing that, since the MADRL framework employs deterministic policies, the centralized critic action-value function $Q_j^{\pi }$ can be approximated as $Q_j^{\pi }(x,a_1,a_2,\dots ,a_N)\approx Q_j^{\pi }(x,a_j)$. However, in multi-agent deep deterministic policy gradient (MADDPG), this approximation during Q-value updates often leads to overestimation of Q-values, particularly in the early stages of training. Such overestimation not only slows convergence but may also lead to sub-optimal policies. Furthermore, in practical applications, environments are often stochastic, which amplifies this issue by introducing additional noise from function approximation errors [43].

To overcome these shortcomings, a multi-agent variant of TD3 (MATD3) is adopted. The MATD3 algorithm extends the principles of TD3 to a multi-agent framework, aiming to improve both sample efficiency and training stability in MADDPG. This is achieved by introducing a double centralized-critic Q-learning mechanism, represented as $Q_{j,w_{1,2}}^{\pi }(x,a_1,a_2,\dots ,a_N)$ for each agent $j\in \mathcal{N}$. The target value $y_j$ of the centralized critic network computed by MATD3 is given as follows:

$$y_j=r_j+\gamma \underset{1,2}{min}{Q}_j^{\pi }\left(x^{\prime },{\mu }_{1,w_{\pi }^{\prime }}\left(s^{\prime }\right)+\epsilon ,\dots ,{\mu }_{N,w_{\pi }^{\prime }}\left(s^{\prime }\right)+\epsilon \right).$$

(29)

The complete procedure of the MATD3-based EMS algorithm is depicted in Algorithm 2.

Algorithm 2 MATD3

1: Initialize replay buffer D and network parameters. 2: For each episode $e=1,2,\dots ,M$: (i) Randomly initialize the action exploration noise. 3: Obtain the initial state variables: $$v_1,T_1,{\mathrm{SOC}}_1.$$ 4: For each time step $t=0$ to $T_{max}$: (ii) Select the actions according to the policy with noise: $$a_j\sim {\mu }_j\left(o_j\right)+ϵ.$$ (iii) Execute the joint actions, observe the reward, and store transitions in D. 5: For each agent $j=1,\dots ,N$: (iv) Randomly sample a minibatch from D and compute $y_j$ using Equation (29). (v) Minimize the critic losses for both critics using Equation (28). (vi) if$tmodd=0$then (vii) Update the actor using the policy gradient Equation (27). (viii) Update the target networks: $$\begin{array}{cc}\hfill w_Q^{\prime }& =\tau w_Q+(1-\tau )w_Q^{\prime },\hfill \\ \hfill w_{\mu }^{\prime }& \hfill =\tau w_{\mu }+(1-\tau )w_{\mu }^{\prime }.\end{array}$$                              end if                 end for           end for     end

The element settings of the energy management strategy (EMS) are given as follows:

$$\mathrm{state}\left(s_j\right)=\{v,T,SOC\},$$

(30)

$$\mathrm{action}\left(a_j\right)=\left\{P_{\mathrm{apu}}\right\},$$

(31)

$$\mathrm{reward}\left(r_j\right)=-\left\{\begin{array}{cc}F\left(t\right),\hfill & SOC\in [SO{C}_{\mathrm{low}},SO{C}_{\mathrm{high}}],\hfill \\ \lambda F\left(t\right)+\mu {\left(\mathrm{\Delta }SOC\left(t\right)\right)}^2,\hfill & \mathrm{otherwise},\hfill \\ \mathrm{\Delta }SOC\left(t\right)=SO{C}_{\mathrm{ref}}-SOC\left(t\right).\hfill \end{array}\right.$$

(32)

where

• v, T, and $\mathrm{SOC}$ represent the velocity, torque, and battery state of charge of vehicle node j, respectively;
• $P_{\mathrm{apu}}$ is the power of the auxiliary power unit;
• $\lambda$ and $\mu$ are the weight coefficients of the reward function;
• $F\left(t\right)$ is the fuel consumption at time t.

The neural network (NN) architecture of the actor-critic model employed in this study comprises three fully connected hidden layers with 128, 64, and 32 neurons, and an output layer with one neuron. All hidden layers use the rectified linear unit (ReLU) activation function. The actor-network output layer employs a sigmoid activation for action generation, whereas the critic output layer employs a linear activation.

The cooperative control in this research refers to a hierarchical, distributed control mechanism in which multiple platoon vehicles and onboard energy sources share information and jointly optimize decisions to achieve system-level objectives such as safety, stability, and energy efficiency. As illustrated in Figure 5, cooperation is realized through a two-stage control architecture consisting of platoon-level motion coordination and powertrain-level energy coordination. At the platoon level, each vehicle node j shares its predicted states and receives assumed outputs from neighboring vehicles through V2V communication, enabling anticipation of future vehicle behaviours rather than isolated trajectory tracking. This cooperative behavior is mathematically embedded in the distributed optimization problem defined in Equation (23). The corresponding cost function explicitly couples neighboring vehicles through inter-vehicle interaction terms in Equation (24) which penalize deviations from neighboring vehicles’ predicted trajectories and enforce coordinated motion. Furthermore, an ADMM-based consensus mechanism ensures that all vehicles iteratively converge to a globally consistent platoon behavior, thereby enabling cooperative longitudinal control rather than independent model-predictive control (MPC) solutions.

Figure 5. Complete architecture of two-stage control.

At the powertrain level, each HEV is modeled as an autonomous agent with state and action spaces defined in Equations (30) and (31) respectively. Cooperation among the onboard energy sources (engine and battery) is enforced through a shared reward function given in Equation (32), which promotes fuel-optimal power sharing while maintaining battery health. Inter-vehicle cooperation during the learning process is achieved using centralized critics, as formulated in Equation (26), and further stabilized through the MATD3 double-critic update in Equation (29), ensuring coordinated policy learning and reduced estimation bias. The ML-DMPC layer generates cooperative velocity and torque predictions that directly influence the energy management decisions. In response, the MATD3-based EMS coordinates power distribution among energy sources while respecting platoon-level motion objectives.

4. Simulation Results and Discussions

The proposed method is trained using the UDDS drive cycle, illustrated in Figure 6, and its performance is validated on standard driving cycles, including HWFET, NEDC, and WLTC. The study considers a heterogeneous HEV platoon with identical powertrains, and the specific parameters of the platoon vehicles are provided in Table 1.

Figure 6. UDDS training drive cycle.

Table 1. Platoon vehicle parameters.

Component	Parameter	Unit	Value
Vehicle	${\mathrm{Mass}}_{1,2,3,4,5}$	kg	1035, 1850, 1935, 1680, 1760
	$f_j$	–	0.01
	$C_d$	–	0.6
	Frontal area (A)	m2	2.23
	$d_0$	m	20
Transmission	${\tau }_j$	s	0.75
	$r_j$	m	0.376
	${\eta }_j$	–	0.95
APU	Maximum power	kW	60
	Maximum torque	Nm	120
	Maximum speed	rpm	4000
Traction motor	Maximum power	kW	50
	Maximum torque	Nm	400
	Maximum speed	rpm	6000
Battery	Capacity	kWh	1.54
	Voltage	V	237
Cost function weights	${\mathrm{\Theta }}_j/{\mathrm{\Gamma }}_j/{\mathrm{\Xi }}_j/{\mathrm{\Sigma }}_j$	–	10/5/10/8

4.1. ML-DMPC-Based Optimization Effectiveness Verification

In evaluating the optimization performance of CACC, the maximum speed deviation, denoted as ${\delta }_{max}\left(u_{c0,j}\right)=|u_{c0}-u_{cj}|$, represents the largest speed difference between the FV and the LV and serves as an indicator of speed-following performance. The maximum distance deviation, ${\delta }_{max}\left(d_{j,j+1}\right)$, is used to assess distance-keeping performance between consecutive FVs. The absolute value of the maximum acceleration, represented as $a_{max}=\left|a_j\right|$, is used to characterize driving comfort. Furthermore, the average energy consumption, $P_{\mathrm{mean}}={\displaystyle \frac{1}{N}}{\sum }_{j=1}^N{P}_j$, is employed to assess the energy-saving effect.

Figure 7 illustrates the speed-following performance under the UDDS drive cycle. The figure shows that each subsequent vehicle (FV) controlled by ML-DMPC effectively tracks the leading vehicle (LV) speed, resulting in a lower speed-following error than with the conventional DMPC controller. In Figure 8, the speed errors between the LV and FVs are presented, with the maximum speed errors for ML-DMPC and DMPC being 1.665 m/s and 1.753 m/s, respectively. The improved speed-following performance and reduced speed errors can mitigate communication delays between platoon vehicles.

Figure 7. Speed curves of (a,b) DMPC (c,d) ML-DMPC.

Figure 8. Speed error between LV and FVs.

From the acceleration curves in Figure 9, it is evident that the proposed ML-DMPC algorithm yields lower and more stable acceleration values (1.54 m/s²) compared with DMPC (1.79 m/s²). The zoomed-in plots show that the ML-DMPC-optimized profile is smoother than that of the traditional DMPC controller. This indicates that ML-DMPC improves driving comfort and vehicle energy efficiency.

Figure 9. Acceleration curves of (a,b) DMPC (c,d) ML-DMPC.

The quantitative advantages of the proposed ML-based approach over classical methods are clearly demonstrated by the percentage reductions reported across all driving cycles in Table 2. Compared to conventional MPC, the ML-DMPC strategy reduces the maximum speed deviation by approximately 36% (UDDS), 23% (HWFET), 24% (NEDC), and 31% (WLTC), with an average reduction of about 28.5% across all cycles. Compared with DMPC, the corresponding reductions are 21%, 6%, 5%, and 15%, yielding an average decrease of approximately 11.8%, confirming significantly enhanced cooperative speed-tracking performance.

Table 2. Summary of the optimization effect of three algorithms.

Driving Cycle	${\mathit{\delta }}_{max}\left({\mathit{u}}_{\mathit{c}0,\mathit{j}}\right)$ (m/s)			${\mathit{a}}_{max}$ (m/s2)			${\mathit{P}}_{\mathbf{mean}}$ (kWh)
	MPC	DMPC	ML-DMPC	MPC	DMPC	ML-DMPC	MPC	DMPC	ML-DMPC
UDDS	2.23	1.80	1.42	2.12	1.79	1.54	7.79	7.13	6.27
HWFET	1.75	1.43	1.34	1.37	0.99	0.92	4.26	3.96	3.31
NEDC	1.82	1.47	1.39	1.62	1.27	0.97	5.84	5.74	5.22
WLTC	2.11	1.71	1.45	1.97	1.68	1.52	12.04	11.49	10.73

Regarding motion smoothness, the maximum longitudinal acceleration is reduced by approximately 27% (UDDS), 33% (HWFET), 40% (NEDC), and 23% (WLTC) relative to MPC, resulting in an average reduction of approximately 30.8%. Compared with DMPC, the reductions are 14%, 7%, 24%, and 10%, corresponding to an average reduction of approximately 13.8%, indicating reduced acceleration transients and smoother longitudinal motion. Additionally, the cooperative motion improvements yield measurable energy benefits. The mean energy consumption is reduced by approximately 20% (UDDS), 22% (HWFET), 11% (NEDC), and 11% (WLTC) relative to MPC, producing an average reduction of about 16%. Compared with DMPC, the corresponding reductions are 12%, 16%, 9%, and 7%, with an average reduction of approximately 11%. These consistent reductions across diverse driving cycles validate the added value of ML-based cooperative control over classical EMS solutions and clearly demonstrate its superiority.

4.2. MATD3-Based EMS Comparative Analysis

In this scenario, DP serves as a benchmark for comparison. The evaluation compares the convergence rates and fuel optimality of the proposed MATD3-based EMS with those of the MADDPG-based EMS across various driving cycles. For algorithm implementation, the DRL framework is developed in Python (https://www.python.org/ (accessed on 5 January 2026)) using TensorFlow and Keras. The hyperparameter settings for neural network training are shown in Table 3. The algorithm is trained for 100 episodes, with each episode corresponding to a HEV driving cycle.

Table 3. Training hyper-parameters of MADRL methods.

Hyper-Parameter	Value	Hyper-Parameter	Value
Replay memory (D)	10,000	Actor learning rate ($a_{\mathrm{lr}}$)	0.001
Mini-batch size	64	Critic learning rate ($c_{\mathrm{lr}}$)	0.001
Discount rate ($\gamma$)	0.90	Target update rate ($\tau$)	0.01

Figure 10 shows the variations in the mean cumulative reward versus episode number, demonstrating learning convergence. The convergence episodes correspond to the average convergence behavior obtained over multiple independent training runs with different random seeds. The convergence points for MADDPG and MATD3 occur at 42 and 31 episodes, respectively. In the initial training phase (Episodes 1–40), both algorithms exhibit large fluctuations due to insufficient experience accumulation. As training progresses, oscillations diminish, resulting in stable reward curves. The MADDPG curve, however, continues to fluctuate beyond its convergence point, attributable to overestimation of the Q-values, as discussed in Section 3.2. Ensuring stable convergence is therefore more challenging for MADDPG than for MATD3. The proposed MATD3-based EMS achieves more effective optimization during learning.

Figure 10. Convergence curves of (a) MADDPG (b) MATD3.

Figure 11 illustrates the SOC trajectories of the EMS methods during the UDDS drive cycle. DP-based global energy management serves as a benchmark for validating optimality. When high power is required, battery electrical power alone is insufficient; thus, the APU must operate in a more efficient region to drive the vehicle and charge the battery. Differences in SOC profiles arise from the distinct optimization mechanisms of each method. A deep battery discharge occurs between 150 s and 320 s due to the large power demand. During this period, DP exhibits a higher discharge rate than multi-agent DRL methods, whereas the proposed MATD3-based EMS demonstrates effective battery utilization with APU support, outperforming MADDPG. Moreover, the SOC-maintenance capability of MATD3 closely matches that of the globally optimal DP control and achieves a higher terminal SOC than MADDPG.

Figure 11. SOC trajectories of different EMS methods.

Figure 12 presents the final SOC values of the two EMS algorithms. Both algorithms yield final SOC values near the reference value (0.60) after approximately 100 episodes, indicating effective constraint handling by the reward function. During Episodes 1–10, SOC values increase and then decline gradually. Notably, MATD3 demonstrates faster convergence to the target SOC and lower fluctuations than MADDPG.

Figure 12. SOC final of (a) MADDPG and (b) MATD3.

Table 4 compares the training performance of DP, MADDPG, and MATD3-based EMSs. DP is the most fuel-efficient because it uses global information as prior knowledge, enabling precise decision-making. MADRL methods, in contrast, rely solely on real-time observations and expertise learned from neural networks. Despite this limitation, MADRL methods offer significantly shorter computation times and are better suited to real-time applications. The table shows that the proposed MATD3 EMS consumes 8.42% less fuel than MADDPG. In addition, the algorithm requires fewer training episodes, which enhances its suitability for real-time control. The MADDPG-based EMS strategy achieves 81.8% fuel economy, whereas MATD3 achieves 91.2% of the DP benchmark on the UDDS driving cycle.

Table 4. Comparison of EMS training performance on UDDS.

Algorithm	Fuel (L/100 km)	Rel. Decrease (%)	Final SOC	Conv. Episode	Fuel Econ. (%)
DP	3.18	–	0.6130	–	100
MADDPG	3.76	–	0.6145	42	81.8
MATD3	3.46	8.42	0.6146	31	91.2

The flexibility and adaptability of the EMS are critical for real-world driving conditions. To further assess HEV performance under the proposed EMS, three representative driving scenarios, such as urban, suburban, and highway, were employed to evaluate the derived policies. Table 5 presents a comparison of different EMS approaches across these driving cycles. In terms of fuel consumption, MATD3 achieves reductions of approximately 10.1% (UDDS), 16.3% (HWFET), 13.1% (NEDC), and 11.9% (WLTC) compared to TD3, yielding an average reduction of about 12.9%. Compared with MADDPG, the corresponding reductions are 7.5%, 4.8%, 6.5%, and 7.2%, yielding an average reduction of approximately 6.5%. Additionally, MATD3 attains fuel consumption within 8–15% of the DP-optimal solution across all cycles, demonstrating near-optimal performance while avoiding the high computational complexity of DP.

Table 5. Comparison of EMS testing performance under different driving cycles.

Driving	Fuel DP	Fuel (L/100 km)			Final SOC			Fuel Economy (%)
Cycle	(L/100 km)	TD3	MADDPG	MATD3	TD3	MADDPG	MATD3	TD3	MADDPG	MATD3
UDDS	3.18	3.85	3.74	3.46	0.593	0.614	0.614	78.9	81.8	91.2
HWFET	2.53	3.32	2.92	2.78	0.601	0.612	0.620	68.8	84.5	90.1
NEDC	2.97	3.82	3.55	3.32	0.608	0.609	0.613	71.4	80.5	88.2
WLTC	4.22	5.23	4.97	4.61	0.587	0.608	0.610	76.1	82.2	90.7

With respect to fuel economy, MATD3 achieves 91.2% (UDDS), 90.1% (HWFET), 88.2% (NEDC), and 90.7% (WLTC) of the DP benchmark, for an average of 90.1%. This significantly exceeds TD3 (73.8% average) and MADDPG (82.3% average), representing average improvements of 16.3% and 7.8%, respectively. Furthermore, MATD3 maintains a stable final SOC across all driving cycles, consistently remaining within a narrow operational window, indicating improved power-split coordination and enhanced battery health preservation relative to baseline reinforcement-learning approaches.

4.3. Overall Two-Stage Model Effectiveness Verification

The generalization and adaptability of EMS are essential for real-world driving conditions. The effectiveness of EMS optimization is significantly affected by the duration of the driving cycle and fluctuations in state-of-charge (SOC) constraints. Therefore, this section re-evaluates the effectiveness of the control strategy in unfamiliar, untrained driving cycles. To further evaluate HEV performance under the proposed policies, a custom test drive cycle is developed by combining standard driving scenarios for urban, suburban, and highway conditions. This newly constructed test cycle is shown in Figure 13.

Figure 13. Testing drive cycle.

Figure 14 presents the SOC tracking results for each strategy. In the new simulation using the test drive cycle, the SOC trajectory of the proposed MLDMPC + MATD3 approach consistently stabilizes around the target value of 0.60, highlighting the method’s strong capability to preserve battery capacity. This suggests that the reward function effectively regulates the battery’s operating range under the new driving conditions. Furthermore, the figure shows that MLDMPC + MATD3 yields the optimal SOC curve during both steady and dynamic driving phases compared with other strategies. Notably, no violations of the battery’s charge/discharge constraints occur with the proposed method throughout the test cycle. In contrast, other strategies exhibit SOC trajectories that fluctuate mainly between 0.45 and 0.55, with the battery’s over-discharge rate.

Figure 14. Battery SOC profiles of EMS approaches during the test drive cycle.

In summary, this two-stage control architecture consists of a high-level DMPC controller that processes inputs such as position, velocity, and torque to compute optimal velocity and torque trajectories over the prediction horizon, thereby optimizing fuel economy, safety, and mobility across the platoon. Simultaneously, a lower-level controller, MATD3, receives predictions from the DMPC controller and manages power distribution between the engine and electric motor for each HEV. The EMS controller’s power-splitting decisions are directly influenced by the velocity and torque predictions provided by the DMPC over the prediction horizon. Traditionally, in DMPC, each node solves an open-loop optimal control problem independently, using only information from its adjacent nodes. The associated cost function penalizes deviations between predicted and reference trajectories, and the optimization concludes in a single iteration.

In contrast, the proposed ML-DMPC controller enhances this process by integrating metric learning into the optimization routine. It employs a distributed ADMM-based algorithm that solves the problem through a two-loop structure: an outer loop enforcing consensus among nodes and an inner loop applying ADMM to iteratively solve unconstrained subproblems until all constraints are satisfied (as outlined in Step 3 of Algorithm 1). As shown in Table 2, the ML-DMPC approach reduces speed prediction errors, resulting in lower average power consumption ($P_{\mathrm{mean}}$). Additionally, the SOC profiles (depicted in Figure 11 and Figure 12) remain stable by leveraging regenerative braking, ultimately achieving a fuel economy above 90% across all drive cycles (refer to Table 5). Furthermore, Figure 14 demonstrates that the proposed model maintains superior performance even during long-duration drive cycles.

5. Proposed Future Work of HIL Architecture for HEV ECU Testing

HIL simulation is a critical methodology for validating and verifying advanced control strategies in HEV platooning and energy management applications. Among available platforms, dSPACE provides a powerful real-time testing environment that enables rapid implementation and evaluation of complex control algorithms. As part of future experimental work, a comprehensive dSPACE-based HIL test platform will be developed to validate the proposed cooperative energy management strategy integrating ML-DMPC and MATD3 controllers. The proposed dSPACE-based HIL architecture for a two-stage control framework is illustrated in Figure 15.

Figure 15. dSPACE-based HIL architecture for two-stage control framework.

The HIL setup will incorporate dSPACE DS1104 and DS1006 real-time processor boards (dSPACE GmbH, Paderborn, Germany), ControlDesk software (https://www.dspace.com/en/pub/home/products/sw/experimentandvisualization/controldesk.cfm, (accessed on 5 January 2026)) for experiment control, dSPACE I/O modules, and hardware-based emulators for battery, internal combustion engine, and electric motor behavior. This configuration enables closed-loop, real-time data exchange between a Toyota Prius power-split ECU and Python-based plant models via Ethernet/CAN. In the planned architecture, the DS1104 board will host the ML-DMPC supervisory layer, which generates high-level velocity-tracking and torque-reference commands that are transmitted to the DS1006 module. Meanwhile, the MATD3 deep reinforcement learning controller, trained offline in Python/TensorFlow, will operate externally on a host PC. The trained policy network will be converted to ONNX format for deployment within the real-time HIL environment. The DRL agent (ECU) will receive real-time feedback signals, including vehicle speed, torque, and battery state of charge, from the DS1006 controller and compute optimal power-split commands for the simulated HEV powertrain.

The inclusion of physical component emulators enhances realism and ensures safe testing by replicating dynamic load characteristics and operational constraints without requiring an actual vehicle. This HIL system provides a robust platform for evaluating fuel-efficiency improvements, battery health utilization, emission levels, disturbance tolerance, and fault resilience under real operating conditions. The setup will facilitate experimental benchmarking of the proposed ML-DMPC + MATD3 control framework against conventional energy management strategies, highlighting its benefits for real-world HEV energy control applications. A dSPACE-based HIL platform integrating DS1104/DS1006 controllers and physical component emulators will enable safe assessment of power-split decisions, battery behavior, and overall HEV performance. The experimental results and benchmarking outcomes from the HIL implementation will be presented in future studies.

6. Conclusions

This paper presents a two-stage energy management control to improve the fuel economy of HEVs. In the first stage, ML-DMPC uses ADMM optimization to forecast the preceding vehicle’s speed with lower predictive errors and to control acceleration, thereby reducing energy consumption. Compared with traditional DMPC, ML-DMPC exhibits improved platoon stability and economic performance. Subsequently, the velocity and torque are provided as inputs to MATD3 to solve the EMS problem for the HEV, yielding an optimal power split. Compared with MADDPG, MATD3 converges faster. The method’s effectiveness is validated across various driving cycles. The simulation results demonstrate that the proposed strategy can achieve a near-optimal solution, confirming its strong adaptability in real-time applications. Future directions include verifying the proposed method across multiple computing units by integrating it into real-time traffic scenarios and HIL validation. A dSPACE-based HIL platform integrating DS1104/DS1006 controllers and physical component emulators will enable safe assessment of power-split decisions, battery behavior, and overall HEV performance. The experimental results and benchmarking outcomes from the HIL implementation will be presented in future studies.