High-Adaptability Driving Mode and Torque Distribution Algorithm Design for Multi-Speed Four-Wheel Drive Electric Vehicle Based on Multi-Agent Deep Reinforcement Learning

Abstract

Multi-motor electric heavy-duty vehicles face significant energy efficiency challenges due to the complex coordination of discrete gear selection and continuous energy flow allocation in the powertrains. This study aims to address this coordination dilemma by proposing a multi-agent deep reinforcement learning energy management strategy (EMS). The framework employs collaborative control across three agents to simultaneously optimize middle axle/rear axle gear shifts (DQN) and power distribution (DDPG), effectively handling the hybrid action space. A specialized rule is integrated to accelerate convergence and enhance real-cycle adaptability. Simulation results on CHTC-TT and CHTC-HT cycles show the proposed strategy achieves only 3.14% and 4.65% higher energy consumption, respectively, compared to a rule-optimized benchmark. This validates its practicality and robustness for real-world electric heavy-duty transportation applications.

1. Introduction

With the challenge of global warming and environmental pollution, energy-saving and electric transport have become a hot topic [1]. Trucks usually have complex powertrain configurations and consume much more energy than small cars; thus, energy consumption is saved [2] and the emission reduction in trucks has attracted great attention. To overcome the shortcomings of battery-powered electric trucks, hybrid trucks and hydrogen electric trucks appeared on the market [3]. EMS plays a critical role in vehicle technology [4]. The purpose of EMS is to minimize the energy consumption of an electric car, by efficiently allocating and managing the vehicle’s energy sources [4,5,6], to improve the energy efficiency and reduce energy consumption [7]. The intelligent decision-making framework balances adaptability to driving cycle with energy consumption control in electric vehicle energy management, providing a crucial intelligent decision-making method for applying multi-agent cooperative strategies in electric powertrain energy allocation and gearshift coordination control [8].

EMS are classified into optimization-based strategies, rule-based strategies, and learning-based strategies. Optimization-based EMS [9] are mainly classified into two methods based on global optimization and instantaneous optimization [10]. Global optimization EMS mainly uses methods such as the Genetic Algorithm [11] (GA), Dynamic Programming [12] (DP), and Pontryagin’s Minimum Principle [13] (PMP). GA cannot find the global optimal solution in some cases, and it is less computationally efficient, with a slower convergence rate and a performance that is more sensitive to the parameter settings. DP [14] is difficult to apply online in real driving, and it is generally used as a reference and cross-reference for strategies. Rule-based EMS can be divided into two types of strategies: deterministic rules designed based on expert experience and those designed through fuzzy logic control. Deterministic rules cannot adapt to various changes and lack flexibility when faced with the influence of multiple factors. The rule formulation and post-calibration rely on reliable engineering experience and expert knowledge [15], which is less flexible and less adaptive, and cannot achieve the theoretically optimal performance [16]. There is a significant inherent contradiction between precision and memory size in rule-based EMS. There is a positive correlation between the degree of policy refinement and the control effect. The detailed rule division can achieve more accurate regulation of the system energy flow and improve the energy utilization efficiency [5]. However, the increase in precision comes at the cost of strategy complexity, which results in more memory resources being consumed in the storage and execution of strategies.

This characteristic is particularly significant in resource-constrained computing environments, and the emergence of learning-based EMS is the right solution to this problem. Learning-based EMS does not rely on experts’ experience, and it obtains the optimal control strategy through data-driven and adaptive learning. Learning-based EMS has good real-time performance and adaptability to different driving cycles. It can optimize computational efficiency while guaranteeing policy accuracy and avoiding the ‘accuracy-memory size’ balance.

However, a single agent has difficulty in satisfying both discrete and continuous actions when dealing with hybrid action spaces. Hybrid action space [17] problems are common in the field of learning vehicle energy management [18]. There are generally three traditional solutions [19] for a hybrid action space. The first one is continuous action discretization, where a continuous action space is turned into a discrete space by performing a split-box/split-barrel operation, and then using discrete action-based algorithms, such as DQN. The second one is the discretized action continuousization: the discrete action space is mapped into a continuous action space by some transformations. Then, adopt a continuous action algorithm, such as the Deep Deterministic Policy Gradient (DDPG); separate optimization: For discrete action, adopt a discrete algorithm; for continuous action, adopt a continuous algorithm. The two algorithms are updated alternately, each calculating its own gradient and finding the discrete and continuous actions that explore the optimum.

Traditional solutions are plagued by numerous issues. The discretization of a continuous action space is prone to cause accuracy degradation, resulting in the algorithm missing the optimal solutions or being trapped in local optima. The continuousization of the discrete action space often brings about unnecessary levels of complexity. This not only heightens the learning difficulty but also renders the policy overly sensitive to minor action changes. As a result, its stability and generalization ability are undermined. Moreover, considering the prevalence of valuable a priori knowledge in most scenarios, the separate optimization of discrete actions and continuous parameters severs their inherent connection. Consequently, the learned strategy lacks integrity, with discrete action selection being uncoordinated with the adjustment of continuous parameters. This incapability of forming an effective strategy combination ultimately reduces both the effectiveness and efficiency of the overall strategy. In other words, it is difficult for a single agent to generate an optimized EMS that deals with continuous and discrete actions at the same time, while multi-agents are required to improve the adaptability of an EMS to a hybrid action space.

In recent years, many academics have proposed new methods to address the hybrid action space problem. For the parameterized Action MDPs (Q-PAMDP) [20], the algorithm assigns a parameter of continuous action to each discrete action. When making decisions, the algorithm selects a discrete action based on the policy ${\mathsf{\pi }}_{\mathsf{\omega }}^{\mathrm{d}}\left(\mathrm{a}\right|\mathrm{s})$. And then the algorithm obtains the action parameter according to the policy ${\mathsf{\pi }}_{\mathsf{\theta }}^{\mathrm{a}}\left(\mathrm{x}\right|\mathrm{s})$. For the Parameterized Action-DDPG (PA-DDPG) [21], the algorithm merges discrete actions and continuous actions in training, using continuous values to represent the weight values of discrete actions. For the Parametrized Deep Q-Networks Learning (PDQN) [22], the algorithm uses a policy network to find the optimal sequential actions corresponding to the discrete actions. Then, the algorithm splices the sequential actions with the states and inputs them into the Q-network to get the weight values of each discrete action. Hybrid Proximal Policy Optimization (HPPO) [21]. The HPPO algorithm evolved from the PPO algorithm. Compared to DQN and DDPG [23], which can only handle discrete or continuous actions, the PPO [24] is an algorithm that can handle both discrete and continuous actions, but the underlying network structure can only handle continuous or discrete actions individually.

These methods involve modifications to the neural network structure based on the original algorithm, increasing the complexity of network design. Multi-agents are involved in the decision-making process simultaneously in the EMS of vehicles. A system consisting of multiple entities is called a multi-agent system. A multi-agent system can achieve complex behaviors far beyond a single individual [25], can solve large-scale complex problems that are difficult to handle by single-agent methods, and are currently mostly applied to communication, collaboration, or competition between independent individuals. In the EMS of vehicles, there are multi-decision hybrid tasks such as gear selection and power allocation, which can be regarded as collaborative tasks of multiple agents.

The structural coupling of hybrid actions in multi-objective EMS tasks differentiates our scenario from conventional single-agent hybrid action problems. In EMS, hybrid actions consist of discrete decisions and continuous control variables, which are not independent but are mutually constrained by objectives. Single-agent methods typically model hybrid actions as a unified parameterized space and optimize them through a single policy network, which faces two critical limitations: First, the curse of dimensionality is exacerbated by the coupling of discrete-continuous variables, leading to slow convergence and poor generalization in high-dimensional action spaces; second, single-agent policies struggle to capture the hierarchical logic of hybrid actions. By contrast, the MARL framework decomposes the hybrid action space into specialized sub-agents (a discrete mode decision agent and a continuous control agent) based on the functional hierarchy of actions, enabling each agent to focus on its own action type. This decomposition is not a trivial design but a targeted solution to the structural complexity of hybrid actions in our task.

This study aims to improve the energy efficiency of a three-motor 11-mode four-wheel drive powertrain. The multi-agent [26] deep reinforcement learning algorithm is used to address the hybrid action space problem in the EMS for a multi-motor multi-speed powertrain in the truck. It retains the traditional agent’s architecture that deals with a single action space, using the algorithm that deals with continuous actions to deal with continuous actions and using the algorithm that deals with discrete actions to deal with discrete actions. A multi-agent system is employed to achieve the collaborative control of hybrid actions related to vehicle gear and power distribution [27], thereby adapting to different driving cycles and driver needs. The power, economy, and safety of the vehicle, as well as the operating state and efficiency of the motor, are considered to formulate the optimal power distribution scheme. Different agents implement the power distribution to the vehicle by selecting reasonable gears and distribution coefficients. Each agent needs to access global information, including the states and actions of other agents, to enable data exchange between multiple agents. Therefore, this multi-agent system is trained using a centralized architecture, which is guided by setting different reward structures and rules to achieve multi-agent collaboration. The main contributions of this paper are as follows.

(1) Multi-agent algorithms are currently applied to scenarios such as collaborative competition between multiple individuals, and in this study, they are applied to a multi-part collaboration of a single individual. The multi-agent deep reinforcement learning strategy is applied to the EMS of a multi-motor–multi-gear–multi-axis vehicle, which solves the problem of the hybrid action space that exists in the EMS of a multi-motor–multi-gear–multi-axis vehicle.

(2) The rule-based pattern control mechanism is used to optimize the agent’s action exploration space, eliminate the irrational exploration of actions by the multi-agent deep reinforcement learning strategy, and improve the performance and learning efficiency of EMS based on the multi-agent.

The development of EMS using multi-agent deep reinforcement learning techniques helps to improve the operational efficiency of multi-motor–multi-gear–multi-axis electric drive systems, and also provides new ideas and methods for the application of deep reinforcement learning to the energy optimization problem of dynamic systems with multi-gear and multi-axis power distribution for complex vehicle configurations. The rest of the paper is organized as follows. Section 2 presents a three-motor–11-mode–four-wheel-drive powertrain, Section 3 describes the method of constructing the EMS for the multi-motor–multi-gear–multi-axle drive system using the multi-agent deep reinforcement learning technique, detailing the processing of the hybrid action space by different algorithmic networks. Section 4 compares and analyzes the EMS based on the multi-agent deep reinforcement algorithm with that based on the optimal pattern graph, and analyzes the significance of the introduction of rule-based optimization for the convergence of the algorithm. Section 5 concludes the whole paper.

2. Powertrain Configuration

The powertrain configuration of the vehicle is shown in Figure 1. The configuration adds a gear configuration on the basis of the two-axle, three-motor vehicle configuration, which optimizes the vehicle’s power domain and further improves the energy utilization efficiency.

The power system is controlled by a dual-axle and three motors, respectively, equipped with multi-gear transmission; the middle axle is a combination of one-motor–three-gear transmission, and the rear axle is equipped with two motors to drive the left and right rear wheels respectively.

Each side of the motor is equipped with a two-speed transmission, which can independently adjust the torque and speed of the left and right rear wheels to improve the vehicle’s handling and stability. And the torque and speed of the motors under straight-line conditions, as well as the transmission gears, are the same; the gear settings are shown in

Table 1. Gear settings.

	Rear: N	Rear: 1	Rear: 2	Rear: 3
Middle: N	0	1	2	3
Middle: 1	4	5	6	7
Middle: 2	8	9	10	11

.

N represents the neutral state, while 1, 2, and 3 denote the gears corresponding to a single motor operation. Additionally, the range from 0 to 11 represents eleven gear combination modes. It should be noted that the gears can only be shifted to adjacent positions, and gear skipping is not allowed. The specifications of the target powertrain are summarized in

Table 2. Powertrain specifications.

Components	Parameter	Value
Vehicle	Tractor weight	11,000 kg
	Trailer weight	29,000 kg
	Gross weight	40,000 kg
	Front area	9.5 m2
	Wheel-radius	0.5226 m
Mot1/Mot2	Rotating mass conversion factor	1.1
	Drag coefficient	0.35
	Rolling resistance coefficient	$\mathrm{f}=\mathrm{c}\mathrm{t}1+\mathrm{c}\mathrm{t}2\ast \mathrm{s}\mathrm{p}\mathrm{d}$
	Rolling resistance parameter1	0.003675
	Rolling resistance parameter2	2.24 × 10−5
	Mot1 peak power	220 KW
	Mot2 peak power	150 KW
	Mot1 peak torque	310 Nm
	Mot2 peak torque	320 Nm
	Mot1 speed	13,000 rpm
	Mot2 speed	13,000 rpm
	Continuous power1	120 KW
	Continuous power2	75 KW
Transmission	i_1; i_2; i_3; i_4; i_5	87.36; 40.20; 19.65; 64.25; 19.89

.

In this study, a backward simulation model is used; the speed and torque requirements are derived from the actual driving cycle. The demand parameter data of each component is deduced from the working principle of each simulated component, based on which the working performance of the vehicle is analyzed. The backward simulation strategy focuses on the evaluation of the economic performance of the drive system and does not require a driver model. Based on the vehicle cycle, the required driving torque is calculated by Equation (1); 21.15 is the unit conversion factor in air resistance calculations:

$$T_d=(\delta ma+mg sin \theta +mgf cos \theta +{\displaystyle \frac{C_{D}A{\nu }^2}{21.15}})R$$

(1)

δ represents the rotating mass conversion coefficient of the car, and the value is 1.1; T_d represents the driving torque required for the vehicle to move, N-m; m represents the vehicle’s overall mass, kg; a represents the acceleration of the car when moving, m/s²; g represents the acceleration of gravity, $9.8 {\mathrm{m}/\mathrm{s}}^2$; ƒ represents the rolling damping coefficient of the road; θ represents the slope of the road, which is not taken into account in the present study, and its value is 0; CD represents the air damping coefficient; A represents the frontal area of the car, m²; v represents the speed of the car, m/s.

In the research stage, we cannot test to get the accurate rolling resistance coefficient value, and so, using the automotive theory empirical formula, we roughly estimate a good road passenger car tire rolling resistance coefficient as follows:

$$f=ct1+ct2\ast spd$$

(2)

ct1 and ct2 are the constant coefficient terms of the rolling resistance coefficient and the speed-related term, respectively, where the speed unit is KM/h. The above table i_1, i_2, and i_3 are the ratios corresponding to the three-speed transmission of the middle axle, and i_4 and i_5 are the ratios corresponding to the two-speed transmission of the rear axle.

3. EMS Based on Deep Reinforcement Learning with Multi-Agent

The vehicle powertrain is a three-motor, multi-gear, four-wheel drive configuration, which needs to distribute the power between the middle and rear axles and select the gear for each motor. The power distribution and gear selection results directly determine the energy consumption for a particular power requirement. The challenge of adopting the reinforcement learning algorithm, i.e., DDPG, to design the EMS is handling the action space, which contains both the continuous action (power distribution between the middle-rear axle) and the discrete action (gear selection for motors).

For the well-known reinforcement learning algorithms, such as DDPG, SAC, and DQN, it is difficult for any single agent to deal with multi-dimensional complex problems. Additional tricks, e.g., Gumble SoftMax [28], are required to output continuous and discrete actions in one agent. However, it is still a trade-off between accuracy and performance. Therefore, this paper studies a multi-agent deep reinforcement learning algorithm to design the EMS for this kind of configuration, which relies on different agents to deal with the continuous and discrete actions, separately. Currently, multi-agent algorithms are mostly applied to communication, collaboration, and competition scenarios of multiple independent entities, and they show obvious flexibility and adaptability advantages in solving real-life complex problems, which are also essential to the learning-based EMS design with a hybrid action space. In this paper, the multi-agent algorithm is applied to control different targets (gear selection and torque distribution) of a single vehicle via communication and collaboration between different control targets, i.e., coordinated control of discrete gears and continuous power distribution in a hybrid action space.

In a single-agent scenario, there is usually only one agent interacting with the environment, and the reward value obtained by the agent performing the same action in the same state is fixed. In a multi-agent environment, multiple agents interact with the environment simultaneously, and real-time communication among agents eliminates independent decision-making by each agent, enabling agents to share information such as states and strategies during the learning process. This paper applies a multi-agent system to control different parts of a single vehicle’s discrete gear selection and continuous torque distribution. Based on the essence of information sharing, unified goals, task allocation, and action synchronization among agents, a coordinated control of the hybrid action space for discrete gears and continuous power distribution is achieved. The vehicle power system is a multi-motor and multi-gear configuration involving power distribution between the middle and rear axles, and the gear selection of each motor also affects power provision and energy consumption. Therefore, the energy management strategy needs to consider the control of power distribution between the middle and rear axles and the middle and rear motors to adjust the output torque and speed of the middle and rear motors in real time to adapt to different driving conditions and driver demands. The proposed multi-agent system consists of three agents: DQN agent 1 for middle axle gear selection, DQN agent 2 for rear axle gear selection, and DDPG agent 3 for power distribution. This framework adopts a centralized training and distributed execution architecture. During centralized training, all agents share the same global state information, including vehicle speed, acceleration, motor torque, wheel speed, and gear position. This full-state sharing mechanism enables each agent to perceive the operational status and decision-making intentions of other agents, avoiding conflicts between gear selection and power distribution. During distributed execution, each agent only uses its own policy network to output decisions in real time without relying on communication among agents. This design ensures efficient collaboration in the hybrid action space and improves the stability and adaptability of the energy management strategy.

3.1. DQN-Based Discrete Gear Selection

Considering the reinforcement learning algorithm DQN can handle discrete outputs, and the gear selection for the powertrain can be taken as discrete actions, this study adopts the classical DQN to control the gear shifting.

Figure 2 shows the framework of the DQN algorithm. The DQN initializes the environment and network parameters before storing the ‘state-action-reward-next_state’ in the replay pool. The experiences are stored and used to train the neural network to approximate the Q-function. Then, in each loop, the agent selects actions based on the policy and observes the feedback from the environment to accumulate rewards. The parameters of the training neural network are iteratively updated to calculate the Q-value of each action. The action with the highest Q-value will be selected to achieve the best strategy. The key to this algorithm is introducing the experience playback mechanism and improving the training efficiency. To stabilize the learning process, DQN employs an independent network to calculate the target Q-value. The motors of the middle axle and rear axle are equipped with multi-speed transmissions to enhance the stability of the vehicle with 11 gears. The selection of motor gears is a discrete action, and the DQN algorithm provides an ideal solution to this problem, due to its adaptability to the discrete action space. Therefore, the DQN algorithm is used for gear selection in the middle axle and rear axle, respectively, to achieve efficient gear control.

3.2. Continuous Torque Distribution Based on DDPG

The DDPG algorithm can generate continuous output, and the power distribution between different motors needs a continuous coefficient action. DDPG is adopted to determine the power distribution between motors in this paper.

Figure 3 shows the DDPG algorithm framework. The DDPG algorithm establishes both the Q-value function and the policy function π, which is also known as the Actor–Critic structure. DDPG inherits the theoretical techniques from DQN: experience replay. Therefore, four networks are involved in the DDPG algorithm: Actor network, target Actor network, Critic network, and target Critic network, in which the Actor network is responsible for outputting the action A. As a deterministic policy function, it does not provide the probability of action, but directly outputs a certain action, i.e., ${\mathrm{A}}_{-}t=\pi (S\_t|\theta \_t^{\wedge }\pi )$. The target Actor network generates the next action A’ based on the data from the empirical playback pool. DDPG updates in the same way as the DQN does, and the Q-value function of DDPG is computed by the Critic network. DDPG uses a coefficient ρ to update the target network parameters softly. Here ρ ≪ 1, indicating that the target network will be updated slowly, improving the stability of learning. To balance the exploration capability and convergence speed of the deterministic strategy, DDPG relies on adjusting the OU noise during training. The update formula is shown below:

$$\{\begin{array}{c}{\theta }^{Q^{\prime }}\leftarrow \rho {\theta }^Q+\left(1-\rho \right){\theta }^{Q^{\prime }}\\ {\theta }^{{\pi }^{\prime }}\leftarrow \rho {\theta }^{\pi }+\left(1-\rho \right){\theta }^{{\pi }^{\prime }}\end{array} \rho \ll 1$$

(3)

In order to improve the energy utilization, the energy flow is allocated for multiple motors between the middle and rear axles in this paper. DQN is suitable for solving discrete problems, and DDPG is suitable for solving continuous problems. In this paper, the action space of the reinforcement learning EMS needs to contain both continuous actions (power distribution coefficients of the middle and rear axles) and discrete actions (gear selection of the motors). Thus, the multi-agent system is chosen to deal with this problem.

3.3. Multi-Agent System Based on DQN-DDPG

The multi-agent reinforcement learning EMS(MARL) algorithm consists of three agents, including two DQN agents using discrete actions to select the transmission gears, and one DDPG agent using continuous actions to determine the power distribution coefficients (0–1) between the motors on the middle and rear axles. Each agent has an independent network of Critic and Actor.

Figure 4 shows the Network Architecture Configuration of the Multi-Agent System that employs a DQN network for gear selection. This approach strikes a balance between feature extraction and computational efficiency, enabling the model to fit the state features of energy management scenarios while avoiding excessive network complexity that could compromise training efficiency. The DDPG network handles torque allocation, employing a diminishing neuron scale of 256-128-64 to progressively compress the features of extracted multi-dimensional observation variables into core characteristics, thereby meeting the requirements for continuous action output. The DQN adopts a lightweight unified architecture to ensure efficiency in discrete decision-making, while the DDPG utilizes a deeper hierarchical network to guarantee precision in continuous control.

As shown in

Table 3. Multi-agent algorithm hyperparameters.

Algorithm	ITEM		VALUE
DQN	LearnRate	DQN1	0.00001
	LearnRate	DQN2	0.00001
	BufferLength	DQN1	1000
	BufferLength	DQN2	1000
	MiniBatchSize	DQN1	128
	MiniBatchSize	DQN2	128
DDPG	LearnRate	DDPG-actor	0.00001
	LearnRate	DDPG-critic	0.0000005
	BufferLength		1 × 104
	Discount		0.999
	MiniBatchSize		128
Train	Noise		0.3
	DeviationDecayRate		1 × 10−4
	Maxepisode		500
	MaxstepsPerEpisode		1800

, these represent the hyperparameters for multi-agent reinforcement learning algorithms. Hyperparameters serve as the core tuning factors determining the training stability, convergence efficiency, and task adaptability of such algorithms. The hyperparameter configurations in the Table have been customized for the characteristics of energy management tasks and the heterogeneous multi-agent framework comprising DQN and DDPG. This hyperparameter configuration system is guided by the core requirements of energy management tasks, implementing differentiated and collaborative designs tailored to the algorithmic properties of heterogeneous agents. This configuration ensures both the training stability and task adaptability of individual agents while achieving collaborative training efficiency among multiple agents. It provides a reliable hyperparameter design paradigm for applying heterogeneous multi-agent reinforcement learning in energy management applications.

Figure 5 shows the CHTC-TT cycle and the CHTC-HT cycle. The EMS reinforcement learning algorithm was trained using the CHTC-TT operating condition and validated based on the CHTC-HT condition. In consequence, three different agents are involved to realize this paper power distribution and gear selection at the same time. The power distribution coefficient is between [0 and 1], represented by $𝜕$, where 0 and 1 represent the rear and middle single-axle drive modes. Coefficients $𝜕$ stands for the ratio of the middle axle motor allocated power to the total required power, while the power allocated to the rear axles is (1 − $𝜕$) × Teq. The two agents that select the engaged gear for the middle and rear axle are [0, 1, 2, 3] and [0, 1, 2], respectively, where 0 represents the neutral gear. Since the three actions are determined by three different agents, it is possible that a specific gear is engaged, but the corresponding allocated power is 0. In other words, the motor does not generate any power, but it is driven by the wheel via the engaged gear with a vehicle speed-related rotation. The aforementioned unreasonable agent exploration inevitably increases the training time and makes the strategy difficult to converge. Therefore, the action selection space should be restricted by specific rules. When the power distribution coefficient determined by DDPG agents is 0 or 1, the corresponding gear selection of the middle or rear axle is directly set to 0, and a single axle drive mode is selected. The action space is set as follows:

$$\begin{gathered} Action Space = \left\{\left(𝜕=0,Mode1=0,Mode2=1,2\right)\right\} \\ \cup \left(𝜕=1,Mode1=1,2,3,Mode2=0\right) \\ \cup \left(𝜕\in \left(\mathrm{0,1}\right), Mode1=1,2,3,Mode2=1,2\right) \end{gathered}$$

(4)

Given that the vehicle state will be changed after mode switching and power distribution in each training step, the energy management problem in this study can be taken as a Markov decision process problem.

Like other reinforcement learning algorithms, the reward plays a key role in agent training quality. Since each agent relies on a different reward to guide the policy updating and the action-based state space changing affected by all the agents, it raises a greater challenge for the multi-agent learning algorithm compared to the single agent-based one. To avoid the significant negative reward or the termination of training, human intervention rules are necessary to correct the agents’ bad exploration. Furthermore, to reduce the impact of external corrections on agents’ exploration, a punishment reward named ‘Change’ is introduced as part of the reward.

Figure 6 shows the rule-based EMS framework to coordinate the different requirements of each agent for convergence. The correction rule module checks the action combinations generated by the multi-agent, adjusting and optimizing them based on the preset rules. If the action combination is found to be unreasonable, the correction rule will modify the action to ensure the feasibility and safety of the action.

The reward terms with different purposes are proportionally introduced to the united reward function with different weightings. The constant term C is a positive encouragement to motivate the agents to complete the whole driving cycle; Q is the energy consumption, and q(t) is the energy consumption of each step; to avoid frequent gear changing and skipping gears, set the gear changing term M1 and M2, respectively, in the rear axle gear reward term, which involves the cross-grade action so M1 is maximal 3, and M2 is maximal 2; the human intervention term Change is to avoid the irrational exploration of the agents. Negative rewards are set for agents to reduce the number of human interventions. Isdone is a penalty item for motors exceeding the limit.

As shown in

Table 4. Multi-agent reward settings.

Item	Multi-Agent Bonus Item Setting
Constant term:	C = 20
Energy consumption items:	Q = (−200) × q(t)
Gear shift item 1:	M1 = (−3) × (1/2/3)
Gear shift item 2:	M2 = (−2) × (1/2)
Human intervention projects:	Change = (−15) × (1/2/3/4)
Penalty term:	Isdone = (−1000)

, the algorithmic reward values are configured as follows: the constant term provides a baseline reward to prevent excessive negative fluctuations caused by other penalty terms, thereby preserving the agent’s fundamental exploration drive and preventing it from abandoning exploration due to excessively low initial rewards.

The primary objective of the energy consumption term is to reduce energy expenditure by penalizing high-energy consumption behaviors, serving as the core reward component for energy management tasks. The gear-shifting term is employed to minimize mechanical wear, mitigate energy consumption fluctuations, and enhance operational stability. The manual intervention component penalizes decisions made through human interference, guiding the agent towards self-sustaining, stable energy levels and reducing reliance on manual input.

The overall goal of the multi-agent algorithm is to minimize energy consumption; however, different agents are trained with slightly different goals. So, different target rewards need to be set to guide different agents in the convergence on the basis of the unchanged overall goal. The DDPG agent distributes coefficients of power between motors, and the R1 rewards focus on reducing energy consumption. The DQN agent performs the selection of the gears between the middle and rear axles. The R2 rewards and the R3 rewards of DQN focus on the rational shift in gears. The reward functions of the three agents are set as R1, R2, and R3, respectively:

$$\begin{gathered} R1=C +Q+ isdone+0.5\ast \left(M1+M2+Change\right) \\ R2=C +Q+ isdone+M1+Change \\ R3=C +Q+ isdone+M2+Change \end{gathered}$$

(5)

Figure 7 shows the MARL-EMS design framework. The training process of the MARL-EMS algorithm is an iterative optimization process. In each training episode, the agents independently select a local action (distribution coefficient, gear selection of the middle axle, or gear selection of the rear axle) based on the state information, such as vehicle speed, demand torque, etc., which are the algorithm output. In order to avoid unreasonable action combinations (e.g., gear selection mismatch, etc.), the algorithm introduces correction rules. The action combinations optimized by the correction rules are fed into the environment for validation. The environment will simulate the operating state of the system based on the new action combination and calculate the corresponding reward value. Eventually, the multi-agent will update its strategy based on the reward value and gradually learn an effective EMS. This process keeps cycling until the algorithm converges on an optimal control strategy.

4. Simulation Results

The performance of the optimal strategy mainly depends on the spacing. The smaller the computational step size, the higher the accuracy. However, a huge amount of computation and storage space requirements are demanded, which is often difficult to achieve in practical applications.

Figure 8 shows the optimal rule-based method of dense table, and Figure 9 shows the optimal rule-based method of sparse table. Dense table spacing can better approximate the theoretical optimal energy consumption. From the comparison of Figure 8 and Figure 9, it can be observed that the dense table is clearer in the definition of boundary values, while the boundary definition of the sparse table is rougher. This leads to the selection of a non-optimal strategy due to the accuracy problem, thus affecting the performance of the optimal mode.

Table 5. Performance of energy consumption with different accuracies under two driving cycles.

Cycle	Speed Accuracy	Torque Accuracy	Distribution Accuracy	Energy Consumption
CHTC-HT	10 km/h	1000 Nm	0.1	122.8
CHTC-HT	1 km/h	1000 Nm	0.1	120.7
CHTC-TT	10 km/h	1000 Nm	0.1	128.7
CHTC-TT	1 km/h	1000 Nm	0.1	128.3

shows the performance of energy consumption with different accuracy under two driving cycles. CHTC-HT and CHTC-TT are chosen as the validation conditions to compare the performance of energy consumption by adjusting the sparsity of the Table with the step size of the speed. It can be observed from the Table that the energy consumption is smaller for a smaller step size. The data in the Table confirms that dense table spacing can better approximate the theoretical optimal energy consumption. The EMS only reflects the theoretical possibility in optimal energy consumption, which is used as a theoretical benchmark for the evaluation of other practical EMS. The performance of the multi-agent algorithm will be evaluated by comparing it with the results obtained from the optimal benchmark.

Under the CHTC_TT standard operating condition, both the rule-based strategy and the multi-agent learning strategy significantly reduce gearshift frequency. The strategy adaptively adjusts the motor drive mode according to vehicle speed and demand torque. To improve the powertrain efficiency and reduce energy consumption, the algorithm tends to select the single-axle drive mode with the middle-axle motor under low-speed conditions. As the vehicle accelerates and the demand for torque increases, the algorithm prefers the dual-axle drive mode under the middle- and high-speed conditions or large torque demand, i.e., all motors work together to meet the greater power demand. The MARL-EMS dynamically adjusts the proportion of power distribution between the middle-axle motor and the rear axle motor to improve the overall powertrain efficiency as much as possible. Under high-speed conditions, since the rear axle is powered by two motors with higher efficiency and larger power reserve, the algorithm usually selects the rear axle drive mode. When the two rear axle motors propel the vehicle, the middle axle motor is ready to provide auxiliary power in standby mode. This adaptive mode selection strategy effectively utilizes the characteristics of different motors at different speeds and torques, which demonstrates the capability of the proposed strategy to achieve energy savings under the CHTC-TT cycle.

The algorithm with rule optimization can guide the multi-agent algorithm to learn in a smaller search space by restricting the range of the motor and the shift strategy, and thus it avoids wasting too much time and computational resources in the ineffective strategy space. It completes the exploration of the whole working condition more quickly.

As shown in Figure 10 and Figure 11, the rule-optimization strategy (RO) is introduced to effectively improve the training efficiency and convergence speed based on the multi-agent reinforcement learning algorithm. Compared with the basic multi-agent algorithm, the MARL-EMS significantly reduces the irrational exploration behaviors with shorter training time and faster convergence speed. The MARL-EMS completes the exploration of the whole driving cycle in about 200 episodes before convergence, while the multi-agent algorithm without rule-based optimization tends converge after 500 episodes with significantly low rewards.

Figure 12, Figure 13, Figure 14 and Figure 15 illustrate the correlation between the engine efficiency map and actual vehicle operating points. The red curve depicts the engine’s outer characteristic curve, representing the maximum torque output at each rotational speed under wide-open throttle conditions. The blue scatter points indicate the actual operating points measured during vehicle operation, reflecting the real-time speed-torque distribution of the engine. The colored contour lines denote different efficiency levels, with labeled values marking the threshold of each efficiency band. Figure 12 and Figure 13 respectively illustrate the operating point distributions of the middle axle and rear axle under the parameter-traversal energy consumption benchmark strategy. The operating points of the middle axle (TM1) are predominantly concentrated within the medium-to-high speed and medium-torque range, specifically at rotational speeds of 6000–12,000 rpm and torques of 50–200 Nm. The overall distribution exhibits characteristics of high rotational speeds and medium loads. The majority of operating points fall within the 90–95% high-efficiency zone of the motor’s efficiency map, with only a small number distributed in the low-speed, low-torque region where efficiency is below 90%. This characteristic indicates that under the energy consumption benchmark strategy, the middle axle motor is prioritized for operation within the high-efficiency range to leverage its efficiency advantages at medium-to-high speeds. The rear axle motor (TM2) exhibits a broader operating point distribution, spanning speeds from 0 to 8000 rpm and torques from 0 to 200 Nm. It displays a notably dense concentration of operating points within the low-speed, low-torque region, with some points achieving efficiencies of merely 85–90%. This distribution reflects the rear axle motor’s role under the baseline strategy in handling greater low-speed, variable-load power output tasks, thereby ensuring system power redundancy. The disparity in operating point distributions between the two motors reveals the functional division within the dual-axle drive system under the energy consumption benchmark strategy: the middle axle motor focuses on the high-efficiency range to optimize energy consumption, while the rear axle motor covers a broad load spectrum to guarantee power redundancy. This outcome not only provides a clear benchmark for evaluating the effectiveness of subsequent multi-level energy management strategies but also offers data-driven support for balancing efficiency prioritization and power redundancy in multi-motor cooperative control designs.

Under the collaborative control of the multi-agent strategy, Figure 14 and Figure 15 respectively illustrate the operating point distributions for the mid-bridge motor TM1 and rear axle motor TM2. Compared to the energy consumption benchmark strategy, these demonstrate significant optimization characteristics, reflecting the core advantage of the multi-agent strategy in enhancing multi-motor collaborative efficiency. The operating points of the mid-bridge motor now span a broader range, extending from low-speed, low-torque conditions to medium-speed, medium-torque scenarios. Notably, the density of operating points in the low-speed segment is markedly higher than that of the baseline strategy. The majority of operating points remain concentrated within the high-efficiency range of 90–95% efficiency. Furthermore, operating points in the low-speed segment are positioned as close as possible to the 90% efficiency isochrone. This demonstrates that the multi-agent strategy successfully broadens the operational scope of the mid-bridge motor while maintaining high energy utilization efficiency. This distribution demonstrates the multi-agent strategy’s dynamic scheduling capability, moving beyond restricting the mid-bridge motor to a fixed high-efficiency range. Instead, it flexibly adjusts its load range according to vehicle requirements while ensuring minimal efficiency loss through intelligent matching. The operating point distribution for the rear axle motor TM2 exhibits distinct clustering characteristics, predominantly concentrated within the medium-speed, medium-torque range of 4000–8000 rpm and 50–150 Nm. Its distribution density significantly exceeds that of the baseline strategy, while operating points in the low-speed, low-torque region are substantially reduced. The vast majority of operating points fall within the high-efficiency range of 90–95% efficiency, with some densely clustered areas approaching the 95% efficiency peak. This represents a marked reduction in the proportion of inefficient points compared to the baseline strategy, directly reflecting enhanced energy utilization efficiency. This distribution demonstrates that the multi-agent strategy actively guides the rear axle motor’s operating points towards the high-efficiency range through dynamic optimization of gear selection and torque coordination. This reduces its operating time in inefficient zones, thereby substantially enhancing overall energy utilization efficiency.

Analysis of the motor operating point diagram reveals significant differences in motor utilization efficiency between the two energy management strategies. As the baseline strategy, the optimal approach fully accounts for the power characteristics and efficiency curves of the mid-rear axle motors, ensuring both operate within the high-efficiency range wherever possible to maximize overall energy utilization efficiency. This is reflected in the rational distribution of the mid-rear axle motor’s operating points within the high-efficiency zone. In contrast, the strategy learned by the multi-agent deep reinforcement learning algorithm exhibits a tendency to prioritize power supply from the rear axle motor, with its operating points predominantly concentrated in the high-efficiency zone where efficiency exceeds 90%.

Figure 16 presents the power distribution pie chart for the optimal strategy, whilst Figure 17 displays that for the multi-agent strategy. These clearly illustrate the differing power output allocations between the mid-bridge motor (TM1) and rear axle motor (TM2) under the two control approaches. From the perspective of power allocation logic, the rule-based strategy concentrates power output on the rear axle motor (TM2). Conversely, the multi-agent strategy employs dynamic cooperative control to reduce the power share of the middle axle motor (TM1) from 20% to 15%, while lowering the rear axle motor’s share from 80% to 85%. This achieves a strategy that favors rear axle power distribution. Under the multi-agent strategy, the operating range of the mid-bridge motor is moderately expanded, with its operating points at low speeds positioned as close as possible to the 90% efficiency isochrone. Conversely, the rear axle motor’s operating points cluster within the high-efficiency range of 4000–8000 rpm and 50–150 Nm torque, significantly reducing inefficient operation at low speeds and low torque. Both motors’ operating points are now more concentrated within the 90–95% high-efficiency range, directly validating the multi-agent strategy’s effectiveness in coordinating hybrid motion space control. Overall, the multi-agent strategy maintains overall system efficiency through dynamic power distribution and operating point optimization. Simultaneously, it reduces operational stress on the rear axle motor via load sharing, achieving a dynamic equilibrium between efficiency prioritization and ample power delivery. This optimization not only provides tangible support for reduced energy consumption and enhanced power response but also offers a viable pathway to improve the long-term reliability of the powertrain system. It fully demonstrates the practical engineering value of multi-agent collaborative control in complex operating conditions. This paper uses the rule-optimal method as a benchmark and verifies the performance of MARL-EMS based on CHTC-TT operating conditions. This article takes the rule-optimal method as the benchmark. Verification of MARL-EMS performance based on CHTC-TT operating conditions. Rule-optimal operating point diagrams (Figure 12 and Figure 13) and MARL-EMS operating point diagrams (Figure 14 and Figure 15) show that there is a significant difference between the two EMSs in terms of motor utilization efficiency. The analysis results are based on the motor operating tracks (Figure 12, Figure 13, Figure 14 and Figure 15) and the power distribution diagrams (Figure 16 and Figure 17). As a baseline strategy, the rule-optimal strategy fully considers the power characteristics and efficiency maps of the middle and rear axle motors, ensuring all of the motors work in the high-efficiency zone to maximize the overall energy efficiency. This is reflected in the reasonable allocation of the working points of the middle and rear axle motors in the high-efficiency interval. In contrast, the strategy obtained by the multi-agent deep reinforcement learning algorithm shows a bias that it tends to prioritize the use of the rear axle motor to provide power, so that its operating points are mainly concentrated in the high-efficiency interval with an efficiency of more than 90%.

Figure 18 shows the efficiency distribution based on the optimized strategy, and the rule-optimal strategy prefers to make the middle-axle motor operate in the high-efficiency region. Figure 19 shows the efficiency distribution based on multi-agent strategies, while the multi-agent strategy tends to enable the rear axle motor to operate in the high-efficiency region. Based on the structural characteristics of the powertrain system and the core control logic of the proposed multi-agent reinforcement learning (MARL)-based energy management strategy (EMS), the inherent preference of the powertrain for rear axle motor utilization arises from the synergistic interaction between its hardware configuration (i.e., a single front motor coupled with dual rear axles) and the multi-objective optimization goals of the MARL-based EMS, rather than arbitrary or unregulated control behaviors. First, from the perspective of powertrain hardware attributes, the dual-motor configuration on the rear axle delivers substantially higher overall power density and a wider high-efficiency operating range compared with the single front motor. Given that the reward function embedded in the MARL framework prioritizes real-time energy efficiency, the discrete-mode decision agents in the multi-agent architecture are naturally inclined to prioritize the activation of dual rear axles during the iterative training process. This control preference extends the duration of the powertrain operating within the high-efficiency region, thereby maximizing the cumulative reward under predefined driving cycle constraints. Second, the inter-agent coordination mechanism inherent in the MARL framework further amplifies this utilization superiority by leveraging the structural merits of the rear axle in torque distribution: the dual rear axles enable fine-grained torque allocation via the continuous control agents, which not only optimizes system-level energy consumption but also improves vehicle driving stability by dynamically adjusting the torque distribution ratio between the two rear axles in response to transient driving conditions.

Figure 20 and Figure 21, illustrating motor efficiency trends, further corroborate this conclusion: under the multi-agent algorithmic strategy, rear axle motor efficiency consistently remains elevated, whereas central-axle motor efficiency remains comparatively subdued, revealing a pronounced imbalance in efficiency distribution. This disparity indicates that while multi-agent deep reinforcement learning methods effectively handle complex mixed action spaces, further algorithmic optimization is required to enhance overall efficiency. For instance, introducing more efficient global reward mechanisms or enhancing multi-agent coordination could optimize motor state synchronization, thereby preventing local optima from hindering the pursuit of global optimality.

Since each agent pursues its own maximum efficiency during the learning process without considering the overall system efficiency, it inevitably fails to achieve the overall rule-optimal operating tracks. To evaluate and analyze the energy performance of MARL-EMS, the energy consumption and motor operating tracks of EMS are compared using the rule-based optimum as a benchmark.

As shown in

Table 6. Energy consumption performance (100 km) and deviation.

Cycle	Rule (kWh/100 km)	Multi-Agent + RO (kWh/100 km)
CHTC-TT	128.3	132.3347	3.14%
CHTC-HT	128.3	134.2718	4.65%

, the energy consumption of the proposed strategy in this paper is only 3.14% higher than the rule-based EMS; the proposed strategy achieves only 4.65% higher energy consumption than the rule-based EMS. The algorithm is able to find more efficient EMS in less time, and this improvement increases the efficiency of multi-agent deep reinforcement learning algorithms. Although the multi-agent strategy ensures the efficiency of the rear axle motor, it reduces the performance of the middle axle motor. The middle axle motor participates in the work only when the rear axle motor is overloaded or needs additional power support. In multi-agent learning processes, there is a certain deviation between the local strategy of a single agent and the global strategy.

5. Conclusions

It is difficult for a single agent to deal with multi-dimensional complex problems, and it cannot handle the complexity of the hybrid action. This study chooses multi-agent deep reinforcement learning algorithms to handle multi-dimensional complex problems and uses different agents to deal with the continuous and discrete actions, respectively.

This study verifies the feasibility of a multi-agent deep reinforcement learning approach to solve the energy management problem of a multi-motor, multi-speed drive powertrain. The control strategy based on multi-agent reinforcement learning designed in this paper successfully handles the complex hybrid action space in this system, i.e., gear selection as well as the cooperative control of inter-axis torque. Traditional rule-based methods often require complex rule design and a large number of parameter adjustments when dealing with this high-dimensional, nonlinear, and strongly coupled hybrid action space.

The multi-agent reinforcement learning algorithm effectively reduces the complexity of the problem by decomposing the system into multiple agents that control different subsystems separately. Each agent learns its own local strategy and achieves the globally optimal energy management goal through inter-agent collaboration. In the energy consumption comparison experiment based on the training cycle, this paper’s strategy is only 3.14% higher than the EMS based on the rule-optimal pattern graph; the proposed strategy achieves only 4.65% higher energy consumption than the rule-optimization EMS. Meanwhile, compared with the multi-agent deep reinforcement learning EMS without rule optimization, the algorithm with rules significantly reduces the unreasonable exploration behaviors during the training process, thus shortening the training time and accelerating the convergence speed. It is shown by the results that the hybrid action space problem existing in complex drive systems is effectively solved by the strategy, and the vehicle energy consumption is reduced to a certain extent. The experimental results show that this method can effectively reduce energy consumption and improve system efficiency while ensuring system stability, which provides a new idea and an effective way for energy management of complex multi-axis drive systems in the future. In addition, our proposed multi-agent architecture has good scalability and robustness, and it can be easily adapted to different numbers of motors, gears, and axes, thus improving the flexibility of system design.

Future research will be carried out in the following aspects:

(1)

Further explore the advantages of deep reinforcement learning algorithms with different combinations of multi-agents to cope with the different power demands of the drive system;

(2)

Further optimize the multi-agent algorithmic strategies to improve the performance of the strategies in terms of energy consumption.