Electric Vehicle Energy Management Under Unknown Disturbances from Undefined Power Demand: Online Co-State Estimation via Reinforcement Learning

Abstract

This paper presents a data-driven energy management scheme for fuel cell and battery electric vehicles, formulated as a constrained optimal control problem. The proposed method employs a co-state network trained using real-time measurements to estimate the control law without requiring prior knowledge of the system model or a complete dataset across the full operating domain. In contrast to conventional reinforcement learning approaches, this method avoids the issue of high dimensionality and does not depend on extensive offline training. Robustness is demonstrated by treating uncertain and time-varying elements, including power consumption from air conditioning systems, variations in road slope, and passenger-related demands, as unknown disturbances. The desired state of charge is defined as a reference trajectory, and the control input is computed while ensuring compliance with all operational constraints. Validation results based on a combined driving profile confirm the effectiveness of the proposed controller in maintaining the battery charge, reducing fluctuations in fuel cell power output, and ensuring reliable performance under practical conditions. Comparative evaluations are conducted against two benchmark controllers: one designed to maintain a constant state of charge and another based on a soft actor–critic learning algorithm.

1. Introduction

The growing demand for sustainable transportation has intensified research in electric vehicles powered by hybrid energy sources [1]. Among these, fuel cell and battery electric vehicles represent a promising solution by combining the high energy density of hydrogen fuel cells with the fast dynamic response of lithium-ion batteries [2,3]. Effective coordination of power flow between these energy sources is critical to achieving performance, efficiency, and durability optimization objectives [4,5]. A key challenge is the development of an energy management system (EMS) that can optimize fuel cell operation while maintaining the battery state of charge within acceptable bounds [6,7]. Conventional energy management strategies often rely on rule-based logic or predictive control methods that depend on accurate models of the vehicle powertrain [8,9]. However, precise modeling of such systems is inherently difficult due to nonlinear dynamics, time-varying operating conditions, and the presence of unknown or uncertain elements, such as auxiliary loads and environmental factors. Moreover, many model-based approaches require prior knowledge of future driving profiles or disturbance patterns, which may not be readily available in real-time applications [10,11].

Model-based EMS for electric vehicles typically rely on detailed mathematical representations of powertrain components, including fuel cells, batteries, electric motors, and auxiliary loads [12,13]. These models facilitate the application of advanced optimization and control techniques, such as model predictive control and dynamic programming, to compute power allocation strategies that improve energy efficiency and prolong component lifespan [14]. By accurately capturing system dynamics and constraints, model-based EMS can predict future states and proactively regulate power flow [15]. However, the performance of these approaches is highly dependent on the accuracy and completeness of the underlying models, which are often difficult to construct due to nonlinear behavior, parameter uncertainties, and variable operating conditions [16]. In addition, the computational demands associated with solving model-based optimization problems in real time may hinder practical implementation, especially under uncertain or rapidly changing environments [17].

The application of reinforcement learning (RL) to EMS in electric vehicles has gained substantial momentum, driven by the increasing need to manage system complexity and uncertainty without reliance on precise mathematical models [18,19]. In particular, actor–critic frameworks have emerged as a promising class of RL algorithms, as they integrate the processes of value estimation and policy optimization, thereby supporting real-time adaptability to dynamic driving environments and power demand fluctuations [20,21]. Compared to traditional control strategies such as rule-based logic or model predictive control (MPC), actor–critic methods have demonstrated superior performance in capturing nonlinear behaviors and learning optimal control policies through direct interaction with the system [22]. Despite these advantages, conventional RL implementations often rely on pretraining with large and diverse datasets, which can constrain their practicality for real-time deployment [23,24]. Moreover, the inherent dimensional complexity of electric vehicle powertrain systems—characterized by high-dimensional state-action spaces—poses challenges related to computational overhead and algorithmic convergence [25,26]. These limitations have motivated the development of more computationally efficient and data-adaptive approaches that can learn effectively in online settings without extensive offline training.

In response to the limitations of model-based methods and conventional actor–critic architectures, data-driven and model-free control approaches have attracted increasing interest due to their ability to learn system behavior directly from real-time measurements [27]. These methods offer robust adaptability under uncertain and varying conditions without relying on explicit system models. This work proposes an adaptive control framework based on a fuzzy rule emulated network, termed MiFREN [28], for real-time power management in fuel cell and battery electric vehicles. The scheme incorporates two learning networks: MiFREN_m, which estimates unknown system dynamics, and MiFREN_s, which approximates the cost gradient necessary for control optimization. Together, these networks enable online adaptation and optimization without prior knowledge of system parameters.

The main contributions of this work are summarized as follows:

• Unlike conventional reinforcement learning schemes [21,22,23], which typically rely on iterative learning and extensive offline training, the proposed approach employs a co-state network that is trained solely using online data in real time. This design enables the formulation of an optimal energy management controller without requiring a comprehensive dataset or prior knowledge of the entire operational domain. Additionally, the framework inherently avoids the curse of dimensionality, making it well-suited for practical deployment in embedded systems.
• By treating power consumption associated with air conditioning systems, time-varying slopes and road conditions, passenger support systems, and other onboard demands as unknown disturbances, the robustness of the proposed scheme is demonstrated both from a practical perspective and through theoretical analysis.
• From the perspective of energy management as a control system, the desired state of charge (SOC) is formulated as the reference trajectory, while the optimal control input is computed using the proposed control law under full operational constraints.

The remainder of this paper is organized as follows. Section 2 presents the problem formulation, where the electric vehicle with energy management is modeled as a class of discrete-time systems, and the optimal solution is addressed from the perspective of a model-free control approach. Section 3 details the design of the proposed controller along with the corresponding analysis. Section 4 provides validation results and comparative evaluations. Finally, the conclusion is presented in Section 5.

2. Problem Formulation with EV-EMS Framework

2.1. A Class of Control Systems Based on Model-Free EV-EMS

For the purposes of this study, the fuel cell/battery electric vehicle is, without loss of generality and within a discrete-time aspect, briefly represented by the power-flow block diagram shown in Figure 1. The drive block denotes the required power $P_d\left(k\right)$ [kW] at sampling index k, as determined by the vehicle velocity $v\left(k\right)$, physical resistances, and longitudinal dynamics.

In this work, the standard model is employed to express the power as a function of the real-time sampling index k and sampling interval $T_s$ [s] as

$$\begin{array}{ccc}\hfill P_d\left(k\right)& =& {\displaystyle \frac{1}{2}}{C}_d{A}_f{\rho }_a{v}^3\left(k\right)+R_i{m}_{0}v\left(k\right){\displaystyle \frac{v\left(k\right)-v(k-1)}{T_s}}\hfill \\ & & +m_{0}gv\left(k\right)sin\left({\theta }_r\left(k\right)\right)+R_r{m}_{0}gv\left(k\right)sin\left({\theta }_r\left(k\right)\right),\hfill \end{array}$$

(1)

where ${\theta }_r\left(k\right)$ denotes the road slope and other parameters are defined as

Table 1. System parameters.

Parameter	Description	Value	Unit	Remark
$C_d$	Aerodynamic drag coefficient	0.3
$A_f$	Fronted area	2.2508	[${\mathrm{m}}^2$]
${\rho }_a$	Air density	1.293	[k/${\mathrm{m}}^3$]
$m_0$	Curb weight	2024	[kg]
$R_i$	Rotational inertia coefficient	1
$R_r$	Rolling resistance coefficient	0.013
g	Gravity acceleration	9.81	[m/${\mathrm{s}}^2$]
${\eta }_{em}$	Motor efficiency	0.9
${\eta }_m$	Mechanical drive efficiency	0.9
${\eta }_{DC/AC}$	Inverter efficiency	0.95
${\eta }_{DC/DC}$	Converter efficiency	0.95
${\eta }_b$	Coulombic efficiency	0.98
$E_0$	Battery capacity	50	[kWh]

.

With respect to a quasi-static fuel cell model, $P_{fc}\left(k\right)$ [kW] denotes the power output at sampling index k, which reflects the instantaneous hydrogen consumption, expressed as ${\dot{m}}_{H_2}\left(k\right)=f_{fc}\left(P_{fc}\left(k\right)\right)$, where $f_{fc}(-)$ is a generally nonlinear function considered unknown in this work. In light of this relationship, and consistent with insights from related studies, the EMS design in this work is directly based on the principle that a reduction in the required power $P_{fc}(k+1)$ leads to a corresponding decrease in hydrogen consumption.

The lithium-ion battery pack is modeled using its standard equivalent circuit, where the output current $I_b\left(k\right)$ is governed by the battery power $P_b\left(k\right)$, such that

$$I_b\left(k\right)={\displaystyle \frac{V_{oc}-\sqrt{V_{oc}^2-4R_i{P}_b\left(k\right)}}{2R_i}},$$

(2)

where $V_{oc}$ is the open circuit voltage and $R_i$ is the internal resistance. It is evident that under discharge conditions, when $P_b\left(k\right)<{\displaystyle \frac{V_{oc}^2}{4R_i}}$, a proportional relationship between the battery power and output current can be established, provided that the state variation is consistent with the direction of the discharge current. By this mean, the change in $SOC$ or $\dot{SOC}$ can be discretized as

$$\dot{SOC}{|}_{t\in [kTs,(k+1)T_s)}\doteq {\displaystyle \frac{SOC(k+1)-SOC\left(k\right)}{T_s}}=-{\eta }_b^{-\mathrm{sign}\left\{P_b\left(k\right)\right\}}{\displaystyle \frac{P_b\left(k\right)}{3,600E_0}},$$

or

$$SOC(k+1)=SOC\left(k\right)-{\eta }_b^{-\mathrm{sign}\left\{P_b\left(k\right)\right\}}{\displaystyle \frac{P_b\left(k\right)}{3,600E_0}}{T}_s,$$

(3)

where ${\eta }_b$ denotes the Coulombic efficiency of the battery pack, and $E_o$ represents its capacity, as specified in Table 1.

The auxiliary power $P_{aux}\left(k\right)$ accounts for the total power consumption associated with the air conditioning system, passenger support utilities, operational functions, and other onboard loads. In this work, $P_{aux}\left(k\right)$ is treated as an unknown disturbance, and, in contrast to previous works, no estimation models or observers are employed.

The electric motor, integrated into the EV powertrain, has an efficiency ${\eta }_{em}$ generally characterized as a function of the output torque ${\tau }_m$ and rotational speed ${\omega }_m$, such that ${\eta }_{em}=f_{{\eta }_{em}}({\tau }_m,{\omega }_m)$, with $f_{{\eta }_{em}}(·)$ defined experimentally. For simplicity and without loss of generality, ${\eta }_{em}$ is specified as listed in Table 1. The motor is driven via a DC/AC inverter and a mechanical transmission, with their respective efficiencies denoted by ${\eta }_{DC/AC}$ and ${\eta }_m$, also provided in Table 1. Furthermore, the electrical energy generated by the fuel cell is delivered through a DC/DC converter, with its efficiency ${\eta }_{DC/DC}$ likewise specified. Based on these considerations, the power balance for the EV powertrain can be expressed as

$${\left({\eta }_m{\eta }_{em}{\eta }_{DC/AC}\right)}^{-\mathrm{sign}\left\{F_{EV}\left(k\right)\right\}}{P}_d\left(k\right)+P_{aux}\left(k\right)={\eta }_{DC/DC}{P}_{fc}\left(k\right)+P_b\left(k\right),$$

(4)

where

$$\begin{array}{ccc}\hfill F_{EV}\left(k\right)& =& {\displaystyle \frac{1}{2}}{C}_d{A}_f{\rho }_a{v}^2\left(k\right)+R_i{m}_0{\displaystyle \frac{v\left(k\right)-v(k-1)}{T_s}}\hfill \\ & & +m_{0}gsin\left({\theta }_r\left(k\right)\right)+R_r{m}_{0}gsin\left({\theta }_r\left(k\right)\right).\hfill \end{array}$$

(5)

It is worth noting that the dynamics presented in Equations (1)–(4) are simplified representations, provided without loss of generality. The complete formulations can be found in the following references: the EV model and gear system in [21,22], the battery model and state of charge (SOC) dynamics in [20], and the fuel cell and motor models in [16]. Nonetheless, these models and associated dynamics are introduced solely for conceptual illustration and validation purposes, as the proposed scheme, which will be discussed in subsequent sections, is entirely model-free and relies exclusively on real-time data from measurable states. For practical implementation, the relevant physical constraints are summarized as follows:

$$\begin{array}{ccc}\hfill P_{fc}^{min}\le & P_{fc}\left(k\right)& \le P_{fc}^{max},\hfill \\ \hfill -\Delta P_{fc}^M\le & \Delta P_{fc}\left(k\right)& \le \Delta P_{fc}^M,\hfill \\ \hfill SO{C}^{min}\le & SOC\left(k\right)& \le SO{C}^{max},\hfill \\ \hfill P_{aux}^{min}\le & P_{aux}\left(k\right)& \le P_{aux}^{max},\hfill \\ \hfill -P_b^M\le & P_b\left(k\right)& \le P_b^M,\hfill \\ \hfill -I_b^M\le & I_b\left(k\right)& \le I_b^M,\hfill \\ \hfill -P_d^{Gen}\le & P_d\left(k\right)& \le P_d^{Drv},\hfill \end{array}$$

(6)

where all corresponding values are detailed in

Table 2. Constraint parameters.

Limit	Value	Unit	Limit	Value	Unit
$P_{fc}^{min}$	0.25	[kW]	$P_{fc}^{max}$	80	[kW]
$\Delta P_{fc}^M$	9	[kW]	$P_b^M$	50	[kW]
$SO{C}^{min}$	0.2	Per Unit	$SO{C}^{max}$	0.9	Per Unit
$P_d^{Gen}$	80	[kW]	$P_d^{Drv}$	100	[kW]
$P_{aux}^{min}$	0.5	[kW]	$P_{aux}^{max}$	20	[kW]
$I_b^M$	50	[A]

. In this work, $P_d^{Gen}$ and $P_d^{Drv}$ are defined to represent the limitations imposed by motor characteristics, such as the maximum torque during both generator and drive modes, as well as constraints associated with the maximum allowable vehicle velocity and its specifications.

2.2. Characterization of the Optimal Solution

Without loss of generality, the dynamics described in Equations (1)–(4) can be regarded as a class of discrete-time control systems, where $SOC\left(k\right)$ denotes the output $y\left(k\right)$, $P_b\left(k\right)$, $P_{fc}\left(k\right)$, $P_d\left(k\right)$, and $v\left(k\right)$ are measurable states $x\left(k\right)\in {\mathbb{R}}^4$, and $P_{aux}\left(k\right)$ is treated as a disturbance $d\left(k\right)$ that requires neither prediction nor direct measurement. The control effort $u\left(k\right)$ is employed as $\Delta P_{fc}\left(k\right)$ such that

$$u\left(k\right)\doteq \Delta P_{fc}\left(k\right)=P_{fc}\left(k\right)-P_{fc}(k-1).$$

(7)

Recalling (4) with (7), it yields

$$\begin{array}{ccc}\hfill P_b\left(k\right)& =& {\left({\eta }_m{\eta }_{em}{\eta }_{DC/AC}\right)}^{-\mathrm{sign}\left\{F_{EV}\left(k\right)\right\}}{P}_d\left(k\right)+P_{aux}\left(k\right)-{\eta }_{DC/DC}{P}_{fc}\left(k\right),\hfill \\ & =& {\left({\eta }_m{\eta }_{em}{\eta }_{DC/AC}\right)}^{-\mathrm{sign}\left\{F_{EV}\left(k\right)\right\}}{P}_d\left(k\right)-{\eta }_{DC/DC}[u\left(k\right)+P_{fc}(k-1)]\hfill \\ & & +d\left(k\right).\hfill \end{array}$$

(8)

By substituting (8) into (3), the resulting expression becomes

$$\begin{array}{ccc}\hfill SOC(k+1)& =& SOC\left(k\right)-{\displaystyle \frac{{\eta }_b^{-\mathrm{sign}\left\{P_b\left(k\right)\right\}}}{3,600E_0}}{T}_s[-{\eta }_{DC/DC}[u\left(k\right)+P_{fc}(k-1)]\hfill \\ & & +{\left({\eta }_m{\eta }_{em}{\eta }_{DC/AC}\right)}^{-\mathrm{sign}\left\{F_{EV}\left(k\right)\right\}}{P}_d\left(k\right)+d\left(k\right)].\hfill \end{array}$$

(9)

With $SOC(k+1)$ considered as the output, the dynamics in Equation (9) can be generalized as a class of non-affine discrete-time systems, described as

$$y(k+1)=f_N(y\left(k\right),x_1\left(k\right),x_2\left(k\right),x_3\left(k\right),x_4\left(k\right),u\left(k\right),d\left(k\right)),$$

(10)

where $f_N(·)$ denotes an unknown function, and only the state vector $x\left(k\right)$ and the output $y\left(k\right)$ are available by design. In this work, the disturbance $d\left(k\right)$ is treated as an unknown but physically bounded time-varying parameter, representing practical variations such as auxiliary power demands or environmental influences, which remain within realistic operational limits.

To address the challenge of handling unknown system dynamics under practical disturbances, this work introduces time-varying functions constructed based on the equivalent modeling framework proposed in [28]. This approach enables the transformation of the original nonlinear dynamics (10) into an affine equivalent representation, thereby facilitating real-time implementation and control synthesis. The resulting system can be expressed as

$$y(k+1)=f_o\left(k\right)+g\left(k\right)u\left(k\right),$$

(11)

where $f_o\left(k\right)$ is the internal dynamics and $g\left(k\right)$ is the input gain. It is worth mentioning that $f_o\left(k\right)$ and $g\left(k\right)$ are unknown nonlinear functions.

At this stage, the control input $u\left(k\right)$ must be designed to drive the system output $y(k+1)$ in Equation (11) to follow the desired trajectory $y_d(k+1)$. In this work, $y_d(k+1)$ corresponds to the reference state-of-charge denoted as $SO{C}_r(k+1)$, such that

$$y_d(k+1)\doteq SO{C}_r(k+1)={\delta }_{SOC}SO{C}_r\left(k\right)+[1-{\delta }_{SOC}]SOC\left(k\right),$$

(12)

where ${\delta }_{SOC}<1$ is a design parameter.

Therefore, the problem is formulated for the control systems when the tracking error $e\left(k\right)$ is defined as

$$e\left(k\right)=y\left(k\right)-y_d\left(k\right).$$

(13)

By utilizing (12) and (11) with (13), it yields

$$\begin{array}{ccc}\hfill e(k+1)& =& y(k+1)-y_d(k+1),\hfill \\ & =& f_o\left(k\right)+g\left(k\right)u\left(k\right)-y_d(k+1),\hfill \\ & =& f\left(k\right)+g\left(k\right)u\left(k\right),\hfill \end{array}$$

(14)

where

$$f\left(k\right)=f_o\left(k\right)-{\delta }_{SOC}SO{C}_r\left(k\right)+[1+{\delta }_{SOC}]SOC\left(k\right).$$

(15)

It is important to emphasize that the affine discrete-time system in (14) has been derived directly to represent the error dynamics, without any transformation from the original plant. Furthermore, the functions $f\left(k\right)$ and $g\left(k\right)$ are assumed to be unknown nonlinear functions.

In accordance with the optimal control design, the long-term cost function $J\left(k\right)$ is defined in this work as

$$J\left(k\right)=\sum _{i=k}^{\infty }r\left(i\right),$$

(16)

where $r\left(i\right)$ is the unity function given as

$$r\left(i\right)=q_r{e}^2\left(i\right)+p_r{u}^2\left(i\right).$$

(17)

$q_r$ and $p_r$ are positive constants. It is clear that $r\left(i\right)=0$ when only $e\left(i\right)=0$ and $u\left(i\right)=0$.

By using $J\left(k\right)$ in (16), we obtain

$$\begin{array}{ccc}\hfill J\left(k\right)& =& r\left(k\right)+\sum _{i=k+1}^{\infty }r\left(i\right),\hfill \\ & =& r\left(k\right)+J(k+1).\hfill \end{array}$$

(18)

To determine the optimal control law, it requires the constraint such that ${\displaystyle \frac{\mathsf{\partial }J\left(k\right)}{\mathsf{\partial }u\left(k\right)}}=0$. That leads to

$$\begin{array}{ccc}\hfill 0& =& {\displaystyle \frac{\mathsf{\partial }r\left(k\right)}{\mathsf{\partial }u\left(k\right)}}+{\displaystyle \frac{\mathsf{\partial }J(k+1)}{\mathsf{\partial }u\left(k\right)}},\hfill \\ & =& 2p_{r}u\left(k\right)+{\displaystyle \frac{\mathsf{\partial }J(k+1)}{\mathsf{\partial }e(k+1)}}{\displaystyle \frac{\mathsf{\partial }e(k+1)}{\mathsf{\partial }u\left(k\right)}}.\hfill \end{array}$$

(19)

Let us recall the error dynamics in (14); thus, the ideal optimal control law $u^{*}\left(k\right)$ is given as

$$u^{*}\left(k\right)=-{\displaystyle \frac{1}{2p_r}}g\left(k\right){\displaystyle \frac{\mathsf{\partial }J(k+1)}{\mathsf{\partial }e(k+1)}}.$$

(20)

The controller in Equation (20) is impractical for implementation, as it requires knowledge of the unknown function $g\left(k\right)$ and the future value of the cost function gradient ${\displaystyle \frac{\mathsf{\partial }J(k+1)}{\mathsf{\partial }e(k+1)}}$. To address this limitation, two key components are proposed: (i) a data-driven scheme is employed to estimate the unknown function $g\left(k\right)$, denoted as $\widehat{g}\left(k\right)$, and (ii) an approximation of the co-state, $\widehat{\lambda }\left(k\right)$, is introduced to predict ${\displaystyle \frac{\mathsf{\partial }J(k+1)}{\mathsf{\partial }e(k+1)}}$. Accordingly, the practical controller developed in this work, is formulated as

$$u\left(k\right)=-{\displaystyle \frac{1}{2p_r}}\widehat{g}\left(k\right)\widehat{\lambda }\left(k\right).$$

(21)

It is worth noting that the control law in Equation (21) involves two time-varying parameters, which will be developed in the following sections, subject to the practical constraints defined in Equation (6).

3. Controller as EMS with MiFREN-Estimators

By considering the control law in Equation (21), it is evident that two main components are required for its implementation: $\widehat{g}\left(k\right)$ and $\widehat{\lambda }\left(k\right)$. In this work, the first adaptive network, denoted as MiFREN_m, is constructed to estimate the unknown function $\widehat{g}\left(k\right)$ based on a data-driven approach. Subsequently, the second network, MiFREN_s, is designed to generate the co-state approximation $\widehat{\lambda }\left(k\right)$.

3.1. Dynamic Equivalent Model

An adaptive network, referred to as MiFREN_m, is employed to estimate the error dynamics described in Equation (14), based on the network architecture illustrated in Figure 2. Accordingly, the equivalent model $\widehat{y}(k+1)$ is formulated as

$$\begin{array}{ccc}\hfill \widehat{y}(k+1)& =& \widehat{f}\left(k\right)+\widehat{g}\left(k\right)u\left(k\right),\hfill \\ & =& {\beta }_f^T\left(k\right)\phi \left(k\right)+{\beta }_g^T\left(k\right)\phi \left(k\right)u\left(k\right),\hfill \end{array}$$

(22)

where ${\beta }_f\left(k\right)\in {\mathbb{R}}^N$ and ${\beta }_g\left(k\right)\in {\mathbb{R}}^N$ are weight parameters of MiFREN and N is the number of IF–THEN rules. $\phi \left(k\right)\in {\mathbb{R}}^N$ is a regression vector for inputs $y\left(k\right)$ and $y(k-1)$. By setting N as negative, Z as zero, and P as positive membership functions as Figure 2, we have $N=9$. Furthermore, it is worth emphasizing that $\widehat{g}\left(k\right)$ required by the control law (21) is determined as

$$\widehat{g}\left(k\right)={\beta }_g^T\left(k\right)\phi \left(k\right).$$

(23)

Thereafter, the learning law is derived to tune parameters ${\beta }_f\left(k\right)$ and ${\beta }_g\left(k\right)$ with the cost function $E_e(k+1)$ defined as

$$E_e(k+1)={\displaystyle \frac{1}{2}}{\tilde{e}}^2(k+1),$$

(24)

where

$$\tilde{e}(k+1)=y(k+1)-\widehat{y}(k+1).$$

(25)

By utilizing the gradient search, learning laws for ${\beta }_f\left(k\right)$ and ${\beta }_g\left(k\right)$ are expressed as

$${\beta }_f(k+1)={\beta }_f(k+1)-{\eta }_e{\displaystyle \frac{\mathsf{\partial }{E}_e(k+1)}{\mathsf{\partial }{\beta }_f(k+1)}},$$

(26)

and

$${\beta }_g(k+1)={\beta }_g(k+1)-{\eta }_e{\displaystyle \frac{\mathsf{\partial }{E}_e(k+1)}{\mathsf{\partial }{\beta }_g(k+1)}},$$

(27)

respectively, where ${\eta }_e$ is the learning rate. Let us employ the chain rule over the estimated error (22); thus, we obtain

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathsf{\partial }{E}_e(k+1)}{\mathsf{\partial }{\beta }_f(k+1)}}& =& {\displaystyle \frac{\mathsf{\partial }{E}_e(k+1)}{\mathsf{\partial }\tilde{e}(k+1)}}{\displaystyle \frac{\mathsf{\partial }\tilde{e}(k+1)}{\mathsf{\partial }\widehat{y}(k+1)}}{\displaystyle \frac{\mathsf{\partial }\widehat{y}(k+1)}{\mathsf{\partial }{\beta }_f(k+1)}},\hfill \\ & =& -\tilde{e}(k+1)\phi \left(k\right),\hfill \end{array}$$

(28)

and

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\mathsf{\partial }{E}_e(k+1)}{\mathsf{\partial }{\beta }_g(k+1)}}& =& {\displaystyle \frac{\mathsf{\partial }{E}_e(k+1)}{\mathsf{\partial }\tilde{e}(k+1)}}{\displaystyle \frac{\mathsf{\partial }\tilde{e}(k+1)}{\mathsf{\partial }\widehat{y}(k+1)}}{\displaystyle \frac{\mathsf{\partial }\widehat{y}(k+1)}{\mathsf{\partial }{\beta }_g(k+1)}},\hfill \\ & =& -\tilde{e}(k+1)u\left(k\right)\phi \left(k\right).\hfill \end{array}$$

(29)

By substitution (26) and (27) with (28) and (29), respectively, learning laws are derived as

$${\beta }_f(k+1)={\beta }_f\left(k\right)+{\eta }_e\tilde{e}(k+1)\phi \left(k\right),$$

(30)

and

$${\beta }_g(k+1)={\beta }_g\left(k\right)+{\eta }_e\tilde{e}(k+1)u\left(k\right)\phi \left(k\right).$$

(31)

Next, the convergence of the proposed learning laws (30) and (31) will be analysed through the learning rate ${\eta }_e$ by the following Lemma.

Lemma 1. 

By utilizing the tracking-error equivalent model (22) with the learning laws (30) and (31) and boundedness of the control effort and weight parameters such that $\left|u\left(k\right)\right|\le u_M$, $\left|\right|{\beta }_f\left(k\right)\left|\right|\le {\beta }_f^M$ and $\left|\right|{\beta }_g\left(k\right)\left|\right|\le {\beta }_g^M$, the convergence of $\tilde{e}(k+1)$ is guaranteed when the learning rate ${\eta }_e$ is employed as the time-varying leaning rate ${\eta }_e\left(k\right)$ given as

$${\eta }_e\left(k\right)={\displaystyle \frac{{\gamma }_e}{[1+|u\left(k\right)u(k-1)\left|\right]{\phi }^T\left(k\right)\phi (k-1)}},$$

(32)

where

$$0<{\gamma }_e<1.$$

(33)

Proof. 

Let us recall the universal function approximation of MiFREN; thus, the dynamics of the tracking error (14) can be rewritten by ideal weight parameters ${\beta }_f^{*}$ and ${\beta }_g^{*}$ as

$$y(k+1)={\phi }^T\left(k\right){\beta }_f^{*}+{\phi }^T\left(k\right){\beta }_g^{*}u\left(k\right)+{\epsilon }_e\left(k\right),$$

(34)

when ${\epsilon }_e\left(k\right)$ is a bounded residual error $|{\epsilon }_e\left(k\right)|\le {\epsilon }_{eM}$. By using (25) with (22) and (34), it yields

$$\begin{array}{ccc}\hfill \tilde{e}(k+1)& =& y(k+1)-\widehat{y}(k+1),\hfill \\ & =& {\phi }^T\left(k\right)[{\beta }_f^{*}-{\beta }_f\left(k\right)]+{\phi }^T\left(k\right)[{\beta }_g^{*}-{\beta }_g\left(k\right)]u\left(k\right)+{\epsilon }_e\left(k\right),\hfill \\ & =& {\phi }^T\left(k\right){\tilde{\beta }}_f\left(k\right)+{\phi }^T\left(k\right){\tilde{\beta }}_g\left(k\right)u\left(k\right)+{\epsilon }_e\left(k\right),\hfill \end{array}$$

(35)

where ${\tilde{\beta }}_f\left(k\right)={\beta }_f^{*}-{\beta }_f\left(k\right)$ and ${\tilde{\beta }}_g\left(k\right)={\beta }_g^{*}-{\beta }_g\left(k\right)$.

Let us recall learning laws (30) and (31); they lead to

$${\tilde{\beta }}_f(k+1)={\tilde{\beta }}_f\left(k\right)-{\eta }_e\tilde{e}(k+1)\phi \left(k\right),$$

(36)

and

$${\tilde{\beta }}_g(k+1)={\tilde{\beta }}_g\left(k\right)-{\eta }_e\tilde{e}(k+1)u\left(k\right)\phi \left(k\right),$$

(37)

respectively. Utilizing one-step back to (36) and (37) and substituting into (35), we obtain

$$\begin{array}{ccc}\hfill \tilde{e}(k+1)& =& {\phi }^T\left(k\right)[{\tilde{\beta }}_f(k-1)-{\eta }_e\tilde{e}\left(k\right)\phi (k-1)]\hfill \\ & & +{\phi }^T\left(k\right)[{\tilde{\beta }}_g(k-1)-{\eta }_e\tilde{e}\left(k\right)u(k-1)\phi (k-1)]u\left(k\right)+{\epsilon }_e\left(k\right),\hfill \\ & =& -{\eta }_e[1+u\left(k\right)u(k-1)]{\phi }^T\left(k\right)\phi (k-1)\tilde{e}\left(k\right)\hfill \\ & & +{\phi }^T\left(k\right){\tilde{\beta }}_f(k-1)+{\phi }^T\left(k\right){\tilde{\beta }}_g(k-1)u\left(k\right)+{\epsilon }_e\left(k\right),\hfill \\ & =& A_{\tilde{e}}\left(k\right)\tilde{e}\left(k\right)+B_{\tilde{e}}\left(k\right),\hfill \end{array}$$

(38)

where

$$A_{\tilde{e}}\left(k\right)=-{\eta }_e[1+u\left(k\right)u(k-1)]{\phi }^T\left(k\right)\phi (k-1),$$

(39)

and

$$B_{\tilde{e}}\left(k\right)={\phi }^T\left(k\right){\tilde{\beta }}_f(k-1)+{\phi }^T\left(k\right){\tilde{\beta }}_g(k-1)u\left(k\right)+{\epsilon }_e\left(k\right).$$

(40)

For the setting of membership functions $\mu (-)\in [0,1]$, it leads to $\left|\right|\phi \left(k\right)\left|\right|\le N$. Thus, it is clear that $B_{\tilde{e}}\left(k\right)$ is bounded as

$$B_{\tilde{e}}\left(k\right)\le 2N[{\beta }_f^M+{\beta }_g^M{u}_M]+{\epsilon }_{eM}.$$

(41)

Thereafter, by substitution ${\eta }_e$ in (39) with the time-varying leaning rate in (32), we have

$$A_{\tilde{e}}\left(k\right)=-{\gamma }_e{\displaystyle \frac{[1+u\left(k\right)u(k-1)]{\phi }^T\left(k\right)\phi (k-1)}{[1+|u\left(k\right)u(k-1)\left|\right]{\phi }^T\left(k\right)\phi (k-1)}}.$$

(42)

Considering the setting of ${\gamma }_e$ in (33), it is obvious that $|A_{\tilde{e}}\left(k\right)|<1$. Thus, $\tilde{e}(k+1)$ is a convergence sequence. The proof is completed. □

At this point, the estimation $\widehat{g}\left(k\right)$ required by the proposed control law (21) has been derived. Therefore, the co-state $\widehat{\lambda }\left(k\right)$ will be established next.

3.2. Co-State Estimation

In this section, the co-state network is constructed by another MiFREN with the network architecture depicted in Figure 3. The tracking error $e\left(k\right)$ and $u\left(k\right)$ are inputs, and the output is the estimated co-state formulated as

$$\widehat{\lambda }\left(k\right)={\beta }_{\lambda }^T\left(k\right){\phi }_{\lambda }\left(k\right),$$

(43)

where ${\beta }_{\lambda }\left(k\right)\in {\mathbb{R}}^N$ is the weight vector and $\phi \left(k\right)\in {\mathbb{R}}^N$ is the input-regression vector.

Let us recall the definition of the co-state $\lambda \left(k\right)$ and its discrete-time approximation such that

$$\lambda \left(k\right)\doteq {\displaystyle \frac{\mathsf{\partial }J(k+1)}{\mathsf{\partial }e(k+1)}}\approx {\displaystyle \frac{J(k+1)-J\left(k\right)}{e(k+1)-e\left(k\right)}}={\displaystyle \frac{\Delta J\left(k\right)}{\Delta e\left(k\right)}},$$

(44)

where $\Delta e\left(k\right)\ne 0$. By utilizing (18), it yields

$$\Delta J\left(k\right)=-r\left(k\right).$$

(45)

Thus, the target co-state ${\lambda }_d\left(k\right)$ employed for tuning the parameter ${\beta }_{\lambda }\left(k\right)$ is formulated as

$${\lambda }_d\left(k\right)=-{\displaystyle \frac{r\left(k\right)}{\Delta e\left(k\right)}}.$$

(46)

Therefore, the error $e_{\lambda }\left(k\right)$ is defined as

$$e_{\lambda }\left(k\right)=\widehat{\lambda }\left(k\right)-{\lambda }_d\left(k\right).$$

(47)

Thus, the learning law of ${\beta }_{\lambda }\left(k\right)$ is given as

$${\beta }_{\lambda }(k+1)={\beta }_{\lambda }\left(k\right)-{\eta }_{\lambda }{\phi }_{\lambda }\left(k\right)e_{\lambda }\left(k\right),$$

(48)

where ${\eta }_{\lambda }$ is the learning rate.

Lemma 2. 

By utilizing the learning law (48), the convergence of the weight parameter ${\beta }_{\lambda }\left(k\right)$ is guaranteed when the learning rate ${\eta }_{\lambda }$ is employed as the time-varying variable ${\eta }_{\lambda }\left(k\right)$ given as

$${\eta }_{\lambda }\left(k\right)={\displaystyle \frac{{\gamma }_{\lambda }}{{\Lambda }_{\lambda }\left(k\right)}},$$

(49)

where

$$0<{\gamma }_{\lambda }<2,$$

(50)

and ${\Lambda }_{\lambda }\left(k\right)$ is an eigenvalue of ${\mathsf{\Psi }}_{\lambda }\left(k\right)$ defined by

$${\mathsf{\Psi }}_{\lambda }\left(k\right)=\left[\begin{array}{cccc}{\phi }_{\lambda ,1}\left(k\right){\phi }_{\lambda ,1}\left(k\right)& {\phi }_{\lambda ,1}\left(k\right){\phi }_{\lambda ,2}\left(k\right)& \cdots & {\phi }_{\lambda ,1}\left(k\right){\phi }_{\lambda ,N}\left(k\right)\\ {\phi }_{\lambda ,2}\left(k\right){\phi }_{\lambda ,1}\left(k\right)& {\phi }_{\lambda ,2}\left(k\right){\phi }_{\lambda ,2}\left(k\right)& \cdots & {\phi }_{\lambda ,2}\left(k\right){\phi }_{\lambda ,N}\left(k\right)\\ ⋮& ⋮& ⋮& ⋮\\ {\phi }_{\lambda ,N}\left(k\right){\phi }_{\lambda ,1}\left(k\right)& {\phi }_{\lambda ,N}\left(k\right){\phi }_{\lambda ,2}\left(k\right)& \cdots & {\phi }_{\lambda ,N}\left(k\right){\phi }_{\lambda ,N}\left(k\right)\end{array}\right].$$

(51)

Proof. 

According to the property of MiFREN, it exists the ideal weight parameter ${\beta }_{\lambda }^{*}$ as

$${\lambda }_d\left(k\right)={\beta }_{\lambda }^{*T}{\phi }_{\lambda }\left(k\right)+{\epsilon }_{\lambda }\left(k\right),$$

(52)

where ${\epsilon }_{\lambda }\left(k\right)$ is a bounded residual error such that $|{\epsilon }_{\lambda }\left(k\right)|<{\epsilon }_{\lambda M}$. Let us substitute (43) and (52) into (47), thus, we have

$$\begin{array}{ccc}\hfill e_{\lambda }\left(k\right)& =& {[{\beta }_{\lambda }\left(k\right)-{\beta }_{\lambda }^{*}]}^T{\phi }_{\lambda }\left(k\right)-{\epsilon }_{\lambda }\left(k\right),\hfill \\ & =& -{\tilde{\beta }}_{\lambda }^T\left(k\right){\phi }_{\lambda }\left(k\right)-{\epsilon }_{\lambda }\left(k\right),\hfill \end{array}$$

(53)

where ${\tilde{\beta }}_{\lambda }\left(k\right)={\beta }_{\lambda }^{*}-{\beta }_{\lambda }\left(k\right)$. By substitution (53) into (48), it yields

$${\beta }_{\lambda }(k+1)={\beta }_{\lambda }\left(k\right)+{\eta }_{\lambda }{\phi }_{\lambda }\left(k\right){\tilde{\beta }}_{\lambda }^T\left(k\right){\phi }_{\lambda }\left(k\right)+{\epsilon }_{\lambda }\left(k\right){\eta }_{\lambda }{\phi }_{\lambda }\left(k\right),$$

(54)

or

$${\tilde{\beta }}_{\lambda }(k+1)={\tilde{\beta }}_{\lambda }\left(k\right)-{\eta }_{\lambda }{\phi }_{\lambda }\left(k\right){\tilde{\beta }}_{\lambda }^T\left(k\right){\phi }_{\lambda }\left(k\right)-{\eta }_{\lambda }{\epsilon }_{\lambda }\left(k\right){\phi }_{\lambda }\left(k\right).$$

(55)

Utilizing matrix algebra, we obtain

$$\begin{array}{ccc}\hfill {\tilde{\beta }}_{\lambda }(k+1)& =& {\tilde{\beta }}_{\lambda }\left(k\right)-{\eta }_{\lambda }{\mathsf{\Psi }}_{\lambda }\left(k\right){\tilde{\beta }}_{\lambda }\left(k\right)-{\eta }_{\lambda }{\epsilon }_{\lambda }\left(k\right){\phi }_{\lambda }\left(k\right),\hfill \\ & =& [1-{\eta }_{\lambda }{\Lambda }_{\lambda }\left(k\right)]{\tilde{\beta }}_{\lambda }\left(k\right)-{\eta }_{\lambda }{\epsilon }_{\lambda }\left(k\right){\phi }_{\lambda }\left(k\right),\hfill \\ & =& A_{\lambda }\left(k\right){\tilde{\beta }}_{\lambda }\left(k\right)+B_{\lambda }\left(k\right),\hfill \end{array}$$

(56)

where

$$A_{\lambda }\left(k\right)=1-{\eta }_{\lambda }{\Lambda }_{\lambda }\left(k\right),$$

(57)

and

$$B_{\lambda }\left(k\right)=-{\eta }_{\lambda }{\epsilon }_{\lambda }\left(k\right){\phi }_{\lambda }\left(k\right).$$

(58)

By setting membership functions and the bounded residual error ${\epsilon }_{\lambda }\left(k\right)$, it is clear that $B_{\lambda }\left(k\right)$ in (58) is also bounded. Furthermore, by recalling the learning rate ${\eta }_{\lambda }$ with (49), it is obvious that $-1<A_{\lambda }\left(k\right)<1$. Thus, ${\tilde{\beta }}_{\lambda }(k+1)$ in (56) is a convergence sequence. The proof is completed. □

For clarity, the block diagram of the proposed scheme is shown in Figure 4, illustrating the flow of key signals within the control structure. Upon receiving all measurable states and output variables, the MiFREN_m network estimates the input gain $\widehat{g}\left(k\right)$, while MiFREN_s computes the co-state $\widehat{\lambda }\left(k\right)$. The control law is then executed based on the reference state of charge $SO{C}_r\left(k\right)$ while accounting for unknown disturbances from driving demand $P_d\left(k\right)$ and auxiliary load $P_{aux}\left(k\right)$, which reflect road and environmental conditions. Here, $P_{fc}\left(k\right)$, $P_d\left(k\right)$, vehicle speed $v\left(k\right)$, and battery power $P_b\left(k\right)$ are treated as measurable states, with $SOC\left(k\right)$ as the output. In contrast, $P_{aux}\left(k\right)$ and the road gradient ${\theta }_r\left(k\right)$ are unmeasurable and modeled as external disturbances. The learning laws for both networks operate directly along the time index k, without iterative updates, ensuring real-time applicability.

4. Validation and Comparative Results

4.1. Validation Results

To implement the proposed scheme, the membership functions for all inputs of both MiFREN_m and MiFREN_s, as illustrated in Figure 5, are designed in accordance with the constraints specified in Equation (6) and Table 2. It is worth noting that, for comparative purposes, the battery power $P_b\left(k\right)$ [kW] is reformulated as the stored energy within the battery, $E_b\left(k\right)$ [kWh], defined as

$$E_b\left(k\right)=E_b(k-1)+P_b\left(k\right){\displaystyle \frac{T_s}{3,600}}.$$

(59)

The velocity trajectory shown in Figure 6 is constructed by sequentially combining standard driving cycles, including UDDS, HWFET, ArtMw150, FTP, ArtRoad, and WLTP2 [2,12,20,25]. These datasets are recorded with a sampling interval of 1 [s]; thus, the sampling time is defined as $T_s=1$ [s]. Based on the selected velocity profiles and corresponding road conditions, the resulting power demand trajectory $P_d\left(k\right)$ is computed and illustrated in Figure 7.

To employ the proposed scheme, the initial battery energy is set to $E_b\left(1\right)=15.5$ [kWh], which determines the initial value of $SOC$ as

$$SOC\left(1\right)={\displaystyle \frac{E_b\left(1\right)}{E_0}}={\displaystyle \frac{15.5}{50}}=0.31.$$

(60)

Subsequently, $SO{C}_r(k+1)$ is generated in real time according to the relation in (12), with ${\delta }_{SOC}=0.95$. Therefore, the learning laws of MiFREN_m, given in Equations (30) and (31), and MiFREN_s, given in Equation (48), are employed with learning rates ${\gamma }_e=0.5$ and ${\gamma }_{\lambda }=0.5$, respectively.

Under the proposed controller, the time-varying behavior of the fuel cell power $P_{fc}\left(k\right)$ is shown in Figure 8, with the corresponding control effort $u\left(k\right)$, represented as $\Delta P_{fc}\left(k\right)$, illustrated in Figure 9. The results indicate that the controller successfully maintains the fuel cell power output at a nearly constant level for the majority of the operation period, while strictly adhering to the lower and upper power constraints defined in Table 2. The battery energy evolution $E_b\left(k\right)$, shown in Figure 10, reveals that despite starting from a low initial energy level, the controller is able to sustain an adequate battery charge throughout the drive cycle. Furthermore, Figure 11 compares the actual state of charge $SOC\left(k\right)$ with the reference trajectory $SO{C}_r\left(k\right)$, demonstrating accurate tracking performance under varying operational conditions. To highlight the adaptive behavior of the proposed networks, Figure 12 illustrates the evolution of the time-varying learning rates ${\eta }_e\left(k\right)$ and ${\eta }_{\lambda }\left(k\right)$, which govern the online adaptation of MiFREN_m and MiFREN_s, respectively. Nonetheless, it is worth remarking that the fluctuations observed in the reference state of charge $SO{C}_r\left(k\right)$ are closely associated with variations in the co-state learning rate ${\eta }_{\lambda }\left(k\right)$, as illustrated in Figure 12. This correlation highlights the effectiveness of the proposed scheme, wherein the adaptive learning mechanism enables the controller to accurately manage the behavior of the actual $SOC\left(k\right)$—particularly under high charge conditions. These learning rates dynamically respond to system variations, contributing to the robustness and real-time adaptability of the control strategy.

4.2. Comparative Results

4.2.1. Comparative Controller A

The comparative controller A is developed based on the concept of maintaining $SOC\left(k\right)$ approximately constant, following the algorithm proposed in [25]. All design parameters are selected in accordance with Table 6 of [25], except that the minibatch size is increased to 256, as suggested by the numerical formulation in [12], to enhance performance. This adjustment is justified by the extended validation period in this work, which spans 6 h compared to only 0.67 h in [25].

In this case, the fuel cell power output $P_{fc}\left(k\right)$ under Controller A is depicted in Figure 13. In comparison with the results obtained using the proposed controller (Figure 8), it is apparent that Controller A induces more pronounced high-frequency fluctuations in $P_{fc}\left(k\right)$. This behavior may impose additional stress on the fuel cell system and could potentially reduce its operational lifespan. Figure 14 presents the battery energy trajectory $E_b\left(k\right)$, indicating that the controller maintains the battery around its nominal energy level throughout the driving cycle. Additionally, Figure 15 shows the evolution of the state of charge $SOC\left(k\right)$, demonstrating that Controller A achieves its design objective of regulating $SOC\left(k\right)$ near a constant value. Nonetheless, it is important to note that this approach may sacrifice smooth fuel cell operation in favor of maintaining a steady battery charge.

4.2.2. Comparative Controller B

To address the issue of high-frequency variations in $P_{fc}\left(k\right)$, the soft actor–critic scheme developed in [2] is adopted as Controller B. All design parameters are chosen in accordance with Table 4 of [2], except for the discount factor, which is set to $\gamma =0.95$ to optimize performance for the current validation case. It is important to note that this approach requires additional information, such as a thermal load model and a detailed description of the air conditioning system, which together allow for a well-defined formulation of $P_{aux}\left(k\right)$. However, in the context of this validation test, $P_{aux}\left(k\right)$ is treated as an unknown disturbance. Consequently, the learning algorithm proposed in [20] is employed to train both the actor and critic networks, with the learning rates reselected to 0.001.

As a result, the power output $P_{fc}\left(k\right)$ generated by Controller B is illustrated in Figure 16. In comparison with Controller A, it is evident that the high-frequency components have been significantly suppressed, indicating smoother fuel cell operation. However, some transient variations are observed during the initial hour of operation, likely due to the adaptation period of the learning-based controller. Furthermore, Figure 17 presents the evolution of $SOC\left(k\right)$ under Controller B. While the state of charge is successfully maintained within the specified operational limits, the $SOC\left(k\right)$ trajectory exhibits slightly larger oscillations compared to those obtained with the proposed controller and Controller A.

5. Conclusions

This work has presented a data-driven energy management strategy for fuel cell and battery electric vehicles, formulated as a constrained optimal control problem. The proposed approach has integrated a co-state network with online learning to estimate the optimal control input in real time, eliminating the need for prior system modeling or complete operational datasets and avoiding dimensionality challenges common in many learning-based methods. Robustness has been achieved by treating unknown and time-varying disturbances—such as auxiliary power consumption, road slope variations, and passenger-related loads—as bounded system uncertainties. The controller has aimed to track a desired SOC trajectory while respecting all physical constraints and adapting effectively to changing operating conditions without requiring offline training or predictive models. Validation results obtained over a six-hour composite driving profile—comprising UDDS, HWFET, ArtMw150, FTP, ArtRoad, and WLTP2 cycles—have demonstrated the following:

• Stable battery operation with SOC maintained within a practical range;
• A significant reduction in high-frequency fluctuations of fuel cell power output compared to benchmark controllers;
• Improved overall energy efficiency relative to constant SOC and soft actor–critic methods.

Comparative analysis against a constant SOC controller and a soft actor–critic algorithm has further confirmed the proposed scheme’s advantages in terms of stability, robustness to unknown disturbances, and real-time computational feasibility. Building upon the benefits of the online learning framework developed in this work, integration with traffic and route information—by leveraging vehicle-to-everything (V2X) and route-based forecasting—has been identified as a promising direction for future research to further enhance predictive capabilities and energy efficiency under real-world driving conditions.