Co-Optimization of Capacity and Operation for Battery-Hydrogen Hybrid Energy Storage Systems Based on Deep Reinforcement Learning and Mixed Integer Programming

Abstract

The hybrid energy storage system (HESS) that combines battery with hydrogen storage exploits complementary power/energy characteristics, but most studies optimize capacity and operation separately, leading to suboptimal overall performance. To address this issue, this paper proposes a bi-level co-optimization framework that integrates deep reinforcement learning (DRL) and mixed integer programming (MIP). The outer layer employs the TD3 algorithm for capacity configuration, while the inner layer uses the Gurobi solver for optimal operation under constraints. On a standalone PV–wind–load-HESS system, the method attains near-optimal quality at dramatically lower runtime. Relative to GA + Gurobi and PSO + Gurobi, the cost is lower by 4.67% and 1.31%, while requiring only 0.52% and 0.58% of their runtime; compared with a direct Gurobi solve, the cost remains comparable while runtime decreases to 0.07%. Sensitivity analysis further validates the model’s robustness under various cost parameters and renewable energy penetration levels. These results indicate that the proposed DRL–MIP cooperation achieves near-optimal solutions with orders of magnitude speedups. This study provides a new DRL–MIP paradigm for efficiently solving strongly coupled bi-level optimization problems in energy systems.

1. Introduction

The hybrid energy storage system (HESS) integrates power-based energy storage technologies, such as supercapacitors, flywheels, and batteries, with energy-based energy storage technologies, such as hydrogen, thermal, compressed-air, and gravity. This combined approach balances fast power response with long-duration energy supply, addressing the shortcomings of single storage technologies. By adjusting power output across different time scales, HESS improves renewable energy utilization and strengthens the overall stability of energy networks. With properly optimized resource allocation, HESS not only reduces overall system investment costs but also extends the service life of storage devices. Owing to these technical and economic merits, HESS has emerged as a prominent research focus in the energy sector and has been widely deployed in renewable energy systems, active distribution networks, electric vehicles, and integrated energy systems.

Existing studies on HESS optimization primarily focus on three aspects: capacity (sizing) optimization, operation (energy management, scheduling, and power allocation) optimization, and the co-optimization of both capacity and operation. This paper systematically reviews and analyzes the current literature from these three perspectives.

(1)

Capacity Optimization

References [1,2,3,4,5] investigated the application of HESS in grid-connected photovoltaic (PV), wind, and wave energy systems, with a focus on capacity optimization to smooth power fluctuations. Reference [6] proposed a multi-objective capacity optimization approach for standalone wind power systems, jointly considering economic performance, reliability, and energy consumption rate. References [7,8] explored HESS deployment in renewable energy systems, optimizing system sizing in response to load demands under varying climatic conditions. Reference [9] examined the role of HESS in islanded microgrids, configuring storage capacity to minimize both overall operating costs and the flexibility-deficiency rate. References [10,11,12,13] focused on integrated energy systems, where capacity optimization of HESS improved system economics and stability while mitigating wind/PV power curtailment and voltage fluctuations.

In summary, HESS capacity optimization models primarily focus on economy, reliability, and equipment operating conditions; however, their operational strategies are mostly rule-based deterministic methods, lacking systematic operation optimization modeling.

(2)

Operation Optimization

References [14,15,16] investigated energy management strategies for HESS in electric vehicles, aiming to improve energy efficiency and driving range. References [17,18,19,20,21,22] focused on renewable energy grid-integration scenarios, optimizing energy management to enhance system stability. References [23,24,25,26,27,28,29,30,31,32,33,34,35] explored the optimal operation of HESS in microgrids to achieve global energy management objectives. References [36,37,38] discussed the scheduling optimization methods of HESS in integrated energy systems.

These studies generally assume that storage capacity is predetermined, concentrating instead on optimizing decision variables at the operational level.

The aforementioned studies consider capacity configuration and operation optimization separately; however, there is a significant coupling relationship between these two aspects. Consequently, an increasing number of studies in recent years have focused on this.

(3)

Co-Optimization of Capacity and Operation

The collaborative optimization of capacity and operation for HESS is inherently a complex mixed-integer nonlinear optimization problem, posing high complexity in both modeling and solution processes. There are three major challenges inherent in the strongly coupled capacity–operation problem of hybrid energy storage systems (HESS):

(1)

Strong coupling between long-term investment and short-term operation;

(2)

High nonlinearity and mixed-integer characteristics;

(3)

Environmental uncertainty and computational complexity.

To this end, this Section of the literature review focuses on model construction methods and solution strategies for the co-optimization of HESS capacity and operation, with a summary of relevant studies presented in

Table 1. Literature Review on Co-Optimization of Capacity and Operation.

Reference	Co-Optimization Module	Solution Methods	Whether Scenario Uncertainty Is Considered	Whether Energy Storage Degradation Is Considered
[39]	Capacity level: nonlinear programming (NLP); Operation level: dynamic programming model (DP)	Capacity level: multi-start space-reduction algorithm; Operation level: dynamic programming algorithm; Interactive iteration between two levels	No	Yes, only battery
[40]	Capacity level: multi-objective NLP; Operation level: DP	Capacity level: multi-objective grey wolf optimization algorithm; Operation level: dynamic programming algorithm; Interactive iteration between two levels	No	Yes, only battery
[41]	Capacity level: NLP; Operation level: mixed-integer linear programming (MILP)	Capacity level: improved PSO–GA hybrid algorithm; Operation level: commercial solver; Interactive iteration between two levels	Partially considered by typical day data based on clustering	No
[42]	Capacity level: NLP; Operation level: MILP	Capacity level: PSO algorithm; Operation level: commercial solver; Interactive iteration between two levels	Partially considered by typical day data based on clustering	Yes, only battery
[43]	Capacity level: multi-objective NLP; Operation level: MILP	Capacity level: NSGA-III algorithm; Operation level: commercial solver; Interactive iteration between two levels	Partially considered by typical day data based on clustering	No
[44]	Capacity level: NLP; Operation level: mixed-integer nonlinear programming (MINLP)	Capacity level: sequential quadratic programming algorithm; Operation level: commercial solver; Interactive iteration between two levels	No	No
[45]	Capacity level: multi-objective NLP; Operation level: MILP	Capacity level: ε-constraint multi-objective optimization algorithm; Operation level: commercial solver; Interactive iteration between two levels	No	No
[46]	Capacity level: NLP; Operation level: MINLP	Transformed into a single-level MILP and solved by a commercial solver	No	Yes, only battery
[47]	Capacity level: NLP; Operation level: MINLP	Transformed into a single-level MILP and solved by a commercial solver	No	No

.

In general, existing studies typically model the co-optimization of HESS capacity and operation as a bi-level or two-stage optimization model. Specifically, the capacity level focuses on capacity configuration optimization, with objectives usually being the minimization of the system’s life-cycle investment cost or comprehensive economic indicators; the operation level emphasizes operation optimization, targeting the minimization of the system’s daily operating cost while maintaining reliability. In terms of modeling, the capacity configuration problem is often formulated as a nonlinear programming model (NLP); due to the need to consider energy storage charging/discharging logic and energy balance constraints, the operation optimization problem is usually modeled as a mixed-integer linear programming (MILP), mixed-integer nonlinear programming (MINLP), or dynamic programming model (DP).

From the perspective of the solution methods of the two-layer optimization model, there are mainly two categories: (1) the two-layer interactive iterative solution method [39,40,41,42,43,44,45]; (2) the method of converting the two-layer model into a single-layer MILP model and then directly using a commercial solver to solve it [46,47]: (1) bi-level interactive iterative solution methods [39,40,41,42,43,44,45]; (2) methods that convert the bi-level model into a single-level MILP model and then solve it directly using commercial solvers [46,47]. Among them, the former is more widely applied. The main reason is that the interactive iterative solution method can better handle the nonlinear characteristics and cross-time-scale coupling problems existing in HESS. In contrast, although the bi-level to single-level method can significantly reduce computational complexity, it often requires introducing simplifying assumptions during the model conversion process, which easily ignores nonlinear coupling relationships and leads to a decline in numerical stability and robustness.

For the capacity layer, existing studies mostly adopt heuristic algorithms (such as PSO, GA, GWO, etc.) [40,41,42,43] to address the nonlinear and non-convex characteristics of the upper-level model; some studies use surrogate model-based space reduction algorithms to reduce computational complexity [39]; other scholars employ deterministic mathematical programming methods (such as sequential quadratic programming algorithms [44], ε-constraint multi-objective optimization algorithms [45]) for solution. In general, deterministic mathematical programming methods can obtain high-precision solutions but are sensitive to initial values and time-consuming; heuristic algorithms have relatively faster solution speeds but often yield suboptimal solutions in complex non-convex spaces; surrogate model methods feature high computational efficiency but their accuracy depends on the quality of approximation. The trade-off between solution accuracy and computational efficiency remains prominent.

For the operation layer, dynamic programming (DP) algorithms [39,40,41] or commercial solvers (such as CPLEX, Gurobi) [42,43,44,45] are commonly used for operation optimization, and the solution methods are relatively mature.

In addition, the volatility of renewable energy (e.g., photovoltaic and wind power) and the uncertainty of load demand result in high uncertainty in system operation [48,49]. Constrained by the model structure, the aforementioned methods usually perform optimization under a single typical scenario, making it difficult to fully reflect the uncertainties of renewable energy output and load fluctuations under multi-scenario conditions.

In recent years, the emerging deep reinforcement learning (DRL) method can learn optimal strategies through multi-scenario training in an interactive environment, thereby capturing complex dynamic characteristics and various uncertainties. It exhibits stronger generalization ability and robustness when dealing with complex future stochastic environments. Meanwhile, once the DRL model is trained, its inference and solution speed is much faster than other methods, featuring significant advantages in real-time decision-making [50].

At the same time, the degradation characteristics of energy storage systems (especially non-battery energy storage) have an important impact on both capacity planning and operation decisions, but current research has relatively insufficient consideration of this issue.

To address the above issues, this paper constructs a hybrid energy storage system (HESS) integrating a battery energy storage system (BESS), electrolyzer (EL), fuel cell (FC), and hydrogen storage tank (HST), and applies it to a standalone hybrid renewable energy system to improve the reliability and economy of power supply. On this basis, a bi-level collaborative optimization method combining deep reinforcement learning and mixed-integer programming is proposed as follows: the outer layer (RL) adaptively learns capacity configuration strategies through interaction with the environment to cope with complex, variable, and uncertain operating environments; the inner layer (MIP) achieves precise operation scheduling and constraint satisfaction under given capacity conditions, thereby fully leveraging the complementary advantages of the two methods. In addition, the degradation characteristics of battery and hydrogen energy storage are explicitly incorporated into the model to further enhance the accuracy of optimization results and engineering applicability.

The main contributions of this paper are as follows:

(1)

A novel bi-level collaborative optimization model for capacity and operation is proposed. In the capacity optimization layer, the model incorporates an adaptive mechanism for net load fluctuations into capacity boundary calculation; in the operation optimization layer, a dynamic operation model considering the degradation processes of battery and hydrogen energy storage is established to improve the model’s accuracy and engineering applicability.

(2)

A novel hybrid solution algorithm that combines RL and mixed-integer programming (MIP) is proposed for the bi-level optimization model. The proposed approach leverages the adaptive learning and environment-perception capabilities of RL to dynamically respond to complex and uncertain scenarios, while employing MIP to ensure accurate optimal operation and strict constraint satisfaction. This hybrid algorithm achieves near-optimal performance with significantly reduced computational time.

(3)

The proposed model is validated through multi-scenario simulations and sensitivity analyses, demonstrating its robustness and generalization capability. The results show that the proposed method maintains stable optimization performance under varying operating conditions and system parameters.

The remainder of this paper is organized as follows. Section 2 describes the configuration and mathematical model of the proposed HESS integrated with hybrid renewable energy systems. Section 3 presents the bi-level collaborative optimization framework that combines deep reinforcement learning (DRL) and mixed-integer programming (MIP). Section 4 discusses the case studies and sensitivity analyses used to validate the proposed method. Finally, Section 5 concludes the paper and outlines future research directions.

2. System Model

The hybrid energy storage system considered in this paper comprises two subsystems, a lithium-ion BESS and a hydrogen energy storage system. The hydrogen energy storage system consists of an EL, a hydrogen storage tank (HST), and an FC, collectively providing power balancing and energy management services for a standalone hybrid renewable energy system, as shown in Figure 1. The system is assumed to operate in a standalone configuration under normal conditions, without considering failure contingencies.

The working principle is as follows. Surplus electricity is converted into hydrogen by the electrolyzer and stored in the hydrogen tank; when electricity is required, the fuel cell reconverts the stored hydrogen into power. The lithium battery primarily provides fast, short-term power response, while the two subsystems operate in a complementary manner to achieve multi-timescale energy management.

To achieve the co-optimization of capacity configuration and operational strategy for the lithium-hydrogen HESS, a bi-level optimization model is developed. The core idea is to decompose the comprehensive optimization problem into two interrelated but hierarchical sub-problems. The outer layer deals with capacity configuration (long-term investment decision), and the inner layer addresses operation optimization (short-term dispatch decision). The outer layer determines the optimal capacities of each storage component and provides boundary constraints to the inner layer; the inner layer, in turn, derives the optimal operational strategy under the given capacity limits and feeds back its objective value to the outer layer as the economic performance metric for investment decisions. The detailed formulation of the bi-level optimization model is presented below.

2.1. Inner-Layer Operation Optimization

This layer primarily addresses the optimal operating strategy of the HESS in typical daily scenarios, subject to the capacity configuration parameters determined by the outer layer. The decision variables include the electrolyzer power, fuel cell power, and battery charging/discharging power, defined as $P_{ele}(t)$, $P_{hfc}(t)$, $P_{bat.\mathrm{c}}(t)$ and $P_{bat.\mathrm{d}}(t)$, respectively. The objective function and constraints of the inner layer are formulated as follows.

2.1.1. Objective Function

The objective of the inner-layer optimization is to minimize the weighted sum of the daily operating cost and power deviation, thereby achieving coordinated optimization of the system’s economic efficiency and reliability.

$$O_{inner}=\min (C_{op}+PD)$$

(1)

where $C_{op}$ denotes the total daily operation and maintenance cost of each component, as calculated by Equation (2). It consists of three parts: the degradation cost of the electrolyzer, the degradation cost of the fuel cell, and the aging cost associated with battery charging and discharging.

$$C_{op}={\displaystyle \sum _{t=1}^T(D_{ele}{P}_{ele}(t)+D_{hfc}{P}_{hfc}(t)+D_{bat}(t)(P_{bat.c}(t)+P_{bat.d}(t))})\times \Delta t$$

(2)

where T represents the length of the optimization time horizon (typically set to 24 h), and the calculation formula for the degradation cost coefficient for the battery is given in Equation (3).

$$D_{bat}(t)={\displaystyle \frac{C_{inv.bat}-R{S}_{bat}}{2DoD(t)ACC(t)\sqrt{{\eta }_{bat.c}{\eta }_{bat.d}}}}$$

(3)

where $C_{inv.bat}$ denotes the battery investment cost, $R{S}_{bat}$ represents the salvage value, $DoD(t)$ denotes the depth of discharge (DoD), $ACC(t)$ denotes the accelerated aging coefficient of battery, ${\eta }_{bat.c}$ and ${\eta }_{bat.d}$ represent the charging and discharging efficiencies of battery, respectively, a and b are the parameters of the battery aging model, with values a = 1.8 × 10⁻³ and b = 2, based on experimental references.

$$DoD(t)=1-SoC(t)$$

(4)

$$ACC(t)={\displaystyle \frac{a}{{\left(DoD(t)\right)}^b}}$$

(5)

The calculation formula for the degradation cost coefficient of electrolyzer $D_{ele}$ is given in Equation (6), where $f_{ele}$ denotes the ratio of maintenance and replacement costs to the investment cost of the electrolyzer, $C_{inv.ele}$ represents the unit power investment cost of the electrolyzer, and $H_{ele}$ denotes the designed service life of the electrolyzer.

$$D_{ele}={\displaystyle \frac{f_{ele}{C}_{inv.ele}}{H_{ele}}}$$

(6)

The calculation formula for the degradation cost coefficient of fuel cell $D_{hfc}$ in Equation (7), where $f_{hfc}$ denotes the ratio of maintenance and replacement cost to the investment cost of the fuel cell, $C_{inv.hfc}$ represents the unit power investment cost of the fuel cell, and $H_{hfc}$ denotes the designed service life of the fuel cell.

$$D_{hfc}={\displaystyle \frac{f_{hfc}{C}_{inv.hfc}}{H_{hfc}}}$$

(7)

$PD$ denotes the power deviation, which is used to evaluate the supply–demand balancing capability of the HESS. It is typically required that PD ≤ 1–2%, and its calculation formula is given as follows:

$$PD={\displaystyle \frac{{\displaystyle \sum _{t=1}^T}|P_{net}(t)-(P_{ele}(t)+P_{hfc}(t)+P_{bat.c}(t)+P_{bat.d}(t))|}{{\displaystyle \sum _{t=1}^{T_2}}|P_{net}(t)|}}$$

(8)

where $P_{net}(t)$ denotes the net load power at time t, calculated as the load demand minus the renewable energy power output, that a positive value indicates a power deficit, whereas a negative value indicates a power surplus.

2.1.2. Constraints

Hydrogen Energy Storage System Constraints

(1)

Power operation constraints.

$$0\le P_{ele}(t)\le u_{ele}(t)\cdot P_{ele}^{rated}$$

(9)

$$0\le P_{hfc}(t)\le u_{hfc}(t)\cdot P_{hfc}^{rated}$$

(10)

(2)

Mutual-exclusion constraint (to prevent the electrolyzer and fuel cell from operating simultaneously).

$$u_{ele}(t)+u_{hfc}(t)\le 1$$

(11)

(3)

Hydrogen storage tank dynamic balance equation.

$$SO{H}_{HT}(t)=SO{H}_{HT}(t-1)+({\displaystyle \frac{G_{ele}(t){\eta }_s}{Q_{HT}}}-{\displaystyle \frac{G_{hfc}(t)}{{\eta }_c{Q}_{HT}}})\cdot \mathsf{\Delta }t$$

(12)

(4)

Hydrogen storage tank state constraints.

$$SO{H}_{HT}^{\min }\le SO{H}_{HT}(t)\le SO{H}_{HT}^{\max }$$

(13)

where the hydrogen production rate and hydrogen consumption rate are given as,

$$G_{ele}(t)={\displaystyle \frac{{\eta }_{ele}{P}_{ele}(t)}{E_{H_2}}}$$

(14)

$$G_{hfc}(t)={\displaystyle \frac{P_{hfc}(t)}{{\eta }_{hfc}{E}_{H_2}}}$$

(15)

(5)

To ensure the feasibility and stability of the system during multi-day continuous operation, periodic constraints are imposed.

$$|SO{H}_{T-1}-SO{H}_0|\le ϵ$$

(16)

(6)

To prevent frequent start–stop cycling of the electrolyzer, which may accelerate its degradation, a start–stop operation constraint is imposed.

$$\sum _{t=0}^{T-1}(startu{p}_{ele,t}+shutdow{n}_{ele,t})\le N_{switch,max}$$

(17)

where the start–stop indicator variable is defined as,

$$startu{p}_{ele,t}\ge u_{ele,t}-u_{ele,t-1}$$

(18)

$$shutdow{n}_{ele,t}\ge u_{ele,t-1}-u_{ele,t}$$

(19)

where $u_{ele}(t)$ and $u_{hfc}(t)$ are the binary variables indicating whether the electrolyzer and the fuel cell are operating, respectively; they are equal to 1 when operating and 0 when shut down. $SO{H}_{HT}$ denotes the state of the hydrogen storage tank; ${\eta }_{ele}$ is the electricity-to-hydrogen conversion efficiency of the electrolyzer; ${\eta }_{hfc}$ is the hydrogen-to-electricity conversion efficiency of the fuel cell; ${\eta }_s$ represents the hydrogen compression storage efficiency, and ${\eta }_c$ denotes the hydrogen decompression efficiency, representing the energy transfer efficiency during the release process from the hydrogen tank to the fuel cell; $E_{H_2}$ is the lower heating value (LHV) of hydrogen (kWh/kg); the unit of $Q_{HT}$ is kilograms (kg); $ϵ$ represents the difference between the starting state and the ending state, typically 10%; and $N_{switch,max}$ denotes the total number of start–stop cycles.

Battery System Constraints

(1)

Power operation constraints

$$0\le P_{bat.\mathrm{c}}(t)\le u_{bat.\mathrm{c}}(t)\cdot P_{bat}^{rated}$$

(20)

$$0\le P_{bat.\mathrm{d}}(t)\le u_{bat.\mathrm{d}}(t)\cdot P_{bat}^{rated}$$

(21)

(2)

Charging/discharging mutual-exclusion constraint (to prevent simultaneous charging and discharging)

$$u_{bat.c}(t)+u_{bat.d}(t)\le 1$$

(22)

(3)

Battery state balance equation

$$So{C}_{bat}(t)=So{C}_{bat}(t-1)+{\displaystyle \frac{{\eta }_{bat.c}{P}_{bat.c}(t)\Delta t}{C_{bat}}}-{\displaystyle \frac{P_{bat.d}(t)\Delta t}{{\eta }_{bat.d}{C}_{bat}}}$$

(23)

(4)

Battery state upper and lower bound constraints

$$So{C}_{bat}^{\min }\le So{C}_{bat}(t)\le So{C}_{bat}^{\max }$$

(24)

To prevent overdraw of the energy storage system (e.g., full discharge or overcharge of the battery) caused by single-day optimization and to ensure the feasibility and stability of the system during multi-day continuous operation, a periodic constraint is imposed.

$$|So{C}_{T-1}-So{C}_0|\le ϵ$$

(25)

where $u_{bat.c}(t)$ and $u_{bat.d}(t)$ are the binary variables representing the charging and discharging states of the battery, respectively; $So{C}_{bat}$ denotes the state of charge (SoC) of the battery; ${\eta }_{bat.c}$ and ${\eta }_{bat.d}$ represent the charging efficiency and discharging efficiency of the battery, respectively.

System Power Balance Equation

$$P_{hfc,t}+P_{bd,t}-P_{ele,t}-P_{bc,t}+\xi =P_{net,t}$$

(26)

where $\xi$ denotes the deficit or surplus power.

2.2. Outer-Layer Capacity Optimization

The outer-layer optimization aims to determine the optimal rated power and capacity configuration for each HESS component, representing a long-term planning decision focused on the economic performance of the system over its entire life cycle. The decision variables include the rated power of the electrolyzer, the rated power of the fuel cell, the capacity of the hydrogen storage tank, the rated power of the battery, and the rated energy capacity of the battery, defined as $P_{ele}^{rated}$, $P_{hfc}^{rated}$, $Q_{HT}$, $P_{bat}^{rated}$ and $C_{bat}$, respectively.

The objective function and constraints of the outer layer are formulated as follows.

2.2.1. Objective Function

The objective of outer-layer optimization is to minimize the total system cost, comprising the daily annualized investment cost and the daily operating cost, while satisfying the system technical constraints and operational reliability requirements.

$$C^{total}=\min ({\displaystyle \frac{1}{365}}\times {\displaystyle \frac{i{(1+i)}^{N_l}}{{(1+i)}^{N_l}-1}}\times INV+C_{op})$$

(27)

$$INV=C_{inv.ele}{P}_{ele}^{rated}+C_{inv.hfc}{P}_{hfc}^{rated}+C_{inv.HT}{Q}_{HT}+C_{inv.bat\_p}{P}_{bat}^{rated}+C_{inv.bat\_c}{C}_{bat}$$

(28)

where $C^{total}$ denotes the minimized daily total cost of the system. INV denotes the investment cost, with its calculation given by Equation (28); $N_l$ denotes the technical lifetime (years), i denotes the discount rate, and $C_{op}$ denotes the minimum operating cost obtained from the inner-layer optimization. $C_{inv}$ denotes the unit investment cost per unit power/capacity for each component.

2.2.2. Constraints

(1)

System reliability constraints

$$PD\le P{D}_{threshold}$$

(29)

(2)

Capacity configuration boundary constraints

$$P_{ele}^{rated}\in (0,|P_{net}^{min}|\times \lambda ]$$

(30)

$$P_{hfc}^{rated}\in (0,P_{net}^{max}\times \lambda ]$$

(31)

$$Q_{HT}\in (0,\max (E_{surplus},E_{deficit})/E_{H_2}\times \lambda ]$$

(32)

$$P_{bat}^{rated}\in (0,\max (P_{net}^{max},\left|P_{net}^{min}\right|)\times \lambda ]$$

(33)

$$C_{bat}\in (0,E_{continuous}^{max}\times \lambda ]$$

(34)

where λ denotes the margin factor. Among the characteristic parameters, $P_{net}^{min}$ denotes the minimum of the net load, $P_{net}^{max}$ denotes the maximum of the net load, $E_{surplus}$ denotes the surplus energy, $E_{deficit}$ denotes the deficit energy, and $E_{continuous}^{max}$ denotes the maximum continuous energy requirement of the system. The corresponding formulas are given in Equations (35)–(39).

$$P_{net}^{min}=\underset{t}{\min }\{P_{net,t}:P_{net,t}<0\}$$

(35)

$$P_{net}^{max}=\underset{t}{\max }\{P_{net,t}:P_{net,t}>0\}$$

(36)

$$E_{deficit}=\sum _{t:P_{net,t}>0}{P}_{net,t}\mathsf{\Delta }t$$

(37)

$$E_{surplus}=\sum _{t:P_{net,t}<0}|P_{net,t}|\mathsf{\Delta }t$$

(38)

$$E_{continuous}^{max}=\max (\underset{t}{\max }\sum _{i=0}^k{P}_{net,t+i}^{+},\underset{t}{\max }\sum _{i=0}^k|P_{net,t+i}^{-}|)$$

(39)

(3)

Battery charging/discharging duration constraints

$${\displaystyle \frac{C_{bat}}{P_{bat}^{rated}}}\ge {\tau }_{min}$$

(40)

where ${\tau }_{min}$ denotes the required minimum charging/discharging duration of the battery system.

3. A Cooperative DRL–MIP Framework for HESS Capacity Configuration and Operation Optimization

To solve the bi-level co-optimization model proposed in Section 2, this study develops a cooperative algorithm that integrates DRL with MIP. The outer layer employs a DRL method, exemplified by the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for adaptive exploration of capacity configuration, while the inner layer formulates the operational optimization as an MIP problem and solves it exactly using Gurobi. In this way, efficient co-optimization of capacity configuration and operation optimization is achieved.

3.1. Collaborative Optimization Mechanism

3.1.1. Outer Layer Design

The HESS capacity configuration problem can be modeled as a Markov Decision Process (MDP), defined by a five-tuple $M=\left(S,A,P,R,\gamma \right)$. Here, S denotes the state space, represented by an 18-dimensional vector comprising load characteristics, the current capacity configuration, and historical optimization information; A denotes the action space, consisting of five continuous adjustment variables for capacity configuration; $P:S\times A\times S\to [0,1]$ denotes the state transition probability, jointly determined by the capacity update rules and the outcomes of the inner-layer optimization; $R:S\times A\to R$ denotes the reward function, constructed based on cost improvements and constraint satisfaction; and $\gamma \in \left[\begin{array}{c}0,1\end{array}\right]$ denotes the discount factor, which balances immediate rewards and long-term returns.

Within this MDP framework, the agent incrementally optimizes the capacity configuration through sequential decisions. At each decision step, the agent observes the current state $s_t\in S$, selects an action $a_t\in A$ to adjust the capacities, and the environment executes the inner-layer optimization based on the updated configuration, returning the reward $r_t$ and the next state $s_{t+1}$. The state-transition process satisfies the Markov property $P\left(s_{t+1}|s_0,a_0,\dots ,s_t,a_t\right)=P\left(s_{t+1}|s_t,a_t\right)$, ensuring that the current decision depends only on the current state, without requiring knowledge of the entire historical trajectory.

3.1.2. Inner Layer Design

With the operating cost and power-balance deviation as the optimization objectives, a constrained model is formulated, and its results are fed back to the outer layer as the reward signal, thereby forming a closed-loop optimization mechanism.

The overall algorithmic workflow is illustrated in Figure 2 and outlined as follows:

Step 1: Algorithm initialization. Configure the network architecture and hyperparameters, including the TD3 framework, training parameters, and the experience replay buffer, and then enter the outer-layer TD3 decision loop.

Step 2: Environment reset and adaptive boundary computation. At the beginning of each episode, reset the environment and compute adaptive bounds based on the typical daily net-load profile to provide reasonable constraint ranges for capacity configuration.

Step 3: Action selection and configuration update. Select actions using the TD3 networks and update the capacity configuration.

Step 4: Inner-layer scheduling optimization. Given the current capacities, use Gurobi to solve a mixed-integer programming model that minimizes the operating cost and power-balance deviation, thereby yielding the optimal operating strategy.

Step 5: Reward calculation and configuration update. If the current solution outperforms the historical best in terms of cost and power-balance constraints, update the best configuration; otherwise, keep it unchanged.

Step 6: Network parameter updates. Store experience samples in the replay buffer and train the TD3 networks by sampling mini-batches, including value-function updates for the critic network and policy-gradient optimization for the actor network.

Step 7: Convergence checks and output. When the number of steps in an episode reaches the preset limit, terminate the episode and output the optimal capacity configuration and operational strategy.

Step 8: Iterative loop. Repeat the above process until the training converges or the preset maximum number of episodes is reached.

This cooperative solution algorithm integrates the exploratory capability of reinforcement learning with the exactness of mathematical programming, achieving coordinated optimality of HESS capacity configuration and operating strategy under strict technical constraints. Its core modules include the following:

• State and action space design

The state space comprises statistical features of the net-load profile (10 dimensions), the normalized values of the current optimal configuration (five dimensions), and historical information features (three dimensions).

$$\mathit{s}=[{\mathit{f}}_{profile},{\mathit{x}}_{normalized},{\mathit{h}}_{history}]\in {ℝ}^{18}$$

(41)

where ${\mathit{f}}_{profile}$ denotes the feature vector of the net-load profile (normalized values of the mean, standard deviation, maximum, minimum, number of deficit periods, number of surplus periods, total deficit energy, total surplus energy, longest consecutive deficit duration, and longest consecutive surplus duration); ${\mathit{x}}_{normalized}$ denotes the normalized value of the current best capacity configuration; and ${\mathit{h}}_{history}$ denotes the normalized value of the historical information features (best daily total cost, PD value, and minimum boundary distance).

The action space is a five-dimensional continuous space, corresponding to the adjustment increments of the capacity configuration: $\Delta P_{ele}^{rated}$, $\Delta P_{hfc}^{rated}$, $\Delta Q_{HT}$, $\Delta P_{bat}^{rated}$ and $\Delta C_{bat}$.

$$\mathit{a}=[a_1,a_2,a_3,a_4,a_5],a_i\in [-1,1]$$

(42)

The update formula for the capacity configuration is,

$$\mathit{s}=[{\mathit{f}}_{profile},{\mathit{x}}_{normalized},{\mathit{h}}_{history}]\in {ℝ}^{18}$$

(43)

2.

Reward function design

The reward function is constructed based on the magnitude of cost improvement and the power-balance constraints. Equations (44) and (45), respectively, quantify the degree of cost improvement or deterioration of the current solution relative to the previous best solution, whereas Equation (46) evaluates whether the current power-balance deviation exceeds the specified threshold.

$$\rho ={\displaystyle \frac{\max (0,C_{best}^{prev}-C_{current})}{C_{best}^{prev}+{ϵ}_1}}$$

(44)

$${\rho }_{neg}={\displaystyle \frac{\max (0,C_{current}-C_{best}^{prev})}{C_{best}^{prev}+{ϵ}_1}}$$

(45)

$${\rho }_{neg}={\displaystyle \frac{\max (0,C_{current}-C_{best}^{prev})}{C_{best}^{prev}+{ϵ}_1}}$$

(46)

When the optimal cost decreases and the power deviation does not exceed the threshold, the reward is computed by Equation (47). If the cost decreases but the power deviation exceeds the threshold, it is calculated by Equation (48). Otherwise, a negative reward is assigned, as given in Equation (49).

$$R=R_{bonus}+\alpha \cdot \min \left({\left(\max \left({\displaystyle \frac{\rho }{s_c}},1.0\right)\right)}^p,R_{cap}\right)$$

(47)

$$R=-\beta \cdot v$$

(48)

$$R=-\gamma (1-e^{-{\rho }_{neg}/s_w})-\beta \cdot v$$

(49)

where $P{D}_{threshold}$ denotes the power deviation threshold; ${ϵ}_1$ denotes a numerical stability term; $R_{bonus}$ denotes the base improvement reward; $\alpha$ denotes the improvement amplification coefficient; $p$ denotes the superlinear exponent; $R_{cap}$ denotes the reward cap; $\gamma$ denotes the penalty coefficient for no improvement; $\beta$ denotes the penalty coefficient for PD violation; $s_c$ and $s_w$ denote the normalization scale parameters.

3.

Network architecture and training strategy

(1)

Network architecture

The Actor network adopts a multilayer fully connected architecture, incorporating LayerNorm and Dropout to enhance generalization. The Critic network also uses a multilayer structure, with the state–action pair as its input.

(2)

TD3 core strategy

Find the optimal actor policy ${\pi }_{\theta }$ that minimizes the expected total system cost $C^{total}(s,a)$, where θ denotes the parameters of the actor network [51].

$$\min {\mathbb{E}}_{s,a\sim {\pi }_{\theta }}[C^{total}(s,a)]$$

(50)

(3)

Network update mechanism

The actor network adopts a multilayer fully connected architecture with LayerNorm and Dropout to enhance generalization. The Critic network also uses a multilayer architecture, taking the state–action pair as input.

$${\mathit{\theta }}_{\mu }\leftarrow {\mathit{\theta }}_{\mu }+{\alpha }_{\mu }{\nabla }_{{\mathit{\theta }}_{\mu }}\mathbb{E}[Q_i(s,{\mu }_{{\mathit{\theta }}_{\mu }}(s))]$$

(51)

$${\theta }_{Q_i}\leftarrow {\theta }_{Q_i}-{\alpha }_Q{\nabla }_{{\mathit{\theta }}_{Q_i}}{L}_i$$

(52)

where $\mu$ denotes the actor network (policy network), describing how the agent selects actions based on the observed state; ${\theta }_{\mu }$ denotes the parameter set of the policy function; ${\alpha }_{\mu }$ the actor learning rate; ${\mu }_{{\mathit{\theta }}_{\mu }}(s)$ denotes the action output by the actor network under state s, i.e., the policy function; and ${\nabla }_{{\mathit{\theta }}_{\mu }}$ denotes the gradient operator with respect to ${\theta }_{\mu }$; $Q_i$ is the i-th critic network’s Q-function (action–value function); ${\theta }_{Q_i}$ represents the parameters of the critic network; ${\alpha }_Q$ the critic learning rate; ${\nabla }_{{\mathit{\theta }}_{Q_i}}$ denotes the gradient operator with respect to ${\theta }_{Q_i}$; $L_i$ denotes the loss function of the critic network, computed as in Equation (55).

$$L_i=\mathbb{E}[{(Q_i(s_t,a_t)-y_t)}^2]$$

(53)

The target Q-value is $y_t$.

$$y_t=r_t+\gamma \underset{i=1,2}{\min }{Q}_i^{\prime }(s_{t+1},{\mu }^{\prime }(s_{t+1})+\mathrm{noise})$$

(54)

where $r_t$ denotes the immediate reward; $\gamma$ denotes the discount factor; $\underset{i=1,2}{\min }{Q}_i^{\prime }$ denotes the minimum of the two Critic network outputs; ${\mu }^{\prime }(s_{t+1})$ denotes the action generated by the target actor network at the next state $s_{t+1}$; $\mathrm{noise}$ denotes the random noise added to the target action.

(4)

Soft update mechanism

The soft update mechanism ensures smooth iteration of the target network parameters, with the update process given by Equations (55) and (56).

$${\theta }_{Q_i}^{\prime }\leftarrow \tau {\theta }_{Q_i}+(1-\tau ){\mathit{\theta }}_{Q_i}^{\prime }$$

(55)

$${\theta }_{\mu }^{\prime }\leftarrow \tau {\theta }_{\mu }+(1-\tau ){\theta }_{\mu }^{\prime }$$

(56)

where $\tau$ denotes the soft update parameter; ${\theta }_{Q_i}^{\prime }$ denotes the parameters of the i-th target critic network; ${\theta }_{\mu }^{\prime }$ denotes the parameters of the target actor network.

(5)

Prioritized experience replay

To improve learning efficiency, a prioritized experience replay mechanism is employed. The core idea is to assign sampling probabilities based on the gap between the current Q-value prediction and the “true” target value (Temporal Difference Error, TDE). The corresponding formulas are given in Equations (57)–(59).

$$P(i)={\displaystyle \frac{p_i^{{\alpha }_{priority}}}{{\displaystyle \sum _k}{p}_k^{{\alpha }_{priority}}}}$$

(57)

$$p_i=|{\delta }_i^{(1)}+{\delta }_i^{(2)}|/2+{ϵ}_{priority}$$

(58)

$${\delta }_i^{(j)}=Q_j(s_i,a_i)-y_i$$

(59)

where $p_i$ denotes the priority of sample i; ${\alpha }_{priority}$ denotes the priority-sampling hyperparameter that controls the influence of the TDE on the sampling probability; ${ϵ}_{priority}$ denotes the priority stability term; ${\delta }_i^{(j)}$ denotes the temporal-difference error.

An importance-sampling weight $w_i$ is introduced to reweight samples and correct the bias induced by non-uniform sampling, where ${\beta }_{priority}$ denotes the importance-sampling hyperparameter that controls the degree of correction.

$$w_i={\left({\displaystyle \frac{1}{N}}\cdot {\displaystyle \frac{1}{P(i)}}\right)}^{{\beta }_{priority}}/\underset{j}{\max }{w}_j$$

(60)

4. Results and Discussion

This Section aims to validate the proposed TD3–Gurobi cooperative algorithm that integrates deep reinforcement learning with mathematical programming. First, the parameter settings of the case study are introduced. Next, the HESS capacity configuration solution obtained by the algorithm and its operating strategy under a typical day are presented and analyzed. Subsequently, three categories of comparative cases are designed from different validation perspectives. Finally, a sensitivity analysis is conducted to examine the impact of key parameters on system configuration and economic performance, thereby discussing the model’s applicability and robustness.

4.1. Case Setting

The optimization object of this study is a hybrid renewable energy system comprising photovoltaics (PV), wind power, local load, and a lithium-hydrogen HESS. The daily curves of PV output, wind output, and load are shown in Figure 3, Figure 4 and Figure 5. System optimization is conducted based on annual net-load data. Different colored lines represent different curves. To more intuitively present load characteristics and to select representative scenarios for validating the algorithm, the annual net-load profiles are clustered by renewable energy penetration into high, medium, and low scenarios, as shown in Figure 6. The main technical and economic parameters of the HESS are summarized in

Table 2. The main economic and technical parameters of the HESS.

Component	Economic/Technical Parameter	Value	Unit
EL (Electrolyzer)	$C_{inv.ele}$	786	USD/kWh
	${\eta }_{ele}$	0.8	- 1
	$f_{ele}$	0.4	-
	$H_{ele}$	70,000	Hour
HFC (Fuel Cell)	$C_{inv.hfc}$	286	USD/kW
	${\eta }_{hfc}$	0.6	-
	$f_{hfc}$	0.3	-
	$H_{hfc}$	30,000	Hour
HT (Hydrogen Tank)	$C_{inv.HT}$	1143	USD/kg
	$SO{H}_{HT}^{\min }$	0	-
	$SO{H}_{HT}^{\max }$	1	-
	${\eta }_s$	0.97	-
	${\eta }_c$	0.98	-
Battery	$C_{inv.bat\_p}$	429	USD/kW
	$C_{inv.bat\_c}$	357	USD/kWh
	${\eta }_{bat.c}$	0.98	-
	${\eta }_{bat.d}$	0.98	-
	$So{C}_{bat}^{\min }$	0.1	-
	$So{C}_{bat}^{\max }$	0.9	-
	$R{S}_{bat}$	10	USD/kWh
HESS (Hybrid Energy Storage System)	$N_l$	20	Year
	i	0.08	-
Other	$E_{H_2}$	33.33	kWh/kg

¹ The symbol “-” indicates no unit.

.

4.2. Algorithmic Solution and Results Analysis

The computing platform used for the solution is configured as follows: an Intel Core i9-14900K processor (24 cores, 32 threads), 128 GB RAM, and an NVIDIA RTX 4090 24 GB discrete GPU.

A total of 1500 daily net-load scenarios, derived from a standard dataset, were employed to train the RL agent. During the testing phase, for clarity of presentation, the net-load profiles were classified into three representative categories (high-, medium-, and low-penetration) using the k-means clustering method (shown in Figure 6). One typical scenario from each category was then selected for detailed analysis.

The training curves of the DRL–Gurobi cooperative algorithm are shown in Figure 7, demonstrating stable convergence. The parameters N (maximum number of training episodes) and N_step (maximum decision steps per episode), as illustrated in Figure 2, were empirically determined through convergence tests to balance training stability and computational efficiency. Specifically, N = 1500 episodes ensures convergence of the TD3 learning curve, while N_step = 150 steps per episode provides sufficient learning opportunities for sequential capacity adjustment decisions in the outer-layer optimization.

Using the proposed cooperative approach, the optimal HESS configuration is obtained. As shown in

Table 3. The optimal configuration results of HESS based on the proposed method.

Decision Variable	Optimized Result	Unit
$P_{ele}^{rated}$	312.23	kW
$P_{hfc}^{rated}$	173.26	kW
$Q_{HT}$	225.90	kg
$P_{bat}^{rated}$	71.60	kW
$C_{bat}$	174.68	kWh
Minimum daily total cost	209.10	USD

, under the condition that the power deviation does not exceed 0.01, the minimum daily total cost is USD 209.10.

To further illustrate the internal operation mechanism of the system, Figure 8 presents the coordinated operation strategy of each HESS component under a representative net-load profile (high-penetration scenario).

Figure 8a illustrates the power profile over the typical day, capturing the dynamic relationships among the electrolyzer, fuel cell, battery charging/discharging, and net load. Figure 8b depicts the evolution of the battery state of charge (SoC) and the state of hydrogen storage tank (SoH).

At night (00:00–06:00 and 18:00–24:00), the net load remains negative. During these periods, the electrolyzer operates at a large scale, converting surplus electricity into hydrogen for storage; battery charging mainly occurs during the low-demand hours, cooperating with the electrolyzer to absorb excess energy.

During the daytime (09:00–12:00 and 15:00–18:00), the net load is markedly positive. The fuel cell and battery discharge jointly support power supply, performing peak shaving and valley filling. The battery SoC shows obvious charge–discharge cycles: charging in the early morning and at night, and rapid discharging during daytime load peaks. The SoC peaks at about 90% and falls to as low as 10%, indicating that the battery undertakes fast and deep regulation tasks.

The SoH exhibits relatively small variations with a gradual rise–fall pattern: it increases at night due to hydrogen production by the electrolyzer and declines during the day as the fuel cell consumes hydrogen for power generation. The changes are smooth, highlighting the long-cycle energy shifting characteristic of hydrogen storage. The battery and hydrogen storage are complementary across time scales; the battery primarily addresses short-term, rapid fluctuations, while hydrogen storage achieves inter-period energy balancing.

4.3. Comparative Analysis

To comprehensively validate the effectiveness of the proposed TD3–Gurobi cooperative optimization framework, three comparative cases are designed from different validation perspectives as follows:

Case 1 (Proposed method): TD3-based deep reinforcement learning in the outer layer and the Gurobi mixed-integer programming solver in the inner layer.

Case 2 (Comparative algorithm verification): The outer layer uses the standard genetic algorithm (GA) and particle swarm optimization algorithm (PSO) combined with the inner layer Gurobi solver. And Gurobi is set to directly solve the two-level optimization problem for comparison. GA and PSO are selected to examine whether the proposed DRL method outperforms mature metaheuristics on high-dimensional continuous decision problems, in terms of solution quality (achieving lower cost) and convergence efficiency (shorter computation time). Although mathematical programming methods can theoretically obtain global optimal solutions, they often face the problem of excessively long solution times or even failure to converge. Therefore, a comparison with traditional deterministic algorithms (Gurobi solvers) aims to demonstrate whether the proposed algorithm can significantly improve computational efficiency while ensuring solution quality.

Case 3 (System architecture validation): Configure a single-storage system only (either battery storage or hydrogen storage) and solve it using the same optimization method as in Case 1. This comparison is designed to verify the advantages of the hybrid energy storage system over single-storage technologies in terms of economic performance and technical efficacy.

The performance of each scheme on key economic and technical indicators is summarized in

Table 4. Performance comparison of different optimization methods and energy storage configurations.

Cases		${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)	Computation Time (s)
Case 1	DRL + Gurobi	312.23	173.26	225.90	71.60	174.68	209.10	1.3
Case 2	GA + Gurobi	262.88	93.19	112.77	133.42	460.72	219.34	250
	PSO + Gurobi	279.96	115.35	146.30	104.32	353.56	211.87	225
	Gurobi	309.16	173.33	220.43	74.39	186.89	208.73	1800
Case 3	Battery-only	-	-	-	383.57	3297.74	473.35	-
	Hydrogen-only	383.57	238.93	76.25	-	-	140.19	-

.

The comparative analysis indicates substantial differences in economic performance and computational efficiency across optimization methods and storage configurations.

Case 1 (TD3 + Gurobi) achieves the most balanced performance, with a total cost of only USD 209.10 and a computation time of just 1.3 s.

In Case 2, the total costs of GA + G and PSO + G are 4.90% and 1.32% higher than Case 1, respectively, and their computation times are 250 s and 225 s, approximately 192× and 173× that of Case 1. This indicates that metaheuristic algorithms converge more slowly and have limited global search capability for complex optimization problems. Although the Gurobi approach attains the lowest total cost, its computation time is about 1385× that of Case 1, implying that while it can theoretically approach the global optimum, its efficiency is too low for complex, non-convex, high-dimensional problems and thus fails to meet rapid decision-making requirements.

Case 3 further validates the structural advantages of the hybrid storage system. Although the battery storage scheme provides fast response, it requires an oversized capacity (3297.74 kWh) to satisfy energy balance, resulting in a total cost of USD 473.35, which is substantially higher than that of the hybrid system. The hydrogen energy storage solution has the lowest cost (USD 140.19), but its response rate cannot meet the system flexibility requirements. In contrast, the hybrid energy storage system achieves complementary capacity configuration between the battery and hydrogen storage, optimizes the balance of power and energy, and simultaneously delivers economy, flexibility, and operational stability.

4.4. Sensitivity Analysis

4.4.1. Sensitivity Analysis of Key Component Costs

To evaluate the robustness of the proposed method and identify the main cost drivers, this study conducted a sensitivity analysis on the costs of key components, including the electrolyzer power, fuel cell power, hydrogen tank, and the power and energy capacity of lithium battery. The corresponding optimal HESS configurations are shown in

Table 5. The sensitivity analysis of electrolyzer power cost on HESS optimal configuration and minimum daily total cost.

${\mathit{C}}_{\mathit{inv}.\mathit{ele}}$ (USD/kW)	${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)
550	319.07	172.27	232.94	64.53	158.19	186.71
629	274.13	109.31	132.36	109.37	386	198.5
707	307.63	156.85	217.02	75.94	194.1	203.3
786	312.23	173.26	225.9	71.6	174.68	209.1
864	270.75	106.74	122.61	112.82	406.87	218.51
943	266.8	102.38	109.69	116.77	435.96	225.15
1021	255.13	93.96	72.45	128.44	521.15	231.62

,

Table 6. The sensitivity analysis of fuel cell power cost on HESS optimal configuration and minimum daily total cost.

${\mathit{C}}_{\mathit{inv}.\mathit{hfc}}$ (USD/kW)	${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)
200	304.59	167.7	210.58	81.9	208.87	205.98
229	317.57	175.64	230.91	68.55	163.37	207.27
257	317.28	173.78	231.34	68.11	162.31	207.79
286	312.23	173.26	225.9	71.6	174.68	209.1
314	318.06	173.25	231.93	65.71	160.49	210.95
343	280.19	108.37	154.28	103.38	338.23	213.84
371	273.75	109.99	132.11	109.82	385.45	215.13

,

Table 7. The sensitivity analysis of hydrogen storage tank cost on HESS optimal configuration and minimum daily total cost.

${\mathit{C}}_{\mathit{inv}.\mathit{HT}}$ (USD/kg)	${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)
800	334.53	181.63	255.82	49.54	120.08	187.76
914	320.07	175.68	234.14	63.54	155.56	194.59
1029	286.16	115.47	164.88	96.73	327.47	206.44
1143	312.23	173.26	225.9	71.6	174.68	209.1
1257	253.81	102.13	65.9	129.77	534.64	215.21
1371	251.5	90.13	59.94	132.91	548.25	216.69
1486	254.86	95.91	57.52	128.72	553.54	219.14

,

Table 8. The sensitivity analysis of lithium battery power cost on HESS optimal configuration and minimum daily total cost.

${\mathit{C}}_{\mathit{inv}.\mathit{bat}\_\mathit{p}}$ (USD/kW)	${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)
300	305.77	169.31	212.97	80.29	203.19	207.81
343	306.89	171.54	215.44	77.49	197.67	208.86
386	306.34	171.57	215.45	78.51	199.52	209.82
429	312.23	173.26	225.9	71.6	174.68	209.1
471	273.55	125.53	131.58	110.09	386.62	213.42
514	314.97	180.85	259.79	57.69	143.77	216.62
557	278.22	146.66	131.77	107.87	386.84	217.57

and

Table 9. The sensitivity analysis of lithium battery capacity cost on HESS optimal configuration and minimum daily total cost.

${\mathit{C}}_{\mathit{inv}.\mathit{bat}\_\mathit{c}}$ (USD/kWh)	${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)
250	249.66	88.36	53.83	134.04	561.86	179.71
286	244.15	99.73	53.04	146.54	611.26	198.23
321	264.4	99.8	101.19	119.17	455.11	203.01
357	312.23	173.26	225.9	71.6	174.68	209.1
393	285.23	150.21	168.27	98.33	304.06	217.79
429	311.65	168.48	225.19	77.5	177.39	218.42
464	325.33	174.73	241.01	58.71	142.63	220.08

, respectively. The details are as follows:

(1)

Sensitivity to Electrolyzer Power Cost

Variations in the electrolyzer power cost have a pronounced impact on system configuration and total cost. When the cost increases from 550 USD/kW to 1021 USD/kW, the average daily total cost rises from USD 186.71 to USD 231.62 (+24.0%). The hydrogen storage tank capacity decreases from 232.94 kg to 72.45 kg (−68.9%), indicating that the system mitigates higher electrolyzer costs by reducing hydrogen storage capacity. Correspondingly, the battery storage capacity increases from 158.19 kWh to 521.15 kWh (+229.4%), reflecting the substitution effect of battery storage for hydrogen storage. These results indicate that the electrolyzer cost is a key determinant of the economic performance of the hybrid storage system.

(2)

Sensitivity to Fuel Cell Power Cost

Changes in the fuel cell power cost have a relatively moderate impact on the system. When the cost increases from 200 USD/kW to 371 USD/kW, the average daily total cost rises only from USD 205.98 to USD 215.13 (+4.4%).

(3)

Sensitivity to Hydrogen Tank Cost

Changes in the hydrogen tank cost significantly affect the system’s configuration strategy. When the cost increases from 800 USD/kg to 1486 USD/kg, the average daily total cost rises from USD 187.76 to USD 219.14 (+16.7%). The hydrogen storage tank capacity decreases from 255.82 kg to 57.52 kg (−77.5%). To compensate for the reduction in hydrogen storage, the battery storage capacity increases from 120.08 kWh to 553.54 kWh (+361.0%), reflecting a cost-driven technology substitution effect and indicating that hydrogen tank cost is a key economic factor influencing long-term storage technology choices.

(4)

Sensitivity to Lithium Battery Power Cost

Variations in the lithium battery power cost have a relatively small effect on the total system cost. When the cost increases from 300 USD/kW to 557 USD/kW, the average daily total cost rises from USD 207.81 to USD 217.57 (+4.7%).

(5)

Sensitivity to Lithium Battery Energy-Capacity Cost

Changes in the lithium battery energy-capacity cost have a pronounced impact on system configuration. When the cost rises from 250 USD/kWh to 464 USD/kWh, the average daily total cost increases from USD 179.71 to USD 220.08 (+22.5%). When the capacity cost is relatively low (250–321 USD/kWh), the system substantially increases the battery energy capacity (up to 611.26 kWh) while reducing investment in hydrogen storage, reflecting a cost-driven technology substitution effect.

Ranked by the magnitude of total cost variation, the cost sensitivities of the components from highest to lowest are as follows: electrolyzer power cost (24.0%) > lithium battery energy-capacity cost (22.5%) > hydrogen tank cost (16.7%) > lithium battery power cost (4.7%) > fuel cell power cost (4.4%).

The proposed algorithm enables robust configuration adjustments in response to changes in component costs, maintaining overall economic performance through a dynamic balance between battery storage and hydrogen storage. When the cost of one storage technology increases, the system automatically raises the configuration share of the other, demonstrating the technological complementarity of the hybrid energy storage system.

4.4.2. Sensitivity Analysis of Renewable Energy Penetration

Optimization is performed separately under the three typical renewable energy penetration scenarios: high, medium, and low. The optimal HESS configuration is shown in

Table 10. The optimal HESS configuration and performance under different renewable energy.

Renewable Energy Penetration Scenario	${\mathit{P}}_{\mathit{ele}}^{\mathit{rated}}$ (kW)	${\mathit{P}}_{\mathit{hfc}}^{\mathit{rated}}$ (kW)	${\mathit{Q}}_{\mathit{HT}}$ (kg)	${\mathit{P}}_{\mathit{bat}}^{\mathit{rated}}$ (kW)	${\mathit{C}}_{\mathit{bat}}$ (kWh)	${\mathit{C}}^{\mathit{total}}$ (USD)
High	312.23	173.26	225.9	71.6	174.68	209.1
Medium	318.37	212.37	51.83	117.86	372.66	203.48
Low	123.18	290.64	1083.62	69.76	327.83	470.96

. The optimal operation results are shown in Figure 8, Figure 9, and Figure 10, respectively.

It can be observed that under the low-penetration scenario, renewable generation is insufficient; the required electrolyzer capacity is relatively small, but a higher-power fuel cell is needed to supplement electricity and ensure supply stability, while large hydrogen reserves are required to address long-term energy deficits. Under the medium-penetration scenario, the battery undertakes more short-term regulation tasks and the hydrogen demand is relatively low; under the medium-penetration scenario, the battery accounts for a higher share in short-term fluctuation mitigation, and both medium and low penetration scenarios require larger battery capacities to balance energy. In the low-penetration case, maintaining stable system operation also necessitates larger-capacity fuel cells and hydrogen storage, leading to a significant increase in cost.

From Figure 8, Figure 9 and Figure 10, the following can be observed.

Under the high-penetration scenario, the battery SoC exhibits a dual-peak charging pattern, reaching about 90% around 03:00–04:00 and 15:00. Discharging mainly coincides with the load peaks at 10:00–11:00 and 19:00–20:00, with the minimum SoC dropping to around 10%. Under medium penetration, the battery SoC can reach nearly 100% and remain there for an extended period, followed by rapid discharging during peak-load hours to about 25%. Under low penetration, the battery charges overnight to approximately 90% and then remains stable, with a gradual daytime discharge to a minimum of roughly 15%.

For the hydrogen storage tank, under high penetration, SoH slowly rises from about 50% to 65%, reflecting the smooth characteristics of long-duration storage. Under medium penetration, SoH quickly increases to around 100%, remains high for a considerable time, and then gradually declines, indicating higher utilization intensity. Under low penetration, SoH varies most gently, remaining essentially within the 45–52% range, implying relatively low utilization.

The TD3–Gurobi cooperative optimization algorithm yields appropriate optimal configurations across all scenarios, demonstrating its robustness and adaptability. As renewable energy penetration increases, the importance of long-duration storage becomes more pronounced, while battery storage consistently provides indispensable fast regulation in all scenarios.

5. Conclusions

This paper proposes a bi-level cooperative optimization framework that integrates deep reinforcement learning (DRL) with mixed-integer programming (MIP) to jointly optimize the capacity configuration and operating strategy of a hybrid battery-hydrogen energy storage system. Targeting the minimization of average daily total cost and power deviation, the method achieves optimal capacity configuration and operation optimization of all system components. The results demonstrate that the proposed approach significantly outperforms conventional algorithms in both computational efficiency and solution quality, thereby verifying the advantages of hybrid storage over single-technology schemes. These findings demonstrate the feasibility and effectiveness of combining DRL with mathematical programming, providing a valuable reference for optimizing the configuration of complex energy systems.

The theoretical contribution lies in providing an efficient solution paradigm that fuses artificial intelligence with operations research for bi-level optimization problems characterized by strong coupling in energy systems. Future work may incorporate uncertainties in wind/PV and load forecasting using stochastic programming or robust optimization to enhance reliability under complex conditions. In addition, future studies will extend the proposed model to account for system resilience under islanded operation and potential component failures. This can be achieved by integrating probabilistic reliability modeling or fault-tolerant control mechanisms into the DRL–MIP framework, enabling a more comprehensive assessment of system robustness. In parallel, exploring diversified business models, such as the participation of storage in ancillary service markets, could further unlock the economic potential of hybrid energy storage systems.